We notice that some of the pulled metrics have a ton of data, and
during app upgrades, we're forming partial buckets that represent
small periods of time but require many bytes of data. We now have an
option to drop these buckets that are too short. Note that we still
have to pull the data to keep the metrics for the next bucket
correct. We include a new field in the value and gauge metric outputs
so that it's easy to tell when a bucket was dropped.
We drop the partial buckets also from anomaly detection since we
should be computing anomalies from the same data that is reported.
Test: Added unit-tests for value and gauge metrics.
Bug: 77925710
Change-Id: Ic370496377c6afd380e02278a6c1ed8b521a2731
APIs that return package usage data (such as the new StatsManager)
must ensure that callers hold both the PACKAGE_USAGE_STATS permission
and the OP_GET_USAGE_STATS app-op.
Add noteOp() method that can be called from native code.
Also add missing security checks on command interface.
Bug: 77662908, 78121728
Test: builds, boots
Change-Id: Ie0d51e4baaacd9d7d36ba0c587ec91a870b9df17
When statsd reconnects to logd, statsd will read all logs from buffer again. To prevent us from
reprocessing old events, we do the following:
1. At any given moment, record the largest timestamp(T_max) and last timestamp (check point) that
we've seen before.
2. When reconnection happens, we look for the check point until we see a new log with a timestamp
larger than T_max.
-> If we found the CP, resume after the CP. Success
-> If we can't find CP, there is definitely log loss. We reset all configs.
Note:
1. Logd has an API to read logs after a certain timestamp. But this api is vulnerable to
time changes from Settings. So we cannot rely on it.
2. If logd inserts a new log (with older timestamp) before CP, we cannot detect it. It's not
possible to detect it without record all timestamps we have seen.
Test: statsd_test
Bug: 77813113
Change-Id: Ic3fdb47230807606ab11dc994cb162194adb8448
When StatsManager fails to connect to statsd, it now throws an exception
for the caller to catch. It also throws an exception of the config being
added is of an unreadable format.
Due to backwards compatibility issues, the old APIs could not be
changed, so new ones were made to replace the old ones. The old ones are
now temporary and will be removed when the compatibility issue is
resolved.
Bug: 77648233
Test: gts-tradefed run gts-dev --module GtsStatsdHostTestCases
Change-Id: Ibea05883a29b9b3ef9927d2f8fe295eb99832ab7
The previous scheme captured periodic snapshots for each config with
complex logic that's unnecessary and wasted memory. We actually don't
need to store any snapshots since we just convert the current state
into a snapshot and also include the deltas (change events) since the
previous report until now.
To make the system more robust, we also include up to 100 of the
deleted apps in the uid map.
Also, fix the wiring of the partial buckets so the metric producers
form partial buckets on both app upgrade and removal, but not on
installation of a new app.
Also, we update StatsCompanionService to also include disabled apps.
Bug: 77607583
Test: Verified unit-tests pass and added new e2e tests.
Change-Id: I98e1f544d6e6571545ae1581c4cebab807596f51
Roughly check the config every hour to see whether the ttl expired.
If so, read the config from disk and recreate the metric manager.
Test: statsd test
BUG: b/77274363
Change-Id: I16838afe5bbe966c3a0f638869751f9b59a5a259
It's tricky to determine the source of the metrics on a device
currently since we can take the union of multiple configs and send
only one giant statsd_config into statsd. We will use the int64 field
to track the sub config id's and the int32 field to track the version
for each sub config, but the fields are named more generically as
annotations.
The annotations are available in both the reports and metadata.
Test: Check that all unit-tests pass on marlin-eng
Bug: 77327261
Change-Id: Ic37c549c8b2991676f69948c515156765c9f5108
The logic of where refractory period enforcement was moved out of the
anomaly tracker and into the metric's predictAnomalyTimestamp. It was
done for ORING, but not for MAX. This fixes MAX.
Bug: 74446029
Test: adb sync data && adb shell data/nativetest64/statsd_test/statsd_test
Change-Id: I51e43c7c132f424af8fe20a37f2ad10cc55b5989
changes are:
1) for pushed atoms, use attribution node in place of uid when
appropriate
2) name changes to be more consistent
Bug: 73823969
Test: manual test
Change-Id: Iacf7186dbd7a2282f7fe481f43dbbf92e1165b47
Mostly to add test to assure the corner cases are covered.
One minor logic change is if two true conditions happen, in the case
when following happen:
(bucket boundary1) -> (condition false) -> (condition true) -> (pull
triggered for the boundary1)
Previously we take the latest. Now we skip the late boundary pull.
Bug: 76384731
Test: unit test
Change-Id: I345c2210a58bf03eb91d65742573073d2668358b
+ change StatsPullerManager internal time units to be consistent
+ use series of alarms for pullers, instead of use setRepeating
Bug: 76223345
Bug: 75970648
Test: cts test
Change-Id: I9e6ac0ce06541f5ceabd2a8fa444e13d40e36983
Previously tried an optimization that results in corrupted proto
output. This changes to a safer approach of storing the snapshot data
in memory and only converting to proto output when the
ProtoOutputStream is provided.
Also fixes a security issue when trying to invoke triggerUidSnapshot
since we forgot to use SCS' permissions.
Test: Added a unit-test to verify output of StatsLogProcessor.
Bug: 76231867
Change-Id: Id410ce3505fda9d71caa71942ef3068b55872c66
Statsd clients may want to set an empty config temporarily, so it's
more convenient to allow them to set an empty config instead of
having to use the removeConfig and then having to remember to call
StatsManager#setDataFetchOperation.
Test: Added unit-tests and check they pass on marlin-eng.
Bug: 74997752
Change-Id: I2e762e5ec01e5a2c9a3469fb330b53fefbd734d6
ProtoOutputStream
- Specify the length of message to avoid libprotoutil from thinking that
we are trying to write bool
- We only attach the previous dump file to the upload file where config
key matches
- Store ConfigMetricsReport (instead of ConfigMetricsReportList) onto
disk
- Stop use stack after scope in StorageManager
- Migrate UidMap to use ProtoOutputStream and renaming variables to
prevent confusion
Bug: 74021554
Bug: 75968524
Test: manual test, statsd_test, CTS tests
Change-Id: Iedf52633d7f5b985f5a934a3fb5a0c3c3b2e7fd1
Various fixes and improvements to statsd's anomaly detection.
Bug: 74607818
Test: make statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test
Change-Id: Ia67a8eb6da0ea9293f698949e1565f7f024a7cb9
We should fail right away if someone forgets to set this field
instead of setting default values since this may lead to hard to
explain bugs in the future (eg, why isn't systemui logging an atom
to a config).
Bug: 74608359
Test: Verified unit-tests still pass on marlin-eng.
Change-Id: Ibd8a6ccbc5cc8d2dfb8a1577c64bf9b49822a2c3
We don't need to parse the proto of uid map, so we use the
ProtoOutputStreame class to generate the binary form of the proto
output that's needed for parsing the uid map data.
Test: Verified unit-tests still pass.
Bug: 74010813
Change-Id: Ia2f7572f3b78bb6f7b60e8b14cf5d65428469ab6
Past buckets had a field mBucketNum, storing the
"bucket number" they represented. Currently, the concept
of bucket number is used by Anomaly Detection (although this
may not be necessary), but these stored values are not used at all,
so removing them will save RAM.
Test: statsd tests still pass
Bug: 74607818
Change-Id: Iacc343bc39c5035f6e2f236c03de1d91606eff4c
Fix the bug when serializing multiple atoms in gauge metric
BUG: b/74159560
Test: new test for ALL_CONDITION_CHANGES sampling method.
Change-Id: I6d33c1efbac92b6e13be2d64c323e090cb1f84aa
use same cpu stats readers with BatteryStats so that
1) both throttle to avoid too frequent pulls
2) cached value is served within throttle interval to avoid double
pulling by both statsd and BatteryStats
To run unit tests:
bit
FrameworksCoreTests:com.android.internal.os.KernelUidCpuFreqTimeReaderTest
bit
FrameworksCoreTests:com.android.internal.os.KernelUidCpuClusterTimeReaderTest
bit
FrameworksCoreTests:com.android.internal.os.KernelUidCpuActiveTimeReaderTest
make -j56 statsd_test && adb sync data && adb shell
/data/nativetest64/statsd_test/statsd_test
Test: cts test, unit test
Bug: 73745189
Bug: 73780619
Bug: 73360959
Merged-In: I10a9bc91ca67fa812f4cd71c4fbd73c1a5ba580e
Change-Id: I10a9bc91ca67fa812f4cd71c4fbd73c1a5ba580e
+ Metrics will do flushIfNeeded() to correctly move the clock and informing
AnomalyTracker the past bucket info, and then clear past buckets.
+ We will still keep the current bucket data for the validity of the future metrics.
Bug: 70571383
Test: statsd_test
Change-Id: Ib13c45574974e7b4e82bd8f305091dc93bda76f5
StateTracker is a special condition tracker that's based on a state atom.
State atoms are annotated in atoms.proto.
The rules for StateTracker:
1. must not have "stop". must have "dimension"
2. must be based on a state atom.
3. it must have the all primary fields and the exclusive state field in its dimension.
For example UidProcessStateTracker, will have output dimension {uid, state}.
Test: unit tests added.
Change-Id: I6b77e58e9fabe61f7326daf929577d8b2cfbf27b
1. StatsdStats does not use the proto object in memory anymore.
2. lite_static -> lite
3. don't use sstream
Bug: 72129300
Test: statsd_test
Change-Id: I8a5adaf222d4d5034e8bf115215fb6dd5f042cac
+ This is useful when we want to build Anomaly detection on wakelocks, but want to whitelist
wakelocks held by some apps that are whitelisted. It reduces the number of matchers needed
in such a config.
+ Also added the ability to match an AID by string name.
Bug: 73897465
Test: unit tests added.
Change-Id: I19315ae4d7d27fc467655d3a29866049cd8c9a2b
