Reports written to disk don't contain the strings used, which will
make this report unusable if there are strings that don't show up
again. We should always include the strings, so this option is
removed entirely.
Also, we hard-coded the wrong number of fields when pulling
ModemActivityInfo. There are actually 10 fields, not 6.
Bug: 79601503
Test: Tested unit-tests pass on marlin-eng.
Change-Id: I6834b096ced77418a9cc2ddd79b08d1c9c447fae
We notice devices uploading a bunch of bytes for the uidmap even if
the device is running an empty config, so there are no actual metrics
to report. This hardcodes some logic to skip the inclusion of the
uidmap if there are exactly 0 metrics.
Bug: 79381210
Test: Tested unit-tests on marlin-eng
Change-Id: I96348235341a7faf15ff57d4d1eccac635a3a999
We observe a single ConfigMetricsReportList can be greater than the
safe size for the binder transaction buffer since we only check the
size of the current metrics in progress, but we also return the
previous reports stored on disk.
This change will attempt to send another ConfigMetricsReportList
as soon as possible if there's already a report on disk.
Also fixes a bug when trying to trigger data fetch before the client
has registered the corresponding dataFetchOperation.
Bug: 79201869
Test: Tested manually on marlin-eng
Change-Id: I2d3677162804a27e7a7a95d482d80c46bd994a67
It only works on eng build. And all code is behind a build flag, so the
code will be stripped out in production builds.
Bug: 78239479
Test: manual
Change-Id: I20ee51822d18e6c77ca324a5327712cbed09593e
1. Hash the strings in metric dimensions.
2. Optimize the timestamp encoding in bucket.
Use bucket num for full bucket and millis for
partial bucket.
3. Encode the dimension path per metric and avoid
deduping it across dimensons.
Test: statsd test
Change-Id: I18f69654de85edb21a9c835c73edead756295e05
BUG: b/77813755
If the data receiver is experiencing delays, there may be a queue of
multiple intents to collect the same data. This timestamp makes it
easy in the receiver to de-dupe these requests to call getData.
Also, we update how StatsCompanionService gets the snapshot by
requesting data for all known apps. I notice that Keep seems to have
a uid active even when it appears uninstalled.
Bug: 77981668
Test: Flashed marlin-eng and manually verified.
Change-Id: I509e19383ec4a5da8746dd0c76ac71a948c6877d
When statsd reconnects to logd, statsd will read all logs from buffer again. To prevent us from
reprocessing old events, we do the following:
1. At any given moment, record the largest timestamp(T_max) and last timestamp (check point) that
we've seen before.
2. When reconnection happens, we look for the check point until we see a new log with a timestamp
larger than T_max.
-> If we found the CP, resume after the CP. Success
-> If we can't find CP, there is definitely log loss. We reset all configs.
Note:
1. Logd has an API to read logs after a certain timestamp. But this api is vulnerable to
time changes from Settings. So we cannot rely on it.
2. If logd inserts a new log (with older timestamp) before CP, we cannot detect it. It's not
possible to detect it without record all timestamps we have seen.
Test: statsd_test
Bug: 77813113
Change-Id: Ic3fdb47230807606ab11dc994cb162194adb8448
The previous scheme captured periodic snapshots for each config with
complex logic that's unnecessary and wasted memory. We actually don't
need to store any snapshots since we just convert the current state
into a snapshot and also include the deltas (change events) since the
previous report until now.
To make the system more robust, we also include up to 100 of the
deleted apps in the uid map.
Also, fix the wiring of the partial buckets so the metric producers
form partial buckets on both app upgrade and removal, but not on
installation of a new app.
Also, we update StatsCompanionService to also include disabled apps.
Bug: 77607583
Test: Verified unit-tests pass and added new e2e tests.
Change-Id: I98e1f544d6e6571545ae1581c4cebab807596f51
Roughly check the config every hour to see whether the ttl expired.
If so, read the config from disk and recreate the metric manager.
Test: statsd test
BUG: b/77274363
Change-Id: I16838afe5bbe966c3a0f638869751f9b59a5a259
+ change StatsPullerManager internal time units to be consistent
+ use series of alarms for pullers, instead of use setRepeating
Bug: 76223345
Bug: 75970648
Test: cts test
Change-Id: I9e6ac0ce06541f5ceabd2a8fa444e13d40e36983
+ Config count is 10 per uid
+ Update the limit for metrics, matchers, conditions, etc.
Test: statsd_test
Bug: 73122377
Change-Id: I3e1adfe318d1354a7c9d1bf484855661aa3a1fc8
We can increase the buffer of metrics we store in statsd memory, but
we still request the clients to call getData when the metrics memory
exceeds 128 KB (previously was 90% of 128 KB).
Bug: 76171061
Test: Test that unit-tests still pass.
Change-Id: I901545b364ed313af8c033ce9b40d3cfadb93213
ProtoOutputStream
- Specify the length of message to avoid libprotoutil from thinking that
we are trying to write bool
- We only attach the previous dump file to the upload file where config
key matches
- Store ConfigMetricsReport (instead of ConfigMetricsReportList) onto
disk
- Stop use stack after scope in StorageManager
- Migrate UidMap to use ProtoOutputStream and renaming variables to
prevent confusion
Bug: 74021554
Bug: 75968524
Test: manual test, statsd_test, CTS tests
Change-Id: Iedf52633d7f5b985f5a934a3fb5a0c3c3b2e7fd1
[memory]
statsd binary size from 664k -> 600k
memory usage 1978k -> 1813k (with no configs)
+ Avoid initialize any static map in statslog.h to avoid many copies of the map in each include.
- Do it in cpp so that it is initialized only in places that use them
[Uid annotation]
+ Uid annotation is needed for extracting uid from dimension for UidCpuPuller.
+ After the change, stand-alone uids don't need to be in field 1 anymore.
+ Also added exclusive bit annotation in AppDied
+ Currently only allow one uid field in an Atom. This is to keep things simple until
we find an exception.
Test: statsd_test
Bug: 73958484
Bug: 72129300
Change-Id: I8a916d5c00d5930e24ae7e0825a57dea19c0e744
We don't need to parse the proto of uid map, so we use the
ProtoOutputStreame class to generate the binary form of the proto
output that's needed for parsing the uid map data.
Test: Verified unit-tests still pass.
Bug: 74010813
Change-Id: Ia2f7572f3b78bb6f7b60e8b14cf5d65428469ab6
Fix the bug when serializing multiple atoms in gauge metric
BUG: b/74159560
Test: new test for ALL_CONDITION_CHANGES sampling method.
Change-Id: I6d33c1efbac92b6e13be2d64c323e090cb1f84aa
+ Metrics will do flushIfNeeded() to correctly move the clock and informing
AnomalyTracker the past bucket info, and then clear past buckets.
+ We will still keep the current bucket data for the validity of the future metrics.
Bug: 70571383
Test: statsd_test
Change-Id: Ib13c45574974e7b4e82bd8f305091dc93bda76f5
This could happen when statsd is disconnected from logd reader. When we reconnect, we are going to
get all events from the buffer again.
Bug: 72379125
Test: manual
Change-Id: Ie0122d5452555500c3bdfc1f905a0b1c646efdf7
We should be using elapsed realtime for most timestamps in statsd
so that the times can only increase monotonically.
Test: Test that statsd builds and unit-tests passes.
Change-Id: I0bb23e89aa9a6dbf6d56a0c23eec77bdd053f29b
its clients have not started to receive broadcasts
This also fixes broken statsd_test's which happens whenever there are
files in /data/misc/stats-data/ which is generated right before reboots.
This would delay the upload time from right after reboot to next upload
cycle but it should not be an issue.
Bug: 73089712
Test: statsd_test
Change-Id: Ida81099c9c9e54804a0c3b3b349096312ef570bc
The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.
+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.
+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
becoming checking if the underlying integers are equal. The integers are stored contiguously
in memory, so it's much faster than previous tree structure.
Start review from FieldValue.h
Test: statsd_test + new unit tests
Bug: 72659059
Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
Sets DEBUG to false everywhere and replaces all ALOGD with VLOG so they
do not print with DEBUG false. Leaves all ALOGI, ALOGW and ALOGE as is.
Test: ran all CTS tests and checked "adb logcat -s statsd" to make sure
it wasn't spammy
Change-Id: Iaa8eb3a0a63723ffe40f94f2815f94df877fd432
We include the start of when the last dump occurred and the current
timestamp. These timestamps are shared across all metrics, so
there's no advantage in duplicating these numbers across all metrics.
Also, we should use elapsed realtime instead of times based on wall
clock, which can jump around and go backwards.
Test: Test that statsd can still build and
adb shell cmd stats dump-report doesn't crash.
Change-Id: I819e5643cee75dfa3e78a58f94c9d61ededa78d7
