+ Metrics will do flushIfNeeded() to correctly move the clock and informing
AnomalyTracker the past bucket info, and then clear past buckets.
+ We will still keep the current bucket data for the validity of the future metrics.
Bug: 70571383
Test: statsd_test
Change-Id: Ib13c45574974e7b4e82bd8f305091dc93bda76f5
This could happen when statsd is disconnected from logd reader. When we reconnect, we are going to
get all events from the buffer again.
Bug: 72379125
Test: manual
Change-Id: Ie0122d5452555500c3bdfc1f905a0b1c646efdf7
We should be using elapsed realtime for most timestamps in statsd
so that the times can only increase monotonically.
Test: Test that statsd builds and unit-tests passes.
Change-Id: I0bb23e89aa9a6dbf6d56a0c23eec77bdd053f29b
its clients have not started to receive broadcasts
This also fixes broken statsd_test's which happens whenever there are
files in /data/misc/stats-data/ which is generated right before reboots.
This would delay the upload time from right after reboot to next upload
cycle but it should not be an issue.
Bug: 73089712
Test: statsd_test
Change-Id: Ida81099c9c9e54804a0c3b3b349096312ef570bc
The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.
+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.
+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
becoming checking if the underlying integers are equal. The integers are stored contiguously
in memory, so it's much faster than previous tree structure.
Start review from FieldValue.h
Test: statsd_test + new unit tests
Bug: 72659059
Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
Sets DEBUG to false everywhere and replaces all ALOGD with VLOG so they
do not print with DEBUG false. Leaves all ALOGI, ALOGW and ALOGE as is.
Test: ran all CTS tests and checked "adb logcat -s statsd" to make sure
it wasn't spammy
Change-Id: Iaa8eb3a0a63723ffe40f94f2815f94df877fd432
We include the start of when the last dump occurred and the current
timestamp. These timestamps are shared across all metrics, so
there's no advantage in duplicating these numbers across all metrics.
Also, we should use elapsed realtime instead of times based on wall
clock, which can jump around and go backwards.
Test: Test that statsd can still build and
adb shell cmd stats dump-report doesn't crash.
Change-Id: I819e5643cee75dfa3e78a58f94c9d61ededa78d7
1/ Avoid unnecessary field/dimension proto construction.
2/ use unordered_map for slicing.
3/ Use dimension fields to compare dimension keys.
Test: all statsd tests passed.
Change-Id: I2f74f78589b7f6ecd0803a2ead822b8d0399f334
+ Bugreport will use the non-verbose mode
+ Reuse the log_msg object in LogReader
+ Add logd errors to StatsdStats
Bug: 72383073
Test: manual + statsd_test
Change-Id: Id5a8b103074d034f5ece3c9831c740d44a5df9cd
- Limit total number of files to 1000
- Limit total size of files to 5MB
- Remove idle files to be deleted after 30 days
Bug: 69854160
Test: manual testing, statsd, statsd_test
Change-Id: I33148a3b7ca11d413ec2495d5c0659f1ba4485c3
Statsd will contain at least one snapshot of the uid map. The
previous design was not very robust in case a snapshot was missing.
Also fixes subtle bug with updating the isolated uid mapping since
this should always be kept up to date even if there are no metrics
being used (since metrics may be added later after the isolated uid
was created).
Test: Checked that unit-tests pass on marlin-eng.
Change-Id: I99754ed9016d980564e409b0946a46b398fd12b7
rather than during the runtime of statsd
The purpose of this change is to prevent causing selinux violation by
trying to mkdir to /data/misc/ directory when statsd doesn't have
permission to do so.
Bug: 71537285
Test: manually tested to make sure that there's no sepolicy violation
Change-Id: I9c4ccecc416f41923c9b24dd44a388d135fecc07
+ predefined "AID_X" will be provided as string type to statsd, and we will translate
to integer uid using the static map.
Test: statsd_test
Change-Id: Ie47d8481e0c456457e6881ebb9cb4ce008e772b8
2/ Handle Subscription for alert.
3/ Support no_report_metric
Bug: 69522276
Test: all statsd unit tests passed.
Change-Id: I851b235f2d149b8602b0cad632d5bf541962f40a
Splits out DurationAnomalyTracker-specific functions into their own
subclass.
Test: the unit tests and CTS tests
Change-Id: Id6eb74d232b4a9c3a932d805d1ba3f0ba43a88b1
+ Add log source whitelist in StatsdConfig
+ Some changes in UidMap API. Listener needs to be wp instead of sp.
+ Update dogfood app config to have log source
+ Increase the stats service thread pool size to 10 (9+1).
TODO: add unit tests(b/70805664). This unit test takes some time to write.
Test: statsd_test & manual
Change-Id: I129b1cc13db5114db7417580962bd7cc4438519d
We use one alarm clock for all pulled atoms.
If metrics from different configs are not aligned,
the clock will be set to repeat at higher and higher
frequency, and consume a lot of battery.
Current implementation assumes a 5min minimum bucket
size. New metric start time is set to be aligned to
the start time of statsd in the next 5min.
So it will ignore events up to 5min.
align puller alarm to minute bundary
Test: unit test
Change-Id: I77ffa3c13de363c780b1000181b9a9b780dd0846
The exception is EventMetricProducer. Each EventMetricProducer will still have a ProtoOutputStream
Because LogEvent comes as a fixed 4K, it's more memory efficient to have an 8k ProtoOutputStream for
storing the events.
Also removed finish() api in MetricProducer, which was intended to use with Dropbox.
Test: statsd_test & dogfood app
Bug: 70393808
Change-Id: I2efe4ecc76a88060a9aa5eb49d1fa6ea60bc5da8
Since there is a separate guardrail for memory used by uid map, we
no longer add the memory from uid map with the memory per each
config's metrics. We also prevent the byte size check from happening
too frequently. In order to mock the MetricsManager, we refactor
some of the existing methods.
Test: Added unit-tests and verified they all pass on marlin.
Change-Id: I15cf105f7d95f4016fdb0443b0a33eebe862cafb
Previously, we always sent a broadcast, even after we have exceeded
the memory limit for this config key. We switch the order so that
we drop the data if the limit is exceeded. If greater than 90% of the
way to the limit, we send the broadcast.
We need to find a way to unit-test this behavior.
Test: N/A.
Change-Id: I6ea40b9e34dceb19805d9af24495d72878f787e0
Checks if current memory usage of uid map is above a configured limit
and if so, we start deleting snapshots. If there are no more
snapshots, we begin deleting two of the deltas. Also records stats
in the guardrail StatsdStats. Also fixes an edge case where a config
is added after the snapshots are added. We request a snapshot of all
installed uid's at that moment. Finally, adds the uid map memory size
when determining if we should send a broadcast to trigger collection.
Test: Added unit-tests and check they pass on marlin.
Change-Id: Id5d86378bd1efe12a06b409164c777c0c6f4e3ab
Fixes a few items in AnomalyTracker, especially to do with what happens
when an anomaly alarm fires.
Test: unit tests still pass
Change-Id: Ia89bd617442e952e587336b890c3ca67430b5e21
+ StatsdStats is the global class that tracks the stats about statsd.
+ Added guardrail for classes that have a map which could potentially grow
unboundedly with the number of logs.
TODO: add unit tests & CTS for StatsdStats, and guardrail
add stats for pulled atoms.
Test: statsd_test
Change-Id: I0ea562de4dd3f6162f7923a9c193420b482c1d51
1. Create intent receiver in StatsCompanionService to listen to shutdown
events.
2. Create StatsWriter class to handle disk writes and deleting files.
3. Update StatsLogProcessor, ConfigManager, and StatsService to handle
files on disk using StatsWriter.
4. Add a wrapper for ConfigMetricsReport.
Still TODO is to be able to add a guardrail to prevent accumulating
excessive amount files on disk, which will be followed up by another
change.
Test: statsd, statsd_test
Change-Id: Ia0b3af315af545daa8b0078b3700c600aa7c285f
2/ Init anomaly from config based on the public language.
3/ Unit tests for anomaly detection in count/gauge producer.
4/ Revisit the duration tracker logic.
Test: unit test passed.
Change-Id: I2423c0e0f05b1e37626954de9e749303423963f2
