- When a log event is matched to the "what" atom of a metric,
MetricProducer parses the event for state primary key(s) and queries
StateTracker for the correct state values
- CountMetricProducer writes state information to the dump report proto
- Added e2e CountMetric tests that push events to StatsLogProcessor and
check that the resulting dump report is correct
Test: bit statsd_test:*
Bug: 136566566
Bug: 142124705
Change-Id: I9ca5b097151c0c73a617f3b16c8c3584b068cd82
We used a whitelist to determine which atoms should have their
timestamps truncated to the nearest 5 minutes. This changes the logic
to a blacklist so that we can get fine grained timestamps from vendor and
mainline atoms. Also reserves a range for atoms that need to be
truncated in the future.
Bug: 134574701
Test: inspected generated statslog.cpp file to make sure it had the
correct blacklist
Test: testdrive on ScreenStateChanged to ensure timestamps are preserved
when they should be
Test: testdrive on CallStateChanged to ensure timestamps are truncated
when they should be.
Change-Id: Id3468542c830cdf41395a94c77f7df0b46cd11b7
When statsd starts or is killed, the conditions are not initialized.
Tracks how many buckets are skipped because the condition was unknown.
This change does not change the behavior of statsd. In the future, we
might want to invalid buckets with a condition that has been partially
unknown.
Bug: 124100912
Test: atest statsd_test
Change-Id: Ie3d2eb5e30c21dda3203ad30f9b486961ff570cf
Creates a hidden api to register puller callbacks for vendor atoms.
Test: manual local test
Test: statsd unit tests
Bug: 119898637
Change-Id: Id28817b8fc718e128adc4e1c6b2e997db84517f9
Whenever a pull fails, we cannot compute a correct metric for this
bucket. Whether the failures happen at the beginning, in the middle or
at the end of the bucket, we cannot trust the data inside the bucket.
Let's say we have a metric with a screen on condition, a bucket size of
4 hours and we have the following events:
- h+0 bucket start -- screen is on -- data pull failed
- h+2 screen off -- pull succeed
- h+3 screen on -- pull succeed
- h+4 bucket end -- screen on -- pull succeed
The current logic will be wrong, it will ignore any data between h+0
and h+2. That timespan might be a huge contributor of the total bucket
value so this is wrong to keep any data from this bucket.
We also extend the concept of invalid buckets to other problems like a
pull being too long.
Bug: 123866830
Test: atest statsd_test
Change-Id: I300ab05cd7582cd2d7af9167de8d99b349071f0d
This adds to PulledAtomStats and creates a new stats atom AtomMetricStats.
Test: none
Bug: 123572169
Change-Id: Ifea682005210a6342a027643e162f8ce7785cb06
1) Refactor pullers and add tests.
2) Add timeout to a puller.
mPullTimeoutNs is intrinsic to puller. A pull taking longer than this is
deemed failed and the data discarded.
A metric or StatsPullerManager requesting a pull should monitor the pull
and have deadlineNs. A successful pull may come later than desired due
to statsd processing delays.
3) Add unit tests to puller now that the base puller is more
complicated.
Bug: 118756964
Test: unit test
Change-Id: I0e5d47e2527391f7beef4b2d06bfd5c2f82f1179
This is savaged from the large cl.
1. Simplify the logic in ValueMetricProducer.
1.1 for pull data on bucket boundary, we pull on bucket end, instead of
playing with timestamp twice.
1.2 for data that require diffing, we keep a rolling diff base that gets
updated.
1.3 Now we check condition in onMatchedLogEventInternalLocked for pushed atoms. For pulled atoms, check before commit time. This was very error prone in P and caused multiple bugs. It is much simpler now.
2. Treat pushed and pulled atoms the same way and share the same
aggregation types.
4. Allow decreasing values for diffing.
5. Allow diffing for pushed atoms.
6. For diff based aggregation, if the diff value is zero, we skip
output.
Bug: 117224984
Bug: 115683963
Bug: 117975256
Bug: 113268259
Test: unit test
Change-Id: I6ee306e9f6e5a166b392c443594704e7d2792ef5
There are an increasing number of requests to log data in complex format to statsd, while the data
is not expected to be parsed or aggregated by statsd and only to be uploaded as events.
Instead of making an exception for each of these cases in a hard coded way, this CL add a feature to
annotate these field in atoms.proto and the stats-log-api-gen tool will produce byte array
interfaces for them.
Note that log_msg does not have byte array type, and only has string type, when statsd receives the
log, these fields are in string type. Only when the atom is written to proto, we will check if this
field should be bytes field and write it to protobuf in message format.
Change-Id: If53dd95c5826710c76d7fe982bf951a435dfc738
Fix: 118386797
Test: unit test & manual test
When statsd pulls data from StatsCompanionService, the data is put into
log_msg format on java side and expanded into LogEvent on cpp side.
There is a lot of dependency on log_msg and liblog to add new types.
There is pending bug to rewrite thta part of the code in statsd to
completely rid the legacy of logd and liblog. But that may not happen
soon.
Now we can support new storage type.
Also no need to specify number of fields in StatsLogEventWrapper cstr,
which is a source of bug in P.
Bug: 115775035
Test: manual test and cts test
Change-Id: Id1f0b033885da6f3bcebe043968061821db48f35
1. Hash the strings in metric dimensions.
2. Optimize the timestamp encoding in bucket.
Use bucket num for full bucket and millis for
partial bucket.
3. Encode the dimension path per metric and avoid
deduping it across dimensons.
Test: statsd test
Change-Id: I18f69654de85edb21a9c835c73edead756295e05
BUG: b/77813755
+ change StatsPullerManager internal time units to be consistent
+ use series of alarms for pullers, instead of use setRepeating
Bug: 76223345
Bug: 75970648
Test: cts test
Change-Id: I9e6ac0ce06541f5ceabd2a8fa444e13d40e36983
The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.
+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.
+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
becoming checking if the underlying integers are equal. The integers are stored contiguously
in memory, so it's much faster than previous tree structure.
Start review from FieldValue.h
Test: statsd_test + new unit tests
Bug: 72659059
Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
Test: added e2e tests for count/duration metrics sliced by fields in condition and with/without links.
Change-Id: Ie34deba68e6780abdde458be3f0ce5284e76a1a2
all pullers have a default 1s cool down before next pull.
We can adjust these later.
Also add puller stats in StatsdStats
Test: unit test
Change-Id: I71894a24c41e059d841591312dbb852f54387b7d
