Commit Graph

138 Commits

Author SHA1 Message Date
Muhammad Qureshi
741d31898a Remove libstatsmetadata
Remove kWhitelistedAtoms usage in MetricsManager and remove:
- atoms_info references from statsd
- libstatsmetadata usages from statsd
- libstatsmetadata library
- atoms_info_writer.h/.cpp
- references to atoms_info in rest of stats-log-api-gen
- allow_from_any_uid annotation

Fixes: 147600720
Fixes: 154856835
Fixes: 148993016

Test: m stats-log-api-gen
Test: m stats-log-api-gen-test &&
out/host/linux-x86/nativetest/stats-log-api-gen-test/stats-log-api-gen-test
Test: m statsd

Change-Id: I2e0579609ff257da934b95cdda397f3ca0ffa1f0
2020-06-18 18:59:12 -07:00
Chen Zhu
253d4f3fe7 Add test config to statsd_test and apply mainline module
test controller to it so that the tests will only be run

when statsd apex is found on device.

Bug: 154845935
Test: m mts && mts-tradefed run mts-statsd
Change-Id: Ieaa8049cb66b7e75daba4f22477e850367821b64
2020-05-21 13:40:36 -07:00
Jeffrey Huang
8650e9aa11 Merge "Fix atest statsd_test" into rvc-dev 2020-04-22 16:41:45 +00:00
Tej Singh
affc08a340 Merge "Split buckets on boot complete" into rvc-dev 2020-04-22 01:16:28 +00:00
Tej Singh
e678cb753a Split buckets on boot complete
Also clean up a bit of code on splitting on app upgrades

Piggy-backed off the app upgrade tests, adding parameterized tests to
also test boot complete event.

Refactored some value metric test code to increase code reuse and
assertions.

Fixed a broken value metric test that had assertions commented out.

Refactored NamedLatch into MultiConditionTrigger to avoid creating a
thread before necessary.

Test: atest statsd_test
Test: push a simple test config, reboot, wait, get data. Made sure
the bucket was split
Bug: 144099206
Bug: 154511974

Change-Id: I73858b5db08e8cda762bd8091b30da8738d1fd88
2020-04-21 15:31:14 -07:00
Jeffrey Huang
fe283e18d5 Fix atest statsd_test
Seems like we need to create a 64bit binary for atest

Test: atest statsd_test
Bug: 154509776
Change-Id: Ibc47183ed478a08c12fff43dcd9dc793a121d0a1
2020-04-20 14:21:40 -07:00
TreeHugger Robot
5aef02fd4a Merge "Fix statsd_test mts build target" into rvc-dev 2020-04-18 00:11:18 +00:00
Jeffrey Huang
9c30a95f5a Fix statsd_test mts build target
The test tried to execute statsd_test when the binary was named
statsd_test32. This fixes the naming issue.

Bug: 153035538
Test: make mts and check the binary name
Change-Id: I13485a95801f9ef71211cd1d36e8ea1fd3bbc0b6
2020-04-17 15:35:33 -07:00
TreeHugger Robot
702537c7ab Merge "Add statsd device tests as mts target" into rvc-dev 2020-04-16 16:59:32 +00:00
Jeffrey Huang
c6b2830e81 Add statsd device tests as mts target
Added multilib to cc_test as a workaround to the build system being
unable to separate 32 and 64bit architectures.

Bug: 153035538
Test: m -j
Change-Id: Ib7f3b01816d7cbd69b3dbf4dbf63ccb22b0c5b6a
2020-04-15 16:06:40 -07:00
Tej Singh
769f35fc0f Add NamedLatch to statsd
This is a sychronizing primitive that is similar to a latch, but a
thread must count down with an identifier. It will be used to make sure
the boot complete, uid map, and all pullers signals are received before
triggering the bucket split.

The latch's countDown operation takes in a string identifier, so that if
the same operation happens twice, it is only counted once.

Bug: 144099206
Test: atest statsd_test

Change-Id: I261a3e50eabbc4998ca30ddf2d67a9a1e788911e
2020-04-14 16:50:12 -07:00
Tej Singh
5d823b30fa Statsd test mapping
Makes a test mapping for statsd so that unit tests run on presubmit.

Changes to make the tests pass:
1. Require root. This is needed to write to disk, since the tests don't
run as statsd's uid

2. Remove AndroidTest.xml file in favor of the autogenerated one.

3. Remove a check in StatsService.test for getUidFromArgs. The test
checked a failure case where we passed a number bigger than INT32_MAX.
However, on a 32 bit device, strtol will return INT32_MAX when an
overflow happens, since it returns a 32 bit number on a 32 bit device.

4. Refactor a lot of e2e tests to sort dimensions, ensuring that the
dimensions are always in order, instead of relying on implicit ordering
of hashing, which can change.

5. Change a long to an int64 in TestActivationsPersistAcrossSystemServerRestart

Test: statsd_test
Bug: 129613474
Change-Id: I80dfa3bfd50ebe6d2c8c0c3ba201f3ad06b68910
2020-04-09 22:20:59 -07:00
Tej Singh
fa49ff93f9 Merge "Move proto, c++, and test libs to static" into rvc-dev 2020-04-08 00:37:49 +00:00
Tej Singh
1d4b4653cd Move proto, c++, and test libs to static
Link libgtest_prod, libprotobuf-cpp-lite, and libc++ statically.
This reduces the size of the statsd apex from 2.8M to 1.6M.

Initial:
blueline:/apex/com.android.os.statsd # du -h
16K	./lost+found
764K	./bin
1.3M	./lib64
8.0K	./etc
600K	./lib
120K	./javalib
2.8M	.

After linking proto lite and libgtest_prod statically (saves ~500kb):
blueline:/apex/com.android.os.statsd # du -h
16K	./lost+found
800K	./bin
876K	./lib64
8.0K	./etc
600K	./lib
120K	./javalib
2.3M	.

After also linking libc++ statically (saves ~700kb):
blueline:/apex/com.android.os.statsd # du -h
16K	./lost+found
0.9M	./bin
296K	./lib64
8.0K	./etc
204K	./lib
120K	./javalib
1.6M	.

After this change, the only libraries in both /lib and /lib64 are:
libstats_jni.so
libstatspull.so
libstatssocket.so

These are the 3 libraries we actually want to ship.

Test: m
Test: bit statsd_test:*
Test: atest GtsStatsdHostTestCases
Test: manually inspecting the size of the apex, statsd binary, and
libraries
Bug: 153349706
Bug: 153349659
Bug: 153350187

Change-Id: I00796fc1cf33e394422c6dc617a8430a5f5c5bbb
2020-04-06 20:01:49 -07:00
Christine Tsai
d4abb2f9ff Merge "Remove StateConditionTracker from statsd" into rvc-dev 2020-04-03 18:13:06 +00:00
tsaichristine
3d8e12a5ef Remove StateConditionTracker from statsd
StateConditionTracker was previously used as a way to slice by state
through a specific SimplePredicate definition. It is currently being
replaced by StateTracker, and no metrics use StateConditionTracker at
this time.

Bug: 136566566
Test: bit statsd_test:*
Change-Id: I1a14747907c23482a1b7500c6d7f07cd2f1d4aed
2020-04-03 09:41:52 -07:00
tsaichristine
1449fa4661 Slice by state in DurationMetricProducer
- Added #onStateChanged logic to DurationMetricProducer
- Only SUM aggregation type supports state slicing.
OringDurationTrackers now track durations keyed by different state
values.
- Added unit tests for DurationMetricProducer state changes

Test: bit statsd_test:*
Bug: 145838038
Bug: 136566566
Change-Id: I87dcc21bb904ef0fa76ea196d60dadae00f310fa
Merged-In: I735c7deee96639c32cad4b907e59b354dfd98d77
2020-04-02 14:53:40 -07:00
Jeffrey Huang
898dda3919 Merge changes Icd1b1e57,Ib9c6b9b4 into rvc-dev
* changes:
  Load Metadata from Disk
  Save metadata to Disk
2020-04-01 18:56:26 +00:00
Jeffrey Huang
475677e771 Load Metadata from Disk
Bug: 148280505
Test: bit statsd_test:*
Test: Manual - Added Config, Tested System Server Crash
Test: Manual - Added Config, Tested Device restart
Change-Id: Icd1b1e57bbf24e4a5676820a2b955f2034d4b37d
2020-03-31 19:10:58 -07:00
Jeffrey Huang
b8f5403d17 Save metadata to Disk
Bug: 148280505
Test: bit statsd_test:*
Change-Id: Ib9c6b9b4f22e7380717b480c7ae4a37bb3364619
2020-03-31 19:10:58 -07:00
Tej Singh
3be093b518 Uid Sandboxing of Pullers
Overall flow of implementation:
1. parsing the config in MetricsManager to store the uids per atom. It
follows the mAllowedLogSources logic very closely
2. MetricsManager register itself as a PullUidProvider with the
PullerManager.
3. Metrics pass the config key when pulling (for both registering
receivers and normal pulls) , and the puller manager gets
the allowed uids from the PullUidProvider for that config.
4. PullerManager keys receivers by <atomId, configKey> so that it can
look up the uids for that atom using the PullUidProvider as well.
5. Added shell subscriber support. Hardcode a default of AID_SYSTEM for
them and also allow packages per atom. This involved adding a second
interface to Pull that simply accepts the uids, since I didnt want to
make the ShellSubscriber a PullUidProvider as well.
6. Change adb shell cmd stats pull-source to allow users to specify a
package. Default to AID_SYSTEM as well.

Notes:
The feature is flagged off right now, since configs do not pass in the
desired package. Another approach could be to hardcode in the current
mapping, but that doesn't work for OEM pulled atoms.

Test: m statsd
Test: bit statsd_test:* with useUids = false
Test: bit statsd_test:* with useUids = true
Bug: 144099783
Bug: 151978258

Change-Id: I4a7481d7402a52b9beb4ea28b102803f9e50e79f
2020-03-27 18:07:24 -07:00
Jeffrey Huang
1bd7fe106a Merge changes I7127dd7d,I84da0cd9 into rvc-dev
* changes:
  Add mAlertToAnomalyTrackerMap to MetricsManager
  Add StatsMetadataList proto
2020-03-25 22:31:02 +00:00
tsaichristine
63143251fe Update the cc_benchmark to use the new socket schema
Also removed unused CreateEvent helper functions in metric_util.cpp

Test: bit statsd_benchmark:*
Bug: 149794614
Change-Id: I14bcc254c44773eb42f749305beadaaf23a19b2b
2020-03-24 16:03:48 -07:00
Jeffrey Huang
8a617e0024 Add StatsMetadataList proto
Bug: 148280505
Test: m -j
Change-Id: I84da0cd9c79dc1073c40a439fe2acff2e3fbc7ec
2020-03-24 12:46:14 -07:00
Jeffrey Huang
3eb84d4da9 Migrate statsd_test to use libstatslog_statsdtest
This is part of the migration to remove libstatslog from statsd

Since we still depend on libstatsmetadata, we cannot fully mock out
all the atoms. So we will create a new libstatslog_statsdtest to house
atoms used only in the tests.

Bug: 150976524
Test: bit statsd_test:*
Change-Id: I6368305eb89b2c35e670e42907a308afd922e604
2020-03-17 11:59:01 -07:00
Jeffrey Huang
74fc4354b5 Remove apex_available from libstatslog
Test: m -j
Bug: 149781190
Change-Id: I765852b1719a3056c06a0965ea6a05a8e8410693
2020-03-10 15:14:27 -07:00
Muhammad Qureshi
d7944d6573 Merge "Remove libstatslog from libstatsmetadata" into rvc-dev 2020-03-06 08:04:46 +00:00
Muhammad Qureshi
9e0f728fe4 Remove libstatslog from libstatsmetadata
Instead of relying on the constants in libstatslog, hardcode them in
atoms_info to get rid of the dependency on libstatslog

New generated atoms_info.cpp:
https://paste.googleplex.com/5779947622760448

Bug: 150417465
Test: m libstatsmetadata
Change-Id: I954c963f1883f889053b63d308c648548de71e56
2020-03-05 17:20:58 -08:00
Tej Singh
b3915ea363 Remove libstatssocket from statsd_test
Statsd test links both libstatssocket and libstatssocket_private. It
shouldn't need to link both, but needs the private lib to be able to
access hidden APIs like getBuffer. So, remove it from statsd_test.

Test: atest statsd_test
Bug: 150435148
Change-Id: I46e71209489c8b1924800680f26e87ef7f4b8598
2020-03-05 14:37:25 -08:00
Tej Singh
eb7d8f4675 Make libstatssocket shared
Also make private versions of the libraries available to link statically
for tests. We should try to figure out a better long term solution.

Test: m
Test: bit statsd_test:*
Test: atest LibStatsPullTests
Test: bit libstatssocket_test:*
Test: atest GtsStatsdHostTestCases
Bug: 149340100

Change-Id: I05b91efab2a657aec75d436575aff4373f86ed3f
2020-02-27 11:38:41 -08:00
Howard Ro
7008dbe5e4 Merge "Revert^2 "Migrate Statsd to the apex"" 2020-02-20 08:00:53 +00:00
Howard Ro
adc510fd78 Revert^2 "Migrate Statsd to the apex"
24bfb40916

Change-Id: I4bda9b6eb759c32b9167ad9d58bdd3d10ff29411
2020-02-20 04:57:00 +00:00
Howard Ro
00306fce6a Merge "Revert "Migrate Statsd to the apex"" 2020-02-20 04:22:02 +00:00
Howard Ro
24bfb40916 Revert "Migrate Statsd to the apex"
Revert "Update statsd binary path for apex"

Revert submission 10356003-statsd_apex

Reason for revert: https://android-build.googleplex.com/builds/quarterdeck?branch=git_master&target=build_test&lkgb=6221366&fkbb=6221428
Reverted Changes:
Ic84e77269:Update init.rc for statsd setup
I6f3fab4b4:Update statsd binary path for apex
Ib4ea98aed:Migrate Statsd to the apex

Change-Id: I089d5b9bbbdc363e27e79e4addd4eec1e4339c48
2020-02-20 04:19:41 +00:00
TreeHugger Robot
d71f157eb6 Merge "Add libstatspull and libstatssocket to the apex" 2020-02-20 02:35:47 +00:00
Muhammad Qureshi
7b31ff81c5 Add libstatspull and libstatssocket to the apex
Also make libstatssocket dependency static in statsd_test temporarily to use
AStatsEvent_getBuffer in tests.

Bug: 145922701
Bug: 145923416
Test: m

Change-Id: Idb8107cfe4f7d08d08c7a242c037fcc63912bb5f
2020-02-19 14:49:43 -08:00
Jeffrey Huang
89e09b58d6 Migrate Statsd to the apex
Bug: 145923087
Test: m -j
Change-Id: Ib4ea98aedef629dd9a2a8764cfd91284a38ba596
2020-02-19 14:13:13 -08:00
TreeHugger Robot
a73ae8bbc4 Merge "Comment out Statsd tests" 2020-02-19 04:57:51 +00:00
Jeffrey Huang
1e4368aa43 Comment out Statsd tests
Added todos to make them ues the new schema.

Bug: 145923087
Test: m -j && bit statsd_test:*
Change-Id: I0749760eb3123407b78b9ace9a93967bac727bf5
2020-02-18 18:36:02 -08:00
Jeffrey Huang
52e90024d3 Allow libstatsmetadata to be linked by statsd
Bug: 145923087
Test: m -j
Change-Id: I8ad5453be7aaefd8be9a20539f35de06209060e3
2020-02-18 14:57:20 -08:00
Ruchir Rastogi
e449b0c185 Move statsd (and tests) to libbinder_ndk
Major changes include:
    - Removing unused permission checks within StatsService. These
      include ENFORCE_DUMP_AND_USAGE_STATS, checkDumpAndUsageStats,
      kOpUsage, and kPermissionUsage.
    - Converting from sp to shared_ptr
    - Using libbinder_ndk functions instead of libbinder functions
      (e.g. for installing death recipients, getting calling uids, etc.)
        - New death recipients were added in StatsService,
          ConfigManager, and SubscriberReporter.
    - Using a unique token (timestamp) to identify shell subscribers
      instead of IResultReceiver because IResultReceiver is not exposed by
      libbinder_ndk. Currently, statsd cannot detect if perfd dies; we
      will fix that later.

Bug: 145232107
Bug: 148609603
Test: m statsd
Test: m statsd_test
Test: bit stastd_test:*
Test: atest GtsStatsdHostTestCases
Change-Id: Ia1fda7280c22320bc4ebc8371acaadbe8eabcbd2
2020-02-14 18:07:37 -08:00
Wonsik Kim
d3f06c5f84 Revert "Move libstatssocket to apex"
This reverts commit 3f17225f75.

Reason for revert: major functional regression
Bug: 149538115
Change-Id: Ic5650a09afcfb735668a6ef35cfefc6c4c915a20
2020-02-14 19:35:04 +00:00
Jeffrey Huang
3f17225f75 Move libstatssocket to apex
Bug: 145923416
Test: m
Test: fastboot flashall
Change-Id: Ieae64fb975bf7986f987e1f27548e041db2c505e
2020-02-13 11:13:57 -08:00
Ruchir Rastogi
56da4c3862 Use StatsDimensionsValueParcel within statsd apex
Because statsd now uses StatsDimensionsValueParcel instead of
StatsDimensionsValue.h/c, statsd no longer has to depend on libservices.

Test: m -j
Test: atest StatsdHostTestCases#testBroadcastSubscriber
Bug: 148604617
Change-Id: I6d65383ccec99f4672d6575232981c0f6cc40fcf
2020-02-07 15:55:11 -08:00
Ruchir Rastogi
86379b14a7 Use cpp aidl_interface within statsd
This CL is part of a larger effort to shift statsd to libbinder_ndk.
This CL specifically accomplishes the following:
    - constructs an aidl_interface soong module that can be used by statsd
      and libstatspull
    - defines the parameters of StatsDimensionsValueParcel
      (conversions to and from StatsDimensionsValueParcel are not yet defined,
      so this CL will break sendSubscriberBroadcast. A later CL will fix this
      issue...)

Test: m -j
Bug: 148682447
Change-Id: I7afa3e21d4c9fc1970e6c393204e1bdb110760fb
2020-02-07 15:40:24 -08:00
Howard Ro
47807e2f01 Statically link libutils
this is still needed for stastd

Bug: 145565901
Test: compiles
Change-Id: I6356fbc8e4c92d3b6dc9742bf661a4fdd58dac47
2020-02-06 23:17:33 -08:00
Jeffrey Huang
0e0313c399 Migrate StatsHal
Test: bit VtsHalStatsV1_0TargetTest:*
Bug: 148794952
Change-Id: I19ff6edf78bd84a95d8c0ae3a1adda99d4232aa5
2020-02-06 10:03:29 -08:00
TreeHugger Robot
0a3a6c179c Merge "Link libstatssocket as a shared library" 2020-02-06 09:40:14 +00:00
TreeHugger Robot
f8df902cbf Merge "GpuStats: migrate to new statsd native puller api" 2020-02-06 07:57:29 +00:00
Howard Ro
e5575c210e Link libstatssocket as a shared library
Library clean up as a part of statsd becoming a module

Bug: 145565901
Test: compiles
Change-Id: I52ef9bfe9bf52a2582e155317b3817da3600b87d
2020-02-05 22:49:39 -08:00