Commit Graph

3607 Commits

Author SHA1 Message Date
Chenjie Yu
e894f95053 Merge "ValueMetric overhaul" 2018-11-07 05:11:44 +00:00
Chenjie Yu
c715b9e0cc ValueMetric overhaul
This is savaged from the large cl.

1. Simplify the logic in ValueMetricProducer.
1.1 for pull data on bucket boundary, we pull on bucket end, instead of
playing with timestamp twice.
1.2 for data that require diffing, we keep a rolling diff base that gets
updated.
1.3 Now we check condition in onMatchedLogEventInternalLocked for pushed atoms. For pulled atoms, check before commit time. This was very error prone in P and caused multiple bugs. It is much simpler now.
2. Treat pushed and pulled atoms the same way and share the same
aggregation types.
4. Allow decreasing values for diffing.
5. Allow diffing for pushed atoms.
6. For diff based aggregation, if the diff value is zero, we skip
output.

Bug: 117224984
Bug: 115683963
Bug: 117975256
Bug: 113268259
Test: unit test
Change-Id: I6ee306e9f6e5a166b392c443594704e7d2792ef5
2018-11-06 17:07:29 -08:00
Adam Bookatz
3f5b30c42e Merge "Minor comment fix in StatsLogProcessor" 2018-11-06 22:41:45 +00:00
Colin Cross
872fd51397 Merge changes from topic "revert-protobuf3.5.2" am: a4680e1ebe am: 3e312d20cd
am: e217b6222c

Change-Id: Ib4c806582d5445c8387054dbc6fe2f6e173f1a97
2018-11-06 11:41:02 -08:00
Colin Cross
43ac27281b Revert "Convert statsd_test and statsd_benchmark to proto lite" am: 47046b80d2 am: 65200cf70e
am: bd7c4efbb6

Change-Id: Id4bab0ce60f08ce358db99c67c58cbef0fe6e589
2018-11-06 11:34:29 -08:00
Colin Cross
e217b6222c Merge changes from topic "revert-protobuf3.5.2" am: a4680e1ebe
am: 3e312d20cd

Change-Id: I096f3b5a5a985ce7926ccb71fa07eda69d078ed3
2018-11-06 11:21:11 -08:00
Colin Cross
bd7c4efbb6 Revert "Convert statsd_test and statsd_benchmark to proto lite" am: 47046b80d2
am: 65200cf70e

Change-Id: I952a9cbcb9bf9d2d479efecc28c5411e31dc9cf6
2018-11-06 11:15:44 -08:00
Bookatz
9cc7b660a9 Minor comment fix in StatsLogProcessor
Test: none; it's just a comment
Change-Id: Iafd9a5264eb13502509e7466c2773b0d5536170c
2018-11-06 10:39:21 -08:00
Adam Bookatz
bab22bc860 Merge "Statsd can dump data as proto to bugreport" 2018-11-06 08:35:47 +00:00
Bookatz
ff71cadecc Statsd can dump data as proto to bugreport
* Creates an incident section for statsd data.
* Allows dump to output statsd data, in proto format.
* Hooks up two statsd outputs to bugreports:
  -statsd report data in proto format
  -statsd metadata (statsdstats) in text format

The incident section does not import stats_log.proto because that turns
out to be extremely difficult: stats_log.proto imports atoms.proto,
which imports more things and is enormous and causes all sorts of
problems. atoms.proto was purposefully never compiled in AOSP, so to
retain that feature, the incident section uses 'bytes' instead of an
actual message. Since this isn't ever read in AOSP (other than testing),
this should be fine.

Bug: 115678461
Test: take a bug report and confirm valid proto
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases -t android.cts.statsd.atom.HostAtomTests#testDumpsysStats

Change-Id: I1c370af7678d1dc7440ce299ea5ea4da6d33832b
2018-11-05 18:52:49 -08:00
Colin Cross
44f74d111f Merge changes from topic "protobuf3.5.2" am: 1d899f6be8 am: 11a4d77516
am: c836696a92

Change-Id: I158185738327cee8f501dcba7e85de66240eaf42
2018-11-04 18:39:37 -08:00
Colin Cross
5d258093a8 Convert statsd_test and statsd_benchmark to proto lite am: e7908d7b62 am: 5742f9ffa7
am: 85201e02df

Change-Id: Id99366ec5900c71ba712dbee99fbefb5b785b827
2018-11-04 18:33:00 -08:00
Colin Cross
c836696a92 Merge changes from topic "protobuf3.5.2" am: 1d899f6be8
am: 11a4d77516

Change-Id: I6ad3772bc16f236c2c5e7ba381593b10292aabde
2018-11-04 18:13:52 -08:00
Colin Cross
85201e02df Convert statsd_test and statsd_benchmark to proto lite am: e7908d7b62
am: 5742f9ffa7

Change-Id: I5ef22cce9bbebe4355915ae3ab158faf14bf2708
2018-11-04 18:08:27 -08:00
Colin Cross
054b0c0e06 Revert "Adapt to google::protobuf::uint64 type change"
This reverts commit fa6bc27df0.

Change-Id: I9f778e6daa9ae0c6080016ab0bd1e6e414ae6061
2018-11-04 17:24:27 -08:00
Colin Cross
47046b80d2 Revert "Convert statsd_test and statsd_benchmark to proto lite"
This reverts commit e7908d7b62.

Change-Id: I585991ab1d5b35930bee6e32b82714aa11fbcb41
2018-11-04 17:24:26 -08:00
Colin Cross
e7908d7b62 Convert statsd_test and statsd_benchmark to proto lite
libplatformprotos for the device is compiled using proto lite, and
statsd_test and statsd_benchmark have protos that reference the
libplatformprotos.  With protobuf 3.5.2 compiling a full proto
against a lite proto doesn't work.  Convert them to lite.  Requires
adding the protobuf internal protos to the sources,
libprotobuf-cpp-lite does not contain the compiled version of
field_options.proto.

Bug: 117607748
Test: m checkbuild
Change-Id: I6a618c4118972c0e5ffb07a361ac6612b9069c5d
2018-11-02 11:35:48 -07:00
Colin Cross
fa6bc27df0 Adapt to google::protobuf::uint64 type change
Protobuf 3.5.2 redefines google::protobuf::uint64 from unsigned long
long to uint64_t, which is sometimes unsigned long and sometimes
unsigned long long.  Use PRIu64 to print it, and add an implementation
of ProtoOutputStream::write for long.

Bug: 117607748
Test: m checkbuild
Change-Id: I48912f58e2f63f4a98c3aefaa4f78cc2c719bb68
2018-11-02 11:35:48 -07:00
TreeHugger Robot
5f2a965757 Merge "record atom pull time" 2018-11-01 21:33:30 +00:00
Chenjie Yu
48ed1cc74f record atom pull time
output looks like this:
********Pulled Atom stats***********
Atom 10003->(total pull)2, (pull from cache)0, (min pull interval)0,
(average pull time nanos)528047, (max pull time nanos)568386, (average
pull delay nanos)4222422, (max pull delay nanos)4498125
Atom 10019->(total pull)2, (pull from cache)0, (min pull interval)0,
(average pull time nanos)4455885, (max pull time nanos)6245938, (average
pull delay nanos)12153985, (max pull delay nanos)13518387
Atom 10020->(total pull)2, (pull from cache)0, (min pull interval)0,
(average pull time nanos)3092005, (max pull time nanos)3223594, (average
pull delay nanos)18301668, (max pull delay nanos)19805783
Atom 10021->(total pull)3, (pull from cache)0, (min pull interval)0,
(average pull time nanos)3151944, (max pull time nanos)4931719, (average
pull delay nanos)17319897, (max pull delay nanos)24455211

Bug: 118757386
Test: unit test
Change-Id: Idbe0e4cc822d75813d2e4511be5e7593fff370d4
2018-11-01 11:30:52 -07:00
Rafal Slawik
272a816155 Add process start time to ProcessMemoryState atom
Also, add the start time to non-additive in statsd.

Bug: 118249210
Test: manually verified values are included in a report
Change-Id: Ib91d4d0f416a1cbd298f4a010e8264a3e8f0ee16
2018-11-01 16:32:58 +00:00
Howard Ro
1260bb4739 Manifest for default implementation of stats hal
Bug: 118509430
Test: builds successfully. logging tests will be conducted once client
implementation is in.

Change-Id: If0085273b43316bce4c266829012a618193d6bd8
2018-10-31 15:08:51 -07:00
Colin Cross
bd26d4363f Convert statsd and incidentd to Android.bp
am: 05c9e5c24a

Change-Id: Ic5b6353777baf66eb77f7fc6b9846fa4adbca186
2018-10-31 14:12:36 -07:00
Olivier Gaillard
936b54b261 Merge "Update the binder calls atom to track the calling app of a binder call." 2018-10-31 16:00:27 +00:00
Colin Cross
05c9e5c24a Convert statsd and incidentd to Android.bp
See build/soong/README.md for more information.

Test: m checkbuild
Merged-In: I8c80878cec46c9299234c6f4c1e0e19efc0a221e
Change-Id: I8c80878cec46c9299234c6f4c1e0e19efc0a221e
2018-10-31 08:01:59 -07:00
Colin Cross
b7fddfc6a6 Merge "Convert statsd and incidentd to Android.bp" 2018-10-30 23:28:28 +00:00
Colin Cross
bc2dc31c4d Convert statsd and incidentd to Android.bp
See build/soong/README.md for more information.

Fixes: 110563449
Test: m checkbuild
Change-Id: I8c80878cec46c9299234c6f4c1e0e19efc0a221e
2018-10-30 13:30:23 -07:00
Olivier Gaillard
720ec5baf9 Update the binder calls atom to track the calling app of a binder call.
Test: manual
Change-Id: I0c63298e56b85eb69ac7f909c12414cc43a964a9
2018-10-30 17:33:00 +00:00
TreeHugger Robot
0b4e17bee2 Merge "idmap2: initial code drop" 2018-10-30 17:00:24 +00:00
Misha Wagner
5a51e004b1 Add atom to StatsCompanionService for retrieving per-thread CPU usage data
N.B.: This calls CpuThreadProcReader synchronously in
StatsCompanionService::pullData. This call takes approximately 50.079ms on a
Pixel 2.

Bug: 111534779
Test: `adb shell cmd stats pull-source 10035` returns correct data

Change-Id: I6fe6d178e28669f10ba9c076cbf19dc443d171c9
2018-10-30 15:34:07 +00:00
Mårten Kongstad
0275123eef idmap2: initial code drop
idmap2 is a reboot of the idmap project. The project aims to

  - use modern C++
  - greatly improve test and debug support
  - interface towards AssetManager2 (instead of AssetManager)
  - provide a solid foundation to add support for new features

To make it easier to verify correctness, this first version of idmap2 is
feature equivalent to idmap. Later versions will add support for new
features such as <overlayable>.

Bug: 78815803
Test: make idmap2_tests
Change-Id: I1d806dc875a493e730ab55d2fdb027618e586d16
2018-10-30 04:37:41 -07:00
Chenjie Yu
c3c30c0b30 add drop byte info in statsdstats
Test: manual test and unit test
Change-Id: I8675b399f83b2bb43be4fc2e6c8f28db37b8863b
2018-10-29 21:48:38 -07:00
TreeHugger Robot
761b0bd7c9 Merge "Adding tests and move launcher.proto" 2018-10-29 21:52:52 +00:00
Gopinath Elanchezhian
4e57b2c35c Merge "Add statsdprotolite in Android.bp" am: f469bd2f68
am: 748b449887

Change-Id: I94293a1cd5ca457224eccaf98ee1c265a3c3ce06
2018-10-29 11:47:54 -07:00
Gopinath Elanchezhian
f469bd2f68 Merge "Add statsdprotolite in Android.bp" 2018-10-29 17:14:46 +00:00
Rafal Slawik
bf67d07be2 Record process start time for native processes
Process start time is recorded to detect whether two memory samples
come from the same process and how long the process was alive.

Bug: 118249210
Test: atest MemoryStatUtilTest and manually verified that data is in
statsd report

Change-Id: I7f49cd8bfc81c5e7e70e4f8b49729632eeec5798
2018-10-29 14:36:37 +00:00
gopinath
49dd4c02e8 Merge "allow statsd pull based on event trigger." am: bd4ae82f66
am: 161b7028a4

Change-Id: Ibf8b0ac7e6f5f0deffb2e0c687eb9aeef4da9423
2018-10-26 21:39:51 -07:00
gopinath
17593b0395 Add statsdprotolite in Android.bp
Bug: b/118482428

Test: make statsdprotolite
Change-Id: I5c27f3017be043b6809699158a94b32155163535
Merged-In: I857957e816506162496192fe0ad1da4dbdfd4efd
2018-10-26 23:16:39 +00:00
gopinath
9d426e141d allow statsd pull based on event trigger.
Bug: b/118482428

Test: make statsdprotolite
Change-Id: I5a40d5e6a409b65ca1639029d5da83c327ed1798
Merged-In: Ia15b1f209945f022edffb9ec5d673317d55d9e4f
2018-10-26 22:22:51 +00:00
Yao Chen
8c43386145 Adding tests and move launcher.proto
Test: statsd_test

Change-Id: I0655f70098d78f8f6dc9381921c4a687d2195bd4
2018-10-26 10:28:05 -07:00
Hyunyoung Song
4f8f072f7c Merge "Add launcher.proto to atoms.proto" 2018-10-26 04:57:58 +00:00
TreeHugger Robot
1cfaeed73b Merge "Log uid info to KeyValuePairAtom" 2018-10-25 23:57:34 +00:00
Hyunyoung Song
c6d6b77ddc Add launcher.proto to atoms.proto
Test: blank
Bug: 113043444

Change-Id: I7e0e72cdf68dcf195188dbab8fe2567fcbd5964e
2018-10-25 15:22:12 -07:00
Howard Ro
1a2a399419 Log uid info to KeyValuePairAtom
Bug: 118153464
Test: verified the data via print-logs in logcat
Change-Id: I74fcde1a7d2b2c37eb6d63e40fb5cb40ebb9cff9
2018-10-25 13:26:42 -07:00
TreeHugger Robot
03b8c0f49c Merge "Allow atoms to log fields in bytes format." 2018-10-25 19:04:17 +00:00
Wei Wang
86d0d9d8e5 Merge "BootAnimation: Use system log" 2018-10-25 03:56:48 +00:00
TreeHugger Robot
c242dac241 Merge "[Screenshot] Accept data space and pixel format for screenshot." 2018-10-25 03:29:31 +00:00
TreeHugger Robot
fd5117d680 Merge "make testdrive log message less verbose" 2018-10-25 02:16:35 +00:00
Yangster-mac
308ea0cda8 Pull process cpu stats atom.
Bug: 113353129
Test: manual + statsd_test

Change-Id: I4dde181e328506c119aa3994b1a243d7a9d61e25
2018-10-25 01:01:16 +00:00
Chenjie Yu
328432d887 make testdrive log message less verbose
1. Use custom TestDriveFormatter for log output.
It should be easy to fine tune it from here.
2. Fine tune logging level

Now the output looks like this:

./out/host/linux-x86/bin/statsd_testdrive 10
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.Utf8$UnsafeProcessor (file:/android2/master/out/host/linux-x86/framework/statsd_testdrive.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.Utf8$UnsafeProcessor
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Pushed the following config to statsd:
id: 12345
event_metric {
  id: 1111
  what: 1234567
}
atom_matcher {
  id: 1234567
  simple_atom_matcher {
    atom_id: 10
  }
}

......

hash_strings_in_metric_report: false

Now please play with the device to trigger the event. All events should be dumped after 1 min ...
Success!
Got following metric data dump:
metric_id: 1111
event_metrics {
  data {

.....

        state: RELEASE
      }
    }
  }
}

Test: manual test
Change-Id: I9fcb09bbd1663f0eae85d428ed45a3b139f62782
2018-10-24 16:26:05 -07:00