EncodedBuffer is used a lot in incidentd. EncodedBuffer uses malloc
internally to acquire memory. Frequently creating and destroying
EncodedBuffer creates memory fragmentation, leading to high memory
usage after taking an incident report.
Also fixes a few other places with lots of malloc/free operations.
This change:
* Creates a pool of EncodedBuffer in incidentd. The saving is
significant. It reduces EncodedBuffer creation from 3 per section to
3 per report.
* Replaces malloc with mmap inside EncodedBuffer. mmap is guaranteed
to be mem page aligned, so there will be no mem fragmentation after
destroying EncodedBuffer.
* Replaces new with mmap inside TombstoneSection
* Forks a process to execute LogSection, because liblog malloc & free
significant amount of memory
Result:
PSS before taking a report: 1295 KB
PSS after taking a report: 1336 KB
Bug: 150311553
Test: heapprofd
Change-Id: I83bd9c969b751c80b2f42747020799bd85d8aae6
Protobuf 3.9.1 redefines google::protobuf::uint64 from unsigned long
long to uint64_t, which is sometimes unsigned long and sometimes
unsigned long long. Use PRIu64 to print it, and add an implementation
of ProtoOutputStream::write for long.
Bug: 117607748
Test: m checkbuild
Exempt-From-Owner-Approval: approved at https://android-review.googlesource.com/q/Ib2d3f4e17857f8ccbbe342ce6678e76b591df510
Change-Id: Ib2d3f4e17857f8ccbbe342ce6678e76b591df510
Previously, all metrics/configs would deactivate on system server death.
Now, active status is restored.
Bug: 129717537
Test: bit statsd_test:*
Test: libprotoutil_test:ProtoOutputStream*
Change-Id: Idf372457f60a931a2d00176a5eab58c534a25e41
* Add explicit to conversion constructors/operators
Bug: 28341362
Test: make with WITH_TIDY=1 DEFAULT_GLOBAL_TIDY_CHECKS=-*,google-explicit-constructor
Change-Id: I31d45e4c6ff9fa69801a14c341f2da66baf565a0
Protobuf 3.5.2 redefines google::protobuf::uint64 from unsigned long
long to uint64_t, which is sometimes unsigned long and sometimes
unsigned long long. Use PRIu64 to print it, and add an implementation
of ProtoOutputStream::write for long.
Bug: 117607748
Test: m checkbuild
Change-Id: I48912f58e2f63f4a98c3aefaa4f78cc2c719bb68
There must be equal number of start/end calls with right token given in
order, if used in wrong order, ProtoOutputStream will have empty output.
Also refactor Android.bp, so the gtest is standalone unit test, it used
to require compile/push libprotoutil.so to device.
Bug: 77342154
Test: atest libprotoutil_test
Change-Id: I0011bbab34c04cb38164d2ed21cd818d52a2ecf9
This section simply gzip a large file and stores result in GZippedFileProto
This greatly improves the size, before gzip, the last kmsg size ~500KB,
after gzip the proto size is ~60KB.
Bug: 73354384
Test: atest incidentd_test and manual on device test
Change-Id: I9bfc2cf07384487671edbffb5f0bd8495608fea6
It is very unlikely the protobuf changes the value in descriptor.h,
and if defines an extra mapping, there are several places to maintain:
1. java-stream,
2. cpp-stream,
3. ProtoOutputStream.java
4. ProtoOutputStream.cpp
5. Privacy.h (GetFieldId)
6. StatsLog to generate field id (type << 32 + field number)
Therefore use the current value in descriptor.h seems reasonable unless
they change that, very very unlikely, they probably will just add new
types, and deprect the existing ones like Group.
Test: test output of dumpsys proto
Change-Id: I6e150ab427851dd3b5dd55d3b273deeed7a0963c
1. Split the parsers to its own file to prevent all the parsers in one
gaint file.
2. Completely get rid of protobuf-cpp-full in incident_helper, use
ProtoOutputStream and cppstream instead, the incident_helper binary is
reduced from ~500K to ~113K.
3. Write data to protobuf even its values are zero/default, the reason
is for example we have a repeated int32 orders = 1; and people
explicitly append 0 so the total repeated field has 10 values, if zero
is not written to serialized data, this repeated field will only have 9
values which is not what we want at first place. This also aligns with
the default protobuf serialization behavior in incident_helper_test.
4. Use Android.bp for protoutil lib since it is not able to depend on
libs compiled by .mk file, it works the other way.
5. Add a new custom message option for streaming_proto, if specified,
the cppstream will create extra metadata to get field ids by field name.
A Table class is created in incident_helper to use it.
Bug: 67860303
Test: unit tested as well as on device test
Change-Id: I8e136fd15f343a4a623d20910ec64b622b478a3e
1. Change the APIs of ProtoOutputStream to be able to write bytes
2. Fix the tests in incidentd, stdout is closed so can't capture the
value, use temporaryFile instead.
Test: N/A
Change-Id: Ibc31f2efd068afc6c06188d92f57ca5a754c3683
The in-place algorithm in ProtoOutputStream will have at most 2 copies
of possible nested message values incidentd is getting which makes it
more ram-efficient, and minimize allocations.
Bug: 65641021
Test: the units pass
Change-Id: Ic3fe99b7e27895bd613a129ba0f12ccfca4af317
Implement ProtoOutputStream.h to write protobuf format data.
Usage of dumping proto:
ProtoOutputStream proto(fd);
proto.write(fieldId, value1); // dump a single value
// start to dump a message:
long long token = proto.start(messageFieldId);
proto.write(nestedField1, nestedValue1);
...
proto.end(token);
fieldId will be generated by protoc-cpp plugin(TBD). It is an encoded uint64_t
value, with 0 - 32 bits as its proto number, 33 - 40 bits reserved for
field type, int32, bool, string, message, etc. and 41 - 43 bits for
single, repeated or packed type. Currently packed field is not
supported.
Bug: 65641021
Test: N/A, need to wait for protoc-cpp plugin and will test in
incident_helper
Change-Id: Ic188615b950235aae0edeee4876b78d31feb5619
Extract protobuf class out and creates EncodedBuffer class
which holds protobuf data.
Next step is to create a ProtoOutputStream and let incident helper
adapt the change as well.
please see frameworks/base/core/java/android/util/proto
Bug: 65641021
Test: unit tested
Change-Id: I0dd343b2e62d60f091c8f857fae3452ec8da6b96
