From 479cc54307616b079ab37a3c2d083379ccfd3fb9 Mon Sep 17 00:00:00 2001 From: Lajos Molnar Date: Wed, 30 Oct 2013 14:02:27 -0700 Subject: [PATCH] MediaCodec: expand documentation on flush/EOS handling and adaptive playback Bug: 11216198 Change-Id: I929954ec381511b9b97e53a91ff6fdff75ca1382 --- media/java/android/media/MediaCodec.java | 64 ++++++++++++++++++++++-- 1 file changed, 59 insertions(+), 5 deletions(-) diff --git a/media/java/android/media/MediaCodec.java b/media/java/android/media/MediaCodec.java index 7bf9e982ed0a2..96e6ab9717a45 100644 --- a/media/java/android/media/MediaCodec.java +++ b/media/java/android/media/MediaCodec.java @@ -94,9 +94,17 @@ import java.util.Map; * and {@link #dequeueOutputBuffer} then transfer ownership from the codec * to the client.

* The client is not required to resubmit/release buffers immediately - * to the codec, the sample code above simply does this for simplicity's sake.

+ * to the codec, the sample code above simply does this for simplicity's sake. + * Nonetheless, it is possible that a codec may hold off on generating + * output buffers until all outstanding buffers have been + * released/resubmitted. + *

* Once the client has an input buffer available it can fill it with data - * and submit it it to the codec via a call to {@link #queueInputBuffer}.

+ * and submit it it to the codec via a call to {@link #queueInputBuffer}. + * Do not submit multiple input buffers with the same timestamp (unless + * it is codec-specific data marked as such using the flag + * {@link #BUFFER_FLAG_CODEC_CONFIG}). + *

* The codec in turn will return an output buffer to the client in response * to {@link #dequeueOutputBuffer}. After the output buffer has been processed * a call to {@link #releaseOutputBuffer} will return it to the codec. @@ -128,18 +136,64 @@ import java.util.Map; * {@link #queueInputBuffer}. The codec will continue to return output buffers * until it eventually signals the end of the output stream by specifying * the same flag ({@link #BUFFER_FLAG_END_OF_STREAM}) on the BufferInfo returned in - * {@link #dequeueOutputBuffer}. + * {@link #dequeueOutputBuffer}. Do not submit additional input buffers after + * signaling the end of the input stream, unless the codec has been flushed, + * or stopped and restarted. *

+ *

Seeking & Adaptive Playback Support

+ * + * You can check if a decoder supports adaptive playback via {@link + * MediaCodecInfo.CodecCapabilities#isFeatureSupported}. Adaptive playback + * is only supported if you configure the codec to decode onto a {@link + * android.view.Surface}. + * + *

For decoders that do not support adaptive playback (including + * when not decoding onto a Surface)

+ * * In order to start decoding data that's not adjacent to previously submitted - * data (i.e. after a seek) it is necessary to {@link #flush} the decoder. + * data (i.e. after a seek) one must {@link #flush} the decoder. * Any input or output buffers the client may own at the point of the flush are * immediately revoked, i.e. after a call to {@link #flush} the client does not * own any buffers anymore. + *

+ * It is important that the input data after a flush starts at a suitable + * stream boundary. The first frame must be able to be decoded completely on + * its own (for most codecs this means an I-frame), and that no frames should + * refer to frames before that first new frame. * Note that the format of the data submitted after a flush must not change, * flush does not support format discontinuities, - * for this a full {@link #stop}, {@link #configure}, {@link #start} + * for this a full {@link #stop}, {@link #configure configure()}, {@link #start} * cycle is necessary. * + *

For decoders that support adaptive playback

+ * + * In order to start decoding data that's not adjacent to previously submitted + * data (i.e. after a seek) it is not necessary to {@link #flush} the + * decoder. + *

+ * It is still important that the input data after the discontinuity starts + * at a suitable stream boundary (e.g. I-frame), and that no new frames refer + * to frames before the first frame of the new input data segment. + *

+ * For some video formats it is also possible to change the picture size + * mid-stream. To do this for H.264, the new Sequence Parameter Set (SPS) and + * Picture Parameter Set (PPS) values must be packaged together with an + * Instantaneous Decoder Refresh (IDR) frame in a single buffer, which then + * can be enqueued as a regular input buffer. + * The client will receive an {@link #INFO_OUTPUT_FORMAT_CHANGED} return + * value from {@link #dequeueOutputBuffer dequeueOutputBuffer()} or + * {@link Callback#onOutputBufferAvailable onOutputBufferAvailable()} + * just after the picture-size change takes place and before any + * frames with the new size have been returned. + *

+ * Be careful when calling {@link #flush} shortly after you have changed + * the picture size. If you have not received confirmation of the picture + * size change, you will need to repeat the request for the new picture size. + * E.g. for H.264 you will need to prepend the PPS/SPS to the new IDR + * frame to ensure that the codec receives the picture size change request. + * + *

States and error handling

+ * *

During its life, a codec conceptually exists in one of the following states: * Initialized, Configured, Executing, Error, Uninitialized, (omitting transitory states * between them). When created by one of the factory methods,