From 049f7b3332d6197d468647f68d892c2f28f1b79d Mon Sep 17 00:00:00 2001 From: David Friedman Date: Fri, 4 Dec 2015 15:35:54 -0800 Subject: [PATCH] Docs: Add Audio (OpenSL ES) section to NDK docs on DAC. Bug: 21405791 Change-Id: I6c9ba5fa12222793fb60ab2cb3497e59b14ed1d7 --- docs/html/ndk/guides/audio/basics.jd | 125 +++ docs/html/ndk/guides/audio/index.jd | 15 + .../ndk/guides/audio/opensl-for-android.jd | 881 ++++++++++++++++++ docs/html/ndk/guides/guides_toc.cs | 10 + 4 files changed, 1031 insertions(+) create mode 100644 docs/html/ndk/guides/audio/basics.jd create mode 100644 docs/html/ndk/guides/audio/index.jd create mode 100644 docs/html/ndk/guides/audio/opensl-for-android.jd diff --git a/docs/html/ndk/guides/audio/basics.jd b/docs/html/ndk/guides/audio/basics.jd new file mode 100644 index 0000000000000..a5f0ff5fe0af6 --- /dev/null +++ b/docs/html/ndk/guides/audio/basics.jd @@ -0,0 +1,125 @@ +page.title=OpenSL ES™ Basics +@jd:body + +
+ +
+ + +

+The Khronos Group's OpenSL ES standard exposes audio features +similar to those in the {@link android.media.MediaPlayer} and {@link android.media.MediaRecorder} +APIs in the Android Java framework. OpenSL ES provides a C language interface as well as +C++ bindings, allowing you to call it from code written in either language. +

+ +

+This page describes how to add these audio APIs into your app's source code, and how to incorporate +them into the build process. +

+ +

Adding OpenSL ES to your App

+ +

+You can call OpenSL ES from both C and C++ code. To add the core OpenSL ES +feature set to your app, include the {@code OpenSLES.h} header file: + +

+
+#include <SLES/OpenSLES.h>
+
+ +

+To add the OpenSL ES +Android extensions as well, include the {@code OpenSLES_Android.h} header file: +

+
+#include <SLES/OpenSLES_Android.h>
+
+ + +

Building and Debugging

+ +

+You can incorporate OpenSL ES into your build by specifying it in the +{@code Android.mk} file that serves as one of the +NDK build system's makefiles. Add the following line to +{@code Android.mk}: +

+ +
+LOCAL_LDLIBS += -lOpenSLES
+
+ +

+For robust debugging, we recommend that you examine the {@code SLresult} value that most of +the OpenSL ES APIs return. You can use +asserts +or more advanced error-handling logic for debugging; neither offers +an inherent advantage for working with OpenSL ES, although one or the other might be more suitable +for a given use case. +

+ +

+We use asserts in our examples, because +they help catch unrealistic conditions that would indicate a coding error. We have used explicit +error handling for other conditions more likely to occur in production. +

+ +

+Many API errors result in a log entry, in addition to a non-zero result code. Such log entries +can provide additional detail that proves especially useful for relatively complex APIs such as + +{@code Engine::CreateAudioPlayer}. +

+ +

+You can view the log either from the command line or from Android Studio. To examine the log from +the command line, type the following: +

+ +
+$ adb logcat
+
+ +

+To examine the log from Android Studio, either click the Logcat tab in the +Debug +window, or click the Devices | logcat tab in the +Android DDMS +window. +

+ +

Samples

+ +

+Supported and tested example code that you can use as a model for your own code resides both locally +and on GitHub. The local examples are located in +{@code platforms/android-9/samples/native-audio/}, under your NDK root installation directory. +On GitHub, they are available from the +{@code android-ndk} +repository, in the + +{@code audio-echo} and + +{@code native-audio} directories. +

+

The Android NDK implementation of OpenSL ES differs +from the reference specification for OpenSL ES 1.0.1 in a number of respects. +These differences are an important reason as to why sample code that +you copy directly from the OpenSL ES reference specification may not work in your +Android app. +

+

+For more information on differences between the reference specification and the +Android implementation, see + +OpenSL ES™ for Android. diff --git a/docs/html/ndk/guides/audio/index.jd b/docs/html/ndk/guides/audio/index.jd new file mode 100644 index 0000000000000..1767337359904 --- /dev/null +++ b/docs/html/ndk/guides/audio/index.jd @@ -0,0 +1,15 @@ +page.title=NDK Audio: Open SL™ ES +@jd:body + +

The NDK package includes an Android-specific implementation of the +Open SL ES API +specification from the Khronos Group. This library +allows you to use C or C++ to implement high-performance, low-latency audio in your game or other +demanding app.

+ +

This section begins by providing some +basic information about the API, including how +to incorporate it into your app. It then explains what you need to know about the +Android-specific implementation +of OpenSL ES, focusing on differences between this implementation and the reference specification. +

\ No newline at end of file diff --git a/docs/html/ndk/guides/audio/opensl-for-android.jd b/docs/html/ndk/guides/audio/opensl-for-android.jd new file mode 100644 index 0000000000000..763da5a059e78 --- /dev/null +++ b/docs/html/ndk/guides/audio/opensl-for-android.jd @@ -0,0 +1,881 @@ +page.title=Native Audio: OpenSL ES™ for Android +@jd:body + +
+ +
+ +

+This page provides details about how the NDK implementation of OpenSL ES™ differs +from the reference specification for OpenSL ES 1.0.1. When using sample code from the +specification, you may need to modify it to work on Android. +

+ +

Features Inherited from the Reference Specification

+ +

+The Android NDK implementation of OpenSL ES inherits much of the feature set from +the reference specification, although with certain limitations. +

+ +

Global entry points

+ +

+OpenSL ES for Android supports all of the global entry points in the Android specification. +These entry points include: +

+ + + +

Objects and interfaces

+ +

+Table 1 shows which objects and interfaces the Android NDK implementation of +OpenSL ES supports. Green cells indicate features available in this implementation. +

+ +

+ Table 1. Android NDK support for objects and interfaces.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeatureAudio playerAudio recorderEngineOutput mix
Bass boostYesNoNoYes
Buffer queueYesNoNoNo
Dynamic interface managementYesYesYesYes
Effect sendYesNoNoNo
EngineNoNoYesNo
Environmental reverbNoNoNoYes
EqualizerYesNoNoYes
Metadata extractionYes: Decode to PCMNoNoNo
Mute soloYesNoNoNo
ObjectYesYesYesYes
PlayYesNoNoNo
Playback rateYesNoNoNo
Prefetch statusYesNoNoNo
Preset reverbNoNoNoYes
RecordNoYesNoNo
SeekYesNoNoNo
VirtualizerYesNoNoYes
VolumeYesNoNoNo
Buffer queue data locatorYes: SourceNoNoNo
I/O device data locatorNoYes: SourceNoNo
Output mix locatorYes: SinkNoNoNo
URI data locatorYes: SourceNoNoNo
+ +The next section explains limitations of some of these features. + +

Limitations

+ +

+Certain limitations apply to the features in Table 1. These limitations +represent differences from the reference specification. The rest of this section provides +information about these differences.

+ +

Dynamic interface management

+ +

+OpenSL ES for Android does not support {@code RemoveInterface} or +{@code ResumeInterface}. +

+ +

Effect combinations: environment reverb and preset reverb

+ +

+You cannot have both environmental reverb and preset reverb on the same output mix. +

+

+The platform might ignore effect requests if it estimates that the +CPU load would be too high. +

+ +

Effect send

+ +

+SetSendLevel() supports a single send level per audio player. +

+ +

Environmental reverb

+ +

+Environmental reverb does not support the reflectionsDelay, +reflectionsLevel, or reverbDelay fields of +the SLEnvironmentalReverbSettings struct. +

+ +

MIME data format

+ +

+You can use the MIME data format only with the URI data locator, and only for an audio +player. You cannot use this data format for an audio recorder. +

+

+The Android implementation of OpenSL ES requires you to initialize mimeType +to either NULL or a valid UTF-8 string. You must also initialize +containerType to a valid value. +In the absence of other considerations, such as portability to other +implementations, or content format that an app cannot identify by header, +we recommend that you +set mimeType to NULL and containerType +to SL_CONTAINERTYPE_UNSPECIFIED. +

+

+OpenSL ES for Android supports the following audio formats, so long as the +Android platform supports them as well:

+ + + +

+For a list of audio formats that Android supports, see +Supported Media Formats. +

+ +

+The following limitations apply to handling of these and other formats in this +implementation of OpenSL ES: +

+ + + +

Object-related methods

+ +

+OpenSL ES for Android does not support the following methods for manipulating objects: +

+ + + +

PCM data format

+ +

+PCM is the only data format you can use with buffer queues. Supported PCM +playback configurations have the following characteristics: +

+ + + +

+The configurations that OpenSL ES for Android supports for recording are +device-dependent; usually, 16,000 Hz mono 16-bit signed is available regardless of device. +

+

+The value of the samplesPerSec field is in units of milliHz, despite the misleading +name. To avoid accidentally using the wrong value, we recommend that you initialize this field using +one of the symbolic constants defined for this purpose, such as {@code SL_SAMPLINGRATE_44_1}. +

+

+Android 5.0 (API level 21) and above support floating-point data. +

+ +

Playback rate

+ +

+An OpenSL ES playback rate indicates the speed at which an +object presents data, expressed in thousandths of normal speed, or per mille. For example, +a playback rate of 1,000 per mille is 1,000/1,000, or normal speed. +A rate range is a closed interval that expresses possible rate ranges. +

+ +

+Support for playback-rate ranges and other capabilities may vary depending +on the platform version and implementation. Your app can determine these capabilities at runtime by +using PlaybackRate::GetRateRange() or +PlaybackRate::GetCapabilitiesOfRate() to query the device. +

+ +

+A device typically supports the same rate range for a data source in PCM format, and a unity rate +range of 1000 per mille to 1000 per mille for other formats: that is, the unity rate range is +effectively a single value. +

+ +

Record

+ +

+OpenSL ES for Android does not support the SL_RECORDEVENT_HEADATLIMIT +or SL_RECORDEVENT_HEADMOVING events. +

+ +

Seek

+ +

+The SetLoop() method enables whole-file looping. To enable looping, +set the startPos parameter to 0, and the value of the endPos parameter +to SL_TIME_UNKNOWN. +

+ +

Buffer queue data locator

+ +

+An audio player or recorder with a data locator for a buffer queue supports PCM data format only. +

+ +

I/O Device data locator

+ +

+OpenSL ES for Android only supports use of an I/O device data locator when you have +specified the locator as the data source for Engine::CreateAudioRecorder(). +Initialize the device data locator using the values contained in the following code snippet. +

+ +
+SLDataLocator_IODevice loc_dev =
+  {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT,
+  SL_DEFAULTDEVICEID_AUDIOINPUT, NULL};
+
+ +

URI data locator

+ +

+OpenSL ES for Android can only use the URI data locator with MIME data format, +and only for an audio player. You cannot use this data format for an audio recorder. It supports +{@code http:} and {@code file:} schemes. It does not support other schemes, such as {@code https:}, +{@code ftp:}, or +{@code content:}. +

+ +

+We have not verified support for {@code rtsp:} with audio on the Android platform. +

+ +

Android Extensions

+ +

+OpenSL ES for Android extends the reference OpenSL ES specification to make it compatible with +Android, and to take advantage of the power and flexibility of the Android platform. +

+ +

+The definition of the API for the Android extensions resides in OpenSLES_Android.h +and the header files that it includes. Consult {@code OpenSLES_Android.h} +for details about these extensions. This file is located under your installation root, in the +{@code platforms/android-<version>/<abi>/include/SLES} directory. Unless otherwise +noted, all interfaces are explicit. +

+ +

+These extensions limit your application's portability to +other OpenSL ES implementations, because they are Android-specific. You can mitigate this issue by +avoiding use of the extensions or by using {@code #ifdef} to exclude them at compile time. +

+ +

+Table 2 shows the Android-specific interfaces and data locators that Android OpenSL ES supports +for each object type. Green cells indicate interfaces and data locators available for each +object type. +

+ +

+ Table 2. Interfaces and data locators, by object type.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeatureAudio playerAudio recorderEngineOutput mix
Android buffer queueYes: Source (decode)NoNoNo
Android configurationYesYesNoNo
Android effectYesNoNoYes
Android effect capabilitiesNoNoYesNo
Android effect sendYesNoNoNo
Android simple buffer queueYes: Source (playback) or sink (decode)YesNoNo
Android buffer queue data locatorYes: Source (decode)NoNoNo
Android file descriptor data locatorYes: SourceNoNoNo
Android simple buffer queue data locatorYes: Source (playback) or sink (decode)Yes: SinkNoNo
+ +

Android configuration interface

+ +

+The Android configuration interface provides a means to set +platform-specific parameters for objects. This interface is different from other OpenSL ES +1.0.1 interfaces in that your app can use it before instantiating the corresponding object; thus, +you can configure the object before instantiating it. The +{@code OpenSLES_AndroidConfiguration.h} header file, which resides at +{@code platforms/android-<version>/<abi>/include/SLES}, +documents the following available configuration keys and values: +

+ + + +

+The following code snippet shows an example of how to set the Android audio stream type on an audio +player: +

+ +
+// CreateAudioPlayer and specify SL_IID_ANDROIDCONFIGURATION
+// in the required interface ID array. Do not realize player yet.
+// ...
+SLAndroidConfigurationItf playerConfig;
+result = (*playerObject)->GetInterface(playerObject,
+    SL_IID_ANDROIDCONFIGURATION, &playerConfig);
+assert(SL_RESULT_SUCCESS == result);
+SLint32 streamType = SL_ANDROID_STREAM_ALARM;
+result = (*playerConfig)->SetConfiguration(playerConfig,
+    SL_ANDROID_KEY_STREAM_TYPE, &streamType, sizeof(SLint32));
+assert(SL_RESULT_SUCCESS == result);
+// ...
+// Now realize the player here.
+
+ +

+You can use similar code to configure the preset for an audio recorder: +

+
+// ... obtain the configuration interface as the first four lines above, then:
+SLuint32 presetValue = SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION;
+result = (*playerConfig)->SetConfiguration(playerConfig,
+    RECORDING_PRESET, &presetValue, sizeof(SLuint32));
+
+ +

Android effects interfaces

+ +

+Android's effect, effect send, and effect capabilities interfaces provide +a generic mechanism for an application to query and use device-specific +audio effects. Device manufacturers should document any available device-specific audio effects +that they provide. +

+ +

Android file descriptor data locator

+ +

+The Android file descriptor data locator permits you to specify the source for an +audio player as an open file descriptor with read access. The data format must be MIME. +

+

+This extension is especially useful in conjunction with the native asset manager, because +the app reads assets from the APK via a file descriptor. +

+ +

Android simple buffer queue data locator and interface

+ +

+The Android simple buffer queue data locator and interface are +identical to those in the OpenSL ES 1.0.1 reference specification, with two exceptions: You +can also use Android simple buffer queues with both audio players and audio recorders. Also, PCM +is the only data format you can use with these queues. +In the reference specification, buffer queues are for audio players only, but +compatible with data formats beyond PCM. +

+

+For recording, your app should enqueue empty buffers. When a registered callback sends +notification that the system has finished writing data to the buffer, the app can +read the buffer. +

+

+Playback works in the same way. For future source code +compatibility, however, we suggest that applications use Android simple +buffer queues instead of OpenSL ES 1.0.1 buffer queues. +

+ +

Dynamic interfaces at object creation

+ +

+For convenience, the Android implementation of OpenSL ES 1.0.1 +permits your app to specify dynamic interfaces when it instantiates an object. +This is an alternative to using DynamicInterfaceManagement::AddInterface() +to add these interfaces after instantiation. +

+ +

Buffer queue behavior

+ +

+The Android implementation does not include the +reference specification's requirement that the play cursor return to the beginning +of the currently playing buffer when playback enters the {@code SL_PLAYSTATE_STOPPED} +state. This implementation can conform to that behavior, or it can leave the location of the play +cursor unchanged. +

+ +

+As a result, your app cannot assume that either behavior occurs. Therefore, +you should explicitly call the BufferQueue::Clear() method after a transition to +SL_PLAYSTATE_STOPPED. Doing so sets the buffer queue to a known state. +

+ +

+Similarly, there is no specification governing whether the trigger for a buffer queue callback must +be a transition to SL_PLAYSTATE_STOPPED or execution of +BufferQueue::Clear(). Therefore, we recommend against creating a dependency on +one or the other; instead, your app should be able to handle both. +

+ +

Reporting of extensions

+

+There are three methods for querying whether the platform supports the Android extensions. These +methods are: +

+ + + +

+Any of these methods returns ANDROID_SDK_LEVEL_<API-level>, +where {@code API-level} is the platform API level; for example, {@code ANDROID_SDK_LEVEL_23}. +A platform API level of 9 or higher means that the platform supports the extensions. +

+ + +

Decode audio to PCM

+ +

+This section describes a deprecated Android-specific extension to OpenSL ES 1.0.1 +for decoding an encoded stream to PCM without immediate playback. +The table below gives recommendations for use of this extension and alternatives. +

+ + + + + + + + + + + + + + + + + + + + + + +
API levelAlternatives
13 and belowAn open-source codec with a suitable license.
14 to 15An open-source codec with a suitable license.
16 to 20 + The {@link android.media.MediaCodec} class or an open-source codec with a suitable license. +
21 and above + NDK MediaCodec in the {@code <media/NdkMedia*.h>} header files, the + {@link android.media.MediaCodec} class, or an open-source codec with a suitable license. +
+ +

+A standard audio player plays back to an audio device, specifying the output mix as the data sink. +The Android extension differs in that an audio player instead +acts as a decoder if the app has specified the data source either as a URI or as an Android +file descriptor data locator described in MIME data format. In such a case, the data sink is +an Android simple buffer queue data locator with PCM data format. +

+ +

+This feature is primarily intended for games to pre-load their audio assets when changing to a +new game level, similar to the functionality that the {@link android.media.SoundPool} class +provides. +

+ +

+The application should initially enqueue a set of empty buffers in the Android simple +buffer queue. After that, the app fills the buffers with with PCM data. The Android simple +buffer queue callback fires after each buffer is filled. The callback handler processes +the PCM data, re-enqueues the now-empty buffer, and then returns. The application is responsible for +keeping track of decoded buffers; the callback parameter list does not include +sufficient information to indicate which buffer contains data or which buffer to enqueue next. +

+ +

+The data source implicitly reports the end of stream (EOS) by delivering a +SL_PLAYEVENT_HEADATEND event at the end of the stream. After the app has decoded +all of the data it received, it makes no further calls to the Android simple buffer queue callback. +

+

+The sink's PCM data format typically matches that of the encoded data source +with respect to sample rate, channel count, and bit depth. However, you can decode to a different +sample rate, channel count, or bit depth. +For information about a provision to detect the actual PCM format, see +Determining the format of decoded PCM data via metadata. +

+

+OpenSL ES for Android's PCM decoding feature supports pause and initial seek; it does not support +volume control, effects, looping, or playback rate. +

+

+Depending on the platform implementation, decoding may require resources +that cannot be left idle. Therefore, we recommend that you make sure to provide +sufficient numbers of empty PCM buffers; otherwise, the decoder starves. This may happen, +for example, if your app returns from the Android simple buffer queue callback without +enqueueing another empty buffer. The result of decoder starvation is +unspecified, but may include: dropping the decoded +PCM data, pausing the decoding process, or terminating the decoder outright. +

+ +

Note: +To decode an encoded stream to PCM but not play back immediately, for apps running on +Android 4.x (API levels 16–20), we recommend using the {@link android.media.MediaCodec} class. +For new applications running on Android 5.0 (API level 21) or higher, we recommend using the NDK +equivalent, {@code <NdkMedia*.h>}. These header files reside under +the {@code media/} directory, under your installation root. +

+ +

Decode streaming ADTS AAC to PCM

+ +

+An audio player acts as a streaming decoder if the data source is an +Android buffer queue data locator with MIME data format, and the data +sink is an Android simple buffer queue data locator with PCM data format. +Configure the MIME data format as follows: +

+ + + +

+This feature is primarily intended for streaming media applications that +deal with AAC audio but need to perform custom audio processing +prior to playback. Most applications that need to decode audio to PCM +should use the method that Decode audio to PCM describes, +as that method is simpler and handles more audio formats. The technique described +here is a more specialized approach, to be used only if both of these +conditions apply: +

+ + + +

+The application should initially enqueue a set of filled buffers in the Android buffer queue. +Each buffer contains one or more complete ADTS AAC frames. +The Android buffer queue callback fires after each buffer is emptied. +The callback handler should refill and re-enqueue the buffer, and then return. +The application need not keep track of encoded buffers; the callback parameter +list includes sufficient information to indicate which buffer to enqueue next. +The end of stream is explicitly marked by enqueuing an EOS item. +After EOS, no more enqueues are permitted. +

+ +

+We recommend that you make sure to provide full +ADTS AAC buffers, to avoid starving the decoder. This may happen, for example, if your app +returns from the Android buffer queue callback without enqueueing another full buffer. +The result of decoder starvation is unspecified. +

+ +

+In all respects except for the data source, the streaming decode method is the same as +the one that Decode audio to PCM describes. +

+

+Despite the similarity in names, an Android buffer queue is not +the same as an Android simple buffer queue. The streaming decoder +uses both kinds of buffer queues: an Android buffer queue for the ADTS +AAC data source, and an Android simple buffer queue for the PCM data +sink. For more information about the Android simple buffer queue API, see Android +simple buffer queue data locator and interface. +For more information about the Android buffer queue API, see the {@code index.html} file in +the {@code docs/Additional_library_docs/openmaxal/} directory under the installation root. +

+ +

Determining the format of decoded PCM data via metadata

+ +

+The SLMetadataExtractionItf interface is part of the reference specification. +However, the metadata keys that indicate the actual format of decoded PCM data are specific to +Android. The OpenSLES_AndroidMetadata.h header file defines these metadata keys. +This header file resides under your installation root, in the +{@code platforms/android-<version>/<abi>/include/SLES} directory. +

+ +

+The metadata key indices are available immediately after +the Object::Realize() method finishes executing. However, the associated values are not +available until after the app decodes the first encoded data. A good +practice is to query for the key indices in the main thread after calling the {@code +Object::Realize} method, and to read the PCM format metadata values in the Android simple +buffer queue callback handler when calling it for the first time. Consult the +example code in the NDK package +for examples of working with this interface. +

+ +

+Metadata key names are stable, but the key indices are not documented, +and are subject to change. An application should not assume that indices +are persistent across different execution runs, and should not assume that +multiple object instances share indices within the same run. +

+ +

Floating-point data

+ +

+An app running on Android 5.0 (API level 21) and higher can supply data to an AudioPlayer in +single-precision, floating-point format. +

+

+In following example code, the {@code Engine::CreateAudioPlayer} method creates an audio player +that uses floating-point data: +

+ +
+#include <SLES/OpenSLES_Android.h>
+...
+SLAndroidDataFormat_PCM_EX pcm;
+pcm.formatType = SL_ANDROID_DATAFORMAT_PCM_EX;
+pcm.numChannels = 2;
+pcm.sampleRate = SL_SAMPLINGRATE_44_1;
+pcm.bitsPerSample = 32;
+pcm.containerSize = 32;
+pcm.channelMask = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT;
+pcm.endianness = SL_BYTEORDER_LITTLEENDIAN;
+pcm.representation = SL_ANDROID_PCM_REPRESENTATION_FLOAT;
+...
+SLDataSource audiosrc;
+audiosrc.pLocator = ...
+audiosrc.pFormat = &pcm;
+
diff --git a/docs/html/ndk/guides/guides_toc.cs b/docs/html/ndk/guides/guides_toc.cs index 981eb5131facd..4c4c64e2f95f8 100644 --- a/docs/html/ndk/guides/guides_toc.cs +++ b/docs/html/ndk/guides/guides_toc.cs @@ -63,6 +63,16 @@ +
  • +
    + Audio
    + +
    • +