+The Khronos Group's OpenSL ES standard exposes audio features +similar to those in the {@link android.media.MediaPlayer} and {@link android.media.MediaRecorder} +APIs in the Android Java framework. OpenSL ES provides a C language interface as well as +C++ bindings, allowing you to call it from code written in either language. +
+ ++This page describes how to add these audio APIs into your app's source code, and how to incorporate +them into the build process. +
+ ++You can call OpenSL ES from both C and C++ code. To add the core OpenSL ES +feature set to your app, include the {@code OpenSLES.h} header file: + +
++#include <SLES/OpenSLES.h> ++ +
+To add the OpenSL ES +Android extensions as well, include the {@code OpenSLES_Android.h} header file: +
++#include <SLES/OpenSLES_Android.h> ++ + +
+You can incorporate OpenSL ES into your build by specifying it in the +{@code Android.mk} file that serves as one of the +NDK build system's makefiles. Add the following line to +{@code Android.mk}: +
+ ++LOCAL_LDLIBS += -lOpenSLES ++ +
+For robust debugging, we recommend that you examine the {@code SLresult} value that most of +the OpenSL ES APIs return. You can use +asserts +or more advanced error-handling logic for debugging; neither offers +an inherent advantage for working with OpenSL ES, although one or the other might be more suitable +for a given use case. +
+ ++We use asserts in our examples, because +they help catch unrealistic conditions that would indicate a coding error. We have used explicit +error handling for other conditions more likely to occur in production. +
+ ++Many API errors result in a log entry, in addition to a non-zero result code. Such log entries +can provide additional detail that proves especially useful for relatively complex APIs such as + +{@code Engine::CreateAudioPlayer}. +
+ ++You can view the log either from the command line or from Android Studio. To examine the log from +the command line, type the following: +
+ ++$ adb logcat ++ +
+To examine the log from Android Studio, either click the Logcat tab in the +Debug +window, or click the Devices | logcat tab in the +Android DDMS +window. +
+ ++Supported and tested example code that you can use as a model for your own code resides both locally +and on GitHub. The local examples are located in +{@code platforms/android-9/samples/native-audio/}, under your NDK root installation directory. +On GitHub, they are available from the +{@code android-ndk} +repository, in the + +{@code audio-echo} and + +{@code native-audio} directories. +
+The Android NDK implementation of OpenSL ES differs +from the reference specification for OpenSL ES 1.0.1 in a number of respects. +These differences are an important reason as to why sample code that +you copy directly from the OpenSL ES reference specification may not work in your +Android app. +
++For more information on differences between the reference specification and the +Android implementation, see + +OpenSL ES™ for Android. diff --git a/docs/html/ndk/guides/audio/index.jd b/docs/html/ndk/guides/audio/index.jd new file mode 100644 index 0000000000000..1767337359904 --- /dev/null +++ b/docs/html/ndk/guides/audio/index.jd @@ -0,0 +1,15 @@ +page.title=NDK Audio: Open SL™ ES +@jd:body + +
The NDK package includes an Android-specific implementation of the +Open SL ES API +specification from the Khronos Group. This library +allows you to use C or C++ to implement high-performance, low-latency audio in your game or other +demanding app.
+ +This section begins by providing some +basic information about the API, including how +to incorporate it into your app. It then explains what you need to know about the +Android-specific implementation +of OpenSL ES, focusing on differences between this implementation and the reference specification. +
\ No newline at end of file diff --git a/docs/html/ndk/guides/audio/opensl-for-android.jd b/docs/html/ndk/guides/audio/opensl-for-android.jd new file mode 100644 index 0000000000000..763da5a059e78 --- /dev/null +++ b/docs/html/ndk/guides/audio/opensl-for-android.jd @@ -0,0 +1,881 @@ +page.title=Native Audio: OpenSL ES™ for Android +@jd:body + ++This page provides details about how the NDK implementation of OpenSL ES™ differs +from the reference specification for OpenSL ES 1.0.1. When using sample code from the +specification, you may need to modify it to work on Android. +
+ ++The Android NDK implementation of OpenSL ES inherits much of the feature set from +the reference specification, although with certain limitations. +
+ ++OpenSL ES for Android supports all of the global entry points in the Android specification. +These entry points include: +
+ ++Table 1 shows which objects and interfaces the Android NDK implementation of +OpenSL ES supports. Green cells indicate features available in this implementation. +
+ ++ Table 1. Android NDK support for objects and interfaces.
+| Feature | +Audio player | +Audio recorder | +Engine | +Output mix | +
|---|---|---|---|---|
| Bass boost | +Yes | +No | +No | +Yes | +
| Buffer queue | +Yes | +No | +No | +No | +
| Dynamic interface management | +Yes | +Yes | +Yes | +Yes | +
| Effect send | +Yes | +No | +No | +No | +
| Engine | +No | +No | +Yes | +No | +
| Environmental reverb | +No | +No | +No | +Yes | +
| Equalizer | +Yes | +No | +No | +Yes | +
| Metadata extraction | +Yes: Decode to PCM | +No | +No | +No | +
| Mute solo | +Yes | +No | +No | +No | +
| Object | +Yes | +Yes | +Yes | +Yes | +
| Play | +Yes | +No | +No | +No | +
| Playback rate | +Yes | +No | +No | +No | +
| Prefetch status | +Yes | +No | +No | +No | +
| Preset reverb | +No | +No | +No | +Yes | +
| Record | +No | +Yes | +No | +No | +
| Seek | +Yes | +No | +No | +No | +
| Virtualizer | +Yes | +No | +No | +Yes | +
| Volume | +Yes | +No | +No | +No | +
| Buffer queue data locator | +Yes: Source | +No | +No | +No | +
| I/O device data locator | +No | +Yes: Source | +No | +No | +
| Output mix locator | +Yes: Sink | +No | +No | +No | +
| URI data locator | +Yes: Source | +No | +No | +No | +
+Certain limitations apply to the features in Table 1. These limitations +represent differences from the reference specification. The rest of this section provides +information about these differences.
+ ++OpenSL ES for Android does not support {@code RemoveInterface} or +{@code ResumeInterface}. +
+ ++You cannot have both environmental reverb and preset reverb on the same output mix. +
++The platform might ignore effect requests if it estimates that the +CPU load would be too high. +
+ +
+SetSendLevel() supports a single send level per audio player.
+
+Environmental reverb does not support the reflectionsDelay,
+reflectionsLevel, or reverbDelay fields of
+the SLEnvironmentalReverbSettings struct.
+
+You can use the MIME data format only with the URI data locator, and only for an audio +player. You cannot use this data format for an audio recorder. +
+
+The Android implementation of OpenSL ES requires you to initialize mimeType
+to either NULL or a valid UTF-8 string. You must also initialize
+containerType to a valid value.
+In the absence of other considerations, such as portability to other
+implementations, or content format that an app cannot identify by header,
+we recommend that you
+set mimeType to NULL and containerType
+to SL_CONTAINERTYPE_UNSPECIFIED.
+
+OpenSL ES for Android supports the following audio formats, so long as the +Android platform supports them as well:
+ ++For a list of audio formats that Android supports, see +Supported Media Formats. +
+ ++The following limitations apply to handling of these and other formats in this +implementation of OpenSL ES: +
+ ++OpenSL ES for Android does not support the following methods for manipulating objects: +
+ ++PCM is the only data format you can use with buffer queues. Supported PCM +playback configurations have the following characteristics: +
+ ++The configurations that OpenSL ES for Android supports for recording are +device-dependent; usually, 16,000 Hz mono 16-bit signed is available regardless of device. +
+
+The value of the samplesPerSec field is in units of milliHz, despite the misleading
+name. To avoid accidentally using the wrong value, we recommend that you initialize this field using
+one of the symbolic constants defined for this purpose, such as {@code SL_SAMPLINGRATE_44_1}.
+
+Android 5.0 (API level 21) and above support floating-point data. +
+ ++An OpenSL ES playback rate indicates the speed at which an +object presents data, expressed in thousandths of normal speed, or per mille. For example, +a playback rate of 1,000 per mille is 1,000/1,000, or normal speed. +A rate range is a closed interval that expresses possible rate ranges. +
+ +
+Support for playback-rate ranges and other capabilities may vary depending
+on the platform version and implementation. Your app can determine these capabilities at runtime by
+using PlaybackRate::GetRateRange() or
+PlaybackRate::GetCapabilitiesOfRate() to query the device.
+
+A device typically supports the same rate range for a data source in PCM format, and a unity rate +range of 1000 per mille to 1000 per mille for other formats: that is, the unity rate range is +effectively a single value. +
+ +
+OpenSL ES for Android does not support the SL_RECORDEVENT_HEADATLIMIT
+or SL_RECORDEVENT_HEADMOVING events.
+
+The SetLoop() method enables whole-file looping. To enable looping,
+set the startPos parameter to 0, and the value of the endPos parameter
+to SL_TIME_UNKNOWN.
+
+An audio player or recorder with a data locator for a buffer queue supports PCM data format only. +
+ +
+OpenSL ES for Android only supports use of an I/O device data locator when you have
+specified the locator as the data source for Engine::CreateAudioRecorder().
+Initialize the device data locator using the values contained in the following code snippet.
+
+SLDataLocator_IODevice loc_dev =
+ {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT,
+ SL_DEFAULTDEVICEID_AUDIOINPUT, NULL};
+
+
++OpenSL ES for Android can only use the URI data locator with MIME data format, +and only for an audio player. You cannot use this data format for an audio recorder. It supports +{@code http:} and {@code file:} schemes. It does not support other schemes, such as {@code https:}, +{@code ftp:}, or +{@code content:}. +
+ ++We have not verified support for {@code rtsp:} with audio on the Android platform. +
+ ++OpenSL ES for Android extends the reference OpenSL ES specification to make it compatible with +Android, and to take advantage of the power and flexibility of the Android platform. +
+ +
+The definition of the API for the Android extensions resides in OpenSLES_Android.h
+and the header files that it includes. Consult {@code OpenSLES_Android.h}
+for details about these extensions. This file is located under your installation root, in the
+{@code platforms/android-<version>/<abi>/include/SLES} directory. Unless otherwise
+noted, all interfaces are explicit.
+
+These extensions limit your application's portability to +other OpenSL ES implementations, because they are Android-specific. You can mitigate this issue by +avoiding use of the extensions or by using {@code #ifdef} to exclude them at compile time. +
+ ++Table 2 shows the Android-specific interfaces and data locators that Android OpenSL ES supports +for each object type. Green cells indicate interfaces and data locators available for each +object type. +
+ ++ Table 2. Interfaces and data locators, by object type.
+| Feature | +Audio player | +Audio recorder | +Engine | +Output mix | +
|---|---|---|---|---|
| Android buffer queue | +Yes: Source (decode) | +No | +No | +No | +
| Android configuration | +Yes | +Yes | +No | +No | +
| Android effect | +Yes | +No | +No | +Yes | +
| Android effect capabilities | +No | +No | +Yes | +No | +
| Android effect send | +Yes | +No | +No | +No | +
| Android simple buffer queue | +Yes: Source (playback) or sink (decode) | +Yes | +No | +No | +
| Android buffer queue data locator | +Yes: Source (decode) | +No | +No | +No | +
| Android file descriptor data locator | +Yes: Source | +No | +No | +No | +
| Android simple buffer queue data locator | +Yes: Source (playback) or sink (decode) | +Yes: Sink | +No | +No | +
+The Android configuration interface provides a means to set +platform-specific parameters for objects. This interface is different from other OpenSL ES +1.0.1 interfaces in that your app can use it before instantiating the corresponding object; thus, +you can configure the object before instantiating it. The +{@code OpenSLES_AndroidConfiguration.h} header file, which resides at +{@code platforms/android-<version>/<abi>/include/SLES}, +documents the following available configuration keys and values: +
+ +SL_ANDROID_STREAM_MEDIA).SL_ANDROID_RECORDING_PRESET_GENERIC).
++The following code snippet shows an example of how to set the Android audio stream type on an audio +player: +
+ ++// CreateAudioPlayer and specify SL_IID_ANDROIDCONFIGURATION +// in the required interface ID array. Do not realize player yet. +// ... +SLAndroidConfigurationItf playerConfig; +result = (*playerObject)->GetInterface(playerObject, + SL_IID_ANDROIDCONFIGURATION, &playerConfig); +assert(SL_RESULT_SUCCESS == result); +SLint32 streamType = SL_ANDROID_STREAM_ALARM; +result = (*playerConfig)->SetConfiguration(playerConfig, + SL_ANDROID_KEY_STREAM_TYPE, &streamType, sizeof(SLint32)); +assert(SL_RESULT_SUCCESS == result); +// ... +// Now realize the player here. ++ +
+You can use similar code to configure the preset for an audio recorder: +
++// ... obtain the configuration interface as the first four lines above, then: +SLuint32 presetValue = SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION; +result = (*playerConfig)->SetConfiguration(playerConfig, + RECORDING_PRESET, &presetValue, sizeof(SLuint32)); ++ +
+Android's effect, effect send, and effect capabilities interfaces provide +a generic mechanism for an application to query and use device-specific +audio effects. Device manufacturers should document any available device-specific audio effects +that they provide. +
+ ++The Android file descriptor data locator permits you to specify the source for an +audio player as an open file descriptor with read access. The data format must be MIME. +
++This extension is especially useful in conjunction with the native asset manager, because +the app reads assets from the APK via a file descriptor. +
+ ++The Android simple buffer queue data locator and interface are +identical to those in the OpenSL ES 1.0.1 reference specification, with two exceptions: You +can also use Android simple buffer queues with both audio players and audio recorders. Also, PCM +is the only data format you can use with these queues. +In the reference specification, buffer queues are for audio players only, but +compatible with data formats beyond PCM. +
++For recording, your app should enqueue empty buffers. When a registered callback sends +notification that the system has finished writing data to the buffer, the app can +read the buffer. +
++Playback works in the same way. For future source code +compatibility, however, we suggest that applications use Android simple +buffer queues instead of OpenSL ES 1.0.1 buffer queues. +
+ +
+For convenience, the Android implementation of OpenSL ES 1.0.1
+permits your app to specify dynamic interfaces when it instantiates an object.
+This is an alternative to using DynamicInterfaceManagement::AddInterface()
+to add these interfaces after instantiation.
+
+The Android implementation does not include the +reference specification's requirement that the play cursor return to the beginning +of the currently playing buffer when playback enters the {@code SL_PLAYSTATE_STOPPED} +state. This implementation can conform to that behavior, or it can leave the location of the play +cursor unchanged. +
+ +
+As a result, your app cannot assume that either behavior occurs. Therefore,
+you should explicitly call the BufferQueue::Clear() method after a transition to
+SL_PLAYSTATE_STOPPED. Doing so sets the buffer queue to a known state.
+
+Similarly, there is no specification governing whether the trigger for a buffer queue callback must
+be a transition to SL_PLAYSTATE_STOPPED or execution of
+BufferQueue::Clear(). Therefore, we recommend against creating a dependency on
+one or the other; instead, your app should be able to handle both.
+
+There are three methods for querying whether the platform supports the Android extensions. These +methods are: +
+ +Engine::QueryNumSupportedExtensions()Engine::QuerySupportedExtension()Engine::IsExtensionSupported()
+Any of these methods returns ANDROID_SDK_LEVEL_<API-level>,
+where {@code API-level} is the platform API level; for example, {@code ANDROID_SDK_LEVEL_23}.
+A platform API level of 9 or higher means that the platform supports the extensions.
+
+This section describes a deprecated Android-specific extension to OpenSL ES 1.0.1 +for decoding an encoded stream to PCM without immediate playback. +The table below gives recommendations for use of this extension and alternatives. +
+ +| API level | +Alternatives | +
|---|---|
| 13 and below | +An open-source codec with a suitable license. | +
| 14 to 15 | +An open-source codec with a suitable license. | +
| 16 to 20 | ++ The {@link android.media.MediaCodec} class or an open-source codec with a suitable license. + | +
| 21 and above | ++ NDK MediaCodec in the {@code <media/NdkMedia*.h>} header files, the + {@link android.media.MediaCodec} class, or an open-source codec with a suitable license. + | +
+A standard audio player plays back to an audio device, specifying the output mix as the data sink. +The Android extension differs in that an audio player instead +acts as a decoder if the app has specified the data source either as a URI or as an Android +file descriptor data locator described in MIME data format. In such a case, the data sink is +an Android simple buffer queue data locator with PCM data format. +
+ ++This feature is primarily intended for games to pre-load their audio assets when changing to a +new game level, similar to the functionality that the {@link android.media.SoundPool} class +provides. +
+ ++The application should initially enqueue a set of empty buffers in the Android simple +buffer queue. After that, the app fills the buffers with with PCM data. The Android simple +buffer queue callback fires after each buffer is filled. The callback handler processes +the PCM data, re-enqueues the now-empty buffer, and then returns. The application is responsible for +keeping track of decoded buffers; the callback parameter list does not include +sufficient information to indicate which buffer contains data or which buffer to enqueue next. +
+ +
+The data source implicitly reports the end of stream (EOS) by delivering a
+SL_PLAYEVENT_HEADATEND event at the end of the stream. After the app has decoded
+all of the data it received, it makes no further calls to the Android simple buffer queue callback.
+
+The sink's PCM data format typically matches that of the encoded data source +with respect to sample rate, channel count, and bit depth. However, you can decode to a different +sample rate, channel count, or bit depth. +For information about a provision to detect the actual PCM format, see +Determining the format of decoded PCM data via metadata. +
++OpenSL ES for Android's PCM decoding feature supports pause and initial seek; it does not support +volume control, effects, looping, or playback rate. +
++Depending on the platform implementation, decoding may require resources +that cannot be left idle. Therefore, we recommend that you make sure to provide +sufficient numbers of empty PCM buffers; otherwise, the decoder starves. This may happen, +for example, if your app returns from the Android simple buffer queue callback without +enqueueing another empty buffer. The result of decoder starvation is +unspecified, but may include: dropping the decoded +PCM data, pausing the decoding process, or terminating the decoder outright. +
+ +Note: +To decode an encoded stream to PCM but not play back immediately, for apps running on +Android 4.x (API levels 16–20), we recommend using the {@link android.media.MediaCodec} class. +For new applications running on Android 5.0 (API level 21) or higher, we recommend using the NDK +equivalent, {@code <NdkMedia*.h>}. These header files reside under +the {@code media/} directory, under your installation root. +
+ ++An audio player acts as a streaming decoder if the data source is an +Android buffer queue data locator with MIME data format, and the data +sink is an Android simple buffer queue data locator with PCM data format. +Configure the MIME data format as follows: +
+ ++This feature is primarily intended for streaming media applications that +deal with AAC audio but need to perform custom audio processing +prior to playback. Most applications that need to decode audio to PCM +should use the method that Decode audio to PCM describes, +as that method is simpler and handles more audio formats. The technique described +here is a more specialized approach, to be used only if both of these +conditions apply: +
+ ++The application should initially enqueue a set of filled buffers in the Android buffer queue. +Each buffer contains one or more complete ADTS AAC frames. +The Android buffer queue callback fires after each buffer is emptied. +The callback handler should refill and re-enqueue the buffer, and then return. +The application need not keep track of encoded buffers; the callback parameter +list includes sufficient information to indicate which buffer to enqueue next. +The end of stream is explicitly marked by enqueuing an EOS item. +After EOS, no more enqueues are permitted. +
+ ++We recommend that you make sure to provide full +ADTS AAC buffers, to avoid starving the decoder. This may happen, for example, if your app +returns from the Android buffer queue callback without enqueueing another full buffer. +The result of decoder starvation is unspecified. +
+ ++In all respects except for the data source, the streaming decode method is the same as +the one that Decode audio to PCM describes. +
++Despite the similarity in names, an Android buffer queue is not +the same as an Android simple buffer queue. The streaming decoder +uses both kinds of buffer queues: an Android buffer queue for the ADTS +AAC data source, and an Android simple buffer queue for the PCM data +sink. For more information about the Android simple buffer queue API, see Android +simple buffer queue data locator and interface. +For more information about the Android buffer queue API, see the {@code index.html} file in +the {@code docs/Additional_library_docs/openmaxal/} directory under the installation root. +
+ +
+The SLMetadataExtractionItf interface is part of the reference specification.
+However, the metadata keys that indicate the actual format of decoded PCM data are specific to
+Android. The OpenSLES_AndroidMetadata.h header file defines these metadata keys.
+This header file resides under your installation root, in the
+{@code platforms/android-<version>/<abi>/include/SLES} directory.
+
+The metadata key indices are available immediately after
+the Object::Realize() method finishes executing. However, the associated values are not
+available until after the app decodes the first encoded data. A good
+practice is to query for the key indices in the main thread after calling the {@code
+Object::Realize} method, and to read the PCM format metadata values in the Android simple
+buffer queue callback handler when calling it for the first time. Consult the
+example code in the NDK package
+for examples of working with this interface.
+
+Metadata key names are stable, but the key indices are not documented, +and are subject to change. An application should not assume that indices +are persistent across different execution runs, and should not assume that +multiple object instances share indices within the same run. +
+ ++An app running on Android 5.0 (API level 21) and higher can supply data to an AudioPlayer in +single-precision, floating-point format. +
++In following example code, the {@code Engine::CreateAudioPlayer} method creates an audio player +that uses floating-point data: +
+ ++#include <SLES/OpenSLES_Android.h> +... +SLAndroidDataFormat_PCM_EX pcm; +pcm.formatType = SL_ANDROID_DATAFORMAT_PCM_EX; +pcm.numChannels = 2; +pcm.sampleRate = SL_SAMPLINGRATE_44_1; +pcm.bitsPerSample = 32; +pcm.containerSize = 32; +pcm.channelMask = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT; +pcm.endianness = SL_BYTEORDER_LITTLEENDIAN; +pcm.representation = SL_ANDROID_PCM_REPRESENTATION_FLOAT; +... +SLDataSource audiosrc; +audiosrc.pLocator = ... +audiosrc.pFormat = &pcm; +diff --git a/docs/html/ndk/guides/guides_toc.cs b/docs/html/ndk/guides/guides_toc.cs index 981eb5131facd..4c4c64e2f95f8 100644 --- a/docs/html/ndk/guides/guides_toc.cs +++ b/docs/html/ndk/guides/guides_toc.cs @@ -63,6 +63,16 @@ +