From 08bd163e2db8c7e073a3d633cbf4cb0f53fe3c14 Mon Sep 17 00:00:00 2001 From: Eino-Ville Talvala Date: Wed, 1 Aug 2018 17:23:09 -0700 Subject: [PATCH] Camera: Documentation updates for calibration and distortion correction - Add more notes on coordinate axes - Add more text on metadata when distortion correction is active - Note that poseTranslation needs to be negated in many use cases - Fix coordinate system references for OIS reporting, add more information - Note that pixel centers at half-integers for the camera API metadata such as lens intrinsics Bug: 79371566 Bug: 74434422 Bug: 109742048 Bug: 109834325 Bug: 109817371 Bug: 112107924 Test: Manual reading of added text Change-Id: I450e80b79ef66ce8d82a4dee835db6abd1e598a3 --- .../camera2/CameraCharacteristics.java | 24 +++-- .../hardware/camera2/CaptureRequest.java | 37 +++++-- .../hardware/camera2/CaptureResult.java | 101 +++++++++++++----- 3 files changed, 125 insertions(+), 37 deletions(-) diff --git a/core/java/android/hardware/camera2/CameraCharacteristics.java b/core/java/android/hardware/camera2/CameraCharacteristics.java index 60e4ce29ab513..d027fd997b95e 100644 --- a/core/java/android/hardware/camera2/CameraCharacteristics.java +++ b/core/java/android/hardware/camera2/CameraCharacteristics.java @@ -1249,7 +1249,9 @@ public final class CameraCharacteristics extends CameraMetadataIf this device is the largest or only camera device with a given facing, then this * position will be (0, 0, 0); a camera device with a lens optical center located 3 cm * from the main sensor along the +X axis (to the right from the user's perspective) will - * report (0.03, 0, 0).

+ * report (0.03, 0, 0). Note that this means that, for many computer vision + * applications, the position needs to be negated to convert it to a translation from the + * camera to the origin.

*

To transform a pixel coordinates between two cameras facing the same direction, first * the source camera {@link CameraCharacteristics#LENS_DISTORTION android.lens.distortion} must be corrected for. Then the source * camera {@link CameraCharacteristics#LENS_INTRINSIC_CALIBRATION android.lens.intrinsicCalibration} needs to be applied, followed by the @@ -1261,7 +1263,8 @@ public final class CameraCharacteristics extends CameraMetadataTo compare this against a real image from the destination camera, the destination camera * image then needs to be corrected for radial distortion before comparison or sampling.

*

When {@link CameraCharacteristics#LENS_POSE_REFERENCE android.lens.poseReference} is GYROSCOPE, then this position is relative to - * the center of the primary gyroscope on the device.

+ * the center of the primary gyroscope on the device. The axis definitions are the same as + * with PRIMARY_CAMERA.

*

Units: Meters

*

Optional - This value may be {@code null} on some devices.

* @@ -1293,13 +1296,15 @@ public final class CameraCharacteristics extends CameraMetadata *

which can then be combined with the camera pose rotation * R and translation t ({@link CameraCharacteristics#LENS_POSE_ROTATION android.lens.poseRotation} and - * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respective) to calculate the + * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respectively) to calculate the * complete transform from world coordinates to pixel * coordinates:

- * 
P = [ K 0   * [ R t
-     *      0 1 ]     0 1 ]
+     * 
P = [ K 0   * [ R -Rt
+     *      0 1 ]      0 1 ]
      * 

-     * 
and with p_w being a point in the world coordinate system
+     * 
(Note the negation of poseTranslation when mapping from camera
+     * to world coordinates, and multiplication by the rotation).

+     * 
With p_w being a point in the world coordinate system
      * and p_s being a point in the camera active pixel array
      * coordinate system, and with the mapping including the
      * homogeneous division by z:

@@ -1321,6 +1326,13 @@ public final class CameraCharacteristics extends CameraMetadata
+     * 
For camera devices, the center of pixel (x,y) is located at
+     * coordinate (x + 0.5, y + 0.5).  So on a device with a
+     * precorrection active array of size (10,10), the valid pixel
+     * indices go from (0,0)-(9,9), and an perfectly-built camera would
+     * have an optical center at the exact center of the pixel grid, at
+     * coordinates (5.0, 5.0), which is the top-left corner of pixel
+     * (5,5).

      * 
Units:
      * Pixels in the
      * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}
diff --git a/core/java/android/hardware/camera2/CaptureRequest.java b/core/java/android/hardware/camera2/CaptureRequest.java
index aca77a5be47e6..0c3fe778dda79 100644
--- a/core/java/android/hardware/camera2/CaptureRequest.java
+++ b/core/java/android/hardware/camera2/CaptureRequest.java
@@ -2522,7 +2522,7 @@ public final class CaptureRequest extends CameraMetadata>
      * outputs will crop horizontally (pillarbox), and 16:9
      * streams will match exactly. These additional crops will
      * be centered within the crop region.

-     * 
If the coordinate system is android.sensor.info.activeArraysSize, the width and height
+     * 
If the coordinate system is {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}, the width and height
      * of the crop region cannot be set to be smaller than
      * floor( activeArraySize.width / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} ) and
      * floor( activeArraySize.height / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} ), respectively.

@@ -2863,8 +2863,14 @@ public final class CaptureRequest extends CameraMetadata>
             new Key("android.statistics.lensShadingMapMode", int.class);
 
     /**
-     * 
A control for selecting whether OIS position information is included in output
-     * result metadata.

+     * 
A control for selecting whether optical stabilization (OIS) position
+     * information is included in output result metadata.

+     * 
Since optical image stabilization generally involves motion much faster than the duration
+     * of individualq image exposure, multiple OIS samples can be included for a single capture
+     * result. For example, if the OIS reporting operates at 200 Hz, a typical camera operating
+     * at 30fps may have 6-7 OIS samples per capture result. This information can be combined
+     * with the rolling shutter skew to account for lens motion during image exposure in
+     * post-processing algorithms.

      * 
Possible values:
      * 
    
      *   
  • {@link #STATISTICS_OIS_DATA_MODE_OFF OFF}
    • 
@@ -3264,14 +3270,28 @@ public final class CaptureRequest extends CameraMetadata>
      * any correction at all would slow down capture rate.  Every output stream will have a
      * similar amount of enhancement applied.
    
      * 
    The correction only applies to processed outputs such as YUV, JPEG, or DEPTH16; it is not
-     * applied to any RAW output. Metadata coordinates such as face rectangles or metering
-     * regions are also not affected by correction.
    
+     * applied to any RAW output.
    
      * 
    This control will be on by default on devices that support this control. Applications
      * disabling distortion correction need to pay extra attention with the coordinate system of
      * metering regions, crop region, and face rectangles. When distortion correction is OFF,
      * metadata coordinates follow the coordinate system of
      * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}. When distortion is not OFF, metadata
-     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.
    
+     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.  The
+     * camera device will map these metadata fields to match the corrected image produced by the
+     * camera device, for both capture requests and results.  However, this mapping is not very
+     * precise, since rectangles do not generally map to rectangles when corrected.  Only linear
+     * scaling between the active array and precorrection active array coordinates is
+     * performed. Applications that require precise correction of metadata need to undo that
+     * linear scaling, and apply a more complete correction that takes into the account the app's
+     * own requirements.
    
+     * 
    The full list of metadata that is affected in this way by distortion correction is:
    
+     * 
      
+     * 
    • {@link CaptureRequest#CONTROL_AF_REGIONS android.control.afRegions}
      • 
+     * 
    • {@link CaptureRequest#CONTROL_AE_REGIONS android.control.aeRegions}
      • 
+     * 
    • {@link CaptureRequest#CONTROL_AWB_REGIONS android.control.awbRegions}
      • 
+     * 
    • {@link CaptureRequest#SCALER_CROP_REGION android.scaler.cropRegion}
      • 
+     * 
    • {@link CaptureResult#STATISTICS_FACES android.statistics.faces}
      • 
+     * 
    
      * 
    Possible values:
      * 
      
      *   
    • {@link #DISTORTION_CORRECTION_MODE_OFF OFF}
      • 
@@ -3282,10 +3302,15 @@ public final class CaptureRequest extends CameraMetadata>
      * {@link CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES android.distortionCorrection.availableModes}
      
      * 
      Optional - This value may be {@code null} on some devices.
      
      *
+     * @see CaptureRequest#CONTROL_AE_REGIONS
+     * @see CaptureRequest#CONTROL_AF_REGIONS
+     * @see CaptureRequest#CONTROL_AWB_REGIONS
      * @see CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES
      * @see CameraCharacteristics#LENS_DISTORTION
+     * @see CaptureRequest#SCALER_CROP_REGION
      * @see CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE
      * @see CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE
+     * @see CaptureResult#STATISTICS_FACES
      * @see #DISTORTION_CORRECTION_MODE_OFF
      * @see #DISTORTION_CORRECTION_MODE_FAST
      * @see #DISTORTION_CORRECTION_MODE_HIGH_QUALITY
diff --git a/core/java/android/hardware/camera2/CaptureResult.java b/core/java/android/hardware/camera2/CaptureResult.java
index 75c27f56c0a63..845f1d3cbcbaa 100644
--- a/core/java/android/hardware/camera2/CaptureResult.java
+++ b/core/java/android/hardware/camera2/CaptureResult.java
@@ -2852,7 +2852,9 @@ public class CaptureResult extends CameraMetadata> {
      * 
      If this device is the largest or only camera device with a given facing, then this
      * position will be (0, 0, 0); a camera device with a lens optical center located 3 cm
      * from the main sensor along the +X axis (to the right from the user's perspective) will
-     * report (0.03, 0, 0).
      
+     * report (0.03, 0, 0).  Note that this means that, for many computer vision
+     * applications, the position needs to be negated to convert it to a translation from the
+     * camera to the origin.
      
      * 
      To transform a pixel coordinates between two cameras facing the same direction, first
      * the source camera {@link CameraCharacteristics#LENS_DISTORTION android.lens.distortion} must be corrected for.  Then the source
      * camera {@link CameraCharacteristics#LENS_INTRINSIC_CALIBRATION android.lens.intrinsicCalibration} needs to be applied, followed by the
@@ -2864,7 +2866,8 @@ public class CaptureResult extends CameraMetadata> {
      * 
      To compare this against a real image from the destination camera, the destination camera
      * image then needs to be corrected for radial distortion before comparison or sampling.
      
      * 
      When {@link CameraCharacteristics#LENS_POSE_REFERENCE android.lens.poseReference} is GYROSCOPE, then this position is relative to
-     * the center of the primary gyroscope on the device.
      
+     * the center of the primary gyroscope on the device. The axis definitions are the same as
+     * with PRIMARY_CAMERA.
      
      * 
      Units: Meters
      
      * 
      Optional - This value may be {@code null} on some devices.
      
      *
@@ -2896,13 +2899,15 @@ public class CaptureResult extends CameraMetadata> {
      *
*

which can then be combined with the camera pose rotation * R and translation t ({@link CameraCharacteristics#LENS_POSE_ROTATION android.lens.poseRotation} and - * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respective) to calculate the + * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respectively) to calculate the * complete transform from world coordinates to pixel * coordinates:

- * 
P = [ K 0   * [ R t
-     *      0 1 ]     0 1 ]
+     * 
P = [ K 0   * [ R -Rt
+     *      0 1 ]      0 1 ]
      * 

-     * 
and with p_w being a point in the world coordinate system
+     * 
(Note the negation of poseTranslation when mapping from camera
+     * to world coordinates, and multiplication by the rotation).

+     * 
With p_w being a point in the world coordinate system
      * and p_s being a point in the camera active pixel array
      * coordinate system, and with the mapping including the
      * homogeneous division by z:

@@ -2924,6 +2929,13 @@ public class CaptureResult extends CameraMetadata> {
      * activeArraySize rectangle), to determine the final pixel
      * coordinate of the world point for processed (non-RAW)
      * output buffers.

+     * 
For camera devices, the center of pixel (x,y) is located at
+     * coordinate (x + 0.5, y + 0.5).  So on a device with a
+     * precorrection active array of size (10,10), the valid pixel
+     * indices go from (0,0)-(9,9), and an perfectly-built camera would
+     * have an optical center at the exact center of the pixel grid, at
+     * coordinates (5.0, 5.0), which is the top-left corner of pixel
+     * (5,5).

      * 
Units:
      * Pixels in the
      * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}
@@ -3188,7 +3200,7 @@ public class CaptureResult extends CameraMetadata> {
      * outputs will crop horizontally (pillarbox), and 16:9
      * streams will match exactly. These additional crops will
      * be centered within the crop region.

-     * 
If the coordinate system is android.sensor.info.activeArraysSize, the width and height
+     * 
If the coordinate system is {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}, the width and height
      * of the crop region cannot be set to be smaller than
      * floor( activeArraySize.width / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} ) and
      * floor( activeArraySize.height / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} ), respectively.

@@ -4077,8 +4089,14 @@ public class CaptureResult extends CameraMetadata> {
             new Key("android.statistics.lensShadingMapMode", int.class);
 
     /**
-     * 
A control for selecting whether OIS position information is included in output
-     * result metadata.

+     * 
A control for selecting whether optical stabilization (OIS) position
+     * information is included in output result metadata.

+     * 
Since optical image stabilization generally involves motion much faster than the duration
+     * of individualq image exposure, multiple OIS samples can be included for a single capture
+     * result. For example, if the OIS reporting operates at 200 Hz, a typical camera operating
+     * at 30fps may have 6-7 OIS samples per capture result. This information can be combined
+     * with the rolling shutter skew to account for lens motion during image exposure in
+     * post-processing algorithms.

      * 
Possible values:
      * 
    
      *   
  • {@link #STATISTICS_OIS_DATA_MODE_OFF OFF}
    • 
@@ -4112,11 +4130,15 @@ public class CaptureResult extends CameraMetadata> {
     /**
      * 
    An array of shifts of OIS samples, in x direction.
    
      * 
    The array contains the amount of shifts in x direction, in pixels, based on OIS samples.
-     * A positive value is a shift from left to right in active array coordinate system. For
-     * example, if the optical center is (1000, 500) in active array coordinates, a shift of
-     * (3, 0) puts the new optical center at (1003, 500).
    
+     * A positive value is a shift from left to right in the pre-correction active array
+     * coordinate system. For example, if the optical center is (1000, 500) in pre-correction
+     * active array coordinates, a shift of (3, 0) puts the new optical center at (1003, 500).
    
      * 
    The number of shifts must match the number of timestamps in
      * android.statistics.oisTimestamps.
    
+     * 
    The OIS samples are not affected by whether lens distortion correction is enabled (on
+     * supporting devices). They are always reported in pre-correction active array coordinates,
+     * since the scaling of OIS shifts would depend on the specific spot on the sensor the shift
+     * is needed.
    
      * 
    Units: Pixels in active array.
    
      * 
    Optional - This value may be {@code null} on some devices.
    
      * @hide
@@ -4127,11 +4149,15 @@ public class CaptureResult extends CameraMetadata> {
     /**
      * 
    An array of shifts of OIS samples, in y direction.
    
      * 
    The array contains the amount of shifts in y direction, in pixels, based on OIS samples.
-     * A positive value is a shift from top to bottom in active array coordinate system. For
-     * example, if the optical center is (1000, 500) in active array coordinates, a shift of
-     * (0, 5) puts the new optical center at (1000, 505).
    
+     * A positive value is a shift from top to bottom in pre-correction active array coordinate
+     * system. For example, if the optical center is (1000, 500) in active array coordinates, a
+     * shift of (0, 5) puts the new optical center at (1000, 505).
    
      * 
    The number of shifts must match the number of timestamps in
      * android.statistics.oisTimestamps.
    
+     * 
    The OIS samples are not affected by whether lens distortion correction is enabled (on
+     * supporting devices). They are always reported in pre-correction active array coordinates,
+     * since the scaling of OIS shifts would depend on the specific spot on the sensor the shift
+     * is needed.
    
      * 
    Units: Pixels in active array.
    
      * 
    Optional - This value may be {@code null} on some devices.
    
      * @hide
@@ -4140,15 +4166,21 @@ public class CaptureResult extends CameraMetadata> {
             new Key("android.statistics.oisYShifts", float[].class);
 
     /**
-     * 
    An array of OIS samples.
    
+     * 
    An array of optical stabilization (OIS) position samples.
    
      * 
    Each OIS sample contains the timestamp and the amount of shifts in x and y direction,
      * in pixels, of the OIS sample.
    
-     * 
    A positive value for a shift in x direction is a shift from left to right in active array
-     * coordinate system. For example, if the optical center is (1000, 500) in active array
-     * coordinates, a shift of (3, 0) puts the new optical center at (1003, 500).
    
-     * 
    A positive value for a shift in y direction is a shift from top to bottom in active array
-     * coordinate system. For example, if the optical center is (1000, 500) in active array
-     * coordinates, a shift of (0, 5) puts the new optical center at (1000, 505).
    
+     * 
    A positive value for a shift in x direction is a shift from left to right in the
+     * pre-correction active array coordinate system. For example, if the optical center is
+     * (1000, 500) in pre-correction active array coordinates, a shift of (3, 0) puts the new
+     * optical center at (1003, 500).
    
+     * 
    A positive value for a shift in y direction is a shift from top to bottom in
+     * pre-correction active array coordinate system. For example, if the optical center is
+     * (1000, 500) in active array coordinates, a shift of (0, 5) puts the new optical center at
+     * (1000, 505).
    
+     * 
    The OIS samples are not affected by whether lens distortion correction is enabled (on
+     * supporting devices). They are always reported in pre-correction active array coordinates,
+     * since the scaling of OIS shifts would depend on the specific spot on the sensor the shift
+     * is needed.
    
      * 
    Optional - This value may be {@code null} on some devices.
    
      */
     @PublicKey
@@ -4578,14 +4610,28 @@ public class CaptureResult extends CameraMetadata> {
      * any correction at all would slow down capture rate.  Every output stream will have a
      * similar amount of enhancement applied.
    
      * 
    The correction only applies to processed outputs such as YUV, JPEG, or DEPTH16; it is not
-     * applied to any RAW output. Metadata coordinates such as face rectangles or metering
-     * regions are also not affected by correction.
    
+     * applied to any RAW output.
    
      * 
    This control will be on by default on devices that support this control. Applications
      * disabling distortion correction need to pay extra attention with the coordinate system of
      * metering regions, crop region, and face rectangles. When distortion correction is OFF,
      * metadata coordinates follow the coordinate system of
      * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}. When distortion is not OFF, metadata
-     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.
    
+     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.  The
+     * camera device will map these metadata fields to match the corrected image produced by the
+     * camera device, for both capture requests and results.  However, this mapping is not very
+     * precise, since rectangles do not generally map to rectangles when corrected.  Only linear
+     * scaling between the active array and precorrection active array coordinates is
+     * performed. Applications that require precise correction of metadata need to undo that
+     * linear scaling, and apply a more complete correction that takes into the account the app's
+     * own requirements.
    
+     * 
    The full list of metadata that is affected in this way by distortion correction is:
    
+     * 
      
+     * 
    • {@link CaptureRequest#CONTROL_AF_REGIONS android.control.afRegions}
      • 
+     * 
    • {@link CaptureRequest#CONTROL_AE_REGIONS android.control.aeRegions}
      • 
+     * 
    • {@link CaptureRequest#CONTROL_AWB_REGIONS android.control.awbRegions}
      • 
+     * 
    • {@link CaptureRequest#SCALER_CROP_REGION android.scaler.cropRegion}
      • 
+     * 
    • {@link CaptureResult#STATISTICS_FACES android.statistics.faces}
      • 
+     * 
    
      * 
    Possible values:
      * 
      
      *   
    • {@link #DISTORTION_CORRECTION_MODE_OFF OFF}
      • 
@@ -4596,10 +4642,15 @@ public class CaptureResult extends CameraMetadata> {
      * {@link CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES android.distortionCorrection.availableModes}
      
      * 
      Optional - This value may be {@code null} on some devices.
      
      *
+     * @see CaptureRequest#CONTROL_AE_REGIONS
+     * @see CaptureRequest#CONTROL_AF_REGIONS
+     * @see CaptureRequest#CONTROL_AWB_REGIONS
      * @see CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES
      * @see CameraCharacteristics#LENS_DISTORTION
+     * @see CaptureRequest#SCALER_CROP_REGION
      * @see CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE
      * @see CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE
+     * @see CaptureResult#STATISTICS_FACES
      * @see #DISTORTION_CORRECTION_MODE_OFF
      * @see #DISTORTION_CORRECTION_MODE_FAST
      * @see #DISTORTION_CORRECTION_MODE_HIGH_QUALITY