[v5,12/16] Documentation: mali-c55: Add Statistics documentation

Message ID 20240529152858.183799-13-dan.scally@ideasonboard.com (mailing list archive)
State Superseded
Headers
Series Add Arm Mali-C55 Image Signal Processor Driver |

Commit Message

Daniel Scally May 29, 2024, 3:28 p.m. UTC
  Add documentation explaining the ability to capture statistics from
the mali-c55 driver's new V4L2 device, as well as the various tap
points from which those statistics can be drawn in the ISP's
processing flow. Additionally add a page detailing the new V4L2
meta format for the mali-c55 statistics.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 Documentation/admin-guide/media/mali-c55.rst  | 60 ++++++++++++++++++-
 .../userspace-api/media/v4l/meta-formats.rst  |  1 +
 .../media/v4l/metafmt-arm-mali-c55.rst        | 28 +++++++++
 3 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
  

Comments

Laurent Pinchart May 30, 2024, 10:43 p.m. UTC | #1
Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:54PM +0100, Daniel Scally wrote:
> Add documentation explaining the ability to capture statistics from
> the mali-c55 driver's new V4L2 device, as well as the various tap
> points from which those statistics can be drawn in the ISP's
> processing flow. Additionally add a page detailing the new V4L2
> meta format for the mali-c55 statistics.
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  Documentation/admin-guide/media/mali-c55.rst  | 60 ++++++++++++++++++-
>  .../userspace-api/media/v4l/meta-formats.rst  |  1 +
>  .../media/v4l/metafmt-arm-mali-c55.rst        | 28 +++++++++
>  3 files changed, 88 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> 
> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
> index cf4176cb1e44..b75437f6e96a 100644
> --- a/Documentation/admin-guide/media/mali-c55.rst
> +++ b/Documentation/admin-guide/media/mali-c55.rst
> @@ -67,10 +67,11 @@ The driver has 4 V4L2 subdevices:
>  - `mali_c55 resizer fr`: The Full-Resolution pipe resizer
>  - `mali_c55 resizer ds`: The Downscale pipe resizer
>  
> -The driver has 2 V4L2 video devices:
> +The driver has 3 V4L2 video devices:
>  
>  - `mali-c55 fr`: The full-resolution pipe's capture device
>  - `mali-c55 ds`: The downscale pipe's capture device
> +- `mali-c55 3a stats`: The 3A statistics capture device
>  
>  Frame sequences are synchronised across to two capture devices, meaning if one
>  pipe is started later than the other the sequence numbers returned in its
> @@ -326,6 +327,63 @@ configured, followed by formats in the appropriate places:
>      # Set format on the video device and stream
>      yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>  
> +.. _mali-c55-3a-stats:
> +
> +Capturing ISP Statistics
> +========================
> +
> +The ISP is capable of producing statistics for consumption by image processing
> +algorithms running in userspace. These statistics can be captured by queueing
> +buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only
> +the :ref:`V4L2_META_FMT_MALI_C55_3A_STATS <v4l2-meta-fmt-mali-c55-3a-stats>`
> +format is supported, so no format-setting need be done:
> +
> +.. code-block:: none
> +
> +    # We assume the media graph has been configured to support RGB565 capture
> +    # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics
> +    # V4L2 device is at /dev/video3
> +
> +    yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \
> +    yavta -c10 -F /dev/video3
> +
> +The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`,
> +but broadly statistics are generated to support three image processing
> +algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus).
> +These stats can be drawn from various places in the Mali C55 ISP pipeline, known
> +as "tap points". This high-level block diagram is intended to explain where in
> +the processing flow the statistics can be drawn from::
> +
> +                  +--> AEXP-2            +----> AEXP-1          +--> AF-0
> +                  |                      +----> AF-1            |
> +                  |                      |                      |
> +      +---------+ |   +--------------+   |   +--------------+   |
> +      |  Input  +-+-->+ Digital Gain +---+-->+ Black Level  +---+---+
> +      +---------+     +--------------+       +--------------+       |
> +  +-----------------------------------------------------------------+
> +  |
> +  |   +--------------+ +---------+       +----------------+
> +  +-->| Sinter Noise +-+  White  +--+--->|  Lens Shading  +--+---------------+
> +      |   Reduction  | | Balance |  |    |                |  |               |
> +      +--------------+ +---------+  |    +----------------+  |               |
> +                                    +---> AEXP-0 (A)         +--> AEXP-0 (B) |
> +  +--------------------------------------------------------------------------+
> +  |
> +  |   +----------------+      +--------------+  +----------------+
> +  +-->|  Tone mapping  +-+--->| Demosaicing  +->+ Purple Fringe  +-+-----------+
> +      |                | |    +--------------+  |   Correction   | |           |
> +      +----------------+ +-> AEXP-IRIDIX        +----------------+ +---> AWB-0 |
> +  +----------------------------------------------------------------------------+
> +  |                    +-------------+        +-------------+
> +  +------------------->|   Colour    +---+--->|    Output   |
> +                       | Correction  |   |    |  Pipelines  |
> +                       +-------------+   |    +-------------+
> +                                         +-->  AWB-1
> +
> +At present all statistics are drawn from the 0th tap point for each algorithm;
> +I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
> +statistics from AF-0. In the future this will be configurable.
> +
>  References
>  ==========
>  .. [1] https://git.linuxtv.org/v4l-utils.git/
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index c23aac823d2c..73c05cc35321 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -15,6 +15,7 @@ These formats are used for the :ref:`metadata` interface only.
>      metafmt-d4xx
>      metafmt-generic
>      metafmt-intel-ipu3
> +    metafmt-arm-mali-c55

Alphabetical order please.

>      metafmt-rkisp1
>      metafmt-uvc
>      metafmt-vivid
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> new file mode 100644
> index 000000000000..219a5dd42d70
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> @@ -0,0 +1,28 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _v4l2-meta-fmt-mali-c55-3a-stats:
> +
> +****************************************
> +V4L2_META_FMT_MALI_C55_3A_STATS ('C55S')
> +****************************************
> +
> +3A Statistics
> +=============

Missing blank line.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +The ISP device collects different statistics over an input bayer frame. Those
> +statistics can be obtained by userspace from the
> +:ref:`mali-c55 3a stats <mali-c55-3a-stats>` metadata capture video node, using
> +the :c:type:`v4l2_meta_format` interface. The buffer contains a single instance
> +of the C structure :c:type:`mali_c55_stats_buffer` defined in
> +``mali-c55-config.h``, so the structure can be obtained from the buffer by:
> +
> +.. code-block:: C
> +
> +	struct mali_c55_stats_buffer *stats =
> +		(struct mali_c55_stats_buffer *)buf;
> +
> +For details of the statistics see :c:type:`mali_c55_stats_buffer`.
> +
> +Arm Mali-C55 uAPI data types
> +============================
> +
> +.. kernel-doc:: include/uapi/linux/media/arm/mali-c55-config.h
  

Patch

diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
index cf4176cb1e44..b75437f6e96a 100644
--- a/Documentation/admin-guide/media/mali-c55.rst
+++ b/Documentation/admin-guide/media/mali-c55.rst
@@ -67,10 +67,11 @@  The driver has 4 V4L2 subdevices:
 - `mali_c55 resizer fr`: The Full-Resolution pipe resizer
 - `mali_c55 resizer ds`: The Downscale pipe resizer
 
-The driver has 2 V4L2 video devices:
+The driver has 3 V4L2 video devices:
 
 - `mali-c55 fr`: The full-resolution pipe's capture device
 - `mali-c55 ds`: The downscale pipe's capture device
+- `mali-c55 3a stats`: The 3A statistics capture device
 
 Frame sequences are synchronised across to two capture devices, meaning if one
 pipe is started later than the other the sequence numbers returned in its
@@ -326,6 +327,63 @@  configured, followed by formats in the appropriate places:
     # Set format on the video device and stream
     yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
 
+.. _mali-c55-3a-stats:
+
+Capturing ISP Statistics
+========================
+
+The ISP is capable of producing statistics for consumption by image processing
+algorithms running in userspace. These statistics can be captured by queueing
+buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only
+the :ref:`V4L2_META_FMT_MALI_C55_3A_STATS <v4l2-meta-fmt-mali-c55-3a-stats>`
+format is supported, so no format-setting need be done:
+
+.. code-block:: none
+
+    # We assume the media graph has been configured to support RGB565 capture
+    # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics
+    # V4L2 device is at /dev/video3
+
+    yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \
+    yavta -c10 -F /dev/video3
+
+The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`,
+but broadly statistics are generated to support three image processing
+algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus).
+These stats can be drawn from various places in the Mali C55 ISP pipeline, known
+as "tap points". This high-level block diagram is intended to explain where in
+the processing flow the statistics can be drawn from::
+
+                  +--> AEXP-2            +----> AEXP-1          +--> AF-0
+                  |                      +----> AF-1            |
+                  |                      |                      |
+      +---------+ |   +--------------+   |   +--------------+   |
+      |  Input  +-+-->+ Digital Gain +---+-->+ Black Level  +---+---+
+      +---------+     +--------------+       +--------------+       |
+  +-----------------------------------------------------------------+
+  |
+  |   +--------------+ +---------+       +----------------+
+  +-->| Sinter Noise +-+  White  +--+--->|  Lens Shading  +--+---------------+
+      |   Reduction  | | Balance |  |    |                |  |               |
+      +--------------+ +---------+  |    +----------------+  |               |
+                                    +---> AEXP-0 (A)         +--> AEXP-0 (B) |
+  +--------------------------------------------------------------------------+
+  |
+  |   +----------------+      +--------------+  +----------------+
+  +-->|  Tone mapping  +-+--->| Demosaicing  +->+ Purple Fringe  +-+-----------+
+      |                | |    +--------------+  |   Correction   | |           |
+      +----------------+ +-> AEXP-IRIDIX        +----------------+ +---> AWB-0 |
+  +----------------------------------------------------------------------------+
+  |                    +-------------+        +-------------+
+  +------------------->|   Colour    +---+--->|    Output   |
+                       | Correction  |   |    |  Pipelines  |
+                       +-------------+   |    +-------------+
+                                         +-->  AWB-1
+
+At present all statistics are drawn from the 0th tap point for each algorithm;
+I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
+statistics from AF-0. In the future this will be configurable.
+
 References
 ==========
 .. [1] https://git.linuxtv.org/v4l-utils.git/
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index c23aac823d2c..73c05cc35321 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -15,6 +15,7 @@  These formats are used for the :ref:`metadata` interface only.
     metafmt-d4xx
     metafmt-generic
     metafmt-intel-ipu3
+    metafmt-arm-mali-c55
     metafmt-rkisp1
     metafmt-uvc
     metafmt-vivid
diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
new file mode 100644
index 000000000000..219a5dd42d70
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
@@ -0,0 +1,28 @@ 
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _v4l2-meta-fmt-mali-c55-3a-stats:
+
+****************************************
+V4L2_META_FMT_MALI_C55_3A_STATS ('C55S')
+****************************************
+
+3A Statistics
+=============
+The ISP device collects different statistics over an input bayer frame. Those
+statistics can be obtained by userspace from the
+:ref:`mali-c55 3a stats <mali-c55-3a-stats>` metadata capture video node, using
+the :c:type:`v4l2_meta_format` interface. The buffer contains a single instance
+of the C structure :c:type:`mali_c55_stats_buffer` defined in
+``mali-c55-config.h``, so the structure can be obtained from the buffer by:
+
+.. code-block:: C
+
+	struct mali_c55_stats_buffer *stats =
+		(struct mali_c55_stats_buffer *)buf;
+
+For details of the statistics see :c:type:`mali_c55_stats_buffer`.
+
+Arm Mali-C55 uAPI data types
+============================
+
+.. kernel-doc:: include/uapi/linux/media/arm/mali-c55-config.h