[v11,25/36] media: Documentation: add multiplexed streams documentation
Commit Message
Add documentation related to multiplexed streams.
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
.../driver-api/media/v4l2-subdev.rst | 8 +
.../userspace-api/media/v4l/dev-subdev.rst | 165 ++++++++++++++++++
2 files changed, 173 insertions(+)
Comments
Hi Tomi
On Tue, Mar 01, 2022 at 06:11:45PM +0200, Tomi Valkeinen wrote:
> Add documentation related to multiplexed streams.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
> .../driver-api/media/v4l2-subdev.rst | 8 +
> .../userspace-api/media/v4l/dev-subdev.rst | 165 ++++++++++++++++++
> 2 files changed, 173 insertions(+)
>
> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
> index 115211cef4d1..80654f1bcac9 100644
> --- a/Documentation/driver-api/media/v4l2-subdev.rst
> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
> @@ -593,6 +593,14 @@ before calling v4l2_subdev_init_finalize():
>
> This shares the driver's private mutex between the controls and the states.
>
> +Streams, multiplexed media pads and internal routing
> +----------------------------------------------------
> +
> +A subdevice driver can implement support for multiplexed streams by setting
Let me start by being picky with a minor thing: the rest of the
documentation seems to use "sub-device". Here you have "sub-device",
"subdevice" and "subdev". I think "sub-device" should be used
everywhere
> +the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
> +centrally managed subdev active state, routing and stream based
> +configuration.
> +
> V4L2 sub-device functions and data structures
> ---------------------------------------------
>
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a67c2749089a..fd042afeddd6 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -503,3 +503,168 @@ source pads.
> :maxdepth: 1
>
> subdev-formats
> +
> +Streams, multiplexed media pads and internal routing
> +----------------------------------------------------
> +
> +Commonly V4L2 subdevices support only separate video streams, that is, each
> +link in the media graph and each pad in a subdevice pass through a single
> +video stream. Thus each pad contains a format configuration for that single
Isn't it the other way around ? A single video stream passes through a
media link.
> +stream. In some cases a subdev can do stream processing and split a stream
> +into two or compose two streams into one, but the inputs and outputs for the
> +subdev are still a single stream per pad.
> +
> +Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
> +bus carries multiple streams. Thus a camera could output two streams, a pixel
> +stream and a metadata stream, and a bridge subdev could route the streams
> +from multiple input pads into a single output pad.
I would use sink/source and not input/output and slightly rephrase
this, because I understand your "bridge subdev" in this example is a
CSI-2 transmitter but "bridge" usually refers to receiver drivers on
the SoC.
What about
Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is,
multiple data streams are transmitted on the same bus, which is
represented by a media link connecting a transmitter source pad with a
sink pad on the receiver. In example, a camera sensor can produce two
distinct streams, a pixel stream and a metadata stream, which are
transmitted on the multiplexed data bus, represented by a media link
which connects the single sensor's source pad with the receiver sink
pad. The stream-aware receiver will de-multiplex the streams received
on the its sink pad and allows to route them individually to one of
its source pads.
Sensor Receiver
+----------+ +--------------+
| Pixel | | ___________(1)
| \_(0) --------(0)/ |
| Meta / | | \___________(2)
| | | |
+----------+ +---------------
(not sure if makes sense to have a drawing here, but... )
> +
> +Subdevice drivers that support multiplexed streams are compatible with
> +non-multiplexed subdev drivers, but, of course, require a routing configuration
> +where the link between those two types of drivers contains only a single
> +stream.
> +
> +Understanding streams
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +A stream is a stream of content (e.g. pixel data or metadata) flowing through
> +the media pipeline from a source (e.g. a sensor) towards the final sink(e.g. a
space between "sink("
> +receiver and demultiplexer in a SoC). Each media link carries all the streams
the enabled streams ?
> +from one end of the link to the other, and subdevices have routing tables which
> +describe how the incoming streams from sink pads are routed to the source
> +pads.
"... from sink pad", singular..
I would rather say "how the incoming streams from the multiplexed sink
pad are routed to source pads".
> +
> +A stream ID (often just "stream") is a media link-local identifier for a
> +stream. In other words, a configuration for a particular stream ID must exist
s/In other words//
Should the configuration on both ends of the link also be identical ?
> +on both sides of a media link, but another stream ID can be used for the same
> +stream at the other side of the subdevice.
I would
A stream ID (often just "stream") is a media link-local identifier for a
stream. The configuration for a stream ID must exist and be identical
on both ends of a media link connecting two multiplexed pads.
and leave the bits about routing out ?
> +
> +A stream at a specific point in the media pipeline is identified with the
> +subdev and a (pad, stream) pair. For subdevices that do not support
> +multiplexed streams the 'stream' is always 0.
> +
> +Configuring streams
> +^^^^^^^^^^^^^^^^^^^
> +
> +The configuration of the streams is done individually for each subdevice and
> +the validity of the streams between subdevices is validated when the pipeline
> +is started.
> +
> +There are three steps in configuring the streams:
> +
> +1) Set up links. Connect the pads between subdevices using the :ref:`Media
> +Controller API <media_controller>`
> +
> +2) Routing. The routing table for the subdevice must be set with
> +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
> +setting the routing table will reset all the stream configurations.
"stream configurations in a media entity".
> +
> +3) Configure streams. Each route endpoint must be configured
> +with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
> +
> +Multiplexed streams setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
> + source pad (pad 0), and outputs two streams, pixel data and metadata.
a single source pad (pad 0) which carries two streams: a pixel data
stream and a metadata stream.
> +
> +- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
> + sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
> + streams.
> +
> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> + connected to the bridge, and four source pads (pads 1-4), going to the DMA
> + engine. The receiver demultiplexes the incoming streams to the four source
> + pads.
> +
> +- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
> + single source pad in the receiver.
> +
> +The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: connect the sensors to the bridge,
> +bridge to the receiver, and the receiver to the DMA engines. This step does
> +not differ from normal non-multiplexed media controller setup.
> +
> +2) Configure routing.
> +
> +.. flat-table:: Sensor routing table (identical on both sensors)
> + :header-rows: 1
> +
> + * - Sink Pad/Stream
> + - Source Pad/Stream
> + - Routing Flags
> + - Comments
> + * - 0/0 (unused)
> + - 0/0
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
> + - Pixel data stream. Source route, i.e. the sink fields are unused.
> + * - 0/0 (unused)
> + - 0/1
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
> + - Metadata stream. Source route, i.e. the sink fields are unused.
> +
> +.. flat-table:: Bridge routing table
> + :header-rows: 1
> +
> + * - Sink Pad/Stream
> + - Source Pad/Stream
> + - Routing Flags
> + - Comments
> + * - 0/0
> + - 2/0
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Pixel data stream from Sensor A
> + * - 0/1
> + - 2/1
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Metadata stream from Sensor A
> + * - 1/0
> + - 2/2
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Pixel data stream from Sensor B
> + * - 1/1
> + - 2/3
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Metadata stream from Sensor B
> +
> +.. flat-table:: Receiver routing table
> + :header-rows: 1
> +
> + * - Sink Pad/Stream
> + - Source Pad/Stream
> + - Routing Flags
> + - Comments
> + * - 0/0
> + - 1/0
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Pixel data stream from Sensor A
> + * - 0/1
> + - 2/0
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Metadata stream from Sensor A
> + * - 0/2
> + - 3/0
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Pixel data stream from Sensor B
> + * - 0/3
> + - 4/0
> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> + - Metadata stream from Sensor B
> +
> +3) Configure streams
> +
> +After configuring the routing table, the next step is configuring the streams.
> +This step is similar to configuring the pads in a non-multiplexed streams
> +setup, with the difference that we need to configure each (pad, stream) pair
> +(i.e. route endpoint) instead of just a pad.
> +
> +A common way to accomplish this is to start from the sensors and propagate the
> +configurations along the stream towards the receiver, using VIDIOC_SUBDEV_S_FMT
using the :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls
> +to configure each stream endpoint in each subdev.
All minors or just suggestions:
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
Thanks
j
> --
> 2.25.1
>
On 16/03/2022 11:04, Jacopo Mondi wrote:
> Hi Tomi
>
> On Tue, Mar 01, 2022 at 06:11:45PM +0200, Tomi Valkeinen wrote:
>> Add documentation related to multiplexed streams.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>> .../driver-api/media/v4l2-subdev.rst | 8 +
>> .../userspace-api/media/v4l/dev-subdev.rst | 165 ++++++++++++++++++
>> 2 files changed, 173 insertions(+)
>>
>> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
>> index 115211cef4d1..80654f1bcac9 100644
>> --- a/Documentation/driver-api/media/v4l2-subdev.rst
>> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
>> @@ -593,6 +593,14 @@ before calling v4l2_subdev_init_finalize():
>>
>> This shares the driver's private mutex between the controls and the states.
>>
>> +Streams, multiplexed media pads and internal routing
>> +----------------------------------------------------
>> +
>> +A subdevice driver can implement support for multiplexed streams by setting
>
> Let me start by being picky with a minor thing: the rest of the
> documentation seems to use "sub-device". Here you have "sub-device",
> "subdevice" and "subdev". I think "sub-device" should be used
> everywhere
I can see lots of all those three in the docs, so I don't really know
which one to pick...
>> +the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
>> +centrally managed subdev active state, routing and stream based
>> +configuration.
>> +
>> V4L2 sub-device functions and data structures
>> ---------------------------------------------
>>
>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> index a67c2749089a..fd042afeddd6 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> @@ -503,3 +503,168 @@ source pads.
>> :maxdepth: 1
>>
>> subdev-formats
>> +
>> +Streams, multiplexed media pads and internal routing
>> +----------------------------------------------------
>> +
>> +Commonly V4L2 subdevices support only separate video streams, that is, each
>> +link in the media graph and each pad in a subdevice pass through a single
>> +video stream. Thus each pad contains a format configuration for that single
>
> Isn't it the other way around ? A single video stream passes through a
> media link.
It's a bit confusing. I meant "pass through" as in "each link and pad
let a single stream pass through". How about:
Commonly V4L2 subdevices support only separate video streams, that is,
only a single stream can pass through a media link and a media pad.
>> +stream. In some cases a subdev can do stream processing and split a stream
>> +into two or compose two streams into one, but the inputs and outputs for the
>> +subdev are still a single stream per pad.
>> +
>> +Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
>> +bus carries multiple streams. Thus a camera could output two streams, a pixel
>> +stream and a metadata stream, and a bridge subdev could route the streams
>> +from multiple input pads into a single output pad.
>
> I would use sink/source and not input/output and slightly rephrase
> this, because I understand your "bridge subdev" in this example is a
> CSI-2 transmitter but "bridge" usually refers to receiver drivers on
> the SoC.
Ok. In DRM, bridges are not the display controllers in the SoC, but the
"bridges" between the SoC and the panel.
> What about
>
> Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is,
> multiple data streams are transmitted on the same bus, which is
> represented by a media link connecting a transmitter source pad with a
> sink pad on the receiver. In example, a camera sensor can produce two
> distinct streams, a pixel stream and a metadata stream, which are
> transmitted on the multiplexed data bus, represented by a media link
> which connects the single sensor's source pad with the receiver sink
> pad. The stream-aware receiver will de-multiplex the streams received
> on the its sink pad and allows to route them individually to one of
> its source pads.
This sounds fine.
> Sensor Receiver
> +----------+ +--------------+
> | Pixel | | ___________(1)
> | \_(0) --------(0)/ |
> | Meta / | | \___________(2)
> | | | |
> +----------+ +---------------
>
> (not sure if makes sense to have a drawing here, but... )
>
>> +
>> +Subdevice drivers that support multiplexed streams are compatible with
>> +non-multiplexed subdev drivers, but, of course, require a routing configuration
>> +where the link between those two types of drivers contains only a single
>> +stream.
>> +
>> +Understanding streams
>> +^^^^^^^^^^^^^^^^^^^^^
>> +
>> +A stream is a stream of content (e.g. pixel data or metadata) flowing through
>> +the media pipeline from a source (e.g. a sensor) towards the final sink(e.g. a
>
> space between "sink("
>
>> +receiver and demultiplexer in a SoC). Each media link carries all the streams
>
> the enabled streams ?
>
>> +from one end of the link to the other, and subdevices have routing tables which
>> +describe how the incoming streams from sink pads are routed to the source
>> +pads.
>
> "... from sink pad", singular..
No, the routing table describes how the streams from the sink pads are
routed to the source pads. Why would it be singular?
> I would rather say "how the incoming streams from the multiplexed sink
> pad are routed to source pads".
>
>> +
>> +A stream ID (often just "stream") is a media link-local identifier for a
>> +stream. In other words, a configuration for a particular stream ID must exist
>
> s/In other words//
Why would you remove it? The sentence explains the previous sentence in
a more verbose way.
> Should the configuration on both ends of the link also be identical ?
If you refer to the format etc. config, the same rules apply here as for
non-multiplexed case. I'm not sure if they need to be identical, but
they have to match. But I'm not really talking about configuration here.
>> +on both sides of a media link, but another stream ID can be used for the same
>> +stream at the other side of the subdevice.
>
> I would
>
> A stream ID (often just "stream") is a media link-local identifier for a
> stream. The configuration for a stream ID must exist and be identical
> on both ends of a media link connecting two multiplexed pads.
You're describing a different thing there. I'm talking about the stream
IDs. I think it becomes clearer if I remove "a configuration for". So:
> A stream ID (often just "stream") is a media link-local identifier for a stream.
> In other words, a particular stream ID must exist on both sides of a media
> link, but another stream ID can be used for the same stream at the other side
> of the subdevice.
So it's explaining the stream IDs, how a single stream can be
represented by multiple stream IDs.
> and leave the bits about routing out ?
>
>> +
>> +A stream at a specific point in the media pipeline is identified with the
>> +subdev and a (pad, stream) pair. For subdevices that do not support
>> +multiplexed streams the 'stream' is always 0.
>> +
>> +Configuring streams
>> +^^^^^^^^^^^^^^^^^^^
>> +
>> +The configuration of the streams is done individually for each subdevice and
>> +the validity of the streams between subdevices is validated when the pipeline
>> +is started.
>> +
>> +There are three steps in configuring the streams:
>> +
>> +1) Set up links. Connect the pads between subdevices using the :ref:`Media
>> +Controller API <media_controller>`
>> +
>> +2) Routing. The routing table for the subdevice must be set with
>> +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
>> +setting the routing table will reset all the stream configurations.
>
> "stream configurations in a media entity".
Ok.
>> +
>> +3) Configure streams. Each route endpoint must be configured
>> +with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
>> +
>> +Multiplexed streams setup example
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +A simple example of a multiplexed stream setup might be as follows:
>> +
>> +- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
>> + source pad (pad 0), and outputs two streams, pixel data and metadata.
>
> a single source pad (pad 0) which carries two streams: a pixel data
> stream and a metadata stream.
Ok.
>> +
>> +- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
>> + sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
>> + streams.
>> +
>> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
>> + connected to the bridge, and four source pads (pads 1-4), going to the DMA
>> + engine. The receiver demultiplexes the incoming streams to the four source
>> + pads.
>> +
>> +- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
>> + single source pad in the receiver.
>> +
>> +The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
>> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
>> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
>> +
>> +To configure this pipeline, the userspace must take the following steps:
>> +
>> +1) Set up media links between entities: connect the sensors to the bridge,
>> +bridge to the receiver, and the receiver to the DMA engines. This step does
>> +not differ from normal non-multiplexed media controller setup.
>> +
>> +2) Configure routing.
>> +
>> +.. flat-table:: Sensor routing table (identical on both sensors)
>> + :header-rows: 1
>> +
>> + * - Sink Pad/Stream
>> + - Source Pad/Stream
>> + - Routing Flags
>> + - Comments
>> + * - 0/0 (unused)
>> + - 0/0
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
>> + - Pixel data stream. Source route, i.e. the sink fields are unused.
>> + * - 0/0 (unused)
>> + - 0/1
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
>> + - Metadata stream. Source route, i.e. the sink fields are unused.
>> +
>> +.. flat-table:: Bridge routing table
>> + :header-rows: 1
>> +
>> + * - Sink Pad/Stream
>> + - Source Pad/Stream
>> + - Routing Flags
>> + - Comments
>> + * - 0/0
>> + - 2/0
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Pixel data stream from Sensor A
>> + * - 0/1
>> + - 2/1
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Metadata stream from Sensor A
>> + * - 1/0
>> + - 2/2
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Pixel data stream from Sensor B
>> + * - 1/1
>> + - 2/3
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Metadata stream from Sensor B
>> +
>> +.. flat-table:: Receiver routing table
>> + :header-rows: 1
>> +
>> + * - Sink Pad/Stream
>> + - Source Pad/Stream
>> + - Routing Flags
>> + - Comments
>> + * - 0/0
>> + - 1/0
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Pixel data stream from Sensor A
>> + * - 0/1
>> + - 2/0
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Metadata stream from Sensor A
>> + * - 0/2
>> + - 3/0
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Pixel data stream from Sensor B
>> + * - 0/3
>> + - 4/0
>> + - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> + - Metadata stream from Sensor B
>> +
>> +3) Configure streams
>> +
>> +After configuring the routing table, the next step is configuring the streams.
>> +This step is similar to configuring the pads in a non-multiplexed streams
>> +setup, with the difference that we need to configure each (pad, stream) pair
>> +(i.e. route endpoint) instead of just a pad.
>> +
>> +A common way to accomplish this is to start from the sensors and propagate the
>> +configurations along the stream towards the receiver, using VIDIOC_SUBDEV_S_FMT
>
> using the :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls
>
>> +to configure each stream endpoint in each subdev.
>
> All minors or just suggestions:
>
> Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
Thanks!
Tomi
Hi Tomi
On Thu, Mar 17, 2022 at 09:18:42AM +0200, Tomi Valkeinen wrote:
> On 16/03/2022 11:04, Jacopo Mondi wrote:
> > Hi Tomi
> >
> > On Tue, Mar 01, 2022 at 06:11:45PM +0200, Tomi Valkeinen wrote:
> > > Add documentation related to multiplexed streams.
> > >
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > > .../driver-api/media/v4l2-subdev.rst | 8 +
> > > .../userspace-api/media/v4l/dev-subdev.rst | 165 ++++++++++++++++++
> > > 2 files changed, 173 insertions(+)
> > >
> > > diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
> > > index 115211cef4d1..80654f1bcac9 100644
> > > --- a/Documentation/driver-api/media/v4l2-subdev.rst
> > > +++ b/Documentation/driver-api/media/v4l2-subdev.rst
> > > @@ -593,6 +593,14 @@ before calling v4l2_subdev_init_finalize():
> > >
> > > This shares the driver's private mutex between the controls and the states.
> > >
> > > +Streams, multiplexed media pads and internal routing
> > > +----------------------------------------------------
> > > +
> > > +A subdevice driver can implement support for multiplexed streams by setting
> >
> > Let me start by being picky with a minor thing: the rest of the
> > documentation seems to use "sub-device". Here you have "sub-device",
> > "subdevice" and "subdev". I think "sub-device" should be used
> > everywhere
>
> I can see lots of all those three in the docs, so I don't really know which
> one to pick...
>
You're right, the number of "subdev" entries is larger that what I
first thought. Let's keep using whatever then :)
@@ -593,6 +593,14 @@ before calling v4l2_subdev_init_finalize():
This shares the driver's private mutex between the controls and the states.
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
V4L2 sub-device functions and data structures
---------------------------------------------
@@ -503,3 +503,168 @@ source pads.
:maxdepth: 1
subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, each
+link in the media graph and each pad in a subdevice pass through a single
+video stream. Thus each pad contains a format configuration for that single
+stream. In some cases a subdev can do stream processing and split a stream
+into two or compose two streams into one, but the inputs and outputs for the
+subdev are still a single stream per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
+bus carries multiple streams. Thus a camera could output two streams, a pixel
+stream and a metadata stream, and a bridge subdev could route the streams
+from multiple input pads into a single output pad.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink(e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the streams
+from one end of the link to the other, and subdevices have routing tables which
+describe how the incoming streams from sink pads are routed to the source
+pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a
+stream. In other words, a configuration for a particular stream ID must exist
+on both sides of a media link, but another stream ID can be used for the same
+stream at the other side of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
+ source pad (pad 0), and outputs two streams, pixel data and metadata.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+ sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+ streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+ connected to the bridge, and four source pads (pads 1-4), going to the DMA
+ engine. The receiver demultiplexes the incoming streams to the four source
+ pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+ single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+ :header-rows: 1
+
+ * - Sink Pad/Stream
+ - Source Pad/Stream
+ - Routing Flags
+ - Comments
+ * - 0/0 (unused)
+ - 0/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+ - Pixel data stream. Source route, i.e. the sink fields are unused.
+ * - 0/0 (unused)
+ - 0/1
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+ - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+ :header-rows: 1
+
+ * - Sink Pad/Stream
+ - Source Pad/Stream
+ - Routing Flags
+ - Comments
+ * - 0/0
+ - 2/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Pixel data stream from Sensor A
+ * - 0/1
+ - 2/1
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Metadata stream from Sensor A
+ * - 1/0
+ - 2/2
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Pixel data stream from Sensor B
+ * - 1/1
+ - 2/3
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+ :header-rows: 1
+
+ * - Sink Pad/Stream
+ - Source Pad/Stream
+ - Routing Flags
+ - Comments
+ * - 0/0
+ - 1/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Pixel data stream from Sensor A
+ * - 0/1
+ - 2/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Metadata stream from Sensor A
+ * - 0/2
+ - 3/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Pixel data stream from Sensor B
+ * - 0/3
+ - 4/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint) instead of just a pad.
+
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver, using VIDIOC_SUBDEV_S_FMT
+to configure each stream endpoint in each subdev.