[v12,00/30] v4l: routing and streams support

  Hi,

This is v12 of the streams series. The v11 can be found from:

https://lore.kernel.org/all/20220301161156.1119557-1-tomi.valkeinen@ideasonboard.com/

My work branch with additional drivers can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git streams/work-v12

And there's also the v4l-utils series v2 to add support to v4l2-ctl and
media-ctl:

https://lore.kernel.org/all/20220714132116.132498-1-tomi.valkeinen@ideasonboard.com/

Changes in v12:

- Rebased on top of upstream (media graph walk changes, subdev active
  state changes)

- V4L2_SUBDEV_FL_MULTIPLEXED renamed to V4L2_SUBDEV_FL_STREAMS

- V4L2_SUBDEV_CAP_MPLEXED renamed to V4L2_SUBDEV_CAP_STREAMS

- Limit Stream ID number to max 63

- Added v4l2_subdev_state_get_stream_crop() and
  v4l2_subdev_state_get_stream_compose()

- Use BIT_ULL() instead of BIT() for 64bit masks.

- Some documentation improvements

- All the patches related to modifying the graph walk have been dropped
  and there is a new implementation of the pipeline construction in
  __media_pipeline_start.

- The change which made media_pipeline_start() take a pad instead of an
  entity has been dropped.

- New pipeline start/stop functions, media_pipeline_alloc_start() &
  media_pipeline_stop_free()

There are still open discussion items wrt. the uAPI. I will post a mail
in reply to this cover letter about those items.

 Tomi

Jacopo Mondi (3):
  media: mc: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation
  media: subdev: Add for_each_active_route() macro

Laurent Pinchart (10):
  media: mc: entity: Merge media_entity_enum_init and
    __media_entity_enum_init
  media: mc: entity: Move media_entity_get_fwnode_pad() out of graph
    walk section
  media: mc: entity: Add media_entity_pipeline() to access the media
    pipeline
  media: mc: entity: Add has_route entity operation and
    media_entity_has_route() helper
  media: mc: entity: Rewrite media_pipeline_start() to support routes
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations
  media: subdev: add v4l2_subdev_routing_validate() helper
  media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper
  media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations
  media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

Sakari Ailus (1):
  media: Add stream to frame descriptor

Tomi Valkeinen (16):
  media: Documentation: mc: add definitions for stream and pipeline
  media: mc: entity: Rename streaming_count -> start_count
  media: mc: entity: add media_pipeline_alloc_start &
    media_pipeline_stop_free
  media: add V4L2_SUBDEV_FL_STREAMS
  media: add V4L2_SUBDEV_CAP_MPLEXED
  media: subdev: add v4l2_subdev_has_route()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add streams to v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper
  media: subdev: use for_each_active_route() in
    v4l2_subdev_init_stream_configs()
  media: subdev: use for_each_active_route() in
    v4l2_link_validate_get_streams()
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

 .clang-format                                 |    1 +
 Documentation/driver-api/media/mc-core.rst    |   19 +-
 .../driver-api/media/v4l2-subdev.rst          |    8 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  175 +++
 .../userspace-api/media/v4l/user-func.rst     |    1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |    5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |    5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |    5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |    5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |    5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |    5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     |  150 +++
 .../media/v4l/vidioc-subdev-g-selection.rst   |    5 +-
 drivers/media/mc/mc-device.c                  |   13 +-
 drivers/media/mc/mc-entity.c                  |  646 +++++++++--
 .../platform/renesas/rcar-vin/rcar-core.c     |    5 +-
 .../platform/renesas/rcar-vin/rcar-dma.c      |    2 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |    4 +-
 drivers/media/platform/ti/omap3isp/isp.c      |    4 +-
 drivers/media/platform/ti/omap3isp/ispvideo.c |    3 +-
 drivers/media/platform/ti/omap3isp/ispvideo.h |   11 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |    3 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |    7 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 1033 ++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |    2 +-
 drivers/staging/media/omap4iss/iss.c          |    4 +-
 drivers/staging/media/omap4iss/iss_video.c    |    3 +-
 drivers/staging/media/omap4iss/iss_video.h    |   11 +-
 include/media/media-device.h                  |   15 -
 include/media/media-entity.h                  |  130 ++-
 include/media/v4l2-subdev.h                   |  381 +++++-
 include/uapi/linux/v4l2-subdev.h              |   88 +-
 33 files changed, 2559 insertions(+), 220 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
  

Hi,

# uAPI

This is both a summary of the uAPI changes for routing and streams 
support and a list of the open questions.

# Subdev Cap

Subdev capability V4L2_SUBDEV_CAP_STREAMS tells the userspace that the 
subdev supports routing and streams.

# Stream ID

We use u64 bitmask inside the kernel to represent a set of streams in a 
pad, each bit representing a stream ID, so the stream ID number is 
currently limited to a max of 63 (i.e. max 64  streams per pad). The 
userspace API doesn't have this limitation at the moment, as the stream 
ID number is passed as u32, and we don't have stream masks in the uAPI 
structs. However, the kernel checks that the Stream IDs given in 
S_ROUTING are < 64.

In theory we can increase this number later, as it won't change the 
uAPI. In practice there will be some challenges in how to do the change 
and how to find all the places in the kernel which depend on this.

If it's clear that 64 streams won't be enough, perhaps we already need 
to take some steps toward this. Maybe hiding the type behind a typedef 
and creating a set of macros/functions which need to be used to access 
the data.

# Routing

```
#define VIDIOC_SUBDEV_G_ROUTING                        _IOWR('V', 38, 
struct v4l2_subdev_routing)
#define VIDIOC_SUBDEV_S_ROUTING                        _IOWR('V', 39, 
struct v4l2_subdev_routing)

#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		_BITUL(1)
#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)

struct v4l2_subdev_route {
	__u32 sink_pad;
	__u32 sink_stream;
	__u32 source_pad;
	__u32 source_stream;
	__u32 flags;
	__u32 reserved[5];
};

struct v4l2_subdev_routing {
	__u32 which;
	__u32 num_routes;
	__u64 routes;
	__u32 reserved[6];
};
```

- There's a maximum of 256 for the number of routes in the routing 
table, but there's nothing in the code that relies on this limit. It is 
there just to limit the memory allocations the kernel will do based on 
the routing table we get from the userspace.

- Using S_ROUTING will overwrite the old routing table. All the 
configurations (e.g. formats) for the streams on that subdevice are lost 
(reset to defaults), so the userspace must reconfigure the subdev after 
S_ROUTING.

- If V4L2_SUBDEV_ROUTE_FL_SOURCE flag is present, the sink_pad and 
sink_stream are ignored. This is used e.g. for sensors.

- I think V4L2_SUBDEV_ROUTE_FL_IMMUTABLE should be dropped. I don't 
really know what it means or how to use it. It was there in the original 
routing patches and so far I have kept it.

- Usually V4L2_SUBDEV_ROUTE_FL_ACTIVE is not strictly needed, as the 
existence of a route in the routing table implies that the route is 
active. However, it can be used for cases where the routing table is not 
freely modifiable. E.g. a sensor that supports both a pixel stream and a 
metadata stream could always have both in its routing table, metadta 
stream inactive by default, and the user could use 
V4L2_SUBDEV_ROUTE_FL_ACTIVE to enable/disable the metadata stream.

# Stream/routing discovery of a device

We currently don't have much for this.

For example:

- How many streams a device can support?
- What kind of routing can the device do?
- What are the streams? If a sensor supports a pixel stream and a 
metadata stream, how does the userspace know which stream ID represents 
which?

I believe this is a very complex issue, as it is perhaps impossible to 
somehow describe complex device's capabilities in a form that the 
userspace could actually use.

Options:
- Just document the device's & driver's behavior and let the userspace 
deal with it. I think this is acceptable, as when streams and routing 
are involved the userspace more or less has to have detailed 
understanding of the HW in question.
- In addition to the above, for source devices we can use the (absence 
of) V4L2_SUBDEV_ROUTE_FL_ACTIVE as described above to inform the 
userspace about the available streams.
- Try to add some kind of discovery mechanism that covers the HW we have 
now, or perhaps only for "simple" pipelines, understanding that it won't 
possibly work for future HWs.

# Selection

I admit I'm not very familiar with the cropping, scaling and composition 
in v4l2 subdevs. I believe the straightforward cases should work fine 
with streams, as there should be no practical difference between a 
legacy case of one sink pad + one source pad (and implicit routing 
between those two), and a stream coming to one pad and routed to a 
source pad.

Possibly problems could arise on cases where a stream is split into two 
streams, or two streams are merged into one stream.

I'd appreciate if someone familiar with the subdev selection could study 
this and if there are problems, explain the setup where it arises.

# Streams

`__u32 stream` field has been added to the following structs:
- v4l2_subdev_format
- v4l2_subdev_crop
- v4l2_subdev_mbus_code_enum
- v4l2_subdev_frame_size_enum
- v4l2_subdev_frame_interval
- v4l2_subdev_frame_interval_enum
- v4l2_subdev_selection

In other words, all the above operations support streams, and for the 
subdevs which do not support streams, they operate on the implicit 
stream ID 0.

  Tomi
  

Moi,

On Wed, Jul 27, 2022 at 01:36:09PM +0300, Tomi Valkeinen wrote:
> Hi,
> 
> This is v12 of the streams series. The v11 can be found from:

Thanks for the update. This continues to be a set that will need changes
before merging, hopefully less so than in the past. But at the same time I
think there are a number of patches early in the set that could be merged
now. Specifically I mean patches 1--5 and 7.

I'll pick those if there are no objections once we have rc1 in media tree.
That should at least make it a little easier to work with the rest.
  

Hi,

On 31/07/2022 23:47, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Jul 27, 2022 at 01:36:09PM +0300, Tomi Valkeinen wrote:
>> Hi,
>>
>> This is v12 of the streams series. The v11 can be found from:
> 
> Thanks for the update. This continues to be a set that will need changes
> before merging, hopefully less so than in the past. But at the same time I
> think there are a number of patches early in the set that could be merged
> now. Specifically I mean patches 1--5 and 7.
> 
> I'll pick those if there are no objections once we have rc1 in media tree.
> That should at least make it a little easier to work with the rest.

I'm fine with that.

Everything up to, and including, patch 9 are kernel-only patches, they 
don't change the uAPI.

  Tomi
  

Hi,

On 03/08/2022 12:03, Milen Mitkov (Consultant) wrote:
> Hey Tomi,
> 
> thank you for providing v12 of the routing and streams support patches! 
> We are using v11 of these to add CSI virtual channels support to the 
> Qualcomm Titan platform (a.k.a qcom-camss driver) and will be moving to 
> the brand new fresh v12 now.
> 
> I ponder several questions with regards to this newly added 
> functionality (not just v12, but all versions in general):
> 
> 1. What are the main benefits of having multiple streams, that can be 
> enabled/disabled/configured, on the same pad vs. say having multiple 
> pads, each of which can be enabled/disabled/configured?

The streams and pads are kind of similar, but also different.

One difference is a conceptual one, that a pad represents some kind of 
physical connector, streams are logical "virtual" pads or connectors.

But perhaps the main practical difference is that you had a hardcoded 
amount of pads, but the amount of streams is dynamic, adjusted based on 
the routing table.

> As far as I understood, in the user-space world each stream on the end 
> of the pipeline gets exposed as a /dev/video%d node. Each node 
> corresponds to a videodev which is wrapped in a media entity that has a 

Yes.

It would be nice to have a videodev that supports multiple streams, but 
at the moment there's no API for that kind of thing. Perhaps in the future.

> single sink pad. So in the end, all streams end up as multiple "stream 
> -> pad" correspondences. I am sure there is some benefit of having 
> multiple streams vs having multiple pads, but I am struggling to put it 
> into exact words.

Consider a bridge device with, say, 2 CSI2 inputs and 2 CSI2 outputs. 
The device can route streams from either input to either output, and 
possibly modify them (say, change virtual channel number).

How many pads would have there? You would need a predefined amount of 
pads, one for each stream. So how many streams? What's a stream? With 
CSI2, we can easily define at least that we can have streams identified 
with the VC and the DT. So, 4 VCs at max, but there can be many DTs. I 
don't remember how wide the DT field is, but lets say 10 DTs. That would 
be max 40 streams per input.

So the above device would need to have 40 * 4 pads to cover "all" 
possible uses. I say "all", because it's not clear how to define a 
stream. If the device in question can, say, split the incoming frame per 
line, and somehow output each of those lines separately, then, 
effectively, there would be as many streams as there are lines. That's a 
silly example, but I just want to highlight the dynamic nature of streams.

> 2. What is your userspace modus operandi with regards to testing these 

I have my own python scripts built on top of kms++. They're really not 
in such a condition that I could share them to others. Maybe I'll at 
some point find the time to clean them up...

> changes? For example, I figured out this much with regards to media-ctl:

Yes, it should be all doable with media-ctl, v4l2-ctl.

> media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->1/1[0x1]]'
> 
> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1 
> stream on the sink pad and route it to 2 streams on its source pad. Is 
> my thinking correct?

Yes, if your HW can do that. I don't have HW that can split (or clone) a 
stream, so it's possible that the use case doesn't work.

> And I also wonder what is your preferred method to open each /dev/video 
> node in userspace concurrently? Are you, say, running 2 or more parallel 
> instances of yavta?

I do it with my script, but yes, I think running multiple yavtas should 
do the trick.

> 3. I assume, that with these changes, it's _strongly_ preferred that the 
> subdevice's active state management is left to the V4L2 API and not kept 
> internally like older drivers do?

A subdev that uses routing/streams _must_ use the subdev active state.

  Tomi
  

On 05/08/2022 18:14, Milen Mitkov (Consultant) wrote:

>  >> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1
>  >> stream on the sink pad and route it to 2 streams on its source pad. Is
>  >> my thinking correct?
>  >
>  >Yes, if your HW can do that. I don't have HW that can split (or clone) a
>  >stream, so it's possible that the use case doesn't work.
> 
> Now here's the main question. We use the CSI decoder (CSID) hardware to 
> split
> one stream from the sensor into 2 or more streams based on datatype or 
> CSI virtual channel.
> 
> Basically, the complete pipeline is something like this, for 2 virtual 
> channels:
> 
>                                             -> ISP line 0 -> videodev 
> /dev/video0
>                                           /
> sensor -> CSIPHY -> CSID ->   
>                                           \
>                                           -> ISP line 1 -> videodev 
> /dev/video1
>                                     
>                                     
> So my idea was to make the CSID subdevice multistream API compliant
> (e.g. V4L2_SUBDEV_FL_STREAMS, manage the active state with the
> V4L2 API v4l2_subdev_get_locked_active_state, take care of routing setup 
> etc.),
> but keep the rest of the devices the way they are.

That's now how the streams support has been designed. Your sensor 
provides two streams, and all the drivers that pass through multiple 
streams needs to be ported to streams API. So in your case, I believe 
everything but the "ISP line" needs to support streams.

> The CSID subdev must take 1 stream on the sink pad and output on 2 
> source pads.
> 
> The routing configuration I use for the CSID subdev looks like this:
> 
> media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->2/0[0x1]]'
> 
> 0 - sink pad, 1 - first source pad, 2 - second source pad
> 
> However, this routing setup fails with the validation in 
> v4l2_link_validate_get_streams().
> The logic there figures these are duplicate streams because because they 
> start at the same sink pad.
> 
> To summarize my questions:
> 
> 1. Is there some sort of restriction that the same sink pad can't be 
> used for more than 1 stream starting from it?

In theory no, but it hasn't been tested. I think this case would mainly 
be cloning of the stream, not really splitting it.

> 2. Is it ok to migrate only one subdevice to the multistream API
> or should all possible subdevices in the pipeline be migrated?

It's ok to mix streams and non-streams subdevices, but the non-streams 
subdevs must only use a single stream. E.g. You could have 4 non-streams 
cameras, each providing a single stream to a bridge. The bridge would 
support streams, and the bridge would send the 4 streams in a single 
CSI-2 bus.

Now, that said, I don't think anything strictly prevents from supporting 
stream splitting, but as I mentioned above, it's not been tested or 
really even considered very much. It's also a bit ambiguous and unclear 
and I'd stay away from it if full streams support makes sense.

I think if a source subdev (sensor) knows that it's providing multiple 
streams, then it should use streams API to provide those. I.e. if the 
sensor is providing different types of data, using VCs or DTs, then 
those are clearly separate streams and the sensor driver must be aware 
of them.

Stream splitting might came into play in situations where the sensor 
provides just a single stream, but a bridge subdev splits it based on 
information the sensor can't be aware of. For example, the sensor 
provides a normal pixel stream, and the bridge subdev splits the frames 
into two halves, sending upper half to output 1 and lower half to output 2.

  Tomi
  

On Mon, Aug 08, 2022 at 09:45:38AM +0300, Tomi Valkeinen wrote:
> On 05/08/2022 18:14, Milen Mitkov (Consultant) wrote:
> 
> > >> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1
> > >> stream on the sink pad and route it to 2 streams on its source pad. Is
> > >> my thinking correct?
> > >
> > > Yes, if your HW can do that. I don't have HW that can split (or clone) a
> > > stream, so it's possible that the use case doesn't work.
> > 
> > Now here's the main question. We use the CSI decoder (CSID) hardware to 
> > split
> > one stream from the sensor into 2 or more streams based on datatype or 
> > CSI virtual channel.
> > 
> > Basically, the complete pipeline is something like this, for 2 virtual 
> > channels:
> > 
> >                               -> ISP line 0 -> videodev /dev/video0
> >                             /
> > sensor -> CSIPHY -> CSID ->
> >                             \
> >                               -> ISP line 1 -> videodev /dev/video1
> >                                     
> >                                     
> > So my idea was to make the CSID subdevice multistream API compliant
> > (e.g. V4L2_SUBDEV_FL_STREAMS, manage the active state with the
> > V4L2 API v4l2_subdev_get_locked_active_state, take care of routing setup 
> > etc.),
> > but keep the rest of the devices the way they are.
> 
> That's now how the streams support has been designed. Your sensor 
> provides two streams, and all the drivers that pass through multiple 
> streams needs to be ported to streams API. So in your case, I believe 
> everything but the "ISP line" needs to support streams.

To add a bit of information here, the important thing to understand is
that streams and physical links are two different concepts. The above
diagram describes the physical links (both outside the SoC, and inside
it). Streams are carried by physical links, and a link can carry
multiple streams (hence the name "multiplexed streams" used in this
patch series). If the sensor outputs image data and embedded data with
two CSI-2 DT on one VC, that's two streams carried over the sensor ->
CSIPHY link, and the same two streams going over the CSIPHY -> CSID
link. The CSID demultiplexes the streams, with one stream going to ISP
line 0 and the other one to ISP line 1. As Tomi explained, every subdev
that deals with multiple streams has to implement the new API. This
includes, in this case, the sensor, the CSIPHY and the CSID.

If the sensor were to output two images in different resolutions over
two VCs, it would conceptually be the same, with two streams. If it were
to output image data, embedded data and black level lines over with 3
DTs over one VC, that would be three streams. And so on.

> > The CSID subdev must take 1 stream on the sink pad and output on 2 
> > source pads.
> > 
> > The routing configuration I use for the CSID subdev looks like this:
> > 
> > media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->2/0[0x1]]'
> > 
> > 0 - sink pad, 1 - first source pad, 2 - second source pad
> > 
> > However, this routing setup fails with the validation in 
> > v4l2_link_validate_get_streams().
> > The logic there figures these are duplicate streams because because they 
> > start at the same sink pad.
> > 
> > To summarize my questions:
> > 
> > 1. Is there some sort of restriction that the same sink pad can't be 
> > used for more than 1 stream starting from it?
> 
> In theory no, but it hasn't been tested. I think this case would mainly 
> be cloning of the stream, not really splitting it.
> 
> > 2. Is it ok to migrate only one subdevice to the multistream API
> > or should all possible subdevices in the pipeline be migrated?
> 
> It's ok to mix streams and non-streams subdevices, but the non-streams 
> subdevs must only use a single stream. E.g. You could have 4 non-streams 
> cameras, each providing a single stream to a bridge. The bridge would 
> support streams, and the bridge would send the 4 streams in a single 
> CSI-2 bus.
> 
> Now, that said, I don't think anything strictly prevents from supporting 
> stream splitting, but as I mentioned above, it's not been tested or 
> really even considered very much. It's also a bit ambiguous and unclear 
> and I'd stay away from it if full streams support makes sense.
> 
> I think if a source subdev (sensor) knows that it's providing multiple 
> streams, then it should use streams API to provide those. I.e. if the 
> sensor is providing different types of data, using VCs or DTs, then 
> those are clearly separate streams and the sensor driver must be aware 
> of them.
> 
> Stream splitting might came into play in situations where the sensor 
> provides just a single stream, but a bridge subdev splits it based on 
> information the sensor can't be aware of. For example, the sensor 
> provides a normal pixel stream, and the bridge subdev splits the frames 
> into two halves, sending upper half to output 1 and lower half to output 2.

We've tested splitting on an i.MX8MP, with two different processing
pipelines capturing the stream produced by a single YUV sensor, in
different resolutions and formats. It works (or at least worked with v11
of the streams series, I'll update the code to v13 and retest).