[v4,0/2] Enumerate all pixels formats

  The goal of this series is to let userland applications enumerate
all the supported pixels formats of a stateless decoder without
setting all the possible codec-dependent control.
That offer a simplest solution for applications to discover
supported pixels formats and possibly let them doing smarter
choice between stateless decoders.

An example of how it can be used in GStreamer to discover the
supported pixels formats for stateless decoder is available here:
https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/commits/v4l2codecs_enum_all_supported_formats?ref_type=heads

changes in version 4:
- Explicitly document that the new flags are targeting mem2mem devices.

changes in version 3:
- Add a flag to inform userspace application that driver
  as take care of the flag.

changes in version 2:
- Clarify documentation.
- Only keep V4L2_FMT_FLAG_ALL_FORMATS flag in ioctl.

Benjamin

Benjamin Gaignard (2):
  media: videodev2: Add flags to unconditionnaly enumerate pixels
    formats
  media: verisilicon: Use V4L2_FMT_FLAG_ENUM_ALL_FORMATS flag

 .../media/v4l/dev-stateless-decoder.rst          |  6 ++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst  | 11 +++++++++++
 .../media/videodev2.h.rst.exceptions             |  2 ++
 drivers/media/platform/verisilicon/hantro_v4l2.c | 16 +++++++++++++---
 drivers/media/v4l2-core/v4l2-ioctl.c             |  3 +++
 include/uapi/linux/videodev2.h                   |  2 ++
 6 files changed, 37 insertions(+), 3 deletions(-)
  

On 17/07/2024 15:14, Benjamin Gaignard wrote:
> The goal of this series is to let userland applications enumerate
> all the supported pixels formats of a stateless decoder without
> setting all the possible codec-dependent control.
> That offer a simplest solution for applications to discover
> supported pixels formats and possibly let them doing smarter
> choice between stateless decoders.
> 
> An example of how it can be used in GStreamer to discover the
> supported pixels formats for stateless decoder is available here:
> https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/commits/v4l2codecs_enum_all_supported_formats?ref_type=heads

So effectively specifying this flag makes ENUM_FMT also return
formats that do not match the bit depth.

So the AV1 (for example) compressed video uses e.g. 8 bit depth, but instead of just
listing only 8 bit uncompressed pixelformats, you want to list them for any
bit depth.

But what is the point of that if the decoder can't decode 8 bit compressed to,
say, 10 bit uncompressed video?

I actually thought that this flag would just list all formats, independent
of the output format (e.g. AV1, H264, etc.), but that does not appear to be
the case? I.e., if capture pixelformat X is only available with AV1, will that still
be listed if the output pixel is set to H264?

I think you need to describe a real use-case here, and I am not convinced about
the name of the flag either.

> 
> changes in version 4:
> - Explicitly document that the new flags are targeting mem2mem devices.
> 
> changes in version 3:
> - Add a flag to inform userspace application that driver
>   as take care of the flag.
> 
> changes in version 2:
> - Clarify documentation.
> - Only keep V4L2_FMT_FLAG_ALL_FORMATS flag in ioctl.
> 
> Benjamin
> 
> Benjamin Gaignard (2):
>   media: videodev2: Add flags to unconditionnaly enumerate pixels
>     formats

I.e.: it is not unconditionally, it still depends on the chosen codec.

Regards,

	Hans

>   media: verisilicon: Use V4L2_FMT_FLAG_ENUM_ALL_FORMATS flag
> 
>  .../media/v4l/dev-stateless-decoder.rst          |  6 ++++++
>  .../userspace-api/media/v4l/vidioc-enum-fmt.rst  | 11 +++++++++++
>  .../media/videodev2.h.rst.exceptions             |  2 ++
>  drivers/media/platform/verisilicon/hantro_v4l2.c | 16 +++++++++++++---
>  drivers/media/v4l2-core/v4l2-ioctl.c             |  3 +++
>  include/uapi/linux/videodev2.h                   |  2 ++
>  6 files changed, 37 insertions(+), 3 deletions(-)
>
  

Le 19/07/2024 à 14:57, Hans Verkuil a écrit :
> On 17/07/2024 15:14, Benjamin Gaignard wrote:
>> The goal of this series is to let userland applications enumerate
>> all the supported pixels formats of a stateless decoder without
>> setting all the possible codec-dependent control.
>> That offer a simplest solution for applications to discover
>> supported pixels formats and possibly let them doing smarter
>> choice between stateless decoders.
>>
>> An example of how it can be used in GStreamer to discover the
>> supported pixels formats for stateless decoder is available here:
>> https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/commits/v4l2codecs_enum_all_supported_formats?ref_type=heads
> So effectively specifying this flag makes ENUM_FMT also return
> formats that do not match the bit depth.
>
> So the AV1 (for example) compressed video uses e.g. 8 bit depth, but instead of just
> listing only 8 bit uncompressed pixelformats, you want to list them for any
> bit depth.
>
> But what is the point of that if the decoder can't decode 8 bit compressed to,
> say, 10 bit uncompressed video?

No decoder will do 8 bits to 10 bits (as far I knows).
The point is to be able to say that decoder could produce 10 bit frames without
setting a full sps/pps for each case (and for each supported codec).

>
> I actually thought that this flag would just list all formats, independent
> of the output format (e.g. AV1, H264, etc.), but that does not appear to be
> the case? I.e., if capture pixelformat X is only available with AV1, will that still
> be listed if the output pixel is set to H264?
>
> I think you need to describe a real use-case here, and I am not convinced about
> the name of the flag either.

I may have miss something but yes the goal is to list all formats independently
of the output format.
When a SoC have multiple decoders for the same codec, knowing the supported formats
is key to select the better one.
Since I will have to do more iteration, feel free to provide a better name for the
flag(s). I'm always bad for naming this kind of thing.


>
>> changes in version 4:
>> - Explicitly document that the new flags are targeting mem2mem devices.
>>
>> changes in version 3:
>> - Add a flag to inform userspace application that driver
>>    as take care of the flag.
>>
>> changes in version 2:
>> - Clarify documentation.
>> - Only keep V4L2_FMT_FLAG_ALL_FORMATS flag in ioctl.
>>
>> Benjamin
>>
>> Benjamin Gaignard (2):
>>    media: videodev2: Add flags to unconditionnaly enumerate pixels
>>      formats
> I.e.: it is not unconditionally, it still depends on the chosen codec.
>
> Regards,
>
> 	Hans
>
>>    media: verisilicon: Use V4L2_FMT_FLAG_ENUM_ALL_FORMATS flag
>>
>>   .../media/v4l/dev-stateless-decoder.rst          |  6 ++++++
>>   .../userspace-api/media/v4l/vidioc-enum-fmt.rst  | 11 +++++++++++
>>   .../media/videodev2.h.rst.exceptions             |  2 ++
>>   drivers/media/platform/verisilicon/hantro_v4l2.c | 16 +++++++++++++---
>>   drivers/media/v4l2-core/v4l2-ioctl.c             |  3 +++
>>   include/uapi/linux/videodev2.h                   |  2 ++
>>   6 files changed, 37 insertions(+), 3 deletions(-)
>>
>
  

On 19/07/2024 15:15, Benjamin Gaignard wrote:
> 
> Le 19/07/2024 à 14:57, Hans Verkuil a écrit :
>> On 17/07/2024 15:14, Benjamin Gaignard wrote:
>>> The goal of this series is to let userland applications enumerate
>>> all the supported pixels formats of a stateless decoder without
>>> setting all the possible codec-dependent control.
>>> That offer a simplest solution for applications to discover
>>> supported pixels formats and possibly let them doing smarter
>>> choice between stateless decoders.
>>>
>>> An example of how it can be used in GStreamer to discover the
>>> supported pixels formats for stateless decoder is available here:
>>> https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/commits/v4l2codecs_enum_all_supported_formats?ref_type=heads
>> So effectively specifying this flag makes ENUM_FMT also return
>> formats that do not match the bit depth.
>>
>> So the AV1 (for example) compressed video uses e.g. 8 bit depth, but instead of just
>> listing only 8 bit uncompressed pixelformats, you want to list them for any
>> bit depth.
>>
>> But what is the point of that if the decoder can't decode 8 bit compressed to,
>> say, 10 bit uncompressed video?
> 
> No decoder will do 8 bits to 10 bits (as far I knows).
> The point is to be able to say that decoder could produce 10 bit frames without
> setting a full sps/pps for each case (and for each supported codec).
> 
>>
>> I actually thought that this flag would just list all formats, independent
>> of the output format (e.g. AV1, H264, etc.), but that does not appear to be
>> the case? I.e., if capture pixelformat X is only available with AV1, will that still
>> be listed if the output pixel is set to H264?
>>
>> I think you need to describe a real use-case here, and I am not convinced about
>> the name of the flag either.
> 
> I may have miss something but yes the goal is to list all formats independently
> of the output format.
> When a SoC have multiple decoders for the same codec, knowing the supported formats
> is key to select the better one.
> Since I will have to do more iteration, feel free to provide a better name for the
> flag(s). I'm always bad for naming this kind of thing.

That really needs to be clarified, since in patch 1/2 it says:

+   * If the ``V4L2_FMT_FLAG_ENUM_ALL_FORMATS`` flag is set the driver must enumerate
+     all the supported formats without taking care of codec-dependent controls
+     set on the ``OUTPUT`` queue. To indicate that the driver has take care of this
+     flag it must set ``V4L2_FMT_FLAG_ALL_FORMATS`` flag for each format while
+     enumerating.

Here it just talks about 'codec-dependent controls set on the ``OUTPUT`` queue', it
doesn't say anything about the compressed pixelformat set for the OUTPUT queue.

And patch 2/2 sets the ignore_depth_match boolean, suggesting also that it is
not listing all formats, but just ignoring a specific check.

But if you list all pixelformats without taking the OUTPUT pixelformat into
account, how do you know which pixelformat is valid for which codec?

Say that only MPEG support NV12 (just for the sake of argument), and that's
what you want to use, you have no way of knowing that NV12 is specific to MPEG,
you would have to try each codec and see if NV12 is supported for that codec.

I just don't see how this can be used in practice.

What exactly is the problem you want to solve? A real-life problem, not a theoretical
one :-)

Regards,

	Hans

> 
> 
>>
>>> changes in version 4:
>>> - Explicitly document that the new flags are targeting mem2mem devices.
>>>
>>> changes in version 3:
>>> - Add a flag to inform userspace application that driver
>>>    as take care of the flag.
>>>
>>> changes in version 2:
>>> - Clarify documentation.
>>> - Only keep V4L2_FMT_FLAG_ALL_FORMATS flag in ioctl.
>>>
>>> Benjamin
>>>
>>> Benjamin Gaignard (2):
>>>    media: videodev2: Add flags to unconditionnaly enumerate pixels
>>>      formats
>> I.e.: it is not unconditionally, it still depends on the chosen codec.
>>
>> Regards,
>>
>>     Hans
>>
>>>    media: verisilicon: Use V4L2_FMT_FLAG_ENUM_ALL_FORMATS flag
>>>
>>>   .../media/v4l/dev-stateless-decoder.rst          |  6 ++++++
>>>   .../userspace-api/media/v4l/vidioc-enum-fmt.rst  | 11 +++++++++++
>>>   .../media/videodev2.h.rst.exceptions             |  2 ++
>>>   drivers/media/platform/verisilicon/hantro_v4l2.c | 16 +++++++++++++---
>>>   drivers/media/v4l2-core/v4l2-ioctl.c             |  3 +++
>>>   include/uapi/linux/videodev2.h                   |  2 ++
>>>   6 files changed, 37 insertions(+), 3 deletions(-)
>>>
>>
>
  

Le 19/07/2024 à 15:37, Hans Verkuil a écrit :
> On 19/07/2024 15:15, Benjamin Gaignard wrote:
>> Le 19/07/2024 à 14:57, Hans Verkuil a écrit :
>>> On 17/07/2024 15:14, Benjamin Gaignard wrote:
>>>> The goal of this series is to let userland applications enumerate
>>>> all the supported pixels formats of a stateless decoder without
>>>> setting all the possible codec-dependent control.
>>>> That offer a simplest solution for applications to discover
>>>> supported pixels formats and possibly let them doing smarter
>>>> choice between stateless decoders.
>>>>
>>>> An example of how it can be used in GStreamer to discover the
>>>> supported pixels formats for stateless decoder is available here:
>>>> https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/commits/v4l2codecs_enum_all_supported_formats?ref_type=heads
>>> So effectively specifying this flag makes ENUM_FMT also return
>>> formats that do not match the bit depth.
>>>
>>> So the AV1 (for example) compressed video uses e.g. 8 bit depth, but instead of just
>>> listing only 8 bit uncompressed pixelformats, you want to list them for any
>>> bit depth.
>>>
>>> But what is the point of that if the decoder can't decode 8 bit compressed to,
>>> say, 10 bit uncompressed video?
>> No decoder will do 8 bits to 10 bits (as far I knows).
>> The point is to be able to say that decoder could produce 10 bit frames without
>> setting a full sps/pps for each case (and for each supported codec).
>>
>>> I actually thought that this flag would just list all formats, independent
>>> of the output format (e.g. AV1, H264, etc.), but that does not appear to be
>>> the case? I.e., if capture pixelformat X is only available with AV1, will that still
>>> be listed if the output pixel is set to H264?
>>>
>>> I think you need to describe a real use-case here, and I am not convinced about
>>> the name of the flag either.
>> I may have miss something but yes the goal is to list all formats independently
>> of the output format.
>> When a SoC have multiple decoders for the same codec, knowing the supported formats
>> is key to select the better one.
>> Since I will have to do more iteration, feel free to provide a better name for the
>> flag(s). I'm always bad for naming this kind of thing.
> That really needs to be clarified, since in patch 1/2 it says:
>
> +   * If the ``V4L2_FMT_FLAG_ENUM_ALL_FORMATS`` flag is set the driver must enumerate
> +     all the supported formats without taking care of codec-dependent controls
> +     set on the ``OUTPUT`` queue. To indicate that the driver has take care of this
> +     flag it must set ``V4L2_FMT_FLAG_ALL_FORMATS`` flag for each format while
> +     enumerating.
>
> Here it just talks about 'codec-dependent controls set on the ``OUTPUT`` queue', it
> doesn't say anything about the compressed pixelformat set for the OUTPUT queue.
>
> And patch 2/2 sets the ignore_depth_match boolean, suggesting also that it is
> not listing all formats, but just ignoring a specific check.
>
> But if you list all pixelformats without taking the OUTPUT pixelformat into
> account, how do you know which pixelformat is valid for which codec?
>
> Say that only MPEG support NV12 (just for the sake of argument), and that's
> what you want to use, you have no way of knowing that NV12 is specific to MPEG,
> you would have to try each codec and see if NV12 is supported for that codec.
>
> I just don't see how this can be used in practice.
>
> What exactly is the problem you want to solve? A real-life problem, not a theoretical
> one :-)

On real-life: on a board with 2 different stateless decoders being able to detect the
one which can decode 10 bits bitstreams without testing all codec-dependent controls.

Regards,
Benjamin

>
> Regards,
>
> 	Hans
>
>>
>>>> changes in version 4:
>>>> - Explicitly document that the new flags are targeting mem2mem devices.
>>>>
>>>> changes in version 3:
>>>> - Add a flag to inform userspace application that driver
>>>>     as take care of the flag.
>>>>
>>>> changes in version 2:
>>>> - Clarify documentation.
>>>> - Only keep V4L2_FMT_FLAG_ALL_FORMATS flag in ioctl.
>>>>
>>>> Benjamin
>>>>
>>>> Benjamin Gaignard (2):
>>>>     media: videodev2: Add flags to unconditionnaly enumerate pixels
>>>>       formats
>>> I.e.: it is not unconditionally, it still depends on the chosen codec.
>>>
>>> Regards,
>>>
>>>      Hans
>>>
>>>>     media: verisilicon: Use V4L2_FMT_FLAG_ENUM_ALL_FORMATS flag
>>>>
>>>>    .../media/v4l/dev-stateless-decoder.rst          |  6 ++++++
>>>>    .../userspace-api/media/v4l/vidioc-enum-fmt.rst  | 11 +++++++++++
>>>>    .../media/videodev2.h.rst.exceptions             |  2 ++
>>>>    drivers/media/platform/verisilicon/hantro_v4l2.c | 16 +++++++++++++---
>>>>    drivers/media/v4l2-core/v4l2-ioctl.c             |  3 +++
>>>>    include/uapi/linux/videodev2.h                   |  2 ++
>>>>    6 files changed, 37 insertions(+), 3 deletions(-)
>>>>
>
  

Hi,

Le vendredi 19 juillet 2024 à 15:47 +0200, Benjamin Gaignard a écrit :
> > What exactly is the problem you want to solve? A real-life problem, not a theoretical
> > one :-)
> 
> On real-life: on a board with 2 different stateless decoders being able to detect the
> one which can decode 10 bits bitstreams without testing all codec-dependent controls.

That leans toward giving an answer for the selected bitstream format though,
since the same driver may do 10bit HEVC without 10bit AV1.

For the use case, both Chromium and GStreamer have a need to categorized
decoders so that we avoid trying to use decoder that can't do that task. More
platforms are getting multiple decoders, and we also need to take into account
the available software decoders.

Just looking at the codec specific profile is insufficient since we need two
conditions to be met.

1. The driver must support 10bit for the specific CODEC (for most codec this is
profile visible)
2. The produced 10bit color format must be supported by userspace

In today's implementation, in order to test this, we'd need to simulate a 10bit
header control, so that when enumerating the formats we get a list of 10bit
(optionally 8bit too, since some decoder can downscale colors) and finally
verify that these pixel formats are know by userspace. This is not impossible,
but very tedious, this proposal we to try and make this easier.

Nicolas
  

Hi,

On 2024-07-19 17:36, Nicolas Dufresne wrote:
> Hi,
> 
> Le vendredi 19 juillet 2024 à 15:47 +0200, Benjamin Gaignard a écrit :
>>> What exactly is the problem you want to solve? A real-life problem, not a theoretical
>>> one :-)
>>
>> On real-life: on a board with 2 different stateless decoders being able to detect the
>> one which can decode 10 bits bitstreams without testing all codec-dependent controls.
> 
> That leans toward giving an answer for the selected bitstream format though,
> since the same driver may do 10bit HEVC without 10bit AV1.
> 
> For the use case, both Chromium and GStreamer have a need to categorized
> decoders so that we avoid trying to use decoder that can't do that task. More
> platforms are getting multiple decoders, and we also need to take into account
> the available software decoders.
> 
> Just looking at the codec specific profile is insufficient since we need two
> conditions to be met.
> 
> 1. The driver must support 10bit for the specific CODEC (for most codec this is
> profile visible)
> 2. The produced 10bit color format must be supported by userspace
> 
> In today's implementation, in order to test this, we'd need to simulate a 10bit
> header control, so that when enumerating the formats we get a list of 10bit
> (optionally 8bit too, since some decoder can downscale colors) and finally
> verify that these pixel formats are know by userspace. This is not impossible,
> but very tedious, this proposal we to try and make this easier.

I have also been wondering what the use-case of this would be, and if it
is something to consider before a FFmpeg v4l2-request hwaccel submission.

I am guessing GStreamer may need to decide what decoder to use prior to
bitstream parsing/decoding has started?

For my re-worked FFmpeg v4l2-request hwaccel series, should hit
ffmpeg-devel list any day now, we try to probe each video device one
by one trying to identify if it will be capable to decode current stream
into a known/supported capture format [1], this typically happen when
header for first slice/frame has been parsed and is used to let driver
select its preferred/optimal capture format. The first device where all
test passes will be used and if none works FFmpeg video players will
typically fallback to use software decoding.

This type of probing may be a little bit limiting and depend too heavy
on the M2M Stateless Video Decoder Interface "It is suggested that the
driver chooses the preferred/optimal format for the current
configuration.".

Would you suggest I change how this probing is happening to make some
more clever detection of what media+video device should be used for a
specific stream with help of this new flag?

[1] https://github.com/Kwiboo/FFmpeg/blob/v4l2request-2024-v2/libavcodec/v4l2_request_probe.c#L373-L424

Regards,
Jonas

> 
> Nicolas
> 
>
  

Hi,

Le vendredi 19 juillet 2024 à 23:59 +0200, Jonas Karlman a écrit :
> Hi,
> 
> On 2024-07-19 17:36, Nicolas Dufresne wrote:
> > Hi,
> > 
> > Le vendredi 19 juillet 2024 à 15:47 +0200, Benjamin Gaignard a écrit :
> > > > What exactly is the problem you want to solve? A real-life problem, not a theoretical
> > > > one :-)
> > > 
> > > On real-life: on a board with 2 different stateless decoders being able to detect the
> > > one which can decode 10 bits bitstreams without testing all codec-dependent controls.
> > 
> > That leans toward giving an answer for the selected bitstream format though,
> > since the same driver may do 10bit HEVC without 10bit AV1.
> > 
> > For the use case, both Chromium and GStreamer have a need to categorized
> > decoders so that we avoid trying to use decoder that can't do that task. More
> > platforms are getting multiple decoders, and we also need to take into account
> > the available software decoders.
> > 
> > Just looking at the codec specific profile is insufficient since we need two
> > conditions to be met.
> > 
> > 1. The driver must support 10bit for the specific CODEC (for most codec this is
> > profile visible)
> > 2. The produced 10bit color format must be supported by userspace
> > 
> > In today's implementation, in order to test this, we'd need to simulate a 10bit
> > header control, so that when enumerating the formats we get a list of 10bit
> > (optionally 8bit too, since some decoder can downscale colors) and finally
> > verify that these pixel formats are know by userspace. This is not impossible,
> > but very tedious, this proposal we to try and make this easier.
> 
> I have also been wondering what the use-case of this would be, and if it
> is something to consider before a FFmpeg v4l2-request hwaccel submission.
> 
> I am guessing GStreamer may need to decide what decoder to use prior to
> bitstream parsing/decoding has started?

In GStreamer, the parsing is happening before the decoder. The parser generates
capabilities that are used during the decoder selection. Though, for performance
reason (GStreamer is plugin based, and loading all the plugins all the time
during playback is too slow), we generate static information about these
decoders and cache the results. This information usually only contains
profile/level and some min/max resolution when available. But being able to
discard profile if (as an example) the produced 10bit pixel format is not
supported by GStreamer is something we want in the long run. Its not essential
of course.

We do that limitation in the plugger that we can't fallback after the first
bitstream as been passed to the decoder, but this could in theory be resolved
and have nothing to do with this proposal. The proposal is just to help
userspace code stay simple and relatively CODEC agnostic.

> 
> For my re-worked FFmpeg v4l2-request hwaccel series, should hit
> ffmpeg-devel list any day now, we try to probe each video device one
> by one trying to identify if it will be capable to decode current stream
> into a known/supported capture format [1], this typically happen when
> header for first slice/frame has been parsed and is used to let driver
> select its preferred/optimal capture format. The first device where all
> test passes will be used and if none works FFmpeg video players will
> typically fallback to use software decoding.
> 
> This type of probing may be a little bit limiting and depend too heavy
> on the M2M Stateless Video Decoder Interface "It is suggested that the
> driver chooses the preferred/optimal format for the current
> configuration.".
> 
> Would you suggest I change how this probing is happening to make some
> more clever detection of what media+video device should be used for a
> specific stream with help of this new flag?

In general, there is only 1 or 2 decoder, so this process should be fast. I
guess if we start seeing large array or decoder (e.g. array of PCIe accelerators
or similar) that could become interesting to cache the information.

I think the difference is that you have the full bitstream header available at
the time you are to decide which decoder to use, I only have a abstract summary
in GStreamer. In short, you don't have to craft anything, which is nice.

Nicolas

> 
> [1] https://github.com/Kwiboo/FFmpeg/blob/v4l2request-2024-v2/libavcodec/v4l2_request_probe.c#L373-L424
> 
> Regards,
> Jonas
> 
> > 
> > Nicolas
> > 
> > 
> 
> _______________________________________________
> Kernel mailing list -- kernel@mailman.collabora.com
> To unsubscribe send an email to kernel-leave@mailman.collabora.com
> This list is managed by https://mailman.collabora.com