[v2,0/1] virtio-media: Add device specification

  Hi,

This a formal attempt of including virtio-media
device specification.

Virtio-media came from a discussion on virtio-dev
mailing list, which lead to presenting virtio-v4l2[1]
specification as an alternative to virtio-video.

Later, virtio-v4l2 was renamed to virtio-media[2]
and published through:

https://github.com/chromeos/virtio-media

The repository above includes a virtio-media driver able
to pass v4l2-compliance when proxying the vivid/vicodec
virtual devices or an actual UVC camera using the
V4L2 vhost device (available in the repository).
Steps to reproduce are also detailed[3].

There is some overlap with virtio-video in regards
to which devices it can handle. However,
as virtio-media will likely be the virtualization
solution for ChromeOS (already landed into the chromeos
organization) and possibly other Google projects for
media devices, it would be desirable to include the
specification in the next virtio release despite
the aforementioned overlap.

The device ID in this document differs from
the ID in the virtio-media project repository.
And it will probably need some discussion on which
would be the correct definitive ID.

Full PDF: https://drive.google.com/file/d/1pNCFP06VoV8Zuyx0aDVQ7HAT4xp-pZ0a/view?usp=sharing
PDF with the media section only: https://drive.google.com/file/d/1sn3NUUeCm46zVyJKHkpytTIgGw1fUt5T/view?usp=sharing

[1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0
[2] https://www.mail-archive.com/virtio-dev@lists.oasis-open.org/msg12665.html
[3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md

Albert Esteve (1):
  virtio-media: Add virtio media device specification

 conformance.tex                           |  13 +-
 content.tex                               |   1 +
 device-types/media/description.tex        | 578 ++++++++++++++++++++++
 device-types/media/device-conformance.tex |  11 +
 device-types/media/driver-conformance.tex |   9 +
 5 files changed, 608 insertions(+), 4 deletions(-)
 create mode 100644 device-types/media/description.tex
 create mode 100644 device-types/media/device-conformance.tex
 create mode 100644 device-types/media/driver-conformance.tex
  

On Fri, Jun 07, 2024 at 10:00:44AM +0200, Albert Esteve wrote:
> Hi,
> 
> This a formal attempt of including virtio-media
> device specification.

This was sent to a wrong mailing list.
Please repost to the correct list.

Albert, when you subscribed to virtio-dev you should
have received a message that read, in part:

When to use this list:
- questions and change proposals for Virtio drivers and devices
  implementing the specification.

When not to use this list:
- questions and change proposals for the Virtio specification,
  including the specification of basic functionality, transports and
  devices (please use the virtio-comment mailing list
  [mailto:virtio-comment@lists.linux.dev] for this).

To do:
- send email preferably in text format (best for archiving)

Not to do:
- copy both virtio-dev and virtio-comment mailing lists (instead, pick
  one)
- send full copies of the virtio spec (in any format)



So what made you send a specification change proposal to
virtio-dev, and also copy the defunct, non-functional
list at virtio-dev@lists.oasis-open.org ?


Was the message unclear? If yes I'd like to improve it.
  

On Fri, Jun 07, 2024 at 11:48:36AM +0200, Albert Esteve wrote:
> 
> 
> On Fri, Jun 7, 2024 at 11:41 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> 
>     On Fri, Jun 07, 2024 at 10:00:44AM +0200, Albert Esteve wrote:
>     > Hi,
>     >
>     > This a formal attempt of including virtio-media
>     > device specification.
> 
>     This was sent to a wrong mailing list.
>     Please repost to the correct list.
> 
>     Albert, when you subscribed to virtio-dev you should
>     have received a message that read, in part:
> 
>     When to use this list:
>     - questions and change proposals for Virtio drivers and devices
>       implementing the specification.
> 
>     When not to use this list:
>     - questions and change proposals for the Virtio specification,
>       including the specification of basic functionality, transports and
>       devices (please use the virtio-comment mailing list
>       [mailto:virtio-comment@lists.linux.dev] for this).
> 
>     To do:
>     - send email preferably in text format (best for archiving)
> 
>     Not to do:
>     - copy both virtio-dev and virtio-comment mailing lists (instead, pick
>       one)
>     - send full copies of the virtio spec (in any format)
> 
> 
> 
>     So what made you send a specification change proposal to
>     virtio-dev, and also copy the defunct, non-functional
>     list at virtio-dev@lists.oasis-open.org ?
> 
> 
>     Was the message unclear? If yes I'd like to improve it.
> 
> 
> My bad.
> The instructions were clear, and I set up the mail to be sent to the
> correct virtio-dev mailing list address.


Hmm no, it's actually virtio-comment@lists.linux.dev for change
proposals for the Virtio specification which is what you have here.


> But I had the configuration
> of the project set to automatically add both of the old addresses to the
> patch. I only noticed after sending, so the damage was already
> made. 
> 
> I will fix it for the next drop.
>  
> 
> 
> 
>     --
>     MST
> 
>
  

Hi Albert,

On 07.06.24 10:00, Albert Esteve wrote:
> Hi,
> 
> This a formal attempt of including virtio-media
> device specification.
> 
> Virtio-media came from a discussion on virtio-dev
> mailing list, which lead to presenting virtio-v4l2[1]
> specification as an alternative to virtio-video.
> 
> Later, virtio-v4l2 was renamed to virtio-media[2]
> and published through:
> 
> https://github.com/chromeos/virtio-media
> 
> The repository above includes a virtio-media driver able
> to pass v4l2-compliance when proxying the vivid/vicodec
> virtual devices or an actual UVC camera using the
> V4L2 vhost device (available in the repository).
> Steps to reproduce are also detailed[3].
> 
> There is some overlap with virtio-video in regards
> to which devices it can handle. However,
> as virtio-media will likely be the virtualization
> solution for ChromeOS (already landed into the chromeos
> organization) and possibly other Google projects for
> media devices, it would be desirable to include the
> specification in the next virtio release despite
> the aforementioned overlap.
> 
> The device ID in this document differs from
> the ID in the virtio-media project repository.
> And it will probably need some discussion on which
> would be the correct definitive ID.
> 
> Full PDF: https://drive.google.com/file/d/1pNCFP06VoV8Zuyx0aDVQ7HAT4xp-pZ0a/view?usp=sharing
> PDF with the media section only: https://drive.google.com/file/d/1sn3NUUeCm46zVyJKHkpytTIgGw1fUt5T/view?usp=sharing
> 
> [1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0
> [2] https://www.mail-archive.com/virtio-dev@lists.oasis-open.org/msg12665.html
> [3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md
> 
> Albert Esteve (1):
>    virtio-media: Add virtio media device specification
> 
>   conformance.tex                           |  13 +-
>   content.tex                               |   1 +
>   device-types/media/description.tex        | 578 ++++++++++++++++++++++
>   device-types/media/device-conformance.tex |  11 +
>   device-types/media/driver-conformance.tex |   9 +
>   5 files changed, 608 insertions(+), 4 deletions(-)
>   create mode 100644 device-types/media/description.tex
>   create mode 100644 device-types/media/device-conformance.tex
>   create mode 100644 device-types/media/driver-conformance.tex

I'd like to add some general considerations:

1. virtio-media device capability discovery is very chatty

V4L2 requires potentially hundreds of system calls to get the full view 
of the device capabilities. This is inherited by virtio-media.
AFAIU V4L2 developers agree there is room for enhancement here, see [1], 
[2]. Also I've been told VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS were 
added for similar reasons.

 From the point of view of virtualization developers like my colleagues 
and me hundreds of hypervisor "exits" are excessive and costly. It can 
noticeably increase boot times, which is something we fight against in 
automotive. AFAIU other virtio developers agree with this, see [3].

In contrast virtio-video has been doing this in one command from day 
one. Yes, the data was incomplete. That's why in draft v9 I added TLV as 
a way to store all possible capabilities. I hope the linux-media folks 
could take a look on it. Maybe this is something, that can be adopted in 
V4L2?

2. virtio-media has a hard dependency on V4L2

There are certainly some "patches" on top of V4L2 in virtio-media, like 
the representation of the extended controls (which actually looks 
similar to the representation of the controls in virtio-video) or 
VIRTIO_MEDIA_EVT_DQBUF. But how far can this go? Is it OK to add your 
own ioctl ids in virtio-media?

AFAIK the linux-media maintainers have been overloaded for years, see 
[4]. Would they be happy to deal with the additional requirements? Would 
virtio community like to have a dependency here?

2.1. an example: format modifiers

There is a patchset aiming at unifying V4L2 pixel formats and extending 
them with DRM format modifiers. It is now at version 7 submitted in 
2023, see [5]. The first version was submitted in 2019, see [6]. Not 
merged yet AFAIK.

In virtio-video I just added them in draft v9.

I'm absolutely not trying to criticize here. I just try to highlight 
that there is a lot of legacy and the process is painful. Right now we 
have an opportunity to design a new API according to the current state 
of the art of the stateful codecs.

3. uncertainty with cameras

AFAIK there is still no agreement about how cameras should be 
virtualized, see [3], [7], [8], [9]. virtio-media provides support for 
cameras in a specific way, which might not be desirable.

4. (minor) is it possible/hard to implement the device in hardware/on a 
micro controller?

This is something I thought about recently, there might be a use-case 
for it in the future. One of the concerns is that dynamic memory 
allocations are IMHO inevitable in virtio-media or virtio-video up to 
draft v8. I think multiple virtqueues in virtio-video draft v9 would 
help here. Not sure yet...

There are also other minor concerns, that are probably tolerable.

[1] 
https://lore.kernel.org/linux-media/20230922100303.GF19112@pendragon.ideasonboard.com/
[2] Page 6 in 
https://hverkuil.home.xs4all.nl/mediasummit2023-pdfs/Hsia-Jun%20Li%20-%20V4L2%20M2M%20EXT%20API%20enhancement.pdf
[3] https://old.linaro.org/blog/the-challenges-of-abstracting-virtio/
[4] 
https://lore.kernel.org/linux-media/4b6b1355-9baa-ff1e-e1c0-89dfdc83ad04@xs4all.nl/
[5] 
https://lore.kernel.org/linux-media/20230206043308.28365-2-ayaka@soulik.info/
[6] 
https://lore.kernel.org/linux-media/20190319145243.25047-1-boris.brezillon@collabora.com/
[7] https://lore.kernel.org/virtio-dev/87354dtp30.fsf@linaro.org/
[8] 
https://lore.kernel.org/linux-media/00f53c06-e66d-aa46-ca4f-c3baab6cf455@xs4all.nl/
[9] 
https://lore.kernel.org/virtio-dev/CAAFQd5BrhDZtFX3vdhBVSLXthe8CykYsZzVQ9ddZxVNvNS3ArA@mail.gmail.com/
  

On Fri, Jun 7, 2024 at 10:29 PM Alexander Gordeev
<alexander.gordeev@opensynergy.com> wrote:
>
> Hi Albert,
>
> On 07.06.24 10:00, Albert Esteve wrote:
> > Hi,
> >
> > This a formal attempt of including virtio-media
> > device specification.
> >
> > Virtio-media came from a discussion on virtio-dev
> > mailing list, which lead to presenting virtio-v4l2[1]
> > specification as an alternative to virtio-video.
> >
> > Later, virtio-v4l2 was renamed to virtio-media[2]
> > and published through:
> >
> > https://github.com/chromeos/virtio-media
> >
> > The repository above includes a virtio-media driver able
> > to pass v4l2-compliance when proxying the vivid/vicodec
> > virtual devices or an actual UVC camera using the
> > V4L2 vhost device (available in the repository).
> > Steps to reproduce are also detailed[3].
> >
> > There is some overlap with virtio-video in regards
> > to which devices it can handle. However,
> > as virtio-media will likely be the virtualization
> > solution for ChromeOS (already landed into the chromeos
> > organization) and possibly other Google projects for
> > media devices, it would be desirable to include the
> > specification in the next virtio release despite
> > the aforementioned overlap.
> >
> > The device ID in this document differs from
> > the ID in the virtio-media project repository.
> > And it will probably need some discussion on which
> > would be the correct definitive ID.
> >
> > Full PDF: https://drive.google.com/file/d/1pNCFP06VoV8Zuyx0aDVQ7HAT4xp-pZ0a/view?usp=sharing
> > PDF with the media section only: https://drive.google.com/file/d/1sn3NUUeCm46zVyJKHkpytTIgGw1fUt5T/view?usp=sharing
> >
> > [1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0
> > [2] https://www.mail-archive.com/virtio-dev@lists.oasis-open.org/msg12665.html
> > [3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md
> >
> > Albert Esteve (1):
> >    virtio-media: Add virtio media device specification
> >
> >   conformance.tex                           |  13 +-
> >   content.tex                               |   1 +
> >   device-types/media/description.tex        | 578 ++++++++++++++++++++++
> >   device-types/media/device-conformance.tex |  11 +
> >   device-types/media/driver-conformance.tex |   9 +
> >   5 files changed, 608 insertions(+), 4 deletions(-)
> >   create mode 100644 device-types/media/description.tex
> >   create mode 100644 device-types/media/device-conformance.tex
> >   create mode 100644 device-types/media/driver-conformance.tex
>
> I'd like to add some general considerations:
>
> 1. virtio-media device capability discovery is very chatty
>
> V4L2 requires potentially hundreds of system calls to get the full view
> of the device capabilities. This is inherited by virtio-media.
> AFAIU V4L2 developers agree there is room for enhancement here, see [1],
> [2]. Also I've been told VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS were
> added for similar reasons.
>
>  From the point of view of virtualization developers like my colleagues
> and me hundreds of hypervisor "exits" are excessive and costly. It can
> noticeably increase boot times, which is something we fight against in
> automotive. AFAIU other virtio developers agree with this, see [3].

There is a simple way to mitigate this: allow an arbitrary number of
commands to be submitted by the driver with each command buffer. The
host processes the commands sequentially and stops at the first error,
if any.

That way, when enumerating formats, the guest can place a bunch of
VIDIOC_ENUM_FMT ioctls covering an arbitrary range 0..n in its command
buffer, and get all these formats in one vmexit instead of n. In the
case of a video decoder supporting 5 input formats, all the formats
are enumerated in 5 exits.

Having this option could also enable other optimizations, like the
guest being able to queue all its CAPTURE buffers in a single vmexit,
although I doubt this would result in a noticeable performance boost.

But at the very least this should address concerns about noisy
enumeration. I haven't noticed any slowness due to format enumeration
with the current scheme, but the option is trivial to enable.

>
> In contrast virtio-video has been doing this in one command from day
> one. Yes, the data was incomplete. That's why in draft v9 I added TLV as
> a way to store all possible capabilities. I hope the linux-media folks
> could take a look on it. Maybe this is something, that can be adopted in
> V4L2?
>
> 2. virtio-media has a hard dependency on V4L2
>
> There are certainly some "patches" on top of V4L2 in virtio-media, like
> the representation of the extended controls (which actually looks
> similar to the representation of the controls in virtio-video) or
> VIRTIO_MEDIA_EVT_DQBUF. But how far can this go? Is it OK to add your
> own ioctl ids in virtio-media?

virtio-media doesn't add any ioctl ID and the extended controls use
the same structure as their V4L2 counterparts.

>
> AFAIK the linux-media maintainers have been overloaded for years, see
> [4]. Would they be happy to deal with the additional requirements? Would
> virtio community like to have a dependency here?

The burden of having a couple new events in virtio-media seems lighter
to me on the V4L2 maintainers than the one of having a whole new video
protocol to maintain.

>
> 2.1. an example: format modifiers
>
> There is a patchset aiming at unifying V4L2 pixel formats and extending
> them with DRM format modifiers. It is now at version 7 submitted in
> 2023, see [5]. The first version was submitted in 2019, see [6]. Not
> merged yet AFAIK.
>
> In virtio-video I just added them in draft v9.
>
> I'm absolutely not trying to criticize here. I just try to highlight
> that there is a lot of legacy and the process is painful. Right now we
> have an opportunity to design a new API according to the current state
> of the art of the stateful codecs.
>
> 3. uncertainty with cameras
>
> AFAIK there is still no agreement about how cameras should be
> virtualized, see [3], [7], [8], [9]. virtio-media provides support for
> cameras in a specific way, which might not be desirable.

There is definitely an agreement on how cameras should work on V4L2 ;
and thanks to that you can operate a camera using virtio-media,
*today*.

I'm sure the virtio-camera specification, driver, and host devices
will be very impressive once they are released ; it's just that I
would enjoy a solution, however imperfect, within my life span.

>
> 4. (minor) is it possible/hard to implement the device in hardware/on a
> micro controller?
>
> This is something I thought about recently, there might be a use-case
> for it in the future. One of the concerns is that dynamic memory
> allocations are IMHO inevitable in virtio-media or virtio-video up to
> draft v8. I think multiple virtqueues in virtio-video draft v9 would
> help here. Not sure yet...

V4L2's MMAP buffer type has that covered.

>
> There are also other minor concerns, that are probably tolerable.
>
> [1]
> https://lore.kernel.org/linux-media/20230922100303.GF19112@pendragon.ideasonboard.com/
> [2] Page 6 in
> https://hverkuil.home.xs4all.nl/mediasummit2023-pdfs/Hsia-Jun%20Li%20-%20V4L2%20M2M%20EXT%20API%20enhancement.pdf
> [3] https://old.linaro.org/blog/the-challenges-of-abstracting-virtio/
> [4]
> https://lore.kernel.org/linux-media/4b6b1355-9baa-ff1e-e1c0-89dfdc83ad04@xs4all.nl/
> [5]
> https://lore.kernel.org/linux-media/20230206043308.28365-2-ayaka@soulik.info/
> [6]
> https://lore.kernel.org/linux-media/20190319145243.25047-1-boris.brezillon@collabora.com/
> [7] https://lore.kernel.org/virtio-dev/87354dtp30.fsf@linaro.org/
> [8]
> https://lore.kernel.org/linux-media/00f53c06-e66d-aa46-ca4f-c3baab6cf455@xs4all.nl/
> [9]
> https://lore.kernel.org/virtio-dev/CAAFQd5BrhDZtFX3vdhBVSLXthe8CykYsZzVQ9ddZxVNvNS3ArA@mail.gmail.com/
>
> --
> Alexander Gordeev
> Senior Software Engineer
>
> OpenSynergy GmbH
> Rotherstr. 20, 10245 Berlin
> www.opensynergy.com