[RFC,v2,0/7] media: v4l2: Add extended fmt and buffer ioctls

  Hello,

This RFC follows the discussion started by Hans [1] a few months back.
It does not try to address all the problem reported in this thread but
instead focuses on the FMT and BUF(S) ioctls.

Note that my primary goal is to unify handling for multiplanar and
singleplanar formats and extend things to support the "single dmabuf
storing all pixel planes" issue.

This version received a bit more testing than the previous one (added
new tests to v4l2-compliance [2] to make sure EXT ioctls work as
expected and also checked that !ext -> ext wrappers work correctly by
running the old tests). Note that I'm not planning to post those
v4l-utils patches on the ML until we've settled down on the userspace
API, unless I'm explicitly asked to do so.

Right now I'm focusing on the case I was primarily interested in:
single dmabuf storing all pixel planes (each being at a different
offset), and it seems that patching the VB2 core to support that is
not a trivial task.

So here are a few questions for V4L/DMABUF experts:
- Can the same dmabuf be mapped several times. I think it's okay apart
  from the extra/needless time spent doing cache maintenance
  operations, but there might be issues if an IOMMU is involved
  (duplicate mappings?). If it's not okay, then we need to find a
  solution to only attach/map the DMABUF once when it's used for
  several planes (this is what I tried to do here [3], but I'm not
  entirely happy with the implementation and started to investigate
  another approach here [4]).
- How should we pass the offset to drivers that were previously using
  the get_cookie() (or the dma_sg wrapper) to retrieve an sg table.
  Adding the offset to the dma_addr or vaddr for vmalloc or dma-contig
  case can be done in the core, but for an sg-table it's a bit more
  complicated. Should drivers access this piece of information
  directly from vb2_plane->dbuf_offset? And in that case, how do we
  make sure drivers don't simply ignore the offset and assume it's
  always zero? 

Few words about the feedback I got from Brian and Nicolas on my v1:

- modifier field has been moved to v4l2_ext_format as suggested
- v4l2_timecode is still not present in v4l2_ext_buffer, but can be
  added back thanks to the extra reserved space
- the ENUMFMT is left as is for now, because I think we want Maxime's
  work on DRM/V4L format unification to land before reworking the
  ioctl (exposing extra info about the format and not only the 4CC
  code?). That also means that there's currently no way to know which
  modifiers are supported
- EXT_FMT/EXT_BUF capability flags to detect whether new ioctls are
  supported or not have not been added yet

Nothing has changed in v3, just rebased patches on top of media/master
so we can discuss it during the Media Summit.

Regards,

Boris

[1]https://www.mail-archive.com/linux-media@vger.kernel.org/msg135729.html
[2]https://github.com/bbrezillon/v4l-utils/commits/master
[3]https://github.com/bbrezillon/linux/commit/4882435f80b05a61827649d55cc0f0cee79680a7
[4]https://github.com/bbrezillon/linux/commit/a415216c6aaab2d51f0bd62270b994c8196ddd90

Boris Brezillon (5):
  media: v4l2: Extend pixel formats to unify single/multi-planar
    handling (and more)
  media: videobuf2: Expose helpers to implement the _ext_fmt and
    _ext_buf hooks
  media: mediabus: Add an helper to convert a ext_pix format to an
    mbus_fmt
  media: vivid: Convert the capture and output drivers to
    EXT_FMT/EXT_BUF
  media: vimc: Implement the ext_fmt and ext_buf hooks

Hans Verkuil (1):
  media: v4l2: Add extended buffer operations

 .../media/common/videobuf2/videobuf2-core.c   |    2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   |  534 ++++----
 drivers/media/platform/vimc/vimc-capture.c    |   65 +-
 drivers/media/platform/vimc/vimc-common.c     |    4 +-
 drivers/media/platform/vimc/vimc-common.h     |    2 +-
 drivers/media/platform/vivid/vivid-core.c     |   30 +-
 drivers/media/platform/vivid/vivid-vid-cap.c  |  171 +--
 drivers/media/platform/vivid/vivid-vid-cap.h  |   15 +-
 drivers/media/platform/vivid/vivid-vid-out.c  |  195 +--
 drivers/media/platform/vivid/vivid-vid-out.h  |   15 +-
 drivers/media/v4l2-core/v4l2-dev.c            |   54 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 1127 +++++++++++++++--
 include/media/v4l2-ioctl.h                    |   63 +
 include/media/v4l2-mediabus.h                 |   22 +
 include/media/videobuf2-core.h                |    6 +-
 include/media/videobuf2-v4l2.h                |   26 +-
 include/uapi/linux/videodev2.h                |  211 +++
 17 files changed, 1897 insertions(+), 645 deletions(-)
  

On Tue,  8 Oct 2019 11:11:13 +0200
Boris Brezillon <boris.brezillon@collabora.com> wrote:

Oops, forgot to update the subject. Should be:

"[RFC PATCH v3 0/6] media: v4l2: Add extended fmt and buffer ioctls"

> Hello,
> 
> This RFC follows the discussion started by Hans [1] a few months back.
> It does not try to address all the problem reported in this thread but
> instead focuses on the FMT and BUF(S) ioctls.
> 
> Note that my primary goal is to unify handling for multiplanar and
> singleplanar formats and extend things to support the "single dmabuf
> storing all pixel planes" issue.
> 
> This version received a bit more testing than the previous one (added
> new tests to v4l2-compliance [2] to make sure EXT ioctls work as
> expected and also checked that !ext -> ext wrappers work correctly by
> running the old tests). Note that I'm not planning to post those
> v4l-utils patches on the ML until we've settled down on the userspace
> API, unless I'm explicitly asked to do so.
> 
> Right now I'm focusing on the case I was primarily interested in:
> single dmabuf storing all pixel planes (each being at a different
> offset), and it seems that patching the VB2 core to support that is
> not a trivial task.
> 
> So here are a few questions for V4L/DMABUF experts:
> - Can the same dmabuf be mapped several times. I think it's okay apart
>   from the extra/needless time spent doing cache maintenance
>   operations, but there might be issues if an IOMMU is involved
>   (duplicate mappings?). If it's not okay, then we need to find a
>   solution to only attach/map the DMABUF once when it's used for
>   several planes (this is what I tried to do here [3], but I'm not
>   entirely happy with the implementation and started to investigate
>   another approach here [4]).
> - How should we pass the offset to drivers that were previously using
>   the get_cookie() (or the dma_sg wrapper) to retrieve an sg table.
>   Adding the offset to the dma_addr or vaddr for vmalloc or dma-contig
>   case can be done in the core, but for an sg-table it's a bit more
>   complicated. Should drivers access this piece of information
>   directly from vb2_plane->dbuf_offset? And in that case, how do we
>   make sure drivers don't simply ignore the offset and assume it's
>   always zero? 
> 
> Few words about the feedback I got from Brian and Nicolas on my v1:
> 
> - modifier field has been moved to v4l2_ext_format as suggested
> - v4l2_timecode is still not present in v4l2_ext_buffer, but can be
>   added back thanks to the extra reserved space
> - the ENUMFMT is left as is for now, because I think we want Maxime's
>   work on DRM/V4L format unification to land before reworking the
>   ioctl (exposing extra info about the format and not only the 4CC
>   code?). That also means that there's currently no way to know which
>   modifiers are supported
> - EXT_FMT/EXT_BUF capability flags to detect whether new ioctls are
>   supported or not have not been added yet
> 
> Nothing has changed in v3, just rebased patches on top of media/master
> so we can discuss it during the Media Summit.
> 
> Regards,
> 
> Boris
> 
> [1]https://www.mail-archive.com/linux-media@vger.kernel.org/msg135729.html
> [2]https://github.com/bbrezillon/v4l-utils/commits/master
> [3]https://github.com/bbrezillon/linux/commit/4882435f80b05a61827649d55cc0f0cee79680a7
> [4]https://github.com/bbrezillon/linux/commit/a415216c6aaab2d51f0bd62270b994c8196ddd90
> 
> Boris Brezillon (5):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add an helper to convert a ext_pix format to an
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
> 
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
> 
>  .../media/common/videobuf2/videobuf2-core.c   |    2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   |  534 ++++----
>  drivers/media/platform/vimc/vimc-capture.c    |   65 +-
>  drivers/media/platform/vimc/vimc-common.c     |    4 +-
>  drivers/media/platform/vimc/vimc-common.h     |    2 +-
>  drivers/media/platform/vivid/vivid-core.c     |   30 +-
>  drivers/media/platform/vivid/vivid-vid-cap.c  |  171 +--
>  drivers/media/platform/vivid/vivid-vid-cap.h  |   15 +-
>  drivers/media/platform/vivid/vivid-vid-out.c  |  195 +--
>  drivers/media/platform/vivid/vivid-vid-out.h  |   15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |   54 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 1127 +++++++++++++++--
>  include/media/v4l2-ioctl.h                    |   63 +
>  include/media/v4l2-mediabus.h                 |   22 +
>  include/media/videobuf2-core.h                |    6 +-
>  include/media/videobuf2-v4l2.h                |   26 +-
>  include/uapi/linux/videodev2.h                |  211 +++
>  17 files changed, 1897 insertions(+), 645 deletions(-)
>
  

Hi Boris,

On Tue, Oct 8, 2019 at 11:11 AM Boris Brezillon
<boris.brezillon@collabora.com> wrote:

> Hello,
> 
> This RFC follows the discussion started by Hans [1] a few months back.
> It does not try to address all the problem reported in this thread but
> instead focuses on the FMT and BUF(S) ioctls.
> 
> Note that my primary goal is to unify handling for multiplanar and
> singleplanar formats and extend things to support the "single dmabuf
> storing all pixel planes" issue.
> 
> This version received a bit more testing than the previous one (added
> new tests to v4l2-compliance [2] to make sure EXT ioctls work as
> expected and also checked that !ext -> ext wrappers work correctly by
> running the old tests). Note that I'm not planning to post those
> v4l-utils patches on the ML until we've settled down on the userspace
> API, unless I'm explicitly asked to do so.
> 

I'm new to V4L2, so I'm sorry if some of my questions don't make sense.

The venus codec can decode to UBWC format so I'm working with this
patch set to see how the modifiers flow will go.  I've been able to
integrate these patches for the most part.

Is the expectation with these new ioctls that they will work seamlessly with
all codecs?  Or that a codec can implement only the ones that are needed?
i.e. I only need to pass the modifier, so could I only implement S_EXT_FMT?
And then is the expectation that I know that for EXT I set the queue to
!MPLANE?

I find the interaction between _MPLANE and !MPLANE to still be a odd with
the new ioctls.  I had thought that I could use the same queue designation
for all the calls.  But for the codec I'm looking at (venus) it is MPLANE only,
so I call REQBUFS with V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE and all of the new
ioctls with V4L2_BUF_TYPE_VIDEO_CAPTURE.  I'm failing to see how to do this
in a backward compatible way.

> Right now I'm focusing on the case I was primarily interested in:
> single dmabuf storing all pixel planes (each being at a different
> offset), and it seems that patching the VB2 core to support that is
> not a trivial task.
> 
> So here are a few questions for V4L/DMABUF experts:
> - Can the same dmabuf be mapped several times. I think it's okay apart
>   from the extra/needless time spent doing cache maintenance
>   operations, but there might be issues if an IOMMU is involved
>   (duplicate mappings?). If it's not okay, then we need to find a
>   solution to only attach/map the DMABUF once when it's used for
>   several planes (this is what I tried to do here [3], but I'm not
>   entirely happy with the implementation and started to investigate
>   another approach here [4]).
> - How should we pass the offset to drivers that were previously using
>   the get_cookie() (or the dma_sg wrapper) to retrieve an sg table.
>   Adding the offset to the dma_addr or vaddr for vmalloc or dma-contig
>   case can be done in the core, but for an sg-table it's a bit more
>   complicated. Should drivers access this piece of information
>   directly from vb2_plane->dbuf_offset? And in that case, how do we
>   make sure drivers don't simply ignore the offset and assume it's
>   always zero? 
> 
> Few words about the feedback I got from Brian and Nicolas on my v1:
> 
> - modifier field has been moved to v4l2_ext_format as suggested
> - v4l2_timecode is still not present in v4l2_ext_buffer, but can be
>   added back thanks to the extra reserved space
> - the ENUMFMT is left as is for now, because I think we want Maxime's
>   work on DRM/V4L format unification to land before reworking the
>   ioctl (exposing extra info about the format and not only the 4CC
>   code?). That also means that there's currently no way to know which
>   modifiers are supported
> - EXT_FMT/EXT_BUF capability flags to detect whether new ioctls are
>   supported or not have not been added yet
> 
> Nothing has changed in v3, just rebased patches on top of media/master
> so we can discuss it during the Media Summit.
> 
> Regards,
> 
> Boris
> 
> [1]https://www.mail-archive.com/linux-media@vger.kernel.org/msg135729.html
> [2]https://github.com/bbrezillon/v4l-utils/commits/master
> [3]https://github.com/bbrezillon/linux/commit/4882435f80b05a61827649d55cc0f0cee79680a7
> [4]https://github.com/bbrezillon/linux/commit/a415216c6aaab2d51f0bd62270b994c8196ddd90
> 
> Boris Brezillon (5):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add an helper to convert a ext_pix format to an
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
> 
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
> 
>  .../media/common/videobuf2/videobuf2-core.c   |    2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   |  534 ++++----
>  drivers/media/platform/vimc/vimc-capture.c    |   65 +-
>  drivers/media/platform/vimc/vimc-common.c     |    4 +-
>  drivers/media/platform/vimc/vimc-common.h     |    2 +-
>  drivers/media/platform/vivid/vivid-core.c     |   30 +-
>  drivers/media/platform/vivid/vivid-vid-cap.c  |  171 +--
>  drivers/media/platform/vivid/vivid-vid-cap.h  |   15 +-
>  drivers/media/platform/vivid/vivid-vid-out.c  |  195 +--
>  drivers/media/platform/vivid/vivid-vid-out.h  |   15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |   54 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 1127 +++++++++++++++--
>  include/media/v4l2-ioctl.h                    |   63 +
>  include/media/v4l2-mediabus.h                 |   22 +
>  include/media/videobuf2-core.h                |    6 +-
>  include/media/videobuf2-v4l2.h                |   26 +-
>  include/uapi/linux/videodev2.h                |  211 +++
>  17 files changed, 1897 insertions(+), 645 deletions(-)
> 
> -- 
> 2.21.0