[v4,0/7] media: mediatek: vcodec: fix v4l2_ctrl_request_complete fail

  v4l2_m2m_buf_done is called in lat work queue, v4l2_ctrl_request_complete
is called in core queue. The request status of output queue will be set to
MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, leading to
output queue request complete fail. Must move v4l2_ctrl_request_complete
in front of v4l2_m2m_buf_done.

Patch 1 setting request complete before buffer done
Patch 2 change flush decode order when stream off
Patch 3 flush decoder before stream off
Patch 4 using input information to get vb2 buffer
Patch 5 store source vb2 buffer
Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove
Patch 7 remove media request checking

---
compared with v3:
- fix flush decoder issue when userspace stream off capture queue firstly
- fluster test result same with v3

compared with v2:
- add patch 5/6/7 to fix decode again issue
- add fluster test result with mt8195 platform(same with no changed):
  1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90
     VP8-TEST-VECTORS 59/61
  2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90
     VP9-TEST-VECTORS 276/305
  3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90
     AV1-TEST-VECTORS 237/239
  4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90
     JVT-AVC_V1       95/135
  5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90
     JCT-VC-HEVC_V1   142/147

compared with v1:
- add patch 2/3/4 to fix timing issue.
---
Yunfei Dong (7):
  media: mediatek: vcodec: setting request complete before buffer done
  media: mediatek: vcodec: change flush decode order when stream off
  media: mediatek: vcodec: flush decoder before stream off
  media: mediatek: vcodec: using input information to get vb2 buffer
  media: mediatek: vcodec: store source vb2 buffer
  media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with
    v4l2_m2m_src_buf_remove
  media: mediatek: vcodec: remove media request checking

 .../mediatek/vcodec/decoder/mtk_vcodec_dec.c  | 44 ++++++++---------
 .../vcodec/decoder/mtk_vcodec_dec_drv.h       |  4 +-
 .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++-----
 .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++----
 .../decoder/vdec/vdec_h264_req_multi_if.c     |  4 +-
 .../decoder/vdec/vdec_hevc_req_multi_if.c     |  4 +-
 .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++----
 .../mediatek/vcodec/decoder/vdec_msg_queue.h  |  4 +-
 8 files changed, 85 insertions(+), 60 deletions(-)
  

Hi Nicolas,

Thanks for your advice.

On Wed, 2024-08-07 at 09:08 -0400, Nicolas Dufresne wrote:
> > 

> Hi Yunfei,
> 
> Le mercredi 07 août 2024 à 16:24 +0800, Yunfei Dong a écrit :
> > v4l2_m2m_buf_done is called in lat work queue,
> > v4l2_ctrl_request_complete
> > is called in core queue. The request status of output queue will be
> > set to
> > MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called,
> > leading to
> > output queue request complete fail. Must move
> > v4l2_ctrl_request_complete
> > in front of v4l2_m2m_buf_done.
> 
> Sebastian and I have analyzed further the issue and the description
> here does
> not seem to match. What happens is that in Stateless decoding, you
> have to set
> header controls out-of-request to negotiate the format at first.
> 
> With VP9 notably, the header control is the only control there is.
> Chromium will
> optimize out this and only attach a bitstream buffer to the request.
> So when the
> buffer is mark to done, it is the last object in the request, which
> implicitly
> mark the request as complete.
> 
> When v4l2_ctrl_request_complete() is later called, the control code
> detect that
> there is no controls in the request. It then creates an empty
> control, but
> attaching an object to a completed request is not allowed.
> 

Whether I can write the commit message as below:

"User space will attach the syntaxes and bit-stream buffer to a media
request for stateless decoding, and the syntax controls are the only
v4l2 control request. The request will be marked to complete status
when the buffer is set to done, then request object will be cleaned
from media request.
 
When v4l2_ctrl_request_complete() is later called, the control request
detect that there is no controls in the request object. It then creates
an empty control request object, but attaching an object to a completed
request is not allowed."

> > Patch 1 setting request complete before buffer done
> > Patch 2 change flush decode order when stream off
> > Patch 3 flush decoder before stream off
> > Patch 4 using input information to get vb2 buffer
> > Patch 5 store source vb2 buffer
> > Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove
> > Patch 7 remove media request checking
> 
> I will give a some testing soon. Can you clarify on if the LAT and
> the CORE
Have you already helped to do the fluster test?

> still runs in parallel after this change ?
> 
Yes, the driver can work in parallel.

Could you please help to review other patches?

> Nicolas
> 
Best Regards,
Yunfei Dong
> > 
> > ---
> > compared with v3:
> > - fix flush decoder issue when userspace stream off capture queue
> > firstly
> > - fluster test result same with v3
> > 
> > compared with v2:
> > - add patch 5/6/7 to fix decode again issue
> > - add fluster test result with mt8195 platform(same with no
> > changed):
> >   1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90
> >      VP8-TEST-VECTORS 59/61
> >   2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90
> >      VP9-TEST-VECTORS 276/305
> >   3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90
> >      AV1-TEST-VECTORS 237/239
> >   4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90
> >      JVT-AVC_V1       95/135
> >   5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90
> >      JCT-VC-HEVC_V1   142/147
> > 
> > compared with v1:
> > - add patch 2/3/4 to fix timing issue.
> > ---
> > Yunfei Dong (7):
> >   media: mediatek: vcodec: setting request complete before buffer
> > done
> >   media: mediatek: vcodec: change flush decode order when stream
> > off
> >   media: mediatek: vcodec: flush decoder before stream off
> >   media: mediatek: vcodec: using input information to get vb2
> > buffer
> >   media: mediatek: vcodec: store source vb2 buffer
> >   media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with
> >     v4l2_m2m_src_buf_remove
> >   media: mediatek: vcodec: remove media request checking
> > 
> >  .../mediatek/vcodec/decoder/mtk_vcodec_dec.c  | 44 ++++++++-------
> > --
> >  .../vcodec/decoder/mtk_vcodec_dec_drv.h       |  4 +-
> >  .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++-
> > ----
> >  .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++----
> >  .../decoder/vdec/vdec_h264_req_multi_if.c     |  4 +-
> >  .../decoder/vdec/vdec_hevc_req_multi_if.c     |  4 +-
> >  .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++----
> >  .../mediatek/vcodec/decoder/vdec_msg_queue.h  |  4 +-
> >  8 files changed, 85 insertions(+), 60 deletions(-)
> > 
> 
>
  

Hey Yunfei,

given this new series by Hans:
https://patchwork.linuxtv.org/project/linux-media/list/?series=13427

we might actually be able to find a more performant solution of the
problem, I'll work on that a bit and give you feedback.

Regards,
Sebastian

On 07.08.2024 16:24, Yunfei Dong wrote:
>v4l2_m2m_buf_done is called in lat work queue, v4l2_ctrl_request_complete
>is called in core queue. The request status of output queue will be set to
>MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, leading to
>output queue request complete fail. Must move v4l2_ctrl_request_complete
>in front of v4l2_m2m_buf_done.
>
>Patch 1 setting request complete before buffer done
>Patch 2 change flush decode order when stream off
>Patch 3 flush decoder before stream off
>Patch 4 using input information to get vb2 buffer
>Patch 5 store source vb2 buffer
>Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove
>Patch 7 remove media request checking
>
>---
>compared with v3:
>- fix flush decoder issue when userspace stream off capture queue firstly
>- fluster test result same with v3
>
>compared with v2:
>- add patch 5/6/7 to fix decode again issue
>- add fluster test result with mt8195 platform(same with no changed):
>  1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90
>     VP8-TEST-VECTORS 59/61
>  2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90
>     VP9-TEST-VECTORS 276/305
>  3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90
>     AV1-TEST-VECTORS 237/239
>  4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90
>     JVT-AVC_V1       95/135
>  5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90
>     JCT-VC-HEVC_V1   142/147
>
>compared with v1:
>- add patch 2/3/4 to fix timing issue.
>---
>Yunfei Dong (7):
>  media: mediatek: vcodec: setting request complete before buffer done
>  media: mediatek: vcodec: change flush decode order when stream off
>  media: mediatek: vcodec: flush decoder before stream off
>  media: mediatek: vcodec: using input information to get vb2 buffer
>  media: mediatek: vcodec: store source vb2 buffer
>  media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with
>    v4l2_m2m_src_buf_remove
>  media: mediatek: vcodec: remove media request checking
>
> .../mediatek/vcodec/decoder/mtk_vcodec_dec.c  | 44 ++++++++---------
> .../vcodec/decoder/mtk_vcodec_dec_drv.h       |  4 +-
> .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++-----
> .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++----
> .../decoder/vdec/vdec_h264_req_multi_if.c     |  4 +-
> .../decoder/vdec/vdec_hevc_req_multi_if.c     |  4 +-
> .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++----
> .../mediatek/vcodec/decoder/vdec_msg_queue.h  |  4 +-
> 8 files changed, 85 insertions(+), 60 deletions(-)
>
>-- 
>2.46.0
>
>
  

Hi Sebastian,

Thanks for your suggestion.
On Fri, 2024-08-23 at 17:45 +0200, Sebastian Fricke wrote:
> Hey Yunfei,
> 
> given this new series by Hans:
> 
https://urldefense.com/v3/__https://patchwork.linuxtv.org/project/linux-media/list/?series=13427__;!!CTRNKA9wMg0ARbw!h4__bSC_RX1erZCq5wvZbjRQrmS9ecXytwjcdLc9sgjBgj3buUC8JrUG78rhtEsXAN6vmZEhM3tWumFQ50R5NcR_1hbqLyYlSg$
>  
> 
> we might actually be able to find a more performant solution of the
> problem, I'll work on that a bit and give you feedback.
> 
I already do so many test for different project with these patches,
whether it's possible to merge these patch firstly, then to call hans's
changes to do stress test.

Best Regards,
Yunfei Dong
> Regards,
> Sebastian
> 
> On 07.08.2024 16:24, Yunfei Dong wrote:
> > v4l2_m2m_buf_done is called in lat work queue,
> > v4l2_ctrl_request_complete
> > is called in core queue. The request status of output queue will be
> > set to
> > MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called,
> > leading to
> > output queue request complete fail. Must move
> > v4l2_ctrl_request_complete
> > in front of v4l2_m2m_buf_done.
> > 
> > Patch 1 setting request complete before buffer done
> > Patch 2 change flush decode order when stream off
> > Patch 3 flush decoder before stream off
> > Patch 4 using input information to get vb2 buffer
> > Patch 5 store source vb2 buffer
> > Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove
> > Patch 7 remove media request checking
> > 
> > ---
> > compared with v3:
> > - fix flush decoder issue when userspace stream off capture queue
> > firstly
> > - fluster test result same with v3
> > 
> > compared with v2:
> > - add patch 5/6/7 to fix decode again issue
> > - add fluster test result with mt8195 platform(same with no
> > changed):
> >  1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90
> >     VP8-TEST-VECTORS 59/61
> >  2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90
> >     VP9-TEST-VECTORS 276/305
> >  3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90
> >     AV1-TEST-VECTORS 237/239
> >  4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90
> >     JVT-AVC_V1       95/135
> >  5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90
> >     JCT-VC-HEVC_V1   142/147
> > 
> > compared with v1:
> > - add patch 2/3/4 to fix timing issue.
> > ---
> > Yunfei Dong (7):
> >  media: mediatek: vcodec: setting request complete before buffer
> > done
> >  media: mediatek: vcodec: change flush decode order when stream off
> >  media: mediatek: vcodec: flush decoder before stream off
> >  media: mediatek: vcodec: using input information to get vb2 buffer
> >  media: mediatek: vcodec: store source vb2 buffer
> >  media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with
> >    v4l2_m2m_src_buf_remove
> >  media: mediatek: vcodec: remove media request checking
> > 
> > .../mediatek/vcodec/decoder/mtk_vcodec_dec.c  | 44 ++++++++------
> > ---
> > .../vcodec/decoder/mtk_vcodec_dec_drv.h       |  4 +-
> > .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++
> > -----
> > .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++----
> > .../decoder/vdec/vdec_h264_req_multi_if.c     |  4 +-
> > .../decoder/vdec/vdec_hevc_req_multi_if.c     |  4 +-
> > .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++----
> > .../mediatek/vcodec/decoder/vdec_msg_queue.h  |  4 +-
> > 8 files changed, 85 insertions(+), 60 deletions(-)
> > 
> > -- 
> > 2.46.0
> > 
> >