From patchwork Wed Apr 6 13:06:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Dufresne X-Patchwork-Id: 82258 Received: from vger.kernel.org ([23.128.96.18]) by www.linuxtv.org with esmtp (Exim 4.92) (envelope-from ) id 1nc7qE-0067QF-O6; Wed, 06 Apr 2022 15:45:07 +0000 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236593AbiDFPrB (ORCPT + 1 other); Wed, 6 Apr 2022 11:47:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236568AbiDFPqw (ORCPT ); Wed, 6 Apr 2022 11:46:52 -0400 Received: from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEBEF451D48 for ; Wed, 6 Apr 2022 06:06:47 -0700 (PDT) Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: nicolas) with ESMTPSA id 426831F441A4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1649250406; bh=YXXK+8OzkUl9s/fzJkIPtxyWoA+96/PK9V7TfydBIcA=; h=Subject:From:To:Date:References:From; b=W56wAQekXx4G/QQSXO/3tqivpT/6RgWammq/Y4KSvR0fVCWESkoKEFoIOAXfaOCFY uzTviQQiAwvQ5rJqbxffynLuQQ0MP6bANN/WzxtZaPEWGNUQGjMRBBD180/y3+utz2 fNucd5qH0eFqfYG8gJ3AyZ9zu1uf+ifDXQ0Z+4sC20sbDKFaAi7WIlUjSehFoG1llm tHcH2HTTDkLDorTXS1F8jA5me0LLHlVx26AEERudy32M6OZyU3nxTODufff5otqSUx g1L/mnL6KBQuUTW3SHnHis5cSzHZgTrJBkQic0Mp3+R6fxTQUraErqfp6jbFpNZUVz cXMuEZxH5doaA== Message-ID: <958e038a6493f6b8035dd2129d25ff61d4c82242.camel@collabora.com> Subject: Fwd: [PATCH v3 00/24] H.264 Field Decoding Support for Frame-based Decoders From: Nicolas Dufresne To: linux-media Date: Wed, 06 Apr 2022 09:06:36 -0400 References: <20220405204426.259074-1-nicolas.dufresne@collabora.com> User-Agent: Evolution 3.44.0 (3.44.0-1.fc36) MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-LSpam-Score: -2.5 (--) X-LSpam-Report: No, score=-2.5 required=5.0 tests=BAYES_00=-1.9,DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,HEADER_FROM_DIFFERENT_DOMAINS=0.5,MAILING_LIST_MULTI=-1,UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no Until now, only Cedrus (a slice base decoder) supported interlaced decoding. In order to support field decoding in our frame-based decoder, the v4l2-h264 library needed adaptation to produce the appropriate reference lists. This patch extends the v4l2-h264 library to produce the larger references list needed to represent fields separately. Hantro, MTK-VCODEC and RKVDEC drivers have been adapted to accommodate the larger lists. Though, only Hantro and RKVDEC actually have HW support for field decoding. So only these two have been updated to make use of the larger lists. All this work has been done using the H.264 specification, LibreELEC downstream kernel patches, Rockchip MPP reference software and Hantro reference software. For reviewers, the following is the map of all commit. Patches that could be merge independently of this serie are marked as independent. Note that the test results do depend on the generic fixes. 01 : Documentation fix (independent) 02-03 : Improving some generic traces (independent) 04 : Minor v4l2-h264 fix (independent) 05-11 : v4l2-h264 field decoding support 12-16 : rkvdec h.264 generic fixes (independent) 17-20 : rkvdec h.264 field decoding support 21-24 : hantro h.264 field decoding support All this work have been tested using GStreamer mainline implementation but also with FFMPEG LibreELEC fork using the testing tool fluster running through the ITU-T H.264 (2016-02) AVCv2 set of bitstream. Before this patch, the scores were: Hantro: FFMPEG: 88/135 GSteamer: 90/135 RKVDEC: FFMPEG: 73/135 GSteamer: 77/135 And after these changes: Hantro: FFMPEG: 118/135 GSteamer: 129/135 RKVDEC: FFMPEG: 118/135 GSteamer: 129/135 Note that a bug in FFMPEG / LibreELEC fork was noticed and fixed with the following change: Some useful links: Detailed Hantro Results: https://gitlab.freedesktop.org/-/snippets/5189 Detailed RKVDEC Results: https://gitlab.freedesktop.org/-/snippets/5253 ITU-T H.264 (2016-02) AVCv2: https://www.itu.int/net/itu-t/sigdb/spevideo/VideoForm-s.aspx?val=102002641 Fluster: https://github.com/fluendo/fluster GStreamer: https://gitlab.freedesktop.org/gstreamer/gstreamer/ FFMPEG Fork: https://github.com/jernejsk/FFmpeg/tree/v4l2-request-hwaccel-4.4 Rockchip MPP: https://github.com/rockchip-linux/mpp Changes in v3: - Improved debug message on timestamp miss-match - Moved H264 SPS validation into rkvdec-h264 - Added more comments around H264 SPS validation - Also validate at streamon (rkvdec start()) - Applied more Review-by and Fixes tag - Fixed Signed-off-by chain in Jonas patch Changes in v2: - Applied most of Sebastian's suggestion in comments and commit messages. - Use a bool for dpb_valid and dpb_bottom in rkvdec - Dropped one wrong typo fix (media: v4l2-mem2mem: Fix typo in trace message) - Dropped Alex fix (media: rkvdec-h264: Don't hardcode SPS/PPS parameters + I will carry this one later, it seems cosmetic Jonas Karlman (5): media: rkvdec: h264: Fix bit depth wrap in pps packet media: rkvdec: h264: Validate and use pic width and height in mbs media: rkvdec: h264: Fix reference frame_num wrap for second field media: rkvdec: Ensure decoded resolution fit coded resolution media: hantro: h264: Make dpb entry management more robust Nicolas Dufresne (18): media: doc: Document dual use of H.264 pic_num/frame_num media: v4l2-mem2mem: Trace on implicit un-hold media: h264: Avoid wrapping long_term_frame_idx media: h264: Use v4l2_h264_reference for reflist media: h264: Increase reference lists size to 32 media: h264: Store current picture fields media: h264: Store all fields into the unordered list media: v4l2: Trace calculated p/b0/b1 initial reflist media: h264: Sort p/b reflist using frame_num media: v4l2: Reorder field reflist media: rkvdec: Stop overclocking the decoder media: rkvdec: h264: Fix dpb_valid implementation media: rkvdec: Move H264 SPS validation in rkvdec-h264 media: rkvdec-h264: Add field decoding support media: rkvdec: Enable capture buffer holding for H264 media: hantro: Stop using H.264 parameter pic_num media: hantro: Add H.264 field decoding support media: hantro: Enable HOLD_CAPTURE_BUF for H.264 Sebastian Fricke (1): media: videobuf2-v4l2: Warn on holding buffers without support .../media/v4l/ext-ctrls-codec-stateless.rst | 10 +- .../media/common/videobuf2/videobuf2-v4l2.c | 7 +- .../mediatek/vcodec/vdec/vdec_h264_req_if.c | 17 +- drivers/media/v4l2-core/v4l2-h264.c | 261 ++++++++++++++---- drivers/media/v4l2-core/v4l2-mem2mem.c | 1 + .../staging/media/hantro/hantro_g1_h264_dec.c | 38 +-- drivers/staging/media/hantro/hantro_h264.c | 119 ++++++-- drivers/staging/media/hantro/hantro_hw.h | 7 +- drivers/staging/media/hantro/hantro_v4l2.c | 25 ++ .../media/hantro/rockchip_vpu2_hw_h264_dec.c | 98 +++---- drivers/staging/media/rkvdec/rkvdec-h264.c | 154 ++++++++--- drivers/staging/media/rkvdec/rkvdec.c | 35 +-- drivers/staging/media/rkvdec/rkvdec.h | 2 + include/media/v4l2-h264.h | 31 ++- 14 files changed, 580 insertions(+), 225 deletions(-) diff --git a/libavcodec/v4l2_request_h264.c b/libavcodec/v4l2_request_h264.c index 88da8f0a2d..394bae0550 100644 --- a/libavcodec/v4l2_request_h264.c +++ b/libavcodec/v4l2_request_h264.c @@ -66,7 +66,7 @@ static void fill_dpb_entry(struct v4l2_h264_dpb_entry *entry, const H264Picture { entry->reference_ts = ff_v4l2_request_get_capture_timestamp(pic->f); entry->pic_num = pic->pic_id; - entry->frame_num = pic->frame_num; + entry->frame_num = pic->long_ref ? pic->pic_id : pic->frame_num; entry->fields = pic->reference & V4L2_H264_FRAME_REF; entry->flags = V4L2_H264_DPB_ENTRY_FLAG_VALID; if (entry->fields)