From patchwork Mon Mar 27 03:13:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?TW91ZHkgSG8gKOS9leWul+WOnyk=?= X-Patchwork-Id: 90811 X-Patchwork-Delegate: hverkuil@xs4all.nl Received: from vger.kernel.org ([23.128.96.18]) by www.linuxtv.org with esmtp (Exim 4.92) (envelope-from ) id 1pgdJS-007kgv-4j; Mon, 27 Mar 2023 03:14:26 +0000 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232242AbjC0DOE (ORCPT + 1 other); Sun, 26 Mar 2023 23:14:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232208AbjC0DN4 (ORCPT ); Sun, 26 Mar 2023 23:13:56 -0400 Received: from mailgw02.mediatek.com (unknown [210.61.82.184]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ACC14C22; Sun, 26 Mar 2023 20:13:45 -0700 (PDT) X-UUID: 5d9403c0cc4d11edb6b9f13eb10bd0fe-20230327 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mediatek.com; s=dk; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From; bh=5dmmAN53knGP8A38vOJ+8chP7YSe5P6lPPiDtcUaDnc=; b=OjrRAIex9hIVUVhmnj8ikieVk4Vi3v3eqYWTIXrFKj8RzWB2lLxY0y5fl5M2mt08+p6YmkItzlhw4nXpvdCrlK6XTrwl9rYmJudhFgpkDmSRMTq+30HYXAv3r+Fu82Efjmev2X30lJaICWhNmBObRJclhUgKv8JMHs4LbL05y6A=; X-CID-P-RULE: Release_Ham X-CID-O-INFO: VERSION:1.1.22,REQID:7f3e2305-ce9d-4763-8e59-dba96ab0db00,IP:0,U RL:0,TC:0,Content:0,EDM:0,RT:0,SF:95,FILE:0,BULK:0,RULE:Release_Ham,ACTION :release,TS:95 X-CID-INFO: VERSION:1.1.22,REQID:7f3e2305-ce9d-4763-8e59-dba96ab0db00,IP:0,URL :0,TC:0,Content:0,EDM:0,RT:0,SF:95,FILE:0,BULK:0,RULE:Spam_GS981B3D,ACTION :quarantine,TS:95 X-CID-META: VersionHash:120426c,CLOUDID:b498fef6-ddba-41c3-91d9-10eeade8eac7,B ulkID:230327111339RI6GJ9G6,BulkQuantity:0,Recheck:0,SF:38|29|28|17|19|48,T C:nil,Content:0,EDM:-3,IP:nil,URL:0,File:nil,Bulk:nil,QS:nil,BEC:nil,COL:0 ,OSI:0,OSA:0,AV:0 X-CID-BVR: 0,NGT X-CID-BAS: 0,NGT,0,_ X-UUID: 5d9403c0cc4d11edb6b9f13eb10bd0fe-20230327 Received: from mtkmbs10n1.mediatek.inc [(172.21.101.34)] by mailgw02.mediatek.com (envelope-from ) (Generic MTA with TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 256/256) with ESMTP id 944995957; Mon, 27 Mar 2023 11:13:37 +0800 Received: from mtkmbs13n2.mediatek.inc (172.21.101.108) by mtkmbs10n2.mediatek.inc (172.21.101.183) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.25; Mon, 27 Mar 2023 11:13:36 +0800 Received: from mtksdccf07.mediatek.inc (172.21.84.99) by mtkmbs13n2.mediatek.inc (172.21.101.73) with Microsoft SMTP Server id 15.2.1118.25 via Frontend Transport; Mon, 27 Mar 2023 11:13:36 +0800 From: Moudy Ho To: Mauro Carvalho Chehab , Matthias Brugger , Hans Verkuil , "Ping-Hsun Wu" CC: , , , , , Moudy Ho Subject: [PATCH v7 01/12] media: platform: mtk-mdp3: fix potential frame size overflow in mdp_try_fmt_mplane() Date: Mon, 27 Mar 2023 11:13:24 +0800 Message-ID: <20230327031335.9663-2-moudy.ho@mediatek.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20230327031335.9663-1-moudy.ho@mediatek.com> References: <20230327031335.9663-1-moudy.ho@mediatek.com> MIME-Version: 1.0 X-MTK: N X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-LSpam-Score: -2.5 (--) X-LSpam-Report: No, score=-2.5 required=5.0 tests=BAYES_00=-1.9,DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,HEADER_FROM_DIFFERENT_DOMAINS=0.5,MAILING_LIST_MULTI=-1,UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no Fix overflow risk when setting certain formats whose frame size exceeds a RGB24 with 7723x7723 resolution. For example, a 7723x7724 RGB24 frame: 1. bpl (byte per line) = 7723 * 3. 2. Overflow occurs when bpl * 7724 * depth. Fixes: 61890ccaefaf ("media: platform: mtk-mdp3: add MediaTek MDP3 driver") Signed-off-by: Moudy Ho --- drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c index 4e84a37ecdfc..36336d169bd9 100644 --- a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c +++ b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c @@ -4,6 +4,7 @@ * Author: Ping-Hsun Wu */ +#include #include #include #include @@ -428,14 +429,15 @@ const struct mdp_format *mdp_try_fmt_mplane(struct v4l2_format *f, u32 bpl = pix_mp->plane_fmt[i].bytesperline; u32 min_si, max_si; u32 si = pix_mp->plane_fmt[i].sizeimage; + u64 di; bpl = clamp(bpl, min_bpl, max_bpl); pix_mp->plane_fmt[i].bytesperline = bpl; - min_si = (bpl * pix_mp->height * fmt->depth[i]) / - fmt->row_depth[i]; - max_si = (bpl * s.max_height * fmt->depth[i]) / - fmt->row_depth[i]; + di = (u64)bpl * pix_mp->height * fmt->depth[i]; + min_si = (u32)div_u64(di, fmt->row_depth[i]); + di = (u64)bpl * s.max_height * fmt->depth[i]; + max_si = (u32)div_u64(di, fmt->row_depth[i]); si = clamp(si, min_si, max_si); pix_mp->plane_fmt[i].sizeimage = si;