From patchwork Tue Apr 23 10:51:37 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Katsuya MATSUBARA X-Patchwork-Id: 18156 X-Patchwork-Delegate: g.liakhovetski@gmx.de Received: from mail.tu-berlin.de ([130.149.7.33]) by www.linuxtv.org with esmtp (Exim 4.72) (envelope-from ) id 1UUaq1-0004T6-P9; Tue, 23 Apr 2013 12:52:41 +0200 X-tubIT-Incoming-IP: 209.132.180.67 Received: from vger.kernel.org ([209.132.180.67]) by mail.tu-berlin.de (exim-4.75/mailfrontend-4) with esmtp id 1UUaq1-0001l8-AI; Tue, 23 Apr 2013 12:52:41 +0200 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756153Ab3DWKwi (ORCPT + 1 other); Tue, 23 Apr 2013 06:52:38 -0400 Received: from mail-pa0-f51.google.com ([209.85.220.51]:56553 "EHLO mail-pa0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756149Ab3DWKwh (ORCPT ); Tue, 23 Apr 2013 06:52:37 -0400 Received: by mail-pa0-f51.google.com with SMTP id jh10so420341pab.10 for ; Tue, 23 Apr 2013 03:52:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:to:cc:subject:date:message-id:x-mailer:in-reply-to :references:x-gm-message-state; bh=dV31Lclp8PT8AL3jEnX4LsRT6DgzHtEMTK/JNPtSgGU=; b=P6GHlzBanjcD3si5SjfQwDLv4CNNyQXT+iTOD0vE5NrxWtJn0DaKbkG08u4ifmq22t 7Cc7BvY5tdePUrs76ydI8Wgg183mi0ZJSR00e7A9EzUZR5SDQGpCaYYqkA3HROJNRguS avxmaOo3xxmQVv20+T2efDrcfmbjLuvcYAJMwOrVEBZdLZ/7nJGnW4TSEsktEhMEMNvb Ca0Qu6VL5c0yjcyfIcMtv3OF19B57kF8pJTKB90+5IF6YF/4kKIhl7uljAypXETMadbJ 7m2sMNJqIw94fQrpjMD5FncE25Rc629ax8jqxUZmdv80SPSqbMampzsNGzEqgdBMvBXA aY+A== X-Received: by 10.68.228.161 with SMTP id sj1mr40448588pbc.151.1366714357277; Tue, 23 Apr 2013 03:52:37 -0700 (PDT) Received: from localhost.localdomain ([219.106.231.132]) by mx.google.com with ESMTPS id ih1sm26463098pbb.44.2013.04.23.03.52.35 (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 23 Apr 2013 03:52:36 -0700 (PDT) From: Katsuya Matsubara To: Guennadi Liakhovetski Cc: Mauro Carvalho Chehab , Hans Verkuil , linux-media@vger.kernel.org, linux-sh@vger.kernel.org, Katsuya Matsubara Subject: [PATCH 3/3] [media] sh_veu: fix the buffer size calculation Date: Tue, 23 Apr 2013 19:51:37 +0900 Message-Id: <1366714297-2784-4-git-send-email-matsu@igel.co.jp> X-Mailer: git-send-email 1.7.0.4 In-Reply-To: <1366714297-2784-1-git-send-email-matsu@igel.co.jp> References: <1366714297-2784-1-git-send-email-matsu@igel.co.jp> X-Gm-Message-State: ALoCoQlVXCl4IVHgfXjkFA5axowvIi3PcZGjnMxMK4N12BBk+ZyguGtXA8O/rWCGQ+u3w1Gfzorv Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-PMX-Version: 5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2013.4.23.104216 X-PMX-Spam: Gauge=IIIIIIII, Probability=8%, Report=' MULTIPLE_RCPTS 0.1, HTML_00_01 0.05, HTML_00_10 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1400_1499 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, URI_ENDS_IN_HTML 0, __ANY_URI 0, __CP_MEDIA_BODY 0, __CP_URI_IN_BODY 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __HAS_X_MAILING_LIST 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MULTIPLE_RCPTS_CC_X2 0, __SANE_MSGID 0, __TO_MALFORMED_2 0, __URI_NO_WWW 0, __URI_NS , __YOUTUBE_RCVD 0' The 'bytesperline' value only indicates the stride of the Y plane if the color format is planar, such as NV12. When calculating the total plane size, the size of CbCr plane must also be considered. Signed-off-by: Katsuya Matsubara --- drivers/media/platform/sh_veu.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c index fa86c6f..358dac5 100644 --- a/drivers/media/platform/sh_veu.c +++ b/drivers/media/platform/sh_veu.c @@ -904,11 +904,11 @@ static int sh_veu_queue_setup(struct vb2_queue *vq, if (ftmp.fmt.pix.width != pix->width || ftmp.fmt.pix.height != pix->height) return -EINVAL; - size = pix->bytesperline ? pix->bytesperline * pix->height : - pix->width * pix->height * fmt->depth >> 3; + size = pix->bytesperline ? pix->bytesperline * pix->height * fmt->depth / fmt->ydepth : + pix->width * pix->height * fmt->depth / fmt->ydepth; } else { vfmt = sh_veu_get_vfmt(veu, vq->type); - size = vfmt->bytesperline * vfmt->frame.height; + size = vfmt->bytesperline * vfmt->frame.height * vfmt->fmt->depth / vfmt->fmt->ydepth; } if (count < 2)