| Message ID | 1681096465-17287-1-git-send-email-wentong.wu@intel.com (mailing list archive) |
|---|---|
| Headers |
Received: from vger.kernel.org ([23.128.96.18])
by www.linuxtv.org with esmtp (Exim 4.92)
(envelope-from <linux-media-owner@vger.kernel.org>)
id 1plhzY-00G3s8-1c; Mon, 10 Apr 2023 03:14:52 +0000
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
id S229498AbjDJDOs (ORCPT <rfc822;mkrufky@linuxtv.org> + 1 other);
Sun, 9 Apr 2023 23:14:48 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58636 "EHLO
lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
with ESMTP id S229689AbjDJDOi (ORCPT
<rfc822;linux-media@vger.kernel.org>); Sun, 9 Apr 2023 23:14:38 -0400
Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31])
by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE488358A
for <linux-media@vger.kernel.org>;
Sun, 9 Apr 2023 20:14:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
d=intel.com; i=@intel.com; q=dns/txt; s=Intel;
t=1681096477; x=1712632477;
h=from:to:cc:subject:date:message-id;
bh=NNiY5mqUuJlBJTlTQeQ9HIXugwnQiWdk5U9gpjaXBdU=;
b=Z0hoswT+CW+LwNFKeTeyS/d5iAqRp9n+5RQnAPMcIJG+lexfE9VGwjaL
EDD01Y3ek9DCsl6742VqD80oOHN13fXvyiKxPfxkEAul1ZGSUQ+mQErN8
tD96PC0kyh4ZveL58cdcWtWBtwEozlmbS+JJyQ9p/dI3AI38OzyvR9GEf
k9q2aLkbddsfbilVGLuUNDV+vGs3huQJh+e+xhmDbaWpZX1b8Y/oJuQ+W
9DaQ3cYiqaupGmfnbUNZYqqA7yXAdZl7QEk3IoM81uWBy+YTpN2MqOuTm
2mIsL+/BK2igskA4Scb1YuMwWM0vqaOpD4j0xQsZTvOKjAczVO55sw3yK
Q==;
X-IronPort-AV: E=McAfee;i="6600,9927,10675"; a="406104110"
X-IronPort-AV: E=Sophos;i="5.98,332,1673942400";
d="scan'208";a="406104110"
Received: from fmsmga005.fm.intel.com ([10.253.24.32])
by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
09 Apr 2023 20:14:37 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=McAfee;i="6600,9927,10675"; a="1017854016"
X-IronPort-AV: E=Sophos;i="5.98,332,1673942400";
d="scan'208";a="1017854016"
Received: from shsensorbuild2.sh.intel.com ([10.239.134.197])
by fmsmga005.fm.intel.com with ESMTP; 09 Apr 2023 20:14:34 -0700
From: Wentong Wu <wentong.wu@intel.com>
To: sakari.ailus@linux.intel.com, hdegoede@redhat.com,
djrscally@gmail.com, laurent.pinchart@ideasonboard.com,
linux-media@vger.kernel.org
Cc: bingbu.cao@linux.intel.com, zhifeng.wang@intel.com,
xiang.ye@intel.com, tian.shu.qiu@intel.com,
Wentong Wu <wentong.wu@intel.com>
Subject: [PATCH v5 0/3] media: pci: intel: ivsc: Add driver of Intel Visual
Sensing Controller(IVSC)
Date: Mon, 10 Apr 2023 11:14:22 +0800
Message-Id: <1681096465-17287-1-git-send-email-wentong.wu@intel.com>
X-Mailer: git-send-email 2.7.4
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,
SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-media.vger.kernel.org>
X-Mailing-List: linux-media@vger.kernel.org
X-LSpam-Score: -2.5 (--)
X-LSpam-Report: No,
score=-2.5 required=5.0 tests=BAYES_00=-1.9,DKIMWL_WL_HIGH=0.001,DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,HEADER_FROM_DIFFERENT_DOMAINS=0.5,MAILING_LIST_MULTI=-1
autolearn=ham autolearn_force=no
|
| Series |
media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
|
|
Message
Wu, Wentong
April 10, 2023, 3:14 a.m. UTC
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
companion chip designed to provide secure and low power vision capability
to IA platforms. IVSC is available in existing commercial platforms from
multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC
interfaces directly with the platform main camera sensor via a CSI-2 link
and processes the image data with the embedded AI engine. The detected
events are sent over I2C to ISH (Intel Sensor Hub) for additional data
fusion from multiple sensors. The fusion results are used to implement
advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system
Since the Image Processing Unit(IPU) used on the host processor needs to
configure the CSI-2 link in normal camera usages, the CSI-2 link and
camera sensor can only be used in mutually-exclusive ways by host IPU and
IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera sensor using the
interfaces exported via v4l2 sub-device.
Switching ownership requires an interface with two different hardware
modules inside IVSC. The software interface to these modules is via Intel
MEI (The Intel Management Engine) commands. These two hardware modules
have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's
ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
IVSC also provides a privacy mode. When privacy mode is turned on, camera
sensor can't be used. This means that both ACE and host IPU can't get
image data. And when this mode is turned on, users are informed via v4l2
control API.
In summary, to acquire ownership of camera by IPU driver, first ACE
module needs to be informed of ownership and then to setup MIPI CSI-2
link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules
inside IVSC.
- ivsc_csi: MEI client driver to send commands and receive notifications
from CSI module.
- ivsc_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release
camera sensor and CSI-2 link.
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------
| Host Processor |
| |
| ----------------- ----------------- --------------- |
| | | | | | | I2C |
| | IPU | | ISH | |camera driver|--| |
| | | | | | | | |
| ----------------- ----------------- --------------- | |
| | | | | |
| | | --------------- | |
| | | | | | |
| | | | IVSC driver | | |
| | | | | | |
| | | --------------- | |
| | | | | |
----------------|-----------------------|----------------------|---------|---
| CSI | I2C |SPI |
| | | |
----------------|-----------------------|---------------- | |
| IVSC | | | |
| | | | |
| ----------------- ----------------- | | |
| | | | | | | |
| | CSI | | ACE | |------| |
| | | | | | |
| ----------------- ----------------- | |
| | | I2C | |
----------------|-----------------------|---------------- |
| CSI | |
| | |
-------------------------------- |
| | I2C |
| camera sensor |-----------------------------|
| |
--------------------------------
---
v5:
- probe mei_csi only after software node has been setup
v4:
- call v4l2_ctrl_handler_free() if setting up the handler failed
- set V4L2_CTRL_FLAG_READ_ONLY for privacy_ctrl
- add dev_warn if failed to switch CSI-2 link to IVSC
- use v4l2_fwnode_endpoint_alloc_parse to get num_data_lanes
- add document about how sensor connected to IVSC is powered
- move lock to mei_ace_send
- check return value for device_link_add
Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
ACPI: delay enumeration of devices with a _DEP pointing to IVSC device
drivers/acpi/scan.c | 2 +
drivers/media/pci/Kconfig | 1 +
drivers/media/pci/intel/Makefile | 2 +
drivers/media/pci/intel/ivsc/Kconfig | 12 +
drivers/media/pci/intel/ivsc/Makefile | 9 +
drivers/media/pci/intel/ivsc/mei_ace.c | 560 ++++++++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_csi.c | 772 +++++++++++++++++++++++++++++++++
7 files changed, 1358 insertions(+)
create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
create mode 100644 drivers/media/pci/intel/ivsc/Makefile
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
Comments
Hi, Please help review this V5 patch set, thanks a lot! BR, Wentong > -----Original Message----- > From: Wu, Wentong <wentong.wu@intel.com> > Sent: Monday, April 10, 2023 11:14 AM > To: sakari.ailus@linux.intel.com; hdegoede@redhat.com; djrscally@gmail.com; > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion > chip designed to provide secure and low power vision capability to IA platforms. > IVSC is available in existing commercial platforms from multiple OEMs. > > The primary use case of IVSC is to bring in context awareness. IVSC interfaces > directly with the platform main camera sensor via a CSI-2 link and processes the > image data with the embedded AI engine. The detected events are sent over I2C > to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The > fusion results are used to implement advanced use cases like: > - Face detection to unlock screen > - Detect user presence to manage backlight setting or waking up system > > Since the Image Processing Unit(IPU) used on the host processor needs to > configure the CSI-2 link in normal camera usages, the CSI-2 link and camera > sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By > default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take > ownership of the CSI-2 link and camera sensor using the interfaces exported via > v4l2 sub-device. > > Switching ownership requires an interface with two different hardware modules > inside IVSC. The software interface to these modules is via Intel MEI (The Intel > Management Engine) commands. These two hardware modules have two > different MEI UUIDs to enumerate. These hardware modules are: > - ACE (Algorithm Context Engine): This module is for algorithm computing when > IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. > This hardware module is used to set ownership of camera sensor. > - CSI (Camera Serial Interface): This module is used to route camera sensor data > either to IVSC or to host for IPU driver and application. > > IVSC also provides a privacy mode. When privacy mode is turned on, camera > sensor can't be used. This means that both ACE and host IPU can't get image > data. And when this mode is turned on, users are informed via v4l2 control API. > > In summary, to acquire ownership of camera by IPU driver, first ACE module > needs to be informed of ownership and then to setup MIPI CSI-2 link for the > camera sensor and IPU. > > Implementation: > There are two different drivers to handle ACE and CSI hardware modules inside > IVSC. > - ivsc_csi: MEI client driver to send commands and receive notifications from > CSI module. > - ivsc_ace: MEI client driver to send commands and get status from ACE module. > Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera > sensor and CSI-2 link. > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor. > ----------------------------------------------------------------------------- > | Host Processor | > | | > | ----------------- ----------------- --------------- | > | | | | | | | I2C | > | | IPU | | ISH | |camera driver|--| | > | | | | | | | | | > | ----------------- ----------------- --------------- | | > | | | | | | > | | | --------------- | | > | | | | | | | > | | | | IVSC driver | | | > | | | | | | | > | | | --------------- | | > | | | | | | > ----------------|-----------------------|----------------------|---------|--- > | CSI | I2C |SPI | > | | | | > ----------------|-----------------------|---------------- | | > | IVSC | | | | > | | | | | > | ----------------- ----------------- | | | > | | | | | | | | > | | CSI | | ACE | |------| | > | | | | | | | > | ----------------- ----------------- | | > | | | I2C | | > ----------------|-----------------------|---------------- | > | CSI | | > | | | > -------------------------------- | > | | I2C | > | camera sensor |-----------------------------| > | | > -------------------------------- > > --- > v5: > - probe mei_csi only after software node has been setup > > v4: > - call v4l2_ctrl_handler_free() if setting up the handler failed > - set V4L2_CTRL_FLAG_READ_ONLY for privacy_ctrl > - add dev_warn if failed to switch CSI-2 link to IVSC > - use v4l2_fwnode_endpoint_alloc_parse to get num_data_lanes > - add document about how sensor connected to IVSC is powered > - move lock to mei_ace_send > - check return value for device_link_add > > Wentong Wu (3): > media: pci: intel: ivsc: Add CSI submodule > media: pci: intel: ivsc: Add ACE submodule > ACPI: delay enumeration of devices with a _DEP pointing to IVSC device > > drivers/acpi/scan.c | 2 + > drivers/media/pci/Kconfig | 1 + > drivers/media/pci/intel/Makefile | 2 + > drivers/media/pci/intel/ivsc/Kconfig | 12 + > drivers/media/pci/intel/ivsc/Makefile | 9 + > drivers/media/pci/intel/ivsc/mei_ace.c | 560 ++++++++++++++++++++++++ > drivers/media/pci/intel/ivsc/mei_csi.c | 772 > +++++++++++++++++++++++++++++++++ > 7 files changed, 1358 insertions(+) > create mode 100644 drivers/media/pci/intel/ivsc/Kconfig > create mode 100644 drivers/media/pci/intel/ivsc/Makefile > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c > > -- > 2.7.4