[v2,00/15] Intel IPU6 and IPU6 input system drivers

  From: Bingbu Cao <bingbu.cao@intel.com>

This patch series adds a driver for Intel IPU6 input system.
IPU6 is the sixth generation of Imaging Processing Unit, it is a PCI
device which can be found in some Intel Client Platforms. User can use
IPU6 to capture images from MIPI camera sensors.

IPU6 has its own firmware which exposes ABIs to driver, and communicates
with CSE to do firmware authentication. IPU6 has its MMU hardware, so
the driver sets up a page table to allow IPU6 DMA to access the system
memory.

IPU6 input system driver uses MC and V4L2 sub-device APIs besides V4L2.
---

v1 -> v2:
  - Add support with latest v4l2 async API
  - Add support for latest bridge and Intel ivsc driver
  - Fix lots of coding style issues not covered by checkpatch.pl
  - Fix several bugs
  - Fix several code issues found by Sparse and cppcheck

RFC -> v1:
  - Add multiplexed streams support
  - Use auxiliary bus to register IPU6 devices
  - Add IPU6 hardware and driver overview documentation
  - Updata IPU6 admin-guide documentation
  - Update number of source pads and video nodes to support
    multiplexed streams

Bingbu Cao (15):
  media: intel/ipu6: add Intel IPU6 PCI device driver
  media: intel/ipu6: add IPU auxiliary devices
  media: intel/ipu6: add IPU6 buttress interface driver
  media: intel/ipu6: CPD parsing for get firmware components
  media: intel/ipu6: add IPU6 DMA mapping API and MMU table
  media: intel/ipu6: add syscom interfaces between firmware and driver
  media: intel/ipu6: input system ABI between firmware and driver
  media: intel/ipu6: add IPU6 CSI2 receiver v4l2 sub-device
  media: intel/ipu6: add the CSI2 DPHY implementation
  media: intel/ipu6: add input system driver
  media: intel/ipu6: input system video capture nodes
  media: add Kconfig and Makefile for IPU6
  MAINTAINERS: add maintainers for Intel IPU6 input system driver
  Documentation: add Intel IPU6 ISYS driver admin-guide doc
  Documentation: add documentation of Intel IPU6 driver and hardware
    overview

 Documentation/admin-guide/media/ipu6-isys.rst |  159 ++
 .../admin-guide/media/ipu6_isys_graph.svg     |  338 +++++
 .../admin-guide/media/ipu6_isys_multi.svg     | 1124 ++++++++++++++
 .../admin-guide/media/v4l-drivers.rst         |    1 +
 .../driver-api/media/drivers/index.rst        |    1 +
 .../driver-api/media/drivers/ipu6.rst         |  205 +++
 MAINTAINERS                                   |   10 +
 drivers/media/pci/intel/Kconfig               |    1 +
 drivers/media/pci/intel/Makefile              |    1 +
 drivers/media/pci/intel/ipu6/Kconfig          |   17 +
 drivers/media/pci/intel/ipu6/Makefile         |   23 +
 drivers/media/pci/intel/ipu6/ipu6-bus.c       |  157 ++
 drivers/media/pci/intel/ipu6/ipu6-bus.h       |   50 +
 drivers/media/pci/intel/ipu6/ipu6-buttress.c  |  900 +++++++++++
 drivers/media/pci/intel/ipu6/ipu6-buttress.h  |  107 ++
 drivers/media/pci/intel/ipu6/ipu6-cpd.c       |  355 +++++
 drivers/media/pci/intel/ipu6/ipu6-cpd.h       |  104 ++
 drivers/media/pci/intel/ipu6/ipu6-dma.c       |  491 ++++++
 drivers/media/pci/intel/ipu6/ipu6-dma.h       |   20 +
 drivers/media/pci/intel/ipu6/ipu6-fw-com.c    |  411 +++++
 drivers/media/pci/intel/ipu6/ipu6-fw-com.h    |   47 +
 drivers/media/pci/intel/ipu6/ipu6-fw-isys.c   |  484 ++++++
 drivers/media/pci/intel/ipu6/ipu6-fw-isys.h   |  570 +++++++
 drivers/media/pci/intel/ipu6/ipu6-isys-csi2.c |  654 ++++++++
 drivers/media/pci/intel/ipu6/ipu6-isys-csi2.h |   81 +
 .../media/pci/intel/ipu6/ipu6-isys-dwc-phy.c  |  533 +++++++
 .../media/pci/intel/ipu6/ipu6-isys-jsl-phy.c  |  237 +++
 .../media/pci/intel/ipu6/ipu6-isys-mcd-phy.c  |  712 +++++++++
 .../media/pci/intel/ipu6/ipu6-isys-queue.c    |  846 +++++++++++
 .../media/pci/intel/ipu6/ipu6-isys-queue.h    |   94 ++
 .../media/pci/intel/ipu6/ipu6-isys-subdev.c   |  379 +++++
 .../media/pci/intel/ipu6/ipu6-isys-subdev.h   |   58 +
 .../media/pci/intel/ipu6/ipu6-isys-video.c    | 1239 +++++++++++++++
 .../media/pci/intel/ipu6/ipu6-isys-video.h    |  129 ++
 drivers/media/pci/intel/ipu6/ipu6-isys.c      | 1345 +++++++++++++++++
 drivers/media/pci/intel/ipu6/ipu6-isys.h      |  201 +++
 drivers/media/pci/intel/ipu6/ipu6-mmu.c       |  828 ++++++++++
 drivers/media/pci/intel/ipu6/ipu6-mmu.h       |   67 +
 .../intel/ipu6/ipu6-platform-buttress-regs.h  |  230 +++
 .../intel/ipu6/ipu6-platform-isys-csi2-reg.h  |  187 +++
 .../media/pci/intel/ipu6/ipu6-platform-regs.h |  179 +++
 drivers/media/pci/intel/ipu6/ipu6.c           |  955 ++++++++++++
 drivers/media/pci/intel/ipu6/ipu6.h           |  352 +++++
 43 files changed, 14882 insertions(+)
 create mode 100644 Documentation/admin-guide/media/ipu6-isys.rst
 create mode 100644 Documentation/admin-guide/media/ipu6_isys_graph.svg
 create mode 100644 Documentation/admin-guide/media/ipu6_isys_multi.svg
 create mode 100644 Documentation/driver-api/media/drivers/ipu6.rst
 create mode 100644 drivers/media/pci/intel/ipu6/Kconfig
 create mode 100644 drivers/media/pci/intel/ipu6/Makefile
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-bus.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-bus.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-buttress.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-buttress.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-cpd.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-cpd.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-dma.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-dma.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-fw-com.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-fw-com.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-fw-isys.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-fw-isys.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-csi2.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-csi2.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-dwc-phy.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-jsl-phy.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-mcd-phy.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-queue.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-queue.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-subdev.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-subdev.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-video.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys-video.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-isys.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-mmu.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-mmu.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-platform-buttress-regs.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-platform-isys-csi2-reg.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6-platform-regs.h
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6.c
 create mode 100644 drivers/media/pci/intel/ipu6/ipu6.h
  

Hi Bingbu,

On 1/9/24 04:51, Bingbu Cao wrote:
> 
> Hans,
> 
> On 1/8/24 10:23 PM, Hans de Goede wrote:
>> Hi Bingbu,
>>
>> On 1/8/24 05:07, Bingbu Cao wrote:
>>> Hans,
>>>
>>> On 11/8/23 7:59 PM, Hans de Goede wrote:
>>>> Hi Bingbu,
>>>>
>>>> On 10/24/23 13:29, bingbu.cao@intel.com wrote:
>>>>> From: Bingbu Cao <bingbu.cao@intel.com>
>>>>>
>>>>> This patch series adds a driver for Intel IPU6 input system.
>>>>> IPU6 is the sixth generation of Imaging Processing Unit, it is a PCI
>>>>> device which can be found in some Intel Client Platforms. User can use
>>>>> IPU6 to capture images from MIPI camera sensors.
>>>>>
>>>>> IPU6 has its own firmware which exposes ABIs to driver, and communicates
>>>>> with CSE to do firmware authentication. IPU6 has its MMU hardware, so
>>>>> the driver sets up a page table to allow IPU6 DMA to access the system
>>>>> memory.
>>>>>
>>>>> IPU6 input system driver uses MC and V4L2 sub-device APIs besides V4L2.
>>>>
>>>> I have been testing this on a TigerLake system, a Dell Latitude 9420
>>>> with ov01a1s and the packed 10 bit bayer pixel fmt: V4L2_PIX_FMT_SGRBG10P,
>>>> which libcamera together with (WIP) software debayer support picks
>>>> by default does not work. There are many many CSI errors in dmesg
>>>> and only the first 10 or so lines of the image show.
>>>>
>>>> Disabling the packed format by removing it from ipu6_isys_pfmts[],
>>>> making libcamera pick the unpacked (every 10 bits per pixel data
>>>> stored in a 16 bit word in output buffer) fixes this.
>>>>
>>>> Are the packed bayer formats supposed to work on Tiger Lake, or
>>>> is this a known issue which Intel's own userspace stack avoids
>>>> by always picking the unpacked format ?
>>>
>>> I just tested the packed bayer on my device (ov01a10) and did not
>>> observe the problems like yours, could you share the isys kernel log?
>>
>> Was this on a device with a Tiger Lake CPU / plain ipu6 (not ipu6ep) ?
> 
> Sorry, it's ipu6ep.
> 
> I have no TigerLake so far.
> The difference between ipu6 and ipu6ep is the MIPI CSI-2 D-PHY. But it
> should not have no impact on the packed format.
> 
> Could you share the debug log on TigerLake?

I don't have access to the machine atm, but I should get it back
in 2 weeks from now. I'll email you debug logs then.

Regards,

Hans
  

Hi folks,

On Tue, Oct 24, 2023 at 07:29:09PM +0800, bingbu.cao@intel.com wrote:
> From: Bingbu Cao <bingbu.cao@intel.com>
> 
> This patch series adds a driver for Intel IPU6 input system.
> IPU6 is the sixth generation of Imaging Processing Unit, it is a PCI
> device which can be found in some Intel Client Platforms. User can use
> IPU6 to capture images from MIPI camera sensors.
> 
> IPU6 has its own firmware which exposes ABIs to driver, and communicates
> with CSE to do firmware authentication. IPU6 has its MMU hardware, so
> the driver sets up a page table to allow IPU6 DMA to access the system
> memory.
> 
> IPU6 input system driver uses MC and V4L2 sub-device APIs besides V4L2.

I've made some changes on top of this (besides some done by Bingbu). I've
pushed the current patches here
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=ipu6>. What's
changed:

- CSI-2 port address calculation simplification.

- Tested (and fixed) driver probing on a Dell XPS 9315 that has IVSC and
  Linux camera description that ignores the IVSC. This works if the
  intel-ipu6 driver is first removed and then re-probed. I'm not quite sure
  why it doesn't re-probe on its own yet. I haven't tried streaming yet.

- The patches are on top of the metadata set. If you want to try without,
  just drop the metadata patches (and the last functional driver patch). It
  might compile. I'll test this soon, too.

So overall there's some progress but work remains to do. I'll post the IVSC
and ACPI patches after a little bit more testing.
  

On Wed, Feb 07, 2024 at 07:00:27AM +0000, Sakari Ailus wrote:
> Hi folks,
> 
> On Tue, Oct 24, 2023 at 07:29:09PM +0800, bingbu.cao@intel.com wrote:
> > From: Bingbu Cao <bingbu.cao@intel.com>
> > 
> > This patch series adds a driver for Intel IPU6 input system.
> > IPU6 is the sixth generation of Imaging Processing Unit, it is a PCI
> > device which can be found in some Intel Client Platforms. User can use
> > IPU6 to capture images from MIPI camera sensors.
> > 
> > IPU6 has its own firmware which exposes ABIs to driver, and communicates
> > with CSE to do firmware authentication. IPU6 has its MMU hardware, so
> > the driver sets up a page table to allow IPU6 DMA to access the system
> > memory.
> > 
> > IPU6 input system driver uses MC and V4L2 sub-device APIs besides V4L2.
> 
> I've made some changes on top of this (besides some done by Bingbu). I've
> pushed the current patches here
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=ipu6>. What's
> changed:
> 
> - CSI-2 port address calculation simplification.
> 
> - Tested (and fixed) driver probing on a Dell XPS 9315 that has IVSC and
>   Linux camera description that ignores the IVSC. This works if the
>   intel-ipu6 driver is first removed and then re-probed. I'm not quite sure
>   why it doesn't re-probe on its own yet. I haven't tried streaming yet.
> 
> - The patches are on top of the metadata set. If you want to try without,
>   just drop the metadata patches (and the last functional driver patch). It
>   might compile. I'll test this soon, too.
> 
> So overall there's some progress but work remains to do. I'll post the IVSC
> and ACPI patches after a little bit more testing.

I've added further patches to the branch and dropped some... Capturing
works now on Dell XPS 9315 with ov01a10 without issues (AFAICT), and that
pipeline includes IVSC. I'm upstreaming the un-media patches to the
respective trees so some of this is going to take a while to merged back to
the media tree.

I've also removed the massive number of extra links from the CSI-2 receiver
subdevs and video nodes. This part remains untested though, the ov01a10
sensor doesn't have support for embedded data (!).

Just FYI.