[alsa-devel,RFC/PATCH,v6,03/12] media: Entities, pads and links

  I wrote:
> I'll see if I can draw up the ALSA-specific media stuff over the weekend.

Sorry, wrong weekend.

Anyway, below are some remarks and a patch.


* Entity types

TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node
in a graph, which does not distinguish it from other entity types
because all entities are part of the topology graph.  I chose "device"
as this type describes entities that are visible as some device node to
other software.

TYPE_EXT describes entities that represent some interface to the
external world, TYPE_INT those that are internal to the entire device.
(I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
to be an even more meaningless name.)


ALSA mixer controls are not directly represented; a better fit for the
architecture of actual devices is that one or more mixer controls can be
associated with an entity.  (This can be done with a field of the mixer
control.)


* Entity properties

There needs to be a mechanism to associate meta-information (properties)
with entities.  This information should be optional and extensible, but,
when being handled inside the kernel, doesn't need to be more than
a read-only blob.  I think that something like ALSA's TLV format (used
for mixer controls) can be used here.  (I'm not mentioning the X-word
here, except to note that the "M" stands for "markup".)


* Entity subtypes

EXT_JACK_ANALOG represents any analog audio and/or video connector.
Properties for audio jacks would be jack type (TRS/RCA), color code,
line level, position, etc.

EXT_JACK_DIGITAL represents a digital connector like S/PDIF (coax/
TOSLINK), ADAT, TDIF, or MADI.

EXT_JACK_BUS represents a bus like FireWire and comes from the USB audio
spec.  (I doubt that any devices with this entitiy will ever exist.)

EXT_INSTRUMENT represents something like an e-guitar, keyboard, or MIDI
controller.  (Instrument entities are typically audio sources and MIDI
sources and sinks, but can also be audio sinks.)

EXT_SPEAKER also includes headphones; there might be made a case for
having those as a separate subtype.

EXT_PLAYER represents a device like a CD/DVD/tape player.  Recorders can
also write to that device, so "player" might not be an ideal name.

EXT_BROADCAST represents devices like TV tuners, satellite receivers,
cable tuners, or radios.

INT_SYNTHESIZER converts MIDI to audio.

INT_NOISE_SOURCE comes from the USB audio spec; this is not an attempt
to describe the characteristics of consumer-grade devices :-) , but
represents an internal noise source for level calibration or measurements.

INT_CONTROLS may have multiple independent controls (this is USB's
Feature Unit); INT_EFFECT may have multiple controls that affect one
single algorithm.

INT_CHANNEL_SPLIT/MERGE are needed for HDAudio devices, whose topology
information has only stereo links.


* Entity specifications

While TYPE_DEVICE entities can be identified by their device node, other
entities typcially have just a numeric ID.  For that, it would be useful
to make do without separate identification and let the driver choose the
entity ID.


Signed-off-by: Clemens Ladisch <clemens@ladisch.de>

--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
  

Hi Clemens,

On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
> I wrote:
> > I'll see if I can draw up the ALSA-specific media stuff over the weekend.
> 
> Sorry, wrong weekend.
> 
> Anyway, below are some remarks and a patch.

Thank you. Please see my comments inline.

> * Entity types
> 
> TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node
> in a graph, which does not distinguish it from other entity types
> because all entities are part of the topology graph.  I chose "device"
> as this type describes entities that are visible as some device node to
> other software.

What this type describes is a device node. Both NODE and DEVICE can be 
confusing in my opinion, but DEVICE_NODE is a bit long.

> TYPE_EXT describes entities that represent some interface to the
> external world, TYPE_INT those that are internal to the entire device.
> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
> to be an even more meaningless name.)

SUBDEV comes from the V4L2 world, and I agree that it might not be a very good 
name.

I'm not sure I would split entities in internal/external categories. I would 
create a category for connectors though.

> ALSA mixer controls are not directly represented; a better fit for the
> architecture of actual devices is that one or more mixer controls can be
> associated with an entity.  (This can be done with a field of the mixer
> control.)

Agreed.

> * Entity properties
> 
> There needs to be a mechanism to associate meta-information (properties)
> with entities.  This information should be optional and extensible, but,
> when being handled inside the kernel, doesn't need to be more than
> a read-only blob.  I think that something like ALSA's TLV format (used
> for mixer controls) can be used here.  (I'm not mentioning the X-word
> here, except to note that the "M" stands for "markup".)

I've been thinking of adding a new ioctl for that. It's something we need to 
draft. The UVC driver will need it, and I'm pretty sure other V4L2 drivers 
would find it useful as well.

> * Entity subtypes
> 
> EXT_JACK_ANALOG represents any analog audio and/or video connector.
> Properties for audio jacks would be jack type (TRS/RCA), color code,
> line level, position, etc.
> 
> EXT_JACK_DIGITAL represents a digital connector like S/PDIF (coax/
> TOSLINK), ADAT, TDIF, or MADI.
> 
> EXT_JACK_BUS represents a bus like FireWire and comes from the USB audio
> spec.  (I doubt that any devices with this entitiy will ever exist.)
> 
> EXT_INSTRUMENT represents something like an e-guitar, keyboard, or MIDI
> controller.  (Instrument entities are typically audio sources and MIDI
> sources and sinks, but can also be audio sinks.)
> 
> EXT_SPEAKER also includes headphones; there might be made a case for
> having those as a separate subtype.

Shouldn't headphones be represented by an EXT_JACK_ANALOG ?

> EXT_PLAYER represents a device like a CD/DVD/tape player.  Recorders can
> also write to that device, so "player" might not be an ideal name.
> 
> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
> cable tuners, or radios.

There's clearly an overlap with V4L here. Hopefully someone from the linux-
media list can comment on this.

> INT_SYNTHESIZER converts MIDI to audio.
> 
> INT_NOISE_SOURCE comes from the USB audio spec; this is not an attempt
> to describe the characteristics of consumer-grade devices :-) , but
> represents an internal noise source for level calibration or measurements.
> 
> INT_CONTROLS may have multiple independent controls (this is USB's
> Feature Unit); INT_EFFECT may have multiple controls that affect one
> single algorithm.

I'd describe this as a feature unit/processing unit then.

> INT_CHANNEL_SPLIT/MERGE are needed for HDAudio devices, whose topology
> information has only stereo links.

Some of those INT entities could also be implemented in dedicated chips, so I 
really think the EXT/INT split doesn't make too much sense. Should we have an 
AUDIO category ?

> * Entity specifications
> 
> While TYPE_DEVICE entities can be identified by their device node, other
> entities typcially have just a numeric ID.

In V4L2 sub-devices have (or rather will have once the media controller 
patches will be integrated) device nodes as well, so exposing that information 
is required.

> For that, it would be useful to make do without separate identification and
> let the driver choose the entity ID.

How would drivers do that ? What if you have two instances of the same chip (a 
video sensor, audio mixer, ...) on the same board ?

> Signed-off-by: Clemens Ladisch <clemens@ladisch.de>
> 
> --- linux/include/linux/media.h
> +++ linux/include/linux/media.h
> @@ -46,16 +46,36 @@ struct media_device_info {
>  #define MEDIA_ENTITY_TYPE_MASK			0x00ff0000
>  #define MEDIA_ENTITY_SUBTYPE_MASK		0x0000ffff
> 
> -#define MEDIA_ENTITY_TYPE_NODE			(1 << MEDIA_ENTITY_TYPE_SHIFT)
> -#define MEDIA_ENTITY_TYPE_NODE_V4L		(MEDIA_ENTITY_TYPE_NODE + 1)
> -#define MEDIA_ENTITY_TYPE_NODE_FB		(MEDIA_ENTITY_TYPE_NODE + 2)
> -#define MEDIA_ENTITY_TYPE_NODE_ALSA		(MEDIA_ENTITY_TYPE_NODE + 3)
> -#define MEDIA_ENTITY_TYPE_NODE_DVB		(MEDIA_ENTITY_TYPE_NODE + 4)
> +#define MEDIA_ENTITY_TYPE_DEVICE		(1 << MEDIA_ENTITY_TYPE_SHIFT)
> +#define MEDIA_ENTITY_TYPE_DEVICE_V4L		(MEDIA_ENTITY_TYPE_DEVICE + 1)
> +#define MEDIA_ENTITY_TYPE_DEVICE_FB		(MEDIA_ENTITY_TYPE_DEVICE + 2)
> +#define MEDIA_ENTITY_TYPE_DEVICE_DVB		(MEDIA_ENTITY_TYPE_DEVICE + 3)
> +#define MEDIA_ENTITY_TYPE_DEVICE_ALSA_PCM	(MEDIA_ENTITY_TYPE_DEVICE + 4)
> +#define MEDIA_ENTITY_TYPE_DEVICE_ALSA_MIDI	(MEDIA_ENTITY_TYPE_DEVICE + 5)
> 
> -#define MEDIA_ENTITY_TYPE_SUBDEV		(2 << MEDIA_ENTITY_TYPE_SHIFT)
> -#define MEDIA_ENTITY_TYPE_SUBDEV_SENSOR		(MEDIA_ENTITY_TYPE_SUBDEV + 1)
> -#define MEDIA_ENTITY_TYPE_SUBDEV_FLASH		(MEDIA_ENTITY_TYPE_SUBDEV + 2)
> -#define MEDIA_ENTITY_TYPE_SUBDEV_LENS		(MEDIA_ENTITY_TYPE_SUBDEV + 3)
> +#define MEDIA_ENTITY_TYPE_EXT			(2 << MEDIA_ENTITY_TYPE_SHIFT)
> +#define MEDIA_ENTITY_TYPE_EXT_SENSOR		(MEDIA_ENTITY_TYPE_EXT + 1)
> +#define MEDIA_ENTITY_TYPE_EXT_FLASH		(MEDIA_ENTITY_TYPE_EXT + 2)
> +#define MEDIA_ENTITY_TYPE_EXT_LENS		(MEDIA_ENTITY_TYPE_EXT + 3)
> +#define MEDIA_ENTITY_TYPE_EXT_JACK_MIDI		(MEDIA_ENTITY_TYPE_EXT + 4)
> +#define MEDIA_ENTITY_TYPE_EXT_JACK_ANALOG	(MEDIA_ENTITY_TYPE_EXT + 5)
> +#define MEDIA_ENTITY_TYPE_EXT_JACK_DIGITAL	(MEDIA_ENTITY_TYPE_EXT + 6)
> +#define MEDIA_ENTITY_TYPE_EXT_JACK_BUS		(MEDIA_ENTITY_TYPE_EXT + 7)
> +#define MEDIA_ENTITY_TYPE_EXT_INSTRUMENT	(MEDIA_ENTITY_TYPE_EXT + 8)
> +#define MEDIA_ENTITY_TYPE_EXT_SPEAKER		(MEDIA_ENTITY_TYPE_EXT + 9)
> +#define MEDIA_ENTITY_TYPE_EXT_MICROPHONE	(MEDIA_ENTITY_TYPE_EXT + 10)
> +#define MEDIA_ENTITY_TYPE_EXT_PLAYER		(MEDIA_ENTITY_TYPE_EXT + 11)
> +#define MEDIA_ENTITY_TYPE_EXT_BROADCAST		(MEDIA_ENTITY_TYPE_EXT + 12)
> +
> +#define MEDIA_ENTITY_TYPE_INT			(3 << MEDIA_ENTITY_TYPE_SHIFT)
> +#define MEDIA_ENTITY_TYPE_INT_SYNTHESIZER	(MEDIA_ENTITY_TYPE_INT + 1)
> +#define MEDIA_ENTITY_TYPE_INT_NOISE_SOURCE	(MEDIA_ENTITY_TYPE_INT + 2)
> +#define MEDIA_ENTITY_TYPE_INT_MIXER		(MEDIA_ENTITY_TYPE_INT + 3)
> +#define MEDIA_ENTITY_TYPE_INT_SELECTOR		(MEDIA_ENTITY_TYPE_INT + 4)
> +#define MEDIA_ENTITY_TYPE_INT_CONTROLS		(MEDIA_ENTITY_TYPE_INT + 5)
> +#define MEDIA_ENTITY_TYPE_INT_EFFECT		(MEDIA_ENTITY_TYPE_INT + 6)
> +#define MEDIA_ENTITY_TYPE_INT_CHANNEL_SPLIT	(MEDIA_ENTITY_TYPE_INT + 7)
> +#define MEDIA_ENTITY_TYPE_INT_CHANNEL_MERGE	(MEDIA_ENTITY_TYPE_INT + 8)
> 
>  #define MEDIA_ENTITY_FLAG_DEFAULT		(1 << 0)
> 
> @@ -72,7 +92,7 @@ struct media_entity_desc {
>  	__u32 reserved[4];
> 
>  	union {
> -		/* Node specifications */
> +		/* Device specifications */
>  		struct {
>  			__u32 major;
>  			__u32 minor;
> @@ -81,11 +101,15 @@ struct media_entity_desc {
>  			__u32 major;
>  			__u32 minor;
>  		} fb;
> -		int alsa;
> +		struct {
> +			__u32 card;
> +			__u32 device;
> +			__s32 subdevice;
> +		} alsa;

I will already incorporate this change, and I'll wait for other opinions on 
the types before changing them.

>  		int dvb;
> 
>  		/* Sub-device specifications */
>  		/* Nothing needed yet */
>  		__u8 raw[184];
>  	};
>  };
  

> Hi Clemens,
>
> On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
>> I wrote:
>> > I'll see if I can draw up the ALSA-specific media stuff over the
>> weekend.
>>
>> Sorry, wrong weekend.
>>
>> Anyway, below are some remarks and a patch.
>
> Thank you. Please see my comments inline.
>
>> * Entity types
>>
>> TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node
>> in a graph, which does not distinguish it from other entity types
>> because all entities are part of the topology graph.  I chose "device"
>> as this type describes entities that are visible as some device node to
>> other software.
>
> What this type describes is a device node. Both NODE and DEVICE can be
> confusing in my opinion, but DEVICE_NODE is a bit long.

What about DEVNODE? I think that would be a good alternative.

>> TYPE_EXT describes entities that represent some interface to the
>> external world, TYPE_INT those that are internal to the entire device.
>> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
>> to be an even more meaningless name.)
>
> SUBDEV comes from the V4L2 world, and I agree that it might not be a very
> good
> name.

SUBDEV refers to a specific type of driver. Within the v4l world it is
well defined. So I prefer to keep this. Perhaps some additional comments
or documentation can be added to clarify this.

> I'm not sure I would split entities in internal/external categories. I
> would
> create a category for connectors though.

I agree. It was always the plan to eventually add connectors, but v4l
didn't really need it (it already has an API to enumerate connectors).

>> ALSA mixer controls are not directly represented; a better fit for the
>> architecture of actual devices is that one or more mixer controls can be
>> associated with an entity.  (This can be done with a field of the mixer
>> control.)
>
> Agreed.
>
>> * Entity properties
>>
>> There needs to be a mechanism to associate meta-information (properties)
>> with entities.  This information should be optional and extensible, but,
>> when being handled inside the kernel, doesn't need to be more than
>> a read-only blob.  I think that something like ALSA's TLV format (used
>> for mixer controls) can be used here.  (I'm not mentioning the X-word
>> here, except to note that the "M" stands for "markup".)
>
> I've been thinking of adding a new ioctl for that. It's something we need
> to
> draft. The UVC driver will need it, and I'm pretty sure other V4L2 drivers
> would find it useful as well.
>
>> * Entity subtypes
>>
>> EXT_JACK_ANALOG represents any analog audio and/or video connector.
>> Properties for audio jacks would be jack type (TRS/RCA), color code,
>> line level, position, etc.
>>
>> EXT_JACK_DIGITAL represents a digital connector like S/PDIF (coax/
>> TOSLINK), ADAT, TDIF, or MADI.
>>
>> EXT_JACK_BUS represents a bus like FireWire and comes from the USB audio
>> spec.  (I doubt that any devices with this entitiy will ever exist.)
>>
>> EXT_INSTRUMENT represents something like an e-guitar, keyboard, or MIDI
>> controller.  (Instrument entities are typically audio sources and MIDI
>> sources and sinks, but can also be audio sinks.)
>>
>> EXT_SPEAKER also includes headphones; there might be made a case for
>> having those as a separate subtype.
>
> Shouldn't headphones be represented by an EXT_JACK_ANALOG ?
>
>> EXT_PLAYER represents a device like a CD/DVD/tape player.  Recorders can
>> also write to that device, so "player" might not be an ideal name.
>>
>> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
>> cable tuners, or radios.

I don't think it is right to talk about 'represents devices'. I'd rephrase
it to 'connects to devices'.

> There's clearly an overlap with V4L here. Hopefully someone from the
> linux-
> media list can comment on this.

I don't think this will be a problem. Initially we probably won't be
enumerating connectors for V4L since it already has its own API for that.

>> INT_SYNTHESIZER converts MIDI to audio.
>>
>> INT_NOISE_SOURCE comes from the USB audio spec; this is not an attempt
>> to describe the characteristics of consumer-grade devices :-) , but
>> represents an internal noise source for level calibration or
>> measurements.
>>
>> INT_CONTROLS may have multiple independent controls (this is USB's
>> Feature Unit); INT_EFFECT may have multiple controls that affect one
>> single algorithm.
>
> I'd describe this as a feature unit/processing unit then.
>
>> INT_CHANNEL_SPLIT/MERGE are needed for HDAudio devices, whose topology
>> information has only stereo links.
>
> Some of those INT entities could also be implemented in dedicated chips,
> so I
> really think the EXT/INT split doesn't make too much sense. Should we have
> an
> AUDIO category ?
>
>> * Entity specifications
>>
>> While TYPE_DEVICE entities can be identified by their device node, other
>> entities typcially have just a numeric ID.
>
> In V4L2 sub-devices have (or rather will have once the media controller
> patches will be integrated) device nodes as well, so exposing that
> information
> is required.
>
>> For that, it would be useful to make do without separate identification
>> and
>> let the driver choose the entity ID.
>
> How would drivers do that ? What if you have two instances of the same
> chip (a
> video sensor, audio mixer, ...) on the same board ?

Regards,

       Hans
  

Hi Hans,

On Tuesday 14 December 2010 13:40:21 Hans Verkuil wrote:
> > On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
> >> * Entity types
> >> 
> >> TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node
> >> in a graph, which does not distinguish it from other entity types
> >> because all entities are part of the topology graph.  I chose "device"
> >> as this type describes entities that are visible as some device node to
> >> other software.
> > 
> > What this type describes is a device node. Both NODE and DEVICE can be
> > confusing in my opinion, but DEVICE_NODE is a bit long.
> 
> What about DEVNODE? I think that would be a good alternative.

Fine with me. Clemens, any opinion on that ?

> >> TYPE_EXT describes entities that represent some interface to the
> >> external world, TYPE_INT those that are internal to the entire device.
> >> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
> >> to be an even more meaningless name.)
> > 
> > SUBDEV comes from the V4L2 world, and I agree that it might not be a very
> > good name.
> 
> SUBDEV refers to a specific type of driver. Within the v4l world it is
> well defined. So I prefer to keep this. Perhaps some additional comments
> or documentation can be added to clarify this.

Should this be clarified by using V4L2_SUBDEV instead then ? What about ALSA 
entities, should they use MEDIA_ENTITY_TYPE_ALSA_* ?

> > I'm not sure I would split entities in internal/external categories. I
> > would create a category for connectors though.
> 
> I agree. It was always the plan to eventually add connectors, but v4l
> didn't really need it (it already has an API to enumerate connectors).
> 
> >> ALSA mixer controls are not directly represented; a better fit for the
> >> architecture of actual devices is that one or more mixer controls can be
> >> associated with an entity.  (This can be done with a field of the mixer
> >> control.)
> > 
> > Agreed.
> > 
> >> * Entity properties
> >> 
> >> There needs to be a mechanism to associate meta-information (properties)
> >> with entities.  This information should be optional and extensible, but,
> >> when being handled inside the kernel, doesn't need to be more than
> >> a read-only blob.  I think that something like ALSA's TLV format (used
> >> for mixer controls) can be used here.  (I'm not mentioning the X-word
> >> here, except to note that the "M" stands for "markup".)
> > 
> > I've been thinking of adding a new ioctl for that. It's something we need
> > to draft. The UVC driver will need it, and I'm pretty sure other V4L2
> > drivers would find it useful as well.
> > 
> >> * Entity subtypes
> >> 
> >> EXT_JACK_ANALOG represents any analog audio and/or video connector.
> >> Properties for audio jacks would be jack type (TRS/RCA), color code,
> >> line level, position, etc.
> >> 
> >> EXT_JACK_DIGITAL represents a digital connector like S/PDIF (coax/
> >> TOSLINK), ADAT, TDIF, or MADI.
> >> 
> >> EXT_JACK_BUS represents a bus like FireWire and comes from the USB audio
> >> spec.  (I doubt that any devices with this entitiy will ever exist.)
> >> 
> >> EXT_INSTRUMENT represents something like an e-guitar, keyboard, or MIDI
> >> controller.  (Instrument entities are typically audio sources and MIDI
> >> sources and sinks, but can also be audio sinks.)
> >> 
> >> EXT_SPEAKER also includes headphones; there might be made a case for
> >> having those as a separate subtype.
> > 
> > Shouldn't headphones be represented by an EXT_JACK_ANALOG ?
> > 
> >> EXT_PLAYER represents a device like a CD/DVD/tape player.  Recorders can
> >> also write to that device, so "player" might not be an ideal name.
> >> 
> >> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
> >> cable tuners, or radios.
> 
> I don't think it is right to talk about 'represents devices'. I'd rephrase
> it to 'connects to devices'.
> 
> > There's clearly an overlap with V4L here. Hopefully someone from the
> > linux-media list can comment on this.
> 
> I don't think this will be a problem. Initially we probably won't be
> enumerating connectors for V4L since it already has its own API for that.

My understanding is that EXT_BROADCAST really represents the TV tuners, ..., 
not the connector they connect to. Some (all ?) of them are definitely V4L2 
subdevs.

> >> INT_SYNTHESIZER converts MIDI to audio.
> >> 
> >> INT_NOISE_SOURCE comes from the USB audio spec; this is not an attempt
> >> to describe the characteristics of consumer-grade devices :-) , but
> >> represents an internal noise source for level calibration or
> >> measurements.
> >> 
> >> INT_CONTROLS may have multiple independent controls (this is USB's
> >> Feature Unit); INT_EFFECT may have multiple controls that affect one
> >> single algorithm.
> > 
> > I'd describe this as a feature unit/processing unit then.
> > 
> >> INT_CHANNEL_SPLIT/MERGE are needed for HDAudio devices, whose topology
> >> information has only stereo links.
> > 
> > Some of those INT entities could also be implemented in dedicated chips,
> > so I really think the EXT/INT split doesn't make too much sense. Should we
> > have an AUDIO category ?
> > 
> >> * Entity specifications
> >> 
> >> While TYPE_DEVICE entities can be identified by their device node, other
> >> entities typcially have just a numeric ID.
> > 
> > In V4L2 sub-devices have (or rather will have once the media controller
> > patches will be integrated) device nodes as well, so exposing that
> > information
> > is required.
> > 
> >> For that, it would be useful to make do without separate identification
> >> and let the driver choose the entity ID.
> > 
> > How would drivers do that ? What if you have two instances of the same
> > chip (a video sensor, audio mixer, ...) on the same board ?
  

Laurent Pinchart wrote:
> On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
>> TYPE_EXT describes entities that represent some interface to the
>> external world, TYPE_INT those that are internal to the entire device.
>> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
>> to be an even more meaningless name.)
> 
> SUBDEV comes from the V4L2 world, and I agree that it might not be a very good 
> name.
> 
> I'm not sure I would split entities in internal/external categories. I would
> create a category for connectors though.

I'm not disagreeing, but what is actually the distinction between types
and subtypes?  ;-)

>> * Entity properties
>> 
>> There needs to be a mechanism to associate meta-information (properties)
>> with entities.  This information should be optional and extensible, but,
>> when being handled inside the kernel, doesn't need to be more than
>> a read-only blob.  I think that something like ALSA's TLV format (used
>> for mixer controls) can be used here.  (I'm not mentioning the X-word
>> here, except to note that the "M" stands for "markup".)
> 
> I've been thinking of adding a new ioctl for that. It's something we need to
> draft. The UVC driver will need it, and I'm pretty sure other V4L2 drivers
> would find it useful as well.

I'm imagining a "read-the-properties" ioctl that just returns the
entity's blob.

>> EXT_SPEAKER also includes headphones; there might be made a case for
>> having those as a separate subtype.
> 
> Shouldn't headphones be represented by an EXT_JACK_ANALOG ?

Headphone jacks are jacks; there are also USB headphones.

>> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
>> cable tuners, or radios.
> 
> There's clearly an overlap with V4L here.

These come from the USB audio spec.  Video devices are indeed likely to
be more detailed than just a single audio source. :)

>> INT_CONTROLS may have multiple independent controls (this is USB's
>> Feature Unit); INT_EFFECT may have multiple controls that affect one
>> single algorithm.
> 
> I'd describe this as a feature unit/processing unit then.

I was aiming for more descriptive names, but I agree that the original
names might be more useful.

> Should we have an AUDIO category ?

Probably not, because there are combined audio/video jacks, any maybe
other entities.

>> * Entity specifications
>> 
>> While TYPE_DEVICE entities can be identified by their device node, other
>> entities typcially have just a numeric ID.
> 
> In V4L2 sub-devices have (or rather will have once the media controller
> patches will be integrated) device nodes as well, so exposing that information
> is required.

USB and HDA entities already have numeric IDs.

>> For that, it would be useful to make do without separate identification and
>> let the driver choose the entity ID.
> 
> How would drivers do that ? What if you have two instances of the same chip
> (a video sensor, audio mixer, ...) on the same board ?

Then those would get different IDs; USB descriptors always describe the
entire device.


Regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
  

Laurent Pinchart wrote:
> On Tuesday 14 December 2010 13:40:21 Hans Verkuil wrote:
>> > On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
>> >> * Entity types
>> >> 
>> >> TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node
>> >> in a graph, which does not distinguish it from other entity types
>> >> because all entities are part of the topology graph.  I chose "device"
>> >> as this type describes entities that are visible as some device node to
>> >> other software.
>> > 
>> > What this type describes is a device node. Both NODE and DEVICE can be
>> > confusing in my opinion, but DEVICE_NODE is a bit long.
>> 
>> What about DEVNODE? I think that would be a good alternative.
> 
> Fine with me. Clemens, any opinion on that ?

Fine with me too.

> > >> TYPE_EXT describes entities that represent some interface to the
> > >> external world, TYPE_INT those that are internal to the entire device.
> > >> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
> > >> to be an even more meaningless name.)
> > > 
> > > SUBDEV comes from the V4L2 world, and I agree that it might not be a very
> > > good name.
> > 
> > SUBDEV refers to a specific type of driver. Within the v4l world it is
> > well defined. So I prefer to keep this. Perhaps some additional comments
> > or documentation can be added to clarify this.
> 
> Should this be clarified by using V4L2_SUBDEV instead then ?

If the "SUBDEV" concept doesn't exist outside V4L, that would indeed be
better.

I don't want to rename things that come out of existing frameworks; this
naming discussion makes sense only for those entity (sub)types that can
be shared between them.  Are there any, besides jacks?

> What about ALSA entities, should they use MEDIA_ENTITY_TYPE_ALSA_* ?

The entity types representing ALSA devices are already named "ALSA".


Regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
  

At Tue, 14 Dec 2010 14:31:55 +0100,
Clemens Ladisch wrote:
> 
> > Should we have an AUDIO category ?
> 
> Probably not, because there are combined audio/video jacks, any maybe
> other entities.

Yes, nowadays HDMI / DP are pretty common, for example.


Takashi
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
  

Hi Clemens,

On Tuesday 14 December 2010 14:31:55 Clemens Ladisch wrote:
> Laurent Pinchart wrote:
> > On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
> >> TYPE_EXT describes entities that represent some interface to the
> >> external world, TYPE_INT those that are internal to the entire device.
> >> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
> >> to be an even more meaningless name.)
> > 
> > SUBDEV comes from the V4L2 world, and I agree that it might not be a very
> > good name.
> > 
> > I'm not sure I would split entities in internal/external categories. I
> > would create a category for connectors though.
> 
> I'm not disagreeing, but what is actually the distinction between types
> and subtypes?  ;-)

The type is currently used to distinguish between entities that stream media 
data from/to memory and other entities. They need to be handled differently in 
the kernel for power management purposes for instance.

I'm not sure if we should create new types, or just remove the type/subtype 
distinction and add a flag somewhere.

> >> * Entity properties
> >> 
> >> There needs to be a mechanism to associate meta-information (properties)
> >> with entities.  This information should be optional and extensible, but,
> >> when being handled inside the kernel, doesn't need to be more than
> >> a read-only blob.  I think that something like ALSA's TLV format (used
> >> for mixer controls) can be used here.  (I'm not mentioning the X-word
> >> here, except to note that the "M" stands for "markup".)
> > 
> > I've been thinking of adding a new ioctl for that. It's something we need
> > to draft. The UVC driver will need it, and I'm pretty sure other V4L2
> > drivers would find it useful as well.
> 
> I'm imagining a "read-the-properties" ioctl that just returns the
> entity's blob.

Martin Rubli has already proposed something similar a while ago on the linux-
media mailing list. His proposal didn't use TLV though.

> >> EXT_SPEAKER also includes headphones; there might be made a case for
> >> having those as a separate subtype.
> > 
> > Shouldn't headphones be represented by an EXT_JACK_ANALOG ?
> 
> Headphone jacks are jacks; there are also USB headphones.

So EXT_SPEAKER are speakers not connected through a jack (USB, internal 
analog, ...) ?

> >> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
> >> cable tuners, or radios.
> > 
> > There's clearly an overlap with V4L here.
> 
> These come from the USB audio spec.  Video devices are indeed likely to
> be more detailed than just a single audio source. :)

Does EXT_BROADCAST represent the TV tuner (or satellite receiver, cable tuner, 
radio tuner, ...) itself, or the connection between the tuner and the rest of 
the device ? Most TV tuner are currently handled by V4L2 and would thus turn 
up as V4L2 subdevs (I'm not sure if that's what we want in the long term, but 
it's at least the current situation).

> >> INT_CONTROLS may have multiple independent controls (this is USB's
> >> Feature Unit); INT_EFFECT may have multiple controls that affect one
> >> single algorithm.
> > 
> > I'd describe this as a feature unit/processing unit then.
> 
> I was aiming for more descriptive names, but I agree that the original
> names might be more useful.
> 
> > Should we have an AUDIO category ?
> 
> Probably not, because there are combined audio/video jacks, any maybe
> other entities.

> >> * Entity specifications
> >> 
> >> While TYPE_DEVICE entities can be identified by their device node, other
> >> entities typcially have just a numeric ID.
> > 
> > In V4L2 sub-devices have (or rather will have once the media controller
> > patches will be integrated) device nodes as well, so exposing that
> > information is required.
> 
> USB and HDA entities already have numeric IDs.

Right. Same for USB Video Class.

We could let drivers set the entity ID, and have the core fill it if the value 
is 0. Non-zero values would have to be checked for uniqueness though. Hans, a 
comment on that ?

> >> For that, it would be useful to make do without separate identification
> >> and let the driver choose the entity ID.
> > 
> > How would drivers do that ? What if you have two instances of the same
> > chip (a video sensor, audio mixer, ...) on the same board ?
> 
> Then those would get different IDs; USB descriptors always describe the
> entire device.
  

Hi Clemens, Laurent, Hans & others!

Clemens Ladisch wrote:
> I wrote:
>> I'll see if I can draw up the ALSA-specific media stuff over the weekend.
> 
> Sorry, wrong weekend.
> 
> Anyway, below are some remarks and a patch.
> 
> 
> * Entity types
> 
> TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node
> in a graph, which does not distinguish it from other entity types
> because all entities are part of the topology graph.  I chose "device"
> as this type describes entities that are visible as some device node to
> other software.
> 
> TYPE_EXT describes entities that represent some interface to the
> external world, TYPE_INT those that are internal to the entire device.
> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
> to be an even more meaningless name.)
> 
> 
> ALSA mixer controls are not directly represented; a better fit for the
> architecture of actual devices is that one or more mixer controls can be
> associated with an entity.  (This can be done with a field of the mixer
> control.)
> 
> 
> * Entity properties
> 
> There needs to be a mechanism to associate meta-information (properties)
> with entities.  This information should be optional and extensible, but,
> when being handled inside the kernel, doesn't need to be more than
> a read-only blob.  I think that something like ALSA's TLV format (used
> for mixer controls) can be used here.  (I'm not mentioning the X-word
> here, except to note that the "M" stands for "markup".)
> 
> 
> * Entity subtypes
> 
> EXT_JACK_ANALOG represents any analog audio and/or video connector.
> Properties for audio jacks would be jack type (TRS/RCA), color code,
> line level, position, etc.
> 
> EXT_JACK_DIGITAL represents a digital connector like S/PDIF (coax/
> TOSLINK), ADAT, TDIF, or MADI.
> 
> EXT_JACK_BUS represents a bus like FireWire and comes from the USB audio
> spec.  (I doubt that any devices with this entitiy will ever exist.)
> 
> EXT_INSTRUMENT represents something like an e-guitar, keyboard, or MIDI
> controller.  (Instrument entities are typically audio sources and MIDI
> sources and sinks, but can also be audio sinks.)
> 
> EXT_SPEAKER also includes headphones; there might be made a case for
> having those as a separate subtype.
> 
> EXT_PLAYER represents a device like a CD/DVD/tape player.  Recorders can
> also write to that device, so "player" might not be an ideal name.
> 
> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
> cable tuners, or radios.
> 
> INT_SYNTHESIZER converts MIDI to audio.
> 
> INT_NOISE_SOURCE comes from the USB audio spec; this is not an attempt
> to describe the characteristics of consumer-grade devices :-) , but
> represents an internal noise source for level calibration or measurements.
> 
> INT_CONTROLS may have multiple independent controls (this is USB's
> Feature Unit); INT_EFFECT may have multiple controls that affect one
> single algorithm.
> 
> INT_CHANNEL_SPLIT/MERGE are needed for HDAudio devices, whose topology
> information has only stereo links.

This naming already has been commented, but what do you think: should
the type explicitly tell what kind of interface, if any, is exported to
user space?

Only MEDIA_ENTITY_NODE_* types do this currently, and
MEDIA_ENTITY_TYPE_SUBDEV_* to some extent, but the way is not consistent
at the moment. MEDIA_ENTITY_NODE_* range has lost of different
interfaces whereas MEDIA_ENTITY_TYPE_SUBDEV_* are basically offering
v4l2_subdev and beyond that, suggesting what kind of controls might be
found from the nodes.

I would expect that the interfaces offered by the character devices
would be somewhat standardised in the end like v4l2_subdev user space
interface.

The types above are mostly describing the role of an entity, which might
be interesting as well.

Regards,
  

> Laurent Pinchart wrote:
>> On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
>>> TYPE_EXT describes entities that represent some interface to the
>>> external world, TYPE_INT those that are internal to the entire device.
>>> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
>>> to be an even more meaningless name.)
>>
>> SUBDEV comes from the V4L2 world, and I agree that it might not be a
>> very good
>> name.
>>
>> I'm not sure I would split entities in internal/external categories. I
>> would
>> create a category for connectors though.
>
> I'm not disagreeing, but what is actually the distinction between types
> and subtypes?  ;-)

The type tells what the behavior is of an entity. E.g., type DEVNODE
represents device node(s) in userspace, V4L2_SUBDEV represents a v4l2
sub-device, etc. The subtype tells whether a V4L2_SUBDEV is a sensor or a
receiver or whatever. Nice to know, but it doesn't change the way
sub-devices work.

In the case of connectors you would create a CONNECTOR type and have a
bunch of subtypes for all the variations of connectors.

That said, I'm not sure whether the distinction is useful for DEVNODEs.
You do need to know the subtype in order to interpret the union correctly.

Laurent, does the MC code test against the DEVNODE type? I.e., does the MC
code ignore the subtype of a DEVNODE, or does it always use it?

Regards,

        Hans
  

Hi Hans,

On Tuesday 14 December 2010 15:51:08 Hans Verkuil wrote:
> > Laurent Pinchart wrote:
> >> On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
> >>> TYPE_EXT describes entities that represent some interface to the
> >>> external world, TYPE_INT those that are internal to the entire device.
> >>> (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems
> >>> to be an even more meaningless name.)
> >> 
> >> SUBDEV comes from the V4L2 world, and I agree that it might not be a
> >> very good
> >> name.
> >> 
> >> I'm not sure I would split entities in internal/external categories. I
> >> would
> >> create a category for connectors though.
> > 
> > I'm not disagreeing, but what is actually the distinction between types
> > and subtypes?  ;-)
> 
> The type tells what the behavior is of an entity. E.g., type DEVNODE
> represents device node(s) in userspace, V4L2_SUBDEV represents a v4l2
> sub-device, etc. The subtype tells whether a V4L2_SUBDEV is a sensor or a
> receiver or whatever. Nice to know, but it doesn't change the way
> sub-devices work.
> 
> In the case of connectors you would create a CONNECTOR type and have a
> bunch of subtypes for all the variations of connectors.
> 
> That said, I'm not sure whether the distinction is useful for DEVNODEs.
> You do need to know the subtype in order to interpret the union correctly.
> 
> Laurent, does the MC code test against the DEVNODE type? I.e., does the MC
> code ignore the subtype of a DEVNODE, or does it always use it?

The MC code uses the DEVNODE type, ignoring the subtype, for power management. 
When a device node is opened all entities in the chain need to be powered up.
  

Laurent Pinchart wrote:
> On Tuesday 14 December 2010 14:31:55 Clemens Ladisch wrote:
> > Laurent Pinchart wrote:
> > > On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
> > >> EXT_SPEAKER also includes headphones; there might be made a case for
> > >> having those as a separate subtype.
> > > 
> > > Shouldn't headphones be represented by an EXT_JACK_ANALOG ?
> > 
> > Headphone jacks are jacks; there are also USB headphones.
> 
> So EXT_SPEAKER are speakers not connected through a jack (USB, internal
> analog, ...) ?

Yes.

When there is jack, the driver often does not know what is connected.

> > >> EXT_BROADCAST represents devices like TV tuners, satellite receivers,
> > >> cable tuners, or radios.
> > > 
> > > There's clearly an overlap with V4L here.
> > 
> > These come from the USB audio spec.  Video devices are indeed likely to
> > be more detailed than just a single audio source. :)
> 
> Does EXT_BROADCAST represent the TV tuner (or satellite receiver, cable tuner,
> radio tuner, ...) itself, or the connection between the tuner and the rest of
> the device ? Most TV tuner are currently handled by V4L2 and would thus turn
> up as V4L2 subdevs (I'm not sure if that's what we want in the long term, but
> it's at least the current situation).

From the point of view of an audio device, this would be just some audio
source, much like a connector.  We don't need this if there is some
better V4L entitity that the USB audio entity can be mapped to.


Regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
  

Hi Clemens,

On Tuesday 14 December 2010 14:49:15 Clemens Ladisch wrote:
> Laurent Pinchart wrote:
> > On Tuesday 14 December 2010 13:40:21 Hans Verkuil wrote:
> >> > On Monday 13 December 2010 17:10:51 Clemens Ladisch wrote:
> >> >> * Entity types
> >> >> 
> >> >> TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a
> >> >> node in a graph, which does not distinguish it from other entity
> >> >> types because all entities are part of the topology graph.  I chose
> >> >> "device" as this type describes entities that are visible as some
> >> >> device node to other software.
> >> > 
> >> > What this type describes is a device node. Both NODE and DEVICE can be
> >> > confusing in my opinion, but DEVICE_NODE is a bit long.
> >> 
> >> What about DEVNODE? I think that would be a good alternative.
> > 
> > Fine with me. Clemens, any opinion on that ?
> 
> Fine with me too.

OK I'll use that name.

> > > >> TYPE_EXT describes entities that represent some interface to the
> > > >> external world, TYPE_INT those that are internal to the entire
> > > >> device. (I'm not sure if that distinction is very useful, but
> > > >> TYPE_SUBDEV seems to be an even more meaningless name.)
> > > > 
> > > > SUBDEV comes from the V4L2 world, and I agree that it might not be a
> > > > very good name.
> > > 
> > > SUBDEV refers to a specific type of driver. Within the v4l world it is
> > > well defined. So I prefer to keep this. Perhaps some additional
> > > comments or documentation can be added to clarify this.
> > 
> > Should this be clarified by using V4L2_SUBDEV instead then ?
> 
> If the "SUBDEV" concept doesn't exist outside V4L, that would indeed be
> better.
> 
> I don't want to rename things that come out of existing frameworks; this
> naming discussion makes sense only for those entity (sub)types that can
> be shared between them.  Are there any, besides jacks?

Some entities like TV tuners play a dual audio/video role. I'm not sure how to 
handle them, I lack experience in that field.

> > What about ALSA entities, should they use MEDIA_ENTITY_TYPE_ALSA_* ?
> 
> The entity types representing ALSA devices are already named "ALSA".

I was talking about the INT_* types. They're ALSA-specific, but have no ALSA 
in the type name.