[for,v5.2] videobuf2-core.c: always reacquire USERPTR memory

  The __prepare_userptr() function made the incorrect assumption that if the
same user pointer was used as the last one for which memory was acquired, then
there was no need to re-acquire the memory. This assumption was never properly
tested, and after doing that it became clear that this was in fact wrong.

Change the behavior to always reacquire memory.

Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reported-by: Tomasz Figa <tfiga@chromium.org>
Cc: <stable@vger.kernel.org>      # for v5.1 and up
---
This should be backported to all stable kernels, unfortunately this patch only
applies cleanly to 5.1, so this has to be backported manually.
---
  

Hi Hans,

Thank you for the patch.

On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> The __prepare_userptr() function made the incorrect assumption that if the
> same user pointer was used as the last one for which memory was acquired, then
> there was no need to re-acquire the memory. This assumption was never properly
> tested, and after doing that it became clear that this was in fact wrong.

Could you explain in the commit message why the assumption is not
correct ?

> Change the behavior to always reacquire memory.

One more reason to not use USERPTR :-)

> Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> Reported-by: Tomasz Figa <tfiga@chromium.org>
> Cc: <stable@vger.kernel.org>      # for v5.1 and up
> ---
> This should be backported to all stable kernels, unfortunately this patch only
> applies cleanly to 5.1, so this has to be backported manually.
> ---
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index 4489744fbbd9..a6400391117f 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -1013,7 +1013,7 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>  	void *mem_priv;
>  	unsigned int plane;
>  	int ret = 0;
> -	bool reacquired = vb->planes[0].mem_priv == NULL;
> +	bool called_cleanup = false;
> 
>  	memset(planes, 0, sizeof(planes[0]) * vb->num_planes);
>  	/* Copy relevant information provided by the userspace */
> @@ -1023,15 +1023,6 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>  		return ret;
> 
>  	for (plane = 0; plane < vb->num_planes; ++plane) {
> -		/* Skip the plane if already verified */
> -		if (vb->planes[plane].m.userptr &&
> -			vb->planes[plane].m.userptr == planes[plane].m.userptr
> -			&& vb->planes[plane].length == planes[plane].length)
> -			continue;
> -
> -		dprintk(3, "userspace address for plane %d changed, reacquiring memory\n",
> -			plane);
> -
>  		/* Check if the provided plane buffer is large enough */
>  		if (planes[plane].length < vb->planes[plane].min_length) {
>  			dprintk(1, "provided buffer size %u is less than setup size %u for plane %d\n",
> @@ -1044,8 +1035,8 @@ static int __prepare_userptr(struct vb2_buffer *vb)
> 
>  		/* Release previously acquired memory if present */
>  		if (vb->planes[plane].mem_priv) {
> -			if (!reacquired) {
> -				reacquired = true;
> +			if (!called_cleanup) {
> +				called_cleanup = true;
>  				vb->copied_timestamp = 0;
>  				call_void_vb_qop(vb, buf_cleanup, vb);
>  			}

Could we do this unconditionally before the loop ?

> @@ -1083,17 +1074,14 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>  		vb->planes[plane].data_offset = planes[plane].data_offset;
>  	}
> 
> -	if (reacquired) {
> -		/*
> -		 * One or more planes changed, so we must call buf_init to do
> -		 * the driver-specific initialization on the newly acquired
> -		 * buffer, if provided.
> -		 */
> -		ret = call_vb_qop(vb, buf_init, vb);
> -		if (ret) {
> -			dprintk(1, "buffer initialization failed\n");
> -			goto err;
> -		}
> +	/*
> +	 * Call buf_init to do the driver-specific initialization on
> +	 * the newly acquired buffer.
> +	 */
> +	ret = call_vb_qop(vb, buf_init, vb);
> +	if (ret) {
> +		dprintk(1, "buffer initialization failed\n");
> +		goto err;
>  	}
> 
>  	ret = call_vb_qop(vb, buf_prepare, vb);
  

On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> Hi Hans,
> 
> Thank you for the patch.
> 
> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>> The __prepare_userptr() function made the incorrect assumption that if the
>> same user pointer was used as the last one for which memory was acquired, then
>> there was no need to re-acquire the memory. This assumption was never properly
>> tested, and after doing that it became clear that this was in fact wrong.
> 
> Could you explain in the commit message why the assumption is not
> correct ?

You can free the memory, then allocate it again and you can get the same pointer,
even though it is not necessarily using the same physical pages for the memory
that the kernel is still using for it.

Worse, you can free the memory, then allocate only half the memory you need and
get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
the original mapping still remains), but this can corrupt userspace memory
causing the application to crash. It's not quite clear to me how the memory can
get corrupted. I don't know enough of those low-level mm internals to understand
the sequence of events.

I have test code for v4l2-compliance available if someone wants to test this.

> 
>> Change the behavior to always reacquire memory.
> 
> One more reason to not use USERPTR :-)

I agree.

Regards,

	Hans

> 
>> Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> Reported-by: Tomasz Figa <tfiga@chromium.org>
>> Cc: <stable@vger.kernel.org>      # for v5.1 and up
>> ---
>> This should be backported to all stable kernels, unfortunately this patch only
>> applies cleanly to 5.1, so this has to be backported manually.
>> ---
>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>> index 4489744fbbd9..a6400391117f 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-core.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
>> @@ -1013,7 +1013,7 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>>  	void *mem_priv;
>>  	unsigned int plane;
>>  	int ret = 0;
>> -	bool reacquired = vb->planes[0].mem_priv == NULL;
>> +	bool called_cleanup = false;
>>
>>  	memset(planes, 0, sizeof(planes[0]) * vb->num_planes);
>>  	/* Copy relevant information provided by the userspace */
>> @@ -1023,15 +1023,6 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>>  		return ret;
>>
>>  	for (plane = 0; plane < vb->num_planes; ++plane) {
>> -		/* Skip the plane if already verified */
>> -		if (vb->planes[plane].m.userptr &&
>> -			vb->planes[plane].m.userptr == planes[plane].m.userptr
>> -			&& vb->planes[plane].length == planes[plane].length)
>> -			continue;
>> -
>> -		dprintk(3, "userspace address for plane %d changed, reacquiring memory\n",
>> -			plane);
>> -
>>  		/* Check if the provided plane buffer is large enough */
>>  		if (planes[plane].length < vb->planes[plane].min_length) {
>>  			dprintk(1, "provided buffer size %u is less than setup size %u for plane %d\n",
>> @@ -1044,8 +1035,8 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>>
>>  		/* Release previously acquired memory if present */
>>  		if (vb->planes[plane].mem_priv) {
>> -			if (!reacquired) {
>> -				reacquired = true;
>> +			if (!called_cleanup) {
>> +				called_cleanup = true;
>>  				vb->copied_timestamp = 0;
>>  				call_void_vb_qop(vb, buf_cleanup, vb);
>>  			}
> 
> Could we do this unconditionally before the loop ?
> 
>> @@ -1083,17 +1074,14 @@ static int __prepare_userptr(struct vb2_buffer *vb)
>>  		vb->planes[plane].data_offset = planes[plane].data_offset;
>>  	}
>>
>> -	if (reacquired) {
>> -		/*
>> -		 * One or more planes changed, so we must call buf_init to do
>> -		 * the driver-specific initialization on the newly acquired
>> -		 * buffer, if provided.
>> -		 */
>> -		ret = call_vb_qop(vb, buf_init, vb);
>> -		if (ret) {
>> -			dprintk(1, "buffer initialization failed\n");
>> -			goto err;
>> -		}
>> +	/*
>> +	 * Call buf_init to do the driver-specific initialization on
>> +	 * the newly acquired buffer.
>> +	 */
>> +	ret = call_vb_qop(vb, buf_init, vb);
>> +	if (ret) {
>> +		dprintk(1, "buffer initialization failed\n");
>> +		goto err;
>>  	}
>>
>>  	ret = call_vb_qop(vb, buf_prepare, vb);
>
  

Hi Hans,

On 2019-06-07 14:01, Hans Verkuil wrote:
> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>> Hi Hans,
>>
>> Thank you for the patch.
>>
>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>> The __prepare_userptr() function made the incorrect assumption that if the
>>> same user pointer was used as the last one for which memory was acquired, then
>>> there was no need to re-acquire the memory. This assumption was never properly
>>> tested, and after doing that it became clear that this was in fact wrong.
>> Could you explain in the commit message why the assumption is not
>> correct ?
> You can free the memory, then allocate it again and you can get the same pointer,
> even though it is not necessarily using the same physical pages for the memory
> that the kernel is still using for it.
>
> Worse, you can free the memory, then allocate only half the memory you need and
> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> the original mapping still remains), but this can corrupt userspace memory
> causing the application to crash. It's not quite clear to me how the memory can
> get corrupted. I don't know enough of those low-level mm internals to understand
> the sequence of events.
>
> I have test code for v4l2-compliance available if someone wants to test this.

I'm interested, I would really like to know what happens in the mm 
subsystem in such case.


Best regards
  

On Fri, Jun 7, 2019 at 9:01 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > Hi Hans,
> >
> > Thank you for the patch.
> >
> > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >> The __prepare_userptr() function made the incorrect assumption that if the
> >> same user pointer was used as the last one for which memory was acquired, then
> >> there was no need to re-acquire the memory. This assumption was never properly
> >> tested, and after doing that it became clear that this was in fact wrong.
> >
> > Could you explain in the commit message why the assumption is not
> > correct ?
>
> You can free the memory, then allocate it again and you can get the same pointer,
> even though it is not necessarily using the same physical pages for the memory
> that the kernel is still using for it.
>
> Worse, you can free the memory, then allocate only half the memory you need and
> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> the original mapping still remains), but this can corrupt userspace memory
> causing the application to crash. It's not quite clear to me how the memory can
> get corrupted. I don't know enough of those low-level mm internals to understand
> the sequence of events.

Chrome specifically didn't keep the mapping between user pointers and
indexes, so it the cache just missed every time. What we noticed was
the put_userptr on the previous userptr at the index being unmapped
apparently caused that memory (often already returned back to the
application) to be corrupted... But we didn't get to the bottom of it
either, as we didn't have any MM expert look at the issue.

The free and realloc scenario just came to my mind when trying to
recall our original problem earlier today.

Best regards,
Tomasz
  

On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> Hi Hans,
> 
> On 2019-06-07 14:01, Hans Verkuil wrote:
>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>> Hi Hans,
>>>
>>> Thank you for the patch.
>>>
>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>> same user pointer was used as the last one for which memory was acquired, then
>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>> tested, and after doing that it became clear that this was in fact wrong.
>>> Could you explain in the commit message why the assumption is not
>>> correct ?
>> You can free the memory, then allocate it again and you can get the same pointer,
>> even though it is not necessarily using the same physical pages for the memory
>> that the kernel is still using for it.
>>
>> Worse, you can free the memory, then allocate only half the memory you need and
>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>> the original mapping still remains), but this can corrupt userspace memory
>> causing the application to crash. It's not quite clear to me how the memory can
>> get corrupted. I don't know enough of those low-level mm internals to understand
>> the sequence of events.
>>
>> I have test code for v4l2-compliance available if someone wants to test this.
> 
> I'm interested, I would really like to know what happens in the mm 
> subsystem in such case.

Here it is:

diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
index be606e48..9abf41da 100644
--- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
+++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
@@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
 	return 0;
 }

-static int captureBufs(struct node *node, const cv4l_queue &q,
+static int captureBufs(struct node *node, cv4l_queue &q,
 		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
 		unsigned &capture_count)
 {
@@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
 				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
 				buf.s_request_fd(buf_req_fds[req_idx]);
 			}
+			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
+				printf("\nidx: %d", buf.g_index());
+				for (unsigned p = 0; p < q.g_num_planes(); p++) {
+					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
+					fflush(stdout);
+					free(buf.g_userptr(p));
+					void *m = calloc(1, q.g_length(p)/2);
+
+					fail_on_test(m == NULL);
+					q.s_userptr(buf.g_index(), p, m);
+					printf("new buf[%d]: %p", p, m);
+					buf.s_userptr(m, p);
+				}
+				printf("\n");
+			}
 			fail_on_test(buf.qbuf(node, q));
 			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
 			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {



Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:

...
Streaming ioctls:
        test read/write: OK
        test blocking wait: OK
        test MMAP (no poll): OK
        test MMAP (select): OK
        test MMAP (epoll): OK
        Video Capture: Frame #000
idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
        Video Capture: Frame #001
idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
        Video Capture: Frame #002
idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
Aborted

Regards,

	Hans
  

On 6/7/19 2:20 PM, Tomasz Figa wrote:
> On Fri, Jun 7, 2019 at 9:01 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>> Hi Hans,
>>>
>>> Thank you for the patch.
>>>
>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>> same user pointer was used as the last one for which memory was acquired, then
>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>
>>> Could you explain in the commit message why the assumption is not
>>> correct ?
>>
>> You can free the memory, then allocate it again and you can get the same pointer,
>> even though it is not necessarily using the same physical pages for the memory
>> that the kernel is still using for it.
>>
>> Worse, you can free the memory, then allocate only half the memory you need and
>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>> the original mapping still remains), but this can corrupt userspace memory
>> causing the application to crash. It's not quite clear to me how the memory can
>> get corrupted. I don't know enough of those low-level mm internals to understand
>> the sequence of events.
> 
> Chrome specifically didn't keep the mapping between user pointers and
> indexes, so it the cache just missed every time. What we noticed was
> the put_userptr on the previous userptr at the index being unmapped
> apparently caused that memory (often already returned back to the
> application) to be corrupted... But we didn't get to the bottom of it
> either, as we didn't have any MM expert look at the issue.

I think this patch needs a bit more work. The put_userptr should happen
before the buffer is dequeued to userspace, not when queuing a new buffer.

I'll make a v2.

Regards,

	Hans

> 
> The free and realloc scenario just came to my mind when trying to
> recall our original problem earlier today.
> 
> Best regards,
> Tomasz
>
  

On 6/7/19 2:23 PM, Hans Verkuil wrote:
> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>> Hi Hans,
>>
>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>> Hi Hans,
>>>>
>>>> Thank you for the patch.
>>>>
>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>> Could you explain in the commit message why the assumption is not
>>>> correct ?
>>> You can free the memory, then allocate it again and you can get the same pointer,
>>> even though it is not necessarily using the same physical pages for the memory
>>> that the kernel is still using for it.
>>>
>>> Worse, you can free the memory, then allocate only half the memory you need and
>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>> the original mapping still remains), but this can corrupt userspace memory
>>> causing the application to crash. It's not quite clear to me how the memory can
>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>> the sequence of events.
>>>
>>> I have test code for v4l2-compliance available if someone wants to test this.
>>
>> I'm interested, I would really like to know what happens in the mm 
>> subsystem in such case.
> 
> Here it is:
> 
> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> index be606e48..9abf41da 100644
> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>  	return 0;
>  }
> 
> -static int captureBufs(struct node *node, const cv4l_queue &q,
> +static int captureBufs(struct node *node, cv4l_queue &q,
>  		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>  		unsigned &capture_count)
>  {
> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>  				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>  				buf.s_request_fd(buf_req_fds[req_idx]);
>  			}
> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> +				printf("\nidx: %d", buf.g_index());
> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> +					fflush(stdout);
> +					free(buf.g_userptr(p));
> +					void *m = calloc(1, q.g_length(p)/2);
> +
> +					fail_on_test(m == NULL);
> +					q.s_userptr(buf.g_index(), p, m);
> +					printf("new buf[%d]: %p", p, m);
> +					buf.s_userptr(m, p);
> +				}
> +				printf("\n");
> +			}
>  			fail_on_test(buf.qbuf(node, q));
>  			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>  			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> 
> 
> 
> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> 
> ...
> Streaming ioctls:
>         test read/write: OK
>         test blocking wait: OK
>         test MMAP (no poll): OK
>         test MMAP (select): OK
>         test MMAP (epoll): OK
>         Video Capture: Frame #000
> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>         Video Capture: Frame #001
> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>         Video Capture: Frame #002
> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> Aborted

To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
then streaming starts and captureBufs is called which basically just calls dqbuf
and qbuf.

Tomasz pointed out that all the pointers in this log are actually different. That's
correct, but here is a log where the old and new buf ptr are the same:

Streaming ioctls:
        test read/write: OK
        test blocking wait: OK
        test MMAP (no poll): OK
        test MMAP (select): OK
        test MMAP (epoll): OK
        Video Capture: Frame #000
idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
        Video Capture: Frame #001
idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
        Video Capture: Frame #002
idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
        Video Capture: Frame #003
idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
Aborted

It's weird that the first log fails that way: if the pointers are different,
then vb2 will call get_userptr and it should discover that the buffer isn't
large enough, causing qbuf to fail. That doesn't seem to happen.

Regards,

	Hans
  

On 6/7/19 2:47 PM, Hans Verkuil wrote:
> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>> Hi Hans,
>>>
>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>> Hi Hans,
>>>>>
>>>>> Thank you for the patch.
>>>>>
>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>> Could you explain in the commit message why the assumption is not
>>>>> correct ?
>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>> even though it is not necessarily using the same physical pages for the memory
>>>> that the kernel is still using for it.
>>>>
>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>> the original mapping still remains), but this can corrupt userspace memory
>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>> the sequence of events.
>>>>
>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>
>>> I'm interested, I would really like to know what happens in the mm 
>>> subsystem in such case.
>>
>> Here it is:
>>
>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>> index be606e48..9abf41da 100644
>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>  	return 0;
>>  }
>>
>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>  		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>  		unsigned &capture_count)
>>  {
>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>  				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>  				buf.s_request_fd(buf_req_fds[req_idx]);
>>  			}
>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>> +				printf("\nidx: %d", buf.g_index());
>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>> +					fflush(stdout);
>> +					free(buf.g_userptr(p));
>> +					void *m = calloc(1, q.g_length(p)/2);
>> +
>> +					fail_on_test(m == NULL);
>> +					q.s_userptr(buf.g_index(), p, m);
>> +					printf("new buf[%d]: %p", p, m);
>> +					buf.s_userptr(m, p);
>> +				}
>> +				printf("\n");
>> +			}
>>  			fail_on_test(buf.qbuf(node, q));
>>  			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>  			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>
>>
>>
>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>
>> ...
>> Streaming ioctls:
>>         test read/write: OK
>>         test blocking wait: OK
>>         test MMAP (no poll): OK
>>         test MMAP (select): OK
>>         test MMAP (epoll): OK
>>         Video Capture: Frame #000
>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>         Video Capture: Frame #001
>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>         Video Capture: Frame #002
>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>> Aborted
> 
> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> then streaming starts and captureBufs is called which basically just calls dqbuf
> and qbuf.
> 
> Tomasz pointed out that all the pointers in this log are actually different. That's
> correct, but here is a log where the old and new buf ptr are the same:
> 
> Streaming ioctls:
>         test read/write: OK
>         test blocking wait: OK
>         test MMAP (no poll): OK
>         test MMAP (select): OK
>         test MMAP (epoll): OK
>         Video Capture: Frame #000
> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>         Video Capture: Frame #001
> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>         Video Capture: Frame #002
> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>         Video Capture: Frame #003
> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> Aborted
> 
> It's weird that the first log fails that way: if the pointers are different,
> then vb2 will call get_userptr and it should discover that the buffer isn't
> large enough, causing qbuf to fail. That doesn't seem to happen.

I think that the reason for this corruption is that the memory pool used
by glibc is now large enough for vb2 to think it can map the full length
of the user pointer into memory, even though only the first half is actually
from the buffer that's allocated. When you capture a frame you just overwrite
a random part of the application's memory pool, causing this invalid pointer.

But that's a matter of garbage in, garbage out. So that's not the issue here.

The real question is what happens when you free the old buffer, allocate a
new buffer, end up with the same userptr, but it's using one or more different
pages for its memory compared to the mapping that the kernel uses.

I managed to reproduce this with v4l2-ctl:

diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
index 28b2b3b9..8f2ed9b5 100644
--- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
+++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
@@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
 		 * has the size that fits the old resolution and might not
 		 * fit to the new one.
 		 */
+		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
+			printf("\nidx: %d", buf.g_index());
+			for (unsigned p = 0; p < q.g_num_planes(); p++) {
+				unsigned *pb = (unsigned *)buf.g_userptr(p);
+				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
+				fflush(stdout);
+				free(buf.g_userptr(p));
+				void *m = calloc(1, q.g_length(p));
+
+				if (m == NULL)
+					return QUEUE_ERROR;
+				q.s_userptr(buf.g_index(), p, m);
+				if (m == buf.g_userptr(p))
+					printf(" identical new buf");
+				buf.s_userptr(m, p);
+			}
+			printf("\n");
+		}
 		if (fd.qbuf(buf) && errno != EINVAL) {
 			fprintf(stderr, "%s: qbuf error\n", __func__);
 			return QUEUE_ERROR;


Load vivid, setup a pure white test pattern:

v4l2-ctl -c test_pattern=6

Now run v4l2-ctl --stream-user and you'll see:

idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
<
idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
<
idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
<
idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
<
idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
<
idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
< 5.00 fps

idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
<
idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf

The first four dequeued buffers are filled with data, after that the
returned buffer is empty because vivid is actually writing to different
memory pages.

With this patch the first pixel is always non-zero.

I wonder if it isn't possible to just check the physical address of
the received user pointer with the physical address of the previous
user pointer. Or something like that. I'll dig around a bit more.

Regards,

	Hans
  

On Fri, Jun 7, 2019 at 10:41 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>> Hi Hans,
> >>>
> >>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>> Hi Hans,
> >>>>>
> >>>>> Thank you for the patch.
> >>>>>
> >>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>> Could you explain in the commit message why the assumption is not
> >>>>> correct ?
> >>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>> even though it is not necessarily using the same physical pages for the memory
> >>>> that the kernel is still using for it.
> >>>>
> >>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>> the original mapping still remains), but this can corrupt userspace memory
> >>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>> the sequence of events.
> >>>>
> >>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>
> >>> I'm interested, I would really like to know what happens in the mm
> >>> subsystem in such case.
> >>
> >> Here it is:
> >>
> >> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >> index be606e48..9abf41da 100644
> >> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>      return 0;
> >>  }
> >>
> >> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>              const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>              unsigned &capture_count)
> >>  {
> >> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>                              buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>                              buf.s_request_fd(buf_req_fds[req_idx]);
> >>                      }
> >> +                    if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >> +                            printf("\nidx: %d", buf.g_index());
> >> +                            for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >> +                                    printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >> +                                    fflush(stdout);
> >> +                                    free(buf.g_userptr(p));
> >> +                                    void *m = calloc(1, q.g_length(p)/2);
> >> +
> >> +                                    fail_on_test(m == NULL);
> >> +                                    q.s_userptr(buf.g_index(), p, m);
> >> +                                    printf("new buf[%d]: %p", p, m);
> >> +                                    buf.s_userptr(m, p);
> >> +                            }
> >> +                            printf("\n");
> >> +                    }
> >>                      fail_on_test(buf.qbuf(node, q));
> >>                      fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>                      if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>
> >>
> >>
> >> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>
> >> ...
> >> Streaming ioctls:
> >>         test read/write: OK
> >>         test blocking wait: OK
> >>         test MMAP (no poll): OK
> >>         test MMAP (select): OK
> >>         test MMAP (epoll): OK
> >>         Video Capture: Frame #000
> >> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>         Video Capture: Frame #001
> >> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>         Video Capture: Frame #002
> >> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >> Aborted
> >
> > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > then streaming starts and captureBufs is called which basically just calls dqbuf
> > and qbuf.
> >
> > Tomasz pointed out that all the pointers in this log are actually different. That's
> > correct, but here is a log where the old and new buf ptr are the same:
> >
> > Streaming ioctls:
> >         test read/write: OK
> >         test blocking wait: OK
> >         test MMAP (no poll): OK
> >         test MMAP (select): OK
> >         test MMAP (epoll): OK
> >         Video Capture: Frame #000
> > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >         Video Capture: Frame #001
> > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >         Video Capture: Frame #002
> > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >         Video Capture: Frame #003
> > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > Aborted
> >
> > It's weird that the first log fails that way: if the pointers are different,
> > then vb2 will call get_userptr and it should discover that the buffer isn't
> > large enough, causing qbuf to fail. That doesn't seem to happen.
>
> I think that the reason for this corruption is that the memory pool used
> by glibc is now large enough for vb2 to think it can map the full length
> of the user pointer into memory, even though only the first half is actually
> from the buffer that's allocated. When you capture a frame you just overwrite
> a random part of the application's memory pool, causing this invalid pointer.
>
> But that's a matter of garbage in, garbage out. So that's not the issue here.
>
> The real question is what happens when you free the old buffer, allocate a
> new buffer, end up with the same userptr, but it's using one or more different
> pages for its memory compared to the mapping that the kernel uses.
>
> I managed to reproduce this with v4l2-ctl:
>
> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> index 28b2b3b9..8f2ed9b5 100644
> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>                  * has the size that fits the old resolution and might not
>                  * fit to the new one.
>                  */
> +               if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> +                       printf("\nidx: %d", buf.g_index());
> +                       for (unsigned p = 0; p < q.g_num_planes(); p++) {
> +                               unsigned *pb = (unsigned *)buf.g_userptr(p);
> +                               printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> +                               fflush(stdout);
> +                               free(buf.g_userptr(p));
> +                               void *m = calloc(1, q.g_length(p));
> +
> +                               if (m == NULL)
> +                                       return QUEUE_ERROR;
> +                               q.s_userptr(buf.g_index(), p, m);
> +                               if (m == buf.g_userptr(p))
> +                                       printf(" identical new buf");
> +                               buf.s_userptr(m, p);
> +                       }
> +                       printf("\n");
> +               }
>                 if (fd.qbuf(buf) && errno != EINVAL) {
>                         fprintf(stderr, "%s: qbuf error\n", __func__);
>                         return QUEUE_ERROR;
>
>
> Load vivid, setup a pure white test pattern:
>
> v4l2-ctl -c test_pattern=6
>
> Now run v4l2-ctl --stream-user and you'll see:
>
> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> <
> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> < 5.00 fps
>
> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> <
> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>
> The first four dequeued buffers are filled with data, after that the
> returned buffer is empty because vivid is actually writing to different
> memory pages.
>
> With this patch the first pixel is always non-zero.
>
> I wonder if it isn't possible to just check the physical address of
> the received user pointer with the physical address of the previous
> user pointer. Or something like that. I'll dig around a bit more.

Hmm, this still wouldn't work, because the first page could be kept by
the userspace, but the rest reallocated. In practice one would need to
check all the pages, so basically the full get_user_pages would have
to be done and the resulting frame vector compared with the old one.
  

Hi Hans,

On 2019-06-07 15:40, Hans Verkuil wrote:
> On 6/7/19 2:47 PM, Hans Verkuil wrote:
>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>>> Thank you for the patch.
>>>>>>
>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>>> Could you explain in the commit message why the assumption is not
>>>>>> correct ?
>>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>>> even though it is not necessarily using the same physical pages for the memory
>>>>> that the kernel is still using for it.
>>>>>
>>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>>> the original mapping still remains), but this can corrupt userspace memory
>>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>>> the sequence of events.
>>>>>
>>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>> I'm interested, I would really like to know what happens in the mm
>>>> subsystem in such case.
>>> Here it is:
>>>
>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>> index be606e48..9abf41da 100644
>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>>   	return 0;
>>>   }
>>>
>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>>   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>>   		unsigned &capture_count)
>>>   {
>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>>   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>>   				buf.s_request_fd(buf_req_fds[req_idx]);
>>>   			}
>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>>> +				printf("\nidx: %d", buf.g_index());
>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>>> +					fflush(stdout);
>>> +					free(buf.g_userptr(p));
>>> +					void *m = calloc(1, q.g_length(p)/2);
>>> +
>>> +					fail_on_test(m == NULL);
>>> +					q.s_userptr(buf.g_index(), p, m);
>>> +					printf("new buf[%d]: %p", p, m);
>>> +					buf.s_userptr(m, p);
>>> +				}
>>> +				printf("\n");
>>> +			}
>>>   			fail_on_test(buf.qbuf(node, q));
>>>   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>>   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>>
>>>
>>>
>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>>
>>> ...
>>> Streaming ioctls:
>>>          test read/write: OK
>>>          test blocking wait: OK
>>>          test MMAP (no poll): OK
>>>          test MMAP (select): OK
>>>          test MMAP (epoll): OK
>>>          Video Capture: Frame #000
>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>>          Video Capture: Frame #001
>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>>          Video Capture: Frame #002
>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>>> Aborted
>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
>> then streaming starts and captureBufs is called which basically just calls dqbuf
>> and qbuf.
>>
>> Tomasz pointed out that all the pointers in this log are actually different. That's
>> correct, but here is a log where the old and new buf ptr are the same:
>>
>> Streaming ioctls:
>>          test read/write: OK
>>          test blocking wait: OK
>>          test MMAP (no poll): OK
>>          test MMAP (select): OK
>>          test MMAP (epoll): OK
>>          Video Capture: Frame #000
>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>>          Video Capture: Frame #001
>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>>          Video Capture: Frame #002
>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>>          Video Capture: Frame #003
>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
>> Aborted
>>
>> It's weird that the first log fails that way: if the pointers are different,
>> then vb2 will call get_userptr and it should discover that the buffer isn't
>> large enough, causing qbuf to fail. That doesn't seem to happen.
> I think that the reason for this corruption is that the memory pool used
> by glibc is now large enough for vb2 to think it can map the full length
> of the user pointer into memory, even though only the first half is actually
> from the buffer that's allocated. When you capture a frame you just overwrite
> a random part of the application's memory pool, causing this invalid pointer.
>
> But that's a matter of garbage in, garbage out. So that's not the issue here.
>
> The real question is what happens when you free the old buffer, allocate a
> new buffer, end up with the same userptr, but it's using one or more different
> pages for its memory compared to the mapping that the kernel uses.
>
> I managed to reproduce this with v4l2-ctl:
>
> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> index 28b2b3b9..8f2ed9b5 100644
> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>   		 * has the size that fits the old resolution and might not
>   		 * fit to the new one.
>   		 */
> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> +			printf("\nidx: %d", buf.g_index());
> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> +				fflush(stdout);
> +				free(buf.g_userptr(p));
> +				void *m = calloc(1, q.g_length(p));
> +
> +				if (m == NULL)
> +					return QUEUE_ERROR;
> +				q.s_userptr(buf.g_index(), p, m);
> +				if (m == buf.g_userptr(p))
> +					printf(" identical new buf");
> +				buf.s_userptr(m, p);
> +			}
> +			printf("\n");
> +		}
>   		if (fd.qbuf(buf) && errno != EINVAL) {
>   			fprintf(stderr, "%s: qbuf error\n", __func__);
>   			return QUEUE_ERROR;
>
>
> Load vivid, setup a pure white test pattern:
>
> v4l2-ctl -c test_pattern=6
>
> Now run v4l2-ctl --stream-user and you'll see:
>
> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> <
> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> <
> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> < 5.00 fps
>
> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> <
> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>
> The first four dequeued buffers are filled with data, after that the
> returned buffer is empty because vivid is actually writing to different
> memory pages.
>
> With this patch the first pixel is always non-zero.

Good catch. The question is weather we treat that as undefined behavior 
and keep the optimization for 'good applications' or assume that every 
broken userspace code has to be properly handled. The good thing is that 
there is still imho no security issue. The physical pages gathered by 
vb2 in worst case belongs to noone else (vb2 is their last user, they 
are not yet returned to free pages pool).

> I wonder if it isn't possible to just check the physical address of
> the received user pointer with the physical address of the previous
> user pointer. Or something like that. I'll dig around a bit more.

Such check won't be so simple. Pages contiguous in the virtual memory 
won't map to pages contiguous in the physical memory, so you would need 
to check every single memory page. Make no sense. It is better to 
reacquire buffer on every queue operation. This indeed show how broken 
the USERPTR related part of v4l2 API is.


Best regards
  

Hi Marek,

On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> On 2019-06-07 15:40, Hans Verkuil wrote:
> > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> >> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>>> Thank you for the patch.
> >>>>>>
> >>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>>> Could you explain in the commit message why the assumption is not
> >>>>>> correct ?
> >>>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>>> even though it is not necessarily using the same physical pages for the memory
> >>>>> that the kernel is still using for it.
> >>>>>
> >>>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>>> the original mapping still remains), but this can corrupt userspace memory
> >>>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>>> the sequence of events.
> >>>>>
> >>>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>> I'm interested, I would really like to know what happens in the mm
> >>>> subsystem in such case.
> >>> Here it is:
> >>>
> >>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>> index be606e48..9abf41da 100644
> >>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>>   	return 0;
> >>>   }
> >>>
> >>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >>> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>>   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>>   		unsigned &capture_count)
> >>>   {
> >>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>>   				buf.s_request_fd(buf_req_fds[req_idx]);
> >>>   			}
> >>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>> +				printf("\nidx: %d", buf.g_index());
> >>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >>> +					fflush(stdout);
> >>> +					free(buf.g_userptr(p));
> >>> +					void *m = calloc(1, q.g_length(p)/2);
> >>> +
> >>> +					fail_on_test(m == NULL);
> >>> +					q.s_userptr(buf.g_index(), p, m);
> >>> +					printf("new buf[%d]: %p", p, m);
> >>> +					buf.s_userptr(m, p);
> >>> +				}
> >>> +				printf("\n");
> >>> +			}
> >>>   			fail_on_test(buf.qbuf(node, q));
> >>>   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>>   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>>
> >>>
> >>>
> >>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>>
> >>> ...
> >>> Streaming ioctls:
> >>>          test read/write: OK
> >>>          test blocking wait: OK
> >>>          test MMAP (no poll): OK
> >>>          test MMAP (select): OK
> >>>          test MMAP (epoll): OK
> >>>          Video Capture: Frame #000
> >>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>>          Video Capture: Frame #001
> >>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>>          Video Capture: Frame #002
> >>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >>> Aborted
> >> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> >> then streaming starts and captureBufs is called which basically just calls dqbuf
> >> and qbuf.
> >>
> >> Tomasz pointed out that all the pointers in this log are actually different. That's
> >> correct, but here is a log where the old and new buf ptr are the same:
> >>
> >> Streaming ioctls:
> >>          test read/write: OK
> >>          test blocking wait: OK
> >>          test MMAP (no poll): OK
> >>          test MMAP (select): OK
> >>          test MMAP (epoll): OK
> >>          Video Capture: Frame #000
> >> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >>          Video Capture: Frame #001
> >> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >>          Video Capture: Frame #002
> >> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >>          Video Capture: Frame #003
> >> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> >> Aborted
> >>
> >> It's weird that the first log fails that way: if the pointers are different,
> >> then vb2 will call get_userptr and it should discover that the buffer isn't
> >> large enough, causing qbuf to fail. That doesn't seem to happen.
> > I think that the reason for this corruption is that the memory pool used
> > by glibc is now large enough for vb2 to think it can map the full length
> > of the user pointer into memory, even though only the first half is actually
> > from the buffer that's allocated. When you capture a frame you just overwrite
> > a random part of the application's memory pool, causing this invalid pointer.
> >
> > But that's a matter of garbage in, garbage out. So that's not the issue here.
> >
> > The real question is what happens when you free the old buffer, allocate a
> > new buffer, end up with the same userptr, but it's using one or more different
> > pages for its memory compared to the mapping that the kernel uses.
> >
> > I managed to reproduce this with v4l2-ctl:
> >
> > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > index 28b2b3b9..8f2ed9b5 100644
> > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> >   		 * has the size that fits the old resolution and might not
> >   		 * fit to the new one.
> >   		 */
> > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > +			printf("\nidx: %d", buf.g_index());
> > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > +				fflush(stdout);
> > +				free(buf.g_userptr(p));
> > +				void *m = calloc(1, q.g_length(p));
> > +
> > +				if (m == NULL)
> > +					return QUEUE_ERROR;
> > +				q.s_userptr(buf.g_index(), p, m);
> > +				if (m == buf.g_userptr(p))
> > +					printf(" identical new buf");
> > +				buf.s_userptr(m, p);
> > +			}
> > +			printf("\n");
> > +		}
> >   		if (fd.qbuf(buf) && errno != EINVAL) {
> >   			fprintf(stderr, "%s: qbuf error\n", __func__);
> >   			return QUEUE_ERROR;
> >
> >
> > Load vivid, setup a pure white test pattern:
> >
> > v4l2-ctl -c test_pattern=6
> >
> > Now run v4l2-ctl --stream-user and you'll see:
> >
> > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > <
> > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > <
> > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > <
> > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > <
> > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > <
> > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > < 5.00 fps
> >
> > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > <
> > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> >
> > The first four dequeued buffers are filled with data, after that the
> > returned buffer is empty because vivid is actually writing to different
> > memory pages.
> >
> > With this patch the first pixel is always non-zero.
> 
> Good catch. The question is weather we treat that as undefined behavior 
> and keep the optimization for 'good applications' or assume that every 
> broken userspace code has to be properly handled.

Given how long we've been saying that USERPTR should be replaced by
DMABUF, I would consider that any userspace code using USERPTR is broken
:-) One could however question whether we were effective at getting that
message across...

> The good thing is that 
> there is still imho no security issue. The physical pages gathered by 
> vb2 in worst case belongs to noone else (vb2 is their last user, they 
> are not yet returned to free pages pool).
> 
> > I wonder if it isn't possible to just check the physical address of
> > the received user pointer with the physical address of the previous
> > user pointer. Or something like that. I'll dig around a bit more.
> 
> Such check won't be so simple. Pages contiguous in the virtual memory 
> won't map to pages contiguous in the physical memory, so you would need 
> to check every single memory page. Make no sense. It is better to 
> reacquire buffer on every queue operation. This indeed show how broken 
> the USERPTR related part of v4l2 API is.
  

On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> Hi Hans,
> 
> On 2019-06-07 15:40, Hans Verkuil wrote:
>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>>>> Thank you for the patch.
>>>>>>>
>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>>>> Could you explain in the commit message why the assumption is not
>>>>>>> correct ?
>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>>>> even though it is not necessarily using the same physical pages for the memory
>>>>>> that the kernel is still using for it.
>>>>>>
>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>>>> the original mapping still remains), but this can corrupt userspace memory
>>>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>>>> the sequence of events.
>>>>>>
>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>>> I'm interested, I would really like to know what happens in the mm
>>>>> subsystem in such case.
>>>> Here it is:
>>>>
>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>> index be606e48..9abf41da 100644
>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>>>   	return 0;
>>>>   }
>>>>
>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>>>   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>>>   		unsigned &capture_count)
>>>>   {
>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>>>   				buf.s_request_fd(buf_req_fds[req_idx]);
>>>>   			}
>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>> +				printf("\nidx: %d", buf.g_index());
>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>>>> +					fflush(stdout);
>>>> +					free(buf.g_userptr(p));
>>>> +					void *m = calloc(1, q.g_length(p)/2);
>>>> +
>>>> +					fail_on_test(m == NULL);
>>>> +					q.s_userptr(buf.g_index(), p, m);
>>>> +					printf("new buf[%d]: %p", p, m);
>>>> +					buf.s_userptr(m, p);
>>>> +				}
>>>> +				printf("\n");
>>>> +			}
>>>>   			fail_on_test(buf.qbuf(node, q));
>>>>   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>>>   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>>>
>>>>
>>>>
>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>>>
>>>> ...
>>>> Streaming ioctls:
>>>>          test read/write: OK
>>>>          test blocking wait: OK
>>>>          test MMAP (no poll): OK
>>>>          test MMAP (select): OK
>>>>          test MMAP (epoll): OK
>>>>          Video Capture: Frame #000
>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>>>          Video Capture: Frame #001
>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>>>          Video Capture: Frame #002
>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>>>> Aborted
>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
>>> then streaming starts and captureBufs is called which basically just calls dqbuf
>>> and qbuf.
>>>
>>> Tomasz pointed out that all the pointers in this log are actually different. That's
>>> correct, but here is a log where the old and new buf ptr are the same:
>>>
>>> Streaming ioctls:
>>>          test read/write: OK
>>>          test blocking wait: OK
>>>          test MMAP (no poll): OK
>>>          test MMAP (select): OK
>>>          test MMAP (epoll): OK
>>>          Video Capture: Frame #000
>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>>>          Video Capture: Frame #001
>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>>>          Video Capture: Frame #002
>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>>>          Video Capture: Frame #003
>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
>>> Aborted
>>>
>>> It's weird that the first log fails that way: if the pointers are different,
>>> then vb2 will call get_userptr and it should discover that the buffer isn't
>>> large enough, causing qbuf to fail. That doesn't seem to happen.
>> I think that the reason for this corruption is that the memory pool used
>> by glibc is now large enough for vb2 to think it can map the full length
>> of the user pointer into memory, even though only the first half is actually
>> from the buffer that's allocated. When you capture a frame you just overwrite
>> a random part of the application's memory pool, causing this invalid pointer.
>>
>> But that's a matter of garbage in, garbage out. So that's not the issue here.
>>
>> The real question is what happens when you free the old buffer, allocate a
>> new buffer, end up with the same userptr, but it's using one or more different
>> pages for its memory compared to the mapping that the kernel uses.
>>
>> I managed to reproduce this with v4l2-ctl:
>>
>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>> index 28b2b3b9..8f2ed9b5 100644
>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>>   		 * has the size that fits the old resolution and might not
>>   		 * fit to the new one.
>>   		 */
>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
>> +			printf("\nidx: %d", buf.g_index());
>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
>> +				fflush(stdout);
>> +				free(buf.g_userptr(p));
>> +				void *m = calloc(1, q.g_length(p));
>> +
>> +				if (m == NULL)
>> +					return QUEUE_ERROR;
>> +				q.s_userptr(buf.g_index(), p, m);
>> +				if (m == buf.g_userptr(p))
>> +					printf(" identical new buf");
>> +				buf.s_userptr(m, p);
>> +			}
>> +			printf("\n");
>> +		}
>>   		if (fd.qbuf(buf) && errno != EINVAL) {
>>   			fprintf(stderr, "%s: qbuf error\n", __func__);
>>   			return QUEUE_ERROR;
>>
>>
>> Load vivid, setup a pure white test pattern:
>>
>> v4l2-ctl -c test_pattern=6
>>
>> Now run v4l2-ctl --stream-user and you'll see:
>>
>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
>> <
>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
>> <
>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
>> <
>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
>> <
>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
>> <
>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
>> < 5.00 fps
>>
>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
>> <
>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>>
>> The first four dequeued buffers are filled with data, after that the
>> returned buffer is empty because vivid is actually writing to different
>> memory pages.
>>
>> With this patch the first pixel is always non-zero.
> 
> Good catch. The question is weather we treat that as undefined behavior 
> and keep the optimization for 'good applications' or assume that every 
> broken userspace code has to be properly handled. The good thing is that 
> there is still imho no security issue. The physical pages gathered by 

Yeah, that scared me for a bit, but it all looks secure.

> vb2 in worst case belongs to noone else (vb2 is their last user, they 
> are not yet returned to free pages pool).

I see three options:

1) just always reacquire the buffer, and if anyone complains about it
   being slower we point them towards DMABUF.

2) keep the current behavior, but document it.

3) as 2), but also add a new buffer flag that forces a reacquire of the
   buffer. This could be valid for DMABUF as well. E.g.:

   V4L2_BUF_FLAG_REACQUIRE

I'm leaning towards the third option since it won't slow down existing
implementations, yet if you do change the userptr every time, then you
can now force this to work safely.

>> I wonder if it isn't possible to just check the physical address of
>> the received user pointer with the physical address of the previous
>> user pointer. Or something like that. I'll dig around a bit more.
> 
> Such check won't be so simple. Pages contiguous in the virtual memory 
> won't map to pages contiguous in the physical memory, so you would need 
> to check every single memory page. Make no sense. It is better to 
> reacquire buffer on every queue operation. This indeed show how broken 
> the USERPTR related part of v4l2 API is.

OK, good to know. Then I'm not going to spend time on that.

Regards,

	Hans
  

On Fri, Jun 7, 2019 at 11:11 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> > Hi Hans,
> >
> > On 2019-06-07 15:40, Hans Verkuil wrote:
> >> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> >>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>>>> Thank you for the patch.
> >>>>>>>
> >>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>>>> Could you explain in the commit message why the assumption is not
> >>>>>>> correct ?
> >>>>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>>>> even though it is not necessarily using the same physical pages for the memory
> >>>>>> that the kernel is still using for it.
> >>>>>>
> >>>>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>>>> the original mapping still remains), but this can corrupt userspace memory
> >>>>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>>>> the sequence of events.
> >>>>>>
> >>>>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>>> I'm interested, I would really like to know what happens in the mm
> >>>>> subsystem in such case.
> >>>> Here it is:
> >>>>
> >>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>> index be606e48..9abf41da 100644
> >>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>>>    return 0;
> >>>>   }
> >>>>
> >>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>>>            const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>>>            unsigned &capture_count)
> >>>>   {
> >>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>                            buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>>>                            buf.s_request_fd(buf_req_fds[req_idx]);
> >>>>                    }
> >>>> +                  if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>>> +                          printf("\nidx: %d", buf.g_index());
> >>>> +                          for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>>> +                                  printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >>>> +                                  fflush(stdout);
> >>>> +                                  free(buf.g_userptr(p));
> >>>> +                                  void *m = calloc(1, q.g_length(p)/2);
> >>>> +
> >>>> +                                  fail_on_test(m == NULL);
> >>>> +                                  q.s_userptr(buf.g_index(), p, m);
> >>>> +                                  printf("new buf[%d]: %p", p, m);
> >>>> +                                  buf.s_userptr(m, p);
> >>>> +                          }
> >>>> +                          printf("\n");
> >>>> +                  }
> >>>>                    fail_on_test(buf.qbuf(node, q));
> >>>>                    fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>>>                    if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>>>
> >>>>
> >>>>
> >>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>>>
> >>>> ...
> >>>> Streaming ioctls:
> >>>>          test read/write: OK
> >>>>          test blocking wait: OK
> >>>>          test MMAP (no poll): OK
> >>>>          test MMAP (select): OK
> >>>>          test MMAP (epoll): OK
> >>>>          Video Capture: Frame #000
> >>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>>>          Video Capture: Frame #001
> >>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>>>          Video Capture: Frame #002
> >>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >>>> Aborted
> >>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> >>> then streaming starts and captureBufs is called which basically just calls dqbuf
> >>> and qbuf.
> >>>
> >>> Tomasz pointed out that all the pointers in this log are actually different. That's
> >>> correct, but here is a log where the old and new buf ptr are the same:
> >>>
> >>> Streaming ioctls:
> >>>          test read/write: OK
> >>>          test blocking wait: OK
> >>>          test MMAP (no poll): OK
> >>>          test MMAP (select): OK
> >>>          test MMAP (epoll): OK
> >>>          Video Capture: Frame #000
> >>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >>>          Video Capture: Frame #001
> >>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >>>          Video Capture: Frame #002
> >>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >>>          Video Capture: Frame #003
> >>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> >>> Aborted
> >>>
> >>> It's weird that the first log fails that way: if the pointers are different,
> >>> then vb2 will call get_userptr and it should discover that the buffer isn't
> >>> large enough, causing qbuf to fail. That doesn't seem to happen.
> >> I think that the reason for this corruption is that the memory pool used
> >> by glibc is now large enough for vb2 to think it can map the full length
> >> of the user pointer into memory, even though only the first half is actually
> >> from the buffer that's allocated. When you capture a frame you just overwrite
> >> a random part of the application's memory pool, causing this invalid pointer.
> >>
> >> But that's a matter of garbage in, garbage out. So that's not the issue here.
> >>
> >> The real question is what happens when you free the old buffer, allocate a
> >> new buffer, end up with the same userptr, but it's using one or more different
> >> pages for its memory compared to the mapping that the kernel uses.
> >>
> >> I managed to reproduce this with v4l2-ctl:
> >>
> >> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >> index 28b2b3b9..8f2ed9b5 100644
> >> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> >>               * has the size that fits the old resolution and might not
> >>               * fit to the new one.
> >>               */
> >> +            if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> >> +                    printf("\nidx: %d", buf.g_index());
> >> +                    for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >> +                            unsigned *pb = (unsigned *)buf.g_userptr(p);
> >> +                            printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> >> +                            fflush(stdout);
> >> +                            free(buf.g_userptr(p));
> >> +                            void *m = calloc(1, q.g_length(p));
> >> +
> >> +                            if (m == NULL)
> >> +                                    return QUEUE_ERROR;
> >> +                            q.s_userptr(buf.g_index(), p, m);
> >> +                            if (m == buf.g_userptr(p))
> >> +                                    printf(" identical new buf");
> >> +                            buf.s_userptr(m, p);
> >> +                    }
> >> +                    printf("\n");
> >> +            }
> >>              if (fd.qbuf(buf) && errno != EINVAL) {
> >>                      fprintf(stderr, "%s: qbuf error\n", __func__);
> >>                      return QUEUE_ERROR;
> >>
> >>
> >> Load vivid, setup a pure white test pattern:
> >>
> >> v4l2-ctl -c test_pattern=6
> >>
> >> Now run v4l2-ctl --stream-user and you'll see:
> >>
> >> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> >> <
> >> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> >> < 5.00 fps
> >>
> >> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> >> <
> >> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> >>
> >> The first four dequeued buffers are filled with data, after that the
> >> returned buffer is empty because vivid is actually writing to different
> >> memory pages.
> >>
> >> With this patch the first pixel is always non-zero.
> >
> > Good catch. The question is weather we treat that as undefined behavior
> > and keep the optimization for 'good applications' or assume that every
> > broken userspace code has to be properly handled. The good thing is that
> > there is still imho no security issue. The physical pages gathered by
>
> Yeah, that scared me for a bit, but it all looks secure.
>
> > vb2 in worst case belongs to noone else (vb2 is their last user, they
> > are not yet returned to free pages pool).
>
> I see three options:
>
> 1) just always reacquire the buffer, and if anyone complains about it
>    being slower we point them towards DMABUF.
>
> 2) keep the current behavior, but document it.
>
> 3) as 2), but also add a new buffer flag that forces a reacquire of the
>    buffer. This could be valid for DMABUF as well. E.g.:
>
>    V4L2_BUF_FLAG_REACQUIRE
>
> I'm leaning towards the third option since it won't slow down existing
> implementations, yet if you do change the userptr every time, then you
> can now force this to work safely.
>

I'd be for 1) or 3) as that would allow Chrome work on mainline.

Also I believe there is still some bug when the pointers don't match,
even if you don't free those pages. I guess some more testing that
includes verifying the contents of previously dequeued buffers could
show something.

> >> I wonder if it isn't possible to just check the physical address of
> >> the received user pointer with the physical address of the previous
> >> user pointer. Or something like that. I'll dig around a bit more.
> >
> > Such check won't be so simple. Pages contiguous in the virtual memory
> > won't map to pages contiguous in the physical memory, so you would need
> > to check every single memory page. Make no sense. It is better to
> > reacquire buffer on every queue operation. This indeed show how broken
> > the USERPTR related part of v4l2 API is.
>
> OK, good to know. Then I'm not going to spend time on that.
>
> Regards,
>
>         Hans
  

Hi Hans,

On 2019-06-07 16:11, Hans Verkuil wrote:
> On 6/7/19 3:55 PM, Marek Szyprowski wrote:
>> On 2019-06-07 15:40, Hans Verkuil wrote:
>>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
>>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>>>>> Thank you for the patch.
>>>>>>>>
>>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>>>>> Could you explain in the commit message why the assumption is not
>>>>>>>> correct ?
>>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>>>>> even though it is not necessarily using the same physical pages for the memory
>>>>>>> that the kernel is still using for it.
>>>>>>>
>>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>>>>> the original mapping still remains), but this can corrupt userspace memory
>>>>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>>>>> the sequence of events.
>>>>>>>
>>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>>>> I'm interested, I would really like to know what happens in the mm
>>>>>> subsystem in such case.
>>>>> Here it is:
>>>>>
>>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>> index be606e48..9abf41da 100644
>>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>>>>    	return 0;
>>>>>    }
>>>>>
>>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>>>>    		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>>>>    		unsigned &capture_count)
>>>>>    {
>>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>    				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>>>>    				buf.s_request_fd(buf_req_fds[req_idx]);
>>>>>    			}
>>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>>> +				printf("\nidx: %d", buf.g_index());
>>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>>>>> +					fflush(stdout);
>>>>> +					free(buf.g_userptr(p));
>>>>> +					void *m = calloc(1, q.g_length(p)/2);
>>>>> +
>>>>> +					fail_on_test(m == NULL);
>>>>> +					q.s_userptr(buf.g_index(), p, m);
>>>>> +					printf("new buf[%d]: %p", p, m);
>>>>> +					buf.s_userptr(m, p);
>>>>> +				}
>>>>> +				printf("\n");
>>>>> +			}
>>>>>    			fail_on_test(buf.qbuf(node, q));
>>>>>    			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>>>>    			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>>>>
>>>>>
>>>>>
>>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>>>>
>>>>> ...
>>>>> Streaming ioctls:
>>>>>           test read/write: OK
>>>>>           test blocking wait: OK
>>>>>           test MMAP (no poll): OK
>>>>>           test MMAP (select): OK
>>>>>           test MMAP (epoll): OK
>>>>>           Video Capture: Frame #000
>>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>>>>           Video Capture: Frame #001
>>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>>>>           Video Capture: Frame #002
>>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>>>>> Aborted
>>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
>>>> then streaming starts and captureBufs is called which basically just calls dqbuf
>>>> and qbuf.
>>>>
>>>> Tomasz pointed out that all the pointers in this log are actually different. That's
>>>> correct, but here is a log where the old and new buf ptr are the same:
>>>>
>>>> Streaming ioctls:
>>>>           test read/write: OK
>>>>           test blocking wait: OK
>>>>           test MMAP (no poll): OK
>>>>           test MMAP (select): OK
>>>>           test MMAP (epoll): OK
>>>>           Video Capture: Frame #000
>>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>>>>           Video Capture: Frame #001
>>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>>>>           Video Capture: Frame #002
>>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>>>>           Video Capture: Frame #003
>>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
>>>> Aborted
>>>>
>>>> It's weird that the first log fails that way: if the pointers are different,
>>>> then vb2 will call get_userptr and it should discover that the buffer isn't
>>>> large enough, causing qbuf to fail. That doesn't seem to happen.
>>> I think that the reason for this corruption is that the memory pool used
>>> by glibc is now large enough for vb2 to think it can map the full length
>>> of the user pointer into memory, even though only the first half is actually
>>> from the buffer that's allocated. When you capture a frame you just overwrite
>>> a random part of the application's memory pool, causing this invalid pointer.
>>>
>>> But that's a matter of garbage in, garbage out. So that's not the issue here.
>>>
>>> The real question is what happens when you free the old buffer, allocate a
>>> new buffer, end up with the same userptr, but it's using one or more different
>>> pages for its memory compared to the mapping that the kernel uses.
>>>
>>> I managed to reproduce this with v4l2-ctl:
>>>
>>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>> index 28b2b3b9..8f2ed9b5 100644
>>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>>>    		 * has the size that fits the old resolution and might not
>>>    		 * fit to the new one.
>>>    		 */
>>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
>>> +			printf("\nidx: %d", buf.g_index());
>>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
>>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
>>> +				fflush(stdout);
>>> +				free(buf.g_userptr(p));
>>> +				void *m = calloc(1, q.g_length(p));
>>> +
>>> +				if (m == NULL)
>>> +					return QUEUE_ERROR;
>>> +				q.s_userptr(buf.g_index(), p, m);
>>> +				if (m == buf.g_userptr(p))
>>> +					printf(" identical new buf");
>>> +				buf.s_userptr(m, p);
>>> +			}
>>> +			printf("\n");
>>> +		}
>>>    		if (fd.qbuf(buf) && errno != EINVAL) {
>>>    			fprintf(stderr, "%s: qbuf error\n", __func__);
>>>    			return QUEUE_ERROR;
>>>
>>>
>>> Load vivid, setup a pure white test pattern:
>>>
>>> v4l2-ctl -c test_pattern=6
>>>
>>> Now run v4l2-ctl --stream-user and you'll see:
>>>
>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
>>> <
>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
>>> <
>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
>>> <
>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
>>> <
>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
>>> <
>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
>>> < 5.00 fps
>>>
>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
>>> <
>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>>>
>>> The first four dequeued buffers are filled with data, after that the
>>> returned buffer is empty because vivid is actually writing to different
>>> memory pages.
>>>
>>> With this patch the first pixel is always non-zero.
>> Good catch. The question is weather we treat that as undefined behavior
>> and keep the optimization for 'good applications' or assume that every
>> broken userspace code has to be properly handled. The good thing is that
>> there is still imho no security issue. The physical pages gathered by
> Yeah, that scared me for a bit, but it all looks secure.
>
>> vb2 in worst case belongs to noone else (vb2 is their last user, they
>> are not yet returned to free pages pool).
> I see three options:
>
> 1) just always reacquire the buffer, and if anyone complains about it
>     being slower we point them towards DMABUF.
>
> 2) keep the current behavior, but document it.
>
> 3) as 2), but also add a new buffer flag that forces a reacquire of the
>     buffer. This could be valid for DMABUF as well. E.g.:
>
>     V4L2_BUF_FLAG_REACQUIRE
>
> I'm leaning towards the third option since it won't slow down existing
> implementations, yet if you do change the userptr every time, then you
> can now force this to work safely.

Is there are valid use case for third variant? I would rather go for second.

There is one more issue related to this. There are many apps which use 
either USERPTR or DMAbuf, but in a bit odd way: they use the same 
buffers all the time, but they ignore buf->index and never match it to 
respective buffer pointers or fds. This makes the current caching 
mechanism useless. Maybe it would make a bit sense do rewrite the 
caching in qbuf to ignore the provided buffer->index?

>>> I wonder if it isn't possible to just check the physical address of
>>> the received user pointer with the physical address of the previous
>>> user pointer. Or something like that. I'll dig around a bit more.
>> Such check won't be so simple. Pages contiguous in the virtual memory
>> won't map to pages contiguous in the physical memory, so you would need
>> to check every single memory page. Make no sense. It is better to
>> reacquire buffer on every queue operation. This indeed show how broken
>> the USERPTR related part of v4l2 API is.
> OK, good to know. Then I'm not going to spend time on that.
>
>
Best regards
  

Hi Hans,

On Fri, Jun 07, 2019 at 04:11:20PM +0200, Hans Verkuil wrote:
> On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> > Hi Hans,
> > 
> > On 2019-06-07 15:40, Hans Verkuil wrote:
> >> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> >>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>>>> Thank you for the patch.
> >>>>>>>
> >>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>>>> Could you explain in the commit message why the assumption is not
> >>>>>>> correct ?
> >>>>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>>>> even though it is not necessarily using the same physical pages for the memory
> >>>>>> that the kernel is still using for it.
> >>>>>>
> >>>>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>>>> the original mapping still remains), but this can corrupt userspace memory
> >>>>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>>>> the sequence of events.
> >>>>>>
> >>>>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>>> I'm interested, I would really like to know what happens in the mm
> >>>>> subsystem in such case.
> >>>> Here it is:
> >>>>
> >>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>> index be606e48..9abf41da 100644
> >>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>>>   	return 0;
> >>>>   }
> >>>>
> >>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>>>   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>>>   		unsigned &capture_count)
> >>>>   {
> >>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>>>   				buf.s_request_fd(buf_req_fds[req_idx]);
> >>>>   			}
> >>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>>> +				printf("\nidx: %d", buf.g_index());
> >>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >>>> +					fflush(stdout);
> >>>> +					free(buf.g_userptr(p));
> >>>> +					void *m = calloc(1, q.g_length(p)/2);
> >>>> +
> >>>> +					fail_on_test(m == NULL);
> >>>> +					q.s_userptr(buf.g_index(), p, m);
> >>>> +					printf("new buf[%d]: %p", p, m);
> >>>> +					buf.s_userptr(m, p);
> >>>> +				}
> >>>> +				printf("\n");
> >>>> +			}
> >>>>   			fail_on_test(buf.qbuf(node, q));
> >>>>   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>>>   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>>>
> >>>>
> >>>>
> >>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>>>
> >>>> ...
> >>>> Streaming ioctls:
> >>>>          test read/write: OK
> >>>>          test blocking wait: OK
> >>>>          test MMAP (no poll): OK
> >>>>          test MMAP (select): OK
> >>>>          test MMAP (epoll): OK
> >>>>          Video Capture: Frame #000
> >>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>>>          Video Capture: Frame #001
> >>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>>>          Video Capture: Frame #002
> >>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >>>> Aborted
> >>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> >>> then streaming starts and captureBufs is called which basically just calls dqbuf
> >>> and qbuf.
> >>>
> >>> Tomasz pointed out that all the pointers in this log are actually different. That's
> >>> correct, but here is a log where the old and new buf ptr are the same:
> >>>
> >>> Streaming ioctls:
> >>>          test read/write: OK
> >>>          test blocking wait: OK
> >>>          test MMAP (no poll): OK
> >>>          test MMAP (select): OK
> >>>          test MMAP (epoll): OK
> >>>          Video Capture: Frame #000
> >>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >>>          Video Capture: Frame #001
> >>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >>>          Video Capture: Frame #002
> >>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >>>          Video Capture: Frame #003
> >>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> >>> Aborted
> >>>
> >>> It's weird that the first log fails that way: if the pointers are different,
> >>> then vb2 will call get_userptr and it should discover that the buffer isn't
> >>> large enough, causing qbuf to fail. That doesn't seem to happen.
> >> I think that the reason for this corruption is that the memory pool used
> >> by glibc is now large enough for vb2 to think it can map the full length
> >> of the user pointer into memory, even though only the first half is actually
> >> from the buffer that's allocated. When you capture a frame you just overwrite
> >> a random part of the application's memory pool, causing this invalid pointer.
> >>
> >> But that's a matter of garbage in, garbage out. So that's not the issue here.
> >>
> >> The real question is what happens when you free the old buffer, allocate a
> >> new buffer, end up with the same userptr, but it's using one or more different
> >> pages for its memory compared to the mapping that the kernel uses.
> >>
> >> I managed to reproduce this with v4l2-ctl:
> >>
> >> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >> index 28b2b3b9..8f2ed9b5 100644
> >> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> >>   		 * has the size that fits the old resolution and might not
> >>   		 * fit to the new one.
> >>   		 */
> >> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> >> +			printf("\nidx: %d", buf.g_index());
> >> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> >> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> >> +				fflush(stdout);
> >> +				free(buf.g_userptr(p));
> >> +				void *m = calloc(1, q.g_length(p));
> >> +
> >> +				if (m == NULL)
> >> +					return QUEUE_ERROR;
> >> +				q.s_userptr(buf.g_index(), p, m);
> >> +				if (m == buf.g_userptr(p))
> >> +					printf(" identical new buf");
> >> +				buf.s_userptr(m, p);
> >> +			}
> >> +			printf("\n");
> >> +		}
> >>   		if (fd.qbuf(buf) && errno != EINVAL) {
> >>   			fprintf(stderr, "%s: qbuf error\n", __func__);
> >>   			return QUEUE_ERROR;
> >>
> >>
> >> Load vivid, setup a pure white test pattern:
> >>
> >> v4l2-ctl -c test_pattern=6
> >>
> >> Now run v4l2-ctl --stream-user and you'll see:
> >>
> >> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> >> <
> >> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> >> <
> >> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> >> < 5.00 fps
> >>
> >> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> >> <
> >> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> >>
> >> The first four dequeued buffers are filled with data, after that the
> >> returned buffer is empty because vivid is actually writing to different
> >> memory pages.
> >>
> >> With this patch the first pixel is always non-zero.
> > 
> > Good catch. The question is weather we treat that as undefined behavior 
> > and keep the optimization for 'good applications' or assume that every 
> > broken userspace code has to be properly handled. The good thing is that 
> > there is still imho no security issue. The physical pages gathered by 
> 
> Yeah, that scared me for a bit, but it all looks secure.
> 
> > vb2 in worst case belongs to noone else (vb2 is their last user, they 
> > are not yet returned to free pages pool).
> 
> I see three options:
> 
> 1) just always reacquire the buffer, and if anyone complains about it
>    being slower we point them towards DMABUF.

That doesn't really help right now as DMABUF has the same property: the
pages are mapped and unmapped every time the buffer is touched by a device.

That could be addressed though, it's not an inherent property of DMABUF
buffer type.

> 
> 2) keep the current behavior, but document it.

I'd favour this. Ideally there should be a way to discard a buffer, and
DESTROY_BUF has been proposed before.

> 
> 3) as 2), but also add a new buffer flag that forces a reacquire of the
>    buffer. This could be valid for DMABUF as well. E.g.:
> 
>    V4L2_BUF_FLAG_REACQUIRE
> 
> I'm leaning towards the third option since it won't slow down existing
> implementations, yet if you do change the userptr every time, then you
> can now force this to work safely.

This would get around the problem of reallocating but in a use case
specific way. DESTROY_BUF would be more generic. But I admit it would
require some rework of vb2 codebase, and possibly drivers as well. So I
think this is reasonable compromise.
  

Hi Marek,

On Fri, Jun 07, 2019 at 04:39:22PM +0200, Marek Szyprowski wrote:
> Hi Hans,
> 
> On 2019-06-07 16:11, Hans Verkuil wrote:
> > On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> >> On 2019-06-07 15:40, Hans Verkuil wrote:
> >>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> >>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>>>>> Thank you for the patch.
> >>>>>>>>
> >>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>>>>> Could you explain in the commit message why the assumption is not
> >>>>>>>> correct ?
> >>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>>>>> even though it is not necessarily using the same physical pages for the memory
> >>>>>>> that the kernel is still using for it.
> >>>>>>>
> >>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>>>>> the original mapping still remains), but this can corrupt userspace memory
> >>>>>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>>>>> the sequence of events.
> >>>>>>>
> >>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>>>> I'm interested, I would really like to know what happens in the mm
> >>>>>> subsystem in such case.
> >>>>> Here it is:
> >>>>>
> >>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>> index be606e48..9abf41da 100644
> >>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>>>>    	return 0;
> >>>>>    }
> >>>>>
> >>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>>>>    		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>>>>    		unsigned &capture_count)
> >>>>>    {
> >>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>>    				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>>>>    				buf.s_request_fd(buf_req_fds[req_idx]);
> >>>>>    			}
> >>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>>>> +				printf("\nidx: %d", buf.g_index());
> >>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >>>>> +					fflush(stdout);
> >>>>> +					free(buf.g_userptr(p));
> >>>>> +					void *m = calloc(1, q.g_length(p)/2);
> >>>>> +
> >>>>> +					fail_on_test(m == NULL);
> >>>>> +					q.s_userptr(buf.g_index(), p, m);
> >>>>> +					printf("new buf[%d]: %p", p, m);
> >>>>> +					buf.s_userptr(m, p);
> >>>>> +				}
> >>>>> +				printf("\n");
> >>>>> +			}
> >>>>>    			fail_on_test(buf.qbuf(node, q));
> >>>>>    			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>>>>    			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>>>>
> >>>>>
> >>>>>
> >>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>>>>
> >>>>> ...
> >>>>> Streaming ioctls:
> >>>>>           test read/write: OK
> >>>>>           test blocking wait: OK
> >>>>>           test MMAP (no poll): OK
> >>>>>           test MMAP (select): OK
> >>>>>           test MMAP (epoll): OK
> >>>>>           Video Capture: Frame #000
> >>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>>>>           Video Capture: Frame #001
> >>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>>>>           Video Capture: Frame #002
> >>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >>>>> Aborted
> >>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> >>>> then streaming starts and captureBufs is called which basically just calls dqbuf
> >>>> and qbuf.
> >>>>
> >>>> Tomasz pointed out that all the pointers in this log are actually different. That's
> >>>> correct, but here is a log where the old and new buf ptr are the same:
> >>>>
> >>>> Streaming ioctls:
> >>>>           test read/write: OK
> >>>>           test blocking wait: OK
> >>>>           test MMAP (no poll): OK
> >>>>           test MMAP (select): OK
> >>>>           test MMAP (epoll): OK
> >>>>           Video Capture: Frame #000
> >>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >>>>           Video Capture: Frame #001
> >>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >>>>           Video Capture: Frame #002
> >>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >>>>           Video Capture: Frame #003
> >>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> >>>> Aborted
> >>>>
> >>>> It's weird that the first log fails that way: if the pointers are different,
> >>>> then vb2 will call get_userptr and it should discover that the buffer isn't
> >>>> large enough, causing qbuf to fail. That doesn't seem to happen.
> >>> I think that the reason for this corruption is that the memory pool used
> >>> by glibc is now large enough for vb2 to think it can map the full length
> >>> of the user pointer into memory, even though only the first half is actually
> >>> from the buffer that's allocated. When you capture a frame you just overwrite
> >>> a random part of the application's memory pool, causing this invalid pointer.
> >>>
> >>> But that's a matter of garbage in, garbage out. So that's not the issue here.
> >>>
> >>> The real question is what happens when you free the old buffer, allocate a
> >>> new buffer, end up with the same userptr, but it's using one or more different
> >>> pages for its memory compared to the mapping that the kernel uses.
> >>>
> >>> I managed to reproduce this with v4l2-ctl:
> >>>
> >>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>> index 28b2b3b9..8f2ed9b5 100644
> >>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> >>>    		 * has the size that fits the old resolution and might not
> >>>    		 * fit to the new one.
> >>>    		 */
> >>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>> +			printf("\nidx: %d", buf.g_index());
> >>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> >>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> >>> +				fflush(stdout);
> >>> +				free(buf.g_userptr(p));
> >>> +				void *m = calloc(1, q.g_length(p));
> >>> +
> >>> +				if (m == NULL)
> >>> +					return QUEUE_ERROR;
> >>> +				q.s_userptr(buf.g_index(), p, m);
> >>> +				if (m == buf.g_userptr(p))
> >>> +					printf(" identical new buf");
> >>> +				buf.s_userptr(m, p);
> >>> +			}
> >>> +			printf("\n");
> >>> +		}
> >>>    		if (fd.qbuf(buf) && errno != EINVAL) {
> >>>    			fprintf(stderr, "%s: qbuf error\n", __func__);
> >>>    			return QUEUE_ERROR;
> >>>
> >>>
> >>> Load vivid, setup a pure white test pattern:
> >>>
> >>> v4l2-ctl -c test_pattern=6
> >>>
> >>> Now run v4l2-ctl --stream-user and you'll see:
> >>>
> >>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> >>> <
> >>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> >>> < 5.00 fps
> >>>
> >>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> >>> <
> >>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> >>>
> >>> The first four dequeued buffers are filled with data, after that the
> >>> returned buffer is empty because vivid is actually writing to different
> >>> memory pages.
> >>>
> >>> With this patch the first pixel is always non-zero.
> >> Good catch. The question is weather we treat that as undefined behavior
> >> and keep the optimization for 'good applications' or assume that every
> >> broken userspace code has to be properly handled. The good thing is that
> >> there is still imho no security issue. The physical pages gathered by
> > Yeah, that scared me for a bit, but it all looks secure.
> >
> >> vb2 in worst case belongs to noone else (vb2 is their last user, they
> >> are not yet returned to free pages pool).
> > I see three options:
> >
> > 1) just always reacquire the buffer, and if anyone complains about it
> >     being slower we point them towards DMABUF.
> >
> > 2) keep the current behavior, but document it.
> >
> > 3) as 2), but also add a new buffer flag that forces a reacquire of the
> >     buffer. This could be valid for DMABUF as well. E.g.:
> >
> >     V4L2_BUF_FLAG_REACQUIRE
> >
> > I'm leaning towards the third option since it won't slow down existing
> > implementations, yet if you do change the userptr every time, then you
> > can now force this to work safely.
> 
> Is there are valid use case for third variant? I would rather go for second.
> 
> There is one more issue related to this. There are many apps which use 
> either USERPTR or DMAbuf, but in a bit odd way: they use the same 
> buffers all the time, but they ignore buf->index and never match it to 
> respective buffer pointers or fds. This makes the current caching 
> mechanism useless. Maybe it would make a bit sense do rewrite the 
> caching in qbuf to ignore the provided buffer->index?

It's the index that is used to refer to a given buffer object. The fact
that we have both queueing and mapping a buffer in the same IOCTL is, well,
something that I think we wouldn't do today.

So I don't think we can change this to work based on a memory address just
like that.
  

On Fri, Jun 07, 2019 at 11:34:35PM +0900, Tomasz Figa wrote:
> On Fri, Jun 7, 2019 at 11:11 PM Hans Verkuil wrote:
> > On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > >> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > >>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > >>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > >>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> > >>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > >>>>>>> Thank you for the patch.
> > >>>>>>>
> > >>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > >>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> > >>>>>>>> same user pointer was used as the last one for which memory was acquired, then
> > >>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> > >>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> > >>>>>>> Could you explain in the commit message why the assumption is not
> > >>>>>>> correct ?
> > >>>>>> You can free the memory, then allocate it again and you can get the same pointer,
> > >>>>>> even though it is not necessarily using the same physical pages for the memory
> > >>>>>> that the kernel is still using for it.
> > >>>>>>
> > >>>>>> Worse, you can free the memory, then allocate only half the memory you need and
> > >>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > >>>>>> the original mapping still remains), but this can corrupt userspace memory
> > >>>>>> causing the application to crash. It's not quite clear to me how the memory can
> > >>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> > >>>>>> the sequence of events.
> > >>>>>>
> > >>>>>> I have test code for v4l2-compliance available if someone wants to test this.
> > >>>>> I'm interested, I would really like to know what happens in the mm
> > >>>>> subsystem in such case.
> > >>>> Here it is:
> > >>>>
> > >>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > >>>> index be606e48..9abf41da 100644
> > >>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > >>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > >>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > >>>>    return 0;
> > >>>>   }
> > >>>>
> > >>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> > >>>> +static int captureBufs(struct node *node, cv4l_queue &q,
> > >>>>            const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > >>>>            unsigned &capture_count)
> > >>>>   {
> > >>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > >>>>                            buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > >>>>                            buf.s_request_fd(buf_req_fds[req_idx]);
> > >>>>                    }
> > >>>> +                  if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > >>>> +                          printf("\nidx: %d", buf.g_index());
> > >>>> +                          for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > >>>> +                                  printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > >>>> +                                  fflush(stdout);
> > >>>> +                                  free(buf.g_userptr(p));
> > >>>> +                                  void *m = calloc(1, q.g_length(p)/2);
> > >>>> +
> > >>>> +                                  fail_on_test(m == NULL);
> > >>>> +                                  q.s_userptr(buf.g_index(), p, m);
> > >>>> +                                  printf("new buf[%d]: %p", p, m);
> > >>>> +                                  buf.s_userptr(m, p);
> > >>>> +                          }
> > >>>> +                          printf("\n");
> > >>>> +                  }
> > >>>>                    fail_on_test(buf.qbuf(node, q));
> > >>>>                    fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > >>>>                    if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > >>>>
> > >>>>
> > >>>>
> > >>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > >>>>
> > >>>> ...
> > >>>> Streaming ioctls:
> > >>>>          test read/write: OK
> > >>>>          test blocking wait: OK
> > >>>>          test MMAP (no poll): OK
> > >>>>          test MMAP (select): OK
> > >>>>          test MMAP (epoll): OK
> > >>>>          Video Capture: Frame #000
> > >>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > >>>>          Video Capture: Frame #001
> > >>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > >>>>          Video Capture: Frame #002
> > >>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > >>>> Aborted
> > >>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > >>> then streaming starts and captureBufs is called which basically just calls dqbuf
> > >>> and qbuf.
> > >>>
> > >>> Tomasz pointed out that all the pointers in this log are actually different. That's
> > >>> correct, but here is a log where the old and new buf ptr are the same:
> > >>>
> > >>> Streaming ioctls:
> > >>>          test read/write: OK
> > >>>          test blocking wait: OK
> > >>>          test MMAP (no poll): OK
> > >>>          test MMAP (select): OK
> > >>>          test MMAP (epoll): OK
> > >>>          Video Capture: Frame #000
> > >>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > >>>          Video Capture: Frame #001
> > >>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > >>>          Video Capture: Frame #002
> > >>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > >>>          Video Capture: Frame #003
> > >>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > >>> Aborted
> > >>>
> > >>> It's weird that the first log fails that way: if the pointers are different,
> > >>> then vb2 will call get_userptr and it should discover that the buffer isn't
> > >>> large enough, causing qbuf to fail. That doesn't seem to happen.
> > >> I think that the reason for this corruption is that the memory pool used
> > >> by glibc is now large enough for vb2 to think it can map the full length
> > >> of the user pointer into memory, even though only the first half is actually
> > >> from the buffer that's allocated. When you capture a frame you just overwrite
> > >> a random part of the application's memory pool, causing this invalid pointer.
> > >>
> > >> But that's a matter of garbage in, garbage out. So that's not the issue here.
> > >>
> > >> The real question is what happens when you free the old buffer, allocate a
> > >> new buffer, end up with the same userptr, but it's using one or more different
> > >> pages for its memory compared to the mapping that the kernel uses.
> > >>
> > >> I managed to reproduce this with v4l2-ctl:
> > >>
> > >> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > >> index 28b2b3b9..8f2ed9b5 100644
> > >> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > >> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > >> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > >>               * has the size that fits the old resolution and might not
> > >>               * fit to the new one.
> > >>               */
> > >> +            if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > >> +                    printf("\nidx: %d", buf.g_index());
> > >> +                    for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > >> +                            unsigned *pb = (unsigned *)buf.g_userptr(p);
> > >> +                            printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > >> +                            fflush(stdout);
> > >> +                            free(buf.g_userptr(p));
> > >> +                            void *m = calloc(1, q.g_length(p));
> > >> +
> > >> +                            if (m == NULL)
> > >> +                                    return QUEUE_ERROR;
> > >> +                            q.s_userptr(buf.g_index(), p, m);
> > >> +                            if (m == buf.g_userptr(p))
> > >> +                                    printf(" identical new buf");
> > >> +                            buf.s_userptr(m, p);
> > >> +                    }
> > >> +                    printf("\n");
> > >> +            }
> > >>              if (fd.qbuf(buf) && errno != EINVAL) {
> > >>                      fprintf(stderr, "%s: qbuf error\n", __func__);
> > >>                      return QUEUE_ERROR;
> > >>
> > >>
> > >> Load vivid, setup a pure white test pattern:
> > >>
> > >> v4l2-ctl -c test_pattern=6
> > >>
> > >> Now run v4l2-ctl --stream-user and you'll see:
> > >>
> > >> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > >> <
> > >> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > >> <
> > >> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > >> <
> > >> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > >> <
> > >> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > >> <
> > >> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > >> < 5.00 fps
> > >>
> > >> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > >> <
> > >> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > >>
> > >> The first four dequeued buffers are filled with data, after that the
> > >> returned buffer is empty because vivid is actually writing to different
> > >> memory pages.
> > >>
> > >> With this patch the first pixel is always non-zero.
> > >
> > > Good catch. The question is weather we treat that as undefined behavior
> > > and keep the optimization for 'good applications' or assume that every
> > > broken userspace code has to be properly handled. The good thing is that
> > > there is still imho no security issue. The physical pages gathered by
> >
> > Yeah, that scared me for a bit, but it all looks secure.
> >
> > > vb2 in worst case belongs to noone else (vb2 is their last user, they
> > > are not yet returned to free pages pool).
> >
> > I see three options:
> >
> > 1) just always reacquire the buffer, and if anyone complains about it
> >    being slower we point them towards DMABUF.
> >
> > 2) keep the current behavior, but document it.
> >
> > 3) as 2), but also add a new buffer flag that forces a reacquire of the
> >    buffer. This could be valid for DMABUF as well. E.g.:
> >
> >    V4L2_BUF_FLAG_REACQUIRE
> >
> > I'm leaning towards the third option since it won't slow down existing
> > implementations, yet if you do change the userptr every time, then you
> > can now force this to work safely.
> >
> 
> I'd be for 1) or 3) as that would allow Chrome work on mainline.

I don't like 3) much, it makes the API and the implementation more
complex just to work around a problem with an API that should not be
used anymore. I'd go for 1), giving even more incentive to stop using
USERPTR.

> Also I believe there is still some bug when the pointers don't match,
> even if you don't free those pages. I guess some more testing that
> includes verifying the contents of previously dequeued buffers could
> show something.
> 
> > >> I wonder if it isn't possible to just check the physical address of
> > >> the received user pointer with the physical address of the previous
> > >> user pointer. Or something like that. I'll dig around a bit more.
> > >
> > > Such check won't be so simple. Pages contiguous in the virtual memory
> > > won't map to pages contiguous in the physical memory, so you would need
> > > to check every single memory page. Make no sense. It is better to
> > > reacquire buffer on every queue operation. This indeed show how broken
> > > the USERPTR related part of v4l2 API is.
> >
> > OK, good to know. Then I'm not going to spend time on that.
  

Le vendredi 07 juin 2019 à 16:58 +0300, Laurent Pinchart a écrit :
> Hi Marek,
> 
> On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > Thank you for the patch.
> > > > > > > > 
> > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > correct ?
> > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > that the kernel is still using for it.
> > > > > > > 
> > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > the sequence of events.
> > > > > > > 
> > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > subsystem in such case.
> > > > > Here it is:
> > > > > 
> > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > index be606e48..9abf41da 100644
> > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > >   	return 0;
> > > > >   }
> > > > > 
> > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > >   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > >   		unsigned &capture_count)
> > > > >   {
> > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > >   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > >   				buf.s_request_fd(buf_req_fds[req_idx]);
> > > > >   			}
> > > > > +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > +				printf("\nidx: %d", buf.g_index());
> > > > > +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > +					fflush(stdout);
> > > > > +					free(buf.g_userptr(p));
> > > > > +					void *m = calloc(1, q.g_length(p)/2);
> > > > > +
> > > > > +					fail_on_test(m == NULL);
> > > > > +					q.s_userptr(buf.g_index(), p, m);
> > > > > +					printf("new buf[%d]: %p", p, m);
> > > > > +					buf.s_userptr(m, p);
> > > > > +				}
> > > > > +				printf("\n");
> > > > > +			}
> > > > >   			fail_on_test(buf.qbuf(node, q));
> > > > >   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > >   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > 
> > > > > 
> > > > > 
> > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > 
> > > > > ...
> > > > > Streaming ioctls:
> > > > >          test read/write: OK
> > > > >          test blocking wait: OK
> > > > >          test MMAP (no poll): OK
> > > > >          test MMAP (select): OK
> > > > >          test MMAP (epoll): OK
> > > > >          Video Capture: Frame #000
> > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > >          Video Capture: Frame #001
> > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > >          Video Capture: Frame #002
> > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > Aborted
> > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > and qbuf.
> > > > 
> > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > 
> > > > Streaming ioctls:
> > > >          test read/write: OK
> > > >          test blocking wait: OK
> > > >          test MMAP (no poll): OK
> > > >          test MMAP (select): OK
> > > >          test MMAP (epoll): OK
> > > >          Video Capture: Frame #000
> > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > >          Video Capture: Frame #001
> > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > >          Video Capture: Frame #002
> > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > >          Video Capture: Frame #003
> > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > Aborted
> > > > 
> > > > It's weird that the first log fails that way: if the pointers are different,
> > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > I think that the reason for this corruption is that the memory pool used
> > > by glibc is now large enough for vb2 to think it can map the full length
> > > of the user pointer into memory, even though only the first half is actually
> > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > a random part of the application's memory pool, causing this invalid pointer.
> > > 
> > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > 
> > > The real question is what happens when you free the old buffer, allocate a
> > > new buffer, end up with the same userptr, but it's using one or more different
> > > pages for its memory compared to the mapping that the kernel uses.
> > > 
> > > I managed to reproduce this with v4l2-ctl:
> > > 
> > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > index 28b2b3b9..8f2ed9b5 100644
> > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > >   		 * has the size that fits the old resolution and might not
> > >   		 * fit to the new one.
> > >   		 */
> > > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > +			printf("\nidx: %d", buf.g_index());
> > > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > +				fflush(stdout);
> > > +				free(buf.g_userptr(p));
> > > +				void *m = calloc(1, q.g_length(p));
> > > +
> > > +				if (m == NULL)
> > > +					return QUEUE_ERROR;
> > > +				q.s_userptr(buf.g_index(), p, m);
> > > +				if (m == buf.g_userptr(p))
> > > +					printf(" identical new buf");
> > > +				buf.s_userptr(m, p);
> > > +			}
> > > +			printf("\n");
> > > +		}
> > >   		if (fd.qbuf(buf) && errno != EINVAL) {
> > >   			fprintf(stderr, "%s: qbuf error\n", __func__);
> > >   			return QUEUE_ERROR;
> > > 
> > > 
> > > Load vivid, setup a pure white test pattern:
> > > 
> > > v4l2-ctl -c test_pattern=6
> > > 
> > > Now run v4l2-ctl --stream-user and you'll see:
> > > 
> > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > <
> > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > <
> > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > <
> > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > <
> > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > <
> > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > < 5.00 fps
> > > 
> > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > <
> > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > 
> > > The first four dequeued buffers are filled with data, after that the
> > > returned buffer is empty because vivid is actually writing to different
> > > memory pages.
> > > 
> > > With this patch the first pixel is always non-zero.
> > 
> > Good catch. The question is weather we treat that as undefined behavior 
> > and keep the optimization for 'good applications' or assume that every 
> > broken userspace code has to be properly handled.
> 
> Given how long we've been saying that USERPTR should be replaced by
> DMABUF, I would consider that any userspace code using USERPTR is broken
> :-) One could however question whether we were effective at getting that
> message across...

Just a reminder that DMABuf is not a replacement for USERPTR. It only
cover a subset in absence of an allocater for it. There is no clean way
to allocate a DMAbuf. Notably, memfds (which could have filled the gap)
are not DMABuf, even though they are they are similar to the buffers
allocated by vivid or uvcvideo.

> 
> > The good thing is that 
> > there is still imho no security issue. The physical pages gathered by 
> > vb2 in worst case belongs to noone else (vb2 is their last user, they 
> > are not yet returned to free pages pool).
> > 
> > > I wonder if it isn't possible to just check the physical address of
> > > the received user pointer with the physical address of the previous
> > > user pointer. Or something like that. I'll dig around a bit more.
> > 
> > Such check won't be so simple. Pages contiguous in the virtual memory 
> > won't map to pages contiguous in the physical memory, so you would need 
> > to check every single memory page. Make no sense. It is better to 
> > reacquire buffer on every queue operation. This indeed show how broken 
> > the USERPTR related part of v4l2 API is.
  

Le vendredi 07 juin 2019 à 16:39 +0200, Marek Szyprowski a écrit :
> Hi Hans,
> 
> On 2019-06-07 16:11, Hans Verkuil wrote:
> > On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > > Thank you for the patch.
> > > > > > > > > 
> > > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > > correct ?
> > > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > > that the kernel is still using for it.
> > > > > > > > 
> > > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > > the sequence of events.
> > > > > > > > 
> > > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > > subsystem in such case.
> > > > > > Here it is:
> > > > > > 
> > > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > index be606e48..9abf41da 100644
> > > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > > >    	return 0;
> > > > > >    }
> > > > > > 
> > > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > > >    		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > > >    		unsigned &capture_count)
> > > > > >    {
> > > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > >    				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > > >    				buf.s_request_fd(buf_req_fds[req_idx]);
> > > > > >    			}
> > > > > > +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > +				printf("\nidx: %d", buf.g_index());
> > > > > > +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > > +					fflush(stdout);
> > > > > > +					free(buf.g_userptr(p));
> > > > > > +					void *m = calloc(1, q.g_length(p)/2);
> > > > > > +
> > > > > > +					fail_on_test(m == NULL);
> > > > > > +					q.s_userptr(buf.g_index(), p, m);
> > > > > > +					printf("new buf[%d]: %p", p, m);
> > > > > > +					buf.s_userptr(m, p);
> > > > > > +				}
> > > > > > +				printf("\n");
> > > > > > +			}
> > > > > >    			fail_on_test(buf.qbuf(node, q));
> > > > > >    			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > > >    			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > > 
> > > > > > ...
> > > > > > Streaming ioctls:
> > > > > >           test read/write: OK
> > > > > >           test blocking wait: OK
> > > > > >           test MMAP (no poll): OK
> > > > > >           test MMAP (select): OK
> > > > > >           test MMAP (epoll): OK
> > > > > >           Video Capture: Frame #000
> > > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > > >           Video Capture: Frame #001
> > > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > > >           Video Capture: Frame #002
> > > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > > Aborted
> > > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > > and qbuf.
> > > > > 
> > > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > > 
> > > > > Streaming ioctls:
> > > > >           test read/write: OK
> > > > >           test blocking wait: OK
> > > > >           test MMAP (no poll): OK
> > > > >           test MMAP (select): OK
> > > > >           test MMAP (epoll): OK
> > > > >           Video Capture: Frame #000
> > > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > > >           Video Capture: Frame #001
> > > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > > >           Video Capture: Frame #002
> > > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > > >           Video Capture: Frame #003
> > > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > > Aborted
> > > > > 
> > > > > It's weird that the first log fails that way: if the pointers are different,
> > > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > > I think that the reason for this corruption is that the memory pool used
> > > > by glibc is now large enough for vb2 to think it can map the full length
> > > > of the user pointer into memory, even though only the first half is actually
> > > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > > a random part of the application's memory pool, causing this invalid pointer.
> > > > 
> > > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > > 
> > > > The real question is what happens when you free the old buffer, allocate a
> > > > new buffer, end up with the same userptr, but it's using one or more different
> > > > pages for its memory compared to the mapping that the kernel uses.
> > > > 
> > > > I managed to reproduce this with v4l2-ctl:
> > > > 
> > > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > index 28b2b3b9..8f2ed9b5 100644
> > > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > > >    		 * has the size that fits the old resolution and might not
> > > >    		 * fit to the new one.
> > > >    		 */
> > > > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > +			printf("\nidx: %d", buf.g_index());
> > > > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > > +				fflush(stdout);
> > > > +				free(buf.g_userptr(p));
> > > > +				void *m = calloc(1, q.g_length(p));
> > > > +
> > > > +				if (m == NULL)
> > > > +					return QUEUE_ERROR;
> > > > +				q.s_userptr(buf.g_index(), p, m);
> > > > +				if (m == buf.g_userptr(p))
> > > > +					printf(" identical new buf");
> > > > +				buf.s_userptr(m, p);
> > > > +			}
> > > > +			printf("\n");
> > > > +		}
> > > >    		if (fd.qbuf(buf) && errno != EINVAL) {
> > > >    			fprintf(stderr, "%s: qbuf error\n", __func__);
> > > >    			return QUEUE_ERROR;
> > > > 
> > > > 
> > > > Load vivid, setup a pure white test pattern:
> > > > 
> > > > v4l2-ctl -c test_pattern=6
> > > > 
> > > > Now run v4l2-ctl --stream-user and you'll see:
> > > > 
> > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > > <
> > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > > <
> > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > > <
> > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > > <
> > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > > <
> > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > > < 5.00 fps
> > > > 
> > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > > <
> > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > > 
> > > > The first four dequeued buffers are filled with data, after that the
> > > > returned buffer is empty because vivid is actually writing to different
> > > > memory pages.
> > > > 
> > > > With this patch the first pixel is always non-zero.
> > > Good catch. The question is weather we treat that as undefined behavior
> > > and keep the optimization for 'good applications' or assume that every
> > > broken userspace code has to be properly handled. The good thing is that
> > > there is still imho no security issue. The physical pages gathered by
> > Yeah, that scared me for a bit, but it all looks secure.
> > 
> > > vb2 in worst case belongs to noone else (vb2 is their last user, they
> > > are not yet returned to free pages pool).
> > I see three options:
> > 
> > 1) just always reacquire the buffer, and if anyone complains about it
> >     being slower we point them towards DMABUF.
> > 
> > 2) keep the current behavior, but document it.
> > 
> > 3) as 2), but also add a new buffer flag that forces a reacquire of the
> >     buffer. This could be valid for DMABUF as well. E.g.:
> > 
> >     V4L2_BUF_FLAG_REACQUIRE
> > 
> > I'm leaning towards the third option since it won't slow down existing
> > implementations, yet if you do change the userptr every time, then you
> > can now force this to work safely.
> 
> Is there are valid use case for third variant? I would rather go for second.
> 
> There is one more issue related to this. There are many apps which use 
> either USERPTR or DMAbuf, but in a bit odd way: they use the same 
> buffers all the time, but they ignore buf->index and never match it to 
> respective buffer pointers or fds. This makes the current caching 
> mechanism useless. Maybe it would make a bit sense do rewrite the 
> caching in qbuf to ignore the provided buffer->index?

Notably GStreamer, which inherited this issue from a public API design
error some 15 years ago. Complaint wise, I don't remember someone
complaining about that, so option 1) would simply make the performance
consistent for the framework.

> 
> > > > I wonder if it isn't possible to just check the physical address of
> > > > the received user pointer with the physical address of the previous
> > > > user pointer. Or something like that. I'll dig around a bit more.
> > > Such check won't be so simple. Pages contiguous in the virtual memory
> > > won't map to pages contiguous in the physical memory, so you would need
> > > to check every single memory page. Make no sense. It is better to
> > > reacquire buffer on every queue operation. This indeed show how broken
> > > the USERPTR related part of v4l2 API is.
> > OK, good to know. Then I'm not going to spend time on that.
> > 
> > 
> Best regards
  

On 6/7/19 4:34 PM, Tomasz Figa wrote:
> On Fri, Jun 7, 2019 at 11:11 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> On 6/7/19 3:55 PM, Marek Szyprowski wrote:
>>> Hi Hans,
>>>
>>> On 2019-06-07 15:40, Hans Verkuil wrote:
>>>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
>>>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>>>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>>>>>> Thank you for the patch.
>>>>>>>>>
>>>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>>>>>> Could you explain in the commit message why the assumption is not
>>>>>>>>> correct ?
>>>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>>>>>> even though it is not necessarily using the same physical pages for the memory
>>>>>>>> that the kernel is still using for it.
>>>>>>>>
>>>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>>>>>> the original mapping still remains), but this can corrupt userspace memory
>>>>>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>>>>>> the sequence of events.
>>>>>>>>
>>>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>>>>> I'm interested, I would really like to know what happens in the mm
>>>>>>> subsystem in such case.
>>>>>> Here it is:
>>>>>>
>>>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>> index be606e48..9abf41da 100644
>>>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>>>>>    return 0;
>>>>>>   }
>>>>>>
>>>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>>>>>            const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>>>>>            unsigned &capture_count)
>>>>>>   {
>>>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>>                            buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>>>>>                            buf.s_request_fd(buf_req_fds[req_idx]);
>>>>>>                    }
>>>>>> +                  if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>>>> +                          printf("\nidx: %d", buf.g_index());
>>>>>> +                          for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>>>> +                                  printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>>>>>> +                                  fflush(stdout);
>>>>>> +                                  free(buf.g_userptr(p));
>>>>>> +                                  void *m = calloc(1, q.g_length(p)/2);
>>>>>> +
>>>>>> +                                  fail_on_test(m == NULL);
>>>>>> +                                  q.s_userptr(buf.g_index(), p, m);
>>>>>> +                                  printf("new buf[%d]: %p", p, m);
>>>>>> +                                  buf.s_userptr(m, p);
>>>>>> +                          }
>>>>>> +                          printf("\n");
>>>>>> +                  }
>>>>>>                    fail_on_test(buf.qbuf(node, q));
>>>>>>                    fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>>>>>                    if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>>>>>
>>>>>>
>>>>>>
>>>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>>>>>
>>>>>> ...
>>>>>> Streaming ioctls:
>>>>>>          test read/write: OK
>>>>>>          test blocking wait: OK
>>>>>>          test MMAP (no poll): OK
>>>>>>          test MMAP (select): OK
>>>>>>          test MMAP (epoll): OK
>>>>>>          Video Capture: Frame #000
>>>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>>>>>          Video Capture: Frame #001
>>>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>>>>>          Video Capture: Frame #002
>>>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>>>>>> Aborted
>>>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
>>>>> then streaming starts and captureBufs is called which basically just calls dqbuf
>>>>> and qbuf.
>>>>>
>>>>> Tomasz pointed out that all the pointers in this log are actually different. That's
>>>>> correct, but here is a log where the old and new buf ptr are the same:
>>>>>
>>>>> Streaming ioctls:
>>>>>          test read/write: OK
>>>>>          test blocking wait: OK
>>>>>          test MMAP (no poll): OK
>>>>>          test MMAP (select): OK
>>>>>          test MMAP (epoll): OK
>>>>>          Video Capture: Frame #000
>>>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>>>>>          Video Capture: Frame #001
>>>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>>>>>          Video Capture: Frame #002
>>>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>>>>>          Video Capture: Frame #003
>>>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
>>>>> Aborted
>>>>>
>>>>> It's weird that the first log fails that way: if the pointers are different,
>>>>> then vb2 will call get_userptr and it should discover that the buffer isn't
>>>>> large enough, causing qbuf to fail. That doesn't seem to happen.
>>>> I think that the reason for this corruption is that the memory pool used
>>>> by glibc is now large enough for vb2 to think it can map the full length
>>>> of the user pointer into memory, even though only the first half is actually
>>>> from the buffer that's allocated. When you capture a frame you just overwrite
>>>> a random part of the application's memory pool, causing this invalid pointer.
>>>>
>>>> But that's a matter of garbage in, garbage out. So that's not the issue here.
>>>>
>>>> The real question is what happens when you free the old buffer, allocate a
>>>> new buffer, end up with the same userptr, but it's using one or more different
>>>> pages for its memory compared to the mapping that the kernel uses.
>>>>
>>>> I managed to reproduce this with v4l2-ctl:
>>>>
>>>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>> index 28b2b3b9..8f2ed9b5 100644
>>>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>>>>               * has the size that fits the old resolution and might not
>>>>               * fit to the new one.
>>>>               */
>>>> +            if (q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>> +                    printf("\nidx: %d", buf.g_index());
>>>> +                    for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>> +                            unsigned *pb = (unsigned *)buf.g_userptr(p);
>>>> +                            printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
>>>> +                            fflush(stdout);
>>>> +                            free(buf.g_userptr(p));
>>>> +                            void *m = calloc(1, q.g_length(p));
>>>> +
>>>> +                            if (m == NULL)
>>>> +                                    return QUEUE_ERROR;
>>>> +                            q.s_userptr(buf.g_index(), p, m);
>>>> +                            if (m == buf.g_userptr(p))
>>>> +                                    printf(" identical new buf");
>>>> +                            buf.s_userptr(m, p);
>>>> +                    }
>>>> +                    printf("\n");
>>>> +            }
>>>>              if (fd.qbuf(buf) && errno != EINVAL) {
>>>>                      fprintf(stderr, "%s: qbuf error\n", __func__);
>>>>                      return QUEUE_ERROR;
>>>>
>>>>
>>>> Load vivid, setup a pure white test pattern:
>>>>
>>>> v4l2-ctl -c test_pattern=6
>>>>
>>>> Now run v4l2-ctl --stream-user and you'll see:
>>>>
>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
>>>> <
>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
>>>> <
>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
>>>> <
>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
>>>> <
>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
>>>> <
>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
>>>> < 5.00 fps
>>>>
>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
>>>> <
>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>>>>
>>>> The first four dequeued buffers are filled with data, after that the
>>>> returned buffer is empty because vivid is actually writing to different
>>>> memory pages.
>>>>
>>>> With this patch the first pixel is always non-zero.
>>>
>>> Good catch. The question is weather we treat that as undefined behavior
>>> and keep the optimization for 'good applications' or assume that every
>>> broken userspace code has to be properly handled. The good thing is that
>>> there is still imho no security issue. The physical pages gathered by
>>
>> Yeah, that scared me for a bit, but it all looks secure.
>>
>>> vb2 in worst case belongs to noone else (vb2 is their last user, they
>>> are not yet returned to free pages pool).
>>
>> I see three options:
>>
>> 1) just always reacquire the buffer, and if anyone complains about it
>>    being slower we point them towards DMABUF.
>>
>> 2) keep the current behavior, but document it.
>>
>> 3) as 2), but also add a new buffer flag that forces a reacquire of the
>>    buffer. This could be valid for DMABUF as well. E.g.:
>>
>>    V4L2_BUF_FLAG_REACQUIRE
>>
>> I'm leaning towards the third option since it won't slow down existing
>> implementations, yet if you do change the userptr every time, then you
>> can now force this to work safely.
>>
> 
> I'd be for 1) or 3) as that would allow Chrome work on mainline.
> 
> Also I believe there is still some bug when the pointers don't match,
> even if you don't free those pages. I guess some more testing that
> includes verifying the contents of previously dequeued buffers could
> show something.

I also realized that there is another problem with USERPTR:

Suppose you queue buffer 1 with pointer X, then dequeue it (the mapping
remains), then queue buffer 2 with the same pointer X: it will now be mapped
again. I've no idea what the result of that will be.

While DMABUF has the same behavior at first glance, after digging deeper
into the dma_buf framework details I see that it is actually refcounting
the mappings and so will not map again, instead it just returns the
existing mapping. So DMABUF is fine.

I think we should either go for option 1 or option 3. My preference is 3,
but it depends on how often USERPTR is used in practice.

Tomasz, Nicolas, do you guys have a better idea about that?

Regards,

	Hans

> 
>>>> I wonder if it isn't possible to just check the physical address of
>>>> the received user pointer with the physical address of the previous
>>>> user pointer. Or something like that. I'll dig around a bit more.
>>>
>>> Such check won't be so simple. Pages contiguous in the virtual memory
>>> won't map to pages contiguous in the physical memory, so you would need
>>> to check every single memory page. Make no sense. It is better to
>>> reacquire buffer on every queue operation. This indeed show how broken
>>> the USERPTR related part of v4l2 API is.
>>
>> OK, good to know. Then I'm not going to spend time on that.
>>
>> Regards,
>>
>>         Hans
  

On 6/7/19 9:43 PM, Nicolas Dufresne wrote:
> Le vendredi 07 juin 2019 à 16:39 +0200, Marek Szyprowski a écrit :
>> Hi Hans,
>>
>> On 2019-06-07 16:11, Hans Verkuil wrote:
>>> On 6/7/19 3:55 PM, Marek Szyprowski wrote:
>>>> On 2019-06-07 15:40, Hans Verkuil wrote:
>>>>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
>>>>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>>>>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>>>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>>>>>>> Thank you for the patch.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>>>>>>> Could you explain in the commit message why the assumption is not
>>>>>>>>>> correct ?
>>>>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>>>>>>> even though it is not necessarily using the same physical pages for the memory
>>>>>>>>> that the kernel is still using for it.
>>>>>>>>>
>>>>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>>>>>>> the original mapping still remains), but this can corrupt userspace memory
>>>>>>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>>>>>>> the sequence of events.
>>>>>>>>>
>>>>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>>>>>> I'm interested, I would really like to know what happens in the mm
>>>>>>>> subsystem in such case.
>>>>>>> Here it is:
>>>>>>>
>>>>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>>> index be606e48..9abf41da 100644
>>>>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>>>>>>    	return 0;
>>>>>>>    }
>>>>>>>
>>>>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>>>>>>    		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>>>>>>    		unsigned &capture_count)
>>>>>>>    {
>>>>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>>>    				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>>>>>>    				buf.s_request_fd(buf_req_fds[req_idx]);
>>>>>>>    			}
>>>>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>>>>> +				printf("\nidx: %d", buf.g_index());
>>>>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>>>>>>> +					fflush(stdout);
>>>>>>> +					free(buf.g_userptr(p));
>>>>>>> +					void *m = calloc(1, q.g_length(p)/2);
>>>>>>> +
>>>>>>> +					fail_on_test(m == NULL);
>>>>>>> +					q.s_userptr(buf.g_index(), p, m);
>>>>>>> +					printf("new buf[%d]: %p", p, m);
>>>>>>> +					buf.s_userptr(m, p);
>>>>>>> +				}
>>>>>>> +				printf("\n");
>>>>>>> +			}
>>>>>>>    			fail_on_test(buf.qbuf(node, q));
>>>>>>>    			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>>>>>>    			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>>>>>>
>>>>>>> ...
>>>>>>> Streaming ioctls:
>>>>>>>           test read/write: OK
>>>>>>>           test blocking wait: OK
>>>>>>>           test MMAP (no poll): OK
>>>>>>>           test MMAP (select): OK
>>>>>>>           test MMAP (epoll): OK
>>>>>>>           Video Capture: Frame #000
>>>>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>>>>>>           Video Capture: Frame #001
>>>>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>>>>>>           Video Capture: Frame #002
>>>>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>>>>>>> Aborted
>>>>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
>>>>>> then streaming starts and captureBufs is called which basically just calls dqbuf
>>>>>> and qbuf.
>>>>>>
>>>>>> Tomasz pointed out that all the pointers in this log are actually different. That's
>>>>>> correct, but here is a log where the old and new buf ptr are the same:
>>>>>>
>>>>>> Streaming ioctls:
>>>>>>           test read/write: OK
>>>>>>           test blocking wait: OK
>>>>>>           test MMAP (no poll): OK
>>>>>>           test MMAP (select): OK
>>>>>>           test MMAP (epoll): OK
>>>>>>           Video Capture: Frame #000
>>>>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>>>>>>           Video Capture: Frame #001
>>>>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>>>>>>           Video Capture: Frame #002
>>>>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>>>>>>           Video Capture: Frame #003
>>>>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
>>>>>> Aborted
>>>>>>
>>>>>> It's weird that the first log fails that way: if the pointers are different,
>>>>>> then vb2 will call get_userptr and it should discover that the buffer isn't
>>>>>> large enough, causing qbuf to fail. That doesn't seem to happen.
>>>>> I think that the reason for this corruption is that the memory pool used
>>>>> by glibc is now large enough for vb2 to think it can map the full length
>>>>> of the user pointer into memory, even though only the first half is actually
>>>>> from the buffer that's allocated. When you capture a frame you just overwrite
>>>>> a random part of the application's memory pool, causing this invalid pointer.
>>>>>
>>>>> But that's a matter of garbage in, garbage out. So that's not the issue here.
>>>>>
>>>>> The real question is what happens when you free the old buffer, allocate a
>>>>> new buffer, end up with the same userptr, but it's using one or more different
>>>>> pages for its memory compared to the mapping that the kernel uses.
>>>>>
>>>>> I managed to reproduce this with v4l2-ctl:
>>>>>
>>>>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>>> index 28b2b3b9..8f2ed9b5 100644
>>>>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>>>>>    		 * has the size that fits the old resolution and might not
>>>>>    		 * fit to the new one.
>>>>>    		 */
>>>>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>>> +			printf("\nidx: %d", buf.g_index());
>>>>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
>>>>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
>>>>> +				fflush(stdout);
>>>>> +				free(buf.g_userptr(p));
>>>>> +				void *m = calloc(1, q.g_length(p));
>>>>> +
>>>>> +				if (m == NULL)
>>>>> +					return QUEUE_ERROR;
>>>>> +				q.s_userptr(buf.g_index(), p, m);
>>>>> +				if (m == buf.g_userptr(p))
>>>>> +					printf(" identical new buf");
>>>>> +				buf.s_userptr(m, p);
>>>>> +			}
>>>>> +			printf("\n");
>>>>> +		}
>>>>>    		if (fd.qbuf(buf) && errno != EINVAL) {
>>>>>    			fprintf(stderr, "%s: qbuf error\n", __func__);
>>>>>    			return QUEUE_ERROR;
>>>>>
>>>>>
>>>>> Load vivid, setup a pure white test pattern:
>>>>>
>>>>> v4l2-ctl -c test_pattern=6
>>>>>
>>>>> Now run v4l2-ctl --stream-user and you'll see:
>>>>>
>>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
>>>>> <
>>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
>>>>> <
>>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
>>>>> <
>>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
>>>>> <
>>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
>>>>> <
>>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
>>>>> < 5.00 fps
>>>>>
>>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
>>>>> <
>>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>>>>>
>>>>> The first four dequeued buffers are filled with data, after that the
>>>>> returned buffer is empty because vivid is actually writing to different
>>>>> memory pages.
>>>>>
>>>>> With this patch the first pixel is always non-zero.
>>>> Good catch. The question is weather we treat that as undefined behavior
>>>> and keep the optimization for 'good applications' or assume that every
>>>> broken userspace code has to be properly handled. The good thing is that
>>>> there is still imho no security issue. The physical pages gathered by
>>> Yeah, that scared me for a bit, but it all looks secure.
>>>
>>>> vb2 in worst case belongs to noone else (vb2 is their last user, they
>>>> are not yet returned to free pages pool).
>>> I see three options:
>>>
>>> 1) just always reacquire the buffer, and if anyone complains about it
>>>     being slower we point them towards DMABUF.
>>>
>>> 2) keep the current behavior, but document it.
>>>
>>> 3) as 2), but also add a new buffer flag that forces a reacquire of the
>>>     buffer. This could be valid for DMABUF as well. E.g.:
>>>
>>>     V4L2_BUF_FLAG_REACQUIRE
>>>
>>> I'm leaning towards the third option since it won't slow down existing
>>> implementations, yet if you do change the userptr every time, then you
>>> can now force this to work safely.
>>
>> Is there are valid use case for third variant? I would rather go for second.
>>
>> There is one more issue related to this. There are many apps which use 
>> either USERPTR or DMAbuf, but in a bit odd way: they use the same 
>> buffers all the time, but they ignore buf->index and never match it to 
>> respective buffer pointers or fds. This makes the current caching 
>> mechanism useless. Maybe it would make a bit sense do rewrite the 
>> caching in qbuf to ignore the provided buffer->index?
> 
> Notably GStreamer, which inherited this issue from a public API design
> error some 15 years ago. Complaint wise, I don't remember someone
> complaining about that, so option 1) would simply make the performance
> consistent for the framework.

After analyzing the DMABUF behavior in this case I realized that the
dma_buf framework refcount the mapping, so it won't map again unless
it's really necessary. So there is (almost) no performance hit for
DMABUF if users do not match dmabuf fds with the buffer index.

So option 1 *would* slow down the USERPTR performance compared to
the other streaming models.

Regards,

	Hans
  

Hi Nicolas,

On Fri, Jun 07, 2019 at 03:38:39PM -0400, Nicolas Dufresne wrote:
> Le vendredi 07 juin 2019 à 16:58 +0300, Laurent Pinchart a écrit :
> > On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> >> On 2019-06-07 15:40, Hans Verkuil wrote:
> >>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> >>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>>>>> Thank you for the patch.
> >>>>>>>> 
> >>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>>>>> Could you explain in the commit message why the assumption is not
> >>>>>>>> correct ?
> >>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>>>>> even though it is not necessarily using the same physical pages for the memory
> >>>>>>> that the kernel is still using for it.
> >>>>>>> 
> >>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>>>>> the original mapping still remains), but this can corrupt userspace memory
> >>>>>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>>>>> the sequence of events.
> >>>>>>> 
> >>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>>>> I'm interested, I would really like to know what happens in the mm
> >>>>>> subsystem in such case.
> >>>>> Here it is:
> >>>>> 
> >>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>> index be606e48..9abf41da 100644
> >>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>>>>   	return 0;
> >>>>>   }
> >>>>> 
> >>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>>>>   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>>>>   		unsigned &capture_count)
> >>>>>   {
> >>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>>   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>>>>   				buf.s_request_fd(buf_req_fds[req_idx]);
> >>>>>   			}
> >>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>>>> +				printf("\nidx: %d", buf.g_index());
> >>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >>>>> +					fflush(stdout);
> >>>>> +					free(buf.g_userptr(p));
> >>>>> +					void *m = calloc(1, q.g_length(p)/2);
> >>>>> +
> >>>>> +					fail_on_test(m == NULL);
> >>>>> +					q.s_userptr(buf.g_index(), p, m);
> >>>>> +					printf("new buf[%d]: %p", p, m);
> >>>>> +					buf.s_userptr(m, p);
> >>>>> +				}
> >>>>> +				printf("\n");
> >>>>> +			}
> >>>>>   			fail_on_test(buf.qbuf(node, q));
> >>>>>   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>>>>   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>>>> 
> >>>>> ...
> >>>>> Streaming ioctls:
> >>>>>          test read/write: OK
> >>>>>          test blocking wait: OK
> >>>>>          test MMAP (no poll): OK
> >>>>>          test MMAP (select): OK
> >>>>>          test MMAP (epoll): OK
> >>>>>          Video Capture: Frame #000
> >>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>>>>          Video Capture: Frame #001
> >>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>>>>          Video Capture: Frame #002
> >>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >>>>> Aborted
> >>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> >>>> then streaming starts and captureBufs is called which basically just calls dqbuf
> >>>> and qbuf.
> >>>> 
> >>>> Tomasz pointed out that all the pointers in this log are actually different. That's
> >>>> correct, but here is a log where the old and new buf ptr are the same:
> >>>> 
> >>>> Streaming ioctls:
> >>>>          test read/write: OK
> >>>>          test blocking wait: OK
> >>>>          test MMAP (no poll): OK
> >>>>          test MMAP (select): OK
> >>>>          test MMAP (epoll): OK
> >>>>          Video Capture: Frame #000
> >>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >>>>          Video Capture: Frame #001
> >>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >>>>          Video Capture: Frame #002
> >>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >>>>          Video Capture: Frame #003
> >>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> >>>> Aborted
> >>>> 
> >>>> It's weird that the first log fails that way: if the pointers are different,
> >>>> then vb2 will call get_userptr and it should discover that the buffer isn't
> >>>> large enough, causing qbuf to fail. That doesn't seem to happen.
> >>> I think that the reason for this corruption is that the memory pool used
> >>> by glibc is now large enough for vb2 to think it can map the full length
> >>> of the user pointer into memory, even though only the first half is actually
> >>> from the buffer that's allocated. When you capture a frame you just overwrite
> >>> a random part of the application's memory pool, causing this invalid pointer.
> >>> 
> >>> But that's a matter of garbage in, garbage out. So that's not the issue here.
> >>> 
> >>> The real question is what happens when you free the old buffer, allocate a
> >>> new buffer, end up with the same userptr, but it's using one or more different
> >>> pages for its memory compared to the mapping that the kernel uses.
> >>> 
> >>> I managed to reproduce this with v4l2-ctl:
> >>> 
> >>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>> index 28b2b3b9..8f2ed9b5 100644
> >>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> >>>   		 * has the size that fits the old resolution and might not
> >>>   		 * fit to the new one.
> >>>   		 */
> >>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>> +			printf("\nidx: %d", buf.g_index());
> >>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> >>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> >>> +				fflush(stdout);
> >>> +				free(buf.g_userptr(p));
> >>> +				void *m = calloc(1, q.g_length(p));
> >>> +
> >>> +				if (m == NULL)
> >>> +					return QUEUE_ERROR;
> >>> +				q.s_userptr(buf.g_index(), p, m);
> >>> +				if (m == buf.g_userptr(p))
> >>> +					printf(" identical new buf");
> >>> +				buf.s_userptr(m, p);
> >>> +			}
> >>> +			printf("\n");
> >>> +		}
> >>>   		if (fd.qbuf(buf) && errno != EINVAL) {
> >>>   			fprintf(stderr, "%s: qbuf error\n", __func__);
> >>>   			return QUEUE_ERROR;
> >>> 
> >>> 
> >>> Load vivid, setup a pure white test pattern:
> >>> 
> >>> v4l2-ctl -c test_pattern=6
> >>> 
> >>> Now run v4l2-ctl --stream-user and you'll see:
> >>> 
> >>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> >>> <
> >>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> >>> <
> >>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> >>> < 5.00 fps
> >>> 
> >>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> >>> <
> >>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> >>> 
> >>> The first four dequeued buffers are filled with data, after that the
> >>> returned buffer is empty because vivid is actually writing to different
> >>> memory pages.
> >>> 
> >>> With this patch the first pixel is always non-zero.
> >> 
> >> Good catch. The question is weather we treat that as undefined behavior 
> >> and keep the optimization for 'good applications' or assume that every 
> >> broken userspace code has to be properly handled.
> > 
> > Given how long we've been saying that USERPTR should be replaced by
> > DMABUF, I would consider that any userspace code using USERPTR is broken
> > :-) One could however question whether we were effective at getting that
> > message across...
> 
> Just a reminder that DMABuf is not a replacement for USERPTR. It only
> cover a subset in absence of an allocater for it. There is no clean way
> to allocate a DMAbuf. Notably, memfds (which could have filled the gap)
> are not DMABuf, even though they are they are similar to the buffers
> allocated by vivid or uvcvideo.

You always have the option to use MMAP to allocate buffers on the V4L2
device. What prevents you from doing so and forces usage of USERPTR ?

> >> The good thing is that 
> >> there is still imho no security issue. The physical pages gathered by 
> >> vb2 in worst case belongs to noone else (vb2 is their last user, they 
> >> are not yet returned to free pages pool).
> >> 
> >>> I wonder if it isn't possible to just check the physical address of
> >>> the received user pointer with the physical address of the previous
> >>> user pointer. Or something like that. I'll dig around a bit more.
> >> 
> >> Such check won't be so simple. Pages contiguous in the virtual memory 
> >> won't map to pages contiguous in the physical memory, so you would need 
> >> to check every single memory page. Make no sense. It is better to 
> >> reacquire buffer on every queue operation. This indeed show how broken 
> >> the USERPTR related part of v4l2 API is.
  

Hi Hans,

On 2019-06-11 09:52, Hans Verkuil wrote:
> On 6/7/19 9:43 PM, Nicolas Dufresne wrote:
>> Le vendredi 07 juin 2019 à 16:39 +0200, Marek Szyprowski a écrit :
>>> Hi Hans,
>>>
>>> On 2019-06-07 16:11, Hans Verkuil wrote:
>>>> On 6/7/19 3:55 PM, Marek Szyprowski wrote:
>>>>> On 2019-06-07 15:40, Hans Verkuil wrote:
>>>>>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
>>>>>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
>>>>>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
>>>>>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
>>>>>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
>>>>>>>>>>> Thank you for the patch.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
>>>>>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
>>>>>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
>>>>>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
>>>>>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
>>>>>>>>>>> Could you explain in the commit message why the assumption is not
>>>>>>>>>>> correct ?
>>>>>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
>>>>>>>>>> even though it is not necessarily using the same physical pages for the memory
>>>>>>>>>> that the kernel is still using for it.
>>>>>>>>>>
>>>>>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
>>>>>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
>>>>>>>>>> the original mapping still remains), but this can corrupt userspace memory
>>>>>>>>>> causing the application to crash. It's not quite clear to me how the memory can
>>>>>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
>>>>>>>>>> the sequence of events.
>>>>>>>>>>
>>>>>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
>>>>>>>>> I'm interested, I would really like to know what happens in the mm
>>>>>>>>> subsystem in such case.
>>>>>>>> Here it is:
>>>>>>>>
>>>>>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>>>> index be606e48..9abf41da 100644
>>>>>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
>>>>>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
>>>>>>>>     	return 0;
>>>>>>>>     }
>>>>>>>>
>>>>>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
>>>>>>>>     		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
>>>>>>>>     		unsigned &capture_count)
>>>>>>>>     {
>>>>>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
>>>>>>>>     				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
>>>>>>>>     				buf.s_request_fd(buf_req_fds[req_idx]);
>>>>>>>>     			}
>>>>>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>>>>>> +				printf("\nidx: %d", buf.g_index());
>>>>>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>>>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
>>>>>>>> +					fflush(stdout);
>>>>>>>> +					free(buf.g_userptr(p));
>>>>>>>> +					void *m = calloc(1, q.g_length(p)/2);
>>>>>>>> +
>>>>>>>> +					fail_on_test(m == NULL);
>>>>>>>> +					q.s_userptr(buf.g_index(), p, m);
>>>>>>>> +					printf("new buf[%d]: %p", p, m);
>>>>>>>> +					buf.s_userptr(m, p);
>>>>>>>> +				}
>>>>>>>> +				printf("\n");
>>>>>>>> +			}
>>>>>>>>     			fail_on_test(buf.qbuf(node, q));
>>>>>>>>     			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
>>>>>>>>     			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
>>>>>>>>
>>>>>>>> ...
>>>>>>>> Streaming ioctls:
>>>>>>>>            test read/write: OK
>>>>>>>>            test blocking wait: OK
>>>>>>>>            test MMAP (no poll): OK
>>>>>>>>            test MMAP (select): OK
>>>>>>>>            test MMAP (epoll): OK
>>>>>>>>            Video Capture: Frame #000
>>>>>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
>>>>>>>>            Video Capture: Frame #001
>>>>>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
>>>>>>>>            Video Capture: Frame #002
>>>>>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
>>>>>>>> Aborted
>>>>>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
>>>>>>> then streaming starts and captureBufs is called which basically just calls dqbuf
>>>>>>> and qbuf.
>>>>>>>
>>>>>>> Tomasz pointed out that all the pointers in this log are actually different. That's
>>>>>>> correct, but here is a log where the old and new buf ptr are the same:
>>>>>>>
>>>>>>> Streaming ioctls:
>>>>>>>            test read/write: OK
>>>>>>>            test blocking wait: OK
>>>>>>>            test MMAP (no poll): OK
>>>>>>>            test MMAP (select): OK
>>>>>>>            test MMAP (epoll): OK
>>>>>>>            Video Capture: Frame #000
>>>>>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
>>>>>>>            Video Capture: Frame #001
>>>>>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
>>>>>>>            Video Capture: Frame #002
>>>>>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
>>>>>>>            Video Capture: Frame #003
>>>>>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
>>>>>>> Aborted
>>>>>>>
>>>>>>> It's weird that the first log fails that way: if the pointers are different,
>>>>>>> then vb2 will call get_userptr and it should discover that the buffer isn't
>>>>>>> large enough, causing qbuf to fail. That doesn't seem to happen.
>>>>>> I think that the reason for this corruption is that the memory pool used
>>>>>> by glibc is now large enough for vb2 to think it can map the full length
>>>>>> of the user pointer into memory, even though only the first half is actually
>>>>>> from the buffer that's allocated. When you capture a frame you just overwrite
>>>>>> a random part of the application's memory pool, causing this invalid pointer.
>>>>>>
>>>>>> But that's a matter of garbage in, garbage out. So that's not the issue here.
>>>>>>
>>>>>> The real question is what happens when you free the old buffer, allocate a
>>>>>> new buffer, end up with the same userptr, but it's using one or more different
>>>>>> pages for its memory compared to the mapping that the kernel uses.
>>>>>>
>>>>>> I managed to reproduce this with v4l2-ctl:
>>>>>>
>>>>>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>>>> index 28b2b3b9..8f2ed9b5 100644
>>>>>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>>>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
>>>>>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
>>>>>>     		 * has the size that fits the old resolution and might not
>>>>>>     		 * fit to the new one.
>>>>>>     		 */
>>>>>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
>>>>>> +			printf("\nidx: %d", buf.g_index());
>>>>>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
>>>>>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
>>>>>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
>>>>>> +				fflush(stdout);
>>>>>> +				free(buf.g_userptr(p));
>>>>>> +				void *m = calloc(1, q.g_length(p));
>>>>>> +
>>>>>> +				if (m == NULL)
>>>>>> +					return QUEUE_ERROR;
>>>>>> +				q.s_userptr(buf.g_index(), p, m);
>>>>>> +				if (m == buf.g_userptr(p))
>>>>>> +					printf(" identical new buf");
>>>>>> +				buf.s_userptr(m, p);
>>>>>> +			}
>>>>>> +			printf("\n");
>>>>>> +		}
>>>>>>     		if (fd.qbuf(buf) && errno != EINVAL) {
>>>>>>     			fprintf(stderr, "%s: qbuf error\n", __func__);
>>>>>>     			return QUEUE_ERROR;
>>>>>>
>>>>>>
>>>>>> Load vivid, setup a pure white test pattern:
>>>>>>
>>>>>> v4l2-ctl -c test_pattern=6
>>>>>>
>>>>>> Now run v4l2-ctl --stream-user and you'll see:
>>>>>>
>>>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
>>>>>> <
>>>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
>>>>>> <
>>>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
>>>>>> <
>>>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
>>>>>> <
>>>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
>>>>>> <
>>>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
>>>>>> < 5.00 fps
>>>>>>
>>>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
>>>>>> <
>>>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
>>>>>>
>>>>>> The first four dequeued buffers are filled with data, after that the
>>>>>> returned buffer is empty because vivid is actually writing to different
>>>>>> memory pages.
>>>>>>
>>>>>> With this patch the first pixel is always non-zero.
>>>>> Good catch. The question is weather we treat that as undefined behavior
>>>>> and keep the optimization for 'good applications' or assume that every
>>>>> broken userspace code has to be properly handled. The good thing is that
>>>>> there is still imho no security issue. The physical pages gathered by
>>>> Yeah, that scared me for a bit, but it all looks secure.
>>>>
>>>>> vb2 in worst case belongs to noone else (vb2 is their last user, they
>>>>> are not yet returned to free pages pool).
>>>> I see three options:
>>>>
>>>> 1) just always reacquire the buffer, and if anyone complains about it
>>>>      being slower we point them towards DMABUF.
>>>>
>>>> 2) keep the current behavior, but document it.
>>>>
>>>> 3) as 2), but also add a new buffer flag that forces a reacquire of the
>>>>      buffer. This could be valid for DMABUF as well. E.g.:
>>>>
>>>>      V4L2_BUF_FLAG_REACQUIRE
>>>>
>>>> I'm leaning towards the third option since it won't slow down existing
>>>> implementations, yet if you do change the userptr every time, then you
>>>> can now force this to work safely.
>>> Is there are valid use case for third variant? I would rather go for second.
>>>
>>> There is one more issue related to this. There are many apps which use
>>> either USERPTR or DMAbuf, but in a bit odd way: they use the same
>>> buffers all the time, but they ignore buf->index and never match it to
>>> respective buffer pointers or fds. This makes the current caching
>>> mechanism useless. Maybe it would make a bit sense do rewrite the
>>> caching in qbuf to ignore the provided buffer->index?
>> Notably GStreamer, which inherited this issue from a public API design
>> error some 15 years ago. Complaint wise, I don't remember someone
>> complaining about that, so option 1) would simply make the performance
>> consistent for the framework.
> After analyzing the DMABUF behavior in this case I realized that the
> dma_buf framework refcount the mapping, so it won't map again unless
> it's really necessary. So there is (almost) no performance hit for
> DMABUF if users do not match dmabuf fds with the buffer index.

Well, not really. If you consider only the first fs/userptr vs. index 
mismatch, you are right, the mapping for the queued buffer already 
exists are will be reused, but this also means that the mapping for the 
buffer which used that index will be freed. Considering the next calls, 
you will end up with a typical map/unmap pattern what really hits the 
performance.

The question is how to implement a smart caching? If we are talking 
about the gstreamer and v4l2 plugin, which afair doesn't even match the 
number of buffers between source and destination between the pipeline 
elements (for example: codec produces 8 buffers, but scaler operates 
only with 2 buffers).

> So option 1 *would* slow down the USERPTR performance compared to
> the other streaming models.

Best regards
  

Le mardi 11 juin 2019 à 13:24 +0300, Laurent Pinchart a écrit :
> Hi Nicolas,
> 
> On Fri, Jun 07, 2019 at 03:38:39PM -0400, Nicolas Dufresne wrote:
> > Le vendredi 07 juin 2019 à 16:58 +0300, Laurent Pinchart a écrit :
> > > On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> > > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > > > Thank you for the patch.
> > > > > > > > > > 
> > > > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > > > correct ?
> > > > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > > > that the kernel is still using for it.
> > > > > > > > > 
> > > > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > > > the sequence of events.
> > > > > > > > > 
> > > > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > > > subsystem in such case.
> > > > > > > Here it is:
> > > > > > > 
> > > > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > index be606e48..9abf41da 100644
> > > > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > > > >   	return 0;
> > > > > > >   }
> > > > > > > 
> > > > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > > > >   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > > > >   		unsigned &capture_count)
> > > > > > >   {
> > > > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > >   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > > > >   				buf.s_request_fd(buf_req_fds[req_idx]);
> > > > > > >   			}
> > > > > > > +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > +				printf("\nidx: %d", buf.g_index());
> > > > > > > +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > > > +					fflush(stdout);
> > > > > > > +					free(buf.g_userptr(p));
> > > > > > > +					void *m = calloc(1, q.g_length(p)/2);
> > > > > > > +
> > > > > > > +					fail_on_test(m == NULL);
> > > > > > > +					q.s_userptr(buf.g_index(), p, m);
> > > > > > > +					printf("new buf[%d]: %p", p, m);
> > > > > > > +					buf.s_userptr(m, p);
> > > > > > > +				}
> > > > > > > +				printf("\n");
> > > > > > > +			}
> > > > > > >   			fail_on_test(buf.qbuf(node, q));
> > > > > > >   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > > > >   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > > > 
> > > > > > > ...
> > > > > > > Streaming ioctls:
> > > > > > >          test read/write: OK
> > > > > > >          test blocking wait: OK
> > > > > > >          test MMAP (no poll): OK
> > > > > > >          test MMAP (select): OK
> > > > > > >          test MMAP (epoll): OK
> > > > > > >          Video Capture: Frame #000
> > > > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > > > >          Video Capture: Frame #001
> > > > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > > > >          Video Capture: Frame #002
> > > > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > > > Aborted
> > > > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > > > and qbuf.
> > > > > > 
> > > > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > > > 
> > > > > > Streaming ioctls:
> > > > > >          test read/write: OK
> > > > > >          test blocking wait: OK
> > > > > >          test MMAP (no poll): OK
> > > > > >          test MMAP (select): OK
> > > > > >          test MMAP (epoll): OK
> > > > > >          Video Capture: Frame #000
> > > > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > > > >          Video Capture: Frame #001
> > > > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > > > >          Video Capture: Frame #002
> > > > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > > > >          Video Capture: Frame #003
> > > > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > > > Aborted
> > > > > > 
> > > > > > It's weird that the first log fails that way: if the pointers are different,
> > > > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > > > I think that the reason for this corruption is that the memory pool used
> > > > > by glibc is now large enough for vb2 to think it can map the full length
> > > > > of the user pointer into memory, even though only the first half is actually
> > > > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > > > a random part of the application's memory pool, causing this invalid pointer.
> > > > > 
> > > > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > > > 
> > > > > The real question is what happens when you free the old buffer, allocate a
> > > > > new buffer, end up with the same userptr, but it's using one or more different
> > > > > pages for its memory compared to the mapping that the kernel uses.
> > > > > 
> > > > > I managed to reproduce this with v4l2-ctl:
> > > > > 
> > > > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > index 28b2b3b9..8f2ed9b5 100644
> > > > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > > > >   		 * has the size that fits the old resolution and might not
> > > > >   		 * fit to the new one.
> > > > >   		 */
> > > > > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > +			printf("\nidx: %d", buf.g_index());
> > > > > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > > > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > > > +				fflush(stdout);
> > > > > +				free(buf.g_userptr(p));
> > > > > +				void *m = calloc(1, q.g_length(p));
> > > > > +
> > > > > +				if (m == NULL)
> > > > > +					return QUEUE_ERROR;
> > > > > +				q.s_userptr(buf.g_index(), p, m);
> > > > > +				if (m == buf.g_userptr(p))
> > > > > +					printf(" identical new buf");
> > > > > +				buf.s_userptr(m, p);
> > > > > +			}
> > > > > +			printf("\n");
> > > > > +		}
> > > > >   		if (fd.qbuf(buf) && errno != EINVAL) {
> > > > >   			fprintf(stderr, "%s: qbuf error\n", __func__);
> > > > >   			return QUEUE_ERROR;
> > > > > 
> > > > > 
> > > > > Load vivid, setup a pure white test pattern:
> > > > > 
> > > > > v4l2-ctl -c test_pattern=6
> > > > > 
> > > > > Now run v4l2-ctl --stream-user and you'll see:
> > > > > 
> > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > > > <
> > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > > > <
> > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > > > <
> > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > > > <
> > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > > > <
> > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > > > < 5.00 fps
> > > > > 
> > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > > > <
> > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > > > 
> > > > > The first four dequeued buffers are filled with data, after that the
> > > > > returned buffer is empty because vivid is actually writing to different
> > > > > memory pages.
> > > > > 
> > > > > With this patch the first pixel is always non-zero.
> > > > 
> > > > Good catch. The question is weather we treat that as undefined behavior 
> > > > and keep the optimization for 'good applications' or assume that every 
> > > > broken userspace code has to be properly handled.
> > > 
> > > Given how long we've been saying that USERPTR should be replaced by
> > > DMABUF, I would consider that any userspace code using USERPTR is broken
> > > :-) One could however question whether we were effective at getting that
> > > message across...
> > 
> > Just a reminder that DMABuf is not a replacement for USERPTR. It only
> > cover a subset in absence of an allocater for it. There is no clean way
> > to allocate a DMAbuf. Notably, memfds (which could have filled the gap)
> > are not DMABuf, even though they are they are similar to the buffers
> > allocated by vivid or uvcvideo.
> 
> You always have the option to use MMAP to allocate buffers on the V4L2
> device. What prevents you from doing so and forces usage of USERPTR ?

If you use MMAP on one v4l2 device, how do you import that into another
v4l2 device ? Now, let's say your source is not a v4l2 device, and uses
virtual memory, how does DMABuf replaces such a use case if you want to
avoid copies and you know your HW can support fast usage of these
randomly allocated buffers ?

> 
> > > > The good thing is that 
> > > > there is still imho no security issue. The physical pages gathered by 
> > > > vb2 in worst case belongs to noone else (vb2 is their last user, they 
> > > > are not yet returned to free pages pool).
> > > > 
> > > > > I wonder if it isn't possible to just check the physical address of
> > > > > the received user pointer with the physical address of the previous
> > > > > user pointer. Or something like that. I'll dig around a bit more.
> > > > 
> > > > Such check won't be so simple. Pages contiguous in the virtual memory 
> > > > won't map to pages contiguous in the physical memory, so you would need 
> > > > to check every single memory page. Make no sense. It is better to 
> > > > reacquire buffer on every queue operation. This indeed show how broken 
> > > > the USERPTR related part of v4l2 API is.
  

Le mardi 11 juin 2019 à 13:56 +0200, Marek Szyprowski a écrit :
> Hi Hans,
> 
> On 2019-06-11 09:52, Hans Verkuil wrote:
> > On 6/7/19 9:43 PM, Nicolas Dufresne wrote:
> > > Le vendredi 07 juin 2019 à 16:39 +0200, Marek Szyprowski a écrit :
> > > > Hi Hans,
> > > > 
> > > > On 2019-06-07 16:11, Hans Verkuil wrote:
> > > > > On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> > > > > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > > > > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > > > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > > > > > Thank you for the patch.
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > > > > > correct ?
> > > > > > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > > > > > that the kernel is still using for it.
> > > > > > > > > > > 
> > > > > > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > > > > > the sequence of events.
> > > > > > > > > > > 
> > > > > > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > > > > > subsystem in such case.
> > > > > > > > > Here it is:
> > > > > > > > > 
> > > > > > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > index be606e48..9abf41da 100644
> > > > > > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > > > > > >     	return 0;
> > > > > > > > >     }
> > > > > > > > > 
> > > > > > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > > > > > >     		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > > > > > >     		unsigned &capture_count)
> > > > > > > > >     {
> > > > > > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > >     				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > > > > > >     				buf.s_request_fd(buf_req_fds[req_idx]);
> > > > > > > > >     			}
> > > > > > > > > +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > > > +				printf("\nidx: %d", buf.g_index());
> > > > > > > > > +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > > > +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > > > > > +					fflush(stdout);
> > > > > > > > > +					free(buf.g_userptr(p));
> > > > > > > > > +					void *m = calloc(1, q.g_length(p)/2);
> > > > > > > > > +
> > > > > > > > > +					fail_on_test(m == NULL);
> > > > > > > > > +					q.s_userptr(buf.g_index(), p, m);
> > > > > > > > > +					printf("new buf[%d]: %p", p, m);
> > > > > > > > > +					buf.s_userptr(m, p);
> > > > > > > > > +				}
> > > > > > > > > +				printf("\n");
> > > > > > > > > +			}
> > > > > > > > >     			fail_on_test(buf.qbuf(node, q));
> > > > > > > > >     			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > > > > > >     			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > > > > > 
> > > > > > > > > ...
> > > > > > > > > Streaming ioctls:
> > > > > > > > >            test read/write: OK
> > > > > > > > >            test blocking wait: OK
> > > > > > > > >            test MMAP (no poll): OK
> > > > > > > > >            test MMAP (select): OK
> > > > > > > > >            test MMAP (epoll): OK
> > > > > > > > >            Video Capture: Frame #000
> > > > > > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > > > > > >            Video Capture: Frame #001
> > > > > > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > > > > > >            Video Capture: Frame #002
> > > > > > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > > > > > Aborted
> > > > > > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > > > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > > > > > and qbuf.
> > > > > > > > 
> > > > > > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > > > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > > > > > 
> > > > > > > > Streaming ioctls:
> > > > > > > >            test read/write: OK
> > > > > > > >            test blocking wait: OK
> > > > > > > >            test MMAP (no poll): OK
> > > > > > > >            test MMAP (select): OK
> > > > > > > >            test MMAP (epoll): OK
> > > > > > > >            Video Capture: Frame #000
> > > > > > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > > > > > >            Video Capture: Frame #001
> > > > > > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > > > > > >            Video Capture: Frame #002
> > > > > > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > > > > > >            Video Capture: Frame #003
> > > > > > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > > > > > Aborted
> > > > > > > > 
> > > > > > > > It's weird that the first log fails that way: if the pointers are different,
> > > > > > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > > > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > > > > > I think that the reason for this corruption is that the memory pool used
> > > > > > > by glibc is now large enough for vb2 to think it can map the full length
> > > > > > > of the user pointer into memory, even though only the first half is actually
> > > > > > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > > > > > a random part of the application's memory pool, causing this invalid pointer.
> > > > > > > 
> > > > > > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > > > > > 
> > > > > > > The real question is what happens when you free the old buffer, allocate a
> > > > > > > new buffer, end up with the same userptr, but it's using one or more different
> > > > > > > pages for its memory compared to the mapping that the kernel uses.
> > > > > > > 
> > > > > > > I managed to reproduce this with v4l2-ctl:
> > > > > > > 
> > > > > > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > index 28b2b3b9..8f2ed9b5 100644
> > > > > > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > > > > > >     		 * has the size that fits the old resolution and might not
> > > > > > >     		 * fit to the new one.
> > > > > > >     		 */
> > > > > > > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > +			printf("\nidx: %d", buf.g_index());
> > > > > > > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > > > > > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > > > > > +				fflush(stdout);
> > > > > > > +				free(buf.g_userptr(p));
> > > > > > > +				void *m = calloc(1, q.g_length(p));
> > > > > > > +
> > > > > > > +				if (m == NULL)
> > > > > > > +					return QUEUE_ERROR;
> > > > > > > +				q.s_userptr(buf.g_index(), p, m);
> > > > > > > +				if (m == buf.g_userptr(p))
> > > > > > > +					printf(" identical new buf");
> > > > > > > +				buf.s_userptr(m, p);
> > > > > > > +			}
> > > > > > > +			printf("\n");
> > > > > > > +		}
> > > > > > >     		if (fd.qbuf(buf) && errno != EINVAL) {
> > > > > > >     			fprintf(stderr, "%s: qbuf error\n", __func__);
> > > > > > >     			return QUEUE_ERROR;
> > > > > > > 
> > > > > > > 
> > > > > > > Load vivid, setup a pure white test pattern:
> > > > > > > 
> > > > > > > v4l2-ctl -c test_pattern=6
> > > > > > > 
> > > > > > > Now run v4l2-ctl --stream-user and you'll see:
> > > > > > > 
> > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > > > > > <
> > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > > > > > < 5.00 fps
> > > > > > > 
> > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > > > > > <
> > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > > > > > 
> > > > > > > The first four dequeued buffers are filled with data, after that the
> > > > > > > returned buffer is empty because vivid is actually writing to different
> > > > > > > memory pages.
> > > > > > > 
> > > > > > > With this patch the first pixel is always non-zero.
> > > > > > Good catch. The question is weather we treat that as undefined behavior
> > > > > > and keep the optimization for 'good applications' or assume that every
> > > > > > broken userspace code has to be properly handled. The good thing is that
> > > > > > there is still imho no security issue. The physical pages gathered by
> > > > > Yeah, that scared me for a bit, but it all looks secure.
> > > > > 
> > > > > > vb2 in worst case belongs to noone else (vb2 is their last user, they
> > > > > > are not yet returned to free pages pool).
> > > > > I see three options:
> > > > > 
> > > > > 1) just always reacquire the buffer, and if anyone complains about it
> > > > >      being slower we point them towards DMABUF.
> > > > > 
> > > > > 2) keep the current behavior, but document it.
> > > > > 
> > > > > 3) as 2), but also add a new buffer flag that forces a reacquire of the
> > > > >      buffer. This could be valid for DMABUF as well. E.g.:
> > > > > 
> > > > >      V4L2_BUF_FLAG_REACQUIRE
> > > > > 
> > > > > I'm leaning towards the third option since it won't slow down existing
> > > > > implementations, yet if you do change the userptr every time, then you
> > > > > can now force this to work safely.
> > > > Is there are valid use case for third variant? I would rather go for second.
> > > > 
> > > > There is one more issue related to this. There are many apps which use
> > > > either USERPTR or DMAbuf, but in a bit odd way: they use the same
> > > > buffers all the time, but they ignore buf->index and never match it to
> > > > respective buffer pointers or fds. This makes the current caching
> > > > mechanism useless. Maybe it would make a bit sense do rewrite the
> > > > caching in qbuf to ignore the provided buffer->index?
> > > Notably GStreamer, which inherited this issue from a public API design
> > > error some 15 years ago. Complaint wise, I don't remember someone
> > > complaining about that, so option 1) would simply make the performance
> > > consistent for the framework.
> > After analyzing the DMABUF behavior in this case I realized that the
> > dma_buf framework refcount the mapping, so it won't map again unless
> > it's really necessary. So there is (almost) no performance hit for
> > DMABUF if users do not match dmabuf fds with the buffer index.
> 
> Well, not really. If you consider only the first fs/userptr vs. index 
> mismatch, you are right, the mapping for the queued buffer already 
> exists are will be reused, but this also means that the mapping for the 
> buffer which used that index will be freed. Considering the next calls, 
> you will end up with a typical map/unmap pattern what really hits the 
> performance.
> 
> The question is how to implement a smart caching? If we are talking 
> about the gstreamer and v4l2 plugin, which afair doesn't even match the 
> number of buffers between source and destination between the pipeline 
> elements (for example: codec produces 8 buffers, but scaler operates 
> only with 2 buffers).

We'd match the size if we knew we could match the buffers index, as we
can't, matching it is useless. The CODEC might have 8 buffers, but will
likely only have 2 buffers travelling between the two devices at one
time as most of these buffers are used as reference during the decoding
process.

> 
> > So option 1 *would* slow down the USERPTR performance compared to
> > the other streaming models.
> 
> Best regards
  

Le mardi 11 juin 2019 à 20:12 -0400, Nicolas Dufresne a écrit :
> Le mardi 11 juin 2019 à 13:56 +0200, Marek Szyprowski a écrit :
> > Hi Hans,
> > 
> > On 2019-06-11 09:52, Hans Verkuil wrote:
> > > On 6/7/19 9:43 PM, Nicolas Dufresne wrote:
> > > > Le vendredi 07 juin 2019 à 16:39 +0200, Marek Szyprowski a écrit :
> > > > > Hi Hans,
> > > > > 
> > > > > On 2019-06-07 16:11, Hans Verkuil wrote:
> > > > > > On 6/7/19 3:55 PM, Marek Szyprowski wrote:
> > > > > > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > > > > > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > > > > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > > > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > > > > > > Thank you for the patch.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > > > > > > correct ?
> > > > > > > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > > > > > > that the kernel is still using for it.
> > > > > > > > > > > > 
> > > > > > > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > > > > > > the sequence of events.
> > > > > > > > > > > > 
> > > > > > > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > > > > > > subsystem in such case.
> > > > > > > > > > Here it is:
> > > > > > > > > > 
> > > > > > > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > > index be606e48..9abf41da 100644
> > > > > > > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > > > > > > >     	return 0;
> > > > > > > > > >     }
> > > > > > > > > > 
> > > > > > > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > > > > > > >     		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > > > > > > >     		unsigned &capture_count)
> > > > > > > > > >     {
> > > > > > > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > > >     				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > > > > > > >     				buf.s_request_fd(buf_req_fds[req_idx]);
> > > > > > > > > >     			}
> > > > > > > > > > +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > > > > +				printf("\nidx: %d", buf.g_index());
> > > > > > > > > > +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > > > > +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > > > > > > +					fflush(stdout);
> > > > > > > > > > +					free(buf.g_userptr(p));
> > > > > > > > > > +					void *m = calloc(1, q.g_length(p)/2);
> > > > > > > > > > +
> > > > > > > > > > +					fail_on_test(m == NULL);
> > > > > > > > > > +					q.s_userptr(buf.g_index(), p, m);
> > > > > > > > > > +					printf("new buf[%d]: %p", p, m);
> > > > > > > > > > +					buf.s_userptr(m, p);
> > > > > > > > > > +				}
> > > > > > > > > > +				printf("\n");
> > > > > > > > > > +			}
> > > > > > > > > >     			fail_on_test(buf.qbuf(node, q));
> > > > > > > > > >     			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > > > > > > >     			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > > > > > > 
> > > > > > > > > > ...
> > > > > > > > > > Streaming ioctls:
> > > > > > > > > >            test read/write: OK
> > > > > > > > > >            test blocking wait: OK
> > > > > > > > > >            test MMAP (no poll): OK
> > > > > > > > > >            test MMAP (select): OK
> > > > > > > > > >            test MMAP (epoll): OK
> > > > > > > > > >            Video Capture: Frame #000
> > > > > > > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > > > > > > >            Video Capture: Frame #001
> > > > > > > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > > > > > > >            Video Capture: Frame #002
> > > > > > > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > > > > > > Aborted
> > > > > > > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > > > > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > > > > > > and qbuf.
> > > > > > > > > 
> > > > > > > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > > > > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > > > > > > 
> > > > > > > > > Streaming ioctls:
> > > > > > > > >            test read/write: OK
> > > > > > > > >            test blocking wait: OK
> > > > > > > > >            test MMAP (no poll): OK
> > > > > > > > >            test MMAP (select): OK
> > > > > > > > >            test MMAP (epoll): OK
> > > > > > > > >            Video Capture: Frame #000
> > > > > > > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > > > > > > >            Video Capture: Frame #001
> > > > > > > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > > > > > > >            Video Capture: Frame #002
> > > > > > > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > > > > > > >            Video Capture: Frame #003
> > > > > > > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > > > > > > Aborted
> > > > > > > > > 
> > > > > > > > > It's weird that the first log fails that way: if the pointers are different,
> > > > > > > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > > > > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > > > > > > I think that the reason for this corruption is that the memory pool used
> > > > > > > > by glibc is now large enough for vb2 to think it can map the full length
> > > > > > > > of the user pointer into memory, even though only the first half is actually
> > > > > > > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > > > > > > a random part of the application's memory pool, causing this invalid pointer.
> > > > > > > > 
> > > > > > > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > > > > > > 
> > > > > > > > The real question is what happens when you free the old buffer, allocate a
> > > > > > > > new buffer, end up with the same userptr, but it's using one or more different
> > > > > > > > pages for its memory compared to the mapping that the kernel uses.
> > > > > > > > 
> > > > > > > > I managed to reproduce this with v4l2-ctl:
> > > > > > > > 
> > > > > > > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > > index 28b2b3b9..8f2ed9b5 100644
> > > > > > > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > > > > > > >     		 * has the size that fits the old resolution and might not
> > > > > > > >     		 * fit to the new one.
> > > > > > > >     		 */
> > > > > > > > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > > +			printf("\nidx: %d", buf.g_index());
> > > > > > > > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > > > > > > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > > > > > > +				fflush(stdout);
> > > > > > > > +				free(buf.g_userptr(p));
> > > > > > > > +				void *m = calloc(1, q.g_length(p));
> > > > > > > > +
> > > > > > > > +				if (m == NULL)
> > > > > > > > +					return QUEUE_ERROR;
> > > > > > > > +				q.s_userptr(buf.g_index(), p, m);
> > > > > > > > +				if (m == buf.g_userptr(p))
> > > > > > > > +					printf(" identical new buf");
> > > > > > > > +				buf.s_userptr(m, p);
> > > > > > > > +			}
> > > > > > > > +			printf("\n");
> > > > > > > > +		}
> > > > > > > >     		if (fd.qbuf(buf) && errno != EINVAL) {
> > > > > > > >     			fprintf(stderr, "%s: qbuf error\n", __func__);
> > > > > > > >     			return QUEUE_ERROR;
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Load vivid, setup a pure white test pattern:
> > > > > > > > 
> > > > > > > > v4l2-ctl -c test_pattern=6
> > > > > > > > 
> > > > > > > > Now run v4l2-ctl --stream-user and you'll see:
> > > > > > > > 
> > > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > > > > > > <
> > > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > > > > > > < 5.00 fps
> > > > > > > > 
> > > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > > > > > > <
> > > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > > > > > > 
> > > > > > > > The first four dequeued buffers are filled with data, after that the
> > > > > > > > returned buffer is empty because vivid is actually writing to different
> > > > > > > > memory pages.
> > > > > > > > 
> > > > > > > > With this patch the first pixel is always non-zero.
> > > > > > > Good catch. The question is weather we treat that as undefined behavior
> > > > > > > and keep the optimization for 'good applications' or assume that every
> > > > > > > broken userspace code has to be properly handled. The good thing is that
> > > > > > > there is still imho no security issue. The physical pages gathered by
> > > > > > Yeah, that scared me for a bit, but it all looks secure.
> > > > > > 
> > > > > > > vb2 in worst case belongs to noone else (vb2 is their last user, they
> > > > > > > are not yet returned to free pages pool).
> > > > > > I see three options:
> > > > > > 
> > > > > > 1) just always reacquire the buffer, and if anyone complains about it
> > > > > >      being slower we point them towards DMABUF.
> > > > > > 
> > > > > > 2) keep the current behavior, but document it.
> > > > > > 
> > > > > > 3) as 2), but also add a new buffer flag that forces a reacquire of the
> > > > > >      buffer. This could be valid for DMABUF as well. E.g.:
> > > > > > 
> > > > > >      V4L2_BUF_FLAG_REACQUIRE
> > > > > > 
> > > > > > I'm leaning towards the third option since it won't slow down existing
> > > > > > implementations, yet if you do change the userptr every time, then you
> > > > > > can now force this to work safely.
> > > > > Is there are valid use case for third variant? I would rather go for second.
> > > > > 
> > > > > There is one more issue related to this. There are many apps which use
> > > > > either USERPTR or DMAbuf, but in a bit odd way: they use the same
> > > > > buffers all the time, but they ignore buf->index and never match it to
> > > > > respective buffer pointers or fds. This makes the current caching
> > > > > mechanism useless. Maybe it would make a bit sense do rewrite the
> > > > > caching in qbuf to ignore the provided buffer->index?
> > > > Notably GStreamer, which inherited this issue from a public API design
> > > > error some 15 years ago. Complaint wise, I don't remember someone
> > > > complaining about that, so option 1) would simply make the performance
> > > > consistent for the framework.
> > > After analyzing the DMABUF behavior in this case I realized that the
> > > dma_buf framework refcount the mapping, so it won't map again unless
> > > it's really necessary. So there is (almost) no performance hit for
> > > DMABUF if users do not match dmabuf fds with the buffer index.
> > 
> > Well, not really. If you consider only the first fs/userptr vs. index 
> > mismatch, you are right, the mapping for the queued buffer already 
> > exists are will be reused, but this also means that the mapping for the 
> > buffer which used that index will be freed. Considering the next calls, 
> > you will end up with a typical map/unmap pattern what really hits the 
> > performance.
> > 
> > The question is how to implement a smart caching? If we are talking 
> > about the gstreamer and v4l2 plugin, which afair doesn't even match the 
> > number of buffers between source and destination between the pipeline 
> > elements (for example: codec produces 8 buffers, but scaler operates 
> > only with 2 buffers).
> 
> We'd match the size if we knew we could match the buffers index, as we
> can't, matching it is useless. The CODEC might have 8 buffers, but will
> likely only have 2 buffers travelling between the two devices at one
> time as most of these buffers are used as reference during the decoding
> process.

I guess you wanted to highlight that caching on the buffer objects
requires userspace to have a specific usage of the API. So even if
there was cross-buffer caching, it's it's limited to the queue size, it
won't work for current implementation in GStreamer.

> 
> > > So option 1 *would* slow down the USERPTR performance compared to
> > > the other streaming models.
> > 
> > Best regards
  

Hi Nicolas,

On Tue, Jun 11, 2019 at 08:09:13PM -0400, Nicolas Dufresne wrote:
> Le mardi 11 juin 2019 à 13:24 +0300, Laurent Pinchart a écrit :
> > On Fri, Jun 07, 2019 at 03:38:39PM -0400, Nicolas Dufresne wrote:
> >> Le vendredi 07 juin 2019 à 16:58 +0300, Laurent Pinchart a écrit :
> >>> On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> >>>> On 2019-06-07 15:40, Hans Verkuil wrote:
> >>>>> On 6/7/19 2:47 PM, Hans Verkuil wrote:
> >>>>>> On 6/7/19 2:23 PM, Hans Verkuil wrote:
> >>>>>>> On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> >>>>>>>> On 2019-06-07 14:01, Hans Verkuil wrote:
> >>>>>>>>> On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> >>>>>>>>>> Thank you for the patch.
> >>>>>>>>>> 
> >>>>>>>>>> On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> >>>>>>>>>>> The __prepare_userptr() function made the incorrect assumption that if the
> >>>>>>>>>>> same user pointer was used as the last one for which memory was acquired, then
> >>>>>>>>>>> there was no need to re-acquire the memory. This assumption was never properly
> >>>>>>>>>>> tested, and after doing that it became clear that this was in fact wrong.
> >>>>>>>>>> Could you explain in the commit message why the assumption is not
> >>>>>>>>>> correct ?
> >>>>>>>>> You can free the memory, then allocate it again and you can get the same pointer,
> >>>>>>>>> even though it is not necessarily using the same physical pages for the memory
> >>>>>>>>> that the kernel is still using for it.
> >>>>>>>>> 
> >>>>>>>>> Worse, you can free the memory, then allocate only half the memory you need and
> >>>>>>>>> get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> >>>>>>>>> the original mapping still remains), but this can corrupt userspace memory
> >>>>>>>>> causing the application to crash. It's not quite clear to me how the memory can
> >>>>>>>>> get corrupted. I don't know enough of those low-level mm internals to understand
> >>>>>>>>> the sequence of events.
> >>>>>>>>> 
> >>>>>>>>> I have test code for v4l2-compliance available if someone wants to test this.
> >>>>>>>> I'm interested, I would really like to know what happens in the mm
> >>>>>>>> subsystem in such case.
> >>>>>>> Here it is:
> >>>>>>> 
> >>>>>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>>>> index be606e48..9abf41da 100644
> >>>>>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> >>>>>>> @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> >>>>>>>   	return 0;
> >>>>>>>   }
> >>>>>>> 
> >>>>>>> -static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>>>> +static int captureBufs(struct node *node, cv4l_queue &q,
> >>>>>>>   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> >>>>>>>   		unsigned &capture_count)
> >>>>>>>   {
> >>>>>>> @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> >>>>>>>   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> >>>>>>>   				buf.s_request_fd(buf_req_fds[req_idx]);
> >>>>>>>   			}
> >>>>>>> +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>>>>>> +				printf("\nidx: %d", buf.g_index());
> >>>>>>> +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>>>>>> +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> >>>>>>> +					fflush(stdout);
> >>>>>>> +					free(buf.g_userptr(p));
> >>>>>>> +					void *m = calloc(1, q.g_length(p)/2);
> >>>>>>> +
> >>>>>>> +					fail_on_test(m == NULL);
> >>>>>>> +					q.s_userptr(buf.g_index(), p, m);
> >>>>>>> +					printf("new buf[%d]: %p", p, m);
> >>>>>>> +					buf.s_userptr(m, p);
> >>>>>>> +				}
> >>>>>>> +				printf("\n");
> >>>>>>> +			}
> >>>>>>>   			fail_on_test(buf.qbuf(node, q));
> >>>>>>>   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> >>>>>>>   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> >>>>>>> 
> >>>>>>> ...
> >>>>>>> Streaming ioctls:
> >>>>>>>          test read/write: OK
> >>>>>>>          test blocking wait: OK
> >>>>>>>          test MMAP (no poll): OK
> >>>>>>>          test MMAP (select): OK
> >>>>>>>          test MMAP (epoll): OK
> >>>>>>>          Video Capture: Frame #000
> >>>>>>> idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> >>>>>>>          Video Capture: Frame #001
> >>>>>>> idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> >>>>>>>          Video Capture: Frame #002
> >>>>>>> idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> >>>>>>> Aborted
> >>>>>> To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> >>>>>> then streaming starts and captureBufs is called which basically just calls dqbuf
> >>>>>> and qbuf.
> >>>>>> 
> >>>>>> Tomasz pointed out that all the pointers in this log are actually different. That's
> >>>>>> correct, but here is a log where the old and new buf ptr are the same:
> >>>>>> 
> >>>>>> Streaming ioctls:
> >>>>>>          test read/write: OK
> >>>>>>          test blocking wait: OK
> >>>>>>          test MMAP (no poll): OK
> >>>>>>          test MMAP (select): OK
> >>>>>>          test MMAP (epoll): OK
> >>>>>>          Video Capture: Frame #000
> >>>>>> idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> >>>>>>          Video Capture: Frame #001
> >>>>>> idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> >>>>>>          Video Capture: Frame #002
> >>>>>> idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> >>>>>>          Video Capture: Frame #003
> >>>>>> idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> >>>>>> Aborted
> >>>>>> 
> >>>>>> It's weird that the first log fails that way: if the pointers are different,
> >>>>>> then vb2 will call get_userptr and it should discover that the buffer isn't
> >>>>>> large enough, causing qbuf to fail. That doesn't seem to happen.
> >>>>> I think that the reason for this corruption is that the memory pool used
> >>>>> by glibc is now large enough for vb2 to think it can map the full length
> >>>>> of the user pointer into memory, even though only the first half is actually
> >>>>> from the buffer that's allocated. When you capture a frame you just overwrite
> >>>>> a random part of the application's memory pool, causing this invalid pointer.
> >>>>> 
> >>>>> But that's a matter of garbage in, garbage out. So that's not the issue here.
> >>>>> 
> >>>>> The real question is what happens when you free the old buffer, allocate a
> >>>>> new buffer, end up with the same userptr, but it's using one or more different
> >>>>> pages for its memory compared to the mapping that the kernel uses.
> >>>>> 
> >>>>> I managed to reproduce this with v4l2-ctl:
> >>>>> 
> >>>>> diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>>>> index 28b2b3b9..8f2ed9b5 100644
> >>>>> --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>>>> +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> >>>>> @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> >>>>>   		 * has the size that fits the old resolution and might not
> >>>>>   		 * fit to the new one.
> >>>>>   		 */
> >>>>> +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> >>>>> +			printf("\nidx: %d", buf.g_index());
> >>>>> +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> >>>>> +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> >>>>> +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> >>>>> +				fflush(stdout);
> >>>>> +				free(buf.g_userptr(p));
> >>>>> +				void *m = calloc(1, q.g_length(p));
> >>>>> +
> >>>>> +				if (m == NULL)
> >>>>> +					return QUEUE_ERROR;
> >>>>> +				q.s_userptr(buf.g_index(), p, m);
> >>>>> +				if (m == buf.g_userptr(p))
> >>>>> +					printf(" identical new buf");
> >>>>> +				buf.s_userptr(m, p);
> >>>>> +			}
> >>>>> +			printf("\n");
> >>>>> +		}
> >>>>>   		if (fd.qbuf(buf) && errno != EINVAL) {
> >>>>>   			fprintf(stderr, "%s: qbuf error\n", __func__);
> >>>>>   			return QUEUE_ERROR;
> >>>>> 
> >>>>> 
> >>>>> Load vivid, setup a pure white test pattern:
> >>>>> 
> >>>>> v4l2-ctl -c test_pattern=6
> >>>>> 
> >>>>> Now run v4l2-ctl --stream-user and you'll see:
> >>>>> 
> >>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> >>>>> <
> >>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> >>>>> <
> >>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> >>>>> <
> >>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> >>>>> <
> >>>>> idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> >>>>> <
> >>>>> idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> >>>>> < 5.00 fps
> >>>>> 
> >>>>> idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> >>>>> <
> >>>>> idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> >>>>> 
> >>>>> The first four dequeued buffers are filled with data, after that the
> >>>>> returned buffer is empty because vivid is actually writing to different
> >>>>> memory pages.
> >>>>> 
> >>>>> With this patch the first pixel is always non-zero.
> >>>> 
> >>>> Good catch. The question is weather we treat that as undefined behavior 
> >>>> and keep the optimization for 'good applications' or assume that every 
> >>>> broken userspace code has to be properly handled.
> >>> 
> >>> Given how long we've been saying that USERPTR should be replaced by
> >>> DMABUF, I would consider that any userspace code using USERPTR is broken
> >>> :-) One could however question whether we were effective at getting that
> >>> message across...
> >> 
> >> Just a reminder that DMABuf is not a replacement for USERPTR. It only
> >> cover a subset in absence of an allocater for it. There is no clean way
> >> to allocate a DMAbuf. Notably, memfds (which could have filled the gap)
> >> are not DMABuf, even though they are they are similar to the buffers
> >> allocated by vivid or uvcvideo.
> > 
> > You always have the option to use MMAP to allocate buffers on the V4L2
> > device. What prevents you from doing so and forces usage of USERPTR ?
> 
> If you use MMAP on one v4l2 device, how do you import that into another
> v4l2 device ?

You can simply export the MMAP buffers on the V4L2 device that has
allocated them, and use DMABUF on the importing device.

> Now, let's say your source is not a v4l2 device, and uses virtual
> memory, how does DMABuf replaces such a use case if you want to avoid
> copies and you know your HW can support fast usage of these randomly
> allocated buffers ?

For this use case you should allocate buffers on the sink, mmap them,
and use the mapped memory on the source side. I agree that not all
sources may support this mode of operation, but that's a design issue
with the source. If we had a dmabuf allocator your problem wouldn't be
solved, as the source would still need to be modified to use it.

> >>>> The good thing is that 
> >>>> there is still imho no security issue. The physical pages gathered by 
> >>>> vb2 in worst case belongs to noone else (vb2 is their last user, they 
> >>>> are not yet returned to free pages pool).
> >>>> 
> >>>>> I wonder if it isn't possible to just check the physical address of
> >>>>> the received user pointer with the physical address of the previous
> >>>>> user pointer. Or something like that. I'll dig around a bit more.
> >>>> 
> >>>> Such check won't be so simple. Pages contiguous in the virtual memory 
> >>>> won't map to pages contiguous in the physical memory, so you would need 
> >>>> to check every single memory page. Make no sense. It is better to 
> >>>> reacquire buffer on every queue operation. This indeed show how broken 
> >>>> the USERPTR related part of v4l2 API is.
  

Le mercredi 12 juin 2019 à 11:17 +0300, Laurent Pinchart a écrit :
> Hi Nicolas,
> 
> On Tue, Jun 11, 2019 at 08:09:13PM -0400, Nicolas Dufresne wrote:
> > Le mardi 11 juin 2019 à 13:24 +0300, Laurent Pinchart a écrit :
> > > On Fri, Jun 07, 2019 at 03:38:39PM -0400, Nicolas Dufresne wrote:
> > > > Le vendredi 07 juin 2019 à 16:58 +0300, Laurent Pinchart a écrit :
> > > > > On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> > > > > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > > > > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > > > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > > > > > Thank you for the patch.
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > > > > > correct ?
> > > > > > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > > > > > that the kernel is still using for it.
> > > > > > > > > > > 
> > > > > > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > > > > > the sequence of events.
> > > > > > > > > > > 
> > > > > > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > > > > > subsystem in such case.
> > > > > > > > > Here it is:
> > > > > > > > > 
> > > > > > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > index be606e48..9abf41da 100644
> > > > > > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > > > > > >   	return 0;
> > > > > > > > >   }
> > > > > > > > > 
> > > > > > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > > > > > >   		const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > > > > > >   		unsigned &capture_count)
> > > > > > > > >   {
> > > > > > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > >   				buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > > > > > >   				buf.s_request_fd(buf_req_fds[req_idx]);
> > > > > > > > >   			}
> > > > > > > > > +			if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > > > +				printf("\nidx: %d", buf.g_index());
> > > > > > > > > +				for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > > > +					printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > > > > > +					fflush(stdout);
> > > > > > > > > +					free(buf.g_userptr(p));
> > > > > > > > > +					void *m = calloc(1, q.g_length(p)/2);
> > > > > > > > > +
> > > > > > > > > +					fail_on_test(m == NULL);
> > > > > > > > > +					q.s_userptr(buf.g_index(), p, m);
> > > > > > > > > +					printf("new buf[%d]: %p", p, m);
> > > > > > > > > +					buf.s_userptr(m, p);
> > > > > > > > > +				}
> > > > > > > > > +				printf("\n");
> > > > > > > > > +			}
> > > > > > > > >   			fail_on_test(buf.qbuf(node, q));
> > > > > > > > >   			fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > > > > > >   			if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > > > > > 
> > > > > > > > > ...
> > > > > > > > > Streaming ioctls:
> > > > > > > > >          test read/write: OK
> > > > > > > > >          test blocking wait: OK
> > > > > > > > >          test MMAP (no poll): OK
> > > > > > > > >          test MMAP (select): OK
> > > > > > > > >          test MMAP (epoll): OK
> > > > > > > > >          Video Capture: Frame #000
> > > > > > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > > > > > >          Video Capture: Frame #001
> > > > > > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > > > > > >          Video Capture: Frame #002
> > > > > > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > > > > > Aborted
> > > > > > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > > > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > > > > > and qbuf.
> > > > > > > > 
> > > > > > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > > > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > > > > > 
> > > > > > > > Streaming ioctls:
> > > > > > > >          test read/write: OK
> > > > > > > >          test blocking wait: OK
> > > > > > > >          test MMAP (no poll): OK
> > > > > > > >          test MMAP (select): OK
> > > > > > > >          test MMAP (epoll): OK
> > > > > > > >          Video Capture: Frame #000
> > > > > > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > > > > > >          Video Capture: Frame #001
> > > > > > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > > > > > >          Video Capture: Frame #002
> > > > > > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > > > > > >          Video Capture: Frame #003
> > > > > > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > > > > > Aborted
> > > > > > > > 
> > > > > > > > It's weird that the first log fails that way: if the pointers are different,
> > > > > > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > > > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > > > > > I think that the reason for this corruption is that the memory pool used
> > > > > > > by glibc is now large enough for vb2 to think it can map the full length
> > > > > > > of the user pointer into memory, even though only the first half is actually
> > > > > > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > > > > > a random part of the application's memory pool, causing this invalid pointer.
> > > > > > > 
> > > > > > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > > > > > 
> > > > > > > The real question is what happens when you free the old buffer, allocate a
> > > > > > > new buffer, end up with the same userptr, but it's using one or more different
> > > > > > > pages for its memory compared to the mapping that the kernel uses.
> > > > > > > 
> > > > > > > I managed to reproduce this with v4l2-ctl:
> > > > > > > 
> > > > > > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > index 28b2b3b9..8f2ed9b5 100644
> > > > > > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > > > > > >   		 * has the size that fits the old resolution and might not
> > > > > > >   		 * fit to the new one.
> > > > > > >   		 */
> > > > > > > +		if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > +			printf("\nidx: %d", buf.g_index());
> > > > > > > +			for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > +				unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > > > > > +				printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > > > > > +				fflush(stdout);
> > > > > > > +				free(buf.g_userptr(p));
> > > > > > > +				void *m = calloc(1, q.g_length(p));
> > > > > > > +
> > > > > > > +				if (m == NULL)
> > > > > > > +					return QUEUE_ERROR;
> > > > > > > +				q.s_userptr(buf.g_index(), p, m);
> > > > > > > +				if (m == buf.g_userptr(p))
> > > > > > > +					printf(" identical new buf");
> > > > > > > +				buf.s_userptr(m, p);
> > > > > > > +			}
> > > > > > > +			printf("\n");
> > > > > > > +		}
> > > > > > >   		if (fd.qbuf(buf) && errno != EINVAL) {
> > > > > > >   			fprintf(stderr, "%s: qbuf error\n", __func__);
> > > > > > >   			return QUEUE_ERROR;
> > > > > > > 
> > > > > > > 
> > > > > > > Load vivid, setup a pure white test pattern:
> > > > > > > 
> > > > > > > v4l2-ctl -c test_pattern=6
> > > > > > > 
> > > > > > > Now run v4l2-ctl --stream-user and you'll see:
> > > > > > > 
> > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > > > > > <
> > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > > > > > <
> > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > > > > > < 5.00 fps
> > > > > > > 
> > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > > > > > <
> > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > > > > > 
> > > > > > > The first four dequeued buffers are filled with data, after that the
> > > > > > > returned buffer is empty because vivid is actually writing to different
> > > > > > > memory pages.
> > > > > > > 
> > > > > > > With this patch the first pixel is always non-zero.
> > > > > > 
> > > > > > Good catch. The question is weather we treat that as undefined behavior 
> > > > > > and keep the optimization for 'good applications' or assume that every 
> > > > > > broken userspace code has to be properly handled.
> > > > > 
> > > > > Given how long we've been saying that USERPTR should be replaced by
> > > > > DMABUF, I would consider that any userspace code using USERPTR is broken
> > > > > :-) One could however question whether we were effective at getting that
> > > > > message across...
> > > > 
> > > > Just a reminder that DMABuf is not a replacement for USERPTR. It only
> > > > cover a subset in absence of an allocater for it. There is no clean way
> > > > to allocate a DMAbuf. Notably, memfds (which could have filled the gap)
> > > > are not DMABuf, even though they are they are similar to the buffers
> > > > allocated by vivid or uvcvideo.
> > > 
> > > You always have the option to use MMAP to allocate buffers on the V4L2
> > > device. What prevents you from doing so and forces usage of USERPTR ?
> > 
> > If you use MMAP on one v4l2 device, how do you import that into another
> > v4l2 device ?
> 
> You can simply export the MMAP buffers on the V4L2 device that has
> allocated them, and use DMABUF on the importing device.
> 
> > Now, let's say your source is not a v4l2 device, and uses virtual
> > memory, how does DMABuf replaces such a use case if you want to avoid
> > copies and you know your HW can support fast usage of these randomly
> > allocated buffers ?
> 
> For this use case you should allocate buffers on the sink, mmap them,
> and use the mapped memory on the source side. I agree that not all
> sources may support this mode of operation, but that's a design issue
> with the source. If we had a dmabuf allocator your problem wouldn't be
> solved, as the source would still need to be modified to use it.

I don't think any of this reflection covers the surface of the
restrictions that V4L2 combined queue/allocation impose on userspace.
One of our very common scenarios requires to capture from one source,
and zero-copy that toward multiple sink. One source could be USB driven
(non v4l2), or network socket. Allocating from one random sink isn't
really working, in fact it would lead to ebusy prior to the orphaning
mechanism that was added recently. Since as long as one buffer of a
device is still active, the driver (and the HW behind) cannot be used
anymore.

Over your N sinks, you maybe have zero-copy for some of them, even if
it's foreign allocation, while others will just fallback to mmap/copy,
and that's could be all right for a specific application. But as the
source does not always have a "dmabuf", USERPTR remains the only option
one could try. I don't know how memfd works, but maybe memfd should be
a memory type in replacement to USERPTR ? I'm really not sure what the
replacement, but I'm quite clear that there is no zero-copy replacement
for it atm.

> 
> > > > > > The good thing is that 
> > > > > > there is still imho no security issue. The physical pages gathered by 
> > > > > > vb2 in worst case belongs to noone else (vb2 is their last user, they 
> > > > > > are not yet returned to free pages pool).
> > > > > > 
> > > > > > > I wonder if it isn't possible to just check the physical address of
> > > > > > > the received user pointer with the physical address of the previous
> > > > > > > user pointer. Or something like that. I'll dig around a bit more.
> > > > > > 
> > > > > > Such check won't be so simple. Pages contiguous in the virtual memory 
> > > > > > won't map to pages contiguous in the physical memory, so you would need 
> > > > > > to check every single memory page. Make no sense. It is better to 
> > > > > > reacquire buffer on every queue operation. This indeed show how broken 
> > > > > > the USERPTR related part of v4l2 API is.
  

On Thu, Jun 13, 2019 at 9:21 AM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le mercredi 12 juin 2019 à 11:17 +0300, Laurent Pinchart a écrit :
> > Hi Nicolas,
> >
> > On Tue, Jun 11, 2019 at 08:09:13PM -0400, Nicolas Dufresne wrote:
> > > Le mardi 11 juin 2019 à 13:24 +0300, Laurent Pinchart a écrit :
> > > > On Fri, Jun 07, 2019 at 03:38:39PM -0400, Nicolas Dufresne wrote:
> > > > > Le vendredi 07 juin 2019 à 16:58 +0300, Laurent Pinchart a écrit :
> > > > > > On Fri, Jun 07, 2019 at 03:55:05PM +0200, Marek Szyprowski wrote:
> > > > > > > On 2019-06-07 15:40, Hans Verkuil wrote:
> > > > > > > > On 6/7/19 2:47 PM, Hans Verkuil wrote:
> > > > > > > > > On 6/7/19 2:23 PM, Hans Verkuil wrote:
> > > > > > > > > > On 6/7/19 2:14 PM, Marek Szyprowski wrote:
> > > > > > > > > > > On 2019-06-07 14:01, Hans Verkuil wrote:
> > > > > > > > > > > > On 6/7/19 1:16 PM, Laurent Pinchart wrote:
> > > > > > > > > > > > > Thank you for the patch.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jun 07, 2019 at 10:45:31AM +0200, Hans Verkuil wrote:
> > > > > > > > > > > > > > The __prepare_userptr() function made the incorrect assumption that if the
> > > > > > > > > > > > > > same user pointer was used as the last one for which memory was acquired, then
> > > > > > > > > > > > > > there was no need to re-acquire the memory. This assumption was never properly
> > > > > > > > > > > > > > tested, and after doing that it became clear that this was in fact wrong.
> > > > > > > > > > > > > Could you explain in the commit message why the assumption is not
> > > > > > > > > > > > > correct ?
> > > > > > > > > > > > You can free the memory, then allocate it again and you can get the same pointer,
> > > > > > > > > > > > even though it is not necessarily using the same physical pages for the memory
> > > > > > > > > > > > that the kernel is still using for it.
> > > > > > > > > > > >
> > > > > > > > > > > > Worse, you can free the memory, then allocate only half the memory you need and
> > > > > > > > > > > > get back the same pointer. vb2 wouldn't notice this. And it seems to work (since
> > > > > > > > > > > > the original mapping still remains), but this can corrupt userspace memory
> > > > > > > > > > > > causing the application to crash. It's not quite clear to me how the memory can
> > > > > > > > > > > > get corrupted. I don't know enough of those low-level mm internals to understand
> > > > > > > > > > > > the sequence of events.
> > > > > > > > > > > >
> > > > > > > > > > > > I have test code for v4l2-compliance available if someone wants to test this.
> > > > > > > > > > > I'm interested, I would really like to know what happens in the mm
> > > > > > > > > > > subsystem in such case.
> > > > > > > > > > Here it is:
> > > > > > > > > >
> > > > > > > > > > diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > > index be606e48..9abf41da 100644
> > > > > > > > > > --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > > +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
> > > > > > > > > > @@ -797,7 +797,7 @@ int testReadWrite(struct node *node)
> > > > > > > > > >       return 0;
> > > > > > > > > >   }
> > > > > > > > > >
> > > > > > > > > > -static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > > > +static int captureBufs(struct node *node, cv4l_queue &q,
> > > > > > > > > >               const cv4l_queue &m2m_q, unsigned frame_count, int pollmode,
> > > > > > > > > >               unsigned &capture_count)
> > > > > > > > > >   {
> > > > > > > > > > @@ -962,6 +962,21 @@ static int captureBufs(struct node *node, const cv4l_queue &q,
> > > > > > > > > >                               buf.s_flags(V4L2_BUF_FLAG_REQUEST_FD);
> > > > > > > > > >                               buf.s_request_fd(buf_req_fds[req_idx]);
> > > > > > > > > >                       }
> > > > > > > > > > +                     if (v4l_type_is_capture(buf.g_type()) && q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > > > > +                             printf("\nidx: %d", buf.g_index());
> > > > > > > > > > +                             for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > > > > +                                     printf(" old buf[%d]: %p ", p, buf.g_userptr(p));
> > > > > > > > > > +                                     fflush(stdout);
> > > > > > > > > > +                                     free(buf.g_userptr(p));
> > > > > > > > > > +                                     void *m = calloc(1, q.g_length(p)/2);
> > > > > > > > > > +
> > > > > > > > > > +                                     fail_on_test(m == NULL);
> > > > > > > > > > +                                     q.s_userptr(buf.g_index(), p, m);
> > > > > > > > > > +                                     printf("new buf[%d]: %p", p, m);
> > > > > > > > > > +                                     buf.s_userptr(m, p);
> > > > > > > > > > +                             }
> > > > > > > > > > +                             printf("\n");
> > > > > > > > > > +                     }
> > > > > > > > > >                       fail_on_test(buf.qbuf(node, q));
> > > > > > > > > >                       fail_on_test(buf.g_flags() & V4L2_BUF_FLAG_DONE);
> > > > > > > > > >                       if (buf.g_flags() & V4L2_BUF_FLAG_REQUEST_FD) {
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Load the vivid driver and just run 'v4l2-compliance -s10' and you'll see:
> > > > > > > > > >
> > > > > > > > > > ...
> > > > > > > > > > Streaming ioctls:
> > > > > > > > > >          test read/write: OK
> > > > > > > > > >          test blocking wait: OK
> > > > > > > > > >          test MMAP (no poll): OK
> > > > > > > > > >          test MMAP (select): OK
> > > > > > > > > >          test MMAP (epoll): OK
> > > > > > > > > >          Video Capture: Frame #000
> > > > > > > > > > idx: 0 old buf[0]: 0x7f71c6e7c010 new buf[0]: 0x7f71c6eb4010
> > > > > > > > > >          Video Capture: Frame #001
> > > > > > > > > > idx: 1 old buf[0]: 0x7f71c6e0b010 new buf[0]: 0x7f71c6e7b010
> > > > > > > > > >          Video Capture: Frame #002
> > > > > > > > > > idx: 0 old buf[0]: 0x7f71c6eb4010 free(): invalid pointer
> > > > > > > > > > Aborted
> > > > > > > > > To clarify: two full size buffers are allocated and queued (that happens in setupUserPtr()),
> > > > > > > > > then streaming starts and captureBufs is called which basically just calls dqbuf
> > > > > > > > > and qbuf.
> > > > > > > > >
> > > > > > > > > Tomasz pointed out that all the pointers in this log are actually different. That's
> > > > > > > > > correct, but here is a log where the old and new buf ptr are the same:
> > > > > > > > >
> > > > > > > > > Streaming ioctls:
> > > > > > > > >          test read/write: OK
> > > > > > > > >          test blocking wait: OK
> > > > > > > > >          test MMAP (no poll): OK
> > > > > > > > >          test MMAP (select): OK
> > > > > > > > >          test MMAP (epoll): OK
> > > > > > > > >          Video Capture: Frame #000
> > > > > > > > > idx: 0 old buf[0]: 0x7f1094e16010 new buf[0]: 0x7f1094e4e010
> > > > > > > > >          Video Capture: Frame #001
> > > > > > > > > idx: 1 old buf[0]: 0x7f1094da5010 new buf[0]: 0x7f1094e15010
> > > > > > > > >          Video Capture: Frame #002
> > > > > > > > > idx: 0 old buf[0]: 0x7f1094e4e010 new buf[0]: 0x7f1094e4e010
> > > > > > > > >          Video Capture: Frame #003
> > > > > > > > > idx: 1 old buf[0]: 0x7f1094e15010 free(): invalid pointer
> > > > > > > > > Aborted
> > > > > > > > >
> > > > > > > > > It's weird that the first log fails that way: if the pointers are different,
> > > > > > > > > then vb2 will call get_userptr and it should discover that the buffer isn't
> > > > > > > > > large enough, causing qbuf to fail. That doesn't seem to happen.
> > > > > > > > I think that the reason for this corruption is that the memory pool used
> > > > > > > > by glibc is now large enough for vb2 to think it can map the full length
> > > > > > > > of the user pointer into memory, even though only the first half is actually
> > > > > > > > from the buffer that's allocated. When you capture a frame you just overwrite
> > > > > > > > a random part of the application's memory pool, causing this invalid pointer.
> > > > > > > >
> > > > > > > > But that's a matter of garbage in, garbage out. So that's not the issue here.
> > > > > > > >
> > > > > > > > The real question is what happens when you free the old buffer, allocate a
> > > > > > > > new buffer, end up with the same userptr, but it's using one or more different
> > > > > > > > pages for its memory compared to the mapping that the kernel uses.
> > > > > > > >
> > > > > > > > I managed to reproduce this with v4l2-ctl:
> > > > > > > >
> > > > > > > > diff --git a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > > index 28b2b3b9..8f2ed9b5 100644
> > > > > > > > --- a/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > > +++ b/utils/v4l2-ctl/v4l2-ctl-streaming.cpp
> > > > > > > > @@ -1422,6 +1422,24 @@ static int do_handle_cap(cv4l_fd &fd, cv4l_queue &q, FILE *fout, int *index,
> > > > > > > >                    * has the size that fits the old resolution and might not
> > > > > > > >                    * fit to the new one.
> > > > > > > >                    */
> > > > > > > > +         if (q.g_memory() == V4L2_MEMORY_USERPTR) {
> > > > > > > > +                 printf("\nidx: %d", buf.g_index());
> > > > > > > > +                 for (unsigned p = 0; p < q.g_num_planes(); p++) {
> > > > > > > > +                         unsigned *pb = (unsigned *)buf.g_userptr(p);
> > > > > > > > +                         printf(" old buf[%d]: %p first pixel: 0x%x", p, buf.g_userptr(p), *pb);
> > > > > > > > +                         fflush(stdout);
> > > > > > > > +                         free(buf.g_userptr(p));
> > > > > > > > +                         void *m = calloc(1, q.g_length(p));
> > > > > > > > +
> > > > > > > > +                         if (m == NULL)
> > > > > > > > +                                 return QUEUE_ERROR;
> > > > > > > > +                         q.s_userptr(buf.g_index(), p, m);
> > > > > > > > +                         if (m == buf.g_userptr(p))
> > > > > > > > +                                 printf(" identical new buf");
> > > > > > > > +                         buf.s_userptr(m, p);
> > > > > > > > +                 }
> > > > > > > > +                 printf("\n");
> > > > > > > > +         }
> > > > > > > >                   if (fd.qbuf(buf) && errno != EINVAL) {
> > > > > > > >                           fprintf(stderr, "%s: qbuf error\n", __func__);
> > > > > > > >                           return QUEUE_ERROR;
> > > > > > > >
> > > > > > > >
> > > > > > > > Load vivid, setup a pure white test pattern:
> > > > > > > >
> > > > > > > > v4l2-ctl -c test_pattern=6
> > > > > > > >
> > > > > > > > Now run v4l2-ctl --stream-user and you'll see:
> > > > > > > >
> > > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x80ea80ea identical new buf
> > > > > > > > <
> > > > > > > > idx: 0 old buf[0]: 0x7f91551cb010 first pixel: 0x0 identical new buf
> > > > > > > > <
> > > > > > > > idx: 1 old buf[0]: 0x7f915515a010 first pixel: 0x0 identical new buf
> > > > > > > > < 5.00 fps
> > > > > > > >
> > > > > > > > idx: 2 old buf[0]: 0x7f91550e9010 first pixel: 0x0 identical new buf
> > > > > > > > <
> > > > > > > > idx: 3 old buf[0]: 0x7f9155078010 first pixel: 0x0 identical new buf
> > > > > > > >
> > > > > > > > The first four dequeued buffers are filled with data, after that the
> > > > > > > > returned buffer is empty because vivid is actually writing to different
> > > > > > > > memory pages.
> > > > > > > >
> > > > > > > > With this patch the first pixel is always non-zero.
> > > > > > >
> > > > > > > Good catch. The question is weather we treat that as undefined behavior
> > > > > > > and keep the optimization for 'good applications' or assume that every
> > > > > > > broken userspace code has to be properly handled.
> > > > > >
> > > > > > Given how long we've been saying that USERPTR should be replaced by
> > > > > > DMABUF, I would consider that any userspace code using USERPTR is broken
> > > > > > :-) One could however question whether we were effective at getting that
> > > > > > message across...
> > > > >
> > > > > Just a reminder that DMABuf is not a replacement for USERPTR. It only
> > > > > cover a subset in absence of an allocater for it. There is no clean way
> > > > > to allocate a DMAbuf. Notably, memfds (which could have filled the gap)
> > > > > are not DMABuf, even though they are they are similar to the buffers
> > > > > allocated by vivid or uvcvideo.
> > > >
> > > > You always have the option to use MMAP to allocate buffers on the V4L2
> > > > device. What prevents you from doing so and forces usage of USERPTR ?
> > >
> > > If you use MMAP on one v4l2 device, how do you import that into another
> > > v4l2 device ?
> >
> > You can simply export the MMAP buffers on the V4L2 device that has
> > allocated them, and use DMABUF on the importing device.
> >
> > > Now, let's say your source is not a v4l2 device, and uses virtual
> > > memory, how does DMABuf replaces such a use case if you want to avoid
> > > copies and you know your HW can support fast usage of these randomly
> > > allocated buffers ?
> >
> > For this use case you should allocate buffers on the sink, mmap them,
> > and use the mapped memory on the source side. I agree that not all
> > sources may support this mode of operation, but that's a design issue
> > with the source. If we had a dmabuf allocator your problem wouldn't be
> > solved, as the source would still need to be modified to use it.
>
> I don't think any of this reflection covers the surface of the
> restrictions that V4L2 combined queue/allocation impose on userspace.
> One of our very common scenarios requires to capture from one source,
> and zero-copy that toward multiple sink. One source could be USB driven
> (non v4l2), or network socket. Allocating from one random sink isn't
> really working, in fact it would lead to ebusy prior to the orphaning
> mechanism that was added recently. Since as long as one buffer of a
> device is still active, the driver (and the HW behind) cannot be used
> anymore.
>
> Over your N sinks, you maybe have zero-copy for some of them, even if
> it's foreign allocation, while others will just fallback to mmap/copy,
> and that's could be all right for a specific application. But as the
> source does not always have a "dmabuf", USERPTR remains the only option
> one could try. I don't know how memfd works, but maybe memfd should be
> a memory type in replacement to USERPTR ? I'm really not sure what the
> replacement, but I'm quite clear that there is no zero-copy replacement
> for it atm.

We have udmabuf now:
https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/dma-buf/udmabuf.c