VideoTexture, only first frame shown (gst-launch/v4l2loopback -> Blender) #48692

New Issue

Mario Mey · 2016-06-21T01:34:42+02:00

Mario Mey commented

2016-06-21 01:34:42 +02:00

System
Ubuntu-Mate 14.04 64x, nVidia GT620.

Issue
I made a 3D scene (in BGE) with two VideoTextures: one shows the webcam image, the other one: a v4l2loopback device capturing second-display screen, using gst-launch-1.0. No matter what command I use (*), when BlenderPlayer starts, it only shows the first frame of the v4l2loopback device. Webcam works well.

What I tested:

Using VLC, webcam and any v4l2loopback works OK. I get smooth playback.
Using Blender, webcam and another capture device (EasyCap), also I get smooth playback of both devices.
Using two gst-launch-1.0 instances, both devices are displayed well in different windows, at the same time (I use "ximagesink" instead of "v4l2sink...")

(*) My command:

gst-launch-1.0 ximagesrc startx=1920 starty=0 endx=2943 endy=768 show-pointer=0 use-damage=0 \
! video/x-raw,framerate=30/1 \
! videoscale method=0 add-borders=false \
! video/x-raw,width=640,height=360 \
! videoflip method=horizontal-flip \
! v4l2sink device=/dev/video1

Also, I tried with the simplest one:

gst-launch videotestsrc ! v4l2sink device=/dev/video1

Blend file: 95-video-texture-v4l2loopback.blend

To reproduce the "issue"

Install v4l2loopback module (https://github.com/umlaeute/v4l2loopback)
Connect a webcam.
Run the simple gst-launch command mentioned above.
Open VLC and test both devices.
Close VLC.
Open the blend file in Blender.
Start Game Engine.

What it should do

One object should show the webcam device and, the other, a video-test-pattern with motion.

I also reported in v4l2loopback GitHub issues page: https://github.com/umlaeute/v4l2loopback/issues/115

**System** Ubuntu-Mate 14.04 64x, nVidia GT620. **Issue** I made a 3D scene (in BGE) with two VideoTextures: one shows the webcam image, the other one: a v4l2loopback device capturing second-display screen, using gst-launch-1.0. No matter what command I use (*), when BlenderPlayer starts, it only shows the first frame of the v4l2loopback device. Webcam works well. What I tested: - Using VLC, webcam and any v4l2loopback works OK. I get smooth playback. - Using Blender, webcam and another capture device (EasyCap), also I get smooth playback of both devices. - Using two gst-launch-1.0 instances, both devices are displayed well in different windows, at the same time (I use "ximagesink" instead of "v4l2sink...") (*) My command: ``` gst-launch-1.0 ximagesrc startx=1920 starty=0 endx=2943 endy=768 show-pointer=0 use-damage=0 \ ! video/x-raw,framerate=30/1 \ ! videoscale method=0 add-borders=false \ ! video/x-raw,width=640,height=360 \ ! videoflip method=horizontal-flip \ ! v4l2sink device=/dev/video1 ``` Also, I tried with the simplest one: ``` gst-launch videotestsrc ! v4l2sink device=/dev/video1 ``` Blend file: [95-video-texture-v4l2loopback.blend](https://archive.blender.org/developer/F317915/95-video-texture-v4l2loopback.blend) **To reproduce the "issue"** - Install v4l2loopback module (https://github.com/umlaeute/v4l2loopback) - Connect a webcam. - Run the simple gst-launch command mentioned above. - Open VLC and test both devices. - Close VLC. - Open the blend file in Blender. - Start Game Engine. **What it should do** - One object should show the webcam device and, the other, a video-test-pattern with motion. I also reported in v4l2loopback GitHub issues page: https://github.com/umlaeute/v4l2loopback/issues/115

Mario Mey commented

2016-06-21 01:34:42 +02:00

Changed status to: 'Open'

Benoit Cousson was assigned by Mario Mey

2016-06-21 01:34:42 +02:00

Mario Mey commented

2016-06-21 01:34:42 +02:00

Added subscriber: @MarioSottile

Mario Mey commented

2016-06-21 01:41:20 +02:00

This file works in Blender > 2.66.
The bug is there and all the later versions. Included the last one 2.77a.

This file works in Blender > 2.66. The bug is there and all the later versions. Included the last one 2.77a.

Mario Mey commented

2016-06-22 01:25:09 +02:00

Added subscriber: @BenoitBolsee

Mario Mey commented

2016-06-22 01:25:09 +02:00

I think "this Benoit" was the one that Dalai told me to add here...

Benoit Cousson was unassigned by Mario Mey

2016-06-22 01:25:35 +02:00

Benoit Bolsee was assigned by Mario Mey

2016-06-22 01:25:35 +02:00

Mario Mey commented

2016-06-22 01:25:35 +02:00

Added subscriber: @benoit-2

Sergey Sharybin commented

2016-06-28 11:50:33 +02:00

Added subscriber: @Sergey

Sergey Sharybin commented

2016-06-28 11:50:33 +02:00

@BenoitBolsee, are you looking into this issue?

Benoit Bolsee commented

2016-06-28 13:17:56 +02:00

not yet, I need to get hold of a webcam to reproduce.

Mario Mey commented

2016-06-28 15:22:50 +02:00

@BenoitBolsee, you don't really need a camera to test the bug. The webcam is there to show that /dev/video0 is working and /dev/video1 isn't.

Just comment these lines and run it:

14: monitor_webcam()
27: bge.logic.webcam_texture.refresh(True)

@BenoitBolsee, you don't really need a camera to test the bug. The webcam is there to show that /dev/video0 is working and /dev/video1 isn't. Just comment these lines and run it: ``` 14: monitor_webcam() 27: bge.logic.webcam_texture.refresh(True) ```

Benoit Bolsee commented

2016-06-29 00:07:02 +02:00

Thanks for the instructions; I was able to reproduce the bug.
It appears that multi-thread caching doesn't work well with that particular v4l source.
I was able to get a smooth output by disabling caching.

I will analyze further and determine if I need to fix caching or disable it or make it selectable by API (caching is known to work with webcam source).

Thanks for the instructions; I was able to reproduce the bug. It appears that multi-thread caching doesn't work well with that particular v4l source. I was able to get a smooth output by disabling caching. I will analyze further and determine if I need to fix caching or disable it or make it selectable by API (caching is known to work with webcam source).

Mario Mey commented

2016-06-29 00:14:41 +02:00

Is there any way to disable caching without modifying code and recompiling...?
If not (and if it seems to be stable to do that), could you make a diff (patch) file for me to compile Blender here?

(I don't want to seem anxious, but it would be very-great if I can make it work here during these days... it's for a particular job)

Thanks!

Is there any way to disable caching without modifying code and recompiling...? If not (and if it seems to be stable to do that), could you make a diff (patch) file for me to compile Blender here? (I don't want to seem anxious, but it would be very-great if I can make it work here during these days... it's for a particular job) Thanks!

Benoit Bolsee commented

2016-06-29 09:49:43 +02:00

Cache is enabled automatically as soon as Blender detects that the CPU has more than 1 core.
To disable it you must change the code. The diff is extremely simple: just comment the two line with m_isThreaded = true; in VideoFFmpeg.cpp:

diff --git a/source/gameengine/VideoTexture/VideoFFmpeg.cpp b/source/gameengine/VideoTexture/VideoFFmpeg.cpp
index 083e9e2..5a470be 100644
--- a/source/gameengine/VideoTexture/VideoFFmpeg.cpp
+++ b/source/gameengine/VideoTexture/VideoFFmpeg.cpp
@@ -581,7 +581,7 @@ void VideoFFmpeg::openFile (char *filename)
 	{
 		*never thread image: there are no frame to read ahead* no need to thread if the system has a single core
-		m_isThreaded =  true;
+		//m_isThreaded =  true;
 	}
 }
 
@@ -673,7 +673,7 @@ void VideoFFmpeg::openCam (char *file, short camIdx)
 	if (BLI_system_thread_count() > 1)
 	{
 		// no need to thread if the system has a single core
-		m_isThreaded =  true;
+		//m_isThreaded =  true;
 	}
 
 	av_dict_free(&formatParams);

Cache is enabled automatically as soon as Blender detects that the CPU has more than 1 core. To disable it you must change the code. The diff is extremely simple: just comment the two line with m_isThreaded = true; in VideoFFmpeg.cpp: ``` diff --git a/source/gameengine/VideoTexture/VideoFFmpeg.cpp b/source/gameengine/VideoTexture/VideoFFmpeg.cpp index 083e9e2..5a470be 100644 --- a/source/gameengine/VideoTexture/VideoFFmpeg.cpp +++ b/source/gameengine/VideoTexture/VideoFFmpeg.cpp @@ -581,7 +581,7 @@ void VideoFFmpeg::openFile (char *filename) { *never thread image: there are no frame to read ahead* no need to thread if the system has a single core - m_isThreaded = true; + //m_isThreaded = true; } } @@ -673,7 +673,7 @@ void VideoFFmpeg::openCam (char *file, short camIdx) if (BLI_system_thread_count() > 1) { // no need to thread if the system has a single core - m_isThreaded = true; + //m_isThreaded = true; } av_dict_free(&formatParams); ```

Mario Mey commented

2016-06-30 12:02:31 +02:00

It worked. ;)

For the moment... Is it safe to disable this caching? Just optimization purpose...?

**It worked.** ;) For the moment... Is it safe to disable this caching? Just optimization purpose...?

Benoit Bolsee commented

2016-06-30 13:27:15 +02:00

Yes, caching is just for optimization. It is safe to disable it.

I did more tests and the problem is caused by the v4l2loopback not providing correct timestamp on frames: vt thinks that the frames are in the past and drops them.
There is a flag to fix this: m_isStreaming; this flags tells vt to ignore timestamp. Then caching works but not smoothly: there is a strange jitter in the stream as if the frames were not properly ordered. It seems that the loopback device doesn't implement a proper frame queue.

I'll investigate further but the best option apparently is to make caching optional by API.

Yes, caching is just for optimization. It is safe to disable it. I did more tests and the problem is caused by the v4l2loopback not providing correct timestamp on frames: vt thinks that the frames are in the past and drops them. There is a flag to fix this: m_isStreaming; this flags tells vt to ignore timestamp. Then caching works but not smoothly: there is a strange jitter in the stream as if the frames were not properly ordered. It seems that the loopback device doesn't implement a proper frame queue. I'll investigate further but the best option apparently is to make caching optional by API.

Mario Mey commented

2016-06-30 17:26:13 +02:00

As I reported this same bug in v4l2loopback issues github page, can I add this information that you are telling to me? Is there any other further information to post there?

Maybe v4l2loopback also can be better by providing the correct timestamp on frames.

As I reported this same bug in v4l2loopback issues github page, can I add this information that you are telling to me? Is there any other further information to post there? Maybe v4l2loopback also can be better by providing the correct timestamp on frames.

Benoit Bolsee commented

2016-06-30 19:15:08 +02:00

Yes you can pass that information to them. They will probably find useful to look at the source code that's doing the caching. It's all in the VideoFFmpeg::cacheThread() function.

I noticed the following anomalies:
av_read_frame() never returns < 0, which would mean 'no frame available at this time' no matter how frequently I call this function and despite the fact that the AVFMT_FLAG_NONBLOCK flag is set in the format context. This is abnormal, v4l2loopback should not return more frames than the target frame rate that is set in the configuration of the device. After decoding with avcodec_decode_video2(), the resulting AVPacket has a dts field set to a fixed value. This is abnormal, I expect dts to be a proper timestamp of the frame. It should advance as a real time clock.

Yes you can pass that information to them. They will probably find useful to look at the source code that's doing the caching. It's all in the VideoFFmpeg::cacheThread() function. I noticed the following anomalies: **av_read_frame() never returns < 0, which would mean 'no frame available at this time' no matter how frequently I call this function and despite the fact that the AVFMT_FLAG_NONBLOCK flag is set in the format context. This is abnormal, v4l2loopback should not return more frames than the target frame rate that is set in the configuration of the device.** After decoding with avcodec_decode_video2(), the resulting AVPacket has a dts field set to a fixed value. This is abnormal, I expect dts to be a proper timestamp of the frame. It should advance as a real time clock.

Mario Mey commented

2016-12-03 13:51:48 +01:00

@BenoitBolsee : I reported this issue in v4l2loopback github page (in June). Now, there is a new version of this module: 0.10.0.

Umlaeute (v4l2loopback dev) ask me for trying this new version with this issue. I installed new version and tested the file with last Blender version... but the issue is still there.

I don't know if Blender "multi-thread caching" issue is still there and it is the main problem... is it resolvable?

Could you try this?

@BenoitBolsee : I reported this issue in v4l2loopback github page (in June). Now, there is a new version of this module: 0.10.0. Umlaeute (v4l2loopback dev) ask me for trying this new version with this issue. I installed new version and tested the file with last Blender version... but the issue is still there. I don't know if Blender "multi-thread caching" issue is still there and it is the main problem... is it resolvable? Could you try this?

Mario Mey commented

2017-06-19 23:59:40 +02:00

I realized that, in june 2016, I wanted to write here... and I wrote in another report (in #46755)! It says:

Wow, it seems to be a big optimization... or a a big bug.

Using the same file, showing the WebCam, there's a big difference in performance between BlenderPlayer with and without caching... but only with WebCam! Look at these window-shots:

Only Webcam, with caching: 40fps.

Only Webcam, without caching (aka, commented thread line): 10-14fps.

Webcam and SecondDisplay-Capture, without caching: also 10-14fps.

Only SecondDisplay-Capture, without caching: 50-60fps.

So, now I have a problem with WebCam. Where is the problem now? :(

Notes:

In the second monitor (TV) there's a blenderplayer instance with a Digital Puppet. The captures below are about a blend file that control the puppet (receive the mouse and keyboard and send OSC messages with input information).
My webcam is a Logitech C170. It captures in 640x480.

I realized that, in june 2016, I wanted to write here... and I wrote in another report (in #46755)! It says: *<start of the comment>* Wow, it seems to be a big optimization... or a a big bug. Using the same file, showing the WebCam, there's a big difference in performance between BlenderPlayer with and without caching... but only with WebCam! Look at these window-shots: Only Webcam, with caching: 40fps. ![Pantallazo-ctrl-0.png](https://archive.blender.org/developer/F636038/Pantallazo-ctrl-0.png) Only Webcam, without caching (aka, commented thread line): 10-14fps. ![Pantallazo-ctrl-1.png](https://archive.blender.org/developer/F636040/Pantallazo-ctrl-1.png) Webcam and SecondDisplay-Capture, without caching: also 10-14fps. ![Pantallazo-ctrl-2.png](https://archive.blender.org/developer/F636042/Pantallazo-ctrl-2.png) Only SecondDisplay-Capture, without caching: 50-60fps. ![Pantallazo-ctrl-3.png](https://archive.blender.org/developer/F636045/Pantallazo-ctrl-3.png) So, now I have a problem with WebCam. Where is the problem now? :( Notes: - In the second monitor (TV) there's a blenderplayer instance with a Digital Puppet. The captures below are about a blend file that control the puppet (receive the mouse and keyboard and send OSC messages with input information). - My webcam is a Logitech C170. It captures in 640x480.

Mario Mey commented

2017-06-20 00:12:03 +02:00

Also, for a reason that I don't understand... this disabling cache workaround is not working in this new Ubuntu version (previous: 14.04, actual: 16.04).

Finally, I compiled Blender in this distro. The line about threading is still commented:

		//m_isThreaded =  true;

... but I run the blend file and it is doing the same as when the report was created: the image is still, not moving at all.

Blender source is the same, same version-commit, v4l2loopback does not have any new commit from that moment (june 2016).

What could be happening?

Finally... where is the problem? in v4l2loopback source code?

Also, for a reason that I don't understand... this disabling cache workaround is not working in this new Ubuntu version (previous: 14.04, actual: 16.04). Finally, I compiled Blender in this distro. The line about threading is still commented: ``` //m_isThreaded = true; ``` ... but I run the blend file and it is doing the same as when the report was created: the image is still, not moving at all. Blender source is the same, same version-commit, v4l2loopback does not have any new commit from that moment (june 2016). What could be happening? Finally... where is the problem? in v4l2loopback source code?

Mario Mey commented

2017-06-20 15:42:00 +02:00

@BenoitBolsee I need your hand.

In v4l2loopback bug report, umlaeute (dev) asks this:

i think one of the issues is that it is unclear who should actually provide the timestamp: the module or the producer? (I tend towards the producer, but I don't know whether there is a single producer out there that actually does it)

@BenoitBolsee I need your hand. In v4l2loopback bug report, umlaeute (dev) asks this: > i think one of the issues is that it is unclear who should actually provide the timestamp: the module or the producer? (I tend towards the producer, but I don't know whether there is a single producer out there that actually does it)

Aaron Carlisle commented

2019-06-29 02:23:35 +02:00

Added subscriber: @Blendify

Aaron Carlisle commented

2019-06-29 02:23:35 +02:00

Changed status from 'Open' to: 'Archived'

Aaron Carlisle closed this issue

2019-06-29 02:23:35 +02:00

Aaron Carlisle commented

2019-06-29 02:23:35 +02:00

This task is being closed because the BGE has been removed in Blender 2.8.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VideoTexture, only first frame shown (gst-launch/v4l2loopback -> Blender) #48692