Page MenuHome

VideoTexture, only first frame shown (gst-launch/v4l2loopback -> Blender)
Closed, ArchivedPublic

Description

System
Ubuntu-Mate 14.04 64x, nVidia GT620.

Issue
I made a 3D scene (in BGE) with two VideoTextures: one shows the webcam image, the other one: a v4l2loopback device capturing second-display screen, using gst-launch-1.0. No matter what command I use (*), when BlenderPlayer starts, it only shows the first frame of the v4l2loopback device. Webcam works well.

What I tested:

  • Using VLC, webcam and any v4l2loopback works OK. I get smooth playback.
  • Using Blender, webcam and another capture device (EasyCap), also I get smooth playback of both devices.
  • Using two gst-launch-1.0 instances, both devices are displayed well in different windows, at the same time (I use "ximagesink" instead of "v4l2sink...")

(*) My command:

gst-launch-1.0 ximagesrc startx=1920 starty=0 endx=2943 endy=768 show-pointer=0 use-damage=0 \
! video/x-raw,framerate=30/1 \
! videoscale method=0 add-borders=false \
! video/x-raw,width=640,height=360 \
! videoflip method=horizontal-flip \
! v4l2sink device=/dev/video1

Also, I tried with the simplest one:

gst-launch videotestsrc ! v4l2sink device=/dev/video1

Blend file:

To reproduce the "issue"

  • Install v4l2loopback module (https://github.com/umlaeute/v4l2loopback)
  • Connect a webcam.
  • Run the simple gst-launch command mentioned above.
  • Open VLC and test both devices.
  • Close VLC.
  • Open the blend file in Blender.
  • Start Game Engine.

What it should do

  • One object should show the webcam device and, the other, a video-test-pattern with motion.

I also reported in v4l2loopback GitHub issues page: https://github.com/umlaeute/v4l2loopback/issues/115

Details

Type
Bug

Event Timeline

Mario Mey (mariomey) raised the priority of this task from to Needs Triage by Developer.
Mario Mey (mariomey) updated the task description. (Show Details)
Mario Mey (mariomey) set Type to Bug.

This file works in Blender > 2.66.
The bug is there and all the later versions. Included the last one 2.77a.

I think "this Benoit" was the one that Dalai told me to add here...

Sergey Sharybin (sergey) lowered the priority of this task from Needs Triage by Developer to Normal.Jun 28 2016, 11:50 AM

@Benoit Bolsee (ben2610), are you looking into this issue?

not yet, I need to get hold of a webcam to reproduce.

@Benoit Bolsee (ben2610), you don't really need a camera to test the bug. The webcam is there to show that /dev/video0 is working and /dev/video1 isn't.

Just comment these lines and run it:

14: monitor_webcam()
27: bge.logic.webcam_texture.refresh(True)

Thanks for the instructions; I was able to reproduce the bug.
It appears that multi-thread caching doesn't work well with that particular v4l source.
I was able to get a smooth output by disabling caching.

I will analyze further and determine if I need to fix caching or disable it or make it selectable by API (caching is known to work with webcam source).

Is there any way to disable caching without modifying code and recompiling...?
If not (and if it seems to be stable to do that), could you make a diff (patch) file for me to compile Blender here?

(I don't want to seem anxious, but it would be very-great if I can make it work here during these days... it's for a particular job)

Thanks!

Cache is enabled automatically as soon as Blender detects that the CPU has more than 1 core.
To disable it you must change the code. The diff is extremely simple: just comment the two line with m_isThreaded = true; in VideoFFmpeg.cpp:

diff --git a/source/gameengine/VideoTexture/VideoFFmpeg.cpp b/source/gameengine/VideoTexture/VideoFFmpeg.cpp
index 083e9e2..5a470be 100644
--- a/source/gameengine/VideoTexture/VideoFFmpeg.cpp
+++ b/source/gameengine/VideoTexture/VideoFFmpeg.cpp
@@ -581,7 +581,7 @@ void VideoFFmpeg::openFile (char *filename)
 	{
 		// never thread image: there are no frame to read ahead
 		// no need to thread if the system has a single core
-		m_isThreaded =  true;
+		//m_isThreaded =  true;
 	}
 }
 
@@ -673,7 +673,7 @@ void VideoFFmpeg::openCam (char *file, short camIdx)
 	if (BLI_system_thread_count() > 1)
 	{
 		// no need to thread if the system has a single core
-		m_isThreaded =  true;
+		//m_isThreaded =  true;
 	}
 
 	av_dict_free(&formatParams);

It worked. ;)

For the moment... Is it safe to disable this caching? Just optimization purpose...?

Yes, caching is just for optimization. It is safe to disable it.

I did more tests and the problem is caused by the v4l2loopback not providing correct timestamp on frames: vt thinks that the frames are in the past and drops them.
There is a flag to fix this: m_isStreaming; this flags tells vt to ignore timestamp. Then caching works but not smoothly: there is a strange jitter in the stream as if the frames were not properly ordered. It seems that the loopback device doesn't implement a proper frame queue.

I'll investigate further but the best option apparently is to make caching optional by API.

As I reported this same bug in v4l2loopback issues github page, can I add this information that you are telling to me? Is there any other further information to post there?

Maybe v4l2loopback also can be better by providing the correct timestamp on frames.

Yes you can pass that information to them. They will probably find useful to look at the source code that's doing the caching. It's all in the VideoFFmpeg::cacheThread() function.

I noticed the following anomalies:

  • av_read_frame() never returns < 0, which would mean 'no frame available at this time' no matter how frequently I call this function and despite the fact that the AVFMT_FLAG_NONBLOCK flag is set in the format context. This is abnormal, v4l2loopback should not return more frames than the target frame rate that is set in the configuration of the device.
  • After decoding with avcodec_decode_video2(), the resulting AVPacket has a dts field set to a fixed value. This is abnormal, I expect dts to be a proper timestamp of the frame. It should advance as a real time clock.

@Benoit Bolsee (ben2610) : I reported this issue in v4l2loopback github page (in June). Now, there is a new version of this module: 0.10.0.

Umlaeute (v4l2loopback dev) ask me for trying this new version with this issue. I installed new version and tested the file with last Blender version... but the issue is still there.

I don't know if Blender "multi-thread caching" issue is still there and it is the main problem... is it resolvable?

Could you try this?

I realized that, in june 2016, I wanted to write here... and I wrote in another report (in T46755)! It says:

<start of the comment>
Wow, it seems to be a big optimization... or a a big bug.

Using the same file, showing the WebCam, there's a big difference in performance between BlenderPlayer with and without caching... but only with WebCam! Look at these window-shots:

Only Webcam, with caching: 40fps.


Only Webcam, without caching (aka, commented thread line): 10-14fps.

Webcam and SecondDisplay-Capture, without caching: also 10-14fps.

Only SecondDisplay-Capture, without caching: 50-60fps.

So, now I have a problem with WebCam. Where is the problem now? :(

Notes:

  • In the second monitor (TV) there's a blenderplayer instance with a Digital Puppet. The captures below are about a blend file that control the puppet (receive the mouse and keyboard and send OSC messages with input information).
  • My webcam is a Logitech C170. It captures in 640x480.

Also, for a reason that I don't understand... this disabling cache workaround is not working in this new Ubuntu version (previous: 14.04, actual: 16.04).

Finally, I compiled Blender in this distro. The line about threading is still commented:

		//m_isThreaded =  true;

... but I run the blend file and it is doing the same as when the report was created: the image is still, not moving at all.

Blender source is the same, same version-commit, v4l2loopback does not have any new commit from that moment (june 2016).

What could be happening?

Finally... where is the problem? in v4l2loopback source code?

@Benoit Bolsee (ben2610) I need your hand.

In v4l2loopback bug report, umlaeute (dev) asks this:

i think one of the issues is that it is unclear who should actually provide the timestamp: the module or the producer? (I tend towards the producer, but I don't know whether there is a single producer out there that actually does it)

Aaron Carlisle (Blendify) closed this task as Archived.Jun 29 2019, 2:23 AM

This task is being closed because the BGE has been removed in Blender 2.8.