Some video format (4:4:4 YUV?) generate huge mem usage with ffmpeg #37166

New Issue

Michael P. · 2013-10-21T16:31:28+02:00

Michael P. commented

2013-10-21 16:31:28 +02:00

%%%--- Operating System, Graphics card ---
32bit, Microsoft Windows XP Professional Build 2600, Service Pack 3
4096 MB, ATI RADEON HD 4890 (RV790 XT)

- Blender version with error, and version that worked ---

tested with:

Blender 2.68 (sub 0), Revision: 58537
Blender 2.69 (sub 1), Revision: 60874
- Short description of error ---
  Working longer time in the VSE causes Blender crash.
- Steps for others to reproduce the error (preferably based on attached .blend file) ---
  Working longer time in the VSE causes Blender crash. Please see attached text and blend files for more.

Note:

I can work "longer", if I click on the "Refresh Sequencer" button before the memory (the value in parentheses) reaches above 500 MB.
I use for the most part footage that was encoded as seen at bottom of this message.

My ffmpeg encoding details:

ffmpeg -i \image-sequence-%4d.png -g 25 -r 25 -c:v libx264 -preset veryslow -b:v 9000k -pass 1 -an -f mp4 NULL && ffmpeg -i \image-sequence-%4d.png -g 25 -r 25 -c:v libx264 -preset veryslow -b:v 9000k -pass 2 \output-file.mp4%%%

%%%--- Operating System, Graphics card --- 32bit, Microsoft Windows XP Professional Build 2600, Service Pack 3 4096 MB, ATI RADEON HD 4890 (RV790 XT) - Blender version with error, and version that worked --- tested with: - Blender 2.68 (sub 0), Revision: 58537 - Blender 2.69 (sub 1), Revision: 60874 - Short description of error --- Working longer time in the VSE causes Blender crash. - Steps for others to reproduce the error (preferably based on attached .blend file) --- Working longer time in the VSE causes Blender crash. Please see attached text and blend files for more. Note: 1. I can work "longer", if I click on the "Refresh Sequencer" button before the memory (the value in parentheses) reaches above 500 MB. 2. I use for the most part footage that was encoded as seen at bottom of this message. My ffmpeg encoding details: ffmpeg -i <path>\image-sequence-%4d.png -g 25 -r 25 -c:v libx264 -preset veryslow -b:v 9000k -pass 1 -an -f mp4 NULL && ffmpeg -i <path>\image-sequence-%4d.png -g 25 -r 25 -c:v libx264 -preset veryslow -b:v 9000k -pass 2 <path>\output-file.mp4%%%

Michael P. commented

2013-10-21 16:31:28 +02:00

Changed status to: 'Open'

Bastien Montagne commented

2013-10-21 16:53:27 +02:00

%%%You are just getting out of memory… Maybe you could try to reduce your memory cache limit (user preferences, System tab). But apart from that, we can’t help you, you should get a 64bit system, or close and re-open blender before it reaches the 1Go limit…

%%%

%%%You are just getting out of memory… Maybe you could try to reduce your memory cache limit (user preferences, System tab). But apart from that, we can’t help you, you should get a 64bit system, or close and re-open blender before it reaches the 1Go limit… %%%

Michael P. commented

2013-10-21 17:08:48 +02:00

%%%Yeah sure a new 64bit OS, more RAM that is one way to solve this. But I don't think that Blender should crash without a warning and "manage" the amount of RAM I have. I also already work with proxies. The RAM limit under the 32bit system is already set to 1024 MB max. Maybe this causes also some problems. And maybe someone else knows more. So it would be nice if this bug stays open until then. Closed means nobody looks into it anymore.%%%

Bastien Montagne commented

2013-10-21 17:37:55 +02:00

%%%If you set your cache limit to 1024MO, there is no wonder why it crashes! Under 32bit OSs, an application has at most 1Go of memory, so you should absolutely not set your cache limit above 512Mo (and most likely even not above 256Mo)!%%%

Michael P. commented

2013-10-21 19:59:54 +02:00

%%%I have 4096 MB RAM and 1 GB graphic card (I know that the total of both won't exceed 4 GB on a 32bit system). In my opinion there is enough RAM. Why I get a RAM crash then? Just asking.

Just tested this with 256 MB and got a crash.

I bet it has something to do with the encoding through ffmpeg. Blender uses probably an older library.%%%

%%%I have 4096 MB RAM and 1 GB graphic card (I know that the total of both won't exceed 4 GB on a 32bit system). In my opinion there is enough RAM. Why I get a RAM crash then? Just asking. Just tested this with 256 MB and got a crash. I bet it has something to do with the encoding through ffmpeg. Blender uses probably an older library.%%%

Bastien Montagne commented

2013-10-21 22:03:33 +02:00

%%%You can have 4, 6 or 8Go of Ram, on a 32bit system an application (like Blender, Gimp, whatever) only can use 1Go, it cannot address more, it’s an OS limitation (also note that XP32 itself cannot use more than 3Go anyway).

And variations in libs versions should not have any effect here… Do you get similar issue with e.g. manipulating big image sequences? What happens if you reduce cache to something as small as 32Mo? Also, could you keep an eye on blender's reported mem usage in your task manager, just before crash? Your crash file reports about 430Mo used by Blender's own allocator, which means (if it crashes at around 1Go of mem used) that more than 500Mo are allocated "elsewhere"…%%%

%%%You can have 4, 6 or 8Go of Ram, on a 32bit system an application (like Blender, Gimp, whatever) only can use 1Go, it cannot address more, it’s an OS limitation (also note that XP32 itself cannot use more than 3Go anyway). And variations in libs versions should not have any effect here… Do you get similar issue with e.g. manipulating big image sequences? What happens if you reduce cache to something as small as 32Mo? Also, could you keep an eye on blender's reported mem usage in your task manager, just before crash? Your crash file reports about 430Mo used by Blender's own allocator, which means (if it crashes at around 1Go of mem used) that more than 500Mo are allocated "elsewhere"…%%%

Michael P. commented

2013-10-21 22:39:38 +02:00

%%%So tested it again with an mp4 File which is 58:04 minutes long and Blender ist stable. Memory Limit set again to 1024 MB. NO crash.

The values when I open the blend file are: Mem: 14.34M (1.21M). Then I scrub and the Mem vlaue gets higher.
When I look at the top in the "Info" window it reads: Mem:525.31M (1.21M) at the end. I would say its "max" or "full".
This memory never gets higher then 550M.

Video details: Decoded Format: Planar 4:2:0 YUV, Codec: H264 - MPEG-4 AVC (part 10) (avc1)

But there is a diffrence with the videos I encoded for my project in ffmpeg.
The "Info" window it reads at start: Mem:7.10M (8.02M). And the the first value changes not much 7.20M but the second a lot (up to 690M !)
and then crash...

Video details: Decoded Format: Planar 4:4:4 YUV, Codec: H264 - MPEG-4 AVC (part 10) (avc1)

I think it could have something to do with "4:4:4 YUV". Older blender versions like 2.67b oder 2.64a don't show that videos anyway...%%%

%%%So tested it again with an mp4 File which is 58:04 minutes long and Blender ist stable. Memory Limit set again to 1024 MB. NO crash. The values when I open the blend file are: Mem: 14.34M (1.21M). Then I scrub and the Mem vlaue gets higher. When I look at the top in the "Info" window it reads: Mem:525.31M (1.21M) at the end. I would say its "max" or "full". This memory never gets higher then 550M. - Video details: Decoded Format: Planar 4:2:0 YUV, Codec: H264 - MPEG-4 AVC (part 10) (avc1) But there is a diffrence with the videos I encoded for my project in ffmpeg. The "Info" window it reads at start: Mem:7.10M (8.02M). And the the first value changes not much 7.20M but the second a lot (up to 690M !) and then crash... - Video details: Decoded Format: Planar 4:4:4 YUV, Codec: H264 - MPEG-4 AVC (part 10) (avc1) I think it could have something to do with "4:4:4 YUV". Older blender versions like 2.67b oder 2.64a don't show that videos anyway...%%%

Bastien Montagne commented

2013-10-21 23:11:36 +02:00

%%%Ah, then now we have something to hunt down! Too late today, but will see if I can reproduce that here (linux64) tomorrow (issue is not actually the crash itself, but that absurd bump of mem usage).%%%

Sergey Sharybin commented

2013-10-28 18:14:09 +01:00

%%%Cache memory limit only affects blender side, it can not FFmpeg keeping different stuff in memory s well. And blender side does not depend on type of your video obviously :)

So i would suspect FFmpeg doing crazy stuff. What does system monitor says about memory usage of blender?%%%

%%%Cache memory limit only affects blender side, it can not FFmpeg keeping different stuff in memory s well. And blender side does not depend on type of your video obviously :) So i would suspect FFmpeg doing crazy stuff. What does system monitor says about memory usage of blender?%%%

Michael P. commented

2013-10-28 19:07:31 +01:00

%%%I attached a windows xp (german) task manager screenshot.
And this is what is happening to the blender.exe memory:
no crash, stable

memory of blender.exe is stable between 668.000 K and 672.000 K (some peeks to 683.000 K but then goes fast back to the mentiond values)
crash, unstable
memory of blender.exe is growing to 1.500.000 K and crashing at 1.754.000 K

I converted my video files to "4:2:0 YUV", but the issue remains.

I also tested this today with an 2 h movie and it is stable.

Other thing than the ffmpeg encoding is the use of the effect strips "speed" and "ajustment layer" and a meta strip in the vse.%%%

%%%I attached a windows xp (german) task manager screenshot. And this is what is happening to the blender.exe memory: no crash, stable - memory of blender.exe is stable between 668.000 K and 672.000 K (some peeks to 683.000 K but then goes fast back to the mentiond values) crash, unstable - memory of blender.exe is growing to 1.500.000 K and crashing at 1.754.000 K I converted my video files to "4:2:0 YUV", but the issue remains. I also tested this today with an 2 h movie and it is stable. Other thing than the ffmpeg encoding is the use of the effect strips "speed" and "ajustment layer" and a meta strip in the vse.%%%

Michael P. commented

2013-10-28 19:36:17 +01:00

%%%1. I have deleted one by one: "speed", "ajustment layer", meta strip. It had no effect. Blender still crashed.
2. I have deleted the largest of my video clips (4 instances, but different parts). Blender memory was at 1.200.000 K. If I duplicate a movie strip it goes up to 1.347.000 K. And if I scrub then it is between 1.347.000 K and 1.363.000 K. No crash yet.%%%

%%%1. I have deleted one by one: "speed", "ajustment layer", meta strip. It had no effect. Blender still crashed. 2. I have deleted the largest of my video clips (4 instances, but different parts). Blender memory was at 1.200.000 K. If I duplicate a movie strip it goes up to 1.347.000 K. And if I scrub then it is between 1.347.000 K and 1.363.000 K. No crash yet.%%%

Michael P. commented

2013-10-28 19:59:59 +01:00

%%%OK. I can redo the crash with other files then mine too. I simply duplicated one movie clip (6 GB large) many times (28+). The memory goes up and up and then blender crashes. With my files its the crash is really fast. And all those files together are only 460 MB.%%%

Sergey Sharybin commented

2013-10-29 12:45:41 +01:00

%%%How many instances of your video strips you've got?

There's a known issue with cacheing which doesn't free FFmpeg buffers for strips at all. So if you've got loads of movie strips you'll run out of memory easily (this FFmpeg buffers are not managed by Blender so it's difficult predict their size). Will try to look into improving cache from this perspective, and this might help your case as well.%%%

%%%How many instances of your video strips you've got? There's a known issue with cacheing which doesn't free FFmpeg buffers for strips at all. So if you've got loads of movie strips you'll run out of memory easily (this FFmpeg buffers are not managed by Blender so it's difficult predict their size). Will try to look into improving cache from this perspective, and this might help your case as well.%%%

Michael P. commented

2013-10-29 13:19:24 +01:00

%%%There are 6 instances of one movie clip. Two of them as a single metastrip. And 6 other movie clips. None of them has more than one instance.
Please note if I click "Refresh Sequencer" button the memory is set back.%%%

%%%There are 6 instances of one movie clip. Two of them as a single metastrip. And 6 other movie clips. None of them has more than one instance. Please note if I click "Refresh Sequencer" button the memory is set back.%%%

Thomas Dinges commented

2014-02-21 01:22:14 +01:00

Added subscriber: @ThomasDinges

Thomas Dinges commented

2014-02-21 01:22:14 +01:00

Any updates here?

I would close this, with a 32bit OS you can only use 2GB per process, and this is the issue here.
32bit is just ancient, especially for 3D Graphics and Video editing. ;)

Any updates here? I would close this, with a 32bit OS you can only use 2GB per process, and this is the issue here. 32bit is just ancient, especially for 3D Graphics and Video editing. ;)

Sergey Sharybin commented

2014-02-21 05:34:01 +01:00

No update yet. But we definitely need to make FFmpeg descriptors managed by the cache system.

Sergey Sharybin commented

2014-05-19 16:58:11 +02:00

Changed status from 'Open' to: 'Archived'

Sergey Sharybin closed this issue

2014-05-19 16:58:11 +02:00

Sergey Sharybin commented

2014-05-19 16:58:11 +02:00

Moving to the TODO list now: http://wiki.blender.org/index.php/Dev:2.5/Source/Development/Todo/Editors#Video_Sequencer

Not actually a bug, just result of FFmpeg frames being non-managed from our side. Would need making some system to keep track on FFmpeg memory usage which is not so trivial.

Thanks for the report, but we'll solve it later as a TODO thing.

Moving to the TODO list now: http://wiki.blender.org/index.php/Dev:2.5/Source/Development/Todo/Editors#Video_Sequencer Not actually a bug, just result of FFmpeg frames being non-managed from our side. Would need making some system to keep track on FFmpeg memory usage which is not so trivial. Thanks for the report, but we'll solve it later as a TODO thing.

ChristopherAnderssarian commented

2020-07-27 16:09:14 +02:00

Added subscriber: @ChristopherAnderssarian

Sybren A. Stüvel referenced this issue

2023-03-30 15:30:09 +02:00

Blender development todo list -- Editors #55366

Sign in to join this conversation.