VSE: Memory leak importing bunch of medias #91110

New Issue

Félix · 2021-09-01T14:21:12+02:00

Félix commented

2021-09-01 14:21:12 +02:00

System Information
Operating system: Windows 10
Graphics card: GTX 1070Ti

Blender Version
Broken: 2.93.4 b7205031ce candidate

Short description of error

RAM increases a continuously while importing new movies when new sounds have no impact.

Exact steps for others to reproduce the error

Start from Video Editing app template and open a script window
Copy/Paste and replace the paths to actual files on your computer:

import bpy

seq_ed = bpy.context.scene.sequence_editor

# Sounds
for t in range(1, 4):
    for s in range(200):
        seq_ed.sequences.new_sound(
            name=f"{t}_{s}",
            filepath="/path/to/sound.wav",
            channel=t,
            frame_start=s*50
        )

# Movies
for t in range(4, 7):
    for s in range(200):
        seq_ed.sequences.new_movie(
            name=f"{t}_{s}",
            filepath="/path/to/movie.mov",
            channel=t,
            frame_start=s*50
        )

Run the script: while it's running sounds import, no memory increase, when movies import start, it goes up. If you wanna have a bigger observation, augment the range from 200 to 1000.

NB
Maybe the following observations are related:

When the import is done, saving the .blend and re-opening by File > Open Recent takes very long (and longer if more medias have been created), when saving, closing and opening the saved .blend is much faster. Looking at the RAM, it decreases very slowly until it reaches a floor, near to the value before the process' start.
Deleting the created sequences right after the importing is done takes long and makes de RAM goes lower and lower until it reaches a floor.
When the import is done, saving, closing, and opening the file clears the RAM and the opening doesn't make the RAM to reach its value at the end of the import. Which makes me think of a memory leak.

Images
I've done some benchs:

Start, save, open recent

Start, delete created seq

Start, save, close blender, open saved .blend

**System Information** Operating system: Windows 10 Graphics card: GTX 1070Ti **Blender Version** Broken: 2.93.4 b7205031cec4 candidate **Short description of error** RAM increases a continuously while importing new movies when new sounds have no impact. **Exact steps for others to reproduce the error** - Start from `Video Editing` app template and open a script window - Copy/Paste and replace the paths to actual files on your computer: ``` import bpy seq_ed = bpy.context.scene.sequence_editor # Sounds for t in range(1, 4): for s in range(200): seq_ed.sequences.new_sound( name=f"{t}_{s}", filepath="/path/to/sound.wav", channel=t, frame_start=s*50 ) # Movies for t in range(4, 7): for s in range(200): seq_ed.sequences.new_movie( name=f"{t}_{s}", filepath="/path/to/movie.mov", channel=t, frame_start=s*50 ) ``` - Run the script: while it's running sounds import, no memory increase, when movies import start, it goes up. If you wanna have a bigger observation, augment the range from 200 to 1000. **NB** Maybe the following observations are related: - When the import is done, saving the .blend and re-opening by File > Open Recent takes very long (and longer if more medias have been created), when saving, closing and opening the saved .blend is much faster. Looking at the RAM, it decreases very slowly until it reaches a floor, near to the value before the process' start. - Deleting the created sequences right after the importing is done takes long and makes de RAM goes lower and lower until it reaches a floor. - When the import is done, saving, closing, and opening the file clears the RAM and the opening doesn't make the RAM to reach its value at the end of the import. Which makes me think of a memory leak. **Images** I've done some benchs: **Start, save, open recent** ![Start_save_reopen.PNG](https://archive.blender.org/developer/F10360399/Start_save_reopen.PNG) **Start, delete created seq** ![Start_save_delete.PNG](https://archive.blender.org/developer/F10360401/Start_save_delete.PNG) **Start, save, close blender, open saved .blend** ![Start_save_close_open.PNG](https://archive.blender.org/developer/F10360403/Start_save_close_open.PNG)

Félix commented

2021-09-01 14:21:12 +02:00

Added subscriber: @Miadim

Philipp Oeser commented

2021-09-01 15:52:27 +02:00

Added subscribers: @iss, @lichtwerk

Philipp Oeser commented

2021-09-01 15:52:27 +02:00

Changed status from 'Needs Triage' to: 'Needs Developer To Reproduce'

Philipp Oeser commented

2021-09-01 15:52:27 +02:00

Isnt this just the memory used to build the Proxies? https://docs.blender.org/manual/en/dev/video_editing/sequencer/sidebar/proxy.html

Once they are done, memory gets slowly freed again.
Opening that file again does not have to build the proxies again, so no memory increase here.

Calling this a memory leak is misleading I think since would only be the case if memory is not freed if blender closes etc.
Blender would also usually report a real leak if started from the commandline in the console.

@iss: this is expected behavior, no?

Isnt this just the memory used to build the Proxies? https://docs.blender.org/manual/en/dev/video_editing/sequencer/sidebar/proxy.html Once they are done, memory gets slowly freed again. Opening that file again does not have to build the proxies again, so no memory increase here. Calling this a `memory leak` is misleading I think since would only be the case if memory is not freed if blender closes etc. Blender would also usually report a real leak if started from the commandline in the console. @iss: this is expected behavior, no?

Félix commented

2021-09-01 16:04:35 +02:00

Yes I hesitated to call it Memory not freed. Maybe it's not a memory leak, but why the memory isn't freed right after the proxies are built? It shouldn't be a flat part remaining until I close, delete or open another .blend. It must freed instantly when the python process is done, no?

Yes I hesitated to call it `Memory not freed`. Maybe it's not a memory leak, but why the memory isn't freed right after the proxies are built? It shouldn't be a flat part remaining until I close, delete or open another .blend. It must freed instantly when the python process is done, no?

Félix commented

2021-09-01 16:18:33 +02:00

Also, I've tried the same steps I listed with Proxy Setup to Manual and I get the exact same result.

Also, I've tried the same steps I listed with Proxy Setup to `Manual` and I get the exact same result.

Richard Antalik commented

2021-09-02 06:00:15 +02:00

Changed status from 'Needs Developer To Reproduce' to: 'Archived'

Richard Antalik closed this issue

2021-09-02 06:00:15 +02:00

Richard Antalik commented

2021-09-02 06:00:15 +02:00

This is not caused by proxy building, even though proxy building would double memory usage. Issue is, that each movie file handle will allocate some memory within ffmpeg. In my case it is about 200MB. This depends on codec settings, but mainly on movie resolution.

I thought this could be bug, because added files don't have to be kept open, which is technically true, but in practice it doesn't make much sense to add movie, open it(for internal purposes), close it and then open it again, because user will try to render it. Also if you would try to render each movie file this memory would be allocated over time anyway. Trouble is that Blender doesn't have idea about this allocated memory, because it is done by external library, so number of open files can't be managed during runtime. Perhaps if amount of open files is limited, but since amount of memory needed is quite variable, this wouldn't work in practice.

So unfortunately this is not even design limitation, but completely outside of our ability to resolve this issue, so I will close this report.

However, as we talked on blender.chat, importer scripts could have ability to skip file opening, which would resolve this issue partially, editing of large amount of movie files is still possible to limited extent (movies can be unloaded manually by pressing Ctrl+R) and rendering does "close" or free all movies file for every frame, so memory usage is kept low. Minor problem here is, that we have argument fit_method that would be disfunctional with fast loading mode (this also can be resolved though).

This is not caused by proxy building, even though proxy building would double memory usage. Issue is, that each movie file handle will allocate some memory within ffmpeg. In my case it is about 200MB. This depends on codec settings, but mainly on movie resolution. I thought this could be bug, because added files don't have to be kept open, which is technically true, but in practice it doesn't make much sense to add movie, open it(for internal purposes), close it and then open it again, because user will try to render it. Also if you would try to render each movie file this memory would be allocated over time anyway. Trouble is that Blender doesn't have idea about this allocated memory, because it is done by external library, so number of open files can't be managed during runtime. Perhaps if amount of open files is limited, but since amount of memory needed is quite variable, this wouldn't work in practice. So unfortunately this is not even design limitation, but completely outside of our ability to resolve this issue, so I will close this report. --- However, as we talked on blender.chat, importer scripts could have ability to skip file opening, which would resolve this issue partially, editing of large amount of movie files is still possible to limited extent (movies can be unloaded manually by pressing Ctrl+R) and rendering does "close" or free all movies file for every frame, so memory usage is kept low. Minor problem here is, that we have argument `fit_method` that would be disfunctional with fast loading mode (this also can be resolved though).

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE: Memory leak importing bunch of medias #91110