Noticeably higher RAM consumption with packed textures #97510

New Issue

Harlekin · 2022-04-21T13:53:07+02:00

Harlekin commented

2022-04-21 13:53:07 +02:00

System Information
Operating system: Has been reported on Mac and Linux, has been observed under Windows as well
Graphics card:

Blender Version
3.1.2 but has been reported as early as January 28

Short description of error
For context, I am reporting on behalf of the SheepIt Renderfarm team.

Our users and we noticed a significant increase in RAM consumption and IO when using packed textures as compared to externally linked textures.
In a test a with a 3 GB Blend file 151GB were read from disk.
The report in January was about a project that could be rendered on a 16 GB Mac with linked textures but would use up to 60 GB when packed.

I will include their final verdict:

I’ve found the issue.

I went through intensive testing, I compressed the textures as best as possible and that made the memory peak go down from 49 gigs to about 20 gigs… that’s pretty good but not much of a solution.

I then found out that what causes most of the massive memory spike lies in Blender’s unpacking algorithm.
So I unpacked the file, turned on Relative Path and zipped the folder with all the files. This made the build time BLAZING FAST (we’re talking mere seconds) and the memory peak dropped to… 3 gigs!!!

We cannot provide you with any of these files so I tried to recreate it. In my tests relative.blend peaked at 850 MB while packed.blend used 1380 MB. I understand that in this case the difference could just be the extraction of the the textures into RAM so I am not really satisfied with this example. I will try to provide a better one if need be.

Exact steps for others to reproduce the error

Unzip attached file
Open relative.blend
Check RAM memory consumption
In {nav File > External Data} click in Pack Resources
Check RAM memory consumption
(Optionally you can load the packed.blend file to skip a few steps)
or
Add objects with PBR materials.
Case A: Reference to the textures locaton
Case B: Pack them into the blend file using File -> External Data -> Pack Resources
Start rendering
bug report.zip

**System Information** Operating system: Has been reported on Mac and Linux, has been observed under Windows as well Graphics card: **Blender Version** 3.1.2 but has been reported as early as January 28 **Short description of error** For context, I am reporting on behalf of the SheepIt Renderfarm team. Our users and we noticed a significant increase in RAM consumption and IO when using packed textures as compared to externally linked textures. In a test a with a 3 GB Blend file 151GB were read from disk. The report in January was about a project that could be rendered on a 16 GB Mac with linked textures but would use up to 60 GB when packed. I will include their final verdict: > I’ve found the issue. > > I went through intensive testing, I compressed the textures as best as possible and that made the memory peak go down from 49 gigs to about 20 gigs… that’s pretty good but not much of a solution. > > I then found out that what causes most of the massive memory spike lies in Blender’s unpacking algorithm. > So I unpacked the file, turned on Relative Path and zipped the folder with all the files. This made the build time BLAZING FAST (we’re talking mere seconds) and the memory peak dropped to… 3 gigs!!! We cannot provide you with any of these files so I tried to recreate it. In my tests relative.blend peaked at 850 MB while packed.blend used 1380 MB. I understand that in this case the difference could just be the extraction of the the textures into RAM so I am not really satisfied with this example. I will try to provide a better one if need be. **Exact steps for others to reproduce the error** - Unzip attached file - Open `relative.blend` - Check RAM memory consumption - In {nav File > External Data} click in `Pack Resources` - Check RAM memory consumption - (Optionally you can load the `packed.blend` file to skip a few steps) or - Add objects with PBR materials. - Case A: Reference to the textures locaton - Case B: Pack them into the blend file using File -> External Data -> Pack Resources - Start rendering [bug report.zip](https://archive.blender.org/developer/F13014155/bug_report.zip)

Harlekin commented

2022-04-21 13:53:07 +02:00

Added subscriber: @randomGuyOnTheInternet

Raimund Klink commented

2022-04-21 14:01:28 +02:00

Added subscriber: @Raimund58

Germano Cavalcante commented

2022-04-22 18:08:08 +02:00

Added subscriber: @mano-wii

Germano Cavalcante commented

2022-04-22 18:08:08 +02:00

Changed status from 'Needs Triage' to: 'Needs Developer To Reproduce'

Germano Cavalcante commented

2022-04-22 18:08:08 +02:00

I can confirm the extra RAM consumption even though the images are not actually being used in Blender.

Memory is allocated in BKE_packedfile_new.

And right after it is written in the Blender file in BKE_packedfile_blend_write.

If data is written to the .blend file, and the packed image is not being used, should it really consume RAM?

If this is a bug or intentional, I imagine it's up to the #core team to inform us.

I can confirm the extra RAM consumption even though the images are not actually being used in Blender. Memory is allocated in `BKE_packedfile_new`. And right after it is written in the Blender file in `BKE_packedfile_blend_write`. If data is written to the .blend file, and the packed image is not being used, should it really consume RAM? If this is a bug or intentional, I imagine it's up to the #core team to inform us.

Harlekin commented

2022-04-22 18:21:02 +02:00

Thanks for taking a look. I should have specified that I mean RAM consumption during rendering specifically, not during the packing

Bastien Montagne commented

2022-04-25 12:21:15 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2022-04-25 12:21:15 +02:00

Extra RAM consumption with packed files is expected, by definition you put the external file's content inside your current Blend file. We could maybe theoretically try to 'clear' the embedded file once the Blend has been written on disk, and have a mechanism to reload it when required, but that would likely be complex to get working properly, since it would be violating quiet a few rules/principles in current data-blocks and blend file handling.

But issue here is related to rendering, and I think this is because CoW copies of IDs (from depsgraph evaluation) currently blindly also duplicate the packed files data! This is obviously wrong, will prepare a patch for this.

Extra RAM consumption with packed files is expected, by definition you put the external file's content inside your current Blend file. We could maybe theoretically try to 'clear' the embedded file once the Blend has been written on disk, and have a mechanism to reload it when required, but that would likely be complex to get working properly, since it would be violating quiet a few rules/principles in current data-blocks and blend file handling. -------------- But issue here is related to rendering, and I think this is because CoW copies of IDs (from depsgraph evaluation) currently blindly also duplicate the packed files data! This is obviously wrong, will prepare a patch for this.

Bastien Montagne commented

2022-04-25 17:23:56 +02:00

Added subscriber: @Sergey

Bastien Montagne commented

2022-04-25 17:23:56 +02:00

hrrrmmm, investigating the code further, images are actually NOT duplicated in depsgraph... Nor during rendering it seems.

So besides the first point in my first comment ("Extra RAM consumption with packed files is expected").

Regarding rendering, when using Cycles there is an optimization for external files, which get directly read by Cycles into its own image buffers, completely by-passing Blender's Image ID /ImBuff system. So if you do not load the image in Blender itself (by e,g displaying it in the Image editor or the 3DView). This is not possible currently for packed Images, which need to be first read by Blender into its own cache (ImBuff buffers system), and then stored again by Cycles into its own data structure, resulting in the image being three times in memory (one as packed data, one as Blender cache, one as Cycles image). Thanks @Sergey for helping understanding what's going on here.

While this is not optimal, there is no bug here. Thanks for the report.

For the record, here are my measurements with provided test file:

Render engine	Before Rendering	Rendering	Difference
Cycles linked:	0.4 G	1.6 G	1.2 G
Cycles packed:	0.9 G	2.6 G	1.7 G

Eevee linked:	0.4 G	3.4 G	3.0 G
Eevee packed:	0.9 G	3.9 G	3.0 G

hrrrmmm, investigating the code further, images are actually NOT duplicated in depsgraph... Nor during rendering it seems. So besides the first point in my first comment ("Extra RAM consumption with packed files is expected"). Regarding rendering, when using Cycles there is an optimization for external files, which get directly read by Cycles into its own image buffers, completely by-passing Blender's Image ID /ImBuff system. So if you do not load the image in Blender itself (by e,g displaying it in the Image editor or the 3DView). This is not possible currently for packed Images, which need to be first read by Blender into its own cache (ImBuff buffers system), and then stored again by Cycles into its own data structure, resulting in the image being three times in memory (one as packed data, one as Blender cache, one as Cycles image). Thanks @Sergey for helping understanding what's going on here. While this is not optimal, there is no bug here. Thanks for the report. ------ For the record, here are my measurements with provided test file: | Render engine | Before Rendering | Rendering | Difference | | ----- | ----- | ----- | ----- | | Cycles linked: | 0.4 G | 1.6 G | 1.2 G | | Cycles packed: | 0.9 G | 2.6 G | 1.7 G | | | | | | | Eevee linked: | 0.4 G | 3.4 G | 3.0 G | | Eevee packed: | 0.9 G | 3.9 G | 3.0 G |

Bastien Montagne commented

2022-04-25 17:24:05 +02:00

Changed status from 'Needs Developer To Reproduce' to: 'Archived'

Bastien Montagne closed this issue

2022-04-25 17:24:05 +02:00

Alessandro Padovani commented

2023-03-04 11:16:52 +01:00

With the actual design, if this is not a bug, then I guess it means that packing is intended for assets distribution, not for production. Once we get a packed asset in a project, we have to unpack it for production.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Noticeably higher RAM consumption with packed textures #97510