Page MenuHome

High Memory Usage when render animation with Persistant Images enabled, and using Fix (T77734) reproducible with Latest 2.90 Blender
Closed, ResolvedPublic

Description

System Information
Tested on

  1. Threadripper 2970WX / 32GB DDR4 / Aorus X399 / 1KW PSU / GTX 1070
  2. Ryzen 3900X / 32GB DDR4 / 850W PSU / 1070 GTX

Operating system:
Windows 10 PRO

Graphics card:
GTX 1070 8GB

Blender Version
Broken 2.90 - 2-JULY 2020
previous editions of blender can see the memory usage problem described in this bug report, but at risk of more frequent crash to desktops, the CTD seems to be fixed with 2.90

Short description of error
Memory usage increases at an insane rate progressively when render animation frames,
The recent fix T77734 has proven to make blender more stable, but an underlining issue is still here where memory usage is seen to be abused

Exact steps for others to reproduce the error

  1. Download my file, https://drive.google.com/file/d/1vrl-kien3uePeQ6oexIHdCU35FSTdGA-/view?usp=sharing
  2. open the blend file
  3. file is setup to to enable the viewporn with materials, ensure that the materials load fully
  4. render as animation
  5. monitor the render frames information, specifically the memory usage in the rendering window.

for additional support I have created some videos
Please view the 3 videos for how the issue responds with 2.81a and 2.90, (2.81a crashes, where 2.90 stays alive to show the memory usage seems to increase way too much)

FAIL Attempt 1 (Blender 2.81a) - https://youtu.be/LXXpIUKanP4
FAIL Attempt 2 (Blender 2.81a) - https://youtu.be/fqIev-diTfU
you can see that blender just simply disapears when trying to render with persistent images enabled.

New Blender 2.90,
Improved, but issues Attempt 3 (Blender 2.90 2-July-2020 download) - https://youtu.be/5NoCXXHsLbw

I have 4 computers with similar spec, the Threadripper 2970WX being most powerful,
I have seen that on the AMD 3900X with 32GB ram and a GTX 1070, I see similar results but with 1 CTD, the threadripper was ran 5 times and never CTD, but says System is out of GPU Host shared memory when it stops at frame 13.

Hi.
I have seen improvements made with the latest 2.90 downloaded today,
Please take the time to view my videos which monitor my computers Main Memory, and CUDA memory.
Persistent images seems to be storing way too much information which clogs up blender and starts acting up.

you will see that with the blender 2.90 video, blender will render a few frames really really fast then there are massive delays before rendering frames.

I have noticed the following, computer system memory increases intensely during the rendering of each frame progressively from say 4-6GB at the start, and ends around 24GB+

When I use persistent images i see huge gains in time rendered for animations in both Windows and Linux, I am truly hoping that this issue where blender seems to be way to memory hungry can be optimized.

in the render preview window you can see that memory usage is approx 23GB at the frame it stops working.
hwinfo will show that the data for the video card become overwhelmed..

I have tested the same on 4 different class systems,
1st is a threadripper 2970WX 1070GTX
2nd is a Ryzen 3900X 32GB RAM 1070GTX
3rd is Intel i7 8600K 32GB RAM 1080TI
4th is an Intel i7 8750H 32GB 1060 GTX laptop

apart from system 2 which does a CTD with 2.90, the other 3 act similar to system 1 which the videos were recorded.

I can see that persistent images when DISABLED, blender will work fine. but I loose performance.

Enable Simplified to limit textures down to 1024 help, but over a long run of frame renders it will end up with same memory full issue.

Many thanks
Shaun

Event Timeline

I am not sure if this has been resolved or not, as I have found multiple entries.

I just wanted to add that I have the same error. I upgraded to 2.83 this morning and immediatly I started getting this issue, where texture images are loaded on every frame even when persistent images is selected and they seem to stack up on every frame. When memory reaches the available limit then the animation stops.
Blender and the machine do not crash but I have to restart blender to reset the memory before trying a new render.

My system settings (although I doubt they are useful for this issue):
Processor: AMD Ryzen 3700x 8-core 3.59GHz
RAM: 16GB
GPU: RTX 2070 Super

Disabling the persistend images of course resolves the issue.

thank you.
kind regards,
Byron

Each frame takes a long time to render. So it is not very feasible for me to try to reproduce the problem.

The scenario described is too time consuming for us to track down, we require the bug reporter to narrow down the problem.

Normally .blend files can be simplified by removing most objects and disabling settings, until the problem reveals itself more clearly.

The frame I tested the memory usage by the system was actually smaller in blender 2.90 (11.2 GB in 2.83 vs 10.6 GB in 2.90).

Hi Germano,

The .blend file I have included and similar
Blend files which are easy to cause this problem to blender usually have high res textures. It's the loading of the textures into memory is what I believe the cause of the issue is.
When persistent images is enabled.. I am not a Dev. But it would appear that the textures maybe being duplicated? I don't understand blenders cycles engine.. but for some reason enable persistent images seems to be adding stuff to the point the system memory is so full blender cannot work anymore. There might be room to tweak or limit how persistent images works?

If the time taken to render the frames is too long you can easily solve that by reducing the samples down to a much lower number try render samples at 5. It should render frames in just a few seconds.

It allows the frames to render much quicker. And you'll see the issues I am speaking about after blender renders approx 13 frames in animation mode. (Ctrl - F12)

The sample size is not important here. It is having persistent images on and watching blender after rendering perhaps 5 frames start using memory in a very loose way.

I also notice that if you render as just using CPU the problem still exsist which leads me to believe this isn't really something to do with cuda memory usage.

The issue described where memory usage is very high by the 13th frame (on my system in the render window with the render timer etc, memory is reported to be using 24,000MB) before it stops on my system.

Using simplified and forcing textures down to 1024 or 512.
Will help but eventually at say render animation maybe the 40th frame will be at 24,000MB..

Persistent images always seems to fill the memory in one way or another.

Difference between render GPU Hybrid vs CPU Only
GPU Hybrid
Lastly, when I mentioned in my original report. I suggest that the blender program slows down after rendering the 4th-5th frame. In my HWinfo window I notice that 8GB video card memory is full (D3D dedicated memory), then the data seems to populate the D3D dynamic memory) which is basically system memory.
CPU Only
Instead of using the memory from the video card, blender starts filling up system memory at a much faster rate!
My system configuration 32GB physical and 32GB virtual memory for total of 64GB commit
Rendering at a sample rate of 5 I reach frame 28 in approx 2 minutes where memory usage is 52,000MB
In this scenario blender CTD

If you have any questions or need help further diagnose the issue please let me know. I am very eager to have this issue explored by blender Dev/expert.

I am not a cycles developer. But I can forward the report to the specialized team.

However, this report is not in the recommended style to be confirmed/forwarding.

The files are very large, it takes a long time to investigate and the problem is not well defined.

I'm tagging the module anyway

Hi shaun,

I took a look into this issue, and it appears that something is off with your file. Indeed when persistent images is enabled, each frame would create more and more render slots for the images, allocating memory until it has it all. But when I dived a bit deeper I've noticed that the image_user objects for the textures in this file are set up as for 100 frame long animations. While I've no idea why this may be this way, and where did you get these materials (they look generated by 3rd party software to me), this does cause the described out-of-memory issue by increasing the memory usage by 100 times or so. While at it I've noticed a little possible issue with Blender that made the situation with animated textures and persistent images even worse, but this is a whole another topic. Anyway, if I'm not mistaken about this problem, here's how to repair the provided test file:

  1. Make sure you have a backup of your work, I don't guarantee anything.
  2. Open the blend file.
  3. Go to the scripting tab.
  4. Press New.
  5. Paste the code from below in the script editor.
  6. On the main menu click Window > Toggle System Console - here you'll see the script output when you run it.
  7. You may run the script in "no fix" mode (explained below) to see all the issues without changing anything. The output will be in the system console.
  8. To fix all the issues run the script in "fix" mode (explained below). You may run it a couple times. When everything is fixed there will be no output.
  9. Save, and you may try rendering. Note that previous renders leave a lot of stuff in memory. So for the best experience reopen the file.

Also note that this script will mess up the real animated textures if there are such (I din't see any in your example). Though if you do use animated textures, I highly recommend against using "persistent images" with them currently.

The script:

import bpy

# Set this to True to correct the situation by making all the textures
# used in the shader nodes 1 frame long.
# Note that every texture user will be made single frame, even ones
# that are meant to be animated.
# Set this to False for read only check.
fix = False


defaults = {
    "frame_offset": 0,
    "frame_start": 1,
    "frame_duration": 1,
    "frame_current": 1,
}


def checkNode(nd):
    if nd.type != "TEX_IMAGE":
        if nd.type == "GROUP":
            for subNd in nd.node_tree.nodes:
                checkNode(subNd)
        return

    isDefault = True
    for k in defaults:
        if getattr(nd.image_user, k) != defaults[k]:
            isDefault = False
            break
            
    if not isDefault:
        print()
        print(f"Material: '{mat.name}'")
        print(f"  Node: '{nd.name}'")
        print(f"    Using texture: {nd.image.name}")
        print(f"    from {nd.image.filepath}")
        for k in defaults:
            print(f"    Has {k}: {getattr(nd.image_user, k)}")
            if fix and getattr(nd.image_user, k) != defaults[k]:
                print(f"    Setting {k} to {defaults[k]}")
                setattr(nd.image_user, k, defaults[k])


print("Checking image users")

for mat in bpy.data.materials:
    if mat.node_tree is None:
        continue
    
    for nd in mat.node_tree.nodes:
        checkNode(nd)

for ng in bpy.data.node_groups:
    checkNode(ng)

print("Done")

To run it in "just check, don't fix" mode leave fix = False. To run it in "check and fix" mode, change it to fix = True.

If the problem proves to be in the file and not Blender code, than please close the issue as invalid so the devs don't waste time trying to reproduce it. Otherwise I have a simplified test file and a tip on what to look for (I've already said it - image_users with 100 frame duration and look for the slots allocation).

Anyways, it was great fun debugging this on an 8GB potato (:

Hi Vincent,

I am glad to say that your script has reconfigured my file and has solved the problems described in my report, blender no longer goes memory crazy, it rendered 150 frames in a matter of minutes,
thank-you so much for your deep dive. would you kindly provide me a few minutes to explain the problem more so I can understand it.

I am actually wanting to know more about this problem that my file was causing to blender?
I dont even know where to find the setting inside the image texture where it was set at 100 frames? is this somewhere in node setup? or only adjustable via script?

I did use a plugin which imports DAZ Projects. I will write to the script developer to see if they can look to see if they can improve/fix or if its intended!

are you able to elaborate on a scenario where I would want to use 100 frames of texture?

thanks again Vincent, I understand the my issue has crossed the bounds of user support, and I do appreciate your time.
I will change the status to completed.

Thank-You
Shaun

Vincent Blankfield (vvv) closed this task as Invalid.Jul 4 2020, 7:18 PM
Vincent Blankfield (vvv) claimed this task.

The problem was that the Image Texture nodes in the material graphs had their image_user objects misconfigured. The image_user (https://docs.blender.org/api/current/bpy.types.ImageUser.html) is an internal object, that lets renderer know how to use a texture. It's not meant to be directly accessed by the user, so it has no direct representation in the UI. It is indirectly accessed when you create a texture. Try adding a texture node to some material, and then set it to some video (or change its source type to Movie or Image Sequence) - you'll see the options like Frames, Start Frame, Offset, etc. So this is used when creating animated textures (here's your use case for a 100 frame texture). But it has no sense with texture type set to Singe Image, as was in the file. There's also an advanced mode of the Outliner, meant for the scripters use, that can show all the internal data structures of a blend file. It's called "Data API" (second dropdown on the left in the outliner header). There you can drill down to the objects in question. The path would be Materials>MATERIAL_NAME>Node Tree>Nodes>TEXTURE_NODE_NAME>Image User. I'll remind that changing stuff directly in the Data API outliner mode will most likely break something if you don't know what you're doing.

So in your case the importer script created those object with wrong parameters. It has set the texture type to Single Image, but duration, start frame and current frame were set up as if it were an animation. Which makes no sense (at least to me). You can see what was set to what values when you run my script on the misconfigured file in fix = False mode. The output (in the system console) will show something like this for each misconfigured texture node:

Material: 'wraps-1'
  Node: '4be_Charmer_gloves_bump_01'
    Using texture: 4be_Charmer_gloves_bump_01
    from D:\Games\DAZ 3D\MyLibrary\Runtime\textures\4blueyes\Charmer\4be_Charmer_gloves_bump_01.jpg
    Has frame_offset: 0
    Has frame_start: 1
    Has frame_duration: 100
    Has frame_current: 0

With this image_user configuration and the Persistent Images option checked the renderer will advance the texture current frame on each animation frame rendered, allocate new slots for the texture images frame, render the frame, don't free the render slots (because persistent images means it). Then on the next frame it will check if all the needed images are loaded - it checks that the image and it's frame are the same. The image frame will differ, so once again it will allocate new render slots, an so on. The memory usage will go through the roof very quickly. Considering that the "single image but multi frame" configuration isn't legal, it can't be said that Blender did something wrong here.

As I've already invested into this investigation, I'm now curious to know how exactly did it happen. Can you drop a link to the exact version of the importer plugin (and maybe the data it was importing) so I can take a look at it? Of course if such sharing isn't restricted by it's license. Just don't upload the files here. A drive link, as in your report, would be perfect. Not sure I will be able to hunt the problem down, as I have no clue what DAZ is, still I'm curious to check it out.

I hope the Blender team will forgive this little off topic conversation here in the name of finding the truth and making the infrastructure around blender more robust.

shaun (toxsickcity) added a comment.EditedJul 5 2020, 6:40 PM

Hi Vincent.

Thanks again so much for follow up. I totally understand that bug tracker is not meant for user support which this is kind of turning into. Thankyou none the less as this issue has plagued me for several versions of blender. Possibly others too so it's good this exsist for someone may search this and get help..

Thomas has joined in and created a bug report here and linked to my report. Not sure why tho. I asked him about it.
Anyways I opened a case for him to look into this at his Bitbucket

https://bitbucket.org/Diffeomorphic/import_daz/issues/122/image_user-objects-for-the-textures-in

Anyway Daz is a 3d program with much free content and lots of paid content. It's no way as powerful as blender as to why many will use Thomas's plugin to directly import a Daz project.

Thomas's plugin is out of this world in terms of powerful features to do what it does. The length and time and work required to bring in Daz content use to be painful he has simplified this to the extent where it's basically a single click! so I thank him also! He has scripted ways for the skeleton to convert to rigify.. seriously he has performed magic..

I know of a site to get the Daz content.. :P I have some 600GB of stuff on my disks and some really sweet models that cycles renders beautifully

Anyways give me a little bit of time to compile up a free use scenario for you to able to play with daz studio

It's all free so if you have spare time and wanted to look at it. It's available free. Go grab it, install it let the download manager download all the free models. I think genesis is there

The content is where money is made I guess. They do give you a few freebies via download manager tho. But the free stuff I don't know of the level texture sizes etc.. most stuff I have are 4k sometimes more.

I will see if I can replicate issue using default content and share a project file.

None the less Thomas seems to have stated he rectified the issue but had no time to test it.

Thanks Vincent.

Shaun.

EDIT: I think what threw me to think this was a blender issue, was I have downloaded .blend files from few websites.. and not only my project files have the issue. It might be lots more people use Daz then I thought. Or maybe in some scenario blender might add image as movie? This is unknown to me.. yeah so that's why I pushed this problem to hard..

Nevermind the request for content. A bug report @Thomas Larsson (thomasl) created explains the situation perfectly. Looks like there is a weird default hardcoded value of 100 frame duration. I've already found where it is, changed, tested - works fine in this usecase. Will post a patch after updating the repos, rebuilding and testing. Will need some devs to look at it, as there may have been some reason for such value I don't know of.

Hi Brecht,

Can you advise me how I can track the progress of this fix?

Has this been applied already to 2.83.2 or 2.90 daily?
Is this the right link to see the actual fixes applied to blendeR?
https://developer.blender.org/T77348

many thanks