blender either crashes immediately after rendering when using multiple denoise and file output nodes, or the output of random denoise nodes saved with the file output node are pure black. #77023

New Issue

michael campbell · 2020-05-24T14:47:47+02:00

michael campbell commented

2020-05-24 14:47:47 +02:00

System Information
Operating system: Windows-10-10.0.17763-SP0 64 Bits
Graphics card: GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.92

Blender Version
Broken: version: 2.90 (sub 2), branch: master, commit date: 2020-05-20 20:56, hash: 8a0840b0ee
Worked: (newest version of Blender that worked as expected)

Short description of error
blender crashes immediately after rendering if multiple render layers are connected to file output nodes

Exact steps for others to reproduce the error
Create a .blend file
set rendering to cuda gpu experimental and render size to 1080p
make a simple scene, box, monkey, light etc.
enable all main render passes (diffuse, glossy, transmission, environment, shadows, volume, denoising data)
make 9 linked copies of the scene (making 10 scenes in total)
place a different camera in each scene and set it as the render camera for that scene.
make an 11th empty scene for the compositor to live in
in the compositor add a render layer node and a file output node, connecting all of the render layer outputs to the file output node. Set the file output node exr multilayer full float rgba.
place 4 denoisers between the render layer node and the file output node
add a layout node and place all nodes inside of it
select the layout and duplicate it 9 times (making 10 in total)
change each of the file outputs to output a different file name
hit render

All the rendering will complete, but as soon as rendering completes it crashes, initially I thought this was because blender is trying to save all 10 exr's simultaneously rather than one at a time, but I'm guessing it's more likely that it's trying to run all denoise nodes simultaneously rather than one at a time?

My system is happy with around 3 instead of 10, so most likely down to trying to calculate expensive nodes simultaneously rather than sequentially.

example of my setup:

**System Information** Operating system: Windows-10-10.0.17763-SP0 64 Bits Graphics card: GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.92 **Blender Version** Broken: version: 2.90 (sub 2), branch: master, commit date: 2020-05-20 20:56, hash: `8a0840b0ee` Worked: (newest version of Blender that worked as expected) **Short description of error** blender crashes immediately after rendering if multiple render layers are connected to file output nodes **Exact steps for others to reproduce the error** Create a .blend file set rendering to cuda gpu experimental and render size to 1080p make a simple scene, box, monkey, light etc. enable all main render passes (diffuse, glossy, transmission, environment, shadows, volume, denoising data) make 9 linked copies of the scene (making 10 scenes in total) place a different camera in each scene and set it as the render camera for that scene. make an 11th empty scene for the compositor to live in in the compositor add a render layer node and a file output node, connecting all of the render layer outputs to the file output node. Set the file output node exr multilayer full float rgba. place 4 denoisers between the render layer node and the file output node add a layout node and place all nodes inside of it select the layout and duplicate it 9 times (making 10 in total) change each of the file outputs to output a different file name hit render All the rendering will complete, but as soon as rendering completes it crashes, initially I thought this was because blender is trying to save all 10 exr's simultaneously rather than one at a time, but I'm guessing it's more likely that it's trying to run all denoise nodes simultaneously rather than one at a time? My system is happy with around 3 instead of 10, so most likely down to trying to calculate expensive nodes simultaneously rather than sequentially. example of my setup: ![image.png](https://archive.blender.org/developer/F8550786/image.png)

michael campbell commented

2020-05-24 14:47:47 +02:00

Added subscriber: @3di

Philipp Oeser commented

2020-05-25 10:02:31 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2020-05-25 10:02:31 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Philipp Oeser commented

2020-05-25 10:02:31 +02:00

Could you please post your example .blend file?
(this just saves time to reproduce -- especially if multiple devs are having a look at this)

Could you please post your example .blend file? (this just saves time to reproduce -- especially if multiple devs are having a look at this)

michael campbell commented

2020-05-25 14:20:58 +02:00

Thanks, blend file attached. The compositor is a little less complex in this one and the file is a lot less heavy, which resulted in it not crashing, but it's still not working. The problem definitely seems to be with the denoise nodes running concurrently, because after render if you try all 10 of the exr images created by the file output node, you'll notice some of the images are missing data because of some of the denoiser nodes failing.

The compositor is in the product compositing scene, open it up hit f12, wait for it to complete, watch the ram usage during rendering and then during the compositing phase where it's trying to denoise and save out the exr's, then using the image input node, try each of the 10 output exr files. You'll notice that some of them have information missing, this is a result of some of the denoise nodes failing. Re-rendering will result in different denoiser nodes failing.

It will probably still crash if you have less than 32gb ram.

Solutions I can think of would be either store the raw renders to the hdd before moving to the next render, and then recalling them one at a time when each file output node is triggered, or alternatively, at the end of each render, execute the full node tree connected to that renderlayers node, clear the memory and then move onto the next renderlayers node/render.

file:

crash with multiple denoise and file out nodes in compositor1.blend

Here's what the scene will look like if all of the denoise nodes work correctly, although I've set it to very only one sample to speed up testing for you:

Thanks, blend file attached. The compositor is a little less complex in this one and the file is a lot less heavy, which resulted in it not crashing, but it's still not working. The problem definitely seems to be with the denoise nodes running concurrently, because after render if you try all 10 of the exr images created by the file output node, you'll notice some of the images are missing data because of some of the denoiser nodes failing. The compositor is in the product compositing scene, open it up hit f12, wait for it to complete, watch the ram usage during rendering and then during the compositing phase where it's trying to denoise and save out the exr's, then using the image input node, try each of the 10 output exr files. You'll notice that some of them have information missing, this is a result of some of the denoise nodes failing. Re-rendering will result in different denoiser nodes failing. It will probably still crash if you have less than 32gb ram. Solutions I can think of would be either store the raw renders to the hdd before moving to the next render, and then recalling them one at a time when each file output node is triggered, or alternatively, at the end of each render, execute the full node tree connected to that renderlayers node, clear the memory and then move onto the next renderlayers node/render. file: [crash with multiple denoise and file out nodes in compositor1.blend](https://archive.blender.org/developer/F8552814/crash_with_multiple_denoise_and_file_out_nodes_in_compositor1.blend) Here's what the scene will look like if all of the denoise nodes work correctly, although I've set it to very only one sample to speed up testing for you: ![image.png](https://archive.blender.org/developer/F8552840/image.png)

michael campbell commented

2020-05-25 14:25:11 +02:00

if you mute 6 of the renderlayers trees then it'll all work usually.

michael campbell commented

2020-05-28 16:34:07 +02:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Philipp Oeser commented

2020-05-29 14:23:55 +02:00

Afraid I cannot really test this (as indeed I am running out of RAM before it finishes)

Sidenote: I am getting this Dependency cycle reported:

Dependency cycle detected:
  NTCompositing Nodetree/Shading Component/MATERIAL_UPDATE() depends on
  NTCompositing Nodetree/Parameters Component/DRIVER(nodes["Group"].inputs[0].default_value) via 'Driver -> Driven Property'
  NTCompositing Nodetree/Shading Component/MATERIAL_UPDATE() via 'RNA Target -> Driver'
Detected 1 dependency cycles

Afraid I cannot really test this (as indeed I am running out of RAM before it finishes) Sidenote: I am getting this Dependency cycle reported: ``` Dependency cycle detected: NTCompositing Nodetree/Shading Component/MATERIAL_UPDATE() depends on NTCompositing Nodetree/Parameters Component/DRIVER(nodes["Group"].inputs[0].default_value) via 'Driver -> Driven Property' NTCompositing Nodetree/Shading Component/MATERIAL_UPDATE() via 'RNA Target -> Driver' Detected 1 dependency cycles ```

michael campbell commented

2020-05-29 14:36:47 +02:00

Hi Philipp. Try muting the contents of 4 or 5 of the layouts, this should then work on 16gb ram, although I'm not sure if that will mean the issue with the denoise nodes rendering out black instead of the denoised image will still happen? Depends if that's a ram issue too.

I'm not sure on the cause of the dependancy cycles unless there's some remnant of a deleted driver still existing in the blend data? Or perhaps that's something to do with the fact grouped nodes exist outside of the bpy.data.scene.node_tree? Maybe try ungrouping the nodes with the driven parameters and see if that issue persists.

Hi Philipp. Try muting the contents of 4 or 5 of the layouts, this should then work on 16gb ram, although I'm not sure if that will mean the issue with the denoise nodes rendering out black instead of the denoised image will still happen? Depends if that's a ram issue too. I'm not sure on the cause of the dependancy cycles unless there's some remnant of a deleted driver still existing in the blend data? Or perhaps that's something to do with the fact grouped nodes exist outside of the bpy.data.scene.node_tree? Maybe try ungrouping the nodes with the driven parameters and see if that issue persists.

Richard Antalik commented

2020-06-17 23:26:53 +02:00

Added subscriber: @iss

Richard Antalik commented

2020-06-17 23:26:53 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Richard Antalik commented

2020-06-17 23:26:53 +02:00

I couldn't reproduce crash with this file. does this happen when you load factory settings? Can you upload debug log using blender_debug_log.cmd file?

I couldn't reproduce crash with this file. does this happen when you load factory settings? Can you upload debug log using `blender_debug_log.cmd` file?

michael campbell commented

2020-06-18 00:23:41 +02:00

Hi Richard, in the example file I posted above, how many of the renderlayers trees/denoiser nodes did you have unmuted? What system specs do you have? Are you definitely in the product compositing scene when you press f12?

Richard Antalik commented

2020-06-18 02:04:41 +02:00

I guess that the issue in my case is that I don't have nvidia GPU. I tried at least... I muted 5 trees as per recommendation for 16GB RAM.

In any case I am requesting this specific info just to exclude influence of externalities like addons and debug log because in 2.90 we see stack trace in case of crash even on windows so this will be something for developers to look at.

I guess that the issue in my case is that I don't have nvidia GPU. I tried at least... I muted 5 trees as per recommendation for 16GB RAM. In any case I am requesting this specific info just to exclude influence of externalities like addons and debug log because in 2.90 we see stack trace in case of crash even on windows so this will be something for developers to look at.

michael campbell commented

2020-06-18 02:07:09 +02:00

Hi Richard. I'll give that a try asap, in the mean time try enabling all 10 trees. GPU shouldn't matter as OIDN is CPU based to the best of my knowledge....unless it's related to having open cl turned on in the compositor settings?

Richard Antalik commented

2020-06-18 02:29:38 +02:00

No crash even with 10 trees. Output image is pink, but I don't quite understand how this setup is supposed to work, so I am not really making any conclusions on that.

michael campbell commented

2020-06-18 02:34:02 +02:00

Output being pink indicates that some of the denoise nodes have failed which means the passes can't be built back together i'd imagine. You can check by checking each of the outputs of the resulting exr file. Some of them will be black instead of the denoised pass.

Fahad Hasan commented

2020-06-18 03:40:42 +02:00

Added subscriber: @MD.FahadHassan

Fahad Hasan commented

2020-06-18 03:40:42 +02:00

No crash in Ryzen 7 16core. 32 GB RAM. GeForce GTX 950. Debain 10. Memory peak was 30,062 . No node was muted.
Possible reason of crash:

Memory maxed out.
Your scene0004 might have some glitch.
I used scene1004 in final compositing area.
Just disconnect the viewer and composite node while rendering. After the render, connect it.

Don't use Backdrop. Instead use image editor>Viewer node. [Just a free advice I give every time ;)]

But I must say, I had to hold my coffee seeing such peak after a long time.

No crash in Ryzen 7 16core. 32 GB RAM. GeForce GTX 950. Debain 10. Memory peak was 30,062 . No node was muted. Possible reason of crash: - Memory maxed out. - Your scene0004 might have some glitch. - I used scene1004 in final compositing area. - Just disconnect the viewer and composite node while rendering. After the render, connect it. # Don't use Backdrop. Instead use image editor>Viewer node. [Just a free advice I give every time ;)] ![Screenshot from 2020-06-18 07-19-28.jpg](https://archive.blender.org/developer/F8629317/Screenshot_from_2020-06-18_07-19-28.jpg) ``` ``` But I must say, I had to hold my coffee seeing such peak after a long time.

michael campbell commented

2020-06-18 04:46:45 +02:00

Hi Fahad, can you confirm that all of the exr's contain the same data on the denoised passes as they do on the un-denoised passes?

I've just managed to complete without the crash, but the problem of some of the denoised passes saving as black persists.

@iss , here's the log files, memory was nearly maxed out, but it didn't crash. Can you check the log files to see what the problem is with the denoise nodes outputting black instead of the denoised pass?

blender_system_info.txt

blender_debug_output.txt

Hi Fahad, can you confirm that all of the exr's contain the same data on the denoised passes as they do on the un-denoised passes? I've just managed to complete without the crash, but the problem of some of the denoised passes saving as black persists. @iss , here's the log files, memory was nearly maxed out, but it didn't crash. Can you check the log files to see what the problem is with the denoise nodes outputting black instead of the denoised pass? [blender_system_info.txt](https://archive.blender.org/developer/F8629431/blender_system_info.txt) [blender_debug_output.txt](https://archive.blender.org/developer/F8629432/blender_debug_output.txt)

michael campbell commented

2020-06-18 04:55:16 +02:00

@iss I noticed that the debug output indicates dependancy cycles between the controls group and the saturation node. Could the denoise node bug be related to this? If so perhaps this is related to the below bug report:

https://developer.blender.org/T77279

EDIT: scrap that, I just tried re-rendering without the driver and most of the denoise nodes outputted black instead of the denoised pass.

@iss I noticed that the debug output indicates dependancy cycles between the controls group and the saturation node. Could the denoise node bug be related to this? If so perhaps this is related to the below bug report: https://developer.blender.org/T77279 EDIT: scrap that, I just tried re-rendering without the driver and most of the denoise nodes outputted black instead of the denoised pass.

michael campbell commented

2020-06-18 05:35:01 +02:00

This one's got a bit more information in it.

blender_debug_output.txt

This one's got a bit more information in it. [blender_debug_output.txt](https://archive.blender.org/developer/F8629582/blender_debug_output.txt)

Fahad Hasan commented

2020-06-18 17:54:30 +02:00

@3di Can you try it with 2.83?

michael campbell commented

2020-06-18 18:07:33 +02:00

@MD.FahadHassan , probably better if one of the devs try that, as they'll probably have a script for replacing the new denoise nodes with the old ones from 2.83

Fahad Hasan commented

2020-06-18 19:37:24 +02:00

Your scene is memory intensive and need too much resources and time. That's why asking.

michael campbell commented

2020-06-18 20:51:48 +02:00

Hi Fahad, I'm not sure I follow. That's the point of this bug report, Blender is crashing or the denoise nodes are failing to output anything due to improper memory handling (I'm guessing). They need to execute consecutively instead of simultaneously, and the result of each denoise node temporarily stored on the HDD instead of ram, that way only the memory required for an individual denoise node is needed at any one time. Once they're all complete then the data can be moved from the HDD back to wherever it's needed, in this case into an image output node for writing of the EXR.

Fahad Hasan commented

2020-06-18 20:58:20 +02:00

Let me test it once again for you. I am asking to test it as devs may not have that much time and hardware resource to compare the result with b2.8 and b2.9 which is AFIK updated with new OIDN library.

Fahad Hasan commented

2020-06-18 21:40:17 +02:00

@3di Result: Tested with B 2.83 and 2.9 (6899cb3c07). Ryzen 7 16 core, 30 GB Ram, GeForce GTX 950. Debian 10. Mem Peak 29,060.

Both version gives denoised data (Named as *_clean pass) properly in all EXRs. Tested with Natron and Blender compositor.
The only fail I had faced is the final comp was pink after render is done. Turning backdrop on/off or connecting-disconnecting or mute/unmute any node or any kind of "one/single" update event in the compositor was necessary to load the Final EXR file in the final comp. That makes sense as the final EXR was absent before the rendering is finished (NULL file with reference).

Hope it helps. Thanks.

@3di Result: Tested with B 2.83 and 2.9 (6899cb3c073e). Ryzen 7 16 core, 30 GB Ram, GeForce GTX 950. Debian 10. Mem Peak 29,060. 1. Both version gives denoised data (Named as *_clean pass) properly in all EXRs. Tested with Natron and Blender compositor. 2. The only fail I had faced is the final comp was pink after render is done. Turning backdrop on/off or connecting-disconnecting or mute/unmute any node or any kind of "one/single" update event in the compositor was necessary to load the Final EXR file in the final comp. That makes sense as the final EXR was absent before the rendering is finished (NULL file with reference). Hope it helps. Thanks.

michael campbell commented

2020-06-18 21:56:18 +02:00

That's great, hopefully the log files I provided will help the devs ascertain the cause of the issue for myself and other people experiencing it (the black output seems to be a common issue). Can you upload the 10 exr files your test created so I can double check all the passes for you?

michael campbell commented

2020-06-18 21:57:21 +02:00

wonder if the problem only happens on intel? That would be funny :)

Fahad Hasan commented

2020-06-18 21:57:57 +02:00

I have just deleted them. It was 1.1 GB.

michael campbell commented

2020-06-18 22:07:23 +02:00

ah ok, no problem. Should be enough info in the log files and above anyway for a dev to investigate the cause.

Evan Wilson commented

2020-06-18 23:04:54 +02:00

Added subscriber: @EAW

Evan Wilson commented

2020-06-18 23:04:54 +02:00

2.90 hasn’t updated to OIDN 1.2.1 just yet, so it is still running the same 1.0 version as 2.83.

Richard Antalik commented

2020-06-18 23:35:40 +02:00

In #77023#957738, @3di wrote:
ah ok, no problem. Should be enough info in the log files and above anyway for a dev to investigate the cause.

Just to clarify, what is the bug we are talking about? I am looking for crash as it is the title of this report, but there is no crash in logs. Blender has normally quit. That is either by user or by some addon somehow. There are lot addons used so I can't really tell what could happen. That is why I want to check if there is any crash when you test with factory settings loaded.

> In #77023#957738, @3di wrote: > ah ok, no problem. Should be enough info in the log files and above anyway for a dev to investigate the cause. Just to clarify, what is the bug we are talking about? I am looking for crash as it is the title of this report, but there is no crash in logs. Blender has normally quit. That is either by user or by some addon somehow. There are lot addons used so I can't really tell what could happen. That is why I want to check if there is any crash when you test with factory settings loaded.

michael campbell changed title from ~~blender crashes immediately after rendering if multiple render layers are connected to file output nodes~~ to blender either crashes immediately after rendering when using multiple denoise and file output nodes, or the output of random denoise nodes saved with the file output node are pure black.

2020-06-18 23:42:42 +02:00

michael campbell commented

2020-06-18 23:48:32 +02:00

I've updated the title. The issue is it either crashes when the render gets to the denoise/file output stage, or some of the denoise nodes output a black image instead of the denoised image....at least they save this way in the exr...so it could be the denoise nodes or the file output.

I haven't been able to reproduce the crash, so perhaps that was related to the build I was using way back when, when I initially reported the bug.

I can't spend any more time on this, as I've already spent three or four days testing and producing files for the devs to look at.

I'm pretty sure it's just a bad implementation in terms of memory management.

I've updated the title. The issue is it either crashes when the render gets to the denoise/file output stage, or some of the denoise nodes output a black image instead of the denoised image....at least they save this way in the exr...so it could be the denoise nodes or the file output. I haven't been able to reproduce the crash, so perhaps that was related to the build I was using way back when, when I initially reported the bug. I can't spend any more time on this, as I've already spent three or four days testing and producing files for the devs to look at. I'm pretty sure it's just a bad implementation in terms of memory management.

Fahad Hasan commented

2020-06-19 01:19:05 +02:00

Removed subscriber: @MD.FahadHassan

Richard Antalik commented

2020-06-22 05:36:40 +02:00

Changed status from 'Needs User Info' to: 'Needs Developer To Reproduce'

Richard Antalik commented

2020-06-22 05:36:40 +02:00

So I have rendered the file again, checked out all exr files, and all seems to be good on my machine. Result was composited correctly that is.

michael campbell commented

2020-06-22 16:39:15 +02:00

In #77023#961844, @iss wrote:
So I have rendered the file again, checked out all exr files, and all seems to be good on my machine. Result was composited correctly that is.

Try increasing the number of un-muted trees to 15, and ensure all denoise nodes are un-muted. It was teetering on the edge at 32gb ram. What differences are there between our hardware and operating system?

> In #77023#961844, @iss wrote: > So I have rendered the file again, checked out all exr files, and all seems to be good on my machine. Result was composited correctly that is. Try increasing the number of un-muted trees to 15, and ensure all denoise nodes are un-muted. It was teetering on the edge at 32gb ram. What differences are there between our hardware and operating system?

Sergey Sharybin commented

2020-06-24 16:26:19 +02:00

Added subscriber: @Sergey

Sergey Sharybin commented

2020-06-24 16:26:19 +02:00

@iss, from reading the discussion seems this is a an out-of-memory issue?

Philipp Oeser commented

2020-06-25 09:27:56 +02:00

Not sure if 5cfbc722d0 possibly changed this for the better?

Not sure if 5cfbc722d0 possibly changed this for the better?

Richard Antalik commented

2020-06-25 10:09:56 +02:00

In #77023#964394, @Sergey wrote:
@iss, from reading the discussion seems this is a an out-of-memory issue?

I wasn't able to reproduce any issue here, so I don't know. Claim is that this file will either crash or some denoise nodes will fail to create output.

> In #77023#964394, @Sergey wrote: > @iss, from reading the discussion seems this is a an out-of-memory issue? I wasn't able to reproduce any issue here, so I don't know. Claim is that this file will either crash or some denoise nodes will fail to create output.

aVersionOfReality commented

2020-08-17 22:06:23 +02:00

Added subscriber: @ColinBarton

Richard Antalik commented

2020-11-26 22:29:37 +01:00

@3di I am going over old reports. Can you still reproduce this issue in latest build form https://builder.blender.org/download/ ?

Jeroen Bakker commented

2021-04-12 14:29:12 +02:00

Added subscriber: @Jeroen-Bakker

Jeroen Bakker commented

2021-04-12 14:29:12 +02:00

Changed status from 'Needs Developer To Reproduce' to: 'Archived'

Jeroen Bakker closed this issue

2021-04-12 14:29:12 +02:00

Jeroen Bakker commented

2021-04-12 14:29:12 +02:00

There is already a mutex guarding multiple calls to OIDN so there is no parallel execution going on. The compositor memory efficiency can be improved and is part of a #81650 (New compositor system proposal)
I wasn't able to reproduce the pink output or crash.

The pink output might indicate something wrong in OIDN. I did add a fix so less memory is allocated during when waiting for the lock. Of course that would lead to less performance. As the performance of denoising is normally slow I don't see that as a problem.

I don't see any reasons to keep this ticket open.

If there is a crash, please add an easy reproducable case. I mean you can always create an overly complex scene that will break any software somehow. That should not be considered a bug.

There is already a mutex guarding multiple calls to OIDN so there is no parallel execution going on. The compositor memory efficiency can be improved and is part of a #81650 (New compositor system proposal) I wasn't able to reproduce the pink output or crash. The pink output might indicate something wrong in OIDN. I did add a fix so less memory is allocated during when waiting for the lock. Of course that would lead to less performance. As the performance of denoising is normally slow I don't see that as a problem. I don't see any reasons to keep this ticket open. If there is a crash, please add an easy reproducable case. I mean you can always create an overly complex scene that will break any software somehow. That should not be considered a bug.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

blender either crashes immediately after rendering when using multiple denoise and file output nodes, or the output of random denoise nodes saved with the file output node are pure black. #77023

Don't use Backdrop. Instead use image editor>Viewer node. [Just a free advice I give every time ;)]