RTX 3080 GPU Memory Leak #90011

New Issue

David Cherrie · 2021-07-21T11:58:25+02:00

David Cherrie commented

2021-07-21 11:58:25 +02:00

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3080/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 471.41

Blender Version
Broken: version: 2.93.1, branch: master, commit date: 2021-06-22 05:57, hash: 1b8d33b18c
Worked: Definitely was working in 2.91, maybe 2.92.

Short description of error
Whilst rendering an animation, hundreds of frames in it will stop rendering with the error "System is out of GPU memory". Rendering each frame as an .png image.

Exact steps for others to reproduce the error
Render an animation in Cycles using GPU compute Optix. Am unsure if this issue is just to do with Optix or if it is isolated to Cycles and not Eevee. It's hard to test since it takes an hour or two for it to reach 150 frames. As I understand, the GPU memory shouldn't rollover between each frame. But currently I can't render animations over night because it stops after an hour.

I've only tried this with one complicated file. I don't really have time to do complicated trouble shooting and testing as each test takes hours to eventuate.

I've googled and saw there has been a few people have this exact same issue on the exact same card.

**System Information** Operating system: Windows-10-10.0.19041-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 3080/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 471.41 **Blender Version** Broken: version: 2.93.1, branch: master, commit date: 2021-06-22 05:57, hash: `1b8d33b18c` Worked: Definitely was working in 2.91, maybe 2.92. **Short description of error** Whilst rendering an animation, hundreds of frames in it will stop rendering with the error "System is out of GPU memory". Rendering each frame as an .png image. **Exact steps for others to reproduce the error** Render an animation in Cycles using GPU compute Optix. Am unsure if this issue is just to do with Optix or if it is isolated to Cycles and not Eevee. It's hard to test since it takes an hour or two for it to reach 150 frames. As I understand, the GPU memory shouldn't rollover between each frame. But currently I can't render animations over night because it stops after an hour. I've only tried this with one complicated file. I don't really have time to do complicated trouble shooting and testing as each test takes hours to eventuate. I've googled and saw there has been a few people have this exact same issue on the exact same card.

David Cherrie commented

2021-07-21 11:58:25 +02:00

Added subscriber: @davidcherrie

Richard Antalik commented

2021-07-22 11:08:26 +02:00

Added subscriber: @iss

Richard Antalik commented

2021-07-22 11:08:26 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Richard Antalik commented

2021-07-22 11:08:26 +02:00

Can you check if it is possible to render from point where it crashes? Just wanted to check if this is leak or there are frames where complexity causes out of memory situation.

David Cherrie commented

2021-07-22 15:55:22 +02:00

Yes I was able to re-render after starting the render again. First time it got to frame 155 and then continued on from that frame which it got to frame 206. I would have expected it to have rendered another 155 frames but it only rendered 51.

The scene I was rendering not much changes throughout the animation. It’s heavy on a particle system with a non-moving crowd which are Make Humans. The only things that are animated in the render is the movement of the camera and 2 faces of an object which have an image sequence playing on them as giant LED screens.

No physics simulations or volumetrics, may be complex to me but probably not to a movie project.

Yes I was able to re-render after starting the render again. First time it got to frame 155 and then continued on from that frame which it got to frame 206. I would have expected it to have rendered another 155 frames but it only rendered 51. The scene I was rendering not much changes throughout the animation. It’s heavy on a particle system with a non-moving crowd which are Make Humans. The only things that are animated in the render is the movement of the camera and 2 faces of an object which have an image sequence playing on them as giant LED screens. No physics simulations or volumetrics, may be complex to me but probably not to a movie project.

Richard Antalik commented

2021-07-23 07:35:29 +02:00

This could be a leak from what you say. Can you upload the file simplified as much as possible? Not sure how large the file is in terms of filesize. But if it's large and it would crash even with textures and caches stripped, that would be best.

David Cherrie commented

2021-07-26 14:07:55 +02:00

I can't actually upload the file due to NDA. However working on another file (which I also can't share) which is totally seperate, after rendering multiple 4K images out over the course of working on the file for a few hours. I got the error message "System out of GPU memory". I escaped, didn't close the file but tried to re-render, same message came up and wouldn't start. I then went from LookDev to the internal shade view (forget what its called), pressed render then it started rendering.

I'll never render whilst looking in Render View in the viewport with cycles, as Blender 1 out of 3 times will either crash or also come up with "System out of GPU memory" but it seems that rendering in 4K especially when working in the document for several hours and rendering out lots of previews to see what it looks like, also results in the same issue that was made from a totally different project. So it's nothing specific with the files.

I have a few generic projects I can share which I was just playing around with. I'll do a test when I can to render an animation out of that and see if the issue occurs, then send the file to you.

I can't actually upload the file due to NDA. However working on another file (which I also can't share) which is totally seperate, after rendering multiple 4K images out over the course of working on the file for a few hours. I got the error message "System out of GPU memory". I escaped, didn't close the file but tried to re-render, same message came up and wouldn't start. I then went from LookDev to the internal shade view (forget what its called), pressed render then it started rendering. I'll never render whilst looking in Render View in the viewport with cycles, as Blender 1 out of 3 times will either crash or also come up with "System out of GPU memory" but it seems that rendering in 4K especially when working in the document for several hours and rendering out lots of previews to see what it looks like, also results in the same issue that was made from a totally different project. So it's nothing specific with the files. I have a few generic projects I can share which I was just playing around with. I'll do a test when I can to render an animation out of that and see if the issue occurs, then send the file to you.

Richard Antalik commented

2021-08-09 04:21:28 +02:00

Sorry I missed notification here. Regarding NDA's usually files can be simplified to extend where only few objects will be left and project contain no confidential material. Not sure if that is case here though. Having generic purpose built file is best.

Having said that, is there any update?

Sorry I missed notification here. Regarding NDA's usually files can be simplified to extend where only few objects will be left and project contain no confidential material. Not sure if that is case here though. Having generic purpose built file is best. Having said that, is there any update?

David Cherrie commented

2021-08-10 15:00:28 +02:00

I have been trying on a file that I am able to share but yesterday had it rendering for 6 hours using Cycles at 1920x1080 and got up to 590 frames with no issues.

Will keep an eye out and see if I come across it again and will try on the original file.

I have been trying on a file that I am able to share but yesterday had it rendering for 6 hours using Cycles at 1920x1080 and got up to 590 frames with no issues. Will keep an eye out and see if I come across it again and will try on the original file.

David Cherrie commented

2021-08-13 08:51:34 +02:00

I have isolated the issue to a linked geometry node. Got 23 frames rendered in the original problematic file before the System is out of GPU Memory. After stopping the render and trying to restart the render, on the first frame I get the GPU Memory message.

I've gone to the file that has the original geometry node and rendered that and it is absolutely fine and has been going for several hours. The files are pretty much the same except one has the original geometry node and the problematic file is linking to that node in the original file. The geometry node in question is a node I use for populating grandstands in my aquatic stadium I have built. Has a seat and combines it with a randomly selected human to sit down.

On Monday I'll try appending rather than linking the geometry node into the problematic project and see if that works. Then I can safely say its an issue to do with linking and not appending geometry nodes.

I have isolated the issue to a linked geometry node. Got 23 frames rendered in the original problematic file before the System is out of GPU Memory. After stopping the render and trying to restart the render, on the first frame I get the GPU Memory message. I've gone to the file that has the original geometry node and rendered that and it is absolutely fine and has been going for several hours. The files are pretty much the same except one has the original geometry node and the problematic file is linking to that node in the original file. The geometry node in question is a node I use for populating grandstands in my aquatic stadium I have built. Has a seat and combines it with a randomly selected human to sit down. On Monday I'll try appending rather than linking the geometry node into the problematic project and see if that works. Then I can safely say its an issue to do with linking and not appending geometry nodes.

Richard Antalik commented

2021-10-14 23:42:38 +02:00

@davidcherrie Hi, just wondering about status of this report, did you manage to identify the issue and can you share sample file?

Raimund Klink commented

2021-11-02 17:33:31 +01:00

Added subscriber: @Raimund58

Raimund Klink commented

2021-11-02 17:33:31 +01:00

Changed status from 'Needs User Info' to: 'Archived'

Raimund Klink closed this issue

2021-11-02 17:33:31 +01:00

Raimund Klink commented

2021-11-02 17:33:31 +01:00

No activity for more than a week. As per the tracker policy we assume the issue is gone and can be closed.

Thanks again for the report. If the problem persists please open a new report with the required information.

No activity for more than a week. As per the tracker policy we assume the issue is gone and can be closed. Thanks again for the report. If the problem persists please open a new report with the required information.

David Cherrie commented

2021-11-10 05:46:22 +01:00

In #90011#1235906, @iss wrote:
@davidcherrie Hi, just wondering about status of this report, did you manage to identify the issue and can you share sample file?

Sorry, this has come up in a new project. I believe its an issue with Optix and switching to CUDA in the same session seems to fix the problem. Switching back to Optix, in the same session, get the "System is out of GPU Memory" on first render. I can share this project with you.

> In #90011#1235906, @iss wrote: > @davidcherrie Hi, just wondering about status of this report, did you manage to identify the issue and can you share sample file? Sorry, this has come up in a new project. I believe its an issue with Optix and switching to CUDA in the same session seems to fix the problem. Switching back to Optix, in the same session, get the "System is out of GPU Memory" on first render. I can share this project with you.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

RTX 3080 GPU Memory Leak #90011