Please see here:
Will there GPU-Memorymanagement ever be as good as it was in Blender 2.62 ?
Please see here:
When reporting issues like this, please attach an example .blend file that shows the issue, this could be caused by many things. Also mention exact blender version you tested, the operating system you use, the graphics card, etc.
My guess is that this may be due to float textures (exr, hdr, tiff), those are now using full float precision rather.
To finish my sentence: full float precision rather than byte precision. This can make quite a big difference, these textures take up 4x more memory then. You can work around that by converting them to png so that they are only in byte precision.
Any more information here? Otherwise will have to close report. I've been investigating this but could not find any other sources of memory increase in Cycles itself besides float images if they are used. Bmesh will use some extra memory for big meshes, and some modifiers like the array modifier will use more memory, but that's unrelated to Cycles and a known issue.
The main tests I did with this (smaler) Scene: http://www.archiviz.ch/download/Kitchen_06052012_01.zip
For the testresults inclusive detailed Errormessages see here: http://www.archiviz.ch/download/GPU_Memorytest_01_06052012.ods
As you can see, when I increase the Renderresolution to 3500x2500 px then Blender crashes.
When I render another Scene, which has mutch more Faces and uses mutch more VRAM than the Scene above, then Blender 2.63 does NOT crash.
The other Scene is this one: http://www.archiviz.ch/download/Living_06052012_01.zip
I do not understand why the smaler Scene crashes at 3500x2500 and bigger one doesn't.
I hope my tests help to explain.
So, from what I understand:
* This is not an issue with newer builds. In 2.63 there was 2.5% more memory usage, but that's hardly a major regression.
* Blender CPU memory usage indeed increased with Bmesh, it's a known issue but should not affect GPU rendering here.
The bug here then is that the more complex scene can render without running out of memory while the simpler one can't. Why that happens I'm not sure, will have to test myself, but GPU memory management can be tricky. There are other things in GPU memory that can take up space, and memory can get fragmented making it impossible to allocate a big enough buffer for a 3500x2500 image.
I intend to optimize memory usage for big renders this week, where instead of progressive rendering the entire image it will render it tile by tile. This should solve the out of memory problem for this case I think. It also helps to disable the Z pass in Render Layers, this will also use up unneeded memory if you don't use it.
Thank you Brecht.
You are right, it's not an issue with newer builds.
I just thought so at first because in my messy testings one Scene worked without problems in 2.62 and a 99% identical Scene did not work in 2.63.
Is there a tool that helps to visualize whats exactly in GPU Memory ?
I only have a monitoringtool that shows me the GPU Memory usage.
Anyway, thank you for your fantastic effort !