Merging objects that use meshes with users count of 1 increases the file size massively #71823

New Issue

roxlu - · 2019-11-23T16:14:44+01:00

roxlu - commented

2019-11-23 16:14:44 +01:00

System Information
Operating system: Linux-5.3.12-arch1-1-x86_64-with-arch 64 Bits
Graphics card: GeForce GTX 980/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.31

Blender Version
Broken: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: 26bd5ebd42
Worked: (optional)

Short description of error
I'm working on a architectural model. I imported the model from Rhino 3D using Collada. The file has about 300K vertices. I'm now merging similar objects by selecting them and pressing CTRL + J. After merging some objects the file size increased from 82MB to 1.4GB. First I thought that this was caused because I was merging objects that were sharing the same mesh data, but when I iterate over the meshes and print mesh.users their value is all 1, for each mesh. I purged all ophaned data too.

As a test, I exported the .blend file to .obj.

The resulting .obj file was ~35MB.
Then I imported the file in a new .blend and saved that one, the file size of this .blend is 57MB.

I've started a discussion here too: https://blenderartists.org/t/massive-file-size-increase-after-joining-meshes-objects/1192317/4

UPDATE 1
I toggled the - [x] compression box when saving and now the file is 18MB. I still think this is a bug as 300K vertices does not result in such a file size increase. Opening that file of 18MB results in ~2.6GB memory usage.
https://imgur.com/a/22hNroI

UPDATE 2
I think this is partly caused by the undo buffer somehow or at least, this issue is twofold; disabling the undo buffer allows me at least to continue working.

What can cause this increase?

Exact steps for others to reproduce the error
I'm using a private 3D model which I can't share. I'm happy to share this privately but I can't post it online.

**System Information** Operating system: Linux-5.3.12-arch1-1-x86_64-with-arch 64 Bits Graphics card: GeForce GTX 980/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.31 **Blender Version** Broken: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: `26bd5ebd42` Worked: (optional) **Short description of error** I'm working on a architectural model. I imported the model from Rhino 3D using Collada. The file has about 300K vertices. I'm now merging similar objects by selecting them and pressing CTRL + J. After merging some objects the file size increased from 82MB to 1.4GB. First I thought that this was caused because I was merging objects that were sharing the same mesh data, but when I iterate over the meshes and print `mesh.users` their value is all 1, for each mesh. I purged all ophaned data too. As a test, I exported the .blend file to .obj. - The resulting .obj file was ~35MB. - Then I imported the file in a new .blend and saved that one, the file size of this .blend is 57MB. I've started a discussion here too: https://blenderartists.org/t/massive-file-size-increase-after-joining-meshes-objects/1192317/4 **UPDATE 1** I toggled the `- [x] compression` box when saving and now the file is 18MB. I still think this is a bug as 300K vertices does not result in such a file size increase. Opening that file of 18MB results in ~2.6GB memory usage. https://imgur.com/a/22hNroI **UPDATE 2** I think this is partly caused by the undo buffer somehow or at least, this issue is twofold; disabling the undo buffer allows me at least to continue working. What can cause this increase? **Exact steps for others to reproduce the error** I'm using a private 3D model which I can't share. I'm happy to share this privately but I can't post it online.

roxlu - commented

2019-11-23 16:14:44 +01:00

Added subscriber: @roxlu

Anthony Edlin commented

2019-11-23 18:22:42 +01:00

Added subscriber: @AnthonyEdlin

Anthony Edlin commented

2019-11-23 18:22:42 +01:00

Yes if you do a series of object joins it's creating a massive undo stack, storing the data multiple times. You can do like you did and disable undo, also if you do not join them 1 by 1 but as one big group it will only be 1 undo. I don't think there is a bug here.

roxlu - commented

2019-11-23 19:43:19 +01:00

Ok, good to know! The crashes were indeed most likely be caused because my PC ran out of mem. But the file size shouldn't change from 18MB to 1.25GB.

roxlu - commented

2019-11-26 09:34:36 +01:00

I just noticed, a day later (?!) that when I open the file my memory goes from 18MB to 466MB which is still al lot but understandable. I removed my /tmp files and I'm not sure if blender uses shared memory, but I think something went wrong with cache files. But that's just a wild guess.

Campbell Barton commented

2019-11-28 08:41:29 +01:00

Changed status from 'Open' to: 'Archived'

Campbell Barton closed this issue

2019-11-28 08:41:29 +01:00

Campbell Barton self-assigned this 2019-11-28 08:41:29 +01:00

Campbell Barton commented

2022-04-26 11:29:06 +02:00

Changed status from 'Archived' to: 'Needs Developer To Reproduce'

Campbell Barton reopened this issue

2022-04-26 11:29:06 +02:00

Campbell Barton commented

2022-04-26 11:45:12 +02:00

Re-opened because closing should always give a response for why.

Philipp Oeser commented

2022-04-26 11:53:31 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2022-04-26 11:53:31 +02:00

OK, we now have #96926 (Merging/joining into single object can massively increase file size) which seems similar and are trying to come up with a repro case there.

So, the reason this was probably closed here is that we dont really have such a repro case, @roxlu : if you can provide one, please share it in #96926 instead.
Will close this one again.

OK, we now have #96926 (Merging/joining into single object can massively increase file size) which seems similar and are trying to come up with a repro case there. So, the reason this was probably closed here is that we dont really have such a repro case, @roxlu : if you can provide one, please share it in #96926 instead. Will close this one again.

Philipp Oeser commented

2022-04-26 11:53:50 +02:00

Closed as duplicate of #96926

Philipp Oeser closed this issue

2022-04-26 11:53:50 +02:00

Mark-Deamer-1 commented

2023-08-26 13:42:48 +02:00

Morning, First post ! I am currently trying to move our teams process away from Maya to Blender, we deal with a massive amount of CAD data for numerous car manufactures, we are experiencing exactly the same issue as above. I've screen captured a simplified version of our workflow to see if it is a bug or if I'm doing something stupid. I'll outline the steps.

Blender Version 3.6.2
Undo's disabled.
Import 1.02 FBX file.
Unparent parts from empties and apply all transforms. ( No increase in memory usage. )
Join similar textured meshes, as more meshes are joined memory usage increases ( in some cases on larger files filling the entire system memory )
Purge all orphaned data and save.
As you can see just these simple operation increase the file size from 2.11gb to 46.7gb, I've tried everything I can think of to find the leak/unused data but I am at a loss.
Reopen Save and as you can see 46.7gb of system memory used but only 2.11gb is my data, this is where it gets odd and I believe Blender is saving a ton of orphaned data or just garbage ( Remember undo's are disabled and I use Clean-ups constantly. )
Export to an obj and the file drops back down to 2.48gb, reopen in blender and all the data is the same and nothing is missing but the file is back to 2.04gb again ! madness. The export and reimport flushes all the unwanted data but is a pain to keep doing. any help would be appreciated. Link below to the screen capture, couldn't get it smaller than 1.2gb at 4k Sorry.

https://1drv.ms/v/s!AoarOZXHLMNWirJ5XDDjIchwGRp2VA?e=fWChO1

A link to a zip below as Onedrive only seems to stream in 1080p

https://1drv.ms/u/s!AoarOZXHLMNWirJ6OKSxqjOvCXjFCg?e=IyhFk2

Good luck.

Morning, First post ! I am currently trying to move our teams process away from Maya to Blender, we deal with a massive amount of CAD data for numerous car manufactures, we are experiencing exactly the same issue as above. I've screen captured a simplified version of our workflow to see if it is a bug or if I'm doing something stupid. I'll outline the steps. Blender Version 3.6.2 Undo's disabled. Import 1.02 FBX file. Unparent parts from empties and apply all transforms. ( No increase in memory usage. ) Join similar textured meshes, as more meshes are joined memory usage increases ( in some cases on larger files filling the entire system memory ) Purge all orphaned data and save. As you can see just these simple operation increase the file size from 2.11gb to 46.7gb, I've tried everything I can think of to find the leak/unused data but I am at a loss. Reopen Save and as you can see 46.7gb of system memory used but only 2.11gb is my data, this is where it gets odd and I believe Blender is saving a ton of orphaned data or just garbage ( Remember undo's are disabled and I use Clean-ups constantly. ) Export to an obj and the file drops back down to 2.48gb, reopen in blender and all the data is the same and nothing is missing but the file is back to 2.04gb again ! madness. The export and reimport flushes all the unwanted data but is a pain to keep doing. any help would be appreciated. Link below to the screen capture, couldn't get it smaller than 1.2gb at 4k Sorry. https://1drv.ms/v/s!AoarOZXHLMNWirJ5XDDjIchwGRp2VA?e=fWChO1 A link to a zip below as Onedrive only seems to stream in 1080p https://1drv.ms/u/s!AoarOZXHLMNWirJ6OKSxqjOvCXjFCg?e=IyhFk2 Good luck.

Mark-Deamer-1 referenced this issue

2023-08-26 13:54:36 +02:00

Merging/joining into single object can massively increase file size #96926

Philipp Oeser referenced this issue

2023-08-28 11:26:45 +02:00

Merging/joining into single object massively increases file size / Fills all available memory. #111564

Philipp Oeser commented

2023-08-28 11:30:51 +02:00

So this was reported again in #111564

And the reason seems to be that joining does not limit the number of UV layers to 8 (as done elsewhere in blender).
So you might end up with hundreds or thousands of UV layers (and each of them occupying memory for a -- now very large -- mesh).

I think this is probably what was the issue in #96926 as well.

I think the situation should still be improved:

either limiting the amount of UV layers upon joining to 8 (as we elsewhere in the UI)
merging the UV layers based upon index (so merge all first uv layers of all sources into one on the destination, merge all second uv layers of all sources into a second uv layer on destination, ...) -- this could be an option on the operator
if the above doesnt work out for some reason, at least spit out a warning in the UI that more than 8 UV layers have been created

I will open another issue for this shortly...

So this was reported again in #111564 And the reason seems to be that joining does not limit the number of UV layers to 8 (as done elsewhere in blender). So you might end up with hundreds or thousands of UV layers (and each of them occupying memory for a -- now very large -- mesh). I think this is probably what was the issue in #96926 as well. I think the situation should still be improved: - either limiting the amount of UV layers upon joining to 8 (as we elsewhere in the UI) - merging the UV layers based upon index (so merge all first uv layers of all sources into one on the destination, merge all second uv layers of all sources into a second uv layer on destination, ...) -- this could be an option on the operator - if the above doesnt work out for some reason, at least spit out a warning in the UI that more than 8 UV layers have been created I will open another issue for this shortly...

Philipp Oeser referenced this issue

2023-08-28 11:31:19 +02:00

Merging/joining into single object can massively increase file size #96926

Philipp Oeser referenced this issue

2023-08-28 13:30:46 +02:00

Joining objects can create more customdata layers (e.g. UVMaps) than allowed, can eat huge amounts of RAM, bloats the filesize #111608

Philipp Oeser commented

2023-08-28 13:35:56 +02:00

I opened #111608 for this.

Sign in to join this conversation.