Mesh import/export with high vertex count is very slow #53236

New Issue

Adam · 2017-11-02T17:57:19+01:00

Adam commented

2017-11-02 17:57:19 +01:00

System Information
Win 7 AMD HD 78xx

Blender Version
Broken: always

Short description of error
Exporting or importing meshes from Blender which have a high number of vertives (> 1 million) take ages to import or export.

Exact steps for others to reproduce the error

Create Suzanne
Use Subsurf until the polycount is in the millions ( 5 -10 million)
Export the mesh (obj or fbx - without UVs/normals will be faster, naturally)
Wait forever ;)

Importing the mesh will take an equally long time.

This is a giant blocker to sculpting workflow in Blender as it makes exporting to for example xNormal extremely time consuming.

**System Information** Win 7 AMD HD 78xx **Blender Version** Broken: always **Short description of error** Exporting or importing meshes from Blender which have a high number of vertives (> 1 million) take ages to import or export. **Exact steps for others to reproduce the error** - Create Suzanne - Use Subsurf until the polycount is in the millions ( 5 -10 million) - Export the mesh (obj or fbx - without UVs/normals will be faster, naturally) - Wait forever ;) Importing the mesh will take an equally long time. This is a giant blocker to sculpting workflow in Blender as it makes exporting to for example xNormal extremely time consuming.

Adam commented

2017-11-02 17:57:19 +01:00

Changed status to: 'Open'

Adam commented

2017-11-02 17:57:19 +01:00

Added subscriber: @Win

Adam commented

2017-11-02 18:05:09 +01:00

T53236_MeshExport.blend

[T53236_MeshExport.blend](https://archive.blender.org/developer/F1087343/T53236_MeshExport.blend)

Peter Boos commented

2017-11-02 19:18:38 +01:00

Added subscriber: @PeterBoos

Peter Boos commented

2017-11-02 19:18:38 +01:00

Changed status from 'Open' to: 'Archived'

Peter Boos closed this issue

2017-11-02 19:18:38 +01:00

Peter Boos self-assigned this 2017-11-02 19:18:38 +01:00

Peter Boos commented

2017-11-02 19:18:38 +01:00

The art of sculpting isnt super small polygons... Its knowing where to put details and where to remove them.
So if your bad ad sculpting then that comes with a cost.
Its not Blender's fault that processing lots of faces take more time, its mostly the speed of your hardware, you could buy SSD's and a faster computer, or better learn to sculpt.

Like for example suzane use subdiv 2 and use smooth surfaces, a final render would look the same.

The art of sculpting isnt super small polygons... Its knowing where to put details and where to remove them. So if your bad ad sculpting then that comes with a cost. Its not Blender's fault that processing lots of faces take more time, its mostly the speed of your hardware, you could buy SSD's and a faster computer, or better learn to sculpt. Like for example suzane use subdiv 2 and use smooth surfaces, a final render would look the same.

Adam commented

2017-11-02 20:01:25 +01:00

I don't understand your charge that I am bad at sculpting. For highly detailed sculpts resolutions of well over 10 million points are common.

That this is not Blender's fault is also wrong. Other software (take zBrush for example) handles large mesh exports in reasonable timeframes of under 30 seconds, not up to 15 minutes.

And of course I'm aware that making a 10 mil vert suzanne is pointless. It was just an example.

Sculpting is also not the only area that the long load times pose an issue. It happens with photogrammetry generated meshes as well.

Please reopen the task, this should be adressed.

I don't understand your charge that I am bad at sculpting. For highly detailed sculpts resolutions of well over 10 million points are common. That this is not Blender's fault is also wrong. Other software (take zBrush for example) handles large mesh exports in reasonable timeframes of under 30 seconds, not up to 15 minutes. And of course I'm aware that making a 10 mil vert suzanne is pointless. It was just an example. Sculpting is also not the only area that the long load times pose an issue. It happens with photogrammetry generated meshes as well. Please reopen the task, this should be adressed.

Peter Boos commented

2017-11-02 20:23:03 +01:00

If this is your regular job, then i stronlgy recommend you a SSD, this is not a bug in code; the waiting time you have is simply caused by the IO speed of your harddisk.
That suszane of 667MB took me about 2 to 3 secconds to load on SSD disk. So we realy cannot blame Blender here, but if it takes you 15 minutes upgrade your hardware, because your Disk IO speed is the bottle neck here.

If this is your regular job, then i stronlgy recommend you a SSD, this is not a bug in code; the waiting time you have is simply caused by the IO speed of your harddisk. That suszane of 667MB took me about 2 to 3 secconds to load on SSD disk. So we realy cannot blame Blender here, but if it takes you 15 minutes upgrade your hardware, because your Disk IO speed is the bottle neck here.

Adam commented

2017-11-02 20:54:19 +01:00

I have a SSD and when I export the model to the SSD as .obj it takes several minutes. So that can not be the problem.

I don't want to overreach but I would like to point out that the file size of the blend file is 667 kilobytes, not 667 megabyte (if by "that Suzanne of 667MB" you mean my example file). So it would load very quickly, the reason being that the geometry is created at runtime and not saved in the file since it is generated by a subsurf modifier.

I have a SSD and when I export the model to the SSD as .obj it takes several minutes. So that can not be the problem. I don't want to overreach but I would like to point out that the file size of the blend file is 667 kilobytes, not 667 megabyte (if by "that Suzanne of 667MB" you mean my example file). So it would load very quickly, the reason being that the geometry is created at runtime and not saved in the file since it is generated by a subsurf modifier.

Germano Cavalcante commented

2017-11-02 21:29:51 +01:00

Added subscriber: @mano-wii

Germano Cavalcante commented

2017-11-02 21:29:51 +01:00

Changed status from 'Archived' to: 'Archived'

Germano Cavalcante commented

2017-11-02 21:29:52 +01:00

The code for exporting obj is written in python and python is usually slow.
By analyzing the export_obj.py code, there are some parts that could be improved using the numpy package.

One of the reasons for the slow code is the line with:
face_index_pairs.sort (key = sort_func)
That is called when the Keep Vertex Order option is checked.

Other reasons for slowness are the loops:

for v in me_verts:;
for f, f_index in face_index_pairs: (2x)
and possibly others, but I got tired of waiting;

Actually this slowness is quite inconvenient, but the code is not broken so the priority is not high.

Currently the developers are focusing their time on the Blender 2.8 Project. So I have no expectations that this problem will be corrected so soon.

In any case, it is good that it is taken into account.

The code for exporting obj is written in python and python is usually slow. By analyzing the `export_obj.py` code, there are some parts that could be improved using the numpy package. One of the reasons for the slow code is the line with: ```face_index_pairs.sort (key = sort_func)``` That is called when the `Keep Vertex Order` option is checked. Other reasons for slowness are the loops: - `for v in me_verts:`; - `for f, f_index in face_index_pairs:` (2x) - and possibly others, but I got tired of waiting; Actually this slowness is quite inconvenient, but the code is not broken so the priority is not high. Currently the developers are focusing their time on the `Blender 2.8 Project`. So I have no expectations that this problem will be corrected so soon. In any case, it is good that it is taken into account.

Ray molenkamp commented

2017-11-02 21:35:02 +01:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2017-11-02 21:35:02 +01:00

@PeterBoos as much fun as yelling at users is that they can't model and their hardware sucks, I decided to sit down and do some tests.

Decided to give it a whirl, exporting to an SSD drive.

testing procedure:

Export from blender .obj which is a common format that virtually all applications know how to read
Import into meshlab
Import into blender
Export from meshlab

With the file from this ticket as is (subdiv level 7):

exporting .obj from blender     : 5m05s (950mb file)
Meshlab export to .obj          : crashed couldn't test. 

Meshlab import of the .obj file : 1m24s 
Blender Import of the .obj file : had to kill it after 5 minutes after it consumed 18GB of ram

changed subdiv level to 5 and payed attention to ram usage:

exporting .obj from blender     : 18s (76mb file) 
Meshlab export to .obj          : 4s 

Meshlab import of the .obj file : 3s / 375mb ram
Blender Import of the .obj file : 18s / 1.3gb ram

changed subdiv level to 6 and payed attention to ram usage:

exporting .obj from blender     : 78s (243mb file) 
Meshlab export to .obj          : 22s  

Meshlab import of the .obj file : 22s / 2.0gb ram
Blender Import of the .obj file : 75s / 4.3gb ram

So in all instances blender was MUCH slower, and used MUCH MUCH more ram compared to other applications while performing the same tasks.

@PeterBoos i'm not sure if you just had a bad day or something else, but generally we try not to be this abusive to users, please try to be a tad more civil in the future?

@PeterBoos as much fun as yelling at users is that they can't model and their hardware sucks, I decided to sit down and do some tests. Decided to give it a whirl, exporting to an SSD drive. testing procedure: 1) Export from blender .obj which is a common format that virtually all applications know how to read 2) Import into meshlab 3) Import into blender 4) Export from meshlab With the file from this ticket as is (subdiv level 7): ``` exporting .obj from blender : 5m05s (950mb file) Meshlab export to .obj : crashed couldn't test. Meshlab import of the .obj file : 1m24s Blender Import of the .obj file : had to kill it after 5 minutes after it consumed 18GB of ram ``` changed subdiv level to 5 and payed attention to ram usage: ``` exporting .obj from blender : 18s (76mb file) Meshlab export to .obj : 4s Meshlab import of the .obj file : 3s / 375mb ram Blender Import of the .obj file : 18s / 1.3gb ram ``` changed subdiv level to 6 and payed attention to ram usage: ``` exporting .obj from blender : 78s (243mb file) Meshlab export to .obj : 22s Meshlab import of the .obj file : 22s / 2.0gb ram Blender Import of the .obj file : 75s / 4.3gb ram ``` So in all instances blender was *MUCH* slower, and used *MUCH* *MUCH* more ram compared to other applications while performing the same tasks. @PeterBoos i'm not sure if you just had a bad day or something else, but generally we try not to be this abusive to users, please try to be a tad more civil in the future?

Adam commented

2017-11-02 21:54:07 +01:00

@LazyDodo @mano-wii Thank you for taking the issue seriously, the initial response was infuriating. It's sad that the issue won't be resolved soon but thank you for the hint with the vertex order, I'll keep that in mind :)

Vuk Gardašević commented

2017-11-02 22:06:53 +01:00

Added subscriber: @VukGardasevic

Vuk Gardašević commented

2017-11-02 22:06:53 +01:00

If you can, try exporting to Collada (dae) using the Collada (Default) exporter.
On a X3 455 (an old AMD 3 core):

it takes around 10-15 seconds for a subdivision 6 Suzanne (~ 2M polygons).
57 seconds for a subdiv 7 Suzanne (~ 8M polygons) ( around 4 GB used in total as the subdivision was applied with the base of 2GB just for the model). The DAE file is 986 MB in size.

If you can, try exporting to Collada (dae) using the Collada (Default) exporter. On a X3 455 (an old AMD 3 core): - it takes around 10-15 seconds for a subdivision 6 Suzanne (~ 2M polygons). - 57 seconds for a subdiv 7 Suzanne (~ 8M polygons) ( around 4 GB used in total as the subdivision was applied with the base of 2GB just for the model). The DAE file is 986 MB in size.

Adam commented

2017-11-02 22:13:27 +01:00

@VukGardasevic Thank you for the tip! My target platform (xnormal) even supports dae.

Peter Boos commented

2017-11-02 22:14:12 +01:00

Changed status from 'Archived' to: 'Open'

Peter Boos reopened this issue

2017-11-02 22:14:12 +01:00

Peter Boos commented

2017-11-02 22:14:12 +01:00

This comment was removed by @PeterBoos

*This comment was removed by @PeterBoos*

Peter Boos commented

2017-11-02 22:24:10 +01:00

i am civil,
Let me give you some more numbers
Apply modifiers then save the file to a Blend file thats 667Mb of raw (but also compressed) blender data,
Without blend saving only applying the modifier, that raw data inside blender its about 1.7 Gig what you see on screen.
that's to be converted to obj since the formats are not the same it takes around 7Gig of data to be processed to become a obj file of more then 900Mb.
well it depends on the horsepower of your machine, over here its about 60 seconds.
And i'm kinda amazed that such a huge file is still processed that fast, for the amounts of data that it has to process.
Maybe add more Ram, or a faster processor (more cores) might help as well. (60 sec tested on Asus K95v a laptop 4 years old now).

So to say blender has a bug its slow, with these data numbers to me seams a rather bold statement.

Though its also subjective "there is no standard for fast" here, upgrading your hardware for sure would help.
You claim 15 minutes what i do in a minute, well i reopen because i only wonder what a python dev might think of this bold claim.
If you reconsider these facts and my earlier advice you might as well close it yourself.

Maybe as i'm not english my language was a bit to formal, i would advice people who start to use blender to go to blender artist.
People there are helping people out with problems like these, this however is a bug report area, not wishing area.
Essentially its working, and it depends on your hardware how fast it works.
And it depends on your workflow how much data you generate.

i am civil, Let me give you some more numbers Apply modifiers then save the file to a Blend file thats 667Mb of raw (but also compressed) blender data, Without blend saving only applying the modifier, that raw data inside blender its about 1.7 Gig what you see on screen. that's to be converted to obj since the formats are not the same it takes around 7Gig of data to be processed to become a obj file of more then 900Mb. well it depends on the horsepower of your machine, over here its about 60 seconds. And i'm kinda amazed that such a huge file is still processed that fast, for the amounts of data that it has to process. Maybe add more Ram, or a faster processor (more cores) might help as well. (60 sec tested on Asus K95v a laptop 4 years old now). So to say blender has a bug its slow, with these data numbers to me seams a rather bold statement. Though its also subjective "there is no standard for fast" here, upgrading your hardware for sure would help. You claim 15 minutes what i do in a minute, well i reopen because i only wonder what a python dev might think of this bold claim. If you reconsider these facts and my earlier advice you might as well close it yourself. Maybe as i'm not english my language was a bit to formal, i would advice people who start to use blender to go to blender artist. People there are helping people out with problems like these, this however is a bug report area, not wishing area. Essentially its working, and it depends on your hardware how fast it works. And it depends on your workflow how much data you generate.

Arto Kitula commented

2017-11-02 22:26:05 +01:00

Added subscriber: @ArtoKitula

Arto Kitula commented

2017-11-02 22:26:05 +01:00

@PeterBoos, There is no point adding more juice to the CPU when python export/import works on single core. Converting import/export scripts to multithreading would help a lot. That's why this is really valid ticket.

Let's start using! http:*assimp.sourceforge.net

@PeterBoos, There is no point adding more juice to the CPU when python export/import works on single core. Converting import/export scripts to multithreading would help a lot. That's why this is really valid ticket. Let's start using! [http:*assimp.sourceforge.net ](http:*assimp.sourceforge.net)

Ray molenkamp commented

2017-11-02 22:33:44 +01:00

Changed status from 'Open' to: 'Archived'

Ray molenkamp closed this issue

2017-11-02 22:33:44 +01:00

Ray molenkamp commented

2017-11-02 22:33:44 +01:00

it's not a valid ticket, the import/export is working as designed, so technically this would be a feature request which doesn't belong on the tracker (so sadly i'm gonna have to abide by tracker rules and close the ticket).

That being said, there's nothing with how formal you were, but you straight up told an end user he didn't know how to do his job, no matter how you phrase that, that's not behavior that is appropriate on the tracker.

it's not a valid ticket, the import/export is working as designed, so technically this would be a feature request which doesn't belong on the tracker (so sadly i'm gonna have to abide by tracker rules and close the ticket). That being said, there's nothing with how formal you were, but you straight up told an end user he didn't know how to do his job, no matter how you phrase that, that's not behavior that is appropriate on the tracker.

Peter Boos commented

2017-11-02 22:52:30 +01:00

ah i see now, at first i responded quickly, i wasnt judging him, but the line was a general statement.
i should have written that poor sculpts, result in poor meshes, i was trying to explain that in general there (like a saying), but not to judge his specific level.

My excuses to him as, i had no intend to blame him there, bad tutor choice of words from my side.

ah i see now, at first i responded quickly, i wasnt judging him, but the line was a general statement. i should have written that poor sculpts, result in poor meshes, i was trying to explain that in general there (like a saying), but not to judge his specific level. My excuses to him as, i had no intend to blame him there, bad tutor choice of words from my side.

Jacob Merrill commented

2020-09-29 22:03:37 +02:00

Added subscriber: @JacobMerrill-1

Jacob Merrill commented

2020-09-29 22:03:37 +02:00

yeah I am using this right now on a production file and it takes ages
almost as long as baking a 4k

yeah I am using this right now on a production file and it takes ages almost as long as baking a 4k

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Mesh import/export with high vertex count is very slow #53236