x3d scene importer inefficient with big files; (with proposed fix) #38261

New Issue

Jonatan Bijl · 2014-01-17T12:35:53+01:00

Jonatan Bijl commented

2014-01-17 12:35:53 +01:00

I recently needed to load a huge wrl file (zipped it was 63MB). That failed on memory issues.
Then I started to look through the x3d scene importer, and found out what it was doing:

read the whole file into a string (!)
split the string into lines, do some filtering, and merge that.
split the string into lines, do some filtering (place { and [ and ] and } on separate lines) , and merge that.
split the string into lines, remove unneeded whitespace , and merge that.
split the string into lines
start processing the lines.

I found it hard to understand how the parser internally worked (very complicated, and no documentation of the AST that is being generated), but figured out that all the functions are passing line-numbers to each other, telling where in the global array of lines they should be reading.

As far as I could see, it is never really needed to look backward more than 1 line. So I used some generator-functions and iterators to read and cleanup the wrl file line by line, and created an object which behaves like an array to replace the lines array.

However, I don't have a wide range of wrl files to check all the features of the importer, nor do I have x3d files to check if that still works.
import_x3d_test.zip
This zip file contains an acceptance test, and a small test wrl file (the real files I can't include due to copyright reasons).
It also contains the version of the importing code that was shipped with 2.69, and my version, and tests whether or not the ASTs that are generated by the two, are the same.

To test it, unzip the zip somewhere, paste some wrl files in the folder, open blender, open comparison.py in blender, and add the names of the wrl files in the TESTFILES list.
It will write the output to files, and print a diff in blender console if there are differences.

I recently needed to load a huge wrl file (zipped it was 63MB). That failed on memory issues. Then I started to look through the x3d scene importer, and found out what it was doing: 1) read the whole file into a string (!) 2) split the string into lines, do some filtering, and merge that. 3) split the string into lines, do some filtering (place { and [ and ] and } on separate lines) , and merge that. 4) split the string into lines, remove unneeded whitespace , and merge that. 5) split the string into lines 6) start processing the lines. I found it hard to understand how the parser internally worked (very complicated, and no documentation of the AST that is being generated), but figured out that all the functions are passing line-numbers to each other, telling where in the global array of lines they should be reading. As far as I could see, it is never really needed to look backward more than 1 line. So I used some generator-functions and iterators to read and cleanup the wrl file line by line, and created an object which behaves like an array to replace the lines array. However, I don't have a wide range of wrl files to check all the features of the importer, nor do I have x3d files to check if that still works. [import_x3d_test.zip](https://archive.blender.org/developer/F69668/import_x3d_test.zip) This zip file contains an acceptance test, and a small test wrl file (the real files I can't include due to copyright reasons). It also contains the version of the importing code that was shipped with 2.69, and my version, and tests whether or not the ASTs that are generated by the two, are the same. To test it, unzip the zip somewhere, paste some wrl files in the folder, open blender, open comparison.py in blender, and add the names of the wrl files in the TESTFILES list. It will write the output to files, and print a diff in blender console if there are differences.

Jonatan Bijl commented

2014-01-17 12:35:53 +01:00

Changed status to: 'Open'

Jonatan Bijl commented

2014-01-17 12:35:53 +01:00

Added subscriber: @jonim8or

Jonatan Bijl commented

2014-01-17 14:18:23 +01:00

I had one part not figured out yet: how to handle referenced files.
Here's a new version of my "test suite"
import_x3d_test.zip

I had one part not figured out yet: how to handle referenced files. Here's a new version of my "test suite" [import_x3d_test.zip](https://archive.blender.org/developer/F69676/import_x3d_test.zip)

Brecht Van Lommel commented

2014-01-17 14:39:07 +01:00

Added subscriber: @ideasman42

Jonatan Bijl commented

2014-01-20 14:29:48 +01:00

Small fix to make multi-line strings in file work. import_x3d.py

Small fix to make multi-line strings in file work. [import_x3d.py](https://archive.blender.org/developer/F71024/import_x3d.py)

Jonatan Bijl commented

2014-04-16 14:36:31 +02:00

My previous version accidently broke the x3d importer (minidom could not handle the line-by-line input).
The x3d parser is now very slow (well, as slow as it used to be) for big files, because it still loads the file into a string first. For wrl files however it is much memory-friendlier.

Here's my new version, with also the PixelTexture code from #39594 integrated into it.
import_x3d.py

My previous version accidently broke the x3d importer (minidom could not handle the line-by-line input). The x3d parser is now very slow (well, as slow as it used to be) for big files, because it still loads the file into a string first. For wrl files however it is much memory-friendlier. Here's my new version, with also the PixelTexture code from #39594 integrated into it. [import_x3d.py](https://archive.blender.org/developer/F85000/import_x3d.py)

Brendon Murphy commented

2016-01-08 07:51:12 +01:00

Added subscriber: @BrendonMurphy

Brendon Murphy commented

2016-01-08 07:51:12 +01:00

Changed status from 'Open' to: 'Archived'

Brendon Murphy closed this issue

2016-01-08 07:51:12 +01:00

Brendon Murphy self-assigned this 2016-01-08 07:51:12 +01:00

Brendon Murphy commented

2016-01-08 07:51:12 +01:00

closing as archived, no activity 1.5 years.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

x3d scene importer inefficient with big files; (with proposed fix) #38261