Page MenuHome

Modifiers automated testing
Needs ReviewPublic

Authored by Habib Gahbiche (zazizizou) on Sun, Jul 28, 8:18 PM.

Details

Summary

This is a small framework for automated modifiers testing. Using the provided script, tests can be generated and maintained very easily (which I plan to do if the task is approved).

The attached .blend file shows an example on how to use the framework and generate a test with two modifiers.
It is assumed that the blend file is in blender_build/lib/tests/modeling.

Diff Detail

Repository
rB Blender
Branch
master
Build Status
Buildable 4481
Build 4481: arc lint + arc unit

Event Timeline

Brecht Van Lommel (brecht) requested changes to this revision.Tue, Jul 30, 12:55 AM

I did not review the test in detail, but there's a few important things missing here in my opinion:

  • It should be possible to copy some command from the failed test log, which will open Blender with the input mesh, reference mesh and resulting mesh visible for comparison.
  • For failing tests, it should be possible to run BLENDER_TEST_UPDATE=1 ctest -R modifier exactly the same as render tests, and this would then update references meshes for failed tests.
  • The list of all modifiers and their parameters can be in a file in tests/python, not embedded in a .blend file. In fact no scripts should be embedded in test .blend files, it makes it hard to search and update them in case of API changes or other refactoring.
  • The meshes used as input for tests should be easy to extend, maybe a small set of meshes in a .blend file can be used for all modifiers. That way it's easy to add tests for specific topology, preservation of data layers, etc.
tests/python/modules/mesh_modifiers_test.py
44 ↗(On Diff #16639)

Can we avoid doing custom parameter parsing here? Any reason not to use a Python dictionary?

83 ↗(On Diff #16639)

Follow Blender Python naming conventions, camel case only for class names: https://wiki.blender.org/wiki/Style_Guide/Python

This revision now requires changes to proceed.Tue, Jul 30, 12:55 AM

These are good points, Brecht. And they equally apply to my mesh_ops_test.py and the bevel and boolean tests that use it. So I should revise them too. Or maybe we should just have one file with both mesh ops and modifier test functions in them.

Thanks Brecht, I will modify the test to fulfil these points.

Habib Gahbiche (zazizizou) updated this revision to Diff 16944.EditedThu, Aug 8, 2:09 PM

This is a work in progress to demonstrate how the class MeshTest can be used to test modifiers.

Operators, as well as a mixture of operators/modifiers can be tested using this class.

Next steps (in different patches):

  • integrate operators
  • write meaningful tests, e.g. use all modifiers with default parameters in random order
  • port existing bevel and boolean tests to use this framework

This test can be used by calling:

BLENDER_VERBOSE=1 ctest -R modifiers
or
BLENDER_VERBOSE=1 BLENDER_TEST_UPDATE=1 ctest -R modifiers to update a failing test.

tests/python/modules/mesh_test.py
30

I use the term 'operation' as a generalization for operator and modifiers. Is there a better word for that?

178

Ideally, I would do something like:

  • select relevant objects.
  • hide all non-selected objects.

This way, blender opens with only failing meshes visible. But I'm not sure how to write a loop to pass it in the command line. Any suggestion on how to do this better? Or is this good enough?

268

When tests are expected to fail, I still return False whereas some other tests return True (e.g. bevel). I think it's better to update the expected object and return False. This way it's more clear that the test was updated. When BLENDER_TEST_UPDATE is set to 1 and nothing gets updated, i.e. tests pass, then we return True.

Thanks! Did not look at the implementation yet, just replying to the comments.

tests/python/modules/mesh_test.py
30

Operation sounds good to me.

178

You could put this code in a separate .py file and then pass arguments like:

blender -P file.py -- a b c

And then read them from sys.argv.

268

To find out which tests were updated, "svn status" is already there. I find it more convenient if it runs the test again after updating and return the result of that.

Otherwise you have to run tests 3 times, once to discover the failure, once to update and once to verify it all passes.

Habib Gahbiche (zazizizou) marked 3 inline comments as done.Thu, Aug 8, 8:03 PM
Habib Gahbiche (zazizizou) added inline comments.
tests/python/modules/mesh_test.py
268

Running the tests 3 times does indeed sound bad. I updated the MeshTest class but it is up to the caller of MeshTest to check if the test has been updated and act accordingly (see modifiers.py)

Habib Gahbiche (zazizizou) marked an inline comment as done.Thu, Aug 8, 8:07 PM
Habib Gahbiche (zazizizou) updated this revision to Diff 16957.

added method is_test_updated() and improved printed command line when a test fails by opening blender with only relevant objects (failing meshes) visible.

  • Implemented operators testing in MeshTest
  • bevel and boolean tests now use the MeshTest testing class. They can be ran using

BLENDER_VERBOSE=1 BLENDER_TEST_UPDATE=1 ctest -R boolean_operator --verbose

Remaining tasks:

  • More modifiers tests (separate patch)
  • More operator tests (separate patch)
tests/python/bevel_operator.py
26–31

Doesn't from modules import mesh_test work, without any changes to sys.path?

That's what we use in cycles_render_tests.py.

32

Why reload?

40

Style: space after comma, no space before ].

166–167

It should run all the tests, not stop on the first failure. Otherwise you don't know what you're updating. Seeing all tests that failed also helps to see patterns to find the cause.

176

Perhaps the logic in this file could be deduplicated, by having something like this:

if __name__ == "__main__":
    ... construct list of MeshTest ...
    mesh_test.run_tests(test_list)

Where mesh_test.run_tests handles the traceback, check for updates, etc.

tests/python/modifiers.py
41

Style: no space before or after brackets.

tests/python/modules/mesh_test.py
123

Style: begin comment with capital, end with dot.

177–179

This can be done without operators:

bpy.data.objects.remove(self.expected_object, do_unlink=True)
193

The test.blend should not be saved when there is no update. Version control should not show the file as modified.

The way I imagined this to work is that it gives you a command to actually run the test. This is needed also to easily reproduce crashes, when it can't save a .blend.

Perhaps the simplest would be to add an additional parameter to the original command line, to run just one test, like:

blender modifiers.blend --python modifiers.py -- --show-test 25

Which could be constructed like:

command = list(sys.argv)
command.remove("--background")
command.append("--")
command.append("--show-test")
command.append("25")

Then bevel_operator.py, boolean_operator and modifiers.py should be simplified to just constructing a list of tests then, and leaving logic including like sys.argv parsing to mesh_test.py. But I think that's a good thing anyway.

It also means you can avoid passing that long Python expressions.

Brecht Van Lommel (brecht) requested changes to this revision.Mon, Aug 12, 11:19 AM
This revision now requires changes to proceed.Mon, Aug 12, 11:19 AM
Habib Gahbiche (zazizizou) marked 9 inline comments as done.Sun, Aug 18, 5:59 PM
Habib Gahbiche (zazizizou) added inline comments.
tests/python/bevel_operator.py
26–31

It doesn't work. I get the error:

ImportError: cannot import name 'mesh_test' from 'modules' (unknown location)s

tests/python/modules/mesh_test.py
193

I think generalising this in the MeshTest class would make it too complex. The general case has to consider

  • N test objects, M operators and P modifiers

But tests usually either test

  • 1 object and N operations or
  • N objects and 1 operation

So I would prefer to keep the failed tests handling on the modifiers.py side and not in the MeshTest class if that's ok with you. To deduplicate code, I introduced the helper classes OperatorTest to handle the typical case of N objects and 1 operator.

Habib Gahbiche (zazizizou) marked 2 inline comments as done.Sun, Aug 18, 6:02 PM
Habib Gahbiche (zazizizou) updated this revision to Diff 17245.

Added helper class OperatorTest to deduplicate code.
Changed style.