Clean up CMake scripts for automated tests #83222

New Issue

Sybren A. Stüvel · 2020-11-30T12:46:19+01:00

Sybren A. Stüvel commented

2020-11-30 12:46:19 +01:00

56aa5b0d8c restructured unit tests, reducing the number of test binaries and optimizing for link time. D9649: Tests: run suites instead of individual test cases changes how tests are grouped & run, optimizing for run time. These changes have caused some duplication and other smells to the CMake code related to automated tests.

After D9649 has been accepted, these need to be cleaned up.

Things to look into:

- Which of the BLENDER_SRC_GTEST_EX, BLENDER_SRC_GTEST, BLENDER_TEST, and BLENDER_TEST_PERFORMANCE are still in use?
- Which of the blender_add_test_executable, blender_add_test_lib, and blender_add_test_suite are still in use?
- The GTest.cmake file contains Google Test code, modified for our use in Blender. It may be clearer if we clean it up a bit more (removing what we don't use) and put it in a Blender-specific file, rather than keeping a copied-but-modified file around.
- There is no overview of how tests are defined, and which macros/functions are used for that.

56aa5b0d8c restructured unit tests, reducing the number of test binaries and optimizing for link time. [D9649: Tests: run suites instead of individual test cases](https://archive.blender.org/developer/D9649) changes how tests are grouped & run, optimizing for run time. These changes have caused some duplication and other smells to the CMake code related to automated tests. After [D9649](https://archive.blender.org/developer/D9649) has been accepted, these need to be cleaned up. Things to look into: - - [ ] Which of the `BLENDER_SRC_GTEST_EX`, `BLENDER_SRC_GTEST`, `BLENDER_TEST`, and `BLENDER_TEST_PERFORMANCE` are still in use? - - [ ] Which of the `blender_add_test_executable`, `blender_add_test_lib`, and `blender_add_test_suite` are still in use? - - [ ] The `GTest.cmake` file contains Google Test code, modified for our use in Blender. It may be clearer if we clean it up a bit more (removing what we don't use) and put it in a Blender-specific file, rather than keeping a copied-but-modified file around. - - [ ] There is no overview of how tests are defined, and which macros/functions are used for that.

Sybren A. Stüvel commented

2020-11-30 12:46:19 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Sybren A. Stüvel commented

2020-11-30 12:46:19 +01:00

Added subscribers: @dr.sybren, @LazyDodo, @Sergey

blender-admin commented

2021-01-11 11:29:30 +01:00

This issue was referenced by 54f8a5dd73

This issue was referenced by 54f8a5dd737388d89332f3074eba01edc9efa8c1

Ankit Meel commented

2022-01-18 11:16:52 +01:00

Added subscriber: @ankitm

Ankit Meel commented

2022-01-18 11:16:52 +01:00

While the other problems with gtest_add_tests are listed in detail at https://www.kitware.com/dynamic-google-test-discovery-in-cmake-3-10/, what irks me right now are these things:
1 cmake rerun after every single change in a test cpp file. The workflow of

change test code -> press run button -> build happens -> test runs ... loop
has changed to
change test code -> press run button -> cmake rerun (this takes more time than the incremental build!) -> press run button again -> xcode processes new changes after cmake rerun (also takes time) -> build happens -> test runs ... loop.

2

Individual tests can still be run if desired by invocation of bin/tests/blender_test --gtest_filter=suitename.testname.

except one has to find *SAN_OPTIONS environment variables, arguments/flags etc (by dry run) and set them manually that code set otherwise.

Some time limitation could be overcome somewhat by running them in parallel -j2. If binary invocation is saving significant time, I assume long running tests like baking/ rendering are not running anyway.

Hierarchy/ grouping tests in targets like say bf_functions_tests problem might be solved by programmatically creating a dummy test fixture for a target. https://groups.google.com/g/googletestframework/c/FhedgGP2-Fs/m/5z10n9pDWeUJ and then using regex.

While the other problems with `gtest_add_tests` are listed in detail at https://www.kitware.com/dynamic-google-test-discovery-in-cmake-3-10/, what irks me right now are these things: 1 cmake rerun after _every single change_ in a test cpp file. The workflow of - change test code -> press run button -> build happens -> test runs ... loop has changed to - change test code -> press run button -> cmake rerun (this takes more time than the incremental build!) -> press run button again -> xcode processes new changes after cmake rerun (also takes time) -> build happens -> test runs ... loop. 2 > Individual tests can still be run if desired by invocation of bin/tests/blender_test --gtest_filter=suitename.testname. except one has to find *SAN_OPTIONS environment variables, arguments/flags etc (by dry run) and set them manually that code set otherwise. Some time limitation could be overcome somewhat by running them in parallel `-j2`. If binary invocation is saving significant time, I assume long running tests like baking/ rendering are not running anyway. Hierarchy/ grouping tests in targets like say `bf_functions_tests` problem might be solved by programmatically creating a dummy test fixture for a target. https://groups.google.com/g/googletestframework/c/FhedgGP2-Fs/m/5z10n9pDWeUJ and then using regex.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Clean up CMake scripts for automated tests #83222