Testing New Code & Refactors¶
Blender CI/CD buildbot tool will help catching many common issues with PRs, such as sompilation failures on specific platforms or many breakages of Blender's unittests.
However, there are still cases that are not (yet) covered by this, and quite often changes to Blender code also require more specific, manual testing.
This page is not about adding unittests. If they are lacking in the area of code being worked on, it is always a good idea to start by adding or improving them!
Building¶
Even though a change may compile fine on a developer's own machine, it does not mean that it will build successfully for others.
The best option is to ask Blender buildbots to build the PR. This will catch already a vast majority of the potential issues, on (almost) all supported platforms. To do so, add a comment to the PR, invoking the blender-bot
: @blender-bot build
.
Note
Triggering a build that way is only possible for developers who have commit access. For others, reviewers will do it as part of the review process.
Changes Affecting Header Includes¶
Changes that affect usages of header files (especially refactors moving parts of the existing code around, and cleanups removing unused headers) should always be checked with particular care, and cannot rely on a sucessful buildbot build as validation.
The reason is that by default, Blender uses unity builds, which can hide broken header dependencies after such changes.
So when modifying includes, local builds should always be done, following these rules:
- Disable
WITH_UNITY_BUILD
CMake option. - Clean build.
Tip
In addition, if possible, using a very recent version of the clang compiler can also help avoiding additional surprises (most recent compilers tend to be way less forgiving, with indirect includes of the std
headers e.g.).
Automated Testings¶
Using Blender buildbots is a very good way to have a global run of all unittests on all supported platforms for a PR.
However, this should be the last stage of a PR testing! Running them locally should be part of the development process.
Another thing that is not (yet) provided by the buildbots is to run these tests with debug + ASAN builds. For the time being, this remains the responsibility of each developer.
Tip
When running unittests on a debug/ASAN build, it is usually better to skip the Cycles ones, as they can take hours to complete on such builds: ctest -E cycles
.
Manual Testings¶
Manual testing covers everything that cannot be automated. For most minor changes, when the area is properly covered by unittests, this can be very simple: open Blender, use a bit the UI related to the affected code paths, and check that nothing unexpected happens.
However, a large part of the codebase is still lacking when it come to unittests. And some things are very complicated to properly and efficiently test that way. That is why manual testing should remain a mandatory step in any significant change.
Break That Code!
The goal of manual testing is to break the new code, to find cases where it does not behave as expected. It is always harder for a developer to break their own just-written code, so involving other people, ideally artists who will actually use it 'in real life', is always a good idea.
Depending on the type of change ('non-behavioral change' refactors and cleanups, vs. bug fixes and new features), it can involve:
- Opening complex existing production blendfiles and ensuring that they still look and behave the same.
- Craft small and simple demo test cases for the affected code paths (as a bonus, these can be a great first step towards proper unittests).
- When there is compatibility involved, open files from older blender, and save files written by the patched Blender, and open them in current releases (including the LTS ones).
- Get artists to try the new feature, and stress-test it.
Ideally, tests should be done using both release builds, and debug + ASAN ones.
- Debug + ASAN Builds: While not always possible (when working on the Cycles renderer e.g.), as much as possible testing should be done with debug + ASAN builds, as it will catch many things that may go under the radar in release builds.
- Release Builds: The main reason to use a release build is to check on performance improvements or regressions. Another sneaky category of issues that may be easier to spot in release builds are certain multi-threading/concurrency ones (as they execute code way faster).
Tip
In the PR pages of projects.blender.org, the blender-bot
can generate temporary packages, that are easy to share with other testers: @blender-bot package
e.g.