Using Ninja's pool feature - building with lower memory usage. #64087

New Issue

Bastien Montagne · 2019-05-02T16:14:41+02:00

Bastien Montagne commented

2019-05-02 16:14:41 +02:00

Building with e.g. 8 parallel tasks a full-featured debug Blender with ASAN and GTests enabled can use over 10GB of RAM at some points (both thanks to some C++ files - over 2.5GB for each cpp process in some cases - and linking - over 3.3GB to link blender and some of its gtests).

Ninja's pools allow to limit the amount of parallel processes, selectively for certain categories of jobs. CMake supports 'natively' settings for 'build' and 'link' jobs, and more refined ones can be used too as needed.

In that patch, I added an extra 'build_heavy' pool, used for the few C++ libs that require a huge amount of RAM (some from Cycles, libmv and openvdb).

With it, and setting NINJA_PARALLEL_COMPILE_HEAVY_JOBS to 1, NINJA_PARALLEL_COMPILE_JOBS to 7, and NINJA_PARALLEL_LINK_JOBS to 1, for me it compiles withing a 4GB memory footprint, with a time penalty of about 30% (from 15 mins to 20 mins).

Setting NINJA_PARALLEL_COMPILE_HEAVY_JOBS to 2, NINJA_PARALLEL_COMPILE_JOBS to 7, and NINJA_PARALLEL_LINK_JOBS to 1, it compiles within a 7GB memory footprint with no measurable time penalty.

To summarize, it allows a RAM usage reduction of 60% for a build time increase of 30% (mostly due to some libraries that are not often rebuilt, and Cycles), or a RAM usage reduction of 30% for no build time penalty.

Building with e.g. 8 parallel tasks a full-featured debug Blender with ASAN and GTests enabled can use over 10GB of RAM at some points (both thanks to some C++ files - over 2.5GB for each cpp process in some cases - and linking - over 3.3GB to link blender and some of its gtests). Ninja's pools allow to limit the amount of parallel processes, selectively for certain categories of jobs. CMake supports 'natively' settings for 'build' and 'link' jobs, and more refined ones can be used too as needed. In that patch, I added an extra 'build_heavy' pool, used for the few C++ libs that require a huge amount of RAM (some from Cycles, libmv and openvdb). With it, and setting `NINJA_PARALLEL_COMPILE_HEAVY_JOBS` to 1, `NINJA_PARALLEL_COMPILE_JOBS` to 7, and `NINJA_PARALLEL_LINK_JOBS` to 1, for me it compiles withing a 4GB memory footprint, with a time penalty of about 30% (from 15 mins to 20 mins). Setting `NINJA_PARALLEL_COMPILE_HEAVY_JOBS` to 2, `NINJA_PARALLEL_COMPILE_JOBS` to 7, and `NINJA_PARALLEL_LINK_JOBS` to 1, it compiles within a 7GB memory footprint with no measurable time penalty. To summarize, it allows a RAM usage reduction of 60% for a build time increase of 30% (mostly due to some libraries that are not often rebuilt, and Cycles), or a RAM usage reduction of 30% for no build time penalty.

Bastien Montagne self-assigned this 2019-05-02 16:14:41 +02:00

Bastien Montagne commented

2019-05-02 16:14:41 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2019-05-02 16:15:15 +02:00

Changed status from 'Open' to: 'Archived'

Bastien Montagne closed this issue

2019-05-02 16:15:15 +02:00

Bastien Montagne commented

2019-05-02 16:15:15 +02:00

Grrr, that was supposed to be a differential… :(

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Using Ninja's pool feature - building with lower memory usage. #64087