Page MenuHome

Remove install_deps script
Needs RevisionPublic

Authored by Sergey Sharybin (sergey) on Jan 22 2020, 4:04 PM.

Details

Summary

While the initial intention of script was quite simple and sounded easy
to be achieved, it turned out to be quite a burden to maintain: all the
weird and wonderful combinations of system-wide and locally compiled
libraries.

Now we have CMake rules to compile all dependencies at their versions
which we know work reliably.

With the CMake based dependencies builder it is possible:

  • Have all libraries at known versions and known ABI. Which also makes it easier to bump versions when needed (no need to check version available at current platform).
  • Follow VFX reference platform libraries.
  • Deliver fully static builds.

One selling point of install_deps vs. CMake based one was minimized
compile time. Now we have Linux libraries in SVN which are recognized
by Blender's build system (this is how we build Blender here at the
studio on many various systems for few months now).

Think Using precompiled libraries should become official way to setup
development environment. Will lower entry barrier for new developers
who wouldn't need to deal with all sort of possible building/linking
errors.

Diff Detail

Repository
rB Blender
Branch
install_deps_remmoval (branched from master)
Build Status
Buildable 6361
Build 6361: arc lint + arc unit

Event Timeline

In general am all for having less to maintain, this afternoon I've been digging into this script and running into problems.

CMake is fine for getting dependencies, but don't think it handles system packages well (installing base development environment for different Linux distros for e.g.).

Although we could try doing this in CMake.

  • Are there instructions of the steps involved for using cmake instead of install_deps.sh ?
  • Would this mean users would be more likely to download and install nearly all the blender deps, even if their distro has the proper version packaged?

installing base development environment for different linux distros for e.g.)

https://developer.blender.org/diffusion/B/browse/master/build_files/build_environment/cmake/check_software.cmake

There might be something missing, but since it "works for me" and in release builder VM (which is a stock CentOS 7.5 install) I can not tell IF anything is missing.

Are there instructions of the steps involved for using cmake instead of install_deps.sh

https://wiki.blender.org/wiki/Building_Blender/Dependencies#make_deps

Would this mean users would be more likely to download and install nearly all the blender deps

Yes.

even if their distro has the proper version packaged?

That is almost never the case. Knowing that all libraries are at versions specified in versions.cmake avoids a lot of time looking for whether some bug is due to older/way newer library used.

One worry i have about this is, Cmake sometimes still goes rogue and picks up stuff outside our lib folder, here's one example but i have seen it mentioned on irc/chat previously as well

One worry i have about this is, Cmake sometimes still goes rogue and picks up stuff outside our lib folder, here's one example but i have seen it mentioned on irc/chat previously as well

Is this about building Blender or its dependencies?
In any way, doesn't sound like something what could only be fixed by install_deps.sh: either with the same probability CMake will pick library outside of /opt (even if we want it to use /opt), or build process in make deps could become more explicit to specify which libraries are used by dependencies (aka, manually provide FOO_LIBRARIES instead of relying on FindFOO, as an example).

Bastien Montagne (mont29) requested changes to this revision.Jan 22 2020, 4:33 PM

Am fully against that proposal, many libs (including heavy ones like python, boost, clang/llvm, OIIO, OCIO, ffmpeg…) are available in compatible versions on almost all modern distros. I do not want to have to keep and maintain a full other copy of those on my system when I can avoid it. Not to mention that most distros package Blender with their available versions of those libs, fully static builds are not well seen in that area. So ensuring Blender keeps working with some major distros's environment is also a good thing.

Imho both tools (install_deps and cmake-based thingy) have their usages and are complementary. Removing install_deps without providing another way to use most of Blender dependencies from distro packages is not acceptable for me.

This revision now requires changes to proceed.Jan 22 2020, 4:33 PM

building blender, with the deps in svn there should be no need for the user to go out and build these things themselves, they can if they want but i don't think it should be the default way

installing base development environment for different linux distros for e.g.)

https://developer.blender.org/diffusion/B/browse/master/build_files/build_environment/cmake/check_software.cmake
There might be something missing, but since it "works for me" and in release builder VM (which is a stock CentOS 7.5 install) I can not tell IF anything is missing.

Testing this, currently has error: P1226

Are there instructions of the steps involved for using cmake instead of install_deps.sh

https://wiki.blender.org/wiki/Building_Blender/Dependencies#make_deps

Would this mean users would be more likely to download and install nearly all the blender deps

Yes.

even if their distro has the proper version packaged?

That is almost never the case. Knowing that all libraries are at versions specified in versions.cmake avoids a lot of time looking for whether some bug is due to older/way newer library used.

As there are more complicated deps, can see there is some value in this. Nevertheless it's a fairly big change.

Personally, I like to use my Linux distros libraries and don't use install_deps.sh or make deps, occasionally I build individual libs from source when needed, as this wont impact me I don't have a strong opinion on this. The issue is what kind of development steps we expect new developers to take.

Is there a way to select deps? if for example you only want to run make lite, can you bypass some of the heavier libraries?

If it's possible to avoid spending a long time downloading large deps which aren't essential, I find this proposal less objectionable.

many libs (including heavy ones like python, boost, clang/llvm, OIIO, OCIO, ffmpeg…) are available in compatible versions on almost all modern distros

  • Python needs to be same version as Blender expects. Arch is probably already on 3.8, while Debian stable is 3.6 at a best.
  • LLVM needs to be the one (a) compatible with OSL (b) not cause major slowdown with OSL (remember mcjit issue)
  • OIIO we had issues when we had to bump version to avoid bad crashes in Blender-specific usage
  • ffmpeg is a constant source of pain when it comes to maintaining multiple versions

So ensuring Blender keeps working with some major distros's environment is also a good thing.

lets officially enable support for arm architecture as well, to help maintainers to their work.

You can also flip the statement: making everybody testing Blender with its dependencies which will be in the final release will improve quality of Blender.

Imho both tools (install_deps and cmake-based thingy) have their usages and are complementary. Removing install_deps without providing another way to use most of Blender dependencies from distro packages is not acceptable for me.

Nobody stops you from using system-wide libraries and compile the ones you are interested in and which are not in your distro repository.

Is there a way to select deps? if for example you only want to run make lite, can you bypass some of the heavier libraries?

No, there is not.
Not even convinced there should be. As a responsible developer you should be testing changes against full feature set (or, at lease, be able to do so).

For me the question is also, who is the target audience?

  • Users want the simplest way to build with all features working correctly, and that's what you get with the precompiled libraries. Helping non-developers troubleshoot issues with their system packages is not a good use of time.
  • For developers that spend a lot of time on Blender, I think precompiled libraries and make deps are also a better workflow. You can easily build different Blender versions and change libraries without affecting other software on your system (which for me was a constant struggle in the past). And of course test things knowing that libraries match what will be in the release.
  • For more casual users, casual developers or package maintainers, I can see the value in having instructions to easily build with system packages only. But in that case, I don't think we should build anything from source and install_deps.sh could be massively simplified. Perhaps even be replaced by a few lines in the build docs per distribution.

As someone who has been through the "compile blender for the first time" process more recently than anyone commenting already, I'll give my opinion.

As a responsible developer you should be testing changes against full feature set (or, at lease, be able to do so).

Often when one of the methods fails and someone posts about the issue on DevTalk the recommendation is to just turn that feature off.

If having fewer methods to build Blender gives developers more time to fix these problems, that's definitely an improvement. But it should be a goal for both make deps and the precompiled dependencies to work better out of the box on all systems, otherwise this isn't an improvement.

The main downside is more downloading and redundant disk usage, but that seems worth it.

No, there is not.
Not even convinced there should be. As a responsible developer you should be testing changes against full feature set (or, at lease, be able to do so).

There are a few reasons to support this,

If any deps fail for any reason, you might want to bypass them.
Currently make deps is failing for me because of an sqlite issue. I have this is installed so I imagine that using my local version would work fine (of course this should be fixed, just an example of a random error).

Building deps is not just a developer building the latest source code. It's sometimes necessary to build older versions, maybe in a year or two one of the web-sites goes offline and downloading fails, or compilers may fail with old code (rare but it happens).

If someone wants to get make deps working on other platforms, it would be useful to start with a subset, or for changing/adding new deps, it's simpler if I can isolate one to check if the changes are working properly.


What's the preferred method of tracking issues in make deps ?

If any deps fail for any reason, you might want to bypass them.

Or just use libraries which were guaranteed to work on that specific version of Blender (aka, use from svn tag).

Currently make deps is failing for me because of an sqlite issue. I have this is installed so I imagine that using my local version would work fine (of course this should be fixed, just an example of a random error).

You will always run into issues like this. No matter if it's shell script, cmake script or anything else. I am quite sure install_deps.sh will fail for an obscure reason on my machine as well.

Investing time into fixing system which has broader audience or system which is crucial for Blender releases feels way less frustrating. Why to keep spending time on maintaining two similar things instead of joining forces and ensure there is one reliable build system?

Building deps is not just a developer building the latest source code. It's sometimes necessary to build older versions, maybe in a year or two one of the web-sites goes offline and downloading fails, or compilers may fail with old code (rare but it happens).

Exactly. This is not solved by install_deps.sh (unless you use --build-all which is not different from make deps).

But then again comes topic of time investment. Is absolutely not uncommon when i need to go few versions back and get that Blender compiled. Having simple svn sw statement is way more efficient than compiling all dependencies needed for that specific Blender version.

If someone wants to get make deps working on other platforms, it would be useful to start with a subset, or for changing/adding new deps, it's simpler if I can isolate one to check if the changes are working properly.

Other platforms? It works on Windows, macOS and Linux.

Get it working on different Linux distro? Go to build_linux/deps type make. You can also compile individual packages like make external_python (which will also compile all its dependencies).

What's the preferred method of tracking issues in make deps ?

Compile single package, single threaded. See above.

For more casual users, casual developers or package maintainers, I can see the value in having instructions to easily build with system packages only. But in that case, I don't think we should build anything from source and install_deps.sh could be massively simplified. Perhaps even be replaced by a few lines in the build docs per distribution.

I don't think platform maintainers will use any script install_deps.sh-alike. They'll rely on dependencies provided by packaging system. For example on debian it would be mk-build-deps --install debian/control. This is if needed to be compiled locally. Builder system will do it "automagically". Most of the time you can also simply do sudo apt-get build-dep blender.

I don't think platform maintainers will use any script install_deps.sh-alike. They'll rely on dependencies provided by packaging system. For example on debian it would be mk-build-deps --install debian/control. This is if needed to be compiled locally. Builder system will do it "automagically". Most of the time you can also simply do sudo apt-get build-dep blender.

That's quite convenient. We can document the command to install blender package dependencies for each distribution, and then they should be picked up automatically.

To me that seems the most useful way to test Blender builds correctly with distribution packages, once we build libraries from source we're not really testing that anyway? Beyond that, the main downside I think is disk space, but I don't see that as a big problem.

I switched from using install_deps.sh to make deps not too long ago. Last week I upgraded my Ubuntu 19.04 → 19.10, and things just kept working. I never had that when using a mixture of Ubuntu packages and builds in /opt/lib.

I agree with @Bastien Montagne (mont29) that saving space is a good idea. However, given the advantages that we would have with maintaining one less way to build Blender, for me the space requirements aren't that big of an issue. I mean, my build_linux directory is already 2.7 GiB and my build_debug directory is 18 GiB. Having <1 GiB in my lib/linux_x86_64 dir is peanuts compared to that.