Self-hosting external libraries packages #86124

New Issue

Dalai Felinto · 2021-03-01T12:14:35+01:00

Dalai Felinto commented

2021-03-01 12:14:35 +01:00

This is a project/proposal to self-host all (most anyways) libraries Blender links to (png, jpg, ...).

This makes Blender less dependent on external hosting solutions. It also makes it easy to comply to GPL source code request.

Host all packages in the svn
make deps to support that
Create a new make source_archive_complete to include those
make deps from the downloaded full package to also work with the local packages

Note: make source_archive_complete should create the following compressed file structure:

$ tar -Jtvf blender-with-libraries-2.92.0.tar.xz

*blender_2.92.0/*blender_2.92.0/source/blender/blenkernel
(...)
*packages/*packages/tiff-4.1.0.tar.gz
(...)

This is a project/proposal to self-host all (most anyways) libraries Blender links to (png, jpg, ...). This makes Blender less dependent on external hosting solutions. It also makes it easy to comply to GPL source code request. * Host all packages in the svn * `make deps` to support that * Create a new `make source_archive_complete` to include those * `make deps` from the downloaded full package to also work with the local packages --- Note: `make source_archive_complete` should create the following compressed file structure: $ `tar -Jtvf blender-with-libraries-2.92.0.tar.xz` ``` *blender_2.92.0/*blender_2.92.0/source/blender/blenkernel (...) *packages/*packages/tiff-4.1.0.tar.gz (...) ```

Dalai Felinto commented

2021-03-01 12:14:35 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Dalai Felinto commented

2021-03-01 12:14:35 +01:00

Added subscriber: @dfelinto

Ray molenkamp was assigned by Dalai Felinto

2021-03-01 16:32:37 +01:00

Dalai Felinto commented

2021-03-01 16:32:37 +01:00

I talked to Ray, he will take from here.

Dalai Felinto commented

2021-03-01 16:37:34 +01:00

Also, @LazyDodo I prepared some cleanup patches: P2005. They are all yours now ;)

Also, @LazyDodo I prepared some cleanup patches: [P2005](https://archive.blender.org/developer/P2005.txt). They are all yours now ;)

Dalai Felinto commented

2021-03-01 16:52:19 +01:00

Added subscriber: @ponderz

Dalai Felinto commented

2021-03-01 16:52:19 +01:00

James wants to keep an eye on it since he will incorporate make package_archive_full to the auto-deployment rig once this is available.

James wants to keep an eye on it since he will incorporate `make package_archive_full` to the auto-deployment rig once this is available.

Dalai Felinto commented

2021-03-02 10:25:17 +01:00

Just to add some extra suggestions that are related to the issue here.
As far as the external world is concerned, those changes mean Blender will make available:

https://download.blender.org/source/blender-2.92.0.tar.xz (40MB)
https://download.blender.org/source/blender-with-libraries-2.92.0.tar.xz (~2GB) - not yet there

My proposal is that on major updates (2.90.0, 2.91.0) both packages are provided. But on incremental minor updates (2.90.1, 2.91.3) only the small package is provided (unless in the rare case where a critical bug was fixed in a library).

Just to add some extra suggestions that are related to the issue here. As far as the external world is concerned, those changes mean Blender will make available: * https://download.blender.org/source/blender-2.92.0.tar.xz (40MB) * https://download.blender.org/source/blender-with-libraries-2.92.0.tar.xz (~2GB) - not yet there My proposal is that on major updates (2.90.0, 2.91.0) both packages are provided. But on incremental minor updates (2.90.1, 2.91.3) only the small package is provided (unless in the rare case where a critical bug was fixed in a library).

Ray molenkamp commented

2021-03-02 16:41:22 +01:00

Just as a sanity check you are not expecting people to download this tarball, run some magic make command and end up with a fully functional blender i hope? Cause that's not the sort of thing we can support.

This may not be common knowledge, but make deps is made by our platform devs, for our platform devs, it's sole purpose is to populate our SVN repo. its target audience is only our platform devs, we offer no support on it and it's highly sensitive to our build environment. if we want to populate the SVN repo with the tar-balls of the 3rd party libs for compliance purposes that be no problem, but it seems unlikely at this point we can/should expand its intended target audience beyond what it currently is.

Just as a sanity check you are not expecting people to download this tarball, run some magic make command and end up with a fully functional blender i hope? Cause that's not the sort of thing we can support. This may not be common knowledge, but `make deps` is made *by* our platform devs, *for* our platform devs, it's sole purpose is to populate our SVN repo. its target audience is only our platform devs, we offer no support on it and it's highly sensitive to *our* build environment. if we want to populate the SVN repo with the tar-balls of the 3rd party libs for compliance purposes that be no problem, but it seems unlikely at this point we can/should expand its intended target audience beyond what it currently is.

Dalai Felinto commented

2021-03-02 16:52:02 +01:00

I would like make deps to work as well as it does now for the platform maintainers.
I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic.

* I would like `make deps` to work as well as it does now for the platform maintainers. * I would love if `make deps` would work out of the box in my Ubuntu computer, but it is a separate topic.

Ray molenkamp commented

2021-03-02 17:28:23 +01:00

I would like make deps to work as well as it does now for the platform maintainers.

I'm unsure what you mean?

I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic.

I am unlikely to let make deps grow into a second install_deps.sh by trying to target every possible environment on the planet, it's working well for us now due to its limited scope, and doing a full run of lib updates is still a multi week multi person endeavor.

> I would like make deps to work as well as it does now for the platform maintainers. I'm unsure what you mean? >I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic. I am unlikely to let make deps grow into a second `install_deps.sh` by trying to target every possible environment on the planet, it's working well for us now due to its limited scope, and doing a full run of lib updates is still a multi week multi person endeavor.

Dalai Felinto commented

2021-03-02 18:38:36 +01:00

What I meant with my first point is that I don't expect make deps to be easier to use in other platforms just because we are distributing tarballs. But I also don't expect it to be harder to use with the tarballs than make deps is at the moment either.

What I meant with my first point is that I don't expect `make deps` to be easier to use in other platforms just because we are distributing tarballs. But I also don't expect it to be harder to use with the tarballs than `make deps` is at the moment either.

Ray molenkamp commented

2021-03-02 18:48:28 +01:00

allright! i'll get on it!

Ray molenkamp commented

2021-03-03 03:37:30 +01:00

Added subscriber: @dr.sybren

Ray molenkamp commented

2021-03-03 03:37:30 +01:00

Initial diff in D10598, I prefer to keep the review there on the technical aspects only, lets keep any non technical discussion here.

makes Blender less dependent on external hosting solutions.

That's a little bit of a stretch, we rely on external hosting when we build the deps, storing the copy that we downloaded at that point in SVN afterwards changes nothing given we still need to get that initial copy.
The only time this is valid is if we ever need to rebuild the libs for an older blender version and some of the library sources are no longer available, seems like a very niche use-case and if this was the sole buy-in, i would have likely not have gone along with this.

It also makes it easy to comply to GPL source code request.

Now this I buy and support. (tip: lead with the better arguments next time)

Host all packages in the svn

I winged the URI where the packages will be in SVN, may need some tweaking and perhaps some checking with operations (is there an official team there?) the initial set will be around 550 MB also unsure if we want just a giant folder with all sources or split it out by blender version?

make deps to support that

Done, but see note above regarding the URL

Create a new make package_archive_full to include those

i'm not familiar with package_archive unsure what this does, source_archive_full seems to make sense in my head, but no idea what package_archive does either way makefiles are not my forte and @dr.sybren will have to assist here.

make deps from the downloaded full package to also work with the local packages

Done.

Initial diff in [D10598](https://archive.blender.org/developer/D10598), I prefer to keep the review there on the technical aspects only, lets keep any non technical discussion here. >makes Blender less dependent on external hosting solutions. That's a little bit of a stretch, we rely on external hosting when we build the deps, storing the copy that we downloaded at that point in SVN afterwards changes nothing given we still need to get that initial copy. The only time this is valid is if we ever need to rebuild the libs for an older blender version and some of the library sources are no longer available, seems like a very niche use-case and if this was the sole buy-in, i would have likely not have gone along with this. >It also makes it easy to comply to GPL source code request. Now this I buy and support. (tip: lead with the better arguments next time) > Host all packages in the svn I winged the URI where the packages will be in SVN, may need some tweaking and perhaps some checking with operations (is there an official team there?) the initial set will be around 550 MB also unsure if we want just a giant folder with all sources or split it out by blender version? > make deps to support that Done, but see note above regarding the URL >Create a new make package_archive_full to include those i'm not familiar with `package_archive` unsure what this does, `source_archive_full` seems to make sense in my head, but no idea what `package_archive` does either way makefiles are not my forte and @dr.sybren will have to assist here. >make deps from the downloaded full package to also work with the local packages Done.

Dalai Felinto commented

2021-03-03 11:39:52 +01:00

It was indeed source_archive I was referring to. Which is the one we distribute. Sorry about the confusion (updated the task description).
I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

1. It was indeed `source_archive` I was referring to. Which is the one we distribute. Sorry about the confusion (updated the task description). 2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

Ray molenkamp commented

2021-03-03 16:00:01 +01:00

Oh yeah the windows USD libs are bigger than that :) still it be nice to ping them given we're doing something new

Danny McGrath commented

2021-03-03 16:45:33 +01:00

Added subscriber: @dmcgrath

Danny McGrath commented

2021-03-03 16:45:33 +01:00

In #86124#1123228, @dfelinto wrote:
2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

As far as the server storage is concerned, SVN is a rather small jail running on the host, and should have plenty of space for 500MB of files (~1-2TB free iirc).

It's more the backups that are a concern, but 500mb is not a problem. It's only when you get into daily GB's of change that the current system is squeezed tight due to deltas, but we have a new system and HDD's waiting to go into the DC to solve that in the near future.

> In #86124#1123228, @dfelinto wrote: > 2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries. As far as the server storage is concerned, SVN is a rather small jail running on the host, and should have plenty of space for 500MB of files (~1-2TB free iirc). It's more the backups that are a concern, but 500mb is not a problem. It's only when you get into daily GB's of change that the current system is squeezed tight due to deltas, but we have a new system and HDD's waiting to go into the DC to solve that in the near future.

Ray molenkamp commented

2021-03-03 16:54:42 +01:00

For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however....

For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however.... For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

Danny McGrath commented

2021-03-03 17:05:39 +01:00

In #86124#1123401, @LazyDodo wrote:
For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however....

This is fine. It's only when you have 50G a day that get into a daily pool that sticks around for a month that you start to run into issues. Even then, the backup settings can be tweaked, but the downside is that Bacula requires a full run every time you change it.

For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, and you don't exclude it from backups (since it's generated anyway), you end up with 50G x 30 days = ~ 1.5TB, which is almost as big as the entire backup pool, currently.

> In #86124#1123401, @LazyDodo wrote: > For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however.... This is fine. It's only when you have 50G a day that get into a daily pool that sticks around for a month that you start to run into issues. Even then, the backup settings can be tweaked, but the downside is that Bacula requires a full run every time you change it. > For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, and you don't exclude it from backups (since it's generated anyway), you end up with 50G x 30 days = ~ 1.5TB, which is almost as big as the entire backup pool, currently.

Ray molenkamp commented

2021-03-03 17:33:10 +01:00

Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes,

oh my, no, it will be nothing like that

>Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, oh my, no, it will be nothing like that

Dalai Felinto commented

2021-03-04 11:56:32 +01:00

I have 2.83 and 2.92 ready to be uploaded (if you don't mind that I did them mostly automatically but partially manually). [they are indeed ~550MB]
I have all the 2.93 libraries ready to be svn committed if you rather me to do it from here (assuming we don't change their names).
For 2.93 onwards we still need the make source_archive_full.
New libraries (after 2.93) should remove the previous one in svn - so at any given time we only have what is required by master.

* I have 2.83 and 2.92 ready to be uploaded (if you don't mind that I did them mostly automatically but partially manually). [they are indeed ~550MB] * I have all the 2.93 libraries ready to be svn committed if you rather me to do it from here (assuming we don't change their names). * For 2.93 onwards we still need the `make source_archive_full`. * New libraries (after 2.93) should remove the previous one in svn - so at any given time we only have what is required by master.

blender-admin commented

2021-03-16 14:10:30 +01:00

This issue was referenced by d07a7697cf

This issue was referenced by d07a7697cfb064f5129469018082f43a8a86d92f

Sybren A. Stüvel commented

2021-03-16 14:31:22 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Sybren A. Stüvel closed this issue

2021-03-16 14:31:22 +01:00

Dalai Felinto commented

2021-03-17 11:25:46 +01:00

Changed status from 'Resolved' to: 'Confirmed'

Dalai Felinto reopened this issue

2021-03-17 11:25:46 +01:00

Dalai Felinto commented

2021-03-17 11:25:46 +01:00

Leaving this open until D10727 is committed.

Leaving this open until [D10727](https://archive.blender.org/developer/D10727) is committed.

blender-admin commented

2021-04-13 17:47:48 +02:00

This issue was referenced by b8479a70c9

This issue was referenced by b8479a70c936c5c59f199a847d679c80196e6c0b

Brecht Van Lommel commented

2021-05-20 18:48:50 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2021-05-20 18:48:50 +02:00

Can we document this? The appropriate wiki page is here:
https://wiki.blender.org/wiki/Building_Blender/Dependencies

Two things are unclear to me:

The policy is for library upgrades. Does bumping a version in versions.cmake need to go along with putting a new source package in svn?
How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive?

Can we document this? The appropriate wiki page is here: https://wiki.blender.org/wiki/Building_Blender/Dependencies Two things are unclear to me: * The policy is for library upgrades. Does bumping a version in `versions.cmake` need to go along with putting a new source package in svn? * How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive?

Ray molenkamp commented

2021-05-20 20:03:21 +02:00

The policy is for library upgrades. Does bumping a version in versions.cmake need to go along with putting a new source package in svn?

Yes, and removal of the old package

How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive?

As of now it's just for bundling, Imeant to do a small tweak to the build scripts to detect the archives in the source bundle and use them, but given it's just a syntactic sugar for the mythical user that builds blender and all deps from source tarball it was pretty low on my prio list and have not gotten to it for 2.93

This whole change is just for compliance reasons, make deps 's target audience is still just our platforms devs, and as of now i'm not aware of any plans to widen that audience.

> The policy is for library upgrades. Does bumping a version in versions.cmake need to go along with putting a new source package in svn? Yes, and removal of the old package > How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive? As of now it's just for bundling, Imeant to do a small tweak to the build scripts to detect the archives in the source bundle and use them, but given it's just a syntactic sugar for the mythical user that builds blender and all deps from source tarball it was pretty low on my prio list and have not gotten to it for 2.93 This whole change is just for compliance reasons, `make deps` 's target audience is still just our platforms devs, and as of now i'm not aware of any plans to widen that audience.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Self-hosting external libraries packages #86124