Also, @Ray molenkamp (LazyDodo) I prepared some cleanup patches: P2005. They are all yours now ;)

James wants to keep an eye on it since he will incorporate make package_archive_full to the auto-deployment rig once this is available.

Just to add some extra suggestions that are related to the issue here.
As far as the external world is concerned, those changes mean Blender will make available:

• https://download.blender.org/source/blender-2.92.0.tar.xz (40MB)
• https://download.blender.org/source/blender-with-libraries-2.92.0.tar.xz (~2GB) - not yet there

My proposal is that on major updates (2.90.0, 2.91.0) both packages are provided. But on incremental minor updates (2.90.1, 2.91.3) only the small package is provided (unless in the rare case where a critical bug was fixed in a library).

Just as a sanity check you are not expecting people to download this tarball, run some magic make command and end up with a fully functional blender i hope? Cause that's not the sort of thing we can support.

This may not be common knowledge, but make deps is made by our platform devs, for our platform devs, it's sole purpose is to populate our SVN repo. its target audience is only our platform devs, we offer no support on it and it's highly sensitive to our build environment. if we want to populate the SVN repo with the tar-balls of the 3rd party libs for compliance purposes that be no problem, but it seems unlikely at this point we can/should expand its intended target audience beyond what it currently is.

• I would like make deps to work as well as it does now for the platform maintainers.
• I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic.

I would like make deps to work as well as it does now for the platform maintainers.

I'm unsure what you mean?

I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic.

I am unlikely to let make deps grow into a second install_deps.sh by trying to target every possible environment on the planet, it's working well for us now due to its limited scope, and doing a full run of lib updates is still a multi week multi person endeavor.

What I meant with my first point is that I don't expect make deps to be easier to use in other platforms just because we are distributing tarballs. But I also don't expect it to be harder to use with the tarballs than make deps is at the moment either.

allright! i'll get on it!

Initial diff in D10598, I prefer to keep the review there on the technical aspects only, lets keep any non technical discussion here.

makes Blender less dependent on external hosting solutions.

That's a little bit of a stretch, we rely on external hosting when we build the deps, storing the copy that we downloaded at that point in SVN afterwards changes nothing given we still need to get that initial copy.
The only time this is valid is if we ever need to rebuild the libs for an older blender version and some of the library sources are no longer available, seems like a very niche use-case and if this was the sole buy-in, i would have likely not have gone along with this.

It also makes it easy to comply to GPL source code request.

Now this I buy and support. (tip: lead with the better arguments next time)

Host all packages in the svn

I winged the URI where the packages will be in SVN, may need some tweaking and perhaps some checking with operations (is there an official team there?) the initial set will be around 550 MB also unsure if we want just a giant folder with all sources or split it out by blender version?

make deps to support that

Done, but see note above regarding the URL

Create a new make package_archive_full to include those

i'm not familiar with package_archive unsure what this does, source_archive_full seems to make sense in my head, but no idea what package_archive does either way makefiles are not my forte and @Sybren A. Stüvel (sybren) will have to assist here.

make deps from the downloaded full package to also work with the local packages

Done.

1. It was indeed source_archive I was referring to. Which is the one we distribute. Sorry about the confusion (updated the task description).

2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

Oh yeah the windows USD libs are bigger than that :) still it be nice to ping them given we're doing something new

In T86124#1123228, @Dalai Felinto (dfelinto) wrote:

2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

As far as the server storage is concerned, SVN is a rather small jail running on the host, and should have plenty of space for 500MB of files (~1-2TB free iirc).

It's more the backups that are a concern, but 500mb is not a problem. It's only when you get into daily GB's of change that the current system is squeezed tight due to deltas, but we have a new system and HDD's waiting to go into the DC to solve that in the near future.

For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however....

For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

In T86124#1123401, @Ray molenkamp (LazyDodo) wrote:

For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however....

This is fine. It's only when you have 50G a day that get into a daily pool that sticks around for a month that you start to run into issues. Even then, the backup settings can be tweaked, but the downside is that Bacula requires a full run every time you change it.

For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, and you don't exclude it from backups (since it's generated anyway), you end up with 50G x 30 days = ~ 1.5TB, which is almost as big as the entire backup pool, currently.

Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes,

oh my, no, it will be nothing like that

• I have 2.83 and 2.92 ready to be uploaded (if you don't mind that I did them mostly automatically but partially manually). [they are indeed ~550MB]
• I have all the 2.93 libraries ready to be svn committed if you rather me to do it from here (assuming we don't change their names).
• For 2.93 onwards we still need the make source_archive_full.
• New libraries (after 2.93) should remove the previous one in svn - so at any given time we only have what is required by master.