Page MenuHome

Investigate adding Git LFS support
Confirmed, NormalPublicTO DO

Description

This task is for investigating what needs to be done to support Git LFS.

  • Needed changes to phabricator configuration
  • Needed changes to git server

Projects to host with this support include

  • Blender Manual (T73394)
  • pre-compiled libraries (not main focus for now)

Relevant links

Event Timeline

Nathan Letwory (jesterking) changed the task status from Needs Triage to Confirmed.Thu, Jan 30, 8:37 AM
Nathan Letwory (jesterking) created this task.

Is this being investigated as a replacement for svn pre-compiled libs, to use in blenders main git repository?

That is one case yes, as noted in the description. While we can say "it works" with SVN there is a distinct disconnect between Git and SVN that adds unnecessary complications to maintaining the structure and having to ensure they stay properly in sync. Not to mention making forensics harder because of the disconnect.

To move pre-compiled libraries I'd propose we don't transfer the entire SVN history, since the history of it isn't recorded in our Git history. Instead we would start with a fresh repository into which the current status is seeded. From the moment we'd move we'd have the potential of recording history. It'll still be the old SVN for pre-lfs in that case.

Tasks like these should be listed on the module page, I've added it there now. I'm not sure which priority this has.

The main benefit I see is about new developers not having to learn about svn. Particularly for the manual and tests repositories, lowering the threshold for contributing to Blender. For me personally as a core developer, I can't think of a tangible benefit, and if history is lost it's going to cost time.

(meta comment, it would be good if tasks like these are presented in terms of problem/solution, it's hard to know what to make of a proposals that simply state a change without any context).


As far as I know having in a separate git repository is necessary, a while back I tried to check-out opentoonz *without* the binary libraries (which were only for Windows/OSX anyway),

It ended up being quite a bit of hassle, requiring making some manual changes to my git-lfs config: see: https://stackoverflow.com/q/42019529/432509

Also, I'm not sure there is a way to checkout only some directories of git-lfs, which is something we'd probably want to be able to do per-platform.


Other question, what are you proposing for the manual - that it moves to git and uses git-lfs for images?

The main benefit I see is about new developers not having to learn about svn.

afaik git-lfs is not available in mainline git, rather than educating them about svn, we now need to educate them about git-lfs

Before making a judgement if git-lfs is worth it compared to svn I'll still need to see if there's any benefits for using it, but my main issue with SVN is latency in the SVN protocol, when i upload lib files a couple of hundred megs each i'll easily reach 1.8-2 megs / second, however boost with it's 20.000 small headers slows down to a crawl

When i uploaded the Win64_vc15 libs recently i took a screenshot when it was done just because it was ridiculous how long it took

Yes that is over 14 hours....

I'd like to do a test run before we decide if it is a good match for us or not.

Important metrics : how long will it take the clients to get the libs as well as how long does it take to commit new libs

another rather important one is, can we control what is being downloaded? right now people will grab the libs they need and that is it (no need to download 4Gb of windows libs for a linux user) i'd like it to stay that way.

Other question, what are you proposing for the manual - that it moves to git and uses git-lfs for images?

This task was inspired by this which I proposed in T73394. I believe this is the biggest uses case of Git + Git LTF, for Blender core it makes less sense as a developer because they only need to check out once and updates are handled via make update.

Before making a judgement if git-lfs is worth it compared to svn I'll still need to see if there's any benefits for using it, but my main issue with SVN is latency in the SVN protocol, when i upload lib files a couple of hundred megs each i'll easily reach 1.8-2 megs / second, however boost with it's 20.000 small headers slows down to a crawl

This is a very compelling reason to switch IMHO.

another rather important one is, can we control what is being downloaded? right now people will grab the libs they need and that is it (no need to download 4Gb of windows libs for a linux user) i'd like it to stay that way.

I think we would need separate repositories per platform

Tasks like these should be listed on the module page, I've added it there now. I'm not sure which priority this has.
The main benefit I see is about new developers not having to learn about svn. Particularly for the manual and tests repositories, lowering the threshold for contributing to Blender. For me personally as a core developer, I can't think of a tangible benefit, and if history is lost it's going to cost time.

The main benefit is not not having to learn separate tools. Having Git _and_ SVN is a minor issue. The disconnect between Git and SVN is the bigger problem wrt the precompiled libraries and blender.git. Yes, there are scripts we use, but that is a pretty fragile solution.

The history won't be lost, the SVN repository would still be there read-only. For archeology that involves pre-LFS it essentially would still be working still as it currently does.

But indeed, T73394 is the initial reason here. And for that purpose we can first concentrate on having Git LFS support added for that, and leave the pre-compiled libraries out of the scope for now. But once we do have Git LFS support I don't see a reason to not revisit this for pre-compiled libraries.