Efficient storage of icons in git #37989

New Issue

Campbell Barton · 2013-12-30T11:56:00+01:00

Campbell Barton commented

2013-12-30 11:56:00 +01:00

Git is storing a full copy of icons for every change (almost 1mb increase to the repo size for every icon edit).

Specifically:
./release/datafiles/blender_icons16.png
./release/datafiles/blender_icons32.png
./release/datafiles/prvicons.png

From SVG source:
./release/freedesktop/icons/blender_icon_source.svg
./release/datafiles/prvicons.svg

Ideally we could use RSVG and render the SVG as a build step, but unfortunately RSVG doesn't support mask's in a way thats compatible with inkscape.

http://sourceforge.net/mailarchive/message.php?msg_id=31778009

So for now I would propose to store each icon file be moved into its own directory and split into uncompressed bitmap files (as an automated python script), Then on build the icons can be re-combined into a single PNG.

This way we can make a change to one icon without having to store a new copy of the entire file each time.

Workflow for creating icons

Basically the same as it is now...

Edit the SVG
Run release/datafiles/blender_icons.sh

Rather then committing the png, commit ./release/datafiles/blender_icons(16/32)/ directory which contains pixmaps

Workflow for building

Basically the same as it is now...

... internally there would be a program like datatoc that would take files from ./release/datafiles/blender_icons(16/32)/ and create a PNG in the build path.

Git is storing a full copy of icons for every change *(almost 1mb increase to the repo size for every icon edit).* Specifically: ./release/datafiles/blender_icons16.png ./release/datafiles/blender_icons32.png ./release/datafiles/prvicons.png From SVG source: ./release/freedesktop/icons/blender_icon_source.svg ./release/datafiles/prvicons.svg Ideally we could use RSVG and render the SVG as a build step, but unfortunately RSVG doesn't support mask's in a way thats compatible with inkscape. http://sourceforge.net/mailarchive/message.php?msg_id=31778009 So for now I would propose to store each icon file be moved into its own directory and split into uncompressed bitmap files (as an automated python script), Then on build the icons can be re-combined into a single PNG. This way we can make a change to one icon without having to store a new copy of the entire file each time. Workflow for creating icons **** *Basically the same as it is now...* * Edit the SVG * Run `release/datafiles/blender_icons.sh` *Rather then committing the png, commit `./release/datafiles/blender_icons(16/32)/` directory which contains pixmaps* Workflow for building **** *Basically the same as it is now...* ... internally there would be a program like `datatoc` that would take files from `./release/datafiles/blender_icons(16/32)/` and create a PNG in the build path.

Campbell Barton commented

2013-12-30 11:56:00 +01:00

Changed status to: 'Open'

Campbell Barton self-assigned this 2013-12-30 11:56:00 +01:00

Campbell Barton commented

2013-12-30 11:56:00 +01:00

Added subscribers: @brecht, @Sergey

Thomas Dinges commented

2013-12-30 12:13:32 +01:00

Added subscriber: @ThomasDinges

Thomas Dinges commented

2013-12-30 12:13:32 +01:00

We would also need the 32x ones of course. (./release/datafiles/blender_icons32).

Sergey Sharybin commented

2013-12-30 12:55:28 +01:00

Notes so far:

Apart blender_icons.sh you would also need blender_icons.bat for Windows. And also make it so OSX works fine (it'll use .sh guess). Wouldn't want that some internal optimization change makes someone's life more difficult.
We've got other binary binary files which are much bigger even (fonts for example, splash) which are being modified on the same time basis as icons.

Ok, cambo says .blend are not compressed so this is not so big issue, but why using 0 compression level for PNG doesn't work for us?

Tweaking icons are gonna to become PITA. Imagine you've tweaked few icons in SVG, generated split PNG. Now what?
All the files might be different from what they where originally (different inkscape versions, different platforms or so).

How would you stage files which you indeed changed? Unless files does have meaningful name, that would end up in rather annoying calculation of icon coords in a grid. Afraid we'll have people who'll commit all the files instead and all our attempts to optimize anything would mean nothing.

Based on previous note, would really recommend using rather natural names for icons (which is easy to do based UI_icons.h i would think) rather than cryptic names like 001.png and so. Would help seeing what's going on after icons modification.
Did someone checked other projects? From quick glance linux/gimp/calligra doesn't do anything specific against binary files in their repo. As in, we're developers and tools are to make our life easier, not we're to make tools life easier! :)

Notes so far: * Apart `blender_icons.sh` you would also need `blender_icons.bat` for Windows. And also make it so OSX works fine (it'll use .sh guess). Wouldn't want that some internal optimization change makes someone's life more difficult. * We've got other binary binary files which are much bigger even (fonts for example, splash) which are being modified on the same time basis as icons. Ok, cambo says .blend are not compressed so this is not so big issue, but why using 0 compression level for PNG doesn't work for us? * Tweaking icons are gonna to become PITA. Imagine you've tweaked few icons in SVG, generated split PNG. Now what? All the files might be different from what they where originally (different inkscape versions, different platforms or so). How would you stage files which you indeed changed? Unless files does have meaningful name, that would end up in rather annoying calculation of icon coords in a grid. Afraid we'll have people who'll commit all the files instead and all our attempts to optimize anything would mean nothing. * Based on previous note, would really recommend using rather natural names for icons (which is easy to do based UI_icons.h i would think) rather than cryptic names like 001.png and so. Would help seeing what's going on after icons modification. * Did someone checked other projects? From quick glance linux/gimp/calligra doesn't do anything specific against binary files in their repo. As in, we're developers and tools are to make our life easier, not we're to make tools life easier! :)

Thomas Dinges commented

2013-12-30 13:04:25 +01:00

What is the longterm prognosis, lets say if we keep modifying the source as usual and keep the binary files as is in 1 year, 2 years?.

It shouldn't matter if its 300 or 400MB in total in a year. IMHO. ;)

What is the longterm prognosis, lets say if we keep modifying the source as usual and keep the binary files as is in 1 year, 2 years?. It shouldn't matter if its 300 or 400MB in total in a year. IMHO. ;)

Campbell Barton commented

2014-01-07 16:23:03 +01:00

@ThomasDinges - disagree, if we add 80mb per year because of binary files this is going to need cleaning up later.

I doubt it would be so much, but rather avoid this if files need updating frequently and it can be avoided without too much trouble.

@ThomasDinges - disagree, if we add 80mb per year because of binary files this is going to need cleaning up later. I doubt it would be so much, but rather avoid this if files need updating frequently and it can be avoided without too much trouble.

Campbell Barton commented

2014-01-10 16:11:59 +01:00

@Sergey, you raise valid points...

The other files we have in our repo are not changing that much (or we can't help it - as with splash),

I've submitted a patch which stored icons in git as small uncompressed bitmaps.

You do raise a good point that some small change in rasterization would update all icons. so I'll add a check for color difference so very minor changes dont cause all pixmaps to regenerate.

@Sergey, you raise valid points... The other files we have in our repo are not changing that much (or we can't help it - as with splash), I've submitted a patch which stored icons in git as small uncompressed bitmaps. You do raise a good point that some small change in rasterization would update all icons. so I'll add a check for color difference so very minor changes dont cause all pixmaps to regenerate.

Campbell Barton removed their assignment 2016-08-11 06:38:46 +02:00

Campbell Barton commented

2016-08-11 06:38:46 +02:00

Removed subscriber: @ideasman42