Page MenuHome

Blender: Going Locale
Open, NormalPublic

Description

Issue

Blender is used by artist around the world [1] however, our efforts do not meet the demand for providing localization for the software and documentation.
The current workflow for both Blender itself and the user manaul are a bit convoluted. For both projects on boarding is difficult, there are no clear steps for how a new translator should get commit writes.
The only translators that we get are the ones that are either brave enough to make the changes and submit patches are create some sort of task/email stating they would like to help.
Communication among translators for a specific language are very limited and it is difficult for multiple translators to work together.

Proposal

Weblate

Blender has grown quite actively over the years and our current method of dealing with translations are not enough to keep up with such a large project.
Most large projects offload this work to an online localization platform. The issue with this for the Blender Foundation is that most of these make use of 3rd party websites.
However, I did find one service, called Weblate [2], that is developed on open source principles can support custom installs.

Weblate is an online localization service with tools to manage and translation projects using an online editor.
The benefits of such a platform are, online editing of translations, communication features to individual language groups,
a translation reviewal system and the ability to easily be able to request access to translations without having to know advanced tools like SVN/Git and sphinx.
A full list of features are explained here [3]. Weblate is used by over a thousand projects including, OpenSuse and Godot Game Engine [4].

For us, the way I see this running is to have weblate installed at https://translate.blender.org which would make it super easy and convenient for translators to get involved in translating.
Weblate integrates directly with version control software, both Git and SVN. For the main translations the use of the SVN repository would be eliminated as Weblate would work directly with the main Git repository.
Changes will be made directly through weblate itself and these changes are then be periodically (or automatically via a webhook) pulled into the repository.
In essence the only thing that should be interacting with the Git repository is Weblate itself. This raises the issue of a workflow change for translators as translations should be translated in the web interface.
However, Weblate has a feature where translators can download translation from Weblate and use a gettext editor workflow [5].

For the user manual things would work in a similar way, except for using Git we can stick to using our SVN repository (although switching to Git would be extremely useful).
There is an issue with the user manual where Weblate uses a single translation file for each language while Sphinx uses multiple translation files per language.
While we can still make this work it introduces a middle step to merge files to a format Weblate can use. This is simple by using msgmerge [6] and godot has worked around this problem [7].

All in all, migrating to a more centralized system like this should improve both the quantity and quality of translations by making translation onboarding easier along with providing a simple online based workflow.

Improved Communication

Another issue that we have now is poor communication our current efforts include the old bf-translations-dev mailing list which only sees a few stray emails a year.
There are a couple of things that I would like to do here:

  • Create a Translations category on devtalk with sub language topics
  • Create a Translations channel on blender.chat

Implementation

Timeline

We want to get this system working sometime around 2.82 which is set to release in 3-4 months.
Implementation details of this proposal will happen immediately has we need time to make this changes and iron out the new workflow.

Tasks

Some initial tasks to get done that I can think of off the top of my head to get this working:

  • Migrate repositories to Git (not required but helpful)
    • Archive old repos and restrict commit access
  • Update the user manual locale structure to use single translation files
    • Write a script similar to Godot's to help manage this
  • Get weblate installed on the server
  • Import the repositories setting up projects and components
  • Update existing translation guides on wiki and user manual)

Questions

There are some things we might want to consider related to how we configure weblate / the workflow we choose.

  • Weblate allows new languages to be added on the fly. Do we want to do this or make users request new languages and languages can be added manually.
  • Weblate can automatically commit changes to the repositories. Do we wan to do this or manually merge changes made in weblate into the repositories.

  1. https://www.blender.org/wp-content/uploads/2013/08/Screen-Shot-2017-08-28-at-14.37.14.png
  2. https://weblate.org/
  3. https://weblate.org/en/features/
  4. https://hosted.weblate.org/projects/godot-engine/godot-docs/
  5. https://docs.weblate.org/en/latest/user/files.html
  6. https://www.gnu.org/software/gettext/manual/html_node/msgmerge-Invocation.html
  7. https://github.com/godotengine/godot-docs-l10n

Details

Type
Design

Event Timeline

Aaron Carlisle (Blendify) lowered the priority of this task from Needs Triage by Developer to Normal.Aug 12 2019, 11:33 PM
Aaron Carlisle (Blendify) created this task.

Hi @Aaron Carlisle (Blendify) ,
I'm the translator of the UI into Spanish (one of the few languages that has been consistently maintained throughout these years), I've handed it alone since 2011, and a couple of times people tried to jump onboard to help, but unfortunately they ultimately never did any significant contribution, for different reasons I think.
I understand what you are looking for with these changes, and it could be a good thing, if done properly.

What I wouldn't like is to lose in the process the simplicity of the actual system, I mean, if I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up! and we'll keep on rolling as always. But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another.

I stress, this is not to say that I don't like the idea, and for sure I understand the purpose behind it, but the task of translating Blender itself (for free) is worth it, if I can focus myself just on doing it for the sake of the community and don't having to lose energy on silly and useless political discussions in the process.
I really like team work, but there should be a smart way of implementing it for these tasks.
In this regard, I think some measures should be in place to assure that the people jumping in will have a certain minimum level of expertise to really be able to "sit around that table" and prove to be a positive force in helping to improve what we currently have.
For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed.
...that kind of things, mainly.

I hope this new initiative works great!
Cheers

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual (for now)?

Or another way of asking: Is anything off the table?

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

@Gabriel Gazzán (gab3d) Thank you for the response, part of my goal here is to get feedback from current translators.

If I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up!

One of the benefits of using a localization platform such as this is that we can adopt a glossary [1] for each project language, which should help solve this very issue.

But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another.

Another benefit of such a system is to add a better mechanism for translation review [2]. Currently reviews are very hard as trying to do patch reviews for translations is extremely tedious and time consuming, not to mention that we do not have enough active contributes to see multiple people working together.

For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed.

This sounds like a good idea for something we can improve on for all areas of contribution in Blender. For Blender itself we have the Quick Hacks project and we can incorporate something similar for translations and even the manual itself.
Some of these quick hacks could also just be routine tasks. Other projects use some sort of "Help wanted" tag for projects that can easily be assigned to new contributors.
I think this would be something nice to look into and that @Nathan Letwory (jesterking) in particular could help guide the foundation towards.

  1. https://docs.weblate.org/en/latest/user/translating.html#glossary
  2. https://docs.weblate.org/en/latest/workflows.html#peer-review

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual (for now)?
Or another way of asking: Is anything off the table?

For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have.

I would say somethings should be off the table though, mainly, API docs and Developer Wiki as anyone who is involved with coding should be an English speakers.

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

In a discussion with @zdy (NGENNGT) in #docs on blender.chat one issue with the manual translations is the lack of quality of translations due to translators need to know proper RST formatting to translate po files.
Because moving to such a system that is online, translators will not be building docs locally and there is concern that the amount of formatting errors would increase.

One way to combat this is to write custom check within Weblate for RST syntax errors. https://docs.weblate.org/en/latest/user/checks.html
We could also improve are documentation stating that care needs to be taken to keep the syntax intact. This has caused issues like T68108

zdy (NGENNGT) added a comment.EditedAug 14 2019, 1:46 AM

In my opinion, there are three requirements for new translators:

  1. Acquaintance with one or more parts of Blender

We gather our translators from BlenderCN community to assure this.

  1. A knowledge of both English and target language

Actually, we don't have a way to qualify this(especially English), since a certificate of English is some kind of privacy.

  1. A knowledge of basic rst syntax and svn.

Since most of Blender users are artists, and not familiar with rst, we summarized a step-by-step tutorial for building local manual, and make it a compulsory task for all Chinese translators. Every contributor should build locally and check the result page before submitting translated files to make sure there is no syntax error in them.

The new system should try to lower the barrier of the requirements above.


I'd like to share some of my suggestions:

  1. rst syntax auto-check

As mentioned above, rst sets the barrier itself, we need to building locally to check and find out the errors in translated texts. So rst syntax auto-check ,suggestion and/or correction will help a lot.

Without this feature, this will be merely a online version of poedit, with some feature like glossary and peer review, of course.

But if this is not to be solved, we still can't assure the quality of translations, or it will be still be hard to attract more qualified volunteer.

  1. Fast entry for translation.

Since the po files will be merged into one huge file, it will be neccesarry to add a fast entry in every manual page to redirect to corresponding strings in weblate.

And we can also add a entry of UI translation for every menu item, like the manual entry for every menu in RMB menu.

This will make contribution more covenient.

[..] translation review [..]

I think this would be something nice to look into and that @Nathan Letwory (jesterking) in particular could help guide the foundation towards.

The proposed tool weblate looks like a good way forward to bring some sense into the translation process.

There should indeed be a good way to vet new translators to ensure we can get the best documentation and translations.

What I'd like to see is a better connection between code changes and the documentation and translation work it may or may not cause.

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual (for now)?
Or another way of asking: Is anything off the table?

For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have.

Lets keep it for now with Blender and user manual. If we find that the tool works well in enabling our translators to do efficient work then we can see if we can tie other services into this system.

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

I'd also like to know the reason for this.

Ideally we move localisation files into a Git repository that would become a submodule of the main blender.git repository. Doing any post-processing of files should indeed be done by the build process. Maybe not by default, but there could be a target that is specifically for handling localisation files and preparing releases.

With the localisation on Git we have a much better connection with Blender, and can look into setting up some workflows where commits trigger tasks for translators, and ideally even documentors. That way we get an automatic list of issues that need to be handled by translators prior to release, and can be handled as changes to Blender happen in a much more intuitive way.

For any concerns about binary files in our repositories: we should enabled git-lfs and put binary files there. It has worked great for many years already, very stable.

I suppose next up would be asking @Dan McGrath (dmcgrath) if we could get weblate hosted soonish.

@Aaron Carlisle (Blendify) do you have a good grasp of the configuration of weblate and the integrations it provides?

Perhaps figure out these things as next step in preparation for a move to weblate for translation work.

So the following with in mind we use weblate at some blender.org address (translate.blender.org sounds good to me)

  • user registration - possible to use phabricator user database? Or some other way to keep these in sync?
  • propose review workflow
  • propose onboarding/testing of prospect translators for quality
  • find instructions to set up the git integration
  • propose how to do move to git from svn
  • propose a time schedule. Could we do it for 2.81 already? Or would it be something that we start now, but target to finalize before 2.82?

Anyway, this looks like a useful move to make. Thanks for writing up!

user registration - possible to use phabricator user database? Or some other way to keep these in sync?

This is a topic I want to bring up for discussion separately ideally shouldnt we be using Blender ID for this and phabricator?

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

Because we want the git repo (which has to be checked out by everybody that wants to build Blender) to be as small as possible. And a tremendous amount of changes in PO files, from week to week, are actually in the 'source' comments (line number changes). those info are mandatory for translation work, but useless if you only want to generate the MOs binaries...

Although the current SVN model has some historical heritage that we could probably get rid of now (e.g. I don't think generating MO's and storing them there is needed anymore...). I would still keep the 'translator' side of things outside of the 'blender' side of thing, but I guess the website could then run the validation/cleanup scripts I am running currently every week when I update the repos... And just use two git repos, one for full raw PO's, and one for proper trimmed out, checked and safe PO's?

Because we want the git repo (which has to be checked out by everybody that wants to build Blender) to be as small as possible. And a tremendous amount of changes in PO files, from week to week, are actually in the 'source' comments (line number changes). those info are mandatory for translation work, but useless if you only want to generate the MOs binaries...

Maybe we can improve out git flow for this. For example if we are worried about the the checkout size of the history then we can use a shallow checkout:
https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt

This can be set up to only grab the files for the latest revision. Translators will be interfacing with weblate so the repository is basically only needed to be read only.
A few admins for example can checkout the full repository elsewhere on disk for purposes such as updating files and such.

We can also just handle this normally as I dont think the repository will be that large with only ~5000 commits so far.

@Nathan Letwory (jesterking) should probably orient us in the best option as the git flow.

Although the current SVN model has some historical heritage that we could probably get rid of now (e.g. I don't think generating MO's and storing them there is needed anymore...).

I agree, these should only be built as part of the build process.

I created two patches; one for the blender source and the other for the add-on to remove mo files from the translations repository.

After these changes, assuming I did not miss anything, the locale directory can be removed from svn.

Note that these will require a workflow change which needs to be updated here:

https://wiki.blender.org/wiki/Process/Translate_Blender

Specifically "How to test your translations - translators" and any reference to blender.mo

Somethings I noticed while testing UI translations:

  • es and es_ES are identical, I propose we remove es_ES
  • We currently have several po-files with no translations, I propose that we remove these also:
    • Amharic (am)
    • Estonian (et)
    • Kazakh (kk)
    • Serbian Latin Chars (sr@latin)
    • Uzbek (uz)
    • Uzbek Cryllic Chars (uz@cryllic)

Doing this will just help the migration process. In the future these can be easily re-added when we actually have translators for them.

@Aaron Carlisle (Blendify)

es and es_ES are identical, I propose we remove es_ES

yes, please do it.
nobody has ever been behind it, since I started with es from scratch back in the day (and es_ES was already at 3%). Since then, I've been simply copying es over es_ES in the hopes that somebody could ever pick it up and develop it, but that never really happened.

I’d rather stay in charge of translation project, including which languages we keep or not… If es and es_ES are identical, then indeed the later should have been removed for a long time.

The empty translations please do not touch for now, I don't see how removing them would make transition easier, imho having them in the web UI might actually trigger some will from users to help translating in those languages.

In any case, I'd rather see the whole web-based translation system up and running *before* we do anything to current one (including docs).

@Aaron Carlisle (Blendify) Am fine adding you to Translations, but why do you need it exactly?

The only reason is to help do the changes I described but if you would like to stay on top of things instead that works too.

In any case, I'd rather see the whole web-based translation system up and running *before* we do anything to current one (including docs).

I am currently trying to investigate an issue I was having getting the Chinese languages imported on a local server.
Once I figure that out I can work with Dan on getting a server set up so we can do public tests.
The only issue the currently with the layout of the svn directory makes it impossible to load the translations into the website.
I ended up making my own repository that I have been using for testing and I guess we can use this also for a public test and switch to a real repo when we are ready.