Page MenuHome

Blender: Going Locale
Open, NormalPublic

Description

Issue

Blender is used by artist around the world [1] however, our efforts do not meet the demand for providing localization for the software and documentation.
The current workflow for both Blender itself and the user manaul are a bit convoluted. For both projects on boarding is difficult, there are no clear steps for how a new translator should get commit writes.
The only translators that we get are the ones that are either brave enough to make the changes and submit patches are create some sort of task/email stating they would like to help.
Communication among translators for a specific language are very limited and it is difficult for multiple translators to work together.

Proposal

Weblate

Blender has grown quite actively over the years and our current method of dealing with translations are not enough to keep up with such a large project.
Most large projects offload this work to an online localization platform. The issue with this for the Blender Foundation is that most of these make use of 3rd party websites.
However, I did find one service, called Weblate [2], that is developed on open source principles can support custom installs.

Weblate is an online localization service with tools to manage and translation projects using an online editor.
The benefits of such a platform are, online editing of translations, communication features to individual language groups,
a translation reviewal system and the ability to easily be able to request access to translations without having to know advanced tools like SVN/Git and sphinx.
A full list of features are explained here [3]. Weblate is used by over a thousand projects including, OpenSuse and Godot Game Engine [4].

For us, the way I see this running is to have weblate installed at https://translate.blender.org which would make it super easy and convenient for translators to get involved in translating.
Weblate integrates directly with version control software, both Git and SVN. For the main translations the use of the SVN repository would be eliminated as Weblate would work directly with the main Git repository.
Changes will be made directly through weblate itself and these changes are then be periodically (or automatically via a webhook) pulled into the repository.
In essence the only thing that should be interacting with the Git repository is Weblate itself. This raises the issue of a workflow change for translators as translations should be translated in the web interface.
However, Weblate has a feature where translators can download translation from Weblate and use a gettext editor workflow [5].

For the user manual things would work in a similar way, except for using Git we can stick to using our SVN repository (although switching to Git would be extremely useful).
There is an issue with the user manual where Weblate uses a single translation file for each language while Sphinx uses multiple translation files per language.
While we can still make this work it introduces a middle step to merge files to a format Weblate can use. This is simple by using msgmerge [6] and godot has worked around this problem [7].

All in all, migrating to a more centralized system like this should improve both the quantity and quality of translations by making translation onboarding easier along with providing a simple online based workflow.

Improved Communication

Another issue that we have now is poor communication our current efforts include the old bf-translations-dev mailing list which only sees a few stray emails a year.
There are a couple of things that I would like to do here:

  • Create a Translations category on devtalk with sub language topics
  • Create a Translations channel on blender.chat

Implementation

Timeline

We want to get this system working sometime around 2.82 which is set to release in 3-4 months.
Implementation details of this proposal will happen immediately has we need time to make this changes and iron out the new workflow.

Tasks

Some initial tasks to get done that I can think of off the top of my head to get this working:

  • Migrate repositories to Git (not required but helpful)
    • Archive old repos and restrict commit access
  • Update the user manual locale structure to use single translation files
    • Write a script similar to Godot's to help manage this
  • Get weblate installed on the server
  • Import the repositories setting up projects and components
  • Update existing translation guides on wiki and user manual)

Questions

There are some things we might want to consider related to how we configure weblate / the workflow we choose.

  • Weblate allows new languages to be added on the fly. Do we want to do this or make users request new languages and languages can be added manually.
  • Weblate can automatically commit changes to the repositories. Do we wan to do this or manually merge changes made in weblate into the repositories.

  1. https://www.blender.org/wp-content/uploads/2013/08/Screen-Shot-2017-08-28-at-14.37.14.png
  2. https://weblate.org/
  3. https://weblate.org/en/features/
  4. https://hosted.weblate.org/projects/godot-engine/godot-docs/
  5. https://docs.weblate.org/en/latest/user/files.html
  6. https://www.gnu.org/software/gettext/manual/html_node/msgmerge-Invocation.html
  7. https://github.com/godotengine/godot-docs-l10n

Details

Type
Design

Related Objects

Event Timeline

Aaron Carlisle (Blendify) triaged this task as Normal priority.

Hi @Aaron Carlisle (Blendify) ,
I'm the translator of the UI into Spanish (one of the few languages that has been consistently maintained throughout these years), I've handed it alone since 2011, and a couple of times people tried to jump onboard to help, but unfortunately they ultimately never did any significant contribution, for different reasons I think.
I understand what you are looking for with these changes, and it could be a good thing, if done properly.

What I wouldn't like is to lose in the process the simplicity of the actual system, I mean, if I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up! and we'll keep on rolling as always. But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another.

I stress, this is not to say that I don't like the idea, and for sure I understand the purpose behind it, but the task of translating Blender itself (for free) is worth it, if I can focus myself just on doing it for the sake of the community and don't having to lose energy on silly and useless political discussions in the process.
I really like team work, but there should be a smart way of implementing it for these tasks.
In this regard, I think some measures should be in place to assure that the people jumping in will have a certain minimum level of expertise to really be able to "sit around that table" and prove to be a positive force in helping to improve what we currently have.
For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed.
...that kind of things, mainly.

I hope this new initiative works great!
Cheers

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual (for now)?

Or another way of asking: Is anything off the table?

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

@Gabriel Gazzán (gab3d) Thank you for the response, part of my goal here is to get feedback from current translators.

If I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up!

One of the benefits of using a localization platform such as this is that we can adopt a glossary [1] for each project language, which should help solve this very issue.

But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another.

Another benefit of such a system is to add a better mechanism for translation review [2]. Currently reviews are very hard as trying to do patch reviews for translations is extremely tedious and time consuming, not to mention that we do not have enough active contributes to see multiple people working together.

For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed.

This sounds like a good idea for something we can improve on for all areas of contribution in Blender. For Blender itself we have the Quick Hacks project and we can incorporate something similar for translations and even the manual itself.
Some of these quick hacks could also just be routine tasks. Other projects use some sort of "Help wanted" tag for projects that can easily be assigned to new contributors.
I think this would be something nice to look into and that @Nathan Letwory (jesterking) in particular could help guide the foundation towards.

  1. https://docs.weblate.org/en/latest/user/translating.html#glossary
  2. https://docs.weblate.org/en/latest/workflows.html#peer-review

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual (for now)?
Or another way of asking: Is anything off the table?

For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have.

I would say somethings should be off the table though, mainly, API docs and Developer Wiki as anyone who is involved with coding should be an English speakers.

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

In a discussion with @zdy (NGENNGT) in #docs on blender.chat one issue with the manual translations is the lack of quality of translations due to translators need to know proper RST formatting to translate po files.
Because moving to such a system that is online, translators will not be building docs locally and there is concern that the amount of formatting errors would increase.

One way to combat this is to write custom check within Weblate for RST syntax errors. https://docs.weblate.org/en/latest/user/checks.html
We could also improve are documentation stating that care needs to be taken to keep the syntax intact. This has caused issues like T68108

zdy (NGENNGT) added a comment.EditedWed, Aug 14, 1:46 AM

In my opinion, there are three requirements for new translators:

  1. Acquaintance with one or more parts of Blender

We gather our translators from BlenderCN community to assure this.

  1. A knowledge of both English and target language

Actually, we don't have a way to qualify this(especially English), since a certificate of English is some kind of privacy.

  1. A knowledge of basic rst syntax and svn.

Since most of Blender users are artists, and not familiar with rst, we summarized a step-by-step tutorial for building local manual, and make it a compulsory task for all Chinese translators. Every contributor should build locally and check the result page before submitting translated files to make sure there is no syntax error in them.

The new system should try to lower the barrier of the requirements above.


I'd like to share some of my suggestions:

  1. rst syntax auto-check

As mentioned above, rst sets the barrier itself, we need to building locally to check and find out the errors in translated texts. So rst syntax auto-check ,suggestion and/or correction will help a lot.

Without this feature, this will be merely a online version of poedit, with some feature like glossary and peer review, of course.

But if this is not to be solved, we still can't assure the quality of translations, or it will be still be hard to attract more qualified volunteer.

  1. Fast entry for translation.

Since the po files will be merged into one huge file, it will be neccesarry to add a fast entry in every manual page to redirect to corresponding strings in weblate.

And we can also add a entry of UI translation for every menu item, like the manual entry for every menu in RMB menu.

This will make contribution more covenient.

[..] translation review [..]

I think this would be something nice to look into and that @Nathan Letwory (jesterking) in particular could help guide the foundation towards.

The proposed tool weblate looks like a good way forward to bring some sense into the translation process.

There should indeed be a good way to vet new translators to ensure we can get the best documentation and translations.

What I'd like to see is a better connection between code changes and the documentation and translation work it may or may not cause.

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual (for now)?
Or another way of asking: Is anything off the table?

For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have.

Lets keep it for now with Blender and user manual. If we find that the tool works well in enabling our translators to do efficient work then we can see if we can tie other services into this system.

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

I'd also like to know the reason for this.

Ideally we move localisation files into a Git repository that would become a submodule of the main blender.git repository. Doing any post-processing of files should indeed be done by the build process. Maybe not by default, but there could be a target that is specifically for handling localisation files and preparing releases.

With the localisation on Git we have a much better connection with Blender, and can look into setting up some workflows where commits trigger tasks for translators, and ideally even documentors. That way we get an automatic list of issues that need to be handled by translators prior to release, and can be handled as changes to Blender happen in a much more intuitive way.

For any concerns about binary files in our repositories: we should enabled git-lfs and put binary files there. It has worked great for many years already, very stable.

I suppose next up would be asking @Dan McGrath (dmcgrath) if we could get weblate hosted soonish.

@Aaron Carlisle (Blendify) do you have a good grasp of the configuration of weblate and the integrations it provides?

Perhaps figure out these things as next step in preparation for a move to weblate for translation work.

So the following with in mind we use weblate at some blender.org address (translate.blender.org sounds good to me)

  • user registration - possible to use phabricator user database? Or some other way to keep these in sync?
  • propose review workflow
  • propose onboarding/testing of prospect translators for quality
  • find instructions to set up the git integration
  • propose how to do move to git from svn
  • propose a time schedule. Could we do it for 2.81 already? Or would it be something that we start now, but target to finalize before 2.82?

Anyway, this looks like a useful move to make. Thanks for writing up!

user registration - possible to use phabricator user database? Or some other way to keep these in sync?

This is a topic I want to bring up for discussion separately ideally shouldnt we be using Blender ID for this and phabricator?