Startup & Core Scripts Code Style #86943

New Issue

Campbell Barton · 2021-03-25T14:55:57+01:00

Campbell Barton commented

2021-03-25 14:55:57 +01:00

Motivation

Currently there are some conventions informally agreed on that have not been documented.

This aims to document the current conventions.

Proposal

https://wiki.blender.org/wiki/Style_Guide/Python should include a section for "Startup & Core Scripts"

Blender comes with scripts which are at the core of Blenders functionality:

release/scripts/startup/, as the name suggests these run on startup and maintained by developers who work across many areas of Blender.
release/scripts/modules/bpy/ the bpy module, the primary way Blender is exposed to Python.

For this reason, scripts that load on startup are intended to be kept fast and simple where possible.

There are some conventions not enforced in other areas of Blender's Python code-base.

Delay imports.

Importing modules not loaded as startup have been moved inside methods to delay loading additional files at startup.

Percentage style string formatting (as opposed to the format method or f-strings).

This creates strings more appropriate for interface translators.

No type hinting.

While this may be evaluated in the future, most UI scripts are simple and don't gain so much from type hints.

Single quotes for RNA rnums such as icon='WARNING' or ob.type == MESH double quotes for text descriptions (all other strings), such as layout.label(text="Text description")

### Motivation Currently there are some conventions informally agreed on that have not been documented. This aims to document the current conventions. ----- ### Proposal https://wiki.blender.org/wiki/Style_Guide/Python should include a section for "Startup & Core Scripts" ---- Blender comes with scripts which are at the core of Blenders functionality: - `release/scripts/startup/`, as the name suggests these run on startup and maintained by developers who work across many areas of Blender. - `release/scripts/modules/bpy/` the bpy module, the primary way Blender is exposed to Python. For this reason, scripts that load on startup are intended to be kept fast and simple where possible. There are some conventions not enforced in other areas of Blender's Python code-base. - Delay imports. ``` Importing modules not loaded as startup have been moved inside methods to delay loading additional files at startup. ``` - Percentage style string formatting (as opposed to the format method or f-strings). ``` This creates strings more appropriate for interface translators. ``` - No type hinting. ``` While this may be evaluated in the future, most UI scripts are simple and don't gain so much from type hints. ``` - Single quotes for RNA rnums such as `icon='WARNING'` or `ob.type == MESH` double quotes for text descriptions (all other strings), such as `layout.label(text="Text description")`

Campbell Barton commented

2021-03-25 14:55:57 +01:00

Added subscriber: @ideasman42

Sybren A. Stüvel commented

2021-03-25 15:19:21 +01:00

Added subscriber: @dr.sybren

Sybren A. Stüvel commented

2021-03-25 15:19:21 +01:00

Delay imports.

I think the practical effect of this is inversely proportional to the commonality of the module. For example, delaying an import of os or addon_utils will hardly have any effect, because the os module is very likely already available in sys.modules['os']. A delayed import only avoids a single dict lookup and name assignment (import os is then equivalent to os = sys.modules['os']). For other, less often used modules, such as webbrowser or rna_xml, I certainly agree with the late import guideline.

Percentage style string formatting (as opposed to the format method or f-strings). This creates strings more appropriate for interface translators.

👍

No type hinting. While this may be evaluated in the future, most UI scripts are simple and don't gain so much from type hints.

I agree, but only to the extent where it's indeed about simple UI code. "startup" does not equate "UI", as operators and other functionality are defined in release/scripts/startup/bl_operators.

For example, release/scripts/startup/bl_operators/object_randomize_transform.py has a function randomize_selected() with rather confusing parameters. Here delta is a boolean, loc, rot, and scale are floats, and scale_even is a boolean again. I found in another part of the code that these loc/rot/scale floats can also be None. Here I feel that type annotations would certainly help readability and understandability. A similar argument could be made for the code in release/scripts/startup/bl_operators/bmesh/find_adjacent.py. So here I disagree with the "no type annotations" rule, because there is a whole lot to gain in these situations.

Single quotes for RNA enums

👍 Not my personal preference, but I can certainly live with this guideline.

> Delay imports. I think the practical effect of this is inversely proportional to the commonality of the module. For example, delaying an import of `os` or `addon_utils` will hardly have any effect, because the `os` module is very likely already available in `sys.modules['os']`. A delayed import only avoids a single dict lookup and name assignment (`import os` is then equivalent to `os = sys.modules['os']`). For other, less often used modules, such as `webbrowser` or `rna_xml`, I certainly agree with the late import guideline. > Percentage style string formatting (as opposed to the format method or f-strings). This creates strings more appropriate for interface translators. :+1: > No type hinting. While this may be evaluated in the future, most UI scripts are simple and don't gain so much from type hints. I agree, but only to the extent where it's indeed about simple UI code. "`startup`" does not equate "UI", as operators and other functionality are defined in `release/scripts/startup/bl_operators`. For example, `release/scripts/startup/bl_operators/object_randomize_transform.py` has a function `randomize_selected()` with rather confusing parameters. Here `delta` is a boolean, `loc`, `rot`, and `scale` are floats, and `scale_even` is a boolean again. I found in another part of the code that these loc/rot/scale floats can also be `None`. Here I feel that type annotations would certainly help readability and understandability. A similar argument could be made for the code in `release/scripts/startup/bl_operators/bmesh/find_adjacent.py`. So here I disagree with the "no type annotations" rule, because there is a whole lot to gain in these situations. > Single quotes for RNA enums :+1: Not my personal preference, but I can certainly live with this guideline.

Campbell Barton commented

2021-03-26 01:56:27 +01:00

In #86943#1136287, @dr.sybren wrote:

Delay imports.

I think the practical effect of this is inversely proportional to the commonality of the module. For example, delaying an import of os or addon_utils will hardly have any effect, because the os module is very likely already available in sys.modules['os']. A delayed import only avoids a single dict lookup and name assignment (import os is then equivalent to os = sys.modules['os']). For other, less often used modules, such as webbrowser or rna_xml, I certainly agree with the late import guideline.

Right, the aim is to avoid loading the module entirely as the dict lookup is negligible.

We could list some exceptions, however I'm wary of complicating rules. If anyone who writes an import needs to consult a table of modules in the wiki (that may change over time) - it risks being impractical to follow.

A simple rule such as "delay imports where possible" is easy to remember, also - for UI code we rarely need modules outside the package anyway, so having a few local imports is hardly a problem, even if a little awkward.

For operators the case to delay imports isn't so straightforward, even so. I don't think delaying imports is really such a hassle.

Percentage style string formatting (as opposed to the format method or f-strings). This creates strings more appropriate for interface translators.

👍

No type hinting. While this may be evaluated in the future, most UI scripts are simple and don't gain so much from type hints.

I agree, but only to the extent where it's indeed about simple UI code. "startup" does not equate "UI", as operators and other functionality are defined in release/scripts/startup/bl_operators.

For example, release/scripts/startup/bl_operators/object_randomize_transform.py has a function randomize_selected() with rather confusing parameters. Here delta is a boolean, loc, rot, and scale are floats, and scale_even is a boolean again. I found in another part of the code that these loc/rot/scale floats can also be None. Here I feel that type annotations would certainly help readability and understandability. A similar argument could be made for the code in release/scripts/startup/bl_operators/bmesh/find_adjacent.py. So here I disagree with the "no type annotations" rule, because there is a whole lot to gain in these situations.

I'm not against type hints in general, I'd rather their inclusion be part of a more formal proposal, so we know what we're getting into and agree on the direction.

Identify places they should be used.
Consider style (when to define our own types, do we want a shared module to import them from - for e.g.).
Ensure the types can be validated.
How comprehensive do we want to be? (is the aim to achieve mypy --strict for e.g.).
Investigate how IDE's/editors can take advantage of this.
Investigate type access from C defined API's.

This doesn't have to be done all at once, if we agree on a plan it can be broken up into steps.

Single quotes for RNA enums

👍 Not my personal preference, but I can certainly live with this guideline.

We could move away from this. Again, I'd rather this be proposed separately from this task. It was already proposed to change during the code-quest and we agreed to keep it this way.

> In #86943#1136287, @dr.sybren wrote: >> Delay imports. > I think the practical effect of this is inversely proportional to the commonality of the module. For example, delaying an import of `os` or `addon_utils` will hardly have any effect, because the `os` module is very likely already available in `sys.modules['os']`. A delayed import only avoids a single dict lookup and name assignment (`import os` is then equivalent to `os = sys.modules['os']`). For other, less often used modules, such as `webbrowser` or `rna_xml`, I certainly agree with the late import guideline. Right, the aim is to avoid loading the module entirely as the dict lookup is negligible. We could list some exceptions, however I'm wary of complicating rules. If anyone who writes an import needs to consult a table of modules in the wiki (that may change over time) - it risks being impractical to follow. A simple rule such as "delay imports where possible" is easy to remember, also - for UI code we rarely need modules outside the package anyway, so having a few local imports is hardly a problem, even if a little awkward. For operators the case to delay imports isn't so straightforward, even so. I don't think delaying imports is really such a hassle. >> Percentage style string formatting (as opposed to the format method or f-strings). This creates strings more appropriate for interface translators. > :+1: > >> No type hinting. While this may be evaluated in the future, most UI scripts are simple and don't gain so much from type hints. > I agree, but only to the extent where it's indeed about simple UI code. "`startup`" does not equate "UI", as operators and other functionality are defined in `release/scripts/startup/bl_operators`. > > For example, `release/scripts/startup/bl_operators/object_randomize_transform.py` has a function `randomize_selected()` with rather confusing parameters. Here `delta` is a boolean, `loc`, `rot`, and `scale` are floats, and `scale_even` is a boolean again. I found in another part of the code that these loc/rot/scale floats can also be `None`. Here I feel that type annotations would certainly help readability and understandability. A similar argument could be made for the code in `release/scripts/startup/bl_operators/bmesh/find_adjacent.py`. So here I disagree with the "no type annotations" rule, because there is a whole lot to gain in these situations. I'm not against type hints in general, I'd rather their inclusion be part of a more formal proposal, so we know what we're getting into and agree on the direction. - Identify places they should be used. - Consider style (when to define our own types, do we want a shared module to import them from - for e.g.). - Ensure the types can be validated. - How comprehensive do we want to be? (is the aim to achieve `mypy --strict` for e.g.). - Investigate how IDE's/editors can take advantage of this. - Investigate type access from C defined API's. This doesn't have to be done all at once, if we agree on a plan it can be broken up into steps. >> Single quotes for RNA enums > :+1: Not my personal preference, but I can certainly live with this guideline. We could move away from this. Again, I'd rather this be proposed separately from this task. It was already proposed to change during the code-quest and we agreed to keep it this way.

Campbell Barton changed title from ~~Startup Scripts Code Style~~ to Startup/Core Scripts Code Style

2021-04-09 08:02:14 +02:00

Campbell Barton self-assigned this 2021-04-09 08:02:14 +02:00

Campbell Barton changed title from ~~Startup/Core Scripts Code Style~~ to Startup & Core Scripts Code Style

2021-04-09 08:05:42 +02:00

Campbell Barton commented

2021-04-09 08:06:22 +02:00

Extended the definition of startup scripts to include the bpy module it's self.

Sybren A. Stüvel commented

2021-04-09 10:27:30 +02:00

In #86943#1136542, @ideasman42 wrote:
We could list some exceptions, however I'm wary of complicating rules. If anyone who writes an import needs to consult a table of modules in the wiki (that may change over time) - it risks being impractical to follow.

I think it's fine to have such a list. People can still delay-import if they want, but chances are most imports aren't going to be that unique and thus without knowing they could already be adhering to that list. The list isn't that hard to make either, we can just look at sys.modules after a factory startup. This is what's loaded anyway, with some Blender-specific modules shortened for brevity. As far as I'm concerned, there is no use in delayed import any of these, especially when they come from Python's stdlib.

abc
addon_utils
atexit
bl_ui (and submodules)
bpy (and submodules)
bpy_extras (and submodules)
bpy_restrict_state
bpy_types
builtins
code
codecs
codeop
collections
collections.abc
console_python
contextlib
copyreg
encodings
encodings.aliases
encodings.latin_1
encodings.utf_8
enum
functools
genericpath
heapq
idprop
idprop.types
importlib
importlib._bootstrap
importlib._bootstrap_external
importlib.abc
importlib.machinery
importlib.util
io
io_anim_bvh
io_curve_svg
io_mesh_ply
io_mesh_stl
io_mesh_uv_layout
io_scene_fbx
io_scene_gltf2
io_scene_obj
io_scene_x3d
itertools
keyingsets_builtins
keyingsets_utils
keyword
linecache
marshal
math
mathutils (and submodules)
nodeitems_builtins
nodeitems_utils
operator
os
os.path
posix
posixpath
re
reprlib
rna_prop_ui
site
sre_compile
sre_constants
sre_parse
stat
sys
time
token
tokenize
traceback
types
typing
typing.io
typing.re
warnings
zipimport

A simple rule such as "delay imports where possible" is easy to remember, also - for UI code we rarely need modules outside the package anyway, so having a few local imports is hardly a problem, even if a little awkward.

For operators the case to delay imports isn't so straightforward, even so. I don't think delaying imports is really such a hassle.

The only downside, and it's a minor one, is that it potentially defers the detectability of errors (I've seen this in practice myself in other Python projects) from loading a module to executing the exact code path that imports the dependency. This means that the "load all addons" unittest is going to be slighly less effective. As I said, this is minor, and IMO doesn't outweigh faster startup.

I'm not against type hints in general, I'd rather their inclusion be part of a more formal proposal, so we know what we're getting into and agree on the direction.

👍

> In #86943#1136542, @ideasman42 wrote: > We could list some exceptions, however I'm wary of complicating rules. If anyone who writes an import needs to consult a table of modules in the wiki (that may change over time) - it risks being impractical to follow. I think it's fine to have such a list. People can still delay-import if they want, but chances are most imports aren't going to be that unique and thus without knowing they could already be adhering to that list. The list isn't that hard to make either, we can just look at `sys.modules` after a factory startup. This is what's loaded anyway, with some Blender-specific modules shortened for brevity. As far as I'm concerned, there is no use in delayed import any of these, especially when they come from Python's stdlib. ``` abc addon_utils atexit bl_ui (and submodules) bpy (and submodules) bpy_extras (and submodules) bpy_restrict_state bpy_types builtins code codecs codeop collections collections.abc console_python contextlib copyreg encodings encodings.aliases encodings.latin_1 encodings.utf_8 enum functools genericpath heapq idprop idprop.types importlib importlib._bootstrap importlib._bootstrap_external importlib.abc importlib.machinery importlib.util io io_anim_bvh io_curve_svg io_mesh_ply io_mesh_stl io_mesh_uv_layout io_scene_fbx io_scene_gltf2 io_scene_obj io_scene_x3d itertools keyingsets_builtins keyingsets_utils keyword linecache marshal math mathutils (and submodules) nodeitems_builtins nodeitems_utils operator os os.path posix posixpath re reprlib rna_prop_ui site sre_compile sre_constants sre_parse stat sys time token tokenize traceback types typing typing.io typing.re warnings zipimport ``` > A simple rule such as "delay imports where possible" is easy to remember, also - for UI code we rarely need modules outside the package anyway, so having a few local imports is hardly a problem, even if a little awkward. > > For operators the case to delay imports isn't so straightforward, even so. I don't think delaying imports is really such a hassle. The only downside, and it's a minor one, is that it potentially defers the detectability of errors (I've seen this in practice myself in other Python projects) from loading a module to executing the exact code path that imports the dependency. This means that the "load all addons" unittest is going to be slighly less effective. As I said, this is minor, and IMO doesn't outweigh faster startup. > I'm not against type hints in general, I'd rather their inclusion be part of a more formal proposal, so we know what we're getting into and agree on the direction. :+1:

Campbell Barton commented

2021-04-09 11:11:32 +02:00

My main concern is avoiding difficult to enforce rules, perhaps tooling could enforce this, although the exact details of modules which are loaded is probably platform spesific.

Regarding detectability of errors, it's a valid point, AFAICS it's a trade-off that that prioritizes startup time.
bl_load_py_modules test should be able to cover these kinds of cases if it doesn't already.

My main concern is avoiding difficult to enforce rules, perhaps tooling could enforce this, although the exact details of modules which are loaded is probably platform spesific. Regarding detectability of errors, it's a valid point, AFAICS it's a trade-off that that prioritizes startup time. `bl_load_py_modules` test should be able to cover these kinds of cases if it doesn't already.

Campbell Barton commented

2021-04-09 11:33:53 +02:00

Added a separate proposal for type hints: #87333 (Proposal for type hint use in Blender's Python scripts)

Bastien Montagne commented

2021-04-09 11:40:35 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2021-04-09 11:40:35 +02:00

In general proposal LGTM.

Some precisions regarding strings. C-style formatting for strings also remains the fastest afaik, there are performances concerns still about the more modern approaches?
For UI translations, format system is also compatible in theory (although it risks more breakage, since translators have already a hard time dealing with the basics of C-string type, and format syntax is less universal). f-strings are fully incompatible, given how they are handled by python.

As for module import, not sure listing sys.modules after a factory startup is that much relevant, since some code will have ran as part of UI scripts and operators after the startup, so don't think this represents modules actually loaded at startup itself? Also add-ons enabled by default should not be taken into account here?

But in general I would be fine if some very common modules from python's stdlib are excluded from this rule, too, like sys, os, importlib...

In general proposal LGTM. Some precisions regarding strings. C-style formatting for strings also remains the fastest afaik, there are performances concerns still about the more modern approaches? For UI translations, `format` system is also compatible in theory (although it risks more breakage, since translators have already a hard time dealing with the basics of C-string type, and format syntax is less universal). f-strings are fully incompatible, given how they are handled by python. As for module import, not sure listing `sys.modules` after a factory startup is that much relevant, since some code will have ran as part of UI scripts and operators *after* the startup, so don't think this represents modules actually loaded at startup itself? Also add-ons enabled by default should not be taken into account here? But in general I would be fine if some very common modules from python's `stdlib` are excluded from this rule, too, like `sys`, `os`, `importlib`...

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Startup & Core Scripts Code Style #86943

Motivation

Proposal