Blender 2.8x / Python, Proposed Changes
Open, NormalPublic

Description

This task is to collect changes (possibly breaking changes), that may go into 2.8x.

Note: Python3.6, builds and runs without any problem with current master it seems, all tests pass OK too, so would not expect any issue switching to it once we are ready.

Proposed

  • Use Python 3.6 Literal String Interpolation for all string formatting.

    Benefit: more readable, and faster evaluation than modulo or format method.

    Update: this is planned some weeks after 2.79 release (allow time for 2.79a if needed). 2.8x branch will also use Python3.6.
  • Use keyword only arguments for internal API's for optional arguments which adjust function behavior (can be decided on case-by case basis, bpy.utils is an example where keyword only arguments can be used for most options)

    Benefit: ability to extend API without worrying about argument order, also more readable code.
  • Support keyword only arguments for RNA API.

    Benefit: We can adjust keyword usage without worrying about callers that rely on argument order.
  • [done] Remove bpy.utils.register_module, already done for default configuration, only addons remaining (see patch to assist with this: P455)

    Benefit: Remove internal code that currently needs to keep track of a classes module when its registered. Also prefer explicit over implicit classes for registration.
  • Replace term median with mean (incorrect use was all over C code and leaked into Python API): Matrix.median_scale, BMFace.calc_center_median, BMFace.calc_center_median_weighted.
  • Rename IDProperty.iteritems() to items(), (convention from Python2, with Python3 items() is default), note that it seems no scripts used iteritems(), though its possible they depend on items() being a list instead of an iterator.
  • Add an 'API version' number in addition to Blender's version system.

    It would be a simple integer, increased every time we change something to Blender API (either directly in py modules, or through RNA API).

    Benefit: while addons in our own repositories (and distributed with Blender) are expected to always be compatible with current master, third party addons could use it to do their own versioning management - current Blender version number (which mostly reflects serious changes done to data model) is not precise enough here.
  • BMesh Operator API currently has enum type arguments which are passes as ints from bmesh.ops API.

    This should be made into something similar to RNA enums (so Python can pass in a string identifier).

    Benefit: Python developer's don't need to guess what magic numbers do.
  • RNA API: Unify handling of singleton custom-data layers.

    Currently there exists Mesh.vertex_paint_masks and Mesh.skin_vertices, these are exposed as lists of layers, but there are only ever single layers.

    The API doesn't yet support a nice way to handle this, but we could add a generic way to expose singleton custom-data layers.
  • Remove addons paths from Python's sys.path, This means addons won't be able to be imported directly.

    Benefit: Avoid polluting the module namespace, since there is no reason add-ons need to be visible globally from any other Python script.

Internals

  • Replace PyAttr_GetItem with Python 3.7's _PyObject_LookupAttr, when we don't want to raise an error.

    Benefit: Correctness, better performance? , search blender's source for comment on this function.

Open Topics

  • Change / remove bpy.app.handlers.scene_update_pre, This is currently running continuously because of changes to scene evaluation, many scripts abuse it to execute functions many times a second.

    Since the original purpose of this callback is no longer working as expected, we should make this run only on scene updates, or - remove it.

    Benefit: Add-on's should remain fast and not slow down Blender by running code continuously.
  • Change behavior or Vector multiply & divide, see discussion on D1743.

    @Sybren A. Stüvel (sybren): We could start using @ for matrix multiplications (of which the dot product is a specialisation); see PEP 465. We can then keep * and / for scalar multiplication/division only.

    Benefit: Ability to perform element-wise division/multiplication.
  • Investigate supporting asyncio.

    Benefit: Better options for performing asynchronous operations.

    Notes: Currently, code that wants to use asyncio has to handle its 'loop kicking' itself, usually by adding its own timer. Idea could be rather to have a single generic 'subloop' timer, only running on demand, to which all scripts could subscribe (probably via a new callback ?).
  • Replace our own auto-completion with Python's for the Py-Console (more a general TODO).

    Benefit: Less code for us to maintain (this isn't and urgent task though).
  • Use type hints (since these are very new, we can take a *wait and see* approach, but seems like these could be useful - and could be automatically set for RNA API's).

    Benefit: Better support code introspection and static checkers.

Details

Type
Design
There are a very large number of changes, so older changes are hidden. Show Older Changes
Campbell Barton (campbellbarton) renamed this task from Blender 2.8x / Python to Blender 2.8x / Python, Proposed Changes.

hi, just to raise some points here.

1/ "Use Python 3.6 Literal String Interpolation for all string formatting."
Q1/ Will the move to Python 3.6 Break many addons?
Q2/ Will this Python 3.6 be available in 2.8 branch immediately?

2/ "Change / remove bpy.app.handlers.scene_update_pre"
Comment: Upon doing a Grep (search for code use in files) the results were 2 addons in contrib & 13 external addons (developed outside Blender) use this code as an important part of their addons code base. I would ask that bpy.app.handlers.scene_update_pre be changed internally to return the same or similar function in a clean way. If it's to be removed I would ask this be held off & changed in 2.8 branch where larger breakages will be more generally more acceptable & expected.

Thanks.

hi, just to raise some points here.

1/ "Use Python 3.6 Literal String Interpolation for all string formatting."
Q1/ Will the move to Python 3.6 Break many addons?

Very unlikely, I don't think Python upgrades caused many problems previously?

Q2/ Will this Python 3.6 be available in 2.8 branch immediately?

Depends, if Python3.6 is released at the time we might, however there isn't some strong reason to link Python3.6 with Blender2.8.

2/ "Change / remove bpy.app.handlers.scene_update_pre"
Comment: Upon doing a Grep (search for code use in files) the results were 2 addons in contrib & 13 external addons (developed outside Blender) use this code as an important part of their addons code base. I would ask that bpy.app.handlers.scene_update_pre be changed internally to return the same or similar function in a clean way. If it's to be removed I would ask this be held off & changed in 2.8 branch where larger breakages will be more generally more acceptable & expected.

Yes, wouldn't do this as part of point release.

Thanks.

Use Python 3.6 Literal String Interpolation for all string formatting.

Didn’t know about that, indeed seems totally worth it!

Use keyword only arguments for internal API's for optional arguments which adjust function behavior (can be decided on case-by case basis, bpy.utils is an example where keyword only arguments can be used for most options)
Support keyword only arguments for RNA API.

Fully agree on this too.

Change / remove bpy.app.handlers.scene_update_pre, This is currently running continuously because of changes to scene evaluation, many scripts abuse it to execute functions many times a second.

Think we should keep it, but only actually call it on real scene update (if at all possible). Current 'main loop'-like behavior is indeed terrible.

Change behavior or Vector multiply & divide, see discussion on D1743.

Agree, just need to decide if we keep * for dot product or not ;)

Investigate supporting asyncio.

Nice TODO, but think this can be done pretty much any time?

Replace our own auto-completion with Python's for the Py-Console (more a general TODO).

Again, nice TODO, but think this can be done pretty much any time?

Use type hints (since these are very new, we can take a *wait and see* approach, but seems like these could be useful - and could be automatically set for RNA API's).

No strong opinion here…

Investigate supporting asyncio.

\o/ yes please!

Agree, just need to decide if we keep * for dot product or not ;)

We could start using @ for matrix multiplications (of which the dot product is a specialisation); see PEP 465. We can then keep * and / for scalar multiplication/division only.

Replace our own auto-completion with Python's for the Py-Console (more a general TODO).

Again, nice TODO, but think this can be done pretty much any time?

I would be so happy if the history management of our Py-Console would be the same as Python, Bash, ZSH and others. In other words: after pressing up a few times, once you press enter, the new entry is at the bottom of the history, and your "history pointer" is there to; pressing down shouldn't do anything (because you're at the bottom of the history), and pressing up should give you the previous command. Right now you're floating somewhere in the middle of your history.

Use type hints (since these are very new, we can take a *wait and see* approach, but seems like these could be useful - and could be automatically set for RNA API's).

No strong opinion here…

Let's start using those. Python code can be very "we get this thing from somewhere and pass it along"-like, and figuring out what it is supposed to be can be challenging. By declaring types (even if Python itself doesn't check) IMO we get easier to read & understand code. Furthermore, IDEs can also use those to aid developers with code completion.

1/ "Use Python 3.6 Literal String Interpolation for all string formatting."
Q1/ Will the move to Python 3.6 Break many addons?

Very unlikely, I don't think Python upgrades caused many problems previously?

I think this was more due to incompatibilities with python c extensions that broke from 3.4 -> 3.5. So render addons that use the renderer's python extension (luxrender, mitsuba prman etc) had to be compiled for 3.5

Add an 'API version' number in addition to Blender's version system.

What's the use case here? Most of the time an addon will follow Blender's official releases. In this case, blender version will be enough for doversion. For the case where the script want to be up to date with master, the scripter can afford some hasattr checks. Instead of adding a burden to Blender devs to remember to bump Python APi version.

hasattr is not enough by far to check for API, it does not handle changes in behavior at all e.g. (we also have an 'hasattr' for DNA, but still need to bump sub-version sometimes).

The problem is not with scripts released with Blender (which are expected to always only work with/for the current master, period), the problem is with third party scripts’ editors who wants their add-on to work with both latest master, and previous releases (and possibly also some other 'code points').

Right now they kind of can do it, but with heap of black magic tricks, instead of being able to just read an extra API rev number.

Extra bonus: we should put that API rev number in generated doc too, that way people know exactly what version of the API is documented online currently (and later, we could even trigger automatic rebuild of API doc based on number change, if/when we have doc building on some buildbot).

I don’t think it would be extra burden to devs, not more than bumping regular sub-version when we change data layout, we do not (or at least, should not :P) change RNA & co that often, besides adding new stuff.

@Sybren A. Stüvel (sybren) using @ is handy but quite a disruptive change.

To make this transition more easily we would be best to disable multiply for cases @ can be used. Get existing scripts working (over some weeks), then add multiply support after.

Replace our own auto-completion with Python's for the Py-Console (more a general TODO).

Again, nice TODO, but think this can be done pretty much any time?

I would be so happy if the history management of our Py-Console would be the same as Python, Bash, ZSH and others. In other words: after pressing up a few times, once you press enter, the new entry is at the bottom of the history, and your "history pointer" is there to; pressing down shouldn't do anything (because you're at the bottom of the history), and pressing up should give you the previous command. Right now you're floating somewhere in the middle of your history.

This is more general development, quite an easy TODO.

Use type hints (since these are very new, we can take a *wait and see* approach, but seems like these could be useful - and could be automatically set for RNA API's).

No strong opinion here…

Let's start using those. Python code can be very "we get this thing from somewhere and pass it along"-like, and figuring out what it is supposed to be can be challenging. By declaring types (even if Python itself doesn't check) IMO we get easier to read & understand code. Furthermore, IDEs can also use those to aid developers with code completion.

Python continues to improve its type declarations, in 3.6x they're quite good, but think its worth seeing how we can get real benefits from them.

For example - Once added, is it possible to run http://mypy-lang.org on Blender's Python code, to give us useful info?

Maybe worth updating a single non-trivial add-on to use them (FBX for example), then see if they give practical benefits.

Let's start using [type hints]. Python code can be very "we get this thing from somewhere and pass it along"-like, and figuring out what it is supposed to be can be challenging. By declaring types (even if Python itself doesn't check) IMO we get easier to read & understand code. Furthermore, IDEs can also use those to aid developers with code completion.

Python continues to improve its type declarations, in 3.6x they're quite good, but think its worth seeing how we can get real benefits from them.

In what way are the benefits I described not "real"?

For example - Once added, is it possible to run http://mypy-lang.org on Blender's Python code, to give us useful info?

I haven't used mypy yet, but it certainly looks interesting.

Maybe worth updating a single non-trivial add-on to use them (FBX for example), then see if they give practical benefits.

I'm already using them in a non-trivial add-on, and it certainly gives practical benefits (as described above).

Let's start using [type hints]. Python code can be very "we get this thing from somewhere and pass it along"-like, and figuring out what it is supposed to be can be challenging. By declaring types (even if Python itself doesn't check) IMO we get easier to read & understand code. Furthermore, IDEs can also use those to aid developers with code completion.

Python continues to improve its type declarations, in 3.6x they're quite good, but think its worth seeing how we can get real benefits from them.

In what way are the benefits I described not "real"?

If we can't actually make use of them in any meaningful way, beyond them acting as type-comments.

For example, if tools to validate types don't work properly when run within Blender.

Code readability is a nice aspect you mention but if we cant validate them its quite weak and likely some are wrong nobody notices.

Another aspect of this is we can have the C-Python API generate type information too, although this will only be useful for editors that can load Blender's Python module (as Eclipse for eg with its remote Python support).

For example - Once added, is it possible to run http://mypy-lang.org on Blender's Python code, to give us useful info?

I haven't used mypy yet, but it certainly looks interesting.

Maybe worth updating a single non-trivial add-on to use them (FBX for example), then see if they give practical benefits.

I'm already using them in a non-trivial add-on, and it certainly gives practical benefits (as described above).

Does it work in Python Editors/IDE's ? - can you give some details on what does/doesn't work?
Are you able to validate that the type signatures are correct?

Does this work for C-API's?


I'm not against this (quite the opposite) just think these are reasonable questions to consider before jumping into using new features across Blender's Python API's.
Especially regarding C/Py-API since it might be quite some work to implement, even some small performance cost. - So if people aren't accessing bpy module directly from their editors, we could save ourselves some hassles and not add support there.

OTOH if this allows good static type checking guarantees that can be tested across all scripts, it's probably worth doing.

Hi, my comment regards the propsed changes to the scene_update_pre handler. To some extent I'm echoing Brendon's concerns - (repeated below).

> 2/ "Change / remove bpy.app.handlers.scene_update_pre"

Comment: Upon doing a Grep (search for code use in files) the results were 2 addons in contrib & 13 external addons (developed outside Blender) use this code as an important part of their addons code base. I would ask that bpy.app.handlers.scene_update_pre be changed internally to return the same or similar function in a clean way. If it's to be removed I would ask this be held off & changed in 2.8 branch where larger breakages will be more generally more acceptable & expected.

Thanks.

Please allow me to present our case and recommendations for this proposal! :D

Firstly let me say that I believe in changing the nature of the scene_update handlers so that they only call their subscribers when a change to the scene has occurred. Right now their current behavior is a little confusing at first as they constantly trigger on every iteration of Blender's main loop. When I first started coding our addon, I believed it would only call my handler when something in the scene, or more generally, in bpy.data changed.

However, the unexpected behaviour of the scene_update handlers actually solved an important problem that made our addon possible! Let me briefly introduce what we do and I'll explain.

Our addon is called crowdrender, we currently have about 150 users that have become members of our website, giving them access to the software.

Our addon allows you to connect machines (we call them nodes) together to do network rendering. It does this in a way that you only have to upload your blend file once, then we stream edits live as the artist works. It basically allows you to render almost immediately on multiple machines even after a small change to the scene. No further uploads of the blend file are necessary. Its very useful for debugging scenes prior to sending them to a render farm or committing to a full render on your own machines, which, crowdrender can network together for you much like netrender does.

We presented at both the blender conference 2016 (https://youtu.be/EF9VDr6UYfk) and also at World Blender Meetup Day. We also had a well received article that got 100 shares on blender nation https://www.blendernation.com/2017/02/21/crowdrender-built-network-rendering-platform/.
So, we have a good deal of engagement with artists and the feedback is that this is something they very much want. So we're working hard to provide it.

Now, though I agree the handler should "do what it says it does", removing the ability to run code frequently in blender poses a risk to our addon. The problem is that without being able to run code frequently, there is no way to poll for incoming messages. When our addon is connected to many computers, there are signals coming in from other machines running blender and our addon.

A good example of a very important use case that demonstrates the need for this is when a node gets switched off or exits. This event is completely in dependent to the session that the user is running. The only way to be aware of this event is to regularly poll a message queue to check for events that have happened outside of the runtime of the current blender session. We act upon messages like this to make sure nodes which are offline are not asked to do a render since they won't respond and their work is given to other active nodes instead.

It would be hard to do this if the only occasions a message could be polled for are when the scene changes. So the fact that the current scene_update handlers do this is VERY useful for us. Removing the ability to poll would make it hard, if not impossible, to poll. However, I see the value in changing the scene_update handlers so that they activate when actual changes to the data occur.

So if this change is going ahead, first, yes, lets do it in 2.8, but also, can we please look a little more closely at why addons use this feature to run code frequently? They may well have very good reasons and the benefits could be substantial. Certainly in our case, being able to network machines together and stream content between blend sessions has been amazing for us and for the projects we're working on. Taking that away would force us back to having to do tedious file uploads on every change.

Now. I understand that the use of the handlers is perhaps not what the authors originally intended, but it is actually very fortunate it does this! So, my second proposal in addition to waiting until 2.8 would be to provide a way to run code frequently, another handler in bpy.app.handlers could be used.

This way rather than abusing the scene_handlers, addon authors could use a dedicated handler. The concern about performance could be addressed in documentation. If there are concerns about frequently running code causing issues, lets have the issues with that documented rather than stopping us from doing it?

I've seen this before, so I am speaking from experience. I am a software engineer by trade and I used to work on full motion flight simulators. These systems have to update at 60Hz. That meant all our code had to be run and returned in 16.6 milli secs. We understood that if we were late, the software would glitch.

Knowing about this restriction helped us all write code that complied. It also worked better that we were allowed the freedom to write code that would be called on every iteration of the main loop, but that we had responsibility to ensure it was fast.

Having the calling frequency restricted was seen as bad design as it either slowed development or resulted in people resorting to ugly hacks.

Look forward to hearing all your thoughts on this :D

James

If we can't actually make use of them in any meaningful way, beyond them acting as type-comments.

For example, if tools to validate types don't work properly when run within Blender.

Code readability is a nice aspect you mention but if we cant validate them its quite weak and likely some are wrong nobody notices.

I agree fully. I've done some experiments with mypy, and it can ignore modules it can't import. Because of this, we should be able to start using it incrementally to check our code.

It does require some changes to some code, though. For example addons/add_curve_sapling/__init__.py performs a from bpy.props import * import; since mypy can't import bpy.props and doesn't know its contents, it raises an error on the use of StringProperty. This can easily avoided by explicitly importing what is being used (which is a good idea anyway).

Another aspect of this is we can have the C-Python API generate type information too, although this will only be useful for editors that can load Blender's Python module (as Eclipse for eg with its remote Python support).

That would be great. If we generate stubs, PyCharm can also handle those, and so can mypy.

Does it work in Python Editors/IDE's ? - can you give some details on what does/doesn't work?
Are you able to validate that the type signatures are correct?

It works, PyCharm will show an error when you, for example, pass a file object where a file name is expected. Of course the function declaration has to contain such annotations.

Does this work for C-API's?

Not sure. If we can generate stubs (i.e. annotated Python code that reflects the functions/classes/types of the C API) it will certainly be supported.

I'm not against this (quite the opposite) just think these are reasonable questions to consider before jumping into using new features across Blender's Python API's.

I certainly agree with that.

Especially regarding C/Py-API since it might be quite some work to implement, even some small performance cost. - So if people aren't accessing bpy module directly from their editors, we could save ourselves some hassles and not add support there.

OTOH if this allows good static type checking guarantees that can be tested across all scripts, it's probably worth doing.

I think that even just generating the stubs will help developers, because IDEs will be able to give much more meaningful information / autocompletion / etc. without having to import the bpy module itself. From there we can take it further by adding & checking type annotations.

@Sybren A. Stüvel (sybren), thanks for the info, seems this is worth supporting, from xxx import * is discouraged by pep8 and our own style guide. Just did a quick check and used of this are mostly in addons (and some tests but in general its not used), places it is used can be fairly easily updated no to.

Generating stubs isn't a problem, RNA can extract this information easily, checking on this mypy supports stubs so nice to know this wont have runtime overhead. (although it could be a bit of a hassle to set this up.... we can try make it relatively pain-free for py developers)

@Jeducious (jameshcrowther), in reply to your post - running Python scripts continuously is just not very scalable. A single addon might use it... but once more then a few start to use this its likely to slow Blender down.

It is in fact a bug that the scene handler acts this way, when it was added originally it was only running on scene updates (IIRC).

So I suppose we could rename it, but I would rather try to find the events you are actually looking for and act on those, its just too easy to misuse a continuous polling function when a more focused check will do (filesystem monitor for example, or check on adding/removing data-blocks).

We could for example - not add this into 2.8x initially, for each use-case this breaks attempt to find an event that solves the use case. Then see if we can do without the handler or not before 2.8x official release.

@Campbell Barton (campbellbarton) I like the approach of not adding this into 2.8 so we can find a way around this, more time would really be appreciated so we can perhaps work together to find a more acceptable way forward :D.

As for the events i am looking for, they are actually outside blender. As I mentioned in my comment above,

when a node gets switched off or exits. This event is completely in dependent to the session that the user is running. The only way to be aware of this event is to regularly poll a message queue to check for events that have happened outside of the runtime of the current blender session.

The 'node' I speak of is another computer running blender that we use to render a part of the image that the user is working on. It sends messages back to the user's session of blender, which our addon detects by polling a message queue. We use one of the scene handlers to do the polling.

So, the event is not triggered by anything within the user's session of blender. Hence the need to poll the message queue and act on anything we find inside it.

Without being able to check our message queue frequently and independently of blender's internal events, our addon won't be able to communicate with other nodes. So, as you can imagine, I'm really keen to find a work around that fulfills the desire to make the scene_handlers better and also still allow our addon to run. :D

Just a note about handlers:
I did not check how it works but ... to avoid misusing them, it would be interesting to add information in console about adding and removing handlers. This would help add-on developers to take more care about using them.

I think there is sufficient information in the blender api, though I think the issue here is that the scene_handler_pre handler does something that is not expected by triggering constantly.

Just a note about handlers:
I did not check how it works but ... to avoid misusing them, it would be interesting to add information in console about adding and removing handlers. This would help add-on developers to take more care about using them.

IMO the console is not the place to educate developers.

This is a bit not related, but several addons still use imp.reload instead of importlib.
Also some example could be added to the python documentation about the proper way of reloading the modules in complex add-ons using __init__.
(like the the case of reloading modules after other imports - i.e. import bpy etc. it will prevent the add-on of registering at least it does in recent builds :)).
The example in the API in Gotchas is not covering that case that is quite often in Blender.

The documentation changes are not urgent, and concerning imp removal it can be made as a separate task.

Concerning * imports i'll start some cleanup in the add-ons, as it also affects using flake8.

This is a bit not related, but several addons still use imp.reload instead of importlib.

In that case reviewers/maintainers should have paid more attention. That function has been deprecated since Python 3.4. In d47e9be4fb I replaced the then-existing obsolete calls.

Also some example could be added to the python documentation about the proper way of reloading the modules in complex add-ons using __init__.

(like the the case of reloading modules after other imports - i.e. `import bpy` etc. it will prevent the add-on of registering at least it does in  recent builds :)).

The example in the API in Gotchas is not covering that case that is quite often in Blender.

I'm sure you have great ideas to include ;-)

The documentation changes are not urgent, and concerning imp removal it can be made as a separate task.

Concerning * imports i'll start some cleanup in the add-ons, as it also affects using flake8.

Great!

Update: Move to Python3.6 is planned after 2.79 release (discussed with @Sergey Sharybin (sergey) @Bastien Montagne (mont29) on IRC).

Scipy can realy help to terrain heightfield generation and image processing addons. Like "Sverchok" one, or "ANT landscape". scipy module and Sverchok

I really like and support the move to Python 3.6 beyond 2.79.

I am specially interested in an improved Python console with dynamic and in context presentation of API usage and navigating the history. Using the annotations is a great step in making this happen.

The following video from PyCon 2017 is a good reference.

Amjith Ramanujam Awesome Command Line Tools PyCon 2017

https://www.youtube.com/watch?v=hJhZhLg3obk

Thank you for all the awesome work.

Satish.

As suggested today on IRC, making robust naming conventions for bmesh operators would help new coders to jump in. It will be very helpful for making the nodal everything work faster. At the moment, the use of for example underscores is pretty random. Having easy to search names will give more time for real coding.

@mathieu menuet (bliblubli), this isn't enough info, are you interested to create a task that includes proposed changes? (something like T51219).

@mathieu menuet (bliblubli), this isn't enough info, are you interested to create a task that includes proposed changes? (something like T51219).

I will if I get enough time to go through.

Hi guys, and particularly @Campbell Barton (campbellbarton), @Sergey Sharybin (sergey) :D merry Christmas to you and the blender devs team :)

So, I just downloaded '2.80 (sub 3)', commit date - b'2017-12-22'. I started work on supporting it in our addon. But there is a key handler that we use that has been removed. bpy.app.handlers.scene_update_post is gone??!?!?!

I have checked the discussion above and I think that most posting here were in favour of keeping it, but also not in its current form, and re-purposing it to actually do what its meant to do, i.e. only trigger when there is a meaningful change to the scene that affects the final output of a render, either to viewport or final.

Let me put forth our case again, for us its critical to have this handler or something that can achieve the same goal. Even in its broken state in 2.79 and previous builds, its crucial to our addon.

Its super important for us to know when the user has made a change to the scene as we then call our addon to go find that change, then we update tens of other computers on local and remote networks with that change so that they are synchronised and can network render immediately. It cuts out costly file uploads and is orders of magnitude faster than even R-sync like algorithms that do file comparison then transfer only deltas.

Here's what our code does in detail, we subscribe a callback to the scene_update_post handler. When it triggers we check collections in the bpy.data main list for "is_updated" being True. If any collections are true, we look for members of the collection that are true and then we quickly log and process what changed and send that through the network. We do not "continually process" its async. It works very similar to the way we believe best practice for addons are spelled out in Blender's python api, which is the addon has a registered function that gets called, runs and then exits. This is what we've been doing since 2013.

I understand that the problem with the scene_update_pre/post handlers constantly triggering is a big issue. In fact this was a big problem for us at first in using it, but we managed to use it regardless. Not having it though, breaks our ability to efficiently tell if something in the scene has changed and what that something is. With no alternative we'll have to poll and that pretty much means using a modal operator with a timer and we're back to constantly running code again, something I believe we're all trying to avoid in 2.8.

Worse, since we'll be forced to do a search on all data to see what has changed since the last run of our operator, the time taken to process will scale linearly with the number of ID blocks in the scene, so more advanced or complicated blend files will get worse performance with the addon enabled.

I think having a proper async method for notification of edits would be ideal. Something like a subscription service, similar to the update handlers, but that can also account for modal operators like grab, rotate, etc and only trigger when those operators actually execute (is_updated gets set True regardless of whether the data is finalised which makes it hard to handle some guy translating an object around on a 5K monitor :D). I'd be more than happy to write requirements for this and contribute to code dev to make it happen.

Also, after we presented this addon at BC2016 last year; we then sat in on the BC2016 "Ask the developers anything session". I asked Sergey about how we could better get feedback from blender's python api that data has changed (you can see his response if you like by checking the youtube video out at this link, I made it link to the moment I ask the question -> https://youtu.be/8rJkm9MKyg8?t=52m30s).

We explained what we were doing and his response was "we'll need to do something smart here". Can I counsel the blender developers that removing our only way of reliably telling what has changed in the scene for view port render is probably not the smartest thing that could have been done! Unless of course there is a replacement that we don't know about. If there is a new solution to tracking data changes for external render engines, we'd love to know about it. Apologies if we've missed announcements on this, we're actively watching bf_python and cycles mailing lists, but not much else besides.

Please think of our users too, its not just us anymore! We've got lots of blender users excited about this and actually using the addon.

Since BC2016, our addon has gained solid traction with the blender community, we've now got 800 website users, 500 have signed up for a free site account to download and use the addon. We developed the latest release (released on the 21/12/2017) with 250 alpha testers who went though 14 revisions of the software with us. Feedback has been very positive and we're assured of a bright future for the addon.

So, can we work together to ensure we can support those who are using the addon now and in the future? We're pretty sure we're up to the challenge :)

Regards

james

There are probably a lot of addon devs who would love to use the scene update handler but don't understand how it works, or they are using it incorrectly. The idealized function of the scene update handlers should still be included in 2.8, just in a less hacky way. The intended function of the scene update handlers should also be included. Specifically, when the scene is updated, the handler should run.

Regarding the hacky use of scene_update, Blender needs a more robust event system. Saving, rendering, and loading a new scene are all important events. However, the current event system is not nearly granular enough.

You can see a similar sort of concept in Jacques "Animation Nodes" system. The nodal execution system has various triggers that are each at a different level of granularity. For Blender as a whole, addon developers really just need a system that is better at tracking WHAT happened and does so at much lower levels than Scene or File. Developers use scene_update to make sure that code is called at certain times. The issue is not that developers want or need to do this. The issue is that addon devs can't employ a relatively standard programming paradigm to be triggered on extremely specific events. Instead, they have to deal with their code basically running all the time, when in reality, it may only need to run 1-5 times in a one hour user session.

As this is still a proposal ...
It would be handy to have an update handler on a collection. To update collections to be used in, for example, prop_search in blender 2.7 one would probably use the scene_update handler as the CollectionProperty doesn't have an update handler.
Moreover we can add handlers into bpy.data collections.

If you remove the scene_update handler in 2.8 how would you expect one to use filtered data in prop_search? A use case that I currently have is to have only value nodes from a node_tree in the prop_search.

Some additionals that might be interesting to look at.

  • Support of functools.partials on get/set and update. This will reduce some code writing.

Might be hard to implement due to the python context sharing, especially after saving and loading.

 def make_unique(self, collection_name, attr_name, value):
     ....

custom_string = bpy.props.StringProperty(update=functools.partial(make_unique, collection_name="objects", attr_name="custom_string")
  • Support of generators or python functions for default
import uuid
def generate_uuid(self):
    ...
uuid = bpy.props.StringProperty(default=generate_uuid)

We use uuid to identify non ID objects when users are allowed to change the name of that object.

  • support of __radd__ in mathutils.Vector
def calc_center_of_polygon(mesh: bpy.types.Mesh, polygon: bpy.types.MeshPolygon):
    return sum([mathutils.Vector(mesh.vertices[vertice].co) for vertice in polygon.vertices]) / len(polygon.vertices)
  • support custom icons for bpy.types.NodeTree. Currently only possible to set custom icons for labels and operations via the draw method. Would be nice to support bl_icon_id. Might be an issue that during loading of a module the icons have not been loaded yet and the icon cannot be determined. Perhaps we should use an own registration for custom icons. so bl_icon_name = "mycustomiconname" together with bl_icon_repository = "mycustomiconrepname".

Python has a PEP for matrix multiplications since Python 3.5. It uses the @ to distinct matrix multiplications from normal multiplications. It is mostly for readability numpy has migrated to support this PEP.

https://www.python.org/dev/peps/pep-0465/

  • also migrate the dot function to @
  • Make sure to not support normal multiplications for a period of time, to give addons devs the time to migrate to the new convention. After that we can reuse normal multiplications for something else.

Note: Will break compatibility of many add-ons.

So, I just downloaded '2.80 (sub 3)', commit date - b'2017-12-22'. I started work on supporting it in our addon. But there is a key handler that we use that has been removed. bpy.app.handlers.scene_update_post is gone??!?!?!

I have checked the discussion above and I think that most posting here were in favour of keeping it, but also not in its current form, and re-purposing it to actually do what its meant to do, i.e. only trigger when there is a meaningful change to the scene that affects the final output of a render, either to viewport or final.

I understand that the problem with the scene_update_pre/post handlers constantly triggering is a big issue. In fact this was a big problem for us at first in using it, but we managed to use it regardless. Not having it though, breaks our ability to efficiently tell if something in the scene has changed and what that something is. With no alternative we'll have to poll and that pretty much means using a modal operator with a timer and we're back to constantly running code again, something I believe we're all trying to avoid in 2.8.

Worse, since we'll be forced to do a search on all data to see what has changed since the last run of our operator, the time taken to process will scale linearly with the number of ID blocks in the scene, so more advanced or complicated blend files will get worse performance with the addon enabled.

I have a similar problem. I'm working on the Renderman for Blender addon, which depends somewhat on scene_update_post for issuing transform edits during IPR, and for updating timestamp information. Removing the function entirely seems like the wrong move.

What's the status on this?

Recently released Toonkit also relies on this function. Remove it without any replacement just make impossible to start porting it for blender 2.8.
Any news about it?

(...) re-purposing it to actually do what its meant to do, i.e. only trigger when there is a meaningful change to the scene that affects the final output of a render, either to viewport or final.

That would be enough for us

Python 3.7 is released: https://docs.python.org/3/whatsnew/3.7.html

Should Blender 2.8 jump directly to Python 3.7 instead of 3.6?
I think it's still a good time to do this change.

+1 for Python 3.7. Now would definitely be a good time.

+1 for Python 3.7. Because type hinting will also be introduced for add-ons PEP 563: Postponed Evaluation of Annotations will be a nice improvement.
And it's kind of funny (okay, to be honest, mainly rather annoying) that the whole vfx and animation industry is still stuck on Python 2.7 and Blender will lead the way with Python 3.7...