Thank you Philipp. Very kind of you. And yes I'm happy now, especially if you can fix it.
I just recently switched from an expensive commercial copy of V-Ray for Rhino to Blender and am really excited what and how much more can be done with it.

In T71908#820445, @Philipp Oeser (lichtwerk) wrote:

@Ryan Guy (rlguy): could you check your file again?
For me, the is no script in ocean_displacement_test.blend, also no "mesh_cache" object...

My apologies, I had uploaded the incorrect file. I have updated the comment with the correct file:

@Ryan Guy (rlguy)
I am not a programmer, so pardon me if this is naive, but should it be

def frame_change_post_handler(scene, depsgraph):

Instead of

def frame_change_post(scene, depsgraph):

See

https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers

And

In T60094#773490, @Sergey Sharybin (sergey) wrote:

There are the following changes now made:

• Dependency graph is always guaranteed to be fully evaluated after handlers for POST depsgraph/frame change (including adding/removing objects, modifying them and so on).

• It is possible to access evaluated versions of datablocks from handlers. See https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers

Currently it's as stable as we can make it. It is indeed true that interface is better be locked, otherwise render thread might cause conflicts with the viewport (due to python handlers possibly modifying data while it's being drawn by the viewport).

There are ways around it, but needs some design changes. Will be handled outside of the bug tracker as a regular development.

Really wish my laptop battery wasn’t right now :)

@

In T71908#820446, @Fran (Prettypicturegirl) wrote:

Thanks Evan,

very kind of you. I am traveling right now, and can do this tomorrow.
Should I file a new bug report with the format you described? Or can I shuffle things around in this bug report?

Thanks,

Fran

You are most welcome!

@Evan Wilson (EAW) The name/identifier of the handler function can be set to any custom name (under the constraints of what is valid in Python). The handler function just needs to be appended to the bpy.app.handlers.frame_change_post list so that Blender runs the function on a frame change.

For example, this should also be valid API usage:

import bpy

def my_custom_handler(scene, depsgraph):
    pass
    

bpy.app.handlers.frame_change_post.append(my_custom_handler)

Thank you for letting me know @Ryan Guy (rlguy)

@Fran (Prettypicturegirl) I have edited the title and description for you.

Perfect, see I'd never had the chance to rephrase tjis way, as I lack the know how in Blender at the moment. Thank you Evan.

For the non-viewport depsgraph the difference is at step 2: there is no copy of new animated property value from evaluated to original.

@Sergey Sharybin (sergey) how would you feel about a way to, instead of updating the original datablock from the handler, update the evaluated datablock? To me this feels more in line with "updating after the animation has run". To avoid breaking the existing API, this could be done from a separate callback, like frame_change_post_animation. The callback could get the evaluated scene, and thus iterate over its evaluated objects.

How would you feel about a way to, instead of updating the original datablock from the handler, update the evaluated datablock?

While this sounds like an easy solution in this specific case, it opens up a huge amount of other issues and inconsistencies:

• Generally, no one should ever edit evaluated datablock. This is what developers are to follow in both C and Python code.
• Currently handlers are acting like a global modification. This means that from a handler you can modify (i.e. procedurally generate) your scene, which will be the same in all the "contexts" (viewport, another window, F12 render).
• Modifications of a datablock are not necessary limited to just overriding scalar properties. You might also want to override ID pointer, which could involve rebuilding relations and pulling new datablock to the graph. Surely, you might get this thing to work, but the next point comes in place.
• You might want to create new datablocks and use them for overriding various things. So your suggestion escalates to support of entirely separate bMain database for the evaluated scene.

This list can go on, with every item going deeper and deeper in design reconsiderations in all areas. So while this is not an invalid plausible path, it needs careful thoughts and clear design put on paper rather than be accepted on a feelings that it solves this specific case.

To avoid breaking the existing API, this could be done from a separate callback, like frame_change_post_animation. The callback could get the evaluated scene, and thus iterate over its evaluated objects.

Please have a look at the existing handler signature. You can access both original and evaluated datablocks since you do have access to the dependency graph. Accessing evaluated objects is not the problem here from the API point of view. What affects on API is If we say that handlers are to modify evaluated datablocks, we are likely to break all existing handlers. One would argue that existing behavior can be preserved, but it will be same limited, it might very quickly get in a way, and having 2 semi-working ways of doing something just adds even more confusion.

Thanks for the clear explanation @Sergey Sharybin (sergey).

I've been thinking about this, and any solution I see keeps boiling down to keeping track of which properties are changed by frame_change_post handlers. That way we can avoid doing a full copy (original → evaluated) in step 5 above and only copy those properties that were updated in the frame_change_post handler. That would keep animated property values intact, and would also allow overriding their values from Python.

Alternatively, it could be possible to make the animation system run but disallow overwriting properties that were changed by frame_change_post. However, this would put a burden on the animation evaluation to check each and every property set, so that'll be bad for performance for sure. And in the end it also boils down to "keep track of which properties were changed".

Keeping track of which properties were changed could be automatically done by RNA, but I suspect that adding even a single boolean check on all RNA setters will negatively impact performance outside of the scope of frame change handlers. Alternatively this could be done explicitly, as something special that's required for frame_change_post handler functions; something like datablock.update_tag_attribute('some_attribute'). I'm also aprehensive of this solution, as it's easy to forget that this is a necessary step, and I had rather Blender do this automatically. However, I would prefer explicitly tagging attributes over worse overall performance.

This approach will not gracefully handle those cases where a single value in DNA is exposed as multiple values in RNA (for example Object.hide_viewport and hide_render)

Modifications of a datablock are not necessary limited to just overriding scalar properties. You might also want to override ID pointer, which could involve rebuilding relations and pulling new datablock to the graph.

Good point. When new datablocks are pulled into the graph, I'm guessing they won't get animated property values now either?

I've been thinking about this, and any solution I see keeps boiling down to keeping track of which properties are changed by frame_change_post handlers. That way we can avoid doing a full copy (original → evaluated) in step 5 above and only copy those properties that were updated in the frame_change_post handler. That would keep animated property values intact, and would also allow overriding their values from Python.

It indeed sounds more like we do need storage of some sort, but it's.. Complicated.

Complication #1: due to the nature of RNA and python scripting you would need to inform all dependency graphs about all property changes. Imagine having importer implemented in Python, ugh :(

Complication #2: Copying becomes more complicated, it needs to know that at some point it need to do one thing, and at another point another thing.

Complication #3: Handlers do not only modify properties via RNA, they might run C function (via bpy utility or via operator) which will perform modification, without using RNA API.

When new datablocks are pulled into the graph, I'm guessing they won't get animated property values now either?

If the handler adds new objects with animation data, yeah, probably it wouldn't. get animation evaluated (since second round of updates only updates tags, without time).

In T71908#928864, @Sergey Sharybin (sergey) wrote:

Complication #1: due to the nature of RNA and python scripting you would need to inform all dependency graphs about all property changes. Imagine having importer implemented in Python, ugh :(

That's only an issue when you want to have existing animation also applied. So the importer would have to be a rather special one, importing some animated data (otherwise it wouldn't have to be run on frame change) but still allowing other animations to be applied as well. I don't know of any importers that do this. Could provide a nice workflow of hand-animating some properties while others are animated by the importer.

Complication #2: Copying becomes more complicated, it needs to know that at some point it need to do one thing, and at another point another thing.

That's only if we want to have this system work everywhere, all the time. From what I can see, it's only an issue in the frame-change-post handler, so applying the changed properties onto the already-evaluated copies could be done in a specialised function. That way simple copies remain simple.

Complication #3: Handlers do not only modify properties via RNA, they might run C function (via bpy utility or via operator) which will perform modification, without using RNA API.

Hmmm yes. We could consider this unsupported (as it is unsupported now as well), and only allow direct RNA prop modifications in a frame-change-post handler.

A possible workaround could be to inspect the C code and figure out which RNA properties it changes, and tagging those for update from the handler in Python. I'm not too thrilled about that approach, though, as it's not friendly and would create unclean code where the C and Python code are tightly coupled. I see this break easily.

That's only an issue when you want to have existing animation also applied. So the importer would have to be a rather special one, importing some animated data (otherwise it wouldn't have to be run on frame change) but still allowing other animations to be applied as well. I don't know of any importers that do this.

You don't want to check whether property is animated or not prior to dependency graph tag. While tagging is not fully cheap, it is way cheaper than checking whether property is animated or not.

Could provide a nice workflow of hand-animating some properties while others are animated by the importer.

Nice workflow would be to "integrate" such properties hand-animation as a part of datablock evaluation. I do not see a scene-global handler which traverses objects a nice workflow.

Such things are not importer-specific, so not sure how this is relevant.

That's only if we want to have this system work everywhere, all the time. From what I can see, it's only an issue in the frame-change-post handler, so applying the changed properties onto the already-evaluated copies could be done in a specialised function. That way simple copies remain simple.

It needs to be a generic enough solution, which will live within design of many areas. You can not be limiting considerations to just "properties change of frame-change-post handler".

Hmmm yes. We could consider this unsupported (as it is unsupported now as well), and only allow direct RNA prop modifications in a frame-change-post handler.

This isn't how the handler is used.

A possible workaround could be to inspect the C code and figure out which RNA properties it changes, and tagging those for update from the handler in Python.

You can't think in terms of RNA properties.
One of usages is to generated meshes, for example, where handler can remove vertices, add new ones.

Long story short: what do I see here so far is an attempts to address some individual aspects of the problem. What I do not see here is a systematic approach which will analyze how functionality is used, how it is expected to behave, how can solution realistically fit existing design of all related areas.

After discussion with @Sergey Sharybin (sergey) we arrived at the following. From what we can see, the following are reasons for having a frame_change_post handler:

1. Reading datablocks. This is doesn't involve any modification, so is not interesting in the scope of this task.
2. Writing to existing datablocks, to either apply a change on top of the original data, or to fully define the state of the datablock.
3. Writing to existing datablocks, to apply a change to the evaluated data (so on top of animation, simulations etc.).
4. Creating new datablocks.

Case 2. is already covered by Blender's current behaviour.

Case 3. could be seen as the frame-change handler being part of the depsgraph evaluation. This could be implemented via customisable Python nodes in the depsgraph. For performance reasons, it's probably a better idea to not do this, and to allow modifying evaluated datablocks from one single function: the frame-change-post handler. However, these changes should not modify (or require modification of) existing relations for sanity's sake. This is likely going to work fine with the current RNA update callbacks. They expect a pointer to original data, and will tag the data for update, but effectively they'll be tagging the evaluated datablock for update. These are not iterated over by the depsgraph, so tagging won't have any impact. For performance & clarity we can explicitly let the tagging functions ignore evaluated datablocks.

Case 4. requires some extra care. We could resolve this by only allowing the creation of new ID datablocks (most importantly Objects) from the frame-change-pre handler. This handler is called after the scene's current frame has been updated, but before the dependency graph evaluation of the new frame.

If I understand what you are trying to prove in your code, then changing this:

mesh_cache_object = bpy.data.objects.get("mesh_cache")

to

mesh_cache_object = [ob.object for ob in depsgraph.object_instances if ob.object.name == "mesh_cache"][0]

will render a static plane of vertices.

I have somewhat similar problem here T77008, though as you can see this line of code does not solve all problems.

In T71908#934122, @Sybren A. Stüvel (sybren) wrote:

After discussion with @Sergey Sharybin (sergey) we arrived at the following.

@Brecht Van Lommel (brecht) Do you have any insight in this? Any preference as to one of the alternatives? See T71908#934122 for context.

Regarding T71908#934122:

More explicitly skipping depsgraph tagging of evaluated datablock sounds good to me.

Disallowing adding relations or creating new datablocks in the post handler I'm not sure about. Such changes require a depsgraph rebuild which may be relatively slow, but it's not clear to me why this is more a problem than in the pre handler. Or which specific current bug this helps with.

It's also unclear to me how these things relate to originally reported problem as described in T71908#916261.

In T71908#991296, @Brecht Van Lommel (brecht) wrote:

Disallowing adding relations or creating new datablocks in the post handler I'm not sure about. Such changes require a depsgraph rebuild which may be relatively slow, but it's not clear to me why this is more a problem than in the pre handler.

If I understand BKE_scene_graph_update_for_newframe() correctly, this would be the order of things when relations change in the post-handler:

1. Update relations
2. Evaluate depsgraph
3. Call post handler
4. Update relations
5. Update relations (this one probably is a no-op)
6. Evaluate depsgraph

When done in the pre handler:

1. Call pre handler
2. Update relations
3. Evaluate depsgraph
4. Update relations

So indeed, it doesn't look like there is much of a difference, apart from the number of operations performed (and thus performance).

It's also unclear to me how these things relate to originally reported problem as described in T71908#916261.

It boils down to the question "when do we allow writing to evaluated datablocks?" So far the answer has been "never", which means that frame-change-post handlers were modifying original data bocks. This in turn led to a conflict between modifications made by the animation system and by the handler (as @Sergey Sharybin (sergey) explained in T71908#916261) , which caused this bug report.

If writing to evaluated datablocks is allowed, that makes it possible to first have the depsgraph evaluation produce an evaluated copy, and then have some Python code that modifies the same evaluated copy. This makes it possible to layer changes on top of each other. Obviously this only applies to the frame-change-post handler, because in the frame-change-pre handler there is no evaluated datablock to modify.

A workaround for some cases is to reassign the animated property from the evaluated datablock to the original one at the end of the script, e.g.:

...

obj.scale = obj.evaluated_get(deps).scale

EDIT:
In this case the following line enables animation of the ocean modifier:

scene.objects["mesh_cache"].modifiers["Ocean"].time = scene.objects["mesh_cache"].evaluated_get(depsgraph).modifiers["Ocean"].time

Could anyone tell me if this issue been fixed in any update? I am experiencing the same problem with 2.83, 2.91, 2.92 alpha. Or if anyone could kindly point me to a workaround I'd really appreciate it. Thanks

The status of this task is "Confirmed, Normal", which means that this is still an open issue.

@Alexander Court (spiegelball) Please don't repeat your own comments. This is not a support forum, but a task/issue tracker for Blender developers.