NLA: Strip Blend-space Transforms to Easily Align Actions #83615

New Issue

Wayde Moss · 2020-12-10T06:57:50+01:00

Wayde Moss commented

2020-12-10 06:57:50 +01:00

(NOTE) Custom build at: https://builder.blender.org/download/temp-nla-strip-alignment/temp-nla-strip-alignment-blender-2.92.0-8118685a831a-windows64.zip

In this build, the preblend transform is in world space but that removes the animators ability to properly change the character's origin of animation. Future builds should have it changed to a more convenient space.

The build won't properly support keyframe remapping when a strip has a non-identity blend-space transform. I wanted to roughly add support before I made the custom build. But I'm also working on other NLA patches at the same time and it was too difficult to properly track everything. This feature will only be properly worked on after D8296: Feature: NLA: Evaluate Whole NLA Stack in Tweak Mode is accepted.

Specific Example

(Rough video example of the blend-space problem and solution)
https://www.youtube.com/watch?v=M9ogJygo2VM
The video was made where IK evaluation wasn't working so everything was baked to FK. Things like feet clipping through the ground due to blending should be ignored as they're a non-problem otherwise.

Problem: There is no way to properly blend two Replace action clips through space. As an example, if we have a through-space Walk cycle blended mid-way into a through-space Run cycle, then there is no way to blend them properly using the NLA only. Let's say the hip bone is the main traversal bone. At the start of the blend, the Walk cycle has the hips at hips.location.x = 10. It blends into the start of the Run cycle, which starts at hips.location.x = 0. Thus the hip blends back to 0 which can be a difficult problem to fix.

Solution: Per strip, we apply a pre-blend transform to the hips. In this example, there will be no blend transform for the Walk cycle strip. The Run cycle strip will have blend_transform->location.x = 10 applied on the hips only. Thus, when the Walk blends into the Run, it won't blend to 0. It will blend properly from hips location.x = 10 to somewhere around the same world-space location for the Run.

Work arounds

Just don't blend at all. Let the Run start after the Walk has completely finished. Use an extra root bone to offset the Run in world-space. Then use extra upper strips to fix the pose discontinuity.
Don't use the NLA system at all. Use a third action where Walk and Run are copy-pasted into it. Manually offset the Run and stitch them together. This is a big hassle. Offsetting the Run alone effectively requires transforming all the hip keyframes in world-space. Blender doesn't support this feature yet nor is it planned.

Base Proposal

In general, the goal is to blend two actions. Two concrete intended use cases are:

Align two Walk actions where the root bone is the main traversal bone and moves the whole character through space. The armature object transform is used to place the character's origin of animation.
Align two Walk action where the root bone is non-animated and may be used to place the character's origin of animation (so it may not be in rest pose). The traversal bones are the Hips and IK legs which moves the character through space.

Both cases share the same problem. Blending with the next strip results in the Walk traversal bones blending back to origin because the NLA blends channels in local space. The solution is to provide a pre-blend strip transform for a specific set of bones (root in case (1), Hips and IK legs in case (2)).

This transform cannot be a World-Space transform as that conflicts with the animator's ability to change the character's origin of animation. One solution is a per-bone Local Space transform. It works for both cases but is a bit inconvenient in case (2), where the Hips and IK legs may be children of the root bone, and then potentially require two unique transforms. Another solution is to specify the parent. The children will be preblend-transformed and the transform will be relative to the Parent Pose Space.

Implementation Ideas

Base Implementation: Add support for a preblend local-space transform that applies to a single bone. The fcurve transformation channels are relative to this transform.

At this point, the animator has core access to everything they need to align multiple actions together. Further improvements are about user friendliness and convenience for authoring the blend transforms.

Add an option where the blended bones are children of the specified bone. The blend transform is then relative to the parent's pose transform.
Operator: Align Strip Preblend Transforms. Align active strip to another based on the current time and a specified (potentially arbitrary) alignment bone.
- Time options: current frame, end of other, start of active.
- Location Alignment options: toggles per X,Y,Z)
- Rotation Alignment options: full orientation, direction (Forward, Right, Up)
- Align to cursor: Allow the cursor to be specified for the location and/or rotation.

In the above video, the use-case for an arbitrary alignment bone can be found near the end where I align the 3rd action based on the feet position. This operator is likely to be frequently used for alignment. The video also shows and example when the location alignment should ignore the upward (Z) axis, when I aligned the 1st action with the 2nd (so the feet don't clip the ground when the 2nd action plays)

Per-Strip toggle: Force Full Influence. When aligning animations, the aligned strip should have full influence so the animator can properly align them.

The video shows me temporarily changing the Blend In property to 0 so I can properly align it. This is inconvenient and error-prone.

Design Questions

Should the idea of PreBlend Transforms be generalized to Strip Modifiers? Does anyone have any other ideas for Strip Modifiers (Mirror Strip Modifier (RCS) ) If not, perhaps generalization should wait until there is more than two use cases?
Where should the UI for the Preblend Transforms be shown? A new tab seems adhoc, the same with a new subpanel... but at the same time this is the first and only strip modifier type. There's no group to place them in.

For my thoughts: see https://developer.blender.org/T83615#1353434

For replace strips, there is a rotation interpolation direction problem. (A character doing a Back Flip blended into a Back-Handspring action clip, video doesn't show this issue) Should we always interpolate along the shortest angle? Is there ever a case where the longer angle is desirable? This is complicated even more since the two strips are animating through the blend which means that even the shortest rotation direction may change over the blend. This problem should only apply to Replace strips. They define absolute poses while all the other blend modes define offsets.

For my thoughts: see https://developer.blender.org/T83615#1353434

(NOTE) Custom build at: https://builder.blender.org/download/temp-nla-strip-alignment/temp-nla-strip-alignment-blender-2.92.0-8118685a831a-windows64.zip In this build, the preblend transform is in world space but that removes the animators ability to properly change the character's origin of animation. Future builds should have it changed to a more convenient space. The build won't properly support keyframe remapping when a strip has a non-identity blend-space transform. I wanted to roughly add support before I made the custom build. But I'm also working on other NLA patches at the same time and it was too difficult to properly track everything. This feature will only be properly worked on after [D8296: Feature: NLA: Evaluate Whole NLA Stack in Tweak Mode](https://archive.blender.org/developer/D8296) is accepted. ___ ### Specific Example (Rough video example of the blend-space problem and solution) https://www.youtube.com/watch?v=M9ogJygo2VM The video was made where IK evaluation wasn't working so everything was baked to FK. Things like feet clipping through the ground due to blending should be ignored as they're a non-problem otherwise. **Problem**: There is no way to properly blend two Replace action clips through space. As an example, if we have a through-space Walk cycle blended mid-way into a through-space Run cycle, then there is no way to blend them properly using the NLA only. Let's say the hip bone is the main traversal bone. At the start of the blend, the Walk cycle has the hips at `hips.location.x = 10`. It blends into the start of the Run cycle, which starts at `hips.location.x = 0`. Thus the hip blends back to 0 which can be a difficult problem to fix. **Solution**: Per strip, we apply a pre-blend transform to the hips. In this example, there will be no blend transform for the Walk cycle strip. The Run cycle strip will have `blend_transform->location.x = 10` applied on the hips only. Thus, when the Walk blends into the Run, it won't blend to 0. It will blend properly from hips `location.x = 10` to somewhere around the same world-space location for the Run. **Work arounds** 1) Just don't blend at all. Let the Run start after the Walk has completely finished. Use an extra root bone to offset the Run in world-space. Then use extra upper strips to fix the pose discontinuity. 2) Don't use the NLA system at all. Use a third action where Walk and Run are copy-pasted into it. Manually offset the Run and stitch them together. This is a big hassle. Offsetting the Run alone effectively requires transforming all the hip keyframes in world-space. Blender doesn't support this feature yet nor is it planned. ## Base Proposal In general, the goal is to blend two actions. Two concrete intended use cases are: 1) Align two Walk actions where the root bone is the main traversal bone and moves the whole character through space. The armature object transform is used to place the character's origin of animation. 2) Align two Walk action where the root bone is non-animated and may be used to place the character's origin of animation (so it may not be in rest pose). The traversal bones are the Hips and IK legs which moves the character through space. Both cases share the same problem. Blending with the next strip results in the Walk traversal bones blending back to origin because the NLA blends channels in local space. The solution is to provide a pre-blend strip transform for a specific set of bones (root in case (1), Hips and IK legs in case (2)). This transform cannot be a World-Space transform as that conflicts with the animator's ability to change the character's origin of animation. One solution is a per-bone Local Space transform. It works for both cases but is a bit inconvenient in case (2), where the Hips and IK legs may be children of the root bone, and then potentially require two unique transforms. Another solution is to specify the parent. The children will be preblend-transformed and the transform will be relative to the Parent Pose Space. ## Implementation Ideas - [ ] **Base Implementation:** Add support for a preblend local-space transform that applies to a single bone. The fcurve transformation channels are relative to this transform. At this point, the animator has core access to everything they need to align multiple actions together. Further improvements are about user friendliness and convenience for authoring the blend transforms. - [ ] Add an option where the blended bones are children of the specified bone. The blend transform is then relative to the parent's pose transform. - [ ] **Operator:** Align Strip Preblend Transforms. Align active strip to another based on the current time and a specified (potentially arbitrary) alignment bone. - [ ] *Time options*: current frame, end of other, start of active. - [ ] *Location Alignment options*: toggles per X,Y,Z) - [ ] *Rotation Alignment options:* full orientation, direction (Forward, Right, Up) - [ ] *Align to cursor:* Allow the cursor to be specified for the location and/or rotation. In the above video, the use-case for an arbitrary alignment bone can be found near the end where I align the 3rd action based on the feet position. This operator is likely to be frequently used for alignment. The video also shows and example when the location alignment should ignore the upward (Z) axis, when I aligned the 1st action with the 2nd (so the feet don't clip the ground when the 2nd action plays) - [ ] Per-Strip toggle: Force Full Influence. When aligning animations, the aligned strip should have full influence so the animator can properly align them. The video shows me temporarily changing the Blend In property to 0 so I can properly align it. This is inconvenient and error-prone. ## Design Questions * Should the idea of PreBlend Transforms be generalized to Strip Modifiers? Does anyone have any other ideas for Strip Modifiers ([Mirror Strip Modifier (RCS) ](https://blender.community/c/rightclickselect/YNgbbc/)) If not, perhaps generalization should wait until there is more than two use cases? * Where should the UI for the Preblend Transforms be shown? A new tab seems adhoc, the same with a new subpanel... but at the same time this is the first and only strip modifier type. There's no group to place them in. For my thoughts: see https://developer.blender.org/T83615#1353434 ## Related problem: For replace strips, there is a rotation interpolation direction problem. (A character doing a Back Flip blended into a Back-Handspring action clip, video doesn't show this issue) Should we always interpolate along the shortest angle? Is there ever a case where the longer angle is desirable? This is complicated even more since the two strips are animating through the blend which means that even the shortest rotation direction may change over the blend. This problem should only apply to Replace strips. They define absolute poses while all the other blend modes define offsets. For my thoughts: see https://developer.blender.org/T83615#1353434

Wayde Moss self-assigned this 2020-12-10 06:57:50 +01:00

Wayde Moss commented

2020-12-10 06:57:50 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Wayde Moss commented

2020-12-10 06:57:50 +01:00

Added subscriber: @wbmoss_dev

Brad Clark commented

2020-12-19 19:29:51 +01:00

Added subscriber: @BClark

Christiaan Moleman commented

2021-01-22 12:08:22 +01:00

Added subscriber: @ChristiaanMoleman

Todor Nikolov commented

2021-09-23 12:04:41 +02:00

Added subscriber: @TodorNikolov

Carlos Barret commented

2021-10-02 08:40:49 +02:00

Added subscriber: @carlosedubarreto

Wayde Moss commented

2022-05-07 04:49:42 +02:00

For replace strips, there is a rotation interpolation direction problem. (A character doing a Back Flip blended into a Back-Handspring action clip, video doesn't show this issue) Should we always interpolate along the shortest angle? Is there ever a case where the longer angle is desirable? This is complicated even more since the two strips are animating through the blend which means that even the shortest rotation direction may change over the blend. This problem should only apply to Replace strips. They define absolute poses while all the other blend modes define offsets.

This problem should probably be solved in a separate patch when the problem is more well understood. So the core patch for adding preblend support shouldn't worry about it yet.

Should the idea of PreBlend Transforms be generalized to Strip Modifiers? Does anyone have any other ideas for Strip Modifiers (Mirror Strip Modifier (RCS)) If not, perhaps generalization should wait until there is more than two use cases?

I don't think we need to make Strip Modifiers until there are more than one use.

Where should the UI for the Preblend Transforms be shown? A new tab seems adhoc, the same with a new subpanel... but at the same time this is the first and only strip modifier type. There's no group to place them in.

I'm leaning towards a subpanel for now until it becomes an issue. It can easily be moved later.

> For replace strips, there is a rotation interpolation direction problem. (A character doing a Back Flip blended into a Back-Handspring action clip, video doesn't show this issue) Should we always interpolate along the shortest angle? Is there ever a case where the longer angle is desirable? This is complicated even more since the two strips are animating through the blend which means that even the shortest rotation direction may change over the blend. This problem should only apply to Replace strips. They define absolute poses while all the other blend modes define offsets. This problem should probably be solved in a separate patch when the problem is more well understood. So the core patch for adding preblend support shouldn't worry about it yet. > Should the idea of PreBlend Transforms be generalized to Strip Modifiers? Does anyone have any other ideas for Strip Modifiers (Mirror Strip Modifier (RCS)) If not, perhaps generalization should wait until there is more than two use cases? I don't think we need to make Strip Modifiers until there are more than one use. >Where should the UI for the Preblend Transforms be shown? A new tab seems adhoc, the same with a new subpanel... but at the same time this is the first and only strip modifier type. There's no group to place them in. I'm leaning towards a subpanel for now until it becomes an issue. It can easily be moved later.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

NLA: Strip Blend-space Transforms to Easily Align Actions #83615

Specific Example

Base Proposal

Implementation Ideas

Design Questions

Related problem: