Object Tracking markers are not scaled with the camera scale visually (but actually are scaled internally???) (With file and video) #74322

New Issue

Manuel · 2020-02-29T21:17:27+01:00

Manuel commented

2020-02-29 21:17:27 +01:00

System Information
Operating system: Windows-10-10.0.18362-SP0 64 Bits
Graphics card: GeForce GTX 750 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 432.00

Blender Version
Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: 77d23b0bd7
Worked: (optional)

Short description of error
Blender's motion tracker is not showing the locations of the object tracking markers correctly if the camera with the solve has a non-1 scale.

(Even using the orientation tools for the camera solve actually scales the camera in blender, scaling the tracking data independent from this value seems to be impossible right now. There are many cases where a camera track needs to be scaled, for example in order to match up with real world dimensions.)

Exact steps for others to reproduce the error

Track footage for camera and object tracking.
Put the camera solve on a blender camera.
Scale the camera to a non-1 value.
The 3d markers of the object track don't represent the coordinates anymore that are being used for an object solve.

I explain my problem and my understanding of what might be happening (and what should be) in this video here:
https://youtu.be/tJG8B2gj2b0

(Please note the image sequence is attached for reference but completely arbitrary and not necessary to understand the problem.)

ExampleSequenceObjectTrack.zip ExampleObjectTrack.blend

**System Information** Operating system: Windows-10-10.0.18362-SP0 64 Bits Graphics card: GeForce GTX 750 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 432.00 **Blender Version** Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: `77d23b0bd7` Worked: (optional) **Short description of error** Blender's motion tracker is not showing the locations of the object tracking markers correctly if the camera with the solve has a non-1 scale. (Even using the orientation tools for the camera solve actually scales the camera in blender, scaling the tracking data independent from this value seems to be impossible right now. There are many cases where a camera track needs to be scaled, for example in order to match up with real world dimensions.) **Exact steps for others to reproduce the error** 1. Track footage for camera and object tracking. 2. Put the camera solve on a blender camera. 3. Scale the camera to a non-1 value. 4. The 3d markers of the object track don't represent the coordinates anymore that are being used for an object solve. I explain my problem and my understanding of what might be happening (and what should be) in this video here: https://youtu.be/tJG8B2gj2b0 (Please note the image sequence is attached for reference but completely arbitrary and not necessary to understand the problem.) [ExampleSequenceObjectTrack.zip](https://archive.blender.org/developer/F8380439/ExampleSequenceObjectTrack.zip) [ExampleObjectTrack.blend](https://archive.blender.org/developer/F8380591/ExampleObjectTrack.blend)

Manuel commented

2020-02-29 21:17:27 +01:00

Added subscriber: @manuell3d

Sean Kennedy commented

2020-02-29 22:46:28 +01:00

Added subscriber: @SeanKennedy

Evan Wilson commented

2020-03-01 16:41:01 +01:00

Added subscriber: @EAW

Evan Wilson commented

2020-03-01 16:41:01 +01:00

Scaling_fixed0001-0100.mp4

ExampleObjectTrack1.blend

I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor.

I set the scaled camera back to 1.0. I then used the coffee table's legs to set and apply the scale to 1m.

I also adjusted the object's tracker scaling the same way, and set it to the 1.6 you approximately had for the camera. I also tinkered with setting and clearing the inverse several times.

I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me. I feel that I kind of lucked into the proper solution, made sure to save it immediately and made the video. This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

[Scaling_fixed0001-0100.mp4](https://archive.blender.org/developer/F8381034/Scaling_fixed0001-0100.mp4) [ExampleObjectTrack1.blend](https://archive.blender.org/developer/F8381032/ExampleObjectTrack1.blend) I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor. I set the scaled camera back to 1.0. I then used the coffee table's legs to set and apply the scale to 1m. I also adjusted the object's tracker scaling the same way, and set it to the 1.6 you approximately had for the camera. I also tinkered with setting and clearing the inverse several times. I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me. I feel that I kind of lucked into the proper solution, made sure to save it immediately and made the video. This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

Manuel commented

2020-03-01 17:12:30 +01:00

Thank you for the quick answer! :)

In #74322#882946, @EAW wrote:
I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor.

Even if that was the case, it shouldn't be. As far as I can tell though, it really practically makes absolutely no difference.

I also tinkered with setting and clearing the inverse several times.

This definetly should not need any tinkering, it's as straight forward as can be in a scale of 1.

I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me.

I feel pretty confident by now that what I explain in the video is more or less what is wrong here.

This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

Really nice of you! Please note this exact scene is just an example because I can't share any of the material we're working on in the studio. So specifically we're trying to find a halfway feasible and reversible workaround for it. However it still definetly is NOT how this is supposed to work.

(For the workaround we're thinking about applying the object solve in a scale 1, then scale the camera. Honestly it's really a complete mess to dance around this flaw and it's really inviting human error...)

Thank you for the quick answer! :) > In #74322#882946, @EAW wrote: > I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor. Even if that was the case, it shouldn't be. As far as I can tell though, it really practically makes absolutely no difference. > I also tinkered with setting and clearing the inverse several times. This definetly should not need any tinkering, it's as straight forward as can be in a scale of 1. > I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me. I feel pretty confident by now that what I explain in the video is more or less what is wrong here. >This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently. Really nice of you! Please note this exact scene is just an example because I can't share any of the material we're working on in the studio. So specifically we're trying to find a halfway feasible and reversible workaround for it. However it still definetly is NOT how this is supposed to work. (For the workaround we're thinking about applying the object solve in a scale 1, then scale the camera. Honestly it's really a complete mess to dance around this flaw and it's really inviting human error...)

Evan Wilson commented

2020-03-02 18:08:18 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Evan Wilson commented

2020-03-02 18:08:18 +01:00

Added subscribers: @Sergey, @sebastian_k

Evan Wilson commented

2020-03-02 18:08:18 +01:00

@manuell3d

Please note this exact scene is just an example because I can't share any of the material we're working on in the studio.

Of course!

I wasn't able to get specific steps of a workaround working in my limited time trying yesterday.

@Sergey and @sebastian_k any ideas about this one? I noticed @Sergey you were working on the tracking constraints a few weeks ago, not sure if this issue looks related.

@manuell3d >Please note this exact scene is just an example because I can't share any of the material we're working on in the studio. Of course! I wasn't able to get specific steps of a workaround working in my limited time trying yesterday. --- @Sergey and @sebastian_k any ideas about this one? I noticed @Sergey you were working on the tracking constraints a few weeks ago, not sure if this issue looks related.

K.nao commented

2020-03-15 21:05:36 +01:00

Added subscriber: @KNAO

K.nao commented

2020-03-15 21:05:36 +01:00

I am seeing the same problem. It worked properly in the 2.79 release, but has happened since the 2.80 release.
When the camera is scaled, the Object Solver is affected by the camera scale and does not maintain the correct position on the Solve.（Windows-10）

I am seeing the same problem. It worked properly in the 2.79 release, but has happened since the 2.80 release. When the camera is scaled, the Object Solver is affected by the camera scale and does not maintain the correct position on the Solve.（Windows-10）

K.nao commented

2020-03-16 08:18:45 +01:00

I reconfirmed. It seems that the object solver is not a problem and the object tracking marker displayed in the viewport overlay is not linked to the camera scale in the first place.
Therefore, even if you adjust the Object Scale of the clip editor by believing in overlay display, if the camera scale is included, the actual position on the display will be different from the actual position.

I reconfirmed. It seems that the object solver is not a problem and the object tracking marker displayed in the viewport overlay is not linked to the camera scale in the first place. Therefore, even if you adjust the Object Scale of the clip editor by believing in overlay display, if the camera scale is included, the actual position on the display will be different from the actual position.

Sebastian Koenig commented

2020-03-16 11:03:00 +01:00

Yeah, that seems to be wrong indeed. Scaling the camera to match the scene is very common, obviously the object track should still work. Seems to be a bug indeed.

Sergey Sharybin commented

2020-03-16 13:02:08 +01:00

Added subscribers: @fclem, @Jeroen-Bakker

Sergey Sharybin commented

2020-03-16 13:02:08 +01:00

The drawing code explicitly uses normalized matrix for tracking object, which cancels out any scale.

@fclem, @Jeroen-Bakker any reason behind this? Or just legacy code which did weird thing? This seems to fix the problem P1301: #74322

diff --git a/source/blender/draw/engines/overlay/overlay_extra.c b/source/blender/draw/engines/overlay/overlay_extra.c
index 49f266291da..53550fb115e 100644
--- a/source/blender/draw/engines/overlay/overlay_extra.c
+++ b/source/blender/draw/engines/overlay/overlay_extra.c
@@ -875,11 +875,9 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb,
   UI_GetThemeColor4ubv(TH_SELECT, text_color_selected);
   UI_GetThemeColor4ubv(TH_TEXT, text_color_unselected);
 
-  float camera_mat- [x]- [x], normal_mat- [x][4];
+  float camera_mat- [x][4];
   BKE_tracking_get_camera_object_matrix(ob, camera_mat);
 
-  normalize_m4_m4(normal_mat, ob->obmat);
-
   LISTBASE_FOREACH (MovieTrackingObject *, tracking_object, &tracking->objects) {
     float tracking_object_mat- [x][4];
 
@@ -889,12 +887,15 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb,
     else {
       const int framenr = BKE_movieclip_remap_scene_to_clip_frame(
           clip, DEG_get_ctime(draw_ctx->depsgraph));
+
       float object_mat- [x][4];
       BKE_tracking_camera_get_reconstructed_interpolate(
           tracking, tracking_object, framenr, object_mat);
 
-      invert_m4(object_mat);
-      mul_m4_m4m4(tracking_object_mat, normal_mat, object_mat);
+      float object_imat- [x][4];
+      invert_m4_m4(object_imat, object_mat);
+
+      mul_m4_m4m4(tracking_object_mat, ob->obmat, object_imat);
     }
 
     ListBase *tracksbase = BKE_tracking_object_get_tracks(tracking, tracking_object);

Maybe @sebastian_k can test :)

The drawing code explicitly uses normalized matrix for tracking object, which cancels out any scale. @fclem, @Jeroen-Bakker any reason behind this? Or just legacy code which did weird thing? This seems to fix the problem [P1301: #74322](https://archive.blender.org/developer/P1301.txt) ``` diff --git a/source/blender/draw/engines/overlay/overlay_extra.c b/source/blender/draw/engines/overlay/overlay_extra.c index 49f266291da..53550fb115e 100644 --- a/source/blender/draw/engines/overlay/overlay_extra.c +++ b/source/blender/draw/engines/overlay/overlay_extra.c @@ -875,11 +875,9 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb, UI_GetThemeColor4ubv(TH_SELECT, text_color_selected); UI_GetThemeColor4ubv(TH_TEXT, text_color_unselected); - float camera_mat- [x]- [x], normal_mat- [x][4]; + float camera_mat- [x][4]; BKE_tracking_get_camera_object_matrix(ob, camera_mat); - normalize_m4_m4(normal_mat, ob->obmat); - LISTBASE_FOREACH (MovieTrackingObject *, tracking_object, &tracking->objects) { float tracking_object_mat- [x][4]; @@ -889,12 +887,15 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb, else { const int framenr = BKE_movieclip_remap_scene_to_clip_frame( clip, DEG_get_ctime(draw_ctx->depsgraph)); + float object_mat- [x][4]; BKE_tracking_camera_get_reconstructed_interpolate( tracking, tracking_object, framenr, object_mat); - invert_m4(object_mat); - mul_m4_m4m4(tracking_object_mat, normal_mat, object_mat); + float object_imat- [x][4]; + invert_m4_m4(object_imat, object_mat); + + mul_m4_m4m4(tracking_object_mat, ob->obmat, object_imat); } ListBase *tracksbase = BKE_tracking_object_get_tracks(tracking, tracking_object); ``` Maybe @sebastian_k can test :)

Sebastian Koenig commented

2020-03-16 14:04:34 +01:00

Yep, that seems to fix it. :)