Page MenuHome

Object Tracking markers are not scaled with the camera scale visually (but actually are scaled internally???) (With file and video)
Closed, ResolvedPublic

Description

System Information
Operating system: Windows-10-10.0.18362-SP0 64 Bits
Graphics card: GeForce GTX 750 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 432.00

Blender Version
Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: rB77d23b0bd76f
Worked: (optional)

Short description of error
Blender's motion tracker is not showing the locations of the object tracking markers correctly if the camera with the solve has a non-1 scale.

(Even using the orientation tools for the camera solve actually scales the camera in blender, scaling the tracking data independent from this value seems to be impossible right now. There are many cases where a camera track needs to be scaled, for example in order to match up with real world dimensions.)

Exact steps for others to reproduce the error

  1. Track footage for camera and object tracking.
  2. Put the camera solve on a blender camera.
  3. Scale the camera to a non-1 value.
  4. The 3d markers of the object track don't represent the coordinates anymore that are being used for an object solve.

I explain my problem and my understanding of what might be happening (and what should be) in this video here:
https://youtu.be/tJG8B2gj2b0

(Please note the image sequence is attached for reference but completely arbitrary and not necessary to understand the problem.)

Event Timeline

I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor.

I set the scaled camera back to 1.0. I then used the coffee table's legs to set and apply the scale to 1m.

I also adjusted the object's tracker scaling the same way, and set it to the 1.6 you approximately had for the camera. I also tinkered with setting and clearing the inverse several times.

I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me. I feel that I kind of lucked into the proper solution, made sure to save it immediately and made the video. This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

Manuel (manuell3d) added a comment.EditedMar 1 2020, 5:12 PM

Thank you for the quick answer! :)

I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor.

Even if that was the case, it shouldn't be. As far as I can tell though, it really practically makes absolutely no difference.

I also tinkered with setting and clearing the inverse several times.

This definetly should not need any tinkering, it's as straight forward as can be in a scale of 1.

I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me.

I feel pretty confident by now that what I explain in the video is more or less what is wrong here.

This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

Really nice of you! Please note this exact scene is just an example because I can't share any of the material we're working on in the studio. So specifically we're trying to find a halfway feasible and reversible workaround for it. However it still definetly is NOT how this is supposed to work.

(For the workaround we're thinking about applying the object solve in a scale 1, then scale the camera. Honestly it's really a complete mess to dance around this flaw and it's really inviting human error...)

Evan Wilson (EAW) changed the task status from Needs Triage to Confirmed.Mar 2 2020, 6:08 PM

@Manuel (manuell3d)

Please note this exact scene is just an example because I can't share any of the material we're working on in the studio.

Of course!

I wasn't able to get specific steps of a workaround working in my limited time trying yesterday.


@Sergey Sharybin (sergey) and @Sebastian Koenig (sebastian_k) any ideas about this one? I noticed @Sergey Sharybin (sergey) you were working on the tracking constraints a few weeks ago, not sure if this issue looks related.

K.nao (nao) added a subscriber: K.nao (nao).EditedMar 15 2020, 9:05 PM

I am seeing the same problem. It worked properly in the 2.79 release, but has happened since the 2.80 release.
When the camera is scaled, the Object Solver is affected by the camera scale and does not maintain the correct position on the Solve.(Windows-10)

I reconfirmed. It seems that the object solver is not a problem and the object tracking marker displayed in the viewport overlay is not linked to the camera scale in the first place.
Therefore, even if you adjust the Object Scale of the clip editor by believing in overlay display, if the camera scale is included, the actual position on the display will be different from the actual position.

Yeah, that seems to be wrong indeed. Scaling the camera to match the scene is very common, obviously the object track should still work. Seems to be a bug indeed.

The drawing code explicitly uses normalized matrix for tracking object, which cancels out any scale.

@Clément Foucault (fclem), @Jeroen Bakker (jbakker) any reason behind this? Or just legacy code which did weird thing? This seems to fix the problem

1diff --git a/source/blender/draw/engines/overlay/overlay_extra.c b/source/blender/draw/engines/overlay/overlay_extra.c
2index 49f266291da..53550fb115e 100644
3--- a/source/blender/draw/engines/overlay/overlay_extra.c
4+++ b/source/blender/draw/engines/overlay/overlay_extra.c
5@@ -875,11 +875,9 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb,
6 UI_GetThemeColor4ubv(TH_SELECT, text_color_selected);
7 UI_GetThemeColor4ubv(TH_TEXT, text_color_unselected);
8
9- float camera_mat[4][4], normal_mat[4][4];
10+ float camera_mat[4][4];
11 BKE_tracking_get_camera_object_matrix(ob, camera_mat);
12
13- normalize_m4_m4(normal_mat, ob->obmat);
14-
15 LISTBASE_FOREACH (MovieTrackingObject *, tracking_object, &tracking->objects) {
16 float tracking_object_mat[4][4];
17
18@@ -889,12 +887,15 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb,
19 else {
20 const int framenr = BKE_movieclip_remap_scene_to_clip_frame(
21 clip, DEG_get_ctime(draw_ctx->depsgraph));
22+
23 float object_mat[4][4];
24 BKE_tracking_camera_get_reconstructed_interpolate(
25 tracking, tracking_object, framenr, object_mat);
26
27- invert_m4(object_mat);
28- mul_m4_m4m4(tracking_object_mat, normal_mat, object_mat);
29+ float object_imat[4][4];
30+ invert_m4_m4(object_imat, object_mat);
31+
32+ mul_m4_m4m4(tracking_object_mat, ob->obmat, object_imat);
33 }
34
35 ListBase *tracksbase = BKE_tracking_object_get_tracks(tracking, tracking_object);

Maybe @Sebastian Koenig (sebastian_k) can test :)

Yep, that seems to fix it. :)

@Sergey Sharybin (sergey) LGTM I don't think this was on purpose.

@Sergey Sharybin (sergey) Ok to commit. Don't know either why it was that way.