Object Tracking markers are not scaled with the camera scale visually (but actually are scaled internally???) (With file and video)
Closed, ResolvedPublic
Actions

Description

System Information
Operating system: Windows-10-10.0.18362-SP0 64 Bits
Graphics card: GeForce GTX 750 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 432.00

Blender Version
Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: rB77d23b0bd76f
Worked: (optional)

Short description of error
Blender's motion tracker is not showing the locations of the object tracking markers correctly if the camera with the solve has a non-1 scale.

(Even using the orientation tools for the camera solve actually scales the camera in blender, scaling the tracking data independent from this value seems to be impossible right now. There are many cases where a camera track needs to be scaled, for example in order to match up with real world dimensions.)

Exact steps for others to reproduce the error

Track footage for camera and object tracking.
Put the camera solve on a blender camera.
Scale the camera to a non-1 value.
The 3d markers of the object track don't represent the coordinates anymore that are being used for an object solve.

I explain my problem and my understanding of what might be happening (and what should be) in this video here:
https://youtu.be/tJG8B2gj2b0

(Please note the image sequence is attached for reference but completely arbitrary and not necessary to understand the problem.)

ExampleSequenceObjectTrack.zip313 MBDownload

ExampleObjectTrack.blend9 MBDownload

Revisions and Commits

rB Blender
	rB07d5b8b0231f Fix T74322: Wrong object bundles with scaled camera

Related Objects

Mentioned In: T74673: Object Tracker Constraint doesn't seem to be working properly
Mentioned Here: P1301 T74322

Event Timeline

Manuel (manuell3d) created this task.Feb 29 2020, 9:17 PM

Manuel (manuell3d) updated the task description. (Show Details)Feb 29 2020, 9:22 PM

Sean Kennedy (hype) added a subscriber: Sean Kennedy (hype).Feb 29 2020, 10:46 PM

Manuel (manuell3d) updated the task description. (Show Details)Mar 1 2020, 4:11 PM

ExampleObjectTrack1.blend10 MBDownload

I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor.

I set the scaled camera back to 1.0. I then used the coffee table's legs to set and apply the scale to 1m.

I also adjusted the object's tracker scaling the same way, and set it to the 1.6 you approximately had for the camera. I also tinkered with setting and clearing the inverse several times.

I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me. I feel that I kind of lucked into the proper solution, made sure to save it immediately and made the video. This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

Thank you for the quick answer! :)

In T74322#882946, @Evan Wilson (EAW) wrote:

I think the way to do it is not to scale the camera in the 3D viewport, but to use the scaling in the movie clip editor.

Even if that was the case, it shouldn't be. As far as I can tell though, it really practically makes absolutely no difference.

I also tinkered with setting and clearing the inverse several times.

This definetly should not need any tinkering, it's as straight forward as can be in a scale of 1.

I will try to come up with the exact order of steps to make it work as expected, as right now that isn’t quite clear even to me.

I feel pretty confident by now that what I explain in the video is more or less what is wrong here.

This is so you can continue your work while I check the commit log and other reports to see if the module owner was working on this specific issue recently.

Really nice of you! Please note this exact scene is just an example because I can't share any of the material we're working on in the studio. So specifically we're trying to find a halfway feasible and reversible workaround for it. However it still definetly is NOT how this is supposed to work.

(For the workaround we're thinking about applying the object solve in a scale 1, then scale the camera. Honestly it's really a complete mess to dance around this flaw and it's really inviting human error...)

@Manuel (manuell3d)

Please note this exact scene is just an example because I can't share any of the material we're working on in the studio.

Of course!

I wasn't able to get specific steps of a workaround working in my limited time trying yesterday.

@Sergey Sharybin (sergey) and @Sebastian Koenig (sebastian_k) any ideas about this one? I noticed @Sergey Sharybin (sergey) you were working on the tracking constraints a few weeks ago, not sure if this issue looks related.

Evan Wilson (EAW) added a project: VFX & Video.Mar 12 2020, 6:26 PM

Evan Wilson (EAW) mentioned this in T74673: Object Tracker Constraint doesn't seem to be working properly.Mar 12 2020, 6:40 PM

I am seeing the same problem. It worked properly in the 2.79 release, but has happened since the 2.80 release.
When the camera is scaled, the Object Solver is affected by the camera scale and does not maintain the correct position on the Solve.（Windows-10）

I reconfirmed. It seems that the object solver is not a problem and the object tracking marker displayed in the viewport overlay is not linked to the camera scale in the first place.
Therefore, even if you adjust the Object Scale of the clip editor by believing in overlay display, if the camera scale is included, the actual position on the display will be different from the actual position.

Yeah, that seems to be wrong indeed. Scaling the camera to match the scene is very common, obviously the object track should still work. Seems to be a bug indeed.

The drawing code explicitly uses normalized matrix for tracking object, which cancels out any scale.

@Clément Foucault (fclem), @Jeroen Bakker (jbakker) any reason behind this? Or just legacy code which did weird thing? This seems to fix the problem

P1301 T74322

1	diff --git a/source/blender/draw/engines/overlay/overlay_extra.c b/source/blender/draw/engines/overlay/overlay_extra.c
2	index 49f266291da..53550fb115e 100644
3	--- a/source/blender/draw/engines/overlay/overlay_extra.c
4	+++ b/source/blender/draw/engines/overlay/overlay_extra.c
5	@@ -875,11 +875,9 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb,
6	UI_GetThemeColor4ubv(TH_SELECT, text_color_selected);
7	UI_GetThemeColor4ubv(TH_TEXT, text_color_unselected);
8
9	- float camera_mat[4][4], normal_mat[4][4];
10	+ float camera_mat[4][4];
11	BKE_tracking_get_camera_object_matrix(ob, camera_mat);
12
13	- normalize_m4_m4(normal_mat, ob->obmat);
14	-
15	LISTBASE_FOREACH (MovieTrackingObject *, tracking_object, &tracking->objects) {
16	float tracking_object_mat[4][4];
17
18	@@ -889,12 +887,15 @@ static void camera_view3d_reconstruction(OVERLAY_ExtraCallBuffers *cb,
19	else {
20	const int framenr = BKE_movieclip_remap_scene_to_clip_frame(
21	clip, DEG_get_ctime(draw_ctx->depsgraph));
22	+
23	float object_mat[4][4];
24	BKE_tracking_camera_get_reconstructed_interpolate(
25	tracking, tracking_object, framenr, object_mat);
26
27	- invert_m4(object_mat);
28	- mul_m4_m4m4(tracking_object_mat, normal_mat, object_mat);
29	+ float object_imat[4][4];
30	+ invert_m4_m4(object_imat, object_mat);
31	+
32	+ mul_m4_m4m4(tracking_object_mat, ob->obmat, object_imat);
33	}
34
35	ListBase *tracksbase = BKE_tracking_object_get_tracks(tracking, tracking_object);

Maybe @Sebastian Koenig (sebastian_k) can test :)

Yep, that seems to fix it. :)

@Sergey Sharybin (sergey) LGTM I don't think this was on purpose.

@Sergey Sharybin (sergey) Ok to commit. Don't know either why it was that way.

Sergey Sharybin (sergey) closed this task as Resolved by committing rB07d5b8b0231f: Fix T74322: Wrong object bundles with scaled camera.Mar 17 2020, 5:49 PM

Sergey Sharybin (sergey) claimed this task.

Sergey Sharybin (sergey) added a commit: rB07d5b8b0231f: Fix T74322: Wrong object bundles with scaled camera.

Object Tracking markers are not scaled with the camera scale visually (but actually are scaled internally???) (With file and video)Closed, ResolvedPublicActions

Description

Revisions and Commits

Related Objects

Event Timeline

Object Tracking markers are not scaled with the camera scale visually (but actually are scaled internally???) (With file and video)
Closed, ResolvedPublic
Actions