world_to_camera_view broken for off-axis projection #74577

Closed
opened 2020-03-09 15:38:26 +01:00 by Erroll Wood · 6 comments

System Information
Operating system: Windows 10
Graphics card: GTX 1070

Blender Version
Broken: 2.82

Short description of error
When a camera has shift_x or shift_y != 0, world_to_camera_view does not work.

We can test this by comparing points projected manually with camera.calc_matrix_camera(), and with the utility world_to_camera_view. When there is no shift_x or shift_y, they line up. But when we change shift_x or shift_y, they are no longer equal.

Projection using the matrix from camera.calc_matrix_camera() seems to be correct. So I think world_to_camera_view() has a bug. This is suprising since the docs suggest that world_to_camera_view() "Takes shift-x/y ... into account"

Exact steps for others to reproduce the error

  1. Open new blend file with default camera and cube
  2. Run this script
import bpy
import numpy as np
from bpy_extras.object_utils import world_to_camera_view

D, C = bpy.data, bpy.context

camera = C.scene.camera
cube = D.objects["Cube"]

def project_with_matrix():

    width, height = C.scene.render.resolution_x, C.scene.render.resolution_y
    projection_matrix = camera.calc_matrix_camera(C.evaluated_depsgraph_get(), x=width, y=height)
    projection_matrix = np.array([list(row) for row in projection_matrix])

    # Cube vertices in camera space
    verts_camspace = np.array([list(camera.matrix_world.inverted() @ v.co) for v in cube.data.vertices])
    
    # Homogenize
    verts_camspace_h = np.hstack([verts_camspace, np.ones((len(verts_camspace), 1))]) 
    
    # Project
    projected = verts_camspace_h.dot(projection_matrix.T)
    
    # Dehomogenize
    projected = projected[:, :2] / projected[:, 3, None] 
    
    # [-1, 1] to [0, 1]
    projected = (projected + 1.0) / 2.0
    
    return projected


def project_with_world_to_camera_view():

    projected = []
    for v in cube.data.vertices:
        projected.append(list(world_to_camera_view(C.scene, camera, v.co).xy))
    
    return np.array(projected)


if __name__ == "__main__":

    camera.data.shift_y = 0.0
    camera.data.shift_x = 0.0
    print("shift_x=0.0, shift_y=0.0:", np.allclose(project_with_matrix(), project_with_world_to_camera_view()))

    camera.data.shift_x = 0.1
    camera.data.shift_y = 0.0
    print("shift_x=0.1, shift_y=0.0:", np.allclose(project_with_matrix(), project_with_world_to_camera_view()))

    camera.data.shift_x = 0.0
    camera.data.shift_y = 0.1
    print("shift_x=0.0, shift_y=0.1:", np.allclose(project_with_matrix(), project_with_world_to_camera_view()))

See this output:

shift_x=0.0, shift_y=0.0: True
shift_x=0.1, shift_y=0.0: False
shift_x=0.0, shift_y=0.1: False

Note: this assumes the default cube has identity transform, so we skip applying it's world matrix

**System Information** Operating system: Windows 10 Graphics card: GTX 1070 **Blender Version** Broken: 2.82 **Short description of error** When a camera has shift_x or shift_y != 0, `world_to_camera_view` does not work. We can test this by comparing points projected manually with `camera.calc_matrix_camera()`, and with the utility `world_to_camera_view`. When there is no shift_x or shift_y, they line up. But when we change shift_x or shift_y, they are no longer equal. Projection using the matrix from `camera.calc_matrix_camera()` seems to be correct. So I think `world_to_camera_view()` has a bug. This is suprising since the docs suggest that `world_to_camera_view()` "Takes shift-x/y ... into account" **Exact steps for others to reproduce the error** 1. Open new blend file with default camera and cube 2. Run this script ``` import bpy import numpy as np from bpy_extras.object_utils import world_to_camera_view D, C = bpy.data, bpy.context camera = C.scene.camera cube = D.objects["Cube"] def project_with_matrix(): width, height = C.scene.render.resolution_x, C.scene.render.resolution_y projection_matrix = camera.calc_matrix_camera(C.evaluated_depsgraph_get(), x=width, y=height) projection_matrix = np.array([list(row) for row in projection_matrix]) # Cube vertices in camera space verts_camspace = np.array([list(camera.matrix_world.inverted() @ v.co) for v in cube.data.vertices]) # Homogenize verts_camspace_h = np.hstack([verts_camspace, np.ones((len(verts_camspace), 1))]) # Project projected = verts_camspace_h.dot(projection_matrix.T) # Dehomogenize projected = projected[:, :2] / projected[:, 3, None] # [-1, 1] to [0, 1] projected = (projected + 1.0) / 2.0 return projected def project_with_world_to_camera_view(): projected = [] for v in cube.data.vertices: projected.append(list(world_to_camera_view(C.scene, camera, v.co).xy)) return np.array(projected) if __name__ == "__main__": camera.data.shift_y = 0.0 camera.data.shift_x = 0.0 print("shift_x=0.0, shift_y=0.0:", np.allclose(project_with_matrix(), project_with_world_to_camera_view())) camera.data.shift_x = 0.1 camera.data.shift_y = 0.0 print("shift_x=0.1, shift_y=0.0:", np.allclose(project_with_matrix(), project_with_world_to_camera_view())) camera.data.shift_x = 0.0 camera.data.shift_y = 0.1 print("shift_x=0.0, shift_y=0.1:", np.allclose(project_with_matrix(), project_with_world_to_camera_view())) ``` See this output: ``` shift_x=0.0, shift_y=0.0: True shift_x=0.1, shift_y=0.0: False shift_x=0.0, shift_y=0.1: False ``` Note: this assumes the default cube has identity transform, so we skip applying it's world matrix
Author

Added subscriber: @errollw

Added subscriber: @errollw

Added subscriber: @mano-wii

Added subscriber: @mano-wii

Changed status from 'Needs Triage' to: 'Confirmed'

Changed status from 'Needs Triage' to: 'Confirmed'

I can confirm.
I tested it with this script and the result is as if the shift-x/y had not been applied:

import bpy
from bpy_extras.object_utils import world_to_camera_view

def project_with_matrix(depsgraph, scene, camera, location):
    from mathutils import Vector

    width, height = scene.render.resolution_x, scene.render.resolution_y
    persp_mat = camera.calc_matrix_camera(depsgraph, x=width, y=height) @ camera.matrix_world.inverted()
    loc = persp_mat @ location.to_4d()
    
    return (loc.xyz / (2 * loc.w)) + Vector((0.5, 0.5, 0.5))

scene = bpy.context.scene
cursor_loc = scene.cursor.location
ob_cam = bpy.context.object
depsgraph = bpy.context.evaluated_depsgraph_get()

print('----------------')
print(world_to_camera_view(scene, ob_cam, cursor_loc))
print(project_with_matrix(depsgraph, scene, ob_cam, cursor_loc))
I can confirm. I tested it with this script and the result is as if the `shift-x/y` had not been applied: ``` import bpy from bpy_extras.object_utils import world_to_camera_view def project_with_matrix(depsgraph, scene, camera, location): from mathutils import Vector width, height = scene.render.resolution_x, scene.render.resolution_y persp_mat = camera.calc_matrix_camera(depsgraph, x=width, y=height) @ camera.matrix_world.inverted() loc = persp_mat @ location.to_4d() return (loc.xyz / (2 * loc.w)) + Vector((0.5, 0.5, 0.5)) scene = bpy.context.scene cursor_loc = scene.cursor.location ob_cam = bpy.context.object depsgraph = bpy.context.evaluated_depsgraph_get() print('----------------') print(world_to_camera_view(scene, ob_cam, cursor_loc)) print(project_with_matrix(depsgraph, scene, ob_cam, cursor_loc)) ```

This issue was referenced by blender/blender@7fcf2e7d4a

This issue was referenced by blender/blender@7fcf2e7d4af1429b461dec92b6d66ff09e9be9f3

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'
Sebastian Parborg self-assigned this 2020-05-19 17:08:06 +02:00
Sign in to join this conversation.
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender-addons#74577
No description provided.