Split of projection matrix for left and right eye #29161

New Issue

Damien Touraine · 2011-11-05T06:36:49+01:00

Damien Touraine commented

2011-11-05 06:36:49 +01:00

%%%Currently, there is only one projection matrix inside blender. This matrix is adapted by a single eye shifting for stereoscopy rendering.
In specific case (CAVE Automatic Virtual Environment), the stereoscopic projection must be adapted to the position of the eyes of the user regarding the scene. For instance, the user can "walk" on the screen and turning on himself.
The default stereoscopic management by BGE is not enough to manage such case. So, we suggest to add two projection matrix to a given camera : one per eye. Both matrices can be addressed by Python API. It will be more simple to adapt the stereoscopy from a Python "plugin".

The patch we propose includes such feature. With the two matrices, it adds an attribute to the camera that is the active rendering eye. As such, every call to the projection matrix of the camera will return the one of the active eye. Before each rendering (inside KX_KetsjiEngine::RenderFrame), the render will set the current eye. Thus each rendering optimization (clipping, scene management, ...) will be computed regarding the current eye.%%%

%%%Currently, there is only one projection matrix inside blender. This matrix is adapted by a single eye shifting for stereoscopy rendering. In specific case (CAVE Automatic Virtual Environment), the stereoscopic projection must be adapted to the position of the eyes of the user regarding the scene. For instance, the user can "walk" on the screen and turning on himself. The default stereoscopic management by BGE is not enough to manage such case. So, we suggest to add two projection matrix to a given camera : one per eye. Both matrices can be addressed by Python API. It will be more simple to adapt the stereoscopy from a Python "plugin". The patch we propose includes such feature. With the two matrices, it adds an attribute to the camera that is the active rendering eye. As such, every call to the projection matrix of the camera will return the one of the active eye. Before each rendering (inside KX_KetsjiEngine::RenderFrame), the render will set the current eye. Thus each rendering optimization (clipping, scene management, ...) will be computed regarding the current eye.%%%

Damien Touraine commented

2011-11-05 06:36:49 +01:00

Changed status to: 'Open'

Damien Touraine commented

2011-11-09 08:54:12 +01:00

%%%Actually, we may also have to split the modelview matrix. Because, among others, OpenGL lighting is computed regarding modelview matrix, but not projection one.
Thus, if the disparity between is voluntarily big (or two different point of view), there might be artifacts in lighting computation.
But, contrary to the projection matrix, the modelisation comes from KX_GameObject. I don't know a smart way to split the modelview matrix.%%%

%%%Actually, we may also have to split the modelview matrix. Because, among others, OpenGL lighting is computed regarding modelview matrix, but not projection one. Thus, if the disparity between is voluntarily big (or two different point of view), there might be artifacts in lighting computation. But, contrary to the projection matrix, the modelisation comes from KX_GameObject. I don't know a smart way to split the modelview matrix.%%%

Dalai Felinto commented

2011-11-09 23:41:20 +01:00

%%%assigning to myself. I will try to look at it next week.
You may be right on the model matrix issue.%%%

%%%assigning to myself. I will try to look at it next week. You may be right on the model matrix issue.%%%

Damien Touraine commented

2011-11-10 07:21:55 +01:00

%%%Concerning the model matrix, we can add a "post camera position" left and right matrices. These matrices are generally equal to identity (unless stereo rendering). The current eye one should be left mutliply by the matrix generated from camera position in the KX_Camera::GetModelviewMatrix() .
But that is an efficient method only if all scene management (clipping planes, bounding spheres, and so on) rely on KX_Camera::GetModelviewMatrix() method to optimize the rendering. I'm not enough confident with blender to be sure that will work.%%%

%%%Concerning the model matrix, we can add a "post camera position" left and right matrices. These matrices are generally equal to identity (unless stereo rendering). The current eye one should be left mutliply by the matrix generated from camera position in the KX_Camera::GetModelviewMatrix() . But that is an efficient method only if all scene management (clipping planes, bounding spheres, and so on) rely on KX_Camera::GetModelviewMatrix() method to optimize the rendering. I'm not enough confident with blender to be sure that will work.%%%

Damien Touraine commented

2011-11-14 13:47:51 +01:00

%%%In the aim of my previous message, I studied a smart way to manage stereoscopic modelview update by python API.
The solution I have found rely on the shift of computing of stereo modelview to Camera rather than RAS_OpenGLRasterizer.
Thus BGE asks for the stereoscopic point of view to the camera (that owns most of the interesting datas for stereo view point calculation). If the user has specified one through python API, thus, the camera provides it. Otherwise, it will give a "default" modelview shifting the same way than RAS_OpenGLRasterizer::SetViewMatrix did before.
The version 2 of the patch %%%

%%%In the aim of my previous message, I studied a smart way to manage stereoscopic modelview update by python API. The solution I have found rely on the shift of computing of stereo modelview to Camera rather than RAS_OpenGLRasterizer. Thus BGE asks for the stereoscopic point of view to the camera (that owns most of the interesting datas for stereo view point calculation). If the user has specified one through python API, thus, the camera provides it. Otherwise, it will give a "default" modelview shifting the same way than RAS_OpenGLRasterizer::SetViewMatrix did before. The version 2 of the patch %%%

Damien Touraine commented

2011-11-14 13:54:13 +01:00

%%%In the aim of my previous message, I studied a smart way to manage stereoscopic modelview update by python API.
The solution I have found rely on the transfer of computing of stereo "shifting" from RAS_OpenGLRasterizer to Camera.
That induce little update of other components (ImageRender, Dome, Light), because RAS_OpenGLRasterizer::setViewMatrix(...) is replaced by KX_Camera::GetStereoMatrix(float eyeSeparation) and RAS_OpenGLRasterizer::SetModelviewMatrix() (both methods are a splitt of the two functionnality of RAS_OpenGLRasterizer::setViewMatrix(...)).

With the version 2 of the patch (see attachments), BGE asks for the stereoscopic point of view to the camera (that owns most of the interesting datas for stereo view point calculation). If the user has specified one through python API, thus, the camera provides it. Otherwise, it will give a "default" modelview shifting (transfer of the code from RAS_OpenGLRasterizer::SetViewMatrix did before).
%%%

%%%In the aim of my previous message, I studied a smart way to manage stereoscopic modelview update by python API. The solution I have found rely on the transfer of computing of stereo "shifting" from RAS_OpenGLRasterizer to Camera. That induce little update of other components (ImageRender, Dome, Light), because RAS_OpenGLRasterizer::setViewMatrix(...) is replaced by KX_Camera::GetStereoMatrix(float eyeSeparation) and RAS_OpenGLRasterizer::SetModelviewMatrix() (both methods are a splitt of the two functionnality of RAS_OpenGLRasterizer::setViewMatrix(...)). With the version 2 of the patch (see attachments), BGE asks for the stereoscopic point of view to the camera (that owns most of the interesting datas for stereo view point calculation). If the user has specified one through python API, thus, the camera provides it. Otherwise, it will give a "default" modelview shifting (transfer of the code from RAS_OpenGLRasterizer::SetViewMatrix did before). %%%

Dalai Felinto commented

2011-11-19 00:53:01 +01:00

%%%Hi Damien, I have some questions about your solution and an overall impression.

I think this is a way too complicate route, when it comes to the user.

Instead of adding the Python calls to left/right/middle project/position matrixes .... why not add a camera.frustrum_shift and do all the calculation internally. This should work even for non-stereo caves (like Jorge's) -- although for those caves the camera.projection_matrix was enough).

Also why to have a middle projection matrix?
The way I see you may need to create an option to set camera.frustrum_orientation too. Although that could be achieved by rotating the camera in Blender and walking 'side-ways'.

Unless you are planning to implement a stereo method different than the 'zero parallax' I think you don't need to bother the user with so many details. What do you think?%%%

%%%Hi Damien, I have some questions about your solution and an overall impression. I think this is a way too complicate route, when it comes to the user. Instead of adding the Python calls to left/right/middle project/position matrixes .... why not add a camera.frustrum_shift and do all the calculation internally. This should work even for non-stereo caves (like Jorge's) -- although for those caves the camera.projection_matrix was enough). Also why to have a middle projection matrix? The way I see you may need to create an option to set camera.frustrum_orientation too. Although that could be achieved by rotating the camera in Blender and walking 'side-ways'. Unless you are planning to implement a stereo method different than the 'zero parallax' I think you don't need to bother the user with so many details. What do you think?%%%

Damien Touraine commented

2011-11-21 07:34:07 +01:00

%%%Hi Dalai,
I'm agree with you: it is a little bit complicated. But, CAVE rendering system is very complicated, too. I remind that, in case of CAVE rendering, the screens are contiguous and only the position and orientation of the screens (regarding whole CAVE system) are used to computer projection and modelisation matrices. We must not rely on "classical" camera parameters such as lens. For instance, "Zooming" (changing the focal distance to magnify or decrease the subject) is not possible. You must "scale" the scene. So, if you do internal calculation of the projection according to given stereo shifting, you may induce overlaps or holes at the intersection of the screens wherever it must be continuous, because a single blender rendering node may not be aware of the other screens.

Thus, we have three solutions:

as you suggest, we can add only both camera.frustrum_shift and camera.frustrum_orientation. Thus, we have to compute, internally, all correct matrices. But, latter, we may have a specific case where this formalism is not valid and the developper would have to send another patch to solve his own constraints. Moreover, the order of multiplying of camera.frustrum_shift and camera.frustrum_orientation may induce mistakes.
we patch blender by allowing blender Python API developper to specify his own modelisation AND projection matrices the way he wants. As such, this developper can do whatever he wants with these matrices. He as a full controll of the OpenGL rendering pipeline. The only restriction he has is to specify his "points of views" according to the camera position.
we include all calculation, including screen coordinates inside rea

In case of CAVE system's, all projection screens are not in the same plane. Most of CAVE include perpendicular screens. Moreover, the positions of the users are tracked to adapt the stereo to the "real" position of the eyes of the users.
My idea, with the blender-cave I'm working on, is to fix the position of the blender camera in the CAVE reference frame (for instance, the camera may look at the "main" screen of the Virtual Environment). Thus, each sreen, that is defined in real world reference frame (that is the same than CAVE one), can be define according to the blender camera. Thus, for a given rendering blender window, we must apply a transformation matrix to adapt it the camera to the attached screen.

The integration of middle matrix was for compatibility with before patch applying (not mixing projection_matrix with new projection_matrix_left/right/middle). But, you are agree : we may remove middle matrices. Implicitely, when the user call the default matrix (stereo_position_matrix or projection_matrix), we work on the "middle" matrix (ie : the matrix that is apply when BGE is not in stereo).

BlenderCAVE that will include all these feature will be release soon as open source software. So a sample of how to integrate CAVE by these complex projection matrices will be provided. We may integrate it inside Blender Python API samples provided natively inside Blender.

Actually, I've a problem with the current python API : with a default 2.6 blender (not modified by our patch), whenever I try to add a scale before frustum matrix, the scene itself does not scale. Is there any scene management that adapt the size of the objects to "perfectly" feet inside the window ? If yes, is there any way to desactivate it ?
%%%

%%%Hi Dalai, I'm agree with you: it is a little bit complicated. But, CAVE rendering system is very complicated, too. I remind that, in case of CAVE rendering, the screens are contiguous and only the position and orientation of the screens (regarding whole CAVE system) are used to computer projection and modelisation matrices. We must not rely on "classical" camera parameters such as lens. For instance, "Zooming" (changing the focal distance to magnify or decrease the subject) is not possible. You must "scale" the scene. So, if you do internal calculation of the projection according to given stereo shifting, you may induce overlaps or holes at the intersection of the screens wherever it must be continuous, because a single blender rendering node may not be aware of the other screens. Thus, we have three solutions: * as you suggest, we can add only both camera.frustrum_shift and camera.frustrum_orientation. Thus, we have to compute, internally, all correct matrices. But, latter, we may have a specific case where this formalism is not valid and the developper would have to send another patch to solve his own constraints. Moreover, the order of multiplying of camera.frustrum_shift and camera.frustrum_orientation may induce mistakes. * we patch blender by allowing blender Python API developper to specify his own modelisation AND projection matrices the way he wants. As such, this developper can do whatever he wants with these matrices. He as a full controll of the OpenGL rendering pipeline. The only restriction he has is to specify his "points of views" according to the camera position. * we include all calculation, including screen coordinates inside rea In case of CAVE system's, all projection screens are not in the same plane. Most of CAVE include perpendicular screens. Moreover, the positions of the users are tracked to adapt the stereo to the "real" position of the eyes of the users. My idea, with the blender-cave I'm working on, is to fix the position of the blender camera in the CAVE reference frame (for instance, the camera may look at the "main" screen of the Virtual Environment). Thus, each sreen, that is defined in real world reference frame (that is the same than CAVE one), can be define according to the blender camera. Thus, for a given rendering blender window, we must apply a transformation matrix to adapt it the camera to the attached screen. The integration of middle matrix was for compatibility with before patch applying (not mixing projection_matrix with new projection_matrix_left/right/middle). But, you are agree : we may remove middle matrices. Implicitely, when the user call the default matrix (stereo_position_matrix or projection_matrix), we work on the "middle" matrix (ie : the matrix that is apply when BGE is not in stereo). BlenderCAVE that will include all these feature will be release soon as open source software. So a sample of how to integrate CAVE by these complex projection matrices will be provided. We may integrate it inside Blender Python API samples provided natively inside Blender. Actually, I've a problem with the current python API : with a default 2.6 blender (not modified by our patch), whenever I try to add a scale before frustum matrix, the scene itself does not scale. Is there any scene management that adapt the size of the objects to "perfectly" feet inside the window ? If yes, is there any way to desactivate it ? %%%

Damien Touraine commented

2011-11-21 07:37:21 +01:00

%%%Misprint for third solution :

we include a specific mode that do all calculation, including screen coordinates inside real world inside blender. This calculation won't rely on Focal Lens to computer projection matrix.%%%

%%%Misprint for third solution : * we include a specific mode that do all calculation, including screen coordinates inside real world inside blender. This calculation won't rely on Focal Lens to computer projection matrix.%%%

Dalai Felinto commented

2011-11-21 21:54:20 +01:00

%%%The thing about focal length is that this is a "render" parameter 'borrowed' by the GE for the stereo calculation.

I know that it doesn't make sense in the GE, but since it's there (and already used by the stereo code as the distance to the zero parallax plane) it's not a huge problem to use. But, now that I think of it, it would be a problem for caves since you would need different "focal lengths" for each projection/wall.

So I would say that allowing custom right/left matrixes may be the way (no middle).
Now, would you prefer to wait after the release of BlenderCAVE to gather some user feedback?

Actually, I've a problem with the current python API : with a default 2.6 blender (not modified by our patch), whenever I try to add a scale before frustum matrix, the scene itself > does not scale. Is there any scene management that adapt the size of the objects to "perfectly" feet inside the window ? If yes, is there any way to desactivate it ?

You mean to scale the objects or the openGL matrixes? have an example I can look at?

%%%

%%%The thing about focal length is that this is a "render" parameter 'borrowed' by the GE for the stereo calculation. I know that it doesn't make sense in the GE, but since it's there (and already used by the stereo code as the distance to the zero parallax plane) it's not a huge problem to use. But, now that I think of it, it would be a problem for caves since you would need different "focal lengths" for each projection/wall. So I would say that allowing custom right/left matrixes may be the way (no middle). Now, would you prefer to wait after the release of BlenderCAVE to gather some user feedback? > Actually, I've a problem with the current python API : with a default 2.6 blender (not modified by our patch), whenever I try to add a scale before frustum matrix, the scene itself > does not scale. Is there any scene management that adapt the size of the objects to "perfectly" feet inside the window ? If yes, is there any way to desactivate it ? You mean to scale the objects or the openGL matrixes? have an example I can look at? %%%

Damien Touraine commented

2011-11-22 08:50:11 +01:00

%%%Actually, I'm thinking of using focal length as a parameter to scale the scene. Because, some scenes (for instance biological or astronomical ones) do not have human size scale (few nanometers or severel light-years) wherever the Virtual Environments have typical human size (about two meters heigth). Each camera and screens will have the same focal length. But, the distance between each point of view and its projection screen will be different and dependent on all CAVE screens. Remind that in blender CAVE there is one blender instance per point of view (modulo the OpenGL quad buffer).

Yes, I think that "middle matrix" will be remove. For the moment, I will wait for feebacks from users fromof our lab and other Virtual Environment (Jorge Gascon, Julian Adenauer) before modifying the patch and resubmit it to this blender patch page.
If you wan't, I can send you the current version. But Jorge will update his Git server (see http://www.gmrv.es/~jgascon/BlenderCave/#download) whenever we have a first working version (few problem of synchronisation of the data for the moment).

Concerning the problem of scaling, I made a mistake: if we want to scale a scene or an object, we need, first, to place the reference frame at its center.
%%%

%%%Actually, I'm thinking of using focal length as a parameter to scale the scene. Because, some scenes (for instance biological or astronomical ones) do not have human size scale (few nanometers or severel light-years) wherever the Virtual Environments have typical human size (about two meters heigth). Each camera and screens will have the same focal length. But, the distance between each point of view and its projection screen will be different and dependent on all CAVE screens. Remind that in blender CAVE there is one blender instance per point of view (modulo the OpenGL quad buffer). Yes, I think that "middle matrix" will be remove. For the moment, I will wait for feebacks from users fromof our lab and other Virtual Environment (Jorge Gascon, Julian Adenauer) before modifying the patch and resubmit it to this blender patch page. If you wan't, I can send you the current version. But Jorge will update his Git server (see http://www.gmrv.es/~jgascon/BlenderCave/#download) whenever we have a first working version (few problem of synchronisation of the data for the moment). Concerning the problem of scaling, I made a mistake: if we want to scale a scene or an object, we need, first, to place the reference frame at its center. %%%

Damien Touraine commented

2012-06-11 10:49:08 +02:00

%%%Hi, we have updated to work with blender 45474. Thus, it is working with the new matrices representation.
Altough this patch applies with few translation on revision 47598, we haven't fully test: we would like to know if this patch would be validated before any further test on recent version of blender ...

Regards

 Damien Touraine%%%

%%%Hi, we have updated to work with blender 45474. Thus, it is working with the new matrices representation. Altough this patch applies with few translation on revision 47598, we haven't fully test: we would like to know if this patch would be validated before any further test on recent version of blender ... Regards ``` Damien Touraine%%%

Julian Eisel commented

2015-02-19 16:52:49 +01:00

Added subscriber: @JulianEisel