VR: Controller Drawing
When in a VR session, it's very important to have visual feedback on the controllers. At least to tell where they are and which direction they point at.
For a start, just having controllers visualized would be sufficient. Visualizing button presses or touchpad interactions should be done too, but is not as important.
Here is what we could do:
- Mixed Reality provides the XR_MSFT_controller_model_preview OpenXR extension for its controllers. It uses GLTF to provide the geometry. If the GLTF add-on is available in Blender, we can use that to obtain the geometry. Other vendors will probably provide geometry this way too, in the future.
- For other controllers, we may have to bundle geometry ourselves somehow.
- There should be a simple fallback solution for when we don't have any geometry data. That may be a simple tube-like shape, or even a simple coordinate axis.
Open Question: With the geometry available, how should controllers be drawn in the viewport?
Controllers have to be drawn to the regular projection-layer of the OpenXR compositor. This means they must be drawn during or after regular viewport drawing. But how exactly to do this?
- Draw controllers as gizmos. Maybe gizmos were not made for such a case (drawing custom geometry in the viewport), but this could work fairly well. Currently shading/lighting would have to be done in a custom shader. Maybe gizmos could somehow use draw-manager shaders though. [Seems there are more and more reasons to integrate gizmos properly with the draw-manager...]
- Have some way to hand the draw-manager custom geometry to draw with its shading.
- Entirely custom OpenGL drawing using draw-handlers.