Page MenuHome

Virtual RealityProject
ActivePublic

Details

Description

Project tag for Virtual Reality support, sub-module of the EEVEE & Viewport module. Later this may be extended to Augmented or Mixed Reality (or more broadly, XR).


Status
Milestones

Milestone 1: Scene Inspection T71347 - Completed

Milestone 1.5: Controller Interaction T77137, T77127, T77128 - In Review (partially merged)

Milestone 2: Continuous Immersive Drawing T71348 - Scheduled

Milestone 1.5, controller interaction, is currently in review (see T88367). The controller interaction milestone consists of multiple parts, including:

  • Extending the Python API to access VR controller poses and inputs
  • Enabling operator execution via default VR actions
  • Providing visual feedback for controllers (aka controller drawing)
  • Navigating the VR scene using controller inputs
  • Supporting selection and basic transformation of objects in VR
Roadmap

See T91627: Virtual Reality: 3.x Series Roadmap.


Team

Commissioner: @Dalai Felinto (dfelinto)
Project leader: @Julian Eisel (Severin)
Project members: @Peter Kim (muxed-reality)

Description

Big picture:

XR (VR/AR/MR) enables an entirely new way to work with computer generated 3D worlds. Instead of trying to take the UIs we’ve already implemented for traditional monitor use and make them work in XR, we should establish an experience on entirely new grounds. Otherwise, what we’ll end up with is just a different, more difficult way to use Blender.
So to enter the new immersive world, we should think about which workflows could benefit the most from XR, and carefully craft a new, improved experience for them. Don’t try to enable huge amounts of features for XR, risking bad outcome - or even total failure - but find out which features matter the most and focus on them first.

Long term, the experience can then be enriched further, with more features added as smaller, more concise projects. Eventually, XR would allow creating 3D content with unseen interactivity. The entirety of Blender’s content creation capabilities available in an immersive 3D experience. However, this better be the long term vision for XR, not the goal for this initial project.

Process:
Proposal is to do a use-case driven development process. That means, we define a number (say 10-20) of specific use cases. These should be chosen wisely, based on what we think has most potential with XR.
First, just vaguely describe what requirements the system has for a use-case, a bit later on, define what tools and options are needed to get the use case done (possibly by a specific persona).

On top of that, the process should be done iteratively. That means, we define short term goals for an iteration, work on them and evaluate afterwards before we get into the next iteration. The iterations may include implementation work. So we don’t do the waterfall-like “design first”, but there should be enough design work done before first implementations are made.

Any implementation should be driven by a use-case (or multiple) that the team agrees is drafted out well enough.

Communication

To contact the members, reach out to them on #xr or #blender-coders on blender.chat.

Resources

Main task: T68998: Virtual Reality (XR/VR/AR/MR)

Recent Activity

Mon, Nov 22

James Hogan (amalon) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

While developing osgXR, I found the best way to debug SteamVR's GL context messing was using ltrace, i.e.:

Mon, Nov 22, 9:31 AM · Add-ons (BF-Blender), Virtual Reality

Sun, Nov 21

Peter Kim (muxed-reality) updated the summary of D13299: XR: Active Tool Gizmo Display.
Sun, Nov 21, 6:48 AM · Virtual Reality

Fri, Nov 12

Simon Lenz (Amudtogal) updated subscribers of T92723: Crash on Linux when starting a VR Session using SteamVR.

indeed Blender does not crash anymore with this workaround!
However, a new problem emerges:

Fri, Nov 12, 12:35 PM · Add-ons (BF-Blender), Virtual Reality
Peter Kim (muxed-reality) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

@Christoph Haag (haagch) Thanks a lot for the information!
Using your workaround, I was able to get SteamVR+OpenGL+NVIDIA working on Windows (actually SteamVR+OpenGL+AMD already worked on Windows, but it's good to know that this combination has issues on Linux).

Fri, Nov 12, 5:28 AM · Add-ons (BF-Blender), Virtual Reality

Thu, Nov 11

Christoph Haag (haagch) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

I can confirm that blender crashes immAttr2f on AMD too with SteamVR. But it works fine with Monado.

Thu, Nov 11, 10:22 PM · Add-ons (BF-Blender), Virtual Reality
Simon Lenz (Amudtogal) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

Unfortunately, I do not know how Blender's OpenGL context works...
I assume someone with more knowledge about the OpenGL implementation needs to look into this, I do not really know where to continue from here...

Thu, Nov 11, 2:34 PM · Add-ons (BF-Blender), Virtual Reality
Simon Lenz (Amudtogal) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

The crash dump is caused by glMapBufferRange which is returning a null pointer.

Thu, Nov 11, 2:19 PM · Add-ons (BF-Blender), Virtual Reality
Simon Lenz (Amudtogal) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

I am not entirely sure whether my SteamVR setup is complete (Steam is annoying me with missing libraries, although I am not sure whether this could be an issue with how steam is called from Blender...), but it seems to start from the Blender instance.
Blender however, crashes right away when starting the VR session (before SteamVR even starts logging/complaining).

Thu, Nov 11, 11:28 AM · Add-ons (BF-Blender), Virtual Reality
Peter Kim (muxed-reality) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

@Simon Lenz (Amudtogal) Thanks for offering, any help would be greatly appreciated.
I believe the problem is that SteamVR's OpenGL swapchain/context (created to submit the VR images) interferes with Blender's own OpenGL context. However, this only seems to occur on NVIDIA GPUs (not AMD) so some driver or architecture difference is probably also a factor.

Thu, Nov 11, 8:39 AM · Add-ons (BF-Blender), Virtual Reality

Wed, Nov 10

Simon Lenz (Amudtogal) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

I have been trying to setup the Valve Index on Linux as well (for some studio work).
While it took a bit of fiddling to get SteamVR loading the right shared libraries (is there a better way to just copying them over to /usr/lib?), I have run into the same crash.

Wed, Nov 10, 3:58 PM · Add-ons (BF-Blender), Virtual Reality

Tue, Nov 9

Peter Kim (muxed-reality) updated the summary of D13153: XR: Mouse Position Simulation.
Tue, Nov 9, 11:47 AM · Virtual Reality

Sat, Nov 6

Peter Kim (muxed-reality) added a project to P2567 VR Actions Example Add-on: Virtual Reality.
Sat, Nov 6, 5:56 PM · Virtual Reality

Thu, Nov 4

Peter Kim (muxed-reality) closed T88367: Review & Merge Controller Support Patches as Resolved.

Since all of the "Controller Support" patches were closed, I think we can mark this as resolved (D10947: XR: Motion Capture is not really part of controller support).
Besides, this was more meant to be a TODO list for 3.0.

Thu, Nov 4, 9:28 AM · Virtual Reality
Peter Kim (muxed-reality) abandoned D10950: XR Controller Support Step 8: Controller Based Transform.

Abandoning this since the invoke_3d/modal_3d approach used by the transform.translate/rotate/resize operators was abandoned in D10945.
Transform functionality is still planned as future patches, however this will be achieved using gizmos and perhaps an additional VR-specific (collision-based grab) operator.

Thu, Nov 4, 9:14 AM · Virtual Reality
Peter Kim (muxed-reality) abandoned D10949: XR Controller Support Step 7: Controller Based Picking.

Abandoning this since the invoke_3d/modal_3d approach used by the view3d.select operators was abandoned in D10945.
However, both the projection-based (OpenGL) and raycast selection methods described here are still planned as future patches.

Thu, Nov 4, 9:06 AM · Virtual Reality
Peter Kim (muxed-reality) abandoned D10945: XR Controller Support Step 6: Invoke/Modal_3d.

Abandoning this since we should use a different method than the invoke_3d/modal_3d approach proposed here to simulate mouse inputs.
This was discussed in chat as well as T88839.
A potential alternative is to have an XR action property (Simulate Mouse or something) that will do the necessary controller->mouse conversion if set during event dispatching.

Thu, Nov 4, 8:55 AM · Virtual Reality
Peter Kim (muxed-reality) updated the summary of D10947: XR: Motion Capture.
Thu, Nov 4, 8:30 AM · Virtual Reality
Peter Kim (muxed-reality) updated the summary of D13102: XR: Motion Capture (Add-on).
Thu, Nov 4, 8:29 AM · Virtual Reality
Peter Kim (muxed-reality) updated the diff for D10947: XR: Motion Capture.

Rebase on master

Thu, Nov 4, 8:20 AM · Virtual Reality

Wed, Nov 3

Jens (Jens.Ne) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

@Peter Kim (muxed-reality) Thank you for looking into this.
And thank you for all the work you are doing for VR! :)

Wed, Nov 3, 7:20 PM · Add-ons (BF-Blender), Virtual Reality
Julian Eisel (Severin) added a comment to T85853: VR Landmark Switcher.

Having buttons for this in the UI indeed seems redundant. But as operators that can be triggered from within the VR session (e.g. via controllers) they can be quite useful I think.

Wed, Nov 3, 2:29 PM · Virtual Reality, Add-ons (Community)
Peter Kim (muxed-reality) updated subscribers of T85853: VR Landmark Switcher.

@Roman Shuvalov (romanshuvalov) I apologize for the extremely late feedback.
I think that the VR landmark switching functionality (operators) proposed in your add-on would be useful in VR (press a controller button to move to the next landmark), however I'm not so sure of its usefulness as a button in the viewport sidebar UI.

Wed, Nov 3, 2:13 PM · Virtual Reality, Add-ons (Community)

Tue, Nov 2

Peter Kim (muxed-reality) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

@Jens (Jens.Ne) Thanks for the report, and good thinking for making a new one due to the different crash log.
Although the crash log does not show anything XR or SteamVR-related, I believe the underlying issue is still the same (using SteamVR with an NVIDIA graphics card in Blender).

Tue, Nov 2, 1:20 PM · Add-ons (BF-Blender), Virtual Reality

Mon, Nov 1

Germano Cavalcante (mano-wii) added a project to T92723: Crash on Linux when starting a VR Session using SteamVR: Add-ons (BF-Blender).

In fact it's not Add-ons (Community) but it's Add-ons (BF-Blender). So adding the tag.

Mon, Nov 1, 9:18 PM · Add-ons (BF-Blender), Virtual Reality
Jens (Jens.Ne) added a comment to T92723: Crash on Linux when starting a VR Session using SteamVR.

The VR Scene Inspection never worked for me on Linux.
The last time I reported about that was in this Task: https://developer.blender.org/T78267
This was closed with a fix for Windows.

Mon, Nov 1, 6:50 PM · Add-ons (BF-Blender), Virtual Reality
Jens (Jens.Ne) edited projects for T92723: Crash on Linux when starting a VR Session using SteamVR, added: Virtual Reality; removed Add-ons (Community).
Mon, Nov 1, 6:48 PM · Add-ons (BF-Blender), Virtual Reality
Germano Cavalcante (mano-wii) merged T92579: I think My Reverb G2 is not working yet into T76082: VR Scene Inspection: It shows me only a pink screen. (It doesn't support AMD GPU).
Mon, Nov 1, 2:50 PM · Virtual Reality

Sat, Oct 30

Marcin (koburzasty) added a watcher for Virtual Reality: Marcin (koburzasty).
Sat, Oct 30, 1:04 PM

Oct 26 2021

Peter Kim (muxed-reality) updated the task description for T91627: Virtual Reality: 3.x Series Roadmap.
Oct 26 2021, 9:32 AM · Virtual Reality
Peter Kim (muxed-reality) edited Description on Virtual Reality.
Oct 26 2021, 9:31 AM
Peter Kim (muxed-reality) closed D11271: VR Add-on: Default Actions, Controller Gizmos.
Oct 26 2021, 6:46 AM · Virtual Reality
Peter Kim (muxed-reality) closed D11501: XR Controller Support Step 5: Navigation.
Oct 26 2021, 6:36 AM · Virtual Reality

Oct 25 2021

Julian Eisel (Severin) accepted D11501: XR Controller Support Step 5: Navigation.

Okay I see, thanks for clarification.

Oct 25 2021, 12:21 PM · Virtual Reality

Oct 24 2021

Peter Kim (muxed-reality) updated the diff for D11271: VR Add-on: Default Actions, Controller Gizmos.

Add base scale property to landmarks

Oct 24 2021, 1:24 AM · Virtual Reality
Peter Kim (muxed-reality) added a comment to D11501: XR Controller Support Step 5: Navigation.

Okay I understand that part. But why do we need 2 scale factors then? XrSessionSettings.base_scale & XrSessionState.navigation_scale? Seems to me (and I'm probably wrong here) that the latter is redundant?

Oct 24 2021, 12:23 AM · Virtual Reality

Oct 23 2021

Jacob Merrill (blueprintrandom) added a comment to D11501: XR Controller Support Step 5: Navigation.

I think one is for overlay elements too not be scaled* like drawn stuff in the vr session / ui feedback

Oct 23 2021, 7:08 PM · Virtual Reality
Julian Eisel (Severin) added a comment to D11501: XR Controller Support Step 5: Navigation.

Sure. Say a user is making a game where the main character is a child, and the user wants to walk through their modeled environment at 0.5 scale to simulate the experience.
If the view is simply zoomed in to reduce the viewer's height, every step in the real world taken by the user will be too large for the character, so this needs to be offset by introducing an additional scale factor.

Okay I understand that part. But why do we need 2 scale factors then? XrSessionSettings.base_scale & XrSessionState.navigation_scale? Seems to me (and I'm probably wrong here) that the latter is redundant?

Oct 23 2021, 6:24 PM · Virtual Reality

Oct 18 2021

Peter Kim (muxed-reality) added a comment to D11501: XR Controller Support Step 5: Navigation.

I don't quite understand the advantage of that over just moving the view in and out as usual in Blender. Could you elaborate a bit more?

Sure. Say a user is making a game where the main character is a child, and the user wants to walk through their modeled environment at 0.5 scale to simulate the experience.
If the view is simply zoomed in to reduce the viewer's height, every step in the real world taken by the user will be too large for the character, so this needs to be offset by introducing an additional scale factor.

Oct 18 2021, 12:04 PM · Virtual Reality
Julian Eisel (Severin) added a comment to D11501: XR Controller Support Step 5: Navigation.

This is done because in VR you may want to maintain viewer scale changes, so the viewer can walk/fly around their scene at a certain scale.

I don't quite understand the advantage of that over just moving the view in and out as usual in Blender. Could you elaborate a bit more?

Oct 18 2021, 11:08 AM · Virtual Reality

Oct 17 2021

Peter Kim (muxed-reality) added a comment to D11501: XR Controller Support Step 5: Navigation.

So there is an actual scale that is applied to the view matrix? This is not something we do in Blender usually, we "zoom" by moving the view in and out. Is there a reason you don't do this here? I'm not surprised that there are issues with the scaling.

Yes there is an actual scale applied to the view matrix (see wm_xr_draw_matrices_create() in wm_xr_draw.c).
This is done because in VR you may want to maintain viewer scale changes, so the viewer can walk/fly around their scene at a certain scale.

Oct 17 2021, 11:34 PM · Virtual Reality
Peter Kim (muxed-reality) updated the diff for D11501: XR Controller Support Step 5: Navigation.

Update with requested changes

Oct 17 2021, 11:07 PM · Virtual Reality
Julian Eisel (Severin) added a comment to D11501: XR Controller Support Step 5: Navigation.

So there is an actual scale that is applied to the view matrix? This is not something we do in Blender usually, we "zoom" by moving the view in and out. Is there a reason you don't do this here? I'm not surprised that there are issues with the scaling.

Oct 17 2021, 6:20 PM · Virtual Reality
Peter Kim (muxed-reality) added a comment to D11501: XR Controller Support Step 5: Navigation.

The scale feature is separate from the navigation, strictly speaking, right? Would prefer that to be committed separately.

Will commit this part separately.

Oct 17 2021, 1:34 PM · Virtual Reality
Peter Kim (muxed-reality) updated the diff for D11501: XR Controller Support Step 5: Navigation.

Update with requested changes

Oct 17 2021, 12:54 PM · Virtual Reality
Nis Madsen (nima01) added a watcher for Virtual Reality: Nis Madsen (nima01).
Oct 17 2021, 1:37 AM

Oct 16 2021

Julian Eisel (Severin) removed a reviewer for D11501: XR Controller Support Step 5: Navigation: Dalai Felinto (dfelinto).
Oct 16 2021, 1:42 PM · Virtual Reality
Julian Eisel (Severin) requested changes to D11501: XR Controller Support Step 5: Navigation.

Mostly requesting smaller changes (coding style, comments, etc.). The whole XR system is already hard to get into, so I'm pushing quite a bit for making things well readable & maintainable at the lower levels. Another dev should be able to jump in and do at least smaller fixes/tweaks without too much trouble.

Oct 16 2021, 1:42 PM · Virtual Reality
Bibin Abraham Jacob (JacAbraham) added a member for Virtual Reality: Bibin Abraham Jacob (JacAbraham).
Oct 16 2021, 7:59 AM

Oct 15 2021

Clément Foucault (fclem) changed the status of T90209: Eevee Alpha Blend artifacts in lookdev and VR with Viewport Denoising enabled from Needs Information from Developers to Confirmed.

@Robert Bragg (rib) You are correct in your assumptions. Thats why most game / realtime engines turned to forward shading pipelines and MSAA for their AA solution (because it is very fast). This is not practical for us as you would loose most of the effects of eevee.

Oct 15 2021, 7:37 PM · Virtual Reality, BF Blender

Oct 12 2021

Julian Eisel (Severin) accepted D11271: VR Add-on: Default Actions, Controller Gizmos.
Oct 12 2021, 11:47 AM · Virtual Reality