This documentation walks you through the features and structure of the passthrough relighting (PTRL) sample. It shows you how your XR Unreal project can be set up such that virtual lights and shadows in your Unreal scene can interact with scene/space anchor objects such as the floor, walls, desks, and so on. This helps your virtual content to blend with the real-world environment seen displayed in passthrough.

Requirements

Follow the MR Utility Kit getting started guide.

Sample app description

Once the application starts, in the menu, go to the Maps tab, select PTRL from the dropdown and click Load Map. You will see lights and shadows interacting with scene objects. You control the movements of the character, Oppy, which projects shadows and highlights.

Those lighting effects are hidden by the Environment Depth, in order to be occluded by real world objects similarly to the rest of the virtual scene.

You can control Oppy’s movement using the left or right controller thumbstick. Moving the thumbstick up moves Oppy in a forward direction relative to the headset.

To make Oppy jump, press the A, X, or the grip buttons. Jumping while moving is also possible, and encouraged in order to jump on scene objects like desks. Holding the jump button makes Oppy jump higher.

A control panel is attached to the left controller and is interactable with a ray shooting from the right controller, using the right trigger to select. Through the panel, you can adjust the render parameters for highlights and shadows, toggle between scene objects and the scene mesh, and change passthrough brightness and depth check.

Showing the control panel and its adjustable options.

Scene overview

The experience is all in a single scene in the Content/Maps folder. The main elements of the scene are:

The character BP_Oppy, containing:

A MRUKBlobShadow component.

A flame floating above its head. It has a point light attached to it, flickering and not casting shadows.

A directional light casting shadows.

A standard VRPawn actor, responsible for moving the cameras and controllers, Oppy inputs, rendering and interacting with the control panel, and containing a PassthroughLayer component.

The OculusXRSceneActor, with a OculusXRSceneGlobalMesh component, gets information from user defined scene anchors and scene mesh, and applies the highlights material to those.

The MRUKLightDispatcher, which is needed for sending the lights information to the highlights material.

Project structure

The core components, MRUKBlobShadow and MRUKLightDispatcher, can be found in the source folder of the MRUK plugin.The Oppy character can be found in the Character folder. The Maps folder contains the PTRL.map with the actual sample.

Adding Passthrough relighting to an MR project

You need to add the MRUtilityKit plugin to your project. Then, add the MRUKLightDispatcher to your scene and assign the MRUKBlobShadow to the game objects casting shadows on Passthrough. To see the highlights, add the highlight material (MI_HighlightsTranslucent) to a mesh.

Add the highlight material to a mesh to see the highlight effects.

In order for this to have an effect in runtime, the user will need to set up their space before running the app. They will need to scan the environment, and define room and scene objects, (for example, a desk).

Multiple point light support

In order to show highlights from more than one point light source, make sure that MPC_Highlights has its Vector Parameters array size at least three times the amount of point lights in your scene, as each point light is represented by three vectors in the material parameter collection.

Highlights and shadows shader

The Highlights material allows objects that use it to collect highlights from point light sources while still being transparent. If objects using this material are aligned with real-life objects, it looks like virtual lights interact with real objects.

The material Highlights achieves this through three steps:

First, the data of the point lights in the scene is sent to MPC_Highlights via MRUKLightDispatcher. Each point light is represented by 3 vectors: one for the light’s position, one for the light’s parameters (AttenuationRadius, LightBrightness, LightFalloffExponent, and UseInverseSquaredFallOff0) and one for the light’s color. Finally, the amount of lights is sent in the scalar parameter TotalLights.

Second, the highlights material definition uses a custom node responsible for calculating the combined highlight for the current pixel. It goes through each light’s data in MPC_Highlights and calculates the highlight for each.

Finally, the depth check is done by comparing the game’s depth with the Environment Depth node, clearing the pixels where the difference between the two is too high.

The depth check is done by comparing the game’s depth with the Environment Depth node, clearing the pixels where the difference between the two is too high.

Note: If you’re using the stock engine, the “EnvironmentDepth” node will not be available. You need to use the Meta fork of Unreal Engine.

Blob shadow

A common alternative to real-time shadows are blob shadows, they are simple blots of color that only take the general shape of the actor into account and are more performant.

To calculate the projected 2D shape of an actor, the MRUKBlobShadow component uses the actor bounds. 2D sizes are used to set the area and rotation of the plane. Check out UMRUKBlobShadowComponent::ComputeOwner2DBounds to see how this is done.

The MRUKBlobShadow component enables shadows (check Oppy actor in the sample scene). It projects a blob shadow on top of the scene meshes so it blends with the material Highlights.

Besides the plane mesh, the BlobShadow object uses a material located in the PTRL folder. Those are only assets required for the BlobShadow component.

Troubleshooting

The M_Highlights material does not work

This is probably because you’re using it with the stock Unreal Engine and the node ScreenPosition is not connected to anything. Make sure to use the Meta fork of Unreal Engine.

Did you find this page helpful?