In the first part of this series, I provided a detailed walkthrough of setting up a project using new Unity XR SDK pipeline for HoloLens 2 development and how to integrate it with the HoloLens 2 toolchain.

In this post, I will continue building on that project by showing you how to set up an HoloLens 2 scene in Unity using the Mixed Reality Toolkit. Finally I will show you how to set up and configure one of the MRTK built-in example projects.

Configuring a scene for the HoloLens 2

Use the project from the previous post in which you configured the project settings, player settings, build settings, and imported the MRTK to use with the new XR SDK pipeline.

Setting up a new scene for the HoloLens only takes a few steps.

1. From the Mixed Reality Toolkit item in the toolbar, select Add to Scene and Configure.

2. This will add the needed MRTK components to your current scene. Verify in the Hierarchy window that your new scene includes the following game objects: Directional Light, MixedReality Toolkit, and Mixed Reality Playspace. Select the MixedReality Toolkit game object.

3. In the Inspector pane for the MixedReality Toolkit game object, there is a dropdown of various configuration profiles. The naming of these profiles is confusing. It is extremely important that you switch from the default DefaultMixedRealityToolkitConfigurationProfile to DefaultXRSDKConfigurationProfile. Without making this change, even basic head tracking will not work for you.

4. Next, click on the Clone button and choose a new pertinent name for your application’s configuration (if  you can’t think of one, then the old standby MyConfigurationProfile will work in a pinch – you can go back and change it later).

5. The MRTK configuration files are set up in a daisy chain fashion, with config files referencing other config files, all of which can be copied and customized. Go ahead and clone the DefaultXRSDKCameraProfile. Rename it to something your can remember (MyCameraProfile will work in a pinch).

6. Save all your changes with Ctrl+S.

Opening an MRTK example project

Being able to test out a working HoloLens 2 application can be instructive. If you followed along with the previous post, you should already have the example scenes imported into your project.

If you missed this step, you can open up the Mixed Reality Feature Tool now and import the Mixed Reality Toolkit Examples.

1. After importing, the MRTK examples are still compressed in a package. In the Project pane, navigate to the Packages folder. Then right click on Projects > Mixed Reality Toolkit Examples and click on View in Package Manager in the context menu.

2. In Package Manager, select the Mixed Reality Toolkit Examples package. This will list all of the compressed MRTK demos to the right.

3. Click on the Import into Project button next to the Demos – HandTracking sample to decompress it.

4. There are a few ways to open your scene. I will demonstrate one of them.Type Ctrl+O on your keyboard (this is equivalent to selecting File | Open Scene on the toolbar). A file explorer window will open up. Navigate to the Assets folder for your Unity project. You will find the HandInteractionExample scene under a folder called Samples. Select it.

The interaction sample is one of the nicest ways to try out the hand tracking capabilities of the HoloLens 2. It still needs to be configured to work with the new XR SDK pipeline, however.

Configuring an MRTK demo scene

Before deploying this scene to a HoloLens 2, you must first configure the scene to use the XR SDK pipeline.

1. Select the MixedRealityToolkit game object in the Hierarchy pane. In the Inspector pane, switch from the default configuration profile to the one you created earlier when you were creating your own scene.

2. Ctrl+S to save your changes.

Preparing the MRTK demo scene for deployment

1. Open up the Build Settings window either by typing Ctrl+Shift+B or by selecting File | Build Settings from the project toolbar.

2. Click on the Add Open Scenes button to add the example scene.

3. Ctrl+S to save your build settings.

4. One of the nicest features of the Mixed Reality Toolkit, going back all the way to the original HoloLens Toolkit it developed out of, is the build feature. Building a project for HoloLens has several involved steps which include building a Visual Studio project for UWP, compiling the UWP project into a Windows Store assembly, and finally deploying the appx to either a HoloLens 2 device or to an emulator. The MRTK build window lets you do all of this from inside the Unity IDE.

From the Mixed Reality Toolkit menu on the toolbar, select Utilities | Build Window. From here, you can build and deploy your application. Alternatively, you can build your appx file and deploy it from the device portal, which is what I usually do.

Summary

This post completes the walkthrough showing you how to set up and then build a HoloLens 2 application in Unity using the new XR SDK pipeline. It is specifically intended for developers who have developed for the HoloLens before and may have missed a few tool cycles, but should be complete enough to also help developers new to spatial computing get quickly up to speed on the praxis of HoloLens 2 development in Unity.

The HoloLens 2 toolchain is under continuous development. In addition, the development of the Mixed Reality Toolkit (MRTK), the developer kit for HoloLens, must be synced with the continuous development of the Unity Game Engine. For these reasons, it is necessary for developers to be careful about using the correct versions of each tool in order to avoid complications.

As of February, 2021, I’m  doing HoloLens development with Visual Studio 2019 v16.5.5, Unity Hub 2.4.2, Unity 2019.4.20f, Mixed Reality Feature Tool 1.0.2102 beta,  MRTK 2.5.4 + MRTK 2.6.0-preview, and Windows 10 Pro 10.0.18363.

MRTK, MR Feature Tool beta,  and the new Unity XR SDK pipeline

The most accurate documentation on getting started with HoloLens 2 development, at this time, is found in the MRTK github repository and on Microsoft’s MR Feature Tool landing page. There are nevertheless some gaps between these two docs that it would be helpful to fill in.

The new Unity XR SDK pipeline is a universal plugin framework that replaces something now known as the legacy Unity XR pipeline. Because of all the moving parts, many people have trouble getting the new pipeline working correctly, especially if they have not kept up with the changes for a while or if they are new to the HoloLens 2.

You will want to download Unity Hub if you don’t have it already.  Unity Hub will help you manage multiple versions of Unity on your development machine. It is fairly common to switch between different versions of Unity if you are working in VR and Mixed Reality as you go back and forth between older versions for maintenance and pursue newer versions for the latest features. As a rule, never upgrade the Unity version of an existing project if things are working well enough.

Create a new unity project

Use the Unity Hub Installs tab to get the latest version of Unity 2019.4, which you will need in order to successfully work with the latest MRTK. Later versions of Unity will not currently work for developing HoloLens 2 applications with the MRTK 2.5.4. Versions earlier than Unity 2018.4 also will not work well.

Some of the documentation mentions using Unity 2020.2 with OpenXR. This is something totally different. Just ignore it.

Start by creating a new Unity 2019.4 project.

When you do this from Unity Hub, you can use the pull down menu to select from any of the versions installed on your development computer.

When your Unity app has been created, open the Unity Package Manager from the Windows menu.

In Package Manager, select the Windows XR Plugin in the left panel. Then click the Install button in the lower left corner of the main panel.

This will also install the following components automatically: XR Interaction Subsystems, XR Legacy Input Helpers, and XR Plugin Management.

*** Notice that the component called Windows Mixed Reality is not installed. Don’t touch it. This is a left over from the legacy Unity XR pipeline and will eventually be removed from Unity. You will use a different method to get the MRTK into your project. ***

Configure project settings for HoloLens 2

You should now edit your project settings. Open the project settings panel by selecting Edit | Project Settings in the menu bar.

1. Select XR Plug-in Management in the left-hand pane and then, in the main window, check off Windows Mixed Reality to select it as your plug-in provider. This lets the new XR SDK pipeline work with your MRTK libraries (which you will import in the next section).

2. Under XR Plug-in Management, select the Windows Mixed Reality entry in the left-hand pane. Make sure Depth Buffer 16 Bit is selected as your Depth Buffer Format and that Shared Depth Buffer is checked off. This will improve the performance of your spatial computing application.

3. Select Quality in the left-hand pane. To improve performance, click the down arrow under the Windows logo to set your default quality setting to Low for your spatial computing application.

4. While you are configuring your project settings, you might as well also import TextMesh Pro. Select TextMesh Pro in the left-hand pane and click on the Import TMP Essentials button in the main window. TMP will be useful for drawing text objects in your spatial computing application.

5. Select Player  in the right-hand pane to edit your player settings. Change the Package name entry to something relevant for your project. (The project name is how the HoloLens identifies your application. If you are quickly prototyping and deploying projects and forget to change the package name, you will get an obscure message about saying your Template3D package is already installed. This is just the default package name on all new Unity projects.)

You are now ready to import the Mixed Reality Toolkit.

Retrieve MRTK components with MR Feature Tool

The HoloLens team has created a new tool called the Mixed Reality Feature Tool for Unity to help you acquire and deploy the correct version of the MRTK to your project.

1. After downloading the feature tool, you can go into settings and check off the Include preview releases box in order to get the 2.6.0-preview.20210220.4 build of MRTK. Alternatively, you can use MRTK version 2.5.4 if you are uncomfortable with using a preview build.

2. Follow the wizard steps to select and download the MRTK features you are interested in. At a minimum, I’d recommend selecting the Toolkit Foundation, Toolkit Extensions, Toolkit Tools, Toolkit Standard Assets, and Toolkit Examples.

3. On the Import features screen, select the path to your Unity project. Validate that everything is correct before importing the selected features into your Unity HoloLens 2 project.

4. Click on the Import button.

Configure build settings for HoloLens 2

As the MRTK libraries are added to your project, your Unity project may hang while it is being refreshed if you still have it open (which is very okay).

1. When the refresh is done, the Mixed Reality Toolkit item will appear your Unity menu bar.

2. At the same time, an MRTK configuration window will also pop up in the Unity IDE. Accept the suggested project setting changes by selecting Apply.

3. Click on File | Build Settings… to open the Build Settings window. Select Universal Windows Platform in the left pane.

4. Upon selecting Universal Windows Platform as your build target, you will be prompted to switch platforms. Click on the Switch Platform button to confirm that you will be building an application for the UWP platform. This will initiate a series of updates to the project that may freeze the IDE as the project is refreshed.

5. After your switch to the Universal Windows Platform, MRTK may prompt you to make additional changes to your project. Apply the recommended changes.

6. For a release deployment, you will want the following build settings:

• Target Device should be HoloLens.
• Architecture is ARM64. This is the processor used in the HoloLens 2.
• Build Type is D3D Project. Any other build type will project a standard 2D UWP window into the HoloLens.
• Build configuration is Master for a release build, rather than a Release, as odd as that seems. This is the most lightweight build you can deploy to your device, and consequently the best performing.
• Compression Method should be LZ4HC for release builds. This takes a lot longer to compile, but is the most efficient in a deployed application.

You can close the Build Settings window.

Summary

This walkthrough is intended to get you through the initial steps for integrating the new Unity XR SDK pipeline with the Mixed Reality Toolkit. For completeness, I will walk you through setting up and deploying a HoloLens scene in the next post.