What the heck just happened to HoloLens?

January 26, 2023January 26, 2023 James Ashley5 Comments

Last Wednesday, on January 18, Microsoft laid off 10,000 employees, or 5% of its workforce. That same day, Bloomberg reported that some of the cuts were targeting the HoloLens hardware team, which had just been moved under Panos Panay in June, 2022 while the software team had been placed under a different organization and the previous head of the HoloLens combined group, Alex Kipman, was maneuvered out of the company.

Over last week, further announcements on social media indicated how thoroughly Microsoft’s MR and VR commitments had been deracinated. It turned out that the entire MRTK engineering group, which created and maintained the SDK and tools for developing on the HoloLens, had been laid off. This was problematic, because if you remove the team supporting the tools people use to develop on the HoloLens, people will stop developing for the HoloLens. It was hard to see this as an accidental by product of cost cutting moves and easy to see it as part of a larger strategic shift at Microsoft.

Microsoft also laid off the employees of AltSpaceVR, central to its Metaverse ambitions, announcing that the site would be shut down on March 10. Microsoft had acquired AtlSpaceVR in 2017.

Various announcements claimed that the work that had been done at AltSpaceVR would be taken up by the Microsoft Mesh team, which is under the Microsoft Teams organization. At the same time, however, there were rumors going around that up to 80% of the Mesh team had also gotten the axe, including some of their product community evangelists.

So to sum up, Microsoft cut deep into HoloLens hardware, the MRTK team, the AltSpaceVR business, and its Mesh team. In addition, they pushed out the organizational exec of the HoloLens team in the summer of 2022 and split his people between two other divisions where they no longer had his protection as the head of a Microsoft fiefdom. The HoloLens – and in turn Microsoft’s investment in Mixed Reality and the Metaverse – was probably already dead at that point. There was hope from the HoloLens developer community that they were simply pausing to see how Apple’s MR strategy would pan out. If there was some possibility that a successful marketing push by Apple would encourage Microsoft to move forward with their headsets, those hopes are now dashed. The cuts have been too deep. There will never be a HoloLens 3.

On the bright side…

Microsoft was first out of the gate to set a standard for what high-end augmented reality headsets would be like – even adapting an old unused term, “mixed reality”, to emphasize the difference between phone based AR and what they were doing.

Previously they had done a similar thing with the Kinect by creating a new market for low-cost 3D depth sensors, which in turn created an ecosystem of alternative vendors, which in turn created a supply chain for 3D components as well as competing technology for 3D capture such as photogrammetry and computer vision, which finally led to the world sensing components today that make untethered VR and self-driving cars possible.

With the HoloLens they helped forge a developer community, changed the priorities of 3D game engine companies like Unity and Unreal, provided competition for up and coming MR vendors like Florida-based Magic Leap, tested out the limits of mixed reality scenarios and proofed out the appetite for passthrough AR, used in the Meta Quest Pro, HTC Vive Elite, and the upcoming Apple MR device, while we all wait on advances in waveguide technology and its alternatives.

In a large sense, Microsoft, Alex Kipman, the thousands of people who worked on the Microsoft HoloLens team as well as the thousands of developers who helped to build out MR experiences for enterprise and commercial products, accomplished their mission. They pushed the tech forward.

The truth is, Microsoft has often been extremely good at helping to build out promising technology but has rarely been good at sticking with technology to the viability phase. The biggest example is being early to tablets and phones, realizing they were waaay too early, and then trying to pick them up again after other tech giants had already cornered the market on these devices. With some notable exceptions, like the Xbox, hardware just isn’t Microsoft’s game and they aren’t comfortable with it.

Which is okay since thanks to their work, Meta, Magicleap, HTC, NVidia and others are stepping into the gap that Microsoft is leaving behind. Advances in MR and VR (“metaverse”) tech and experience design will be made at those companies. The laid off HoloLens workers will be snatched up by these other companies and the developer community will adapt to building for these other hardware devices.

While more work needs to be done, the MRTK in both its version 2 and version 3 flavors, provide a good way for MR developers and companies invested in mixed reality to pivot and port to new devices.

Pivot and port

There are several constructive steps that can be taken over the next few months to continue to push MR forward. The first is renaming and refocusing the HoloDevelopers slack community. This has been the most successful and lively meeting spaces over the past six years for sharing mixed reality news, knowledge and gossip. Thank goodness it never got moved over the Teams, as was once proposed. But it does need to be renamed, since it now covers a much broader MR ecosystem than just the Microsoft HoloLens, and it needs some financial support to enable searching of the archives for past, now hidden information, about how to get things done. No one should be re-inventing the wheel simply because we can’t search the archives.

The next thing that needs to be done is to unravel the MRTK situation. In principle the MRTK is an API layer that will target multiple devices. One of the targets of the MRTK developed over the past couple of years is OpenXR, which in turn is also an API layer that targets multiple devices. (It’s confusing, I know, and I’ve been planning to write a Foucauldian analysis of soft power exercised through API dominance for about a year to explain it.)

There are also two versions of the MRTK, v2 and v3, both of which work with OpenXR. In principle, if you have an app that sits on top of MRTK, and it targets OpenXR, then you should be able to repoint your app at another device, such as the various Meta passthrough AR devices or the Magic Leap 2 MR device, and have it mostly work.

Here are some kinks.

An OpenXR implementation requires that particular hardware device vendors create plugins that map the OpenXR API to their particular HMDs. This can be done more or less well. It can be done in its entirety or only partially. Magicleap, for instance, has a beta plugin available for the Unity implementation of OpenXR, but this still isn’t done, yet (please hurry Magicleap!).
There are platform specific features that haven’t been generalized in OpenXR. For instance, Microsoft has a World Locking system that worked with its World Anchors system to make world anchors not drift so much. But the world locking system sits outside of OpenXR.
MRTK3 hasn’t been published in anything other than preview versions. The team has been laid off a couple of weeks before the first planned release.
For this reason, not many apps are using MRTK3. Also for this reason, it is unlikely that anyone will try to port their apps from MRTK2 to MRTK3, which is an untrivial task.
Some have expressed a hope that the community will pick up the work and support of MRTK3, which was an open source project almost exclusively managed and worked on by Microsoft employees. The problem here is that this hasn’t historically happened. Open source projects are rarely community supported, but require someone to be paid to do it. When Microsoft dropped support of the early HoloLens Toolkit in 2017, it was only two independent developers, rather than a large pool of indie devs sharing the work, that did the majority of the labor involved in expanding it and rearchitecting it into MRTK2.
A re-porting strategy is vital for the MR ecosystem to thrive. Startups need to be able to show that they are not hardware dependent and can get up and running again on a new device over the next three to six months.
Additionally, there are hundreds of HoloLens apps, most not on any public store, available to be ported to alternative headset platforms. And every HMD platform currently has a strong need for more apps.
But none of this can happen without a consensus on whether the ecosystem will be adopting MRTK2 going forward or MRTK3. And it can’t happen unless there is an ongoing commitment to support the MRTK source code.
There are two aspects to the MRTK that make it vital to the ongoing progress of mixed reality. I’ve already discussed the importance of a porting strategy.

The second key feature of the MRTKs is the interaction samples. These are for all practical purposes the best and in some sense only user interfaces available in MR and VR. If you need to enter data or push a button in either mixed reality or the metaverse, these are the tools you should use. They were carefully designed, following user research, but a team at Microsoft led by Julia Schwartz. They are amazing.

But they are also in a big sense reference samples. They need further work to optimize performance and to smooth out usability on a variety of platforms.

It is possible that another company – Unity, Meta, Mgicleap, etc – could step in and develop a new set of tools with ongoing maintenance. But at this point there isn’t.

Summing up

To sum up:

The HoloLens is dead. It has been for about six months.
… but it helped to create a community as well as a device ecosystem that goes on.
The community at HoloDevelopers needs some funding and a second wind, but it has grown organically to be a central repository of knowledge about the development and design of MR apps.
The MRTKs require some hard choices and then a lot of love to make them work well across hardware platforms.
The interaction samples of the MRTKs are a national treasure and also need a lot of love from the community.
We need lots of blog posts and videos covering how to port HoloLens apps to the MQP (Oculus), HTC Elite, Magicleap, and eventually the Apple MR device. In the process we can identify the gaps and issues involved in porting and try to fix them.
Go hug a laid off Microsoft HoloLens employee if you can. I have high confidence they will all land well because they are highly skilled people in a field Microsoft is dropping in favor of generative AI (a reasonable move) but it’s still going to be a tough few months emotionally until they do.
While you are at it, maybe go join the Holodevelopers slack group and hug a non-Microsoft developer, too. They’ll all be fine, too, but its tough to see the work you’ve been doing for the past seven years suddenly drop out from under you.
Off the top of my head, here are some great ones to reach out to: Sean Ong, Joost van Schaik, Dennis Vroegop, Jason Odom, Stephen Hodgson, Simon Jackson, Vincent Guigui, Rene Schulte, Lucas Rizzotto, Andras Velvart, Sky Zhou, Huy Le, Eric Provencher, Lance Larsen, Dwayne Lamb, Charles Poole, Dino Fejzagic, and tons of others I can’t recall right away but that you will hopefully remind me of. They are all heroes.
The ride continues. Just not at Microsoft.

the New XR SDK Pipeline with HoloLens 2: Part 2

March 3, 2021March 22, 2021 James Ashley1 Comment

In the first part of this series, I provided a detailed walkthrough of setting up a project using new Unity XR SDK pipeline for HoloLens 2 development and how to integrate it with the HoloLens 2 toolchain.

In this post, I will continue building on that project by showing you how to set up an HoloLens 2 scene in Unity using the Mixed Reality Toolkit. Finally I will show you how to set up and configure one of the MRTK built-in example projects.

Configuring a scene for the HoloLens 2

Use the project from the previous post in which you configured the project settings, player settings, build settings, and imported the MRTK to use with the new XR SDK pipeline.

Setting up a new scene for the HoloLens only takes a few steps.

1. From the Mixed Reality Toolkit item in the toolbar, select Add to Scene and Configure.

2. This will add the needed MRTK components to your current scene. Verify in the Hierarchy window that your new scene includes the following game objects: Directional Light, MixedReality Toolkit, and Mixed Reality Playspace. Select the MixedReality Toolkit game object.

3. In the Inspector pane for the MixedReality Toolkit game object, there is a dropdown of various configuration profiles. The naming of these profiles is confusing. It is extremely important that you switch from the default DefaultMixedRealityToolkitConfigurationProfile to DefaultXRSDKConfigurationProfile. Without making this change, even basic head tracking will not work for you.

4. Next, click on the Clone button and choose a new pertinent name for your application’s configuration (if you can’t think of one, then the old standby MyConfigurationProfile will work in a pinch – you can go back and change it later).

5. The MRTK configuration files are set up in a daisy chain fashion, with config files referencing other config files, all of which can be copied and customized. Go ahead and clone the DefaultXRSDKCameraProfile. Rename it to something your can remember (MyCameraProfile will work in a pinch).

6. Save all your changes with Ctrl+S.

Opening an MRTK example project

Being able to test out a working HoloLens 2 application can be instructive. If you followed along with the previous post, you should already have the example scenes imported into your project.

If you missed this step, you can open up the Mixed Reality Feature Tool now and import the Mixed Reality Toolkit Examples.

1. After importing, the MRTK examples are still compressed in a package. In the Project pane, navigate to the Packages folder. Then right click on Projects > Mixed Reality Toolkit Examples and click on View in Package Manager in the context menu.

2. In Package Manager, select the Mixed Reality Toolkit Examples package. This will list all of the compressed MRTK demos to the right.

3. Click on the Import into Project button next to the Demos – HandTracking sample to decompress it.

4. There are a few ways to open your scene. I will demonstrate one of them.Type Ctrl+O on your keyboard (this is equivalent to selecting File | Open Scene on the toolbar). A file explorer window will open up. Navigate to the Assets folder for your Unity project. You will find the HandInteractionExample scene under a folder called Samples. Select it.

The interaction sample is one of the nicest ways to try out the hand tracking capabilities of the HoloLens 2. It still needs to be configured to work with the new XR SDK pipeline, however.

Configuring an MRTK demo scene

Before deploying this scene to a HoloLens 2, you must first configure the scene to use the XR SDK pipeline.

1. Select the MixedRealityToolkit game object in the Hierarchy pane. In the Inspector pane, switch from the default configuration profile to the one you created earlier when you were creating your own scene.

2. Ctrl+S to save your changes.

Preparing the MRTK demo scene for deployment

1. Open up the Build Settings window either by typing Ctrl+Shift+B or by selecting File | Build Settings from the project toolbar.

2. Click on the Add Open Scenes button to add the example scene.

3. Ctrl+S to save your build settings.

4. One of the nicest features of the Mixed Reality Toolkit, going back all the way to the original HoloLens Toolkit it developed out of, is the build feature. Building a project for HoloLens has several involved steps which include building a Visual Studio project for UWP, compiling the UWP project into a Windows Store assembly, and finally deploying the appx to either a HoloLens 2 device or to an emulator. The MRTK build window lets you do all of this from inside the Unity IDE.

From the Mixed Reality Toolkit menu on the toolbar, select Utilities | Build Window. From here, you can build and deploy your application. Alternatively, you can build your appx file and deploy it from the device portal, which is what I usually do.

Summary

This post completes the walkthrough showing you how to set up and then build a HoloLens 2 application in Unity using the new XR SDK pipeline. It is specifically intended for developers who have developed for the HoloLens before and may have missed a few tool cycles, but should be complete enough to also help developers new to spatial computing get quickly up to speed on the praxis of HoloLens 2 development in Unity.

The New XR SDK Pipeline with HoloLens 2: A Walkthrough

February 28, 2021March 22, 2021 James Ashley2 Comments

The HoloLens 2 toolchain is under continuous development. In addition, the development of the Mixed Reality Toolkit (MRTK), the developer kit for HoloLens, must be synced with the continuous development of the Unity Game Engine. For these reasons, it is necessary for developers to be careful about using the correct versions of each tool in order to avoid complications.

As of February, 2021, I’m doing HoloLens development with Visual Studio 2019 v16.5.5, Unity Hub 2.4.2, Unity 2019.4.20f, Mixed Reality Feature Tool 1.0.2102 beta, MRTK 2.5.4 + MRTK 2.6.0-preview, and Windows 10 Pro 10.0.18363.

MRTK, MR Feature Tool beta, and the new Unity XR SDK pipeline

The most accurate documentation on getting started with HoloLens 2 development, at this time, is found in the MRTK github repository and on Microsoft’s MR Feature Tool landing page. There are nevertheless some gaps between these two docs that it would be helpful to fill in.

The new Unity XR SDK pipeline is a universal plugin framework that replaces something now known as the legacy Unity XR pipeline. Because of all the moving parts, many people have trouble getting the new pipeline working correctly, especially if they have not kept up with the changes for a while or if they are new to the HoloLens 2.

You will want to download Unity Hub if you don’t have it already. Unity Hub will help you manage multiple versions of Unity on your development machine. It is fairly common to switch between different versions of Unity if you are working in VR and Mixed Reality as you go back and forth between older versions for maintenance and pursue newer versions for the latest features. As a rule, never upgrade the Unity version of an existing project if things are working well enough.

Create a new unity project

Use the Unity Hub Installs tab to get the latest version of Unity 2019.4, which you will need in order to successfully work with the latest MRTK. Later versions of Unity will not currently work for developing HoloLens 2 applications with the MRTK 2.5.4. Versions earlier than Unity 2018.4 also will not work well.

Some of the documentation mentions using Unity 2020.2 with OpenXR. This is something totally different. Just ignore it.

Start by creating a new Unity 2019.4 project.

When you do this from Unity Hub, you can use the pull down menu to select from any of the versions installed on your development computer.

When your Unity app has been created, open the Unity Package Manager from the Windows menu.

In Package Manager, select the Windows XR Plugin in the left panel. Then click the Install button in the lower left corner of the main panel.

This will also install the following components automatically: XR Interaction Subsystems, XR Legacy Input Helpers, and XR Plugin Management.

*** Notice that the component called Windows Mixed Reality is not installed. Don’t touch it. This is a left over from the legacy Unity XR pipeline and will eventually be removed from Unity. You will use a different method to get the MRTK into your project. ***

Configure project settings for HoloLens 2

You should now edit your project settings. Open the project settings panel by selecting Edit | Project Settings in the menu bar.

1. Select XR Plug-in Management in the left-hand pane and then, in the main window, check off Windows Mixed Reality to select it as your plug-in provider. This lets the new XR SDK pipeline work with your MRTK libraries (which you will import in the next section).

2. Under XR Plug-in Management, select the Windows Mixed Reality entry in the left-hand pane. Make sure Depth Buffer 16 Bit is selected as your Depth Buffer Format and that Shared Depth Buffer is checked off. This will improve the performance of your spatial computing application.

3. Select Quality in the left-hand pane. To improve performance, click the down arrow under the Windows logo to set your default quality setting to Low for your spatial computing application.

4. While you are configuring your project settings, you might as well also import TextMesh Pro. Select TextMesh Pro in the left-hand pane and click on the Import TMP Essentials button in the main window. TMP will be useful for drawing text objects in your spatial computing application.

5. Select Player in the right-hand pane to edit your player settings. Change the Package name entry to something relevant for your project. (The project name is how the HoloLens identifies your application. If you are quickly prototyping and deploying projects and forget to change the package name, you will get an obscure message about saying your Template3D package is already installed. This is just the default package name on all new Unity projects.)

You are now ready to import the Mixed Reality Toolkit.

Retrieve MRTK components with MR Feature Tool

The HoloLens team has created a new tool called the Mixed Reality Feature Tool for Unity to help you acquire and deploy the correct version of the MRTK to your project.

1. After downloading the feature tool, you can go into settings and check off the Include preview releases box in order to get the 2.6.0-preview.20210220.4 build of MRTK. Alternatively, you can use MRTK version 2.5.4 if you are uncomfortable with using a preview build.

2. Follow the wizard steps to select and download the MRTK features you are interested in. At a minimum, I’d recommend selecting the Toolkit Foundation, Toolkit Extensions, Toolkit Tools, Toolkit Standard Assets, and Toolkit Examples.

3. On the Import features screen, select the path to your Unity project. Validate that everything is correct before importing the selected features into your Unity HoloLens 2 project.

4. Click on the Import button.

Configure build settings for HoloLens 2

As the MRTK libraries are added to your project, your Unity project may hang while it is being refreshed if you still have it open (which is very okay).

1. When the refresh is done, the Mixed Reality Toolkit item will appear your Unity menu bar.

2. At the same time, an MRTK configuration window will also pop up in the Unity IDE. Accept the suggested project setting changes by selecting Apply.

3. Click on File | Build Settings… to open the Build Settings window. Select Universal Windows Platform in the left pane.

4. Upon selecting Universal Windows Platform as your build target, you will be prompted to switch platforms. Click on the Switch Platform button to confirm that you will be building an application for the UWP platform. This will initiate a series of updates to the project that may freeze the IDE as the project is refreshed.

5. After your switch to the Universal Windows Platform, MRTK may prompt you to make additional changes to your project. Apply the recommended changes.

6. For a release deployment, you will want the following build settings:

Target Device should be HoloLens.
Architecture is ARM64. This is the processor used in the HoloLens 2.
Build Type is D3D Project. Any other build type will project a standard 2D UWP window into the HoloLens.
Build configuration is Master for a release build, rather than a Release, as odd as that seems. This is the most lightweight build you can deploy to your device, and consequently the best performing.
Compression Method should be LZ4HC for release builds. This takes a lot longer to compile, but is the most efficient in a deployed application.

You can close the Build Settings window.

Summary

This walkthrough is intended to get you through the initial steps for integrating the new Unity XR SDK pipeline with the Mixed Reality Toolkit. For completeness, I will walk you through setting up and deploying a HoloLens scene in the next post.

10 Questions With Charles Poole

August 22, 2019 James Ashley

Charles Poole, the owner of IS Studios, is currently one of the most experienced mixed reality developers in the business. Like many of the other people well known for their development chops on the HoloLens and Magic Leap One, he fell into it accidentally. Through a combination of determination and blind luck, as well as the ability to pick up a new UX paradigm that requires technical acumen with both .NET and Unity, he is currently one of those rare people with 3+ years of hands-on MR design, development and project management experience. You’ll have to ask him yourself for the full story, but it basically comes down – as with so many others – to getting his hands on a very expensive device and learning to make it hum (ideally using spatial audio).

Charles is soft spoken and kind. One of the very interesting things about his background is that he is a mathematician – and so in that small subclass of software developers who actually knows math! There’s nothing nicer in the world of programming than having a friend you can hit up when you are having problems with an algorithm or with your matrix math.

What movie has left the most lasting impression on you?

Hackers, I think watching Hackers in 95/96 shaped my childhood and later choices when it came to education and what I spent my time on.

What is the earliest video game you remember playing?

Super Mario Brothers on the NES, or Sky Kid, also on NES. I remember playing it for hours just to get to the 3rd or 4th level, then watching my father get a lot further.

Who is the person who has most influenced the way you think?

Maybe Buckminster Fuller, Neal Stephenson maybe Michael Crichton. I read a lot as a child, I feel as though all the views I was exposed to through fiction and non-fiction had a big influence in how I see the world and approach problems. In general the problems seem really big, cause a lot of drama, are entertaining to read and experience, then the solution just happens to come together from a character that has the experience to pull a solution out of their ass.

When was the last time you changed your mind about something?

A big one recently, and kind of mild, was using Photon for the multiplayer aspects of my work. I was against Photon for a long time, I wanted to be in control of every aspect of what I was building. So I’d do things like make a custom socket server, write the server in Dark rift, use WebRTC. One of the most important things about freelancing is using every tool you have to accelerate development, while keeping it altogether. We had to make a decision recently about a multiplayer backend that could scale to thousands of users, but still be self hosted, and the time-frame was extremely compressed, so I revisited Photon, specifically PUN2 which had been released since the last time I had used PUN, and it felt like it had come a long way in the time since I had used it last.

Simpler and more personal – My daughter’s kindergarten teacher had been pushing for her to repeat kindergarten. I was staunchly against it, she was getting top marks, won the science fair over 5th graders, and with something she had actually done and came up with on her own, we only bought the materials. But she just wasn’t emotionally ready for the pace to get quicker in first grade, and her teacher made her excited about helping out for another year. So, we agreed to have her repeat kindergarten, because she loves to learn, and we didn’t want to make school into something she hated.

What’s a skill people assume you have but that you are terrible at?

Managing my time, I’m terrible at managing my time, I tend to get sucked into a project and neglect everything else. I would work every day from 9am – 9pm or later. I had to step back and put a rigid stop time on my day so I would spend time with my kids and not just work through their whole childhood.

What inspires you to learn?

I want to do everything myself, and push myself outside my developer comfort zone everyday. I’ll say ‘yes’ to things just for the challenge of figuring it out.

What do you need to believe in order to get through the day?

That things can only get better. I started off this dev journey making a thousand bucks a month, living in a tiny apartment with my wife and two kids. Every day, week, month feels like things have gotten better for us, at some point I want to turn around and help make other people’s lives better too.

What’s a view that you hold but can’t defend?

That anything is possible with enough hard work. I have an applied math background, and have seen sparks of insight and intuition I know I’d never have, but I still feel like I’d get there eventually if I put enough hours into it.

What will the future killer Mixed Reality app do?

Something agent based, an intelligent agent that acts as your exocortex. AI/ML is the future of Human Computer Interaction, the killer app won’t feel like an app, it will just be part of your life.

What book have you recommended the most?

Rainbow’s End by Vernor Vinge, it’s shaping up to be the most prescient book I’ve read. It was written in 2006 but the trends he wrote about are what we’re starting to see today, the nascent AR technology.

12 Questions With Simon “Darkside” Jackson

August 8, 2019August 8, 2019 James Ashley

Simon is one of the main contributors to the Microsoft MRTK framework for HoloLens and also to the XRTK framework for cross-platform mixed reality development. He is the author of several technical books on Unity. He is keeper of the flame on the Unity-UI Extensions source code.

Simon basically really intimidates me. He knows the Microsoft coding stack as well as the Unity stack, which makes him formidable. He’s currently working on extending the XRTK framework to support the Oculus Quest, which means if you have built your HoloLens or Magic Leap app on the XRTK, your app will automagically also run on the Quest thanks to Simon. That’s some seriously cool stuff.

He also happens to be a very nice person who is genuinely concerned about the well being of the people around him – which I found out the easy way over many online and in-person interactions. I’m not totally sure why he promotes himself as of the Darkside since he is clearly more of a Gray Jedi – but that’s not one of the 10 questions, so we may never know. Without further ado, here are Simon’s answers to the 10 Questions:

What movie has left the most lasting impression on you?

“The Matrix, it shows us how to stand tall, to face adversity with strength and uncover meaning in this world we call life.”

What is the earliest video game you remember playing?

“Given I have to recognise I’m getting old, my earliest game I recall was Pong on the Atari 2600. First game console our family owned. First games would be the penny shuffle machines in the arcades of old .”

Who is the person who has most influenced the way you think?

“William Shatner, for showing us how to boldly go and give us a glimpse of the world I’d like to see us aspire to.”

When was the last time you changed your mind about something?

“Whenever the wife decides something and I have no other option but to agree.”

What’s a skill people assume you have but that you are terrible at?

“Recruiters are constantly sending me offers for jobs developing in JavaScript or Java, which I’ve avoided for most of my developer life.”

What inspires you to learn?

“My life’s goal is to always learn something new each and every day, to grow and develop. If we no longer aspire to develop ourselves we cease to be.”

What do you need to believe in order to get through the day?

“I have to believe the coffee will not run out, else the world becomes a much more vicious place. I also hope to defeat ignorance, but ignorance always finds new ways to baffle me.”

What’s a view that you hold but can’t defend?

“I have long held the belief that humankind will eventually realise its insignificance and start to work towards the betterment of ourselves and the planet we live on. However, I’m proven wrong each and every day (for now). Basically, I want the world of Star Trek, not the world of Star Wars.”

What will the future killer Mixed Reality app do?

“Once mixed reality technology finally becomes affordable enough and cool enough to wear all day long, I believe the killer experience will be something that integrates with our everyday. An app/experience that will enrich the world around us, show us new sights and experiences, and offer us new ways to interact. Be it a simple experience that adds wonder to a shopping centre experience, or uses geo location whilst visiting historic sights and completely immerse us whilst learning (in stead of just reading signs as we do now).”

What book have you recommended the most?

“Snowcrash by Neal Stephenson, it opens up so many new possibilities and levitates towards the dangers of being “plugged in” too much. Giving us a sense of wonder and danger in equal measure, leading us to live in a world augmented by technology but not driven by it.”

And then Simon volunteered two more unsolicited questions:

Favourite quote?

“The definition of insanity is doing the same thing over and over again and expecting a different result.
— Albert Einstein (as well as others).”

Most used phrase?

“Because… unity.”

10 Questions with Suzanne Borders

July 31, 2019July 31, 2019 James Ashley

Suzanne is the CEO of BadVR – which IMO wins the prize for best company name and probably could easily make a top 10 list for band names, also. Suzanne’s company works with the fascinating world of data visualizations in VR and MR. She is also the recipient of one of the coveted 2019 Magic Leap grants and is a member of Magic Leap’s Independent Creator Program. I met her briefly at an MR event in Mountain View, CA in early 2019 and besides being an amazing advocate for the importance of true 3D data visualizations in spatial experiences, has successfully shown everyone how to be a leader and promoter of mixed reality in the XR world.

What movie has left the most lasting impression on you?

This one is tough! I’m a huge film buff and there have been so many movies that have deeply impacted me and altered my understanding of the universe.

That being said, I think the most impactful film I’ve ever watched is “The Holy Mountain” by Alejandro Jodorowsky. It’s such an explosion of creativity, a surrealistic fever dream that functions on so many levels as a commentary on the human desire to seek truth and enlightenment. Jodorowsky is unlike any other filmmaker out there, a true magician that makes film into high art without losing the ability to make impactful statements about the universal human condition. Any of his films could really be considered my favorite but “The Holy Mountain’ in particular speaks to me the most because it best captures the hero’s journey; our collective desire to seek something greater from life than what we’re given. A lot of surrealistic film is just weird for the sake of being weird and therefore loses impact because it doesn’t use the symbolism of surrealism to make any sort of deeper statement. Jodorowsky is a surrealist in the best sense of the term – all his bizarre unexpected images convey meaning and activate archetypical feelings, drives, and desires in his audience. He’s a master of the subconscious and knows how to access and wield communicative power in this area. Because of this, he’s my creative hero and I look to his work often for inspiration, especially when attempting to craft products that have the ability to touch user’s subconscious. I think this is key when unlocking broad market appeal for products or film or art in general. To really touch and impact a wide audience the experience, the artist or creator must touch on, and involve, a universal archetype. Jodorowsky’s films taught me this lesson and showed me how to execute on it. I want to give a big shout-out and thank you to my filmmaker friend Ryan, who introduced me to them. He, in many ways, has fundamentally changed how I approach any creative challenge by showing me Jodo’s work.

Beyond “The Holy Mountain,” I’m a big fan of “Belladonna of Sadness” (you will not find a more beautifully animated film ever), “Apocalypse Now” (Brando as Kurtz and his monologue at the end talking about the clarity of evil is a perennial favorite; combining Conrad’s “The Heart of Darkness” with the Vietnam war was a stroke of pure genius), “Funeral Parade of Roses” (a Japanese film that sets the ancient story of Oedipus into the transgender alternative subculture in 1960s Japan; I love it for its ability to utilize archetypical images and stories in an unexpected and creative way), “Hiroshima Mon Amour” (any media by Marguerite Duras is an automatic favorite), and “Last Tango in Paris” (I adore Brando, he’s an absolute legend, and this film touches on so many truths of the human existence, our longing for connection, the power of anonymity, my own personal life makes this film more powerful to me than it will to many, but none the less I adore it). And of course, the visual style and occult symbolism of Dario Argento’s films is a forever favorite (“Suspiria” being the pinnacle of Argento’s work IMO).

Lastly, Fellini’s “8 1/2” was the first film I watched as a child that really unlocked for me the power of cinema and storytelling. Prior to watching it, I had dismissed film as some inferior commercial medium. I saw it as cheap mindless entertainment for the masses without substance or meaning. For me at that time, my understanding of film was limited to boring and poorly made summer blockbusters. I remember clearly popping in the 8 1/2 VHS tape at age 17 without any expectation, just another mindless story to pass the long summer hours of adolescence. But the story that jumped out from the screen – starting with Fellini’s infamous opening dream sequence – absolutely captivated me. I found myself profoundly touched at the end of the film, crying even, and realized that I had been changed forever for having watched it. The message of the film – our flawed desire for human connection and all the broken and dysfunctional ways we pursue it – resonated with me at such a level that I have, decades later, never forgotten that moment. From that point on, I considered film and storytelling a high art that held the potential to change the world. Of course, not all film or stories rise to this potential and I’ve continued to be disappointed by mainstream commercial film in such a major way that I don’t even engage with it anymore. But 8 1/2 made me realize the potential of film as a medium for spiritual transformation. It showed me the power of storytelling had to bring humanity together and demonstrated the medium’s ability to hold up to the audience a mirror of themselves, helping them pursue a deeper understanding of both themselves the world around them.

Obviously, I adore film. It is one of my biggest sources of creative inspiration for all my technical work. I love immersive tech because one builds experiences, not screens. MR holds the same potential to affect deep spiritual change and transformation in users and that interests me immensely.

What is the earliest video game you remember playing?

LOOM! I remember playing it on the first computer my father bought for our family, when I was 6 or 7 years old. I remember spending hours and hours sitting in front of the computer playing, captivated by the beautiful game art. LucasFilm games are the best, but in particular Loom really did it for me. I loved (and still love) that the primary way Bobbin Threadbare (main character) interacted with his world was through music and sound. Such an original and creative idea!

Plus, you could cast spells to literally rip apart the fabric of existence, calling forth the lord of the dead, ripping open cemeteries to speak to the souls of the deceased. You could exist beyond space and time and your character could visit this beautiful lake floating in the void, populated by swans who spoke to you in parables of truth. As a goth kid and a lover of poetry, this was beyond transformative for me. I wanted to live in Loom! Additionally, the game came with this amazing backstory about a world full of guilds and weavers of destiny. I used to listen to the backstory tape, complete with a dramatic reenactment, and pretend I was Bobbin Threadbare. Loom will forever be my favorite game of all time.

Myst is a very close second!

Who is the person who has most influenced the way you think?

This is a difficult one – there have been so many amazing mentors in my life and each one of them has taught me something important, about myself, about my experience of the world.

As mentioned, Jodorowsky has been a major influence on me and all that I create. I’ve followed him around the world and I’ve actually met him in real life. I was fortunate enough to have him read my tarot in Paris and that reading truly changed my life. I won’t go into details because it was a deeply personal reading, but it transformed me without doubt. I also was lucky enough to meet him again at the Egyptian Theater in Los Angeles and at this event he dropped many nuggets of wisdom as well.

I’ve also learned a lot from the coterie or filmmaker friends that I’ve developed here in Los Angeles. The one in particular who introduce me to Jodorowsky has taught me a lot about the creative journey. He’s taught me how to dive into my creative subconscious to identify those valuable universal, broadly resonate true ideas. I’ve always been fascinated with the ability to broadly affect so many different types of people with one single idea and I wanted to translate that to my products. When you talk to someone who wrote or directed a hugely successful film, you find they have this ability to take a concept and distill it down into its most basic form. However, instead of that process being reductive or simplistic, you find that this distillation strengthens the idea and makes it more crystalline and clear and most importantly, universally accessible. The ability to take complex, nuanced, ideas and make them resonate with the broadest audience possible is one that I value highly. I’m very glad to have had a group of people who’ve helped teach me this skill. Regardless of the difference in our industries.

When was the last time you changed your mind about something?

Whew boy, I change my mind all the time, constantly, on a second to second basis! I’m always ingesting data about my world, through experiences, books, travel, websites, music, films, poems, and products. Even subconsciously, my mind is always picking up on new data about my world, which then changes my understanding of the universe. Plus, I believe everything constantly changes, so I have to keep pace with this change and adjust my thoughts and theories to mesh with the latest information.

A system that runs off absolutes and stasis is brittle and bound for failure. Only by being nimble and changeable can any system truly be strong and resilient. As such, I agree very much with Nassim Nicholas Taleb and his concept of anti-fragility. Anti-fragility involves growth through stress and I’d like to think all of my internal world models fall into this category by being responsive in real time to new data that stresses their limits, structures, and boundaries.

What’s a skill people assume you have but that you are terrible at?

Everyone assumes I’m skilled at math because my company works with data. But I’m actually numerically dyslexic (yes, it’s a real thing) and numbers have always been a real struggle for me. That’s one of the many inspirations for BadVR – my desire to work with data but my lack of technical skill with which to do so. I am in many ways the non-technical person for whom my product is built; I am my user. This gives me the power and the passion to build and also gives me the empathy needed to deliver an effective product that makes data accessible to everyone.

Of course, being acutely aware of this shortcoming I’ve assembled a team of very highly talented mathematical geniuses that augment my own weaknesses. So, to allay any question about my company’s ability to deliver a highly technical product, I want to underscore the idea that my company is not solely comprised of me. The heart and soul of BadVR is our team, and they are deeply capable in all the ways that I am not. That variety of skills and talent is what makes us powerful. We all balance each other’s weaknesses and strengths and, in doing so, create something better than any of us could ever achieve independently.

What inspires you to learn?

I don’t need inspiration for this! I’m endlessly curious about everything, all the time. I never learned to stopped asking “why?” Learning is my default state of being. Anytime I see anything, or experience anything, it inspires me to ask more questions, to dig deeper, to understand further. My google search history is full of things like “how did dinosaurs procreate? What is dirt? Why is dark meat dark?” I just wonder and google and learn all the time. Every experience is an impetus for learning; a reason to dive into the whys, hows, and whats of yet another line of inquiry.

What do you need to believe in order to get through the day?

I have to believe that life doesn’t end with death. That I will again see the people I love who I’ve lost. If there isn’t an afterlife or there isn’t an alternate timeline we’re we meet again, I can’t continue. I’ve lost too many loved ones to be able to function without the belief that I will see them again. It goes without saying then that I believe in reincarnation, in the broadest sense. I strongly believe that the people we love never leave us and that in some way we end up back together. It’s not an evidence-based belief – besides anecdotal evidence anyways – but I must believe it. I do believe it. I will always believe it. Other, the loneliness is crushing, overwhelming; the feeling akin to being forever a planetary stranger at the very end of the world.

What’s a view that you hold but can’t defend?

I have plenty of beliefs that don’t have scientific, evidence-based support. I can always defend every belief I have if you allow anecdotal evidence or emotional appeals. Some examples include my belief in the tarot, in astrology, in dream work, psychic powers, aliens, the collective subconscious, Bigfoot, the Missouri Skunk Ape, and ghosts. I’d be more than happy to argue their existence on an emotional and anecdotal level with anyone. But science of course doesn’t support or embrace such parapsychology and cryptozoology. This doesn’t stop me from believing, though. Many of the most important questions in life cannot be answered by science. I think the scientific method is important for lesser questions but for the big questions of life like “Why are we here? What is our purpose? What is the meaning of life?” — science fails. I’m more interested in the answers offered by faith and spirituality than I am in the answers offered by science, for these sorts of questions. In the face of the eternal, science can seem so small and pedantic. But of course, for the mundane it is very important.

What will the future killer Mixed Reality app do?

Visualize data and allow for immersive analysis! Data is the killer app for mixed reality. I firmly believe that, and I fully believe my company, BadVR, will be the industry standard tool for working with data immersively. I may be biased as BadVR is my company, but hey that’s what I believe! Our unique approach, mixing art with logic, the abstract with the concrete, is exactly the way this product needs to be approached. In the future, everyone will be able to easily see and interact with incredibly large, abstract and geospatial datasets with ease. We will think of data as an oracle; a source of truth. It’s important that everyone be able to access such a powerful product, which is a major focus of BadVR – universal accessibility.

What book have you recommended the most?

“The Panic Fables” by Alejandro Jodorowsky. A book of spiritual comics that delivers small truths via 1-page comics. It’s an easy entry point into the Jodo-sphere!

“Narcopolis” by Jeet Thayil. One of my all-time favorite passages can be found in this novel. It’s about a large cast of characters who frequent an opium den in Bombay (before it became Mumbai). Thayil is one of the few writers who can write prose that reads like poetry. I am a forever a huge fan!

“The Hour of the Star” by Clarice Lispector. She deconstructs language and storytelling to deliver a narrative about a poor Brazilian girl and her search for meaning and transcendence in a world that doesn’t want or even see her. It is a visceral gut-punch of truth. Anything by Lispector is wonderful, but this story in particular is my favorite.

I will leave you with a quote from Lispector:

“I do not know much. But there are certain advantages in not knowing. Like virgin territory, the mind is free of preconceptions. Everything I do not know forms the greater part of me. And with this I understand everything. The things I do not know constitute my truth.”

Magic Leap One vs HoloLens v1 Comparison

October 8, 2018October 15, 2018 James Ashley14 Comments

I’m currently sitting in my room at the L.A. Grand Hotel waiting for the L.E.A.P. conference to start. I’ve been holding off on this comparison post because I had promised Dennis Vroegop I would give it first as a talk at the Techorama Netherlands conference – which I did last week. I will do a feature comparison based on publicly available information, then highlight features unique to the Magic Leap, and then distinguish subtle but important differences that only become apparent from spending months with these devices at the developer level. Finally I want to point out design improvements in the Magic Leap that are so good for Mixed Reality that I predict they will be incorporated into the next version of HoloLens.

Keep in mind that this is a comparison of two different generations of devices. The Magic Leap One is coming out two years after the HoloLens and would be expected to be better. At the same time, the HoloLens v2 is being released some time in 2019 and can be expected to be better still.

1. Field of View

In raw numbers, the field of view of the Magic Leap One is approximately 25% better than the HoloLens. The HoloLens field of view is estimated to be about 29-30 degrees wide and 17 degrees high. The Magic Leap One is 40 degrees wide by 30 degrees high. There is a corresponding difference in resolution, with the HoloLens offering 1268 by 720 per eye and the Magic Leap One providing 1280 by 960 per eye.

The Magic Leap One uses the same wave guide display technology that the HoloLens does, however, so how did they pump up the FOV? First, the ML1 has a more powerful battery than the HoloLens does, and it’s often been claimed by Microsoft that FOV is largely dependent on the power of the projection. This is probably offset, though, by the fact that the ML1 is using more power to project in two planes instead of only one like the HoloLens does (with 6 Waveguide layers compared to 4 in the HoloLens).

Another trick is that the waveguides in the Magic Leap are closer to the wearer’s eyes than they are in the HoloLens. As a consequence, you can wear glasses underneath the HoloLens while you cannot do so comfortably under the Magic Leap device.

In addition to this, Jasper Brekelmans and Dennis Vroegop suggested over coffees along the Amstel River (in a conversation about David Copperfield) that because one’s peripheral vision is closed off in the ML1, the perceived FOV may be even larger than the actual. The theory behind this is that, due to the widespread use of glasses, we have become used to not paying attention to our peripheral vision so much and consequently are comfortable with this tunneling of our vision.

Blocking off the peripheral field of view might cause issues in certain industrial settings, but the general effect is that what you can see as a proportion of your overall FOV is much larger in the ML1 than it is in the HoloLens. Or another way of putting this is that the empty areas of your FOV, as a proportion of your available FOV, is much smaller than it is in the HoloLens.

On top of this, the aspect ratio of the FOV in the ML1 is much taller than in the HoloLens, which may end up doing a better job at accommodating vertical seccaddic movements of the eyes.

Because of the narrower gap between the device and the wearer’s eyes, the Magic Leap can’t accommodate glasses as the Hololens can. To compensate, Magic Leap is developing relationships with online eyeglass manufacturers to provide prescription inserts that can be placed in front of the waveguides and magnetically lock into place. There’s some controversy over whether this is a good or a bad thing. Some developers have expressed concern that this will make demoing Magic Leap at events more difficult than demoing HoloLens, since those with poor vision will either not be able to participate or, alternatively, we will be forced to carry around a large suitcase of prescription inserts to every event.

On the other hand, when I think of what MR will be like in the future, I tend to think of them resembling real glasses (and not electronic contacts, which simply scare me). When they reach the size and ubiquity of modern glasses, it will make sense for each person to have their own personalized device with their appropriate prescription. Magic Leap is on the right track in this case. It’s just in the intervening period that we have to figure out how to share our limited, expensive devices with others.

	HoloLens v1	Magic Leap One
Price	$3000 – $5000	$2300
OS	Windows	Android variant
Field of View	~30 deg x 17 deg	40 deg x 30 deg
Resolution	1268 x 720 per eye	1280 x 960 per eye
Depth Sensor	Time of Flight	Time of Flight
Display Type	Wave Guide	Wave Guide
Hand Gestures Recognized	2	9
Underlying comic book technology	Light Engines	Light Fields
Controller	Click	6 DOF
Hand tracking	limited	fingers (3 joints each)
Processing unit	Above nose	Light pack
Audio	Spatial Sound	Spatial Sound

2. Hardware Specs (It’s all about the battery)

HoloLens v1	Magic Leap One
Intel Atom x5-Z8100 1.04 GHz Intel Airmont (14nm) 4 Logical Processors 64-bit capable	NVIDIA® Tegra X2 SOC 2 Denver 2.0 64-bit cores + 4 ARM Cortex A57 64-bit cores (2 A57’s and 1 Denver accessible to applications)
8086h (Intel)	GPU. NVIDIA Pascal™, 256 CUDA cores; Graphic APIs: OpenGL 4.5, Vulkan, OpenGL ES 3.3+
2GB RAM	8GB RAM
64GB Storage	128GB Storage

The Magic Leap One is overall a much beefier machine than the current HoloLens. While both the HoloLens and the Magic Leap One advertise a 3 hour battery life, these can mean vastly different things. In order to drive all of its extra hardware, the Magic Leap One needs a much beefier battery. The ML1 is powered by a twin-cell battery with 36.77 Wh, running at 3.83 V. The HoloLens has a 1.65 Wh battery.

For overall performance, the larger battery means the world meshes (i.e. surface reconstruction, world mapping) are much denser and more frequently updated on the Magic Leap than on the HoloLens. The Time-of-Flight depth camera can fire off more frequently and for longer periods.

The larger battery and beefier specs also translate to much better 3D performance. The HoloLens is able to run 30,000 polygons at 60 fps. Beyond that, the fps begins to drop. The Magic Leap runs upwards of 1 million polygons at 60 fps.

On the downside, that more powerful battery rig needs a fan to cool it whereas the HoloLens is passively cooled. In laboratory and medical scenarios where a sterile environment must be maintained, active cooling with a fan could be a problem.

3. The HoloLens and Tracking

The HoloLens uses 4 monochrome cameras (“environment aware sensors”), an accelerometer, magnetometer and gyroscope in a sensor fusion configuration, and a custom HPU to perform head tracking. The Magic Leap one has a similar setup minus the HPU.

The HoloLens tracking is still somewhat better than the ML1’s. It loses tracking less frequently and digital content is less jittery when seen up close or while the wearer is in motion.

Overall, though, tracking performance is fairly close between the two devices.

4. Magic Leap Extras

The ML1 has a couple of features that are simply outside of the box. One is the eye tracking. There are inward facing cameras that track the wearer’s eye movements as invisible IR flashes.

The tracking is not continuous and is captured at a much lower resolution level than the displays. While they shouldn’t be used for direct user interactions, they are great for providing context for other interactions. It would be great if someone would write a keyboard that uses eye tracking to select keys. In the meantime, I wrote this heat vision demo that uses eye tracking to burn the walls of my house — I think of it as “Superman with a Migraine”. Note the eye-blink tracking.

The other cool extra in the Magic Leap is two planes of focus. Most VR devices have a single plane of focus at infinity. The HoloLens has a single plane of focus set at two meters.

In the magic leap one, when you look at near objects, objects further away (on the outer plane) seem to go out of focus. When you look at objects close up, the objects further away go out of focus. I would guess that the close plane is around a meter and the out one about 3 meters but I’m not really sure. In the Lumin OS .91, there is also a sporadic green shift in the near plane (which I expect will be fixed soon).

5. The Tether

The Magic Leap One is made up of two parts: the Light Pack and the Light Wear. They are connected by a cable. The Light Wear contains all the sensors, projectors and displays while the Light Wear, worn at the hip, contains all the computer bits and the battery.

This is an engineering choice that allows for a much larger power source. Without the tether solution, a large battery would not be possible. Without the large battery, the ML1’s enhanced depth sensing, improved graphics processing and larger field of view would not be possible.

In addition, this design makes the Magic Leap a much more comfortable fit on the head. The weight distribution is better than on the HoloLens, it is lighter, and it doesn’t require extra straps.

The tether solution is actually so effective that I would be surprised if the HoloLens v2 does not follow a similar design. The original one-piece “tetherless” solution Microsoft came up with for the HoloLens was visionary, but severely limiting.

6. Developing

If you have ever developed in Unity for the Android (or really any other device) then you know how to develop for Magic Leap in Unity. You press a button and your app compiles to an .mpk image (Android uses “.apk” file extensions). If your device is attached, you can deploy directly by clicking on “build and run”.

Magic Leap apps can also be built with the Unreal Engine.

HoloLens apps run on a Unity player sandboxed in a UWP app. The development cycle consequently involves exporting your HoloLens app as a Visual Studio project targeting UWP and then building and deploying in UWP. In general (and it may just be me) this has been tedious.

It became even worse when the immersive WinMR devices (or occluded WinMR – basically Microsoft VR) devices came out last year and the basic tools used for HoloLens development, known as the HoloLens Toolkit and then the Mixed Reality Toolkit, was expanded to supported both kinds of device. Because of some issues with Unity, building for WinMR required certain versions of Unity and above while developing for HoloLens required certain versions of Unity and below. And this state went on for several months to the point that finding the correct Windows SDK paired with the right MRTK version paired with the correct Unity version became a closely kept alchemical formula passed from developer to developer.

This experience may not be the same for everyone but it left me a bit traumatized. By contrast, Magic Leap development is simply a pleasure. I can build and see the results very quickly in my device. I can wear the device for hours at a time. I typically only stop development when the ML battery runs down and I have to let it recharge. I don’t have a Magic Leap Hub, which would allow me to charge while I dev, but I intend to get one.

The Magic Leap toolkit is still not quite as capable as the open source Mixed Reality Toolkit managed by Stephen Hodgson and others.

The Magic Leap also has a simulator rather than an emulator for developing without a device. This actually makes sense since the Hololens emulator runs the HoloLens OS in a virtual machine, which might be tricky given the much larger specs of the Magic Leap.

7. Interactions

The Magic Leap supports robust hand and gesture tracking as well as a 6DOF controller. The DOF in 6DOF stands for degrees of freedom. We know not only the direction the controller is pointing in (3DOF) but also its position.

I love the controller. I love it so much it made me finally admit to myself that I hate the HoloLens tap gesture. No one ever gets it right. It’s awkward. It’s uncomfortable and makes me feel like I’m performing a kung fu move.

By contrast, a controller just makes sense. The UX for MR, I believe, should always support three layers of interactions. Mixed reality UX should support hand gestures for ease of use. It should fall back to the controller for precision movements. It should finally fall back on the delta pad on the controller for accessibility.

For all of my antipathy toward the HoloLens tap, however, I have to say I miss the HoloLens bloom gesture (escape), which I keep trying to use in Magic Leap to no avail. Instead, in Magic Leap holding the controller’s Home button for three seconds is the escape gesture, which I don’t really like. It also bothers me that hand gestures aren’t supported in the core desktop (the Icon grid) – but this is still the Creator’s Edition (translation: dev edition) after all.

[Late edit thanks to SH: it should also be pointed out that the Lumin OS (the desktop layer) currently doesn’t support hand gestures, which I find baffling. For now, you can’t get past the login and other initial screens without a paired phone or a controller.]

Summing Up

So is the Magic Leap One better than the HoloLens v1? Oh yes. By leaps and bounds.

1. The development workflow is much more straight forward and pleasant.

2. The increased battery size and beefier hardware makes it possible to do things, performance wise, that the HoloLens tended to stop us from doing. Phone and tablet level experiences are doable now.

3. The Magic Leap One has a much better interaction model than the HoloLens does. How did anyone ever do MR without a controller? (Actually, everyone used an XBox controller in the end in order to get any sort of real work done, but we don’t talk about that much.)

Is it time to jump back into Mixed Reality development?

If you spent $3.2K to $5K for a HoloLens, then you owe it to yourself to spend $2,300 for a Magic Leap. It’s the device you originally wanted. The HoloLens was a brilliant device back in 2016 and really the first of its kind, but it had limitations. Many of the projects you were never able to realize in HoloLens (in the small dev community that developed around HoloLens, we all know what these are) are now doable with the improved Magic Leap specs. Additionally, your enterprise stories are much easier to sell with the controller. Instead of spending 5 minutes of your precious pitch time explaining how tap works, you can now just let your potential investors and clients go straight into the demo with a controller they basically already know how to use.

Is there a future in spatial computing?

Now there is. There was a brief pause between 2016 and the middle of 2018, but we currently have two great devices available with another shoe dropping soon. Microsoft will be coming out with a HoloLens v2 sometime in the first half of 2019 which I would predict will implement the tethered design Magic Leap is using. This will be an improvement over the current Magic Leap which in turn will be driven to improve its own tech.

Microsoft has an advantage because it started this journey back in the Kinect days and has the resources of Microsoft Research to draw on. Magic Leap has an advantage because, well, they aren’t Microsoft and don’t face the internal political problems a large tech giant does (though no doubt they have their own). More importantly, they have their own U.S.-based production lines (as well as production lines in Mexico) and are less reliant on China, which hopefully means they are capable of much quicker turn-arounds and initial SKU production.

When do we get smaller devices that wear like glasses?

I have no idea, but try to think in terms of 3, 5, 10 years. We always overestimate what can be done in 3 years but always underestimate how much things will change in 10. Somewhere in the middle, we will intersect with our MR futures.

Your comments, corrections and criticisms are welcome in the comments below. I’ll try to keep up with them and incorporate what you say into the main article as appropriate.

HoloLens and MR Device Dev Advisory Mar-2018

March 28, 2018March 29, 2018 James Ashley1 Comment

I’m currently doing HoloLens development with VS 2017 v15.6.4, Unity 2017.3.1p3, MRTK 2017.1.2, and W10 17128 (Insider Build).

Unity 2017.3.1p3, a patch release, includes the 2017.3.1p1 fix for hologram instability:

(993880) – XR: Fixed stabilization plane not getting set correctly via the SetFocusPointForFrame() API, resulting in poor hologram stabilization and color separation on HoloLens.

There continues to be uncertainty about whether this fixes all the stabilization problems or not – though it’s definitely better than it has been over the past several months.

UnityWebRequest continues to always return true for the isNetworkError property. Use isError or isHttpError instead. The older WWW class probably shouldn’t be used anymore. There are reports that media downloads aren’t working with UnityWebRequest while other file types are.

So if you have something working now, and have work-arounds in place, you probably shouldn’t upgrade. I know of HoloLens developers who are still very happy working in the older Unity 5.6.3.

April is shaping up to be very interesting. According to Unity, they will be releasing Unity 2018.1.0 then. For UWP/HoloLens developers, this means the addition of the .NET Standard 2.0 API compatibility level. .NET Standard 2.0 can be thought of as the set of APIs commonly supported by both .NET Core 2.0 (what UWP uses) and .NET Framework 4.6.1 (used for Windows apps and in the IDE). By supporting this, Unity 2018.1.0 should provide us with the ability to write much more common MonoBehaviour script code that works in both the IDE and on the HoloLens without using precompiler directives.

Of course, this is only useful if the HoloLens actually supports .NET Core 2.0, which is why the announcement of the RS4 Technical Preview is such a big deal. This is the first major firmware update for the HoloLens since release, and brings with it all the changes to Windows platform since the Anniversary Update (build 10.0.14393) which was also known as RS1 and which supports .NET Core 1.0.

Redstone 4 (build 10.0.17133), also known as the Spring Creators Update, is supposed to drop for PCs in mid-April. Which coincidentally is also when Unity 2018.1.0 is supposed to drop. So it would not be out of the question to expect a version of RS4 for HoloLens to drop at around the same time.

What sets RS4 for HoloLens apart from RS4 for Windows? For one thing, on the HoloLens we will have access to a new feature called Research Mode, providing access to low level sensor data such as the ToF depth camera and potentially the 4 mono cameras and the microphones. This in turn can be used to try out new algorithms beyond what the HoloLens already currently uses for data analysis.

On the UI front, the MR Design Labs interface tools have finally been integrated into the dev branch of the Mixed Reality Toolkit. Fingers crossed that this will make its way into the main branch in April also.

Finally, Magic Leap’s mixed reality headset, dubbed the Magic Leap One, had its debut at GDC this month. They also opened their creator portal to all developers, with links to documentation, the Lumin SDK, a special version of Unity 2018 to develop ML apps and a simulator to test gesture and controller interactions.

In the interest of full disclosure, I’ve been developing for the Magic Leap for a while under NDAs and inside a locked room ensorcelled by eldritch spells. It’s a great device and finally creates some good competition for the HoloLens team at Microsoft.

The first reaction among people working with the HoloLens and occluded MR devices may be to be defensive and tribal. Please resist this instinct.

A second, well-funded device like the Magic Leap One means all that much more marketing dollars from both Microsoft and Magic Leap spent on raising the profile of Mixed Reality (or Spatial Computing, as ML is calling it). It means healthy competition between the two device makers that will encourage both companies to improve their tech in efforts to grow and hold large swaths of the AR market. It also means a new device to which most of your spatial development skills will easily transfer. In other words, this is a good thing my MR homies. Embrace it.

And from the development side, there are lots of things to like about Magic Leap. Lumin is Linux/Mono based, which means a higher level of compatibility between the platform and pre-existing Unity assets from the Asset store. It also supports development in Unreal. Lastly, it also supports development on a Mac, potentially offering a way for crossover between the design, gaming and enterprise dev worlds. This in turn raises interest in high-end AR and will make people take a second look at HoloLens and the occluded MR devices.

It doesn’t take a weatherman to know it’s going to be a great summer for Mixed Reality / Spatial Computing developers.

HoloLens and MR Device Dev Advisory Jan-2018

January 23, 2018March 28, 2018 James Ashley1 Comment

I’ve come to accept that doing HoloLens and MR Device development means working with constant issues. As long as I can stay on top of what these issues are, I feel less like pulling out my hair. And I’m not just a member of the hair club for men – I also want to help you avoid hair loss with monthly updates.

I’m currently doing HoloLens development with VS 2017 v15.3.3, Unity 2017.2.0f3, MRTK 2017.1.2, and W10 17074 (Insider Build).

This month saw the release of Unity 2017.3.0f3, which introduced a fix but also some new bugs. The fix is to the HoloLens stabilization issues introduced in December’s Unity 2017.2.1 release which caused holograms to be jittery. In Unity 2017.3.0f3, a new player setting in the editor called shared depth buffer fixes this. Just expand the Windows Mixed Reality node under XR Settings and check off Enable Depth Buffer Sharing. On the other hand, this seems to conflict with the stabilization logic in the MRTK, so you may see some jumpiness (but no jitteriness!) so you’ll want to remove that older logic from your code.

2017.3.0f3 also introduced problems with WWW (Unity’s older internet communication class). Basically, it doesn’t work when running in UWP anymore (though it does on other platforms and in the editor), so if your code or any assets you may be depending on for internet communication depend on WWW, you’ll have issues.

If you are using older stable builds of the MR Toolkit (up to 2017.1.2), you’ll start getting warnings and errors in 2017.2.0f03 and above about outdated APIs. Unity introduces API changes with each monthly release. The UnityEngine.VR.WSA namespace, for instance, is now UnityEngine.XR.WSA (sometimes the Unity editor will automatically fix this for you when you migrate a project but often it doesn’t). In a couple of cases (like in the HandGuidance class) you’ll notice that the InteractionManager APIs have changed.

UnityEngine.XR.WSA.Input.InteractionManager.SourceLost += InteractionManager_SourceLost;
UnityEngine.XR.WSA.Input.InteractionManager.SourceUpdated += InteractionManager_SourceUpdated;
UnityEngine.XR.WSA.Input.InteractionManager.SourceReleased += InteractionManager_SourceReleased;

becomes:

UnityEngine.XR.WSA.Input.InteractionManager.InteractionSourceLost += InteractionManager_SourceLost;
UnityEngine.XR.WSA.Input.InteractionManager.InteractionSourceUpdated += InteractionManager_SourceUpdated;
UnityEngine.XR.WSA.Input.InteractionManager.InteractionSourceReleased += InteractionManager_SourceReleased;

The signature of the event handlers also change. Just follow Visual Studio Intellisense’s notes on how to fix the signatures.

A Guide to Online HoloLens Tutorials

November 30, 2017December 2, 2017 James Ashley1 Comment

There are lots of great video tutorials and advanced HoloLens materials online that even people who work with HoloLens aren’t always aware of. I’d like to fix that in this post.

1. The Fundamentals

lil

If you are still working through the basics with the HoloLens, then I highly recommend the course that Dennis Vroegop and I did for LinkedIn Learning: App Development for Microsoft HoloLens. We approached it with the goal of providing developers with everything we wish we had known when we started working with HoloLens in early 2016. The course was filmed in a studio at the Lynda.com campus in Carpinteria, California, so the overall quality is considerably higher than most other courses you’ll find.

2. The Mixed Reality Toolkit (HoloToolkit)

Once you understand the fundamentals of working with the HoloLens, the thing to learn is the ins-and-outs of the Mixed Reality Toolkit, which is the open source SDK for working with the HoloLens APIs. Stephen Hodgson, a Mixed Reality developer at Valorem, is one of the maintainers of (and probably biggest developer on) the MRTK. He does live streams on Saturdays to address people’s questions about the toolkit. His first two hour-long streamcasts cover the MRTK Input module:

#1 Input 1

#2 Input 2

The next three deal with Sharing Services:

#3 Sharing 1

#4 Sharing 2

#5 Sharing 3

These courses provide the deepest dive you’re ever likely to get about developing for HoloLens.

3. HoloLens Game Tutorial

Sometimes it is helpful to have a project to work through from start to finish. Chad Carter provides this with a multipart series on game development for Mixed Reality. So far there are five lessons … but the course is ongoing and well worth keeping up with.

#1 Setup

#2 Core Game Logic

#3 The Game Controller

#4 Motion Controllers

#5 Keeping Score

4. Scale and Rotation System

Jason Odom’s tutorial series deals with using Unity effectively for HoloLens. It brings home the realization that most of 3D development revolves around moving, resizing, hiding and revealing objects. It’s written for an older version of the toolkit, so some things will have changed since then. By the way, Jason’s theme song for this series is an ear worm. Consider yourself warned.

#1 Setup

#2 Scale and Rotate Manager

#3 Scale and Rotate Class

#4 Scale and Rotate Class part 2

#5 Scale and Rotate Class part 3

#6 Scale and Rotate More Manager Changes

#7 Scale and Rotate Temporary Insanity

#8 Scale and Rotate Q & A

5. HoloLens Academy

There’s also, of course, Microsoft’s official tutorial series known as the HoloLens Academy. It’s thorough and if you follow through the lessons, you’ll gain a broad understanding of the capabilities of the HoloLens device. One thing to keep in mind is that the tutorials are not always synced up with the latest MRTK so don’t get frustrated when you encounter a divergence between what the tutorials tell you to do and what you find in the MRTK, which is being updated at a much more rapid rate than the tutorials are.

6. Summing up

You’re probably now wondering if watching all these videos will make you a HoloLens expert. First of all, expertise isn’t something that you arrive at overnight. It takes time and effort.

Second of all – yeah. Pretty much. HoloLens development is a very niche field and it hasn’t been around for very long. It has plenty of quirks but all these videos will address those quirks either directly or obliquely. If you follow all these videos, you’ll know most of everything I know about the HoloLens, which is kinda a lot.

So have fun, future expert!