What the heck just happened to HoloLens?

Last Wednesday, on January 18, Microsoft laid off 10,000 employees, or 5% of its workforce. That same day, Bloomberg reported that some of the cuts were targeting the HoloLens hardware team, which had just been moved under Panos Panay in June, 2022 while the software team had been placed under a different organization and the previous head of the HoloLens combined group, Alex Kipman, was maneuvered out of the company.

Over last week, further announcements on social media indicated how thoroughly Microsoft’s MR and VR commitments had been deracinated. It turned out that the entire MRTK engineering group, which created and maintained the SDK and tools for developing on the HoloLens, had been laid off. This was problematic, because if you remove the team supporting the tools people use to develop on the HoloLens, people will stop developing for the HoloLens. It was hard to see this as an accidental by product of cost cutting moves and easy to see it as part of a larger strategic shift at Microsoft.

Microsoft also laid off the employees of AltSpaceVR, central to its Metaverse ambitions, announcing that the site would be shut down on March 10. Microsoft had acquired AtlSpaceVR in 2017.

Various announcements claimed that the work that had been done at AltSpaceVR would be taken up by the Microsoft Mesh team, which is under the Microsoft Teams organization. At the same time, however, there were rumors going around that up to 80% of the Mesh team had also gotten the axe, including some of their product community evangelists.

So to sum up, Microsoft cut deep into HoloLens hardware, the MRTK team, the AltSpaceVR business, and its Mesh team. In addition, they pushed out the organizational exec of the HoloLens team in the summer of 2022 and split his people between two other divisions where they no longer had his protection as the head of a Microsoft fiefdom. The HoloLens – and in turn Microsoft’s investment in Mixed Reality and the Metaverse – was probably already dead at that point. There was hope from the HoloLens developer community that they were simply pausing to see how Apple’s MR strategy would pan out. If there was some possibility that a successful marketing push by Apple would encourage Microsoft to move forward with their headsets, those hopes are now dashed. The cuts have been too deep. There will never be a HoloLens 3.

On the bright side…

Microsoft was first out of the gate to set a standard for what high-end augmented reality headsets would be like – even adapting an old unused term, “mixed reality”, to emphasize the difference between phone based AR and what they were doing.

Previously they had done a similar thing with the Kinect by creating a new market for low-cost 3D depth sensors, which in turn created an ecosystem of alternative vendors, which in turn created a supply chain for 3D components as well as competing technology for 3D capture such as photogrammetry and computer vision, which finally led to the world sensing components today that make untethered VR and self-driving cars possible.

With the HoloLens they helped forge a developer community, changed the priorities of 3D game engine companies like Unity and Unreal, provided competition for up and coming MR vendors like Florida-based Magic Leap, tested out the limits of mixed reality scenarios and proofed out the appetite for passthrough AR, used in the Meta Quest Pro, HTC Vive Elite, and the upcoming Apple MR device, while we all wait on advances in waveguide technology and its alternatives.

In a large sense, Microsoft, Alex Kipman, the thousands of people who worked on the Microsoft HoloLens team as well as the thousands of developers who helped to build out MR experiences for enterprise and commercial products, accomplished their mission. They pushed the tech forward.

The truth is, Microsoft has often been extremely good at helping to build out promising technology but has rarely been good at sticking with technology to the viability phase. The biggest example is being early to tablets and phones, realizing they were waaay too early, and then trying to pick them up again after other tech giants had already cornered the market on these devices. With some notable exceptions, like the Xbox, hardware just isn’t Microsoft’s game and they aren’t comfortable with it.

Which is okay since thanks to their work, Meta, Magicleap, HTC, NVidia and others are stepping into the gap that Microsoft is leaving behind. Advances in MR and VR (“metaverse”) tech and experience design will be made at those companies. The laid off HoloLens workers will be snatched up by these other companies and the developer community will adapt to building for these other hardware devices.

While more work needs to be done, the MRTK in both its version 2 and version 3 flavors, provide a good way for MR developers and companies invested in mixed reality to pivot and port to new devices.

Pivot and port

There are several constructive steps that can be taken over the next few months to continue to push MR forward. The first is renaming and refocusing the HoloDevelopers slack community. This has been the most successful and lively meeting spaces over the past six years for sharing mixed reality news, knowledge and gossip. Thank goodness it never got moved over the Teams, as was once proposed. But it does need to be renamed, since it now covers a much broader MR ecosystem than just the Microsoft HoloLens, and it needs some financial support to enable searching of the archives for past, now hidden information, about how to get things done. No one should be re-inventing the wheel simply because we can’t search the archives.

The next thing that needs to be done is to unravel the MRTK situation. In principle the MRTK is an API layer that will target multiple devices. One of the targets of the MRTK developed over the past couple of years is OpenXR, which in turn is also an API layer that targets multiple devices. (It’s confusing, I know, and I’ve been planning to write a Foucauldian analysis of soft power exercised through API dominance for about a year to explain it.)

There are also two versions of the MRTK, v2 and v3, both of which work with OpenXR. In principle, if you have an app that sits on top of MRTK, and it targets OpenXR, then you should be able to repoint your app at another device, such as the various Meta passthrough AR devices or the Magic Leap 2 MR device, and have it mostly work.

Here are some kinks.

1. An OpenXR implementation requires that particular hardware device vendors create plugins that map the OpenXR API to their particular HMDs. This can be done more or less well. It can be done in its entirety or only partially. Magicleap, for instance, has a beta plugin available for the Unity implementation of OpenXR, but this still isn’t done, yet (please hurry Magicleap!).
2. There are platform specific features that haven’t been generalized in OpenXR. For instance, Microsoft has a World Locking system that worked with its World Anchors system to make world anchors not drift so much. But the world locking system sits outside of OpenXR.
3. MRTK3 hasn’t been published in anything other than preview versions. The team has been laid off a couple of weeks before the first planned release.
4. For this reason, not many apps are using MRTK3. Also for this reason, it is unlikely that anyone will try to port their apps from MRTK2 to MRTK3, which is an untrivial task.
5. Some have expressed a hope that the community will pick up the work and support of MRTK3, which was an open source project almost exclusively managed and worked on by Microsoft employees. The problem here is that this hasn’t historically happened. Open source projects are rarely community supported, but require someone to be paid to do it. When Microsoft dropped support of the early HoloLens Toolkit in 2017, it was only two independent developers, rather than a large pool of indie devs sharing the work, that did the majority of the labor involved in expanding it and rearchitecting it into MRTK2.
6. A re-porting strategy is vital for the MR ecosystem to thrive. Startups need to be able to show that they are not hardware dependent and can get up and running again on a new device over the next three to six months.
7. Additionally, there are hundreds of HoloLens apps, most not on any public store, available to be ported to alternative headset platforms. And every HMD platform currently has a strong need for more apps.
8. But none of this can happen without a consensus on whether the ecosystem will be adopting MRTK2 going forward or MRTK3. And it can’t happen unless there is an ongoing commitment to support the MRTK source code.
9. There are two aspects to the MRTK that make it vital to the ongoing progress of mixed reality. I’ve already discussed the importance of a porting strategy.

The second key feature of the MRTKs is the interaction samples. These are for all practical purposes the best and in some sense only user interfaces available in MR and VR. If you need to enter data or push a button in either mixed reality or the metaverse, these are the tools you should use. They were carefully designed, following user research, but a team at Microsoft led by Julia Schwartz. They are amazing.

But they are also in a big sense reference samples. They need further work to optimize performance and to smooth out usability on a variety of platforms.

It is possible that another company – Unity, Meta, Mgicleap, etc – could step in and develop a new set of tools with ongoing maintenance. But at this point there isn’t.

Summing up

To sum up:

1. The HoloLens is dead. It has been for about six months.
2. … but it helped to create a community as well as a device ecosystem that goes on.
3. The community at HoloDevelopers needs some funding and a second wind, but it has grown organically to be a central repository of knowledge about the development and design of MR apps.
4. The MRTKs require some hard choices and then a lot of love to make them work well across hardware platforms.
5. The interaction samples of the MRTKs are a national treasure and also need a lot of love from the community.
6. We need lots of blog posts and videos covering how to port HoloLens apps to the MQP (Oculus), HTC Elite, Magicleap, and eventually the Apple MR device. In the process we can identify the gaps and issues involved in porting and try to fix them.
7. Go hug a laid off Microsoft HoloLens employee if you can. I have high confidence they will all land well because they are highly skilled people in a field Microsoft is dropping in favor of generative AI (a reasonable move) but it’s still going to be a tough few months emotionally until they do.
8. While you are at it, maybe go join the Holodevelopers slack group and hug a non-Microsoft developer, too. They’ll all be fine, too, but its tough to see the work you’ve been doing for the past seven years suddenly drop out from under you.
9. Off the top of my head, here are some great ones to reach out to: Sean Ong, Joost van Schaik, Dennis Vroegop, Jason Odom, Stephen Hodgson, Simon Jackson, Vincent Guigui, Rene Schulte, Lucas Rizzotto, Andras Velvart, Sky Zhou, Huy Le, Eric Provencher, Lance Larsen, Dwayne Lamb, Charles Poole, Dino Fejzagic, and tons of others I can’t recall right away but that you will hopefully remind me of. They are all heroes.
10. The ride continues. Just not at Microsoft.