Video capture tips for the Apple Vision Pro

You can take a snapshot of a spatial computing experience in the Apple Vision Pro simply by pressing both the crown button and the top button at the same time.

IMG_0039

This doesn’t work for video capture, unfortunately. Video capture on the Apple Vision Pro takes spatial video of the actual world, only, and won’t create a composite video of the physical and the digital worlds.

If you want to take a video of a spatial computing experience, you will need to stream what you see to a MacBook or an iPhone, iPad or AppleTV.

There are two ways to record from the Apple Vision Pro.

  1. Mirror to another device (stream and then record).
  2. Use the developer capture tool in Reality Composer Pro (record and then stream).

Mirroring is the more flexible way of doing a recording and allows you to record a video of indefinite length. This is great if you want to do a lot of takes or just want a lot of footage to use.

Using Reality Composer Pro is more complicated and only lets you capture 60 seconds of video at a time. The advantage capturing video this way is that the quality of the video is much higher, at 30 FPS using 10-bit HEVC.

All the advice I’m passing on to you, by the way, I learned from Huy Le, who is an amazing cinematographer of digital experiences.

Mirror to another device

You can mirror what you see inside the Apple Vision Pro to a MacBook, iPhone, iPad or an Apple TV. This is a really handy feature for doing demos or even if you just want to guide someone through using the device or using an app for the first time. It’s also great for user testing your apps.

In order to mirror, you will need to have both the Apple Vision Pro and the device you are streaming to on the same WiFi network.

IMG_0044

1. Inside the Apple Vision Pro, look up – way, way up. A down arrow in a circle will appear at the top of your view. Focus your eyes on it and tap.

IMG_0045

2. Next, select the third icon from the left showing two ellipses stacked on top of each other. This is the control center button.

IMG_0048

3. On the next screen, select the fourth icon from the left showing two rounded rectangles overlapping. This is the “mirror my view” icon.

IMG_0047

4. The final screen will provide you with a list of devices on your WiFi network that you can stream to. Tap on one of these to select it.

5. You will now have a steady 2D video with very little lag streaming to your device. I typically stream to my MacBook. From the MacBook there are two common ways to record your stream. One is to use the built in recording tool. Type Ctrl-Shift-5 to start recording (it is a good idea to start your recording before you begin streaming). The built-in tool is convenient but sometimes it is hard to know if it is recording or not. The second way to record your stream is to download OBS, which is a popular open source screen capture tool, and just use that. OBS gives you a lot of fine-grain control over your recording.

6. Because the Apple Vision Pro can shake while you are wearing it, it’s a good idea to run your footage through video editing software like Adobe Premier Pro and apply a stabilizer effect to it.

Reality Composer Pro

For the highest quality capture (for instance if you are doing promotional footage of your app) you should use the developer capture tool in Reality Composer Pro.

1. To use developer capture, your AVP and your MacBook should be on the same WiFi network. Additionally, the AVP needs to be paired with your MacBook. You may also need to have “developer mode” turned on on your Vision Pro.

Screenshot 2024-02-20 at 9.31.07 AM

2. To open Reality Composer Pro, you first need to open XCode on your MacBook. (If you don’t have XCode 15 installed, you can download it from the Apple site.) Select “Open Developer Tool” on the XCode menu. Then select “Reality Composer Pro”.

Screenshot 2024-02-20 at 9.31.23 AM

3. In Reality Composer Pro, select “Developer Capture…” from the File menu.

Screenshot 2024-02-20 at 9.31.37 AM

4. Select your Apple Vision Pro device  from the drop down menu and click on the record button. This will record on device and then push the completed video to your MacBook when it is done.

5. When the recording is finished it will be deposited on your MacBook desktop as a QuickTime movie.

A note about video capture of in-app video – you can’t

You cannot use either of these techniques to capture a spatial computing experience that incorporates video. If you try to do video capture of apps like the IMAX app or the Disney Plus app, the in-app video will cut out when you begin recording. This may be a DRM feature – but I suspect it might actually be some sort of performance lock on the video buffer.

The (Missing) Apple Vision Pro Developer Ecosystem

Here is an image depicting a unique ecosystem where software developers replace traditional fauna. You can see developers engaging in various activities within a lush green forest and alongside a clear blue river. Some are sitting with their laptops on the riverbank, others are having discussions under the trees, and a few are walking along the forest trails, all in a blend of casual and work attire. The sunlight filtering through the canopy creates a beautiful interplay of light and shadows on the forest floor, highlighting the fusion of nature and technology.

When the app store became available on the iPhone 3G in July of 2008, it had 500 native apps available. When the Apple Vision Pro comes out on February 2, 2024, it will reportedly have around 200* native apps available for download.

These numbers are important in understanding how what is occurring this week fits into the origin myth of the smartphone in late capitalist popular culture.

Here I don’t use the term “myth” to mean that what we understand about the origins of the smartphone are untrue – that the app store was Apple’s “killer app”, that a sui generis army of developers became enamored of the device and built a developer ecosystem around it, that the number of native apps, mostly games, built for the iPhone grew rapidly, even exponentially (to 100,000 by 2010), that someone attached a long tail to the app store beast so that a handful of popular apps captured the majority of the money flooding into the app store, and so on. This is our established understanding of what happened from 2008 to the present. I don’t intend to undermine it.

Rather, I’m interested in how this first cycle of the Apple device story, the original course, affects the second cycle, the ricorso. After Apple reached a saturation point for its devices in its primary, affluent markets, it was able to shift and make profits from the secondary market by selling re-branded older phones to the second and third world. But towards the end of the 20-teens, it was clear that even these secondary markets had hit a saturation point. People were not updating their phones as frequently as they used to, worldwide, and this would eventually hit the bottom line. The corso was reaching its period of decline; the old king was wounded; and in this period of transition the golden bough needs to be recovered in order to renew the technological landscape.

fisher_king

In February 2024, Tim Cook, the king of the orchard, is restarting Apple by embracing an iPhone replacement as he prepares to hand his kingdom over to a new king. The roadmap for this is occurring in an almost ritualistic manner, with commercials that echo the best moments of the previous device revolution.

If I am reaching deeply into the collective unconscious to explain what is happening today, it is because modern marketing is essentially myth creation. And Apple has to draw up all of this mythical sense of the importance of the smartphone in order to recreate what it had in 2008. It must draw mixed reality developers into its field of influence in order to create a store that will sustain it through the next 25 years, or risk a permanent decline.

The original 2008 app store, despite a pretty terrible iPhone SDK developer experience, was successful because developers believed in the promise that they could become app millionaires. Secondarily, developers also jumped in because they believed in the romance around the Apple developer ecosystem as a nexus for artists and dreamers.

I believe Apple believes its success is dependent on making an orchard for developers to create in. As beautiful as the new Vision Pro is, it can only be an occasion for the greatness of others. And in order to attract developers into this orchard, Apple must convince them that this has all happened before, in 2008, and that there will be a golden period in the beginning when any app will make money because there are so few apps available – fruit will drop from the trees into developers’ hands and fish will leap from the rivers into their laps. There is a flaw in this myth that, oddly, will ultimately confirm Apple’s preferred narrative.

First and foremost, any developer in the mixed reality space — that is, the space where development for the HoloLens, Magic Leap or the latest generations of Meta Quest occurs – understands that there is a vast crevasse separating the world of MR developers and the world of Apple developers. All the current Apple tools: XCode, SwiftUI, the various “kits”, are built for flat applications. On the other side, development for mixed reality headsets has been done on relatively mature game engines and for the most part on the Unity Engine. These developers understand spatial user interactions and how to move and light 3D objects around a given space, but they don’t know the first thing about those Apple tools. And the learning curve is huge in either direction. It will take a leap of faith for developers to attempt it.

coding_gnomes

There was originally some excitement about a Unity toolset for Apple Vision Pro mixed reality apps called Polyspatial which seemed promising. Initially, however, it was suggested that the price for using it might be in the tens of thousands**. Later still, it appears that much of the team working on it was affected by the recent Unity layoffs. Had it succeeded, it would have offered a bridge for mixed reality developers to cross. But from the early previews, it appears to still be a work in progress and might still take 6 months or longer to get to a mature point.

I’ve only broken the potential Apple Vision Pro developer ecosystem into two tribes, but there are many additional tribes in this landscape. The current Apple development ecosystem isn’t homogenous, nor is the mixed reality dev ecosystem. Some MR devs come from the games industry and for them MR may be a short detour in their careers. Some MR devs got their big breaks doing original development for the HoloLens in 2016 – these are probably the most valuable devs to have because they have seen 9 years of ups and downs in this industry. There are digital agency natives, who tend to dabble in lots of technologies. There are also veterans of the Creative Technologist movement of 2013, though most of these have gone on to work in AI related fields. The important thing is that none of these people work like any of the others. They have different workflows and different things they find important in coding. They may not even like each other.

Even more vexing — unless you simply want to create a port of an app from another platform — you will probably need a combination of all of these people in order to create something great in the Apple Vision Pro. This isn’t easy. And because it isn’t easy, it will take a lot more time for the Apple Vision Pro to grow its store of native apps than anticipated. This is going to be a long road.

wheels

So how long will it take Apple devs to get their heads inside spatial computing? And how long will it take coders with MR experience to learn the Apple tools? To be generous, let’s say that if a small team is very focused and works together well, and if they start today, they will be able to skill up in about five months. They will need an additional 3 months to design and build a worthwhile app. Supposing some overlap between learning and building, let’s say this comes to six months. This is still six months from launch, or sometime in August, before the number of Vision Pro apps starts to pick up.

This is six months in which the availability of native apps for the Vision Pro will be relatively low and in which any decent app will have a good chance of making serious money. The current inventory for Vision Pros in 2024 is estimated to be around 400,000. 400,000 people with Apple Vision Pros, assuming they sell out (and Apple has already sold 200,000 before launch) will be looking for things to do with their devices. It’s a good bet that someone who has paid approximately $2,800 for a spatial computing headset will be willing to spend a few hundred dollars more for apps for their device.

And let’s say an average app will go for $5. Assuming just a quarter of available app purchasers will be interested in buying your app, you could easily make $500,000. This is a decent return for a few months of learning a new software platform. And the sooner you learn it, the more likely you are to be at the root of the long tail rather than at its tip.

Which is to say, even if you attempt to escape the myth that Apple is creating for itself, you will eventually find your way back to it. Such is the nature of myths and the hero’s journey. They are always true in a self-fulfilling way.

_________________________________

* On 2/2/24, the AVP was released with 600 native apps.

** Currently Polyspatial and VisionOS packages for Unity require a $2000/seat/yr Unity Pro license.

New York Augmented Reality Meetup presentation

I did a presentation on ‘Before Ubiquity’ to the New York AR Meetup in November. It went rather well and I hope you like it. Andrew Agus, who’s been keeping the AR flame burning on the east coast for many years — read the original blog post and thought it could be repackaged as a talk – he was right!

The New York Augmented Reality group is one of the OG’s of AR. Among many notable meetings, fireworks flew in the April 11, 2022 “Great Display Debate” between Jeri Ellsworth and Karl Guttag. You’ll come away knowing a lot more about AR and VR after watching it.

https://www.youtube.com/watch?v=NeECmKWjwqg&t=4s

Are Prompt Engineering Jobs at Risk because of A.I.?

As you know, many AR/MR developers left the field last year to become prompt engineers. I heard some bad news today, though. Apparently A.I. is starting to put prompt engineers out of work.

A.I. is now being used to take normal written commands — sometimes called “natural” language – sometimes called by non-specialists “language” — and processing it into more effective generative A.I. prompts. There’s even a rumor circulating that some of the prompts being used to train this new generation of A.I. come from the prompt engineers themselves. As if every time they use a prompt, it is somehow being recorded.

Can big tech really get away with stealing other people’s work and using it to turn a profit like this?

On the other hand, my friend Miroslav, a crypto manager from Zagreb, says these concerns are overblown. While some entry level prompt engineering jobs might go away, he told me, A.I. can never replace the more sophisticated prompt engineering roles that aren’t strictly learned by rote off of a YouTube channel.

“A.I. simply doesn’t have the emotional intelligence to perform advanced tasks like creating instructional YouTube videos about prompt engineering. Content creation jobs like these will always be safe.”

Learning to Program for the Apple Vision Pro

Learning to program for the Apple Vision Pro can be broken into 4 parts:

  • Learning Swift
  • Learning SwiftUI
  • Learning RealityKit
  • Learning Reality Composer Pro (RCP)

There are lots of good book and video resources for learning Swift and SwiftUI. Not so much for RealityKit or Reality Composer Pro, unfortunately.

If you want to go all out, you should get a subscription to https://www.oreilly.com/ for $499 per yr. This gives you access to the back catalogs of many of the leading technical book publishers.

Swift

To get started on Swift, iOS Programming for Beginners by Ahmad Sahar is pretty good. iOS 17 Programming for Beginners though the official Apple documentation will get you to the same place: https://developer.apple.com/documentation/swift

If you want to dive deeper into Swift, after learning the basics, the Big Nerd Ranch Guide, 3rd edition, is a good read: Swift Programming: The Big Nerd Ranch Guide

But if you want to learn Swift trivia to stump your friends, then I highly recommend Hands-On Design Patterns with Swift by Florent Vilmart and Giordano Scalzo: Hands-On Design Patterns with Swift

SwiftUI

SwiftUI is the library you would use to create menus and 2D interafaces. There are 2 books that are great for this, both by Wallace Wang. Start with Beginning iPhone Development with SwiftUI and once you get that down, continue, naturally, with Wallace’s Pro iPhone Development with SwiftUI .

RealityKit

Even though Apple first introduced RealityKit in 2017, very few iPhone devs actually use it. So even though there are quite a few books on SceneKit and ARKit, RealityKit learning resources are few and far between. Udemy has a course by Mohammed Azam that is fairly well rated. It’s on my shelf but I have yet to start it. Building Augmented Reality Apps in RealityKit & ARKit

Reality Composer Pro

Reality Composer Pro is sort of an improved version of Reality Composer and sort of something totally different. It is a tool that sits somewhere between the coder and the artist – you can create models (or “entities”) with it. But if you’re an artist, you are probably more likely to import your models from Maya or another 3D modeling tool. As a software developer, you can then attach components that you have coded to these entities.

There are no books about it and the resources available for the previous Reality Composer aren’t really much help. You’ll need to work through Apple’s WWDC videos and documentation to learn to use RCP:

WWDC 2023 videos:

Meet Reality Composer Pro

Explore materials in Reality Composer Pro

Work with Reality Composer Pro content in Xcode

Apple documentation:

Designing RealityKit content with Reality Composer Pro

About Polyspatial

If you are coming at this from the HoloLens or Magic Leap, then you are probably more comfortable working in Unity. Unity projects can, under the right circumstances, deploy to the VisionOS Simulator. You should just need the VisionOS support packages to get this working. Polyspatial is a tool that allows you to convert Unity shaders and materials into VisionOS’s native shader framework — and I think this is only needed if you are building mixed reality apps, not for VR (fully immersive) apps.

In general, though, I think you are always better off going native for performance and features. While it may seem like you can use Unity Polyspatial to do in VisionOS everything you are used to doing on other AR platforms, these tools ultimately sit on top of RealityKit once they are deployed. So if what you are trying to do isn’t supported in RealityKit, I’m not sure how it would actually work.

Before Ubiquity

ubiquitous AR

I’ve been developing for augmented reality head-mounted displays for about eight years. I first tried the original HoloLens in 2015. Then I got to purchase a developer unit in 2016 and started doing contract work with it. Later I was in the early pre-release dev program for Magicleap’s original device, which led to more work on the HoloLens 2, Magicleap 2, and Meta Quest Pro. I continue to work in AR HMDs today.

The original community around the HoloLens was amazing. We were all competing for the same work, but at the same time, we only had each other to turn to when we needed to talk to someone who understood what we were going through. So we were all sort of frenemies, except that because Microsoft was notoriously tight-lipped with their information about the device, we helped each other out on difficult programming tricks and tricky AR UI concepts – and this made us friends as well.

In those early days, we all thought AR, and our millions in riches (oh what a greedy lot we were), were just around the corner. But we never quite managed to turn that corner. Instead we had to begin devising theories around what that corner was going to look like, what the signs would be as we approached that corner, and what would happen after we made the corner. Basically, we had to become more stringent in our analyses.

Out of this, one big idea that came to the fore was “AR Ubiquity”. This comes out of the observation that monumental technological change happens slowly and incrementally, until it suddenly happens all at once. So at some point, we believe, everyone will just be wearing AR headsets instead of carrying smartphones. (This is also known, in some quarters, as the “Inflection Point”.)

Planning, consequently, should be based less on how we get to that point, or even when it will happen; and more about how to prepare for “AR Ubiquity” and what we will do afterwards. So AR Ubiquity, in this planning model, can come in 3 years, or in 7 years, or maybe for the most skeptical of us in 20 years. It doesn’t really matter because the important work is not in divining when it will happen (or even who will make it happen) but instead in 1) what it will look like and 2) what we can do — as developers, as startups, as corporations — once it arrives.

meta-history

Once we arrive at a discourse about the implications of “AR Ubiquity” rather than trying to forecast when it will happen, we  are engaging with a grand historical narrative about the transformative power of tech – which is a happy place for me because I used to do research on philosophical meta-history in grad school – though admittedly I wasn’t very good at it.

“AR Ubiquity”, according to the tenets of meta-history, can at the same time both be a theory about how the world works and also a motif in a story about how we fit into the technological world. Both ways of looking at it can provide valuable insights. As a theory we want to know how we can verify (or falsify) it. As a story element, we want to know what it means. In order to discover what it means, in turn, we can excavate it for other mythical elements it resembles and draws upon. (Meta-history, it should be acknowledged, can lead to bad ideas when done poorly and probably worse ideas when it is done well. So please take this with a grain of salt (a phrase which itself has an interesting history, it is worth noting).

I can recall three variations on the theme of disruptive (or revolutionary) historical change. There’s the narrative of the apocalyptic event that you only notice once it has already happened. There’s the narrative of the prophesied event that never actually happens but is always about to. And then there’s the heralded event, which has two beats: one to announce that it is about to happen, and another when it does happen. We long thought AR would follow model A, it currently looks like it is following model B, and I hope it will turn out that we are living through storyline C. Let’s unpack this a bit.

A or B

Model A

Apocalyptic history, as told in zombie movies and TV shows, generally have unknown origins. The hero wakes up after the fateful event has already happened, often in a hospital room, and over the course of the narrative, she may or may not discover whether it was caused by a sick monkey who escaped a viral lab, or by climate change, or by aliens. There’s also the version of apocalyptic history that circulates in Evangelical Christian eschatology known as The Rapture. In the Book of Revelations (which New Yorker writer Adam Gopnik calls the most cinematic Michael Bey ready book of the Bible), St. John of Patmos has a vision in which 144,000 faithful are taken into heaven.  In popular media, people wake up to find that millions of the virtuous elect have suddenly disappeared while they have been left behind to try to pick up the pieces and figure out what to do in a changed world.

In the less dramatic intellectual upheavals described in Thomas Kuhn’s The Structure of Scientific Revolutions, you can start your scientific career believing that an element known as phlogiston is released during combustion, then end it believing that phlogiston doesn’t exist and instead oxygen is added to a combusted material when it is burned (it is oxidized). Or you might start believing that the sun revolves around the earth and end up a few decades later laughing at such beliefs, to the point that it is hard to understand how anyone ever believed such outlandish things in the first place.

It’s a bit like the way we try to remember pulling out paper maps to drive somewhere new in our car, or shoving cassette tapes into our Walkmans, or having to type on physical keys on our phones – or even recalling phones that were used just for phone calls. It seems like a different age. And maybe AR glasses will be the same way . One day it seems fantastical and the next we’ll have difficulty remembering how we got things done with those quaint “smart” phones before we got our slick augmented reality glasses.

waiting for godot

Model B

 The history of waiting might best be captured by the term Millennialism, which describes both Jewish and Christian belief in the return of a Messiah after a thousand years. The study of millennialist beliefs often cover both the social changes that occur in anticipation of a Millennialist event as well as the consequent recalculation of calendars that occurs when an anticipated date has passed and finally the slow realization that nothing is going to happen, after all.

But there are non-theistic analogs to Millennialism that share some common traits such as the Cargo Cult in Fiji or later UFO cults like the Heaven’s Gate movement in the 90’s. Marxism could also be described as a sort of Millenarist cult that promised a Paradise that adherents came to learn would never arrive. One wonders at what point, in each of these belief systems, people first began to lose faith and then decided to simply play along while lacking actual conviction. The analogy can be stretched to belief in concepts like tulip bulb mania, NFTs, bitcoin, and other bubble economies where conviction eventually becomes less important than the realization that everyone else is equally cynical. In the end, it is cynicism that maintains economic bubbles and millenarist belief systems rather than faith.

I don’t think belief in AR Ubiquity is a millenarist cult, yet. It certainly hasn’t reach the stage of widespread cynicism, though it has been in a constant hype cycle over the past decade as new device announcements serve to refresh excitement about the technology. But even this excitement is making way, in a healthy manner, for a dose of skepticism over the latest announcements from Meta and Apple. There’s a hope for the best but expect the worst attitude in the air that I find refreshing, even if I don’t subscribe to it, myself.

the silver surfer

Model C

The last paradigm for disruptive history comes in the form of a herald and the thing he is the herald for. St. John the Baptist is the herald of the Messiah, Jesus Christ, for example. And Silver Surfer is the herald for Galactus. One is a forerunner for good tidings while the other is a harbinger of doom.

The forerunner isn’t necessary for the revolutionary event itself. That will happen in any case. The forerunner is there to let us know that something is coming and to point out where we should be looking for it. And there is just something very human about wanting to be aware of something before it happens so we can more fully savor its arrival.

ar_forever

Model C is the scenario I find most likely and how I imagine AR Ubiquity actually happening.

First we’ll have access to a device that demonstrates an actually useable AR headset with actually useful features. This will be the “it’s for real” moment that dispels the millenarist anxiety that we’re all being taken for a ride.

The “it’s real” moment will then set a bar for hardware manufacturers to work against. The forerunner device becomes the target all AR HMD companies strive to match, once someone has shown them what works, and within a few years we will have the actual arrival of AR Ubiquity.

At this time, reviews of the Apple Vision Pro and the Meta Quest 3 suggest that either could be this harbinger headset. I have my doubts about the Meta Quest 3 because I’m not sure how much better it can be than the MQ2 and the Meta Quest Pro, especially since it has removed eye tracking, which was a key feature of MQP and made the hand tracking more useful.

The AVP, on the other hand, has had such spectacular reviews that one begins to wonder if  it isn’t too good to be true.

But if the reviews can be taken at face value, then AR Ubiquity, or at least a herald that shows us it is possible, might be closer than we think.

I’m just proposing a small tweak to the standard model of how augmented reality headsets will replace smartphones. We’ve been assuming that the first device that convinces consumers to purchase AR headsets will also immediately set off this transition from one device category to the other. But maybe this is going to occur in two steps. First a headset will appear that validates the theory that headsets can replace handsets. Then a second device will rotate the gears of history and lead us into this highly anticipated new technological age.

AWE Presentation

My colleague, Astrini Sie, and I delivered a talk called Porting HoloLens Apps to Other Platforms at AWE 2023. Astrini is an AI researcher, so we threaded AI and AR together to see what AI can teach AR.

Here is the synopsis and a link.

https://www.youtube.com/watch?v=BkE3y_uSfa8

“Although Microsoft has substantially withdrawn from its Mixed Reality and Metaverse ambitions, this left behind a sizable catalog of community built enterprise apps and games, as well as a toolset, the MRTK, on which they were developed. In this talk, we will walk you through the steps required to port a HoloLens app built on one of the MRTK versions to other platforms such as the Magic Leap 2 and the Meta Quest Pro. We hope to demonstrate that, due to clever engineering, whole ecosystems can be moved from one platform to other newer platforms, where they can continue to evolve and thrive.”

Tip: Update your Magic Leap SDK the easy way

In the early HTK and then MRTK days, updating a project with the latest version of the HoloLens software dev kit was always a little scary, and most developers tended to avoid doing it. Because the toolkit was often volatile, updates were hard to do, sometimes involved breaking changes, and then were extremely hard to back out of. As a rule, the version number of the MRTK you started a project with was going to be the version number of the MRTK you would be finishing it with.

Magic Leap has been coming out with SDK updates at the rate of about one a month in 2023. Fortunately they came out with a tool last month, the Magic Leap Setup Tool published in the Unity Asset Store, that makes this pretty painless, both for upgrading and downgrading.

https://assetstore.unity.com/packages/tools/integration/magic-leap-setup-tool-194780

ml_menu

Once you’ve installed the Setup Tool in your Magic Leap project, it will appear in your Unity toolbar. Magic Leap just came out with a their new v1.2.0 SDK yesterday. I used the Magic Leap Hub tool to download all the latest bits and also update my Magic Leap headset. To update one of my projects built on the previous v1.2.0-dev2 SDK, all I did was go to the project menubar, pull down the Magic Leap tab, and select Project Setup Tool.

tool

Next, I clicked on the “Update SDK” button to set a new Magic Leap SDK folder.

new_sdk

I switched my SDK folder from “v1.2.0-dev2” to “v1.2.0”. I got a small dialog asking if I wanted to switch out my Magic Leap package. I replied OK.

update

Next step, go into Package Manger and update the Magic Leap Unity SDK package if you need to (the picture above shows upgrading from 1.5.0 to 1.6.0).

prefs

The last step is to go into your Preferences and check the External Tools | Magic Leap tab. You’ll want to reference the right Magic Leap directory here in order to make sure you have the right ML Simulator associated with your Unity editor.

As a long time admirer of the HoloLens MRTK, I have to say this is a distinct improvement.

What the heck just happened to HoloLens?

IMG_1119

Last Wednesday, on January 18, Microsoft laid off 10,000 employees, or 5% of its workforce. That same day, Bloomberg reported that some of the cuts were targeting the HoloLens hardware team, which had just been moved under Panos Panay in June, 2022 while the software team had been placed under a different organization and the previous head of the HoloLens combined group, Alex Kipman, was maneuvered out of the company.

mrtk_layoff

Over last week, further announcements on social media indicated how thoroughly Microsoft’s MR and VR commitments had been deracinated. It turned out that the entire MRTK engineering group, which created and maintained the SDK and tools for developing on the HoloLens, had been laid off. This was problematic, because if you remove the team supporting the tools people use to develop on the HoloLens, people will stop developing for the HoloLens. It was hard to see this as an accidental by product of cost cutting moves and easy to see it as part of a larger strategic shift at Microsoft.

Microsoft also laid off the employees of AltSpaceVR, central to its Metaverse ambitions, announcing that the site would be shut down on March 10. Microsoft had acquired AtlSpaceVR in 2017.

Various announcements claimed that the work that had been done at AltSpaceVR would be taken up by the Microsoft Mesh team, which is under the Microsoft Teams organization. At the same time, however, there were rumors going around that up to 80% of the Mesh team had also gotten the axe, including some of their product community evangelists.

jessem

So to sum up, Microsoft cut deep into HoloLens hardware, the MRTK team, the AltSpaceVR business, and its Mesh team. In addition, they pushed out the organizational exec of the HoloLens team in the summer of 2022 and split his people between two other divisions where they no longer had his protection as the head of a Microsoft fiefdom. The HoloLens – and in turn Microsoft’s investment in Mixed Reality and the Metaverse – was probably already dead at that point. There was hope from the HoloLens developer community that they were simply pausing to see how Apple’s MR strategy would pan out. If there was some possibility that a successful marketing push by Apple would encourage Microsoft to move forward with their headsets, those hopes are now dashed. The cuts have been too deep. There will never be a HoloLens 3.

On the bright side…

Microsoft was first out of the gate to set a standard for what high-end augmented reality headsets would be like – even adapting an old unused term, “mixed reality”, to emphasize the difference between phone based AR and what they were doing.

Previously they had done a similar thing with the Kinect by creating a new market for low-cost 3D depth sensors, which in turn created an ecosystem of alternative vendors, which in turn created a supply chain for 3D components as well as competing technology for 3D capture such as photogrammetry and computer vision, which finally led to the world sensing components today that make untethered VR and self-driving cars possible.

With the HoloLens they helped forge a developer community, changed the priorities of 3D game engine companies like Unity and Unreal, provided competition for up and coming MR vendors like Florida-based Magic Leap, tested out the limits of mixed reality scenarios and proofed out the appetite for passthrough AR, used in the Meta Quest Pro, HTC Vive Elite, and the upcoming Apple MR device, while we all wait on advances in waveguide technology and its alternatives.

In a large sense, Microsoft, Alex Kipman, the thousands of people who worked on the Microsoft HoloLens team as well as the thousands of developers who helped to build out MR experiences for enterprise and commercial products, accomplished their mission. They pushed the tech forward.

The truth is, Microsoft has often been extremely good at helping to build out promising technology but has rarely been good at sticking with technology to the viability phase. The biggest example is being early to tablets and phones, realizing they were waaay too early, and then trying to pick them up again after other tech giants had already cornered the market on these devices. With some notable exceptions, like the Xbox, hardware just isn’t Microsoft’s game and they aren’t comfortable with it.

Which is okay since thanks to their work, Meta, Magicleap, HTC, NVidia and others are stepping into the gap that Microsoft is leaving behind. Advances in MR and VR (“metaverse”) tech and experience design will be made at those companies. The laid off HoloLens workers will be snatched up by these other companies and the developer community will adapt to building for these other hardware devices.

While more work needs to be done, the MRTK in both its version 2 and version 3 flavors, provide a good way for MR developers and companies invested in mixed reality to pivot and port to new devices.

Pivot and port

There are several constructive steps that can be taken over the next few months to continue to push MR forward. The first is renaming and refocusing the HoloDevelopers slack community. This has been the most successful and lively meeting spaces over the past six years for sharing mixed reality news, knowledge and gossip. Thank goodness it never got moved over the Teams, as was once proposed. But it does need to be renamed, since it now covers a much broader MR ecosystem than just the Microsoft HoloLens, and it needs some financial support to enable searching of the archives for past, now hidden information, about how to get things done. No one should be re-inventing the wheel simply because we can’t search the archives.

The next thing that needs to be done is to unravel the MRTK situation. In principle the MRTK is an API layer that will target multiple devices. One of the targets of the MRTK developed over the past couple of years is OpenXR, which in turn is also an API layer that targets multiple devices. (It’s confusing, I know, and I’ve been planning to write a Foucauldian analysis of soft power exercised through API dominance for about a year to explain it.)

There are also two versions of the MRTK, v2 and v3, both of which work with OpenXR. In principle, if you have an app that sits on top of MRTK, and it targets OpenXR, then you should be able to repoint your app at another device, such as the various Meta passthrough AR devices or the Magic Leap 2 MR device, and have it mostly work.

Here are some kinks.

  1. An OpenXR implementation requires that particular hardware device vendors create plugins that map the OpenXR API to their particular HMDs. This can be done more or less well. It can be done in its entirety or only partially. Magicleap, for instance, has a beta plugin available for the Unity implementation of OpenXR, but this still isn’t done, yet (please hurry Magicleap!).
  2. There are platform specific features that haven’t been generalized in OpenXR. For instance, Microsoft has a World Locking system that worked with its World Anchors system to make world anchors not drift so much. But the world locking system sits outside of OpenXR.
  3. MRTK3 hasn’t been published in anything other than preview versions. The team has been laid off a couple of weeks before the first planned release.
  4. For this reason, not many apps are using MRTK3. Also for this reason, it is unlikely that anyone will try to port their apps from MRTK2 to MRTK3, which is an untrivial task.
  5. Some have expressed a hope that the community will pick up the work and support of MRTK3, which was an open source project almost exclusively managed and worked on by Microsoft employees. The problem here is that this hasn’t historically happened. Open source projects are rarely community supported, but require someone to be paid to do it. When Microsoft dropped support of the early HoloLens Toolkit in 2017, it was only two independent developers, rather than a large pool of indie devs sharing the work, that did the majority of the labor involved in expanding it and rearchitecting it into MRTK2.
  6. A re-porting strategy is vital for the MR ecosystem to thrive. Startups need to be able to show that they are not hardware dependent and can get up and running again on a new device over the next three to six months.
  7. Additionally, there are hundreds of HoloLens apps, most not on any public store, available to be ported to alternative headset platforms. And every HMD platform currently has a strong need for more apps.
  8. But none of this can happen without a consensus on whether the ecosystem will be adopting MRTK2 going forward or MRTK3. And it can’t happen unless there is an ongoing commitment to support the MRTK source code.
  9. There are two aspects to the MRTK that make it vital to the ongoing progress of mixed reality. I’ve already discussed the importance of a porting strategy.

Hand Interaction Examples 1

The second key feature of the MRTKs is the interaction samples. These are for all practical purposes the best and in some sense only user interfaces available in MR and VR. If you need to enter data or push a button in either mixed reality or the metaverse, these are the tools you should use. They were carefully designed, following user research, but a team at Microsoft led by Julia Schwartz. They are amazing.

But they are also in a big sense reference samples. They need further work to optimize performance and to smooth out usability on a variety of platforms.

It is possible that another company – Unity, Meta, Mgicleap, etc – could step in and develop a new set of tools with ongoing maintenance. But at this point there isn’t.

Summing up

To sum up:

  1. The HoloLens is dead. It has been for about six months.
  2. … but it helped to create a community as well as a device ecosystem that goes on.
  3. The community at HoloDevelopers needs some funding and a second wind, but it has grown organically to be a central repository of knowledge about the development and design of MR apps.
  4. The MRTKs require some hard choices and then a lot of love to make them work well across hardware platforms.
  5. The interaction samples of the MRTKs are a national treasure and also need a lot of love from the community.
  6. We need lots of blog posts and videos covering how to port HoloLens apps to the MQP (Oculus), HTC Elite, Magicleap, and eventually the Apple MR device. In the process we can identify the gaps and issues involved in porting and try to fix them.
  7. Go hug a laid off Microsoft HoloLens employee if you can. I have high confidence they will all land well because they are highly skilled people in a field Microsoft is dropping in favor of generative AI (a reasonable move) but it’s still going to be a tough few months emotionally until they do.
  8. While you are at it, maybe go join the Holodevelopers slack group and hug a non-Microsoft developer, too. They’ll all be fine, too, but its tough to see the work you’ve been doing for the past seven years suddenly drop out from under you.
  9. Off the top of my head, here are some great ones to reach out to: Sean Ong, Joost van Schaik, Dennis Vroegop, Jason Odom, Stephen Hodgson, Simon Jackson, Vincent Guigui, Rene Schulte, Lucas Rizzotto, Andras Velvart, Sky Zhou, Huy Le, Eric Provencher, Lance Larsen, Dwayne Lamb, Charles Poole, Dino Fejzagic, and tons of others I can’t recall right away but that you will hopefully remind me of. They are all heroes.
  10. The ride continues. Just not at Microsoft.

Immersion and the Star Wars Galactic Star Cruiser

IMG_0284

In the second week of March, I took my family to the Galactic Starcruiser at Disneyworld in Orlando, Florida, informally known as the Star Wars Hotel. The Starcruiser is a two-night immersive Star Wars experience with integrated storylines, themed meals, costumes, rides and games. For those familiar with the Disneyworld vacation experience, it should be pointed out that even though the Star Wars themed Galaxy’s Edge area in Hollywood Studios, it isn’t a resort hotel. Instead, it can best be thought of as a ride in itself.

20220307_152237

The design is that of a cruise ship, with a dining hall and helm in the “front” and an engine room in the “back”, and a space bar off of the main muster area. The NPCs and the staff never break character, but work hard to maintain the illusion that we are all on real space cruise. Besides humans, the “crew” is also staffed with aliens and robots – two essential aspects of Star Wars theming.

IMG_0165

In line with the cruise experience, you even do a one-day excursion to a nearby alien planet. I’ve had trouble writing about this experience because it felt highly personal, lighting off areas of my child brain that were set aside for space travel fantasies. At the same time, it is also very nerdy, and the intersection of the highly nerdy and the highly personal is dangerous territory. Nevertheless, it being May 4th today, I felt I could not longer put it off.

How you do Immersion?

“Immersion” is the touchstone for what people and tech companies are calling the Metaverse. Part of this is a carry over from VR pitching, and was key to explaining why being inside a virtual reality experience was different and better than simply playing a 3D video game with a flat screen and a controller.

IMG_0103

But the term “immersion” hides as much as it reveals. How can “immersion” be a distinguishing feature of virtual reality when it is already a built-in aspect of real reality? What makes for effective immersion? What are the benefits of immersion? Why would anyone pay to be immersed in someone else’s reality? Is immersion a way to telling a story or is storyline a component of an immersive experience?

20220307_144400

A Russian doll aspect of the starcruiser is the “Simulation Room” which, in the storyline of the ship, is an augmented area in that recreates the climate of the planet the ship is headed toward. The room is equipped with an open roof which happens to perfectly simulate the weather in central Florida. The room also happens to be where the Saja scholars provide instruction on Jedi history and philosophy.

Space Shrimp (finding the familiar in the unfamiliar)

I’m the sort of person who finds it hard to every be present in the moment. I’m either anticipating and planning for the next day, the next week, the next few years, or I am reliving events from the past which I wish had gone better (or wish had never happened at all).

20220307_150017

For two and a half days on this trip, I was fully captivated by the imaginary world I was living through. There wasn’t a moment after about the first hour when I was thinking about anything but the mission I was on and the details of the world I was in. I didn’t feel tempted to check my phone or know what was happening in the outside world.

20220308_180003

An immersive experience, it seems to me, is one that can make you forget about the world in this way, by replacing it with a more captivating world and not letting go of you. I’ve been going over in my head the details of the star wars experience that make this work and I think the blue shrimp we had for dinner one night is the perfect metaphor for how Disney accomplishes immersion.

IMG_0231

To create immersion, there can be nothing that references the outside world. The immersive experience must be self-contained and everyone in the immersive experience, from cabin boy to captain, must only reference things inside the world of the starcruiser. Fortunately Star Wars is a pre-designed universe. This helps in providing the various details that are self-referential and remind us of the world of the movies rather than the world of the world.

IMG_0125

A great example of this is the industrial overhead shower spout and the frosted glass sliding shower door in our cabin. They are small details but harken back to the design aesthetic of the star wars movies, which contain, surprisingly, a lot of blue tinted frosted glass.

IMG_0159

This extends to the food. All the food is themed, in a deconstructionist tour de force, to appear twisted and alien. We drank blue milk and ate bantha steaks. We feasted on green milk and salads made from the vegetation found on the planet Falucia.

IMG_0278

And here there is a difficulty. Humans have a built-in sense of disgust of strange foods that at some point protected our ancestors from accidentally poisoning themselves. And so each item of food had to indicate, through appearance or the name given on the menu, what it was an analog of in the real world. I often found myself unable to enjoy a dish until I could identify what it was meant to be (the lobster bisque was especially difficult to identify).

20220308_185244(1)

What I took from this was that for immersion to work, things have to be self-referential but cannot be totally unfamiliar. As strange as each dish looked, it had to be, like the blue shrimp, analogous with something people knew from the real world outside the ship. Without these analogical connections, the food will tend to create aversion and anxiety instead of the sense of immersion intended.

IMG_0250

One take way is that as odd as the food sometimes looked, the food analogs were always meals familiar to Americans. Things common to other parts of the world, like chicken feet or durian fruit or balut, would not go over well even though they taste good (to many people).

IMG_0285

A second take away is that the galactic food has to be really, really good. In modern American cuisine, it is typical to provide the story behind the food explaining each ingredient’s purpose, where it comes from and how to use it in the dish (is it a salad or a garnish?). The galactic food can’t provide these value-add story points and only has fictitious ones.

IMG_0268(1)

In the case of the food served on the starcruiser, then, each dish has to stand on its own merits, without the usual restaurant storytelling elements that contribute to the overall sense that you are eating something expensive and worthy of that expense. Instead, each dish requires us to taste, smell, and feel the food in our mouths and decide if we liked it or not. I don’t think I’ve ever had to do that before.

World building – (decrepit futurism)

The world of Star Wars is one of decrepit futurism. It is a world of wonders in decline.

IMG_0169

There are other kinds of futurism like the streamlined retro-futurism of the 30s and 50s or contemporary Afro-futurism. The decrepit futurism of Star Wars takes a utopic society and dirties it up, both aesthetically and morally. The original Star Wars starts off at the dissolution of the Senate marking a political decline. George Lucas doubles down on this in the prequels making this also a spiritual decline in which the Jedi are corrupted by a malignant influence and end up bringing about the fall of their own order. The story of the sequels (which is the period in which the galactic space voyage takes place) is about the difficultly and maybe impossibility of restoring the universe to its once great heights.

IMG_0127

As beautiful and polished as all the surfaces are on the star cruiser, the ship is over 200 years old and has undergone massive renovations. Despite this, the engines continue to provide trouble (which you get to help fix). Meanwhile, the political situation in the galaxy in general and on the destination planet in particular is fraught, demanding that voyagers choose which set of storylines they will pursue. Will they help the resistance or be complicit with the First Order? Or will they opt out of this choice and instead be a Han Solo-like rogue pursuing profit amid the disorder?

IMG_0292

The metaphysics of the Star Wars universe is essentially fallibilist and flawed – which in turn opens the way for moral growth and discovery.

The decrepit futurism of Star Wars has always seemed to me to be one of the things that makes it work best because it artfully dodges the question of why things aren’t better in a technologically advanced society. Decrepit futurism says that things once were (our preconceptions of what the future and progress entails is preserved) but have fallen from the state of grace through a Sith corruption. In falling short, the future comes down to the level where the rest of us live.

IMG_0295

It’s also probably why Luke, in the last trilogy, never gets to be the sort of teacher we hoped he would be to Rey. The only notion we have of the greatness and wisdom of a true Jedi master comes from glimpses we get through Yoda in The Empire Strikes Back, but he is only able to achieve this level of wisdom by losing everything. Greatness in Star Wars is always something implied but never seen.

Storytelling (narrative as an organizing principle)

Much is made of storytelling and narrative in the world of immersive experiences. Some people talk as if immersion is simply a medium for storytelling – but I think it is the other way around. Immersion is created out of world building and design that distract us from our real lives. The third piece of immersion is storytelling.

IMG_0136

But one thing I discovered on the Galactic Starcruiser is that the stories in an immersive experience don’t have to be all that great – they don’t have to have the depth of a Dostoevsky novel. Instead they can be at the level of a typical MMORPG. They can be as simple as go into the basement and kill rats to get more information. Hack a wall terminal to get a new mission. Follow the McGuffin to advance the storyline.

IMG_0007

Narrative in an immersive experience is not magic. It’s just a way of organizing time and actions for people, much the way mathematical formulas organize the relationship between numbers or physics theorems organize the interactions of physical bodies. Narratives help us keep the thread while lots of other things are going on around us.

The main difficulty of a live theater narrative, like the one on the starcruiser, is that the multiple story lines have to work well together and work even if people are not always paying attention or even following multiple plots at the same time. Additionally, at some point, all of the storylines must converge. In this case, keeping things simple is probably the only way to go.

2022-03-09_20-_20Star_20Wars_20Galactic_20Starcruiser_20-_20Star_20Wars_20Galactic_20Starcruiser_5

Crafting a narrative for immersive experiences, it seems to me, is a craft rather than an art. It doesn’t have to provide any revelations or tell us truths about ourselves. It just has to get people from one place in time to another.

The real art, of course, is that exercised by the actors who must tell these stories over and over and improvise when guests throw them a curve ball while keeping within the general outline of the overarching narrative. And being able to do this for 3 days at a time is a special gift.

Westworld vs the Metaverse (what is immersion)

Using the Galactic Starcruiser as the exemplar of an immersive experience, I wanted to go back to the question of how immersion in VR is different from immersion in reality. To put it another way, what is the difference between Westworld and the Metaverse?

2022-03-08_20-_20Disneys_20Hollywood_20Studios_20-_20Millennium_20Falcon_13

There seems to be something people are after when they get excited about the Metaverse and I think it’s at bottom the ability to simulate a fantasy. Back when robots were all the rage (about the time Star Wars was originally made in the 70s) Michael Crichton captured this desire for fantasy in his film Westworld. The circle of reference is complete when one realizes that Chrichton based his robots on the animatronics at Disneyland and Disneyworld.

IMG_0156(1)

So what’s the difference between Westworld and the Metaverse? One of the complaints about the Metaverse (and more specifically VR) is that the lack of haptic feedback diminishes the experience. The real world, of course, is full of haptic feedback. More than this, it is also full of flavors and smells, which you cannot currently get from the Metaverse. It can also be full of people that can improvise around your personal choices so that the experience never glitches. This provides a more open world type of experience, whereas the Metaverse as it currently stands will have a lot of experiences on rails.

From all this, it seems as if the Metaverse aspires to be Westworld (or even the Galactic Starcruiser) but inevitably falls short sensuously and dynamically.

20220307_133505

The outstanding thing about the Metaverse, though, is that it can be mass produced – precisely because it is digital and not real. The Starcruiser is prohibitively expensive dinner theater which I was able to pull off through some dumb luck with crypto currencies. It’s wonderful and if you can afford it I highly encourage you to go on that voyage into your childhood.

IMG_0226

The Metaverse, on the other hand, is Westworld-style immersion for the masses. The bar to entry for VR is relatively low compared to a real immersive experience. Now all we have to do is get the world building, design, and storylines right.