Hyperreality – The Imaginative Universal

What the heck just happened to HoloLens?

January 26, 2023January 26, 2023 James Ashley5 Comments

Last Wednesday, on January 18, Microsoft laid off 10,000 employees, or 5% of its workforce. That same day, Bloomberg reported that some of the cuts were targeting the HoloLens hardware team, which had just been moved under Panos Panay in June, 2022 while the software team had been placed under a different organization and the previous head of the HoloLens combined group, Alex Kipman, was maneuvered out of the company.

Over last week, further announcements on social media indicated how thoroughly Microsoft’s MR and VR commitments had been deracinated. It turned out that the entire MRTK engineering group, which created and maintained the SDK and tools for developing on the HoloLens, had been laid off. This was problematic, because if you remove the team supporting the tools people use to develop on the HoloLens, people will stop developing for the HoloLens. It was hard to see this as an accidental by product of cost cutting moves and easy to see it as part of a larger strategic shift at Microsoft.

Microsoft also laid off the employees of AltSpaceVR, central to its Metaverse ambitions, announcing that the site would be shut down on March 10. Microsoft had acquired AtlSpaceVR in 2017.

Various announcements claimed that the work that had been done at AltSpaceVR would be taken up by the Microsoft Mesh team, which is under the Microsoft Teams organization. At the same time, however, there were rumors going around that up to 80% of the Mesh team had also gotten the axe, including some of their product community evangelists.

So to sum up, Microsoft cut deep into HoloLens hardware, the MRTK team, the AltSpaceVR business, and its Mesh team. In addition, they pushed out the organizational exec of the HoloLens team in the summer of 2022 and split his people between two other divisions where they no longer had his protection as the head of a Microsoft fiefdom. The HoloLens – and in turn Microsoft’s investment in Mixed Reality and the Metaverse – was probably already dead at that point. There was hope from the HoloLens developer community that they were simply pausing to see how Apple’s MR strategy would pan out. If there was some possibility that a successful marketing push by Apple would encourage Microsoft to move forward with their headsets, those hopes are now dashed. The cuts have been too deep. There will never be a HoloLens 3.

On the bright side…

Microsoft was first out of the gate to set a standard for what high-end augmented reality headsets would be like – even adapting an old unused term, “mixed reality”, to emphasize the difference between phone based AR and what they were doing.

Previously they had done a similar thing with the Kinect by creating a new market for low-cost 3D depth sensors, which in turn created an ecosystem of alternative vendors, which in turn created a supply chain for 3D components as well as competing technology for 3D capture such as photogrammetry and computer vision, which finally led to the world sensing components today that make untethered VR and self-driving cars possible.

With the HoloLens they helped forge a developer community, changed the priorities of 3D game engine companies like Unity and Unreal, provided competition for up and coming MR vendors like Florida-based Magic Leap, tested out the limits of mixed reality scenarios and proofed out the appetite for passthrough AR, used in the Meta Quest Pro, HTC Vive Elite, and the upcoming Apple MR device, while we all wait on advances in waveguide technology and its alternatives.

In a large sense, Microsoft, Alex Kipman, the thousands of people who worked on the Microsoft HoloLens team as well as the thousands of developers who helped to build out MR experiences for enterprise and commercial products, accomplished their mission. They pushed the tech forward.

The truth is, Microsoft has often been extremely good at helping to build out promising technology but has rarely been good at sticking with technology to the viability phase. The biggest example is being early to tablets and phones, realizing they were waaay too early, and then trying to pick them up again after other tech giants had already cornered the market on these devices. With some notable exceptions, like the Xbox, hardware just isn’t Microsoft’s game and they aren’t comfortable with it.

Which is okay since thanks to their work, Meta, Magicleap, HTC, NVidia and others are stepping into the gap that Microsoft is leaving behind. Advances in MR and VR (“metaverse”) tech and experience design will be made at those companies. The laid off HoloLens workers will be snatched up by these other companies and the developer community will adapt to building for these other hardware devices.

While more work needs to be done, the MRTK in both its version 2 and version 3 flavors, provide a good way for MR developers and companies invested in mixed reality to pivot and port to new devices.

Pivot and port

There are several constructive steps that can be taken over the next few months to continue to push MR forward. The first is renaming and refocusing the HoloDevelopers slack community. This has been the most successful and lively meeting spaces over the past six years for sharing mixed reality news, knowledge and gossip. Thank goodness it never got moved over the Teams, as was once proposed. But it does need to be renamed, since it now covers a much broader MR ecosystem than just the Microsoft HoloLens, and it needs some financial support to enable searching of the archives for past, now hidden information, about how to get things done. No one should be re-inventing the wheel simply because we can’t search the archives.

The next thing that needs to be done is to unravel the MRTK situation. In principle the MRTK is an API layer that will target multiple devices. One of the targets of the MRTK developed over the past couple of years is OpenXR, which in turn is also an API layer that targets multiple devices. (It’s confusing, I know, and I’ve been planning to write a Foucauldian analysis of soft power exercised through API dominance for about a year to explain it.)

There are also two versions of the MRTK, v2 and v3, both of which work with OpenXR. In principle, if you have an app that sits on top of MRTK, and it targets OpenXR, then you should be able to repoint your app at another device, such as the various Meta passthrough AR devices or the Magic Leap 2 MR device, and have it mostly work.

Here are some kinks.

An OpenXR implementation requires that particular hardware device vendors create plugins that map the OpenXR API to their particular HMDs. This can be done more or less well. It can be done in its entirety or only partially. Magicleap, for instance, has a beta plugin available for the Unity implementation of OpenXR, but this still isn’t done, yet (please hurry Magicleap!).
There are platform specific features that haven’t been generalized in OpenXR. For instance, Microsoft has a World Locking system that worked with its World Anchors system to make world anchors not drift so much. But the world locking system sits outside of OpenXR.
MRTK3 hasn’t been published in anything other than preview versions. The team has been laid off a couple of weeks before the first planned release.
For this reason, not many apps are using MRTK3. Also for this reason, it is unlikely that anyone will try to port their apps from MRTK2 to MRTK3, which is an untrivial task.
Some have expressed a hope that the community will pick up the work and support of MRTK3, which was an open source project almost exclusively managed and worked on by Microsoft employees. The problem here is that this hasn’t historically happened. Open source projects are rarely community supported, but require someone to be paid to do it. When Microsoft dropped support of the early HoloLens Toolkit in 2017, it was only two independent developers, rather than a large pool of indie devs sharing the work, that did the majority of the labor involved in expanding it and rearchitecting it into MRTK2.
A re-porting strategy is vital for the MR ecosystem to thrive. Startups need to be able to show that they are not hardware dependent and can get up and running again on a new device over the next three to six months.
Additionally, there are hundreds of HoloLens apps, most not on any public store, available to be ported to alternative headset platforms. And every HMD platform currently has a strong need for more apps.
But none of this can happen without a consensus on whether the ecosystem will be adopting MRTK2 going forward or MRTK3. And it can’t happen unless there is an ongoing commitment to support the MRTK source code.
There are two aspects to the MRTK that make it vital to the ongoing progress of mixed reality. I’ve already discussed the importance of a porting strategy.

The second key feature of the MRTKs is the interaction samples. These are for all practical purposes the best and in some sense only user interfaces available in MR and VR. If you need to enter data or push a button in either mixed reality or the metaverse, these are the tools you should use. They were carefully designed, following user research, but a team at Microsoft led by Julia Schwartz. They are amazing.

But they are also in a big sense reference samples. They need further work to optimize performance and to smooth out usability on a variety of platforms.

It is possible that another company – Unity, Meta, Mgicleap, etc – could step in and develop a new set of tools with ongoing maintenance. But at this point there isn’t.

Summing up

To sum up:

The HoloLens is dead. It has been for about six months.
… but it helped to create a community as well as a device ecosystem that goes on.
The community at HoloDevelopers needs some funding and a second wind, but it has grown organically to be a central repository of knowledge about the development and design of MR apps.
The MRTKs require some hard choices and then a lot of love to make them work well across hardware platforms.
The interaction samples of the MRTKs are a national treasure and also need a lot of love from the community.
We need lots of blog posts and videos covering how to port HoloLens apps to the MQP (Oculus), HTC Elite, Magicleap, and eventually the Apple MR device. In the process we can identify the gaps and issues involved in porting and try to fix them.
Go hug a laid off Microsoft HoloLens employee if you can. I have high confidence they will all land well because they are highly skilled people in a field Microsoft is dropping in favor of generative AI (a reasonable move) but it’s still going to be a tough few months emotionally until they do.
While you are at it, maybe go join the Holodevelopers slack group and hug a non-Microsoft developer, too. They’ll all be fine, too, but its tough to see the work you’ve been doing for the past seven years suddenly drop out from under you.
Off the top of my head, here are some great ones to reach out to: Sean Ong, Joost van Schaik, Dennis Vroegop, Jason Odom, Stephen Hodgson, Simon Jackson, Vincent Guigui, Rene Schulte, Lucas Rizzotto, Andras Velvart, Sky Zhou, Huy Le, Eric Provencher, Lance Larsen, Dwayne Lamb, Charles Poole, Dino Fejzagic, and tons of others I can’t recall right away but that you will hopefully remind me of. They are all heroes.
The ride continues. Just not at Microsoft.

Immersion and the Star Wars Galactic Star Cruiser

May 4, 2022May 5, 2022 James Ashley

In the second week of March, I took my family to the Galactic Starcruiser at Disneyworld in Orlando, Florida, informally known as the Star Wars Hotel. The Starcruiser is a two-night immersive Star Wars experience with integrated storylines, themed meals, costumes, rides and games. For those familiar with the Disneyworld vacation experience, it should be pointed out that even though the Star Wars themed Galaxy’s Edge area in Hollywood Studios, it isn’t a resort hotel. Instead, it can best be thought of as a ride in itself.

The design is that of a cruise ship, with a dining hall and helm in the “front” and an engine room in the “back”, and a space bar off of the main muster area. The NPCs and the staff never break character, but work hard to maintain the illusion that we are all on real space cruise. Besides humans, the “crew” is also staffed with aliens and robots – two essential aspects of Star Wars theming.

In line with the cruise experience, you even do a one-day excursion to a nearby alien planet. I’ve had trouble writing about this experience because it felt highly personal, lighting off areas of my child brain that were set aside for space travel fantasies. At the same time, it is also very nerdy, and the intersection of the highly nerdy and the highly personal is dangerous territory. Nevertheless, it being May 4th today, I felt I could not longer put it off.

How you do Immersion?

“Immersion” is the touchstone for what people and tech companies are calling the Metaverse. Part of this is a carry over from VR pitching, and was key to explaining why being inside a virtual reality experience was different and better than simply playing a 3D video game with a flat screen and a controller.

But the term “immersion” hides as much as it reveals. How can “immersion” be a distinguishing feature of virtual reality when it is already a built-in aspect of real reality? What makes for effective immersion? What are the benefits of immersion? Why would anyone pay to be immersed in someone else’s reality? Is immersion a way to telling a story or is storyline a component of an immersive experience?

A Russian doll aspect of the starcruiser is the “Simulation Room” which, in the storyline of the ship, is an augmented area in that recreates the climate of the planet the ship is headed toward. The room is equipped with an open roof which happens to perfectly simulate the weather in central Florida. The room also happens to be where the Saja scholars provide instruction on Jedi history and philosophy.

Space Shrimp (finding the familiar in the unfamiliar)

I’m the sort of person who finds it hard to every be present in the moment. I’m either anticipating and planning for the next day, the next week, the next few years, or I am reliving events from the past which I wish had gone better (or wish had never happened at all).

For two and a half days on this trip, I was fully captivated by the imaginary world I was living through. There wasn’t a moment after about the first hour when I was thinking about anything but the mission I was on and the details of the world I was in. I didn’t feel tempted to check my phone or know what was happening in the outside world.

An immersive experience, it seems to me, is one that can make you forget about the world in this way, by replacing it with a more captivating world and not letting go of you. I’ve been going over in my head the details of the star wars experience that make this work and I think the blue shrimp we had for dinner one night is the perfect metaphor for how Disney accomplishes immersion.

To create immersion, there can be nothing that references the outside world. The immersive experience must be self-contained and everyone in the immersive experience, from cabin boy to captain, must only reference things inside the world of the starcruiser. Fortunately Star Wars is a pre-designed universe. This helps in providing the various details that are self-referential and remind us of the world of the movies rather than the world of the world.

A great example of this is the industrial overhead shower spout and the frosted glass sliding shower door in our cabin. They are small details but harken back to the design aesthetic of the star wars movies, which contain, surprisingly, a lot of blue tinted frosted glass.

This extends to the food. All the food is themed, in a deconstructionist tour de force, to appear twisted and alien. We drank blue milk and ate bantha steaks. We feasted on green milk and salads made from the vegetation found on the planet Falucia.

And here there is a difficulty. Humans have a built-in sense of disgust of strange foods that at some point protected our ancestors from accidentally poisoning themselves. And so each item of food had to indicate, through appearance or the name given on the menu, what it was an analog of in the real world. I often found myself unable to enjoy a dish until I could identify what it was meant to be (the lobster bisque was especially difficult to identify).

What I took from this was that for immersion to work, things have to be self-referential but cannot be totally unfamiliar. As strange as each dish looked, it had to be, like the blue shrimp, analogous with something people knew from the real world outside the ship. Without these analogical connections, the food will tend to create aversion and anxiety instead of the sense of immersion intended.

One take way is that as odd as the food sometimes looked, the food analogs were always meals familiar to Americans. Things common to other parts of the world, like chicken feet or durian fruit or balut, would not go over well even though they taste good (to many people).

A second take away is that the galactic food has to be really, really good. In modern American cuisine, it is typical to provide the story behind the food explaining each ingredient’s purpose, where it comes from and how to use it in the dish (is it a salad or a garnish?). The galactic food can’t provide these value-add story points and only has fictitious ones.

In the case of the food served on the starcruiser, then, each dish has to stand on its own merits, without the usual restaurant storytelling elements that contribute to the overall sense that you are eating something expensive and worthy of that expense. Instead, each dish requires us to taste, smell, and feel the food in our mouths and decide if we liked it or not. I don’t think I’ve ever had to do that before.

World building – (decrepit futurism)

The world of Star Wars is one of decrepit futurism. It is a world of wonders in decline.

There are other kinds of futurism like the streamlined retro-futurism of the 30s and 50s or contemporary Afro-futurism. The decrepit futurism of Star Wars takes a utopic society and dirties it up, both aesthetically and morally. The original Star Wars starts off at the dissolution of the Senate marking a political decline. George Lucas doubles down on this in the prequels making this also a spiritual decline in which the Jedi are corrupted by a malignant influence and end up bringing about the fall of their own order. The story of the sequels (which is the period in which the galactic space voyage takes place) is about the difficultly and maybe impossibility of restoring the universe to its once great heights.

As beautiful and polished as all the surfaces are on the star cruiser, the ship is over 200 years old and has undergone massive renovations. Despite this, the engines continue to provide trouble (which you get to help fix). Meanwhile, the political situation in the galaxy in general and on the destination planet in particular is fraught, demanding that voyagers choose which set of storylines they will pursue. Will they help the resistance or be complicit with the First Order? Or will they opt out of this choice and instead be a Han Solo-like rogue pursuing profit amid the disorder?

The metaphysics of the Star Wars universe is essentially fallibilist and flawed – which in turn opens the way for moral growth and discovery.

The decrepit futurism of Star Wars has always seemed to me to be one of the things that makes it work best because it artfully dodges the question of why things aren’t better in a technologically advanced society. Decrepit futurism says that things once were (our preconceptions of what the future and progress entails is preserved) but have fallen from the state of grace through a Sith corruption. In falling short, the future comes down to the level where the rest of us live.

It’s also probably why Luke, in the last trilogy, never gets to be the sort of teacher we hoped he would be to Rey. The only notion we have of the greatness and wisdom of a true Jedi master comes from glimpses we get through Yoda in The Empire Strikes Back, but he is only able to achieve this level of wisdom by losing everything. Greatness in Star Wars is always something implied but never seen.

Storytelling (narrative as an organizing principle)

Much is made of storytelling and narrative in the world of immersive experiences. Some people talk as if immersion is simply a medium for storytelling – but I think it is the other way around. Immersion is created out of world building and design that distract us from our real lives. The third piece of immersion is storytelling.

But one thing I discovered on the Galactic Starcruiser is that the stories in an immersive experience don’t have to be all that great – they don’t have to have the depth of a Dostoevsky novel. Instead they can be at the level of a typical MMORPG. They can be as simple as go into the basement and kill rats to get more information. Hack a wall terminal to get a new mission. Follow the McGuffin to advance the storyline.

Narrative in an immersive experience is not magic. It’s just a way of organizing time and actions for people, much the way mathematical formulas organize the relationship between numbers or physics theorems organize the interactions of physical bodies. Narratives help us keep the thread while lots of other things are going on around us.

The main difficulty of a live theater narrative, like the one on the starcruiser, is that the multiple story lines have to work well together and work even if people are not always paying attention or even following multiple plots at the same time. Additionally, at some point, all of the storylines must converge. In this case, keeping things simple is probably the only way to go.

Crafting a narrative for immersive experiences, it seems to me, is a craft rather than an art. It doesn’t have to provide any revelations or tell us truths about ourselves. It just has to get people from one place in time to another.

The real art, of course, is that exercised by the actors who must tell these stories over and over and improvise when guests throw them a curve ball while keeping within the general outline of the overarching narrative. And being able to do this for 3 days at a time is a special gift.

Westworld vs the Metaverse (what is immersion)

Using the Galactic Starcruiser as the exemplar of an immersive experience, I wanted to go back to the question of how immersion in VR is different from immersion in reality. To put it another way, what is the difference between Westworld and the Metaverse?

There seems to be something people are after when they get excited about the Metaverse and I think it’s at bottom the ability to simulate a fantasy. Back when robots were all the rage (about the time Star Wars was originally made in the 70s) Michael Crichton captured this desire for fantasy in his film Westworld. The circle of reference is complete when one realizes that Chrichton based his robots on the animatronics at Disneyland and Disneyworld.

So what’s the difference between Westworld and the Metaverse? One of the complaints about the Metaverse (and more specifically VR) is that the lack of haptic feedback diminishes the experience. The real world, of course, is full of haptic feedback. More than this, it is also full of flavors and smells, which you cannot currently get from the Metaverse. It can also be full of people that can improvise around your personal choices so that the experience never glitches. This provides a more open world type of experience, whereas the Metaverse as it currently stands will have a lot of experiences on rails.

From all this, it seems as if the Metaverse aspires to be Westworld (or even the Galactic Starcruiser) but inevitably falls short sensuously and dynamically.

The outstanding thing about the Metaverse, though, is that it can be mass produced – precisely because it is digital and not real. The Starcruiser is prohibitively expensive dinner theater which I was able to pull off through some dumb luck with crypto currencies. It’s wonderful and if you can afford it I highly encourage you to go on that voyage into your childhood.

The Metaverse, on the other hand, is Westworld-style immersion for the masses. The bar to entry for VR is relatively low compared to a real immersive experience. Now all we have to do is get the world building, design, and storylines right.

Simulations and Simulacra

March 21, 2021August 5, 2022 James Ashley

In a 2010 piece for The New Yorker called Painkiller Deathstreak , the novelist Nicholson Baker reported on his efforts to enter the world of console video games with forays into triple-A titles such as Call of Duty: World at War, Halo 3: ODST, God of War III, Uncharted 2: Among Thieves, and Red Dead Redemption.

“[T]he games can be beautiful. The ‘maps’ or ‘levels’—that is, the three-dimensional physical spaces in which your character moves and acts—are sometimes wonders of explorable specificity. You’ll see an edge-shined, light-bloomed, magic-hour gilded glow on a row of half-wrecked buildings and you’ll want to stop for a few minutes just to take it in. But be careful—that’s when you can get shot by a sniper.”

In his journey through worlds rendered on what was considered high-end graphics a decade ago, Nicholson discovered both the frustrations of playing war games against 13 year olds (currently they would be old enough to be stationed in Afghanistan) as well as the peace to be found in virtual environments like Red Dead Redemption’s Western simulator.

“But after an exhausting day of shooting and skinning and looting and dying comes the real greatness of this game: you stand outside, off the trail, near Hanging Rock, utterly alone, in the cool, insect-chirping enormity of the scrublands, feeling remorse for your many crimes, with a gigantic predawn moon silvering the cacti and a bounty of several hundred dollars on your head. A map says there’s treasure to be found nearby, and that will happen in time, but the best treasure of all is early sunrise. Red Dead Redemption has some of the finest dawns and dusks in all of moving pictures.”

I was reminded of this essay yesterday when Youtube’s algorithms served up a video of Red Dead Redemption 2 (the sequel to the game Nicholson wrote about) being rendered in 8K on an NVidia 3090 graphics card with raytracing turned on.

The encroachment of simulations upon the real world, to the point that they not only look as good as the real world (real?) but in some aspects even better, has interestingly driven the development of the sorts of AI algorithms that serve these videos up to us on our computers. Simulations require mathematical calculations that cannot be done as accurately or as fast on standard CPUs. This is why hardcore gamers pay upwards of a thousand dollars for bleeding edge graphics cards that are specially designed to perform floating point calculations.

These types of calculations, interestingly, are also required for working with large data sets for machine learning. The algorithms that steer our online interests, after all, are just simulations themselves, designed to replicate aspects of the real world in order to make predictions about what sorts of videos (based on a predictive model of human behavior honed to our particular tastes) are most likely to increase our dwell time on Youtube.

Simulations, models and algorithms at this point are all interchangeable terms. The best computer chess programs may or may not understand how chess players think (this is a question for the philosophers). What cannot be denied is that they adequately simulate a master chess player that can beat all the other chess players in the world. Other programs model the stock market and tune them back into the past to see how accurate they are as simulations, then tune them into the future in order to find out what will happen tomorrow – at which point we call them algorithms. Like memory, presence and anticipation for us meatware beings, simulation, model and algorithm make up the false consciousness of AIs.

Simulacra and Simulation, Jean Baudrillard’s 1981 treatise on virtual reality, opens with an analysis of the George Luis Borges short story On Exactitude in Science, about imperial cartographers who strive after precision by creating ever larger and larger maps, until the maps eventually achieve a one-to-one scale, becoming exact even as they overtake their intended purpose.

“The territory no longer precedes the map, nor survives it. Henceforth, it is the map that precedes the territory – precession of simulacra – it is the map that engenders the territory and if we were to revive the fable today, it would be the territory whose shreds are slowly rotting across the map. It is the real, and not the map, whose vestiges subsist here and there, in the deserts which are no longer those of the Empire, but our own. The desert of the real itself.”

I was thinking of Baudrillard and Borges this morning when, by coincidence, Youtube served up a video of comparative map sizes in video games. Even as rendering versimilitude has been one way to gauge the increasing realism of video games, the size of game worlds has been another. A large world provides depth and variety – a simulation of the depth and happenstance we expect in reality – that increases the immersiveness of the game.

Space exploration games like No Man’s Sky and Elite Dangerous attempt to simulate all of known space as your playing ground, while Microsoft’s Flight Simulator uses data from Bing Maps to allow you to fly over the entire earth. In each case, the increased size is achieved by surrendering on detail. But this setback is temporary, and over time we will be able to match the extent of these simulations with detail, also, until the difference between the real and the model of the real is negligible.

One of the key difficulties with VR adoption (and to some extent the superiority of AR) is the falling anxiety everyone experiences as they move around in virtual reality. The suspicion that there are hidden objects in the world that the VR experience does not reveal to us prevents us from being fully immersed in the game – except in the case of the highly popular horror genre VR games in which inspiring anxiety is a mark of success. As the movements continue to both increase the detail of our simulations of the real world – to the point of simulating the living room sofa and the kitchen cabinet – and expand the coverage of our simulations across the world so there is no surveillable surface that can escape the increasing exactness of our model, we will eventually overcome VR anxiety. At that point, we will be able to walk around in our VR goggles without ever being afraid of tripping over objects, because there will be a one-to-one correspondence between what we see and what we feel. AR and VR will be indistinguishable at that exacting point, and we will at last be able to tread upon the sands of the desert of the real.

When GameStop Killed XBox One Kinect

February 2, 2021August 5, 2022 James Ashley2 Comments

If you look up the Xbox One Kinect (informally known as the Kinect 2) on the GameStop website, you’ll read in the product description that “[t]he best Xbox One experience is with Kinect.”

Over the course of the Xbox One’s life, there were approximately 38 games that supported Kinect body tracking. None of them were triple-A games. This is out of 2682 games for the Xbox One. While Microsoft initially planned to require that the Kinect be always on, by the time of the Xbox One’s release on November 2013, this requirement was removed. By the summer of 2014, Microsoft unbundled the Kinect from their game console, allowing people to purchase the Xbox One at a lower price point that was more competitive with PlayStation 4. The final blow came in late 2015, when Microsoft removed their Kinect support for navigating the Xbox dashboard.

Before going into some theories on what happened to the Kinect, I wanted to give my “they’re all dirty” metaphor for the recent rise and fall of the GameStop stock price. The weak GameStop business was being shorted by hedge funds. Small investors gathered on Reddit decided to fight this by pumping money into GameStop stocks in order to inflate the price artificially. They typically used the app Robinhood, which doesn’t charge trading fees, to do this. In the end, the hedge funds appear to have hedged their best, because even as they lost money on their shorts, they made money by fulfilling the trades coming through Robinhood from these reddit investors.

Isn’t this the plot of Mel Brooks’ The Producers? While the purpose of the stock market is supposed to be efficiently moving investor money into the hands of companies in order to create value, short-selling is a speculative financial instrument to allow people to bet that certain companies will fail. Like Leo Bloom, hedge funds like Melvin Capital and Citadel recognized that sometimes you can make more money with a failed venture than with a successful one.

In order to improve the odds of failure, Leo Bloom and Max Bialystock stack the deck by finding the worst script, the worst director and the worst cast for their Broadway show. Similarly, in order to improve the odds of driving down the price of GameStop stock, Citadel let people know that they were shorting the stock. Who would invest in a company that Wall Street big guns were trying to destroy?

The problem for The Producers is that the worst play, Springtime for Hitler, the worst director (who turned it into a Busby Berkeley style musical), and the worst cast (drugged addled hippies), come together to create something that people can enjoy ironically. The play is so bad, it is good.

The worst director and worst cast in the GameStop saga are the Robinhood app and the reddit community /wallstreetbets. Robinhood allows (and encourages) inexperienced investors to bet against Wall Street professionals, which is about as successful as betting against the house in Las Vegas. /wallstreetbets, in turn, allows users to try out betting systems. The latest one depends on treating the stock market ironically, assuming that investment is primarily about manipulating markets rather than finding good companies to invest in. The only difference between /wallstreetbets and the hedge funds, is that one is made up of market outsiders and the other by insiders. Late capitalism. Post-truth investment.

There was a time when GameStop wasn’t just a carcass being fought over by carrion feeders looking for a quick meal. In 2013, GameSpot was a quickly growing company that made its money reselling second-hand console game disks.

In the lead up to the release of the XBox One, it turns out that Microsoft was attempting to kill this aftermarket. Even into the middle of 2013, Microsoft was considering dropping the optical drive from its hardware altogether and making the purchase of games completely cloud-based, like Steam.

It is clear from the confusion around the May, 2013 Xbox One reveal that this idea had lingering ramifications for the strategy around connectivity. Two requirements for a digital only game distribution system are a need for all consoles to be online, at least part of the time, and complex digital licensing verification systems. It turned out that the aftermarket in video games, brokered through third-parties like GameStop, was a much bigger deal than Microsoft realized and their inability to explain how people would be able to exchange and sell used games inspired one of the great marketing trolls of all time, when Sony created a commercial demonstrating how to exchange PlayStation games.

Today any teenager can explain to you the market forces that are destroying GameStop’s business model. There is no need for a company to provide an aftermarket for video games when no one uses disks anymore. Everything is digital in 2021 and everything is online. Almost like an act of revenge for 2013, Microsoft is even strong arming its Microsoft Gold subscribers to upgrade to the Xbox Game Pass by raising prices for the former. Xbox Game Pass allows users to have access to a broad range of games without having to buy those games individually, including the top games from the past two to three years.

Microsoft was ahead of its time in 2013. But what made it want to get rid of disks? One theory is that without a disk drive, Microsoft would have been able to drop the launch price of its console by $50. As it turned out, the Xbox with a disk drive and bundled with an Xbox Kinect, brought the initial price of an Xbox One to $499. The Sony PlayStation 4 launched at a $399 price point.

This one hundred dollar difference turned out to be nearly fatal for the Xbox, which was forced to unbundle the Kinect 2 from its Xbox One by the middle of 2014, finally making their console competitive on price with the PlayStation. It was even able to undercut the price of the PlayStation by selling an unbundled Xbox One for $349 shortly after. This suggests that without an optical drive, the Xbox might have sold for only $50 more than the PlayStation 4, or even for the same price, at launch, while including a key differentiator with the Kinect.

Why did Microsoft insist on bundling the Kinect with the Xbox One in the first place? The problem for Microsoft was that in order to make the Kinect successful, it needed triple-A game companies to create games that used it. But this entails extra design and development costs for game companies. There is no way they would take on this additional cost without a guarantee of a user base that owned Kinect devices. There was a virtuous circle – or perhaps a vicious one – in which game makers need players with Kinects before they will create games for the Kinect, while console buyers need to be shown games that highlight the Kinect before they will buy a console that requires them to buy a Kinect. In the end, neither of these things happened.

There was an underlying reason that Microsoft wanted to get Kinects into consumer living rooms. While the Kinect’s primary feature is its body tracking, which could be used as a controller for playing games and navigating screens, it’s secondary feature is a directional microphone plugged into Microsoft’s cutting edge speech recognition. It could have become an essential interface between consumers and the commercial internet, with Microsoft as the essential broker for these transactions and interactions.

As usual Microsoft was ahead of its time, and even as it quickly killed the Kinect in 2014, Amazon was releasing its own natural language devices built around Alexa, which soon expanded into a tool for not only accessing data on the internet, but also for integrating with services and controlling home devices.

But alas, GameStop created an aftermarket for game disks, that prevented Microsoft from getting rid of its Xbox One optical drive, that caused the Xbox One to lose on price to the PlayStation 4, that caused the XBox to drop the Kinect, that caused Microsoft to cede the living room device market to Amazon.