Gestural Grammatology – The Imaginative Universal

Before Ubiquity

October 5, 2023October 13, 2023 James Ashley1 Comment

ubiquitous AR

I’ve been developing for augmented reality head-mounted displays for about eight years. I first tried the original HoloLens in 2015. Then I got to purchase a developer unit in 2016 and started doing contract work with it. Later I was in the early pre-release dev program for Magicleap’s original device, which led to more work on the HoloLens 2, Magicleap 2, and Meta Quest Pro. I continue to work in AR HMDs today.

The original community around the HoloLens was amazing. We were all competing for the same work, but at the same time, we only had each other to turn to when we needed to talk to someone who understood what we were going through. So we were all sort of frenemies, except that because Microsoft was notoriously tight-lipped with their information about the device, we helped each other out on difficult programming tricks and tricky AR UI concepts – and this made us friends as well.

In those early days, we all thought AR, and our millions in riches (oh what a greedy lot we were), were just around the corner. But we never quite managed to turn that corner. Instead we had to begin devising theories around what that corner was going to look like, what the signs would be as we approached that corner, and what would happen after we made the corner. Basically, we had to become more stringent in our analyses.

Out of this, one big idea that came to the fore was “AR Ubiquity”. This comes out of the observation that monumental technological change happens slowly and incrementally, until it suddenly happens all at once. So at some point, we believe, everyone will just be wearing AR headsets instead of carrying smartphones. (This is also known, in some quarters, as the “Inflection Point”.)

Planning, consequently, should be based less on how we get to that point, or even when it will happen; and more about how to prepare for “AR Ubiquity” and what we will do afterwards. So AR Ubiquity, in this planning model, can come in 3 years, or in 7 years, or maybe for the most skeptical of us in 20 years. It doesn’t really matter because the important work is not in divining when it will happen (or even who will make it happen) but instead in 1) what it will look like and 2) what we can do — as developers, as startups, as corporations — once it arrives.

meta-history

Once we arrive at a discourse about the implications of “AR Ubiquity” rather than trying to forecast when it will happen, we are engaging with a grand historical narrative about the transformative power of tech – which is a happy place for me because I used to do research on philosophical meta-history in grad school – though admittedly I wasn’t very good at it.

“AR Ubiquity”, according to the tenets of meta-history, can at the same time both be a theory about how the world works and also a motif in a story about how we fit into the technological world. Both ways of looking at it can provide valuable insights. As a theory we want to know how we can verify (or falsify) it. As a story element, we want to know what it means. In order to discover what it means, in turn, we can excavate it for other mythical elements it resembles and draws upon. (Meta-history, it should be acknowledged, can lead to bad ideas when done poorly and probably worse ideas when it is done well. So please take this with a grain of salt (a phrase which itself has an interesting history, it is worth noting).

I can recall three variations on the theme of disruptive (or revolutionary) historical change. There’s the narrative of the apocalyptic event that you only notice once it has already happened. There’s the narrative of the prophesied event that never actually happens but is always about to. And then there’s the heralded event, which has two beats: one to announce that it is about to happen, and another when it does happen. We long thought AR would follow model A, it currently looks like it is following model B, and I hope it will turn out that we are living through storyline C. Let’s unpack this a bit.

A or B

Model A

Apocalyptic history, as told in zombie movies and TV shows, generally have unknown origins. The hero wakes up after the fateful event has already happened, often in a hospital room, and over the course of the narrative, she may or may not discover whether it was caused by a sick monkey who escaped a viral lab, or by climate change, or by aliens. There’s also the version of apocalyptic history that circulates in Evangelical Christian eschatology known as The Rapture. In the Book of Revelations (which New Yorker writer Adam Gopnik calls the most cinematic Michael Bey ready book of the Bible), St. John of Patmos has a vision in which 144,000 faithful are taken into heaven. In popular media, people wake up to find that millions of the virtuous elect have suddenly disappeared while they have been left behind to try to pick up the pieces and figure out what to do in a changed world.

In the less dramatic intellectual upheavals described in Thomas Kuhn’s The Structure of Scientific Revolutions, you can start your scientific career believing that an element known as phlogiston is released during combustion, then end it believing that phlogiston doesn’t exist and instead oxygen is added to a combusted material when it is burned (it is oxidized). Or you might start believing that the sun revolves around the earth and end up a few decades later laughing at such beliefs, to the point that it is hard to understand how anyone ever believed such outlandish things in the first place.

It’s a bit like the way we try to remember pulling out paper maps to drive somewhere new in our car, or shoving cassette tapes into our Walkmans, or having to type on physical keys on our phones – or even recalling phones that were used just for phone calls. It seems like a different age. And maybe AR glasses will be the same way . One day it seems fantastical and the next we’ll have difficulty remembering how we got things done with those quaint “smart” phones before we got our slick augmented reality glasses.

waiting for godot

Model B

The history of waiting might best be captured by the term Millennialism, which describes both Jewish and Christian belief in the return of a Messiah after a thousand years. The study of millennialist beliefs often cover both the social changes that occur in anticipation of a Millennialist event as well as the consequent recalculation of calendars that occurs when an anticipated date has passed and finally the slow realization that nothing is going to happen, after all.

But there are non-theistic analogs to Millennialism that share some common traits such as the Cargo Cult in Fiji or later UFO cults like the Heaven’s Gate movement in the 90’s. Marxism could also be described as a sort of Millenarist cult that promised a Paradise that adherents came to learn would never arrive. One wonders at what point, in each of these belief systems, people first began to lose faith and then decided to simply play along while lacking actual conviction. The analogy can be stretched to belief in concepts like tulip bulb mania, NFTs, bitcoin, and other bubble economies where conviction eventually becomes less important than the realization that everyone else is equally cynical. In the end, it is cynicism that maintains economic bubbles and millenarist belief systems rather than faith.

I don’t think belief in AR Ubiquity is a millenarist cult, yet. It certainly hasn’t reach the stage of widespread cynicism, though it has been in a constant hype cycle over the past decade as new device announcements serve to refresh excitement about the technology. But even this excitement is making way, in a healthy manner, for a dose of skepticism over the latest announcements from Meta and Apple. There’s a hope for the best but expect the worst attitude in the air that I find refreshing, even if I don’t subscribe to it, myself.

Model C

The last paradigm for disruptive history comes in the form of a herald and the thing he is the herald for. St. John the Baptist is the herald of the Messiah, Jesus Christ, for example. And Silver Surfer is the herald for Galactus. One is a forerunner for good tidings while the other is a harbinger of doom.

The forerunner isn’t necessary for the revolutionary event itself. That will happen in any case. The forerunner is there to let us know that something is coming and to point out where we should be looking for it. And there is just something very human about wanting to be aware of something before it happens so we can more fully savor its arrival.

Model C is the scenario I find most likely and how I imagine AR Ubiquity actually happening.

First we’ll have access to a device that demonstrates an actually useable AR headset with actually useful features. This will be the “it’s for real” moment that dispels the millenarist anxiety that we’re all being taken for a ride.

The “it’s real” moment will then set a bar for hardware manufacturers to work against. The forerunner device becomes the target all AR HMD companies strive to match, once someone has shown them what works, and within a few years we will have the actual arrival of AR Ubiquity.

At this time, reviews of the Apple Vision Pro and the Meta Quest 3 suggest that either could be this harbinger headset. I have my doubts about the Meta Quest 3 because I’m not sure how much better it can be than the MQ2 and the Meta Quest Pro, especially since it has removed eye tracking, which was a key feature of MQP and made the hand tracking more useful.

The AVP, on the other hand, has had such spectacular reviews that one begins to wonder if it isn’t too good to be true.

But if the reviews can be taken at face value, then AR Ubiquity, or at least a herald that shows us it is possible, might be closer than we think.

I’m just proposing a small tweak to the standard model of how augmented reality headsets will replace smartphones. We’ve been assuming that the first device that convinces consumers to purchase AR headsets will also immediately set off this transition from one device category to the other. But maybe this is going to occur in two steps. First a headset will appear that validates the theory that headsets can replace handsets. Then a second device will rotate the gears of history and lead us into this highly anticipated new technological age.

Unmasking, Optics, and Surveillance 1/n

January 27, 2021February 4, 2021 James Ashley

How do you deal with people who refuse to wear masks?

According to fedscoop, tracking down rioters from the January 6th Capitol invasion will be easy due to three reason:

rioters typically didn’t wear masks
rioters photographed, videoed, and streamed their insurrection
surveillance software is extremely good at analyzing photographs and videos for facial matches
(as an aside, facial recognition software is better with white faces than with minority faces. the overwhelming majority of the rioters were white – and men.)

One way to make sense of this is to realize that masking has taken on mythic overtones in America’s culture wars and the Trump supporters who came to attend rallies in the capital, before they became rioters in the Capitol, are anti-mask. Then when they became a mob and invaded the home of the legislative branch of government, they simply didn’t have masks on them.

On the other hand, the rioters seemed anxious to be seen, livestreaming what they perceived as a revolution as it was occurring. If there was no COVID, it seems likely the rioters would have done the same thing and, potentially, there was more masking than there would have otherwise been because of the pandemic.

There are then two plausible reasons rioters didn’t wear masks. First, the rioting was a surprise to most of them and most of them hadn’t known that they would end up breaking the law. Second, they didn’t see themselves as breaking the law, but thought they were on the same side as the police, the president, and other lawful authorities.

At some point, not wearing COVID masks overlaps with not wearing criminal masks, the first from the belief that COVID is not real and the second out of the belief that breaking into the Capitol is not a crime. But surely, deep inside, there is the suspicion for these people that both the disease and the crime are real.

This inherent conflict between wanting to hide our true selves while also wanting to reveal ourselves online is at the heart of the societal changes driven by social media like Twitter and Facebook. We know that these companies make their money by surveilling our online behavior and selling our information. Yet we see this as a fair trade because they give us the ability to be heard and connect with other people who think like us.

The structural artifact created is that unwanted surveillance is inextricable from the opportunity for identitarian expression.

For Capitol rioters, being observed is the natural corollary to being observed.

Due to the bad optics of the rioting of the U.S. Capitol, some Trump supporters are now disavowing the rioters and attempting to unmask them as Antifa agents pretending to be militia/3 percenters/bougaloo bois/ proud bois/ white supremacists.

In this final turn, the ideology critique tradition that runs through Nietzsche, Freud, Marx, critical theory and eventually critical race theory, reaches an apex of sorts – unmasking as a tactic for erasing one’s tracks, even when everything has been caught on film.

In 1983 David Copperfield made the Statue of Liberty disappear on live television. It was similar to many other disappearing tricks he had performed over the years, but the scale and the fact that it was being filmed made it seem all the more inexplicable. According to some debunkers, however, the fact that it was filmed, and that we all have a bias toward believing what we see with our own eyes, made it actually easier for Copperfield to create his illusion.

As a software developer working with virtual reality, computer vision and artificial intelligence, and also as a former philosophy student, the intersection of these three themes, unmasking, optics and surveillance, are a rich mine for me. In the next few days I want to take each of these concepts apart philosophically and historically, in isolation and in relation to each other, and destrukt them to see what falls out. I want to address Kant’s distinction between the private and public spheres in What Is Enlightenment? while also covering the role of the unmasking motif in Scooby-Doo, naturlich. I want to dig into why magicians never reveal their tricks and why politicians never admit they are wrong. Along the way, if I am feeling particularly self-destructive, I want to touch on Critical Race Theory, cancel culture, right wing safe spaces, the politics of personal destruction, nuclear options and redemption through art vs salvation through politics.

Philosophical Classics for Nerds

January 4, 2021March 22, 2021 James Ashley

It is 60 days after the day I thought the U.S. presidential election would have been settled … and yet. Intellectually, I recognize the outrageousness of the situation, based on the Constitution, based on my high school civics lessons, and based on my memories of the 2000 presidential election between Bush and Gore when everyone felt that any wrong move or overreach back then would have threatened the stability of the republic and the rule of law.

At the same time I have become inured to the cray-cray and as I listen today to recordings of President Trump’s corrupt, self-serving call to Georgia Secretary of State Raffensperger, I find that my intellectual recognition that norms are being broken (the norm that we trust the democratic system, the norm that we should all abide by the rules as they are written , the norm that we should assiduously avoid tampering with ‘the process’ in any way) is not accompanied by the familiar gut uneasiness that signals to humans that norms have been disturbed. That thing that makes up “common sense”, a unanimity between thought and feeling, is missing for me due to four and more years of gaslighting.

When common sense breaks down in this way, there are generally two possible causes. Either you have gone crazy or everyone else has. Like Ingrid Bergman in that George Cukor film, our first instinct is to look for a Joseph Cotton to reassure us that we are right and Charles Boyer is wrong. What always causes me dread about that movie, though, is the notion that things wouldn’t have gone so well had Ingrid Bergman not been gorgeous and drawn Cotton’s gaze and concern.

In another film from a parallel universe, Cotton might have ignored Bergman, and she would have withdrawn from the world, into herself, and pursued a hobby she had full control over, like crochet, or woodworking, or cosplaying. Many do.

Over the past two decades, nerdiness has shifted from being a character flaw into a virtue, from something tacitly acknowledged into a lifestyle to be pursued. The key characteristic of “nerdiness” is the willingness to allow a passion to bloom into an obsession to the point of wanting to know every trivial and quadrivial aspect of a subject. True nerdiness is achieved when we take a matter just that bit too far, when friendships are broken over opinions concerning the Star Wars prequels, or when marriages are split over the classification of a print font.

The loss of the sensus communis can also mark the point where mere thought becomes philosophical. The hallmark of philosophical reflection is that moment when the familiar suddenly becomes unfamiliar and then demands our gaze with new fascination, like Ingrid Bergman suddenly drawing Joseph Cotton’s attention. For Heidegger this was the uncanniness of the world. For Husserl it was the epoche in which we bring into question the givenness of the world. And for Plato it is the desire for one’s lover, which one transfers to beauty in general, and finally to Truth itself.

Philosophers as a rule take things too far. They say forbidden things. They draw unexpected conclusions. They examine all the nooks and crannies of thought, exhaustively, to reach the conclusions they reach, often to the boredom of their audience. They were nerds before we knew what nerds were.

Even in the world of philosophy, however, there are books and ideas that used to be considered too important to overlook but too nerdy to be made central to the discipline. Instead, they have existed on the margins of philosophy waiting for a moment when the Zeitgeist was ready to receive them.

Here are five works of speculative philosophy whose time, I believe has come.

Simulacra and Simulations by Jean Baudrillard – This book describes virtual reality, a bit like William Gibson did with Neuromancer, before it was really a thing. The Wachowskis cite it as an inspiration for The Matrix and even put phrases from this work in one of Morpheus’s monologues. It is blessedly a short work that captures the essence of our virtual world today from a distance of almost half a century (it was written in 1983). No one should be working in tech today without understanding what Baudrillard meant by “the desert of the real.”

Reasons and Persons by Derek Parfit – Parfit took apart the notion of identity using thought experiments drawn from science fiction. One of his most striking arguments, introduced in Part Three of his work, in a section called Simple Teletransportation and the Branch-Line Case, Parfit posits a machine that allows speed-of-light travel by scanning a person into data, sending that data to another planet, and then reconstituting that data as matter to recreate the original person. Of course, we have to destroy the original copy during this process of teletransportation. Parfit toys with our intuitions of what it means to be a person in order to arrive at philosophical gold. If the reader is troubled by this scenario of murder and cloning cum teleportation, Parfit is able to point out that this is what we go through in our lives. How much of the matter we were born with is still a part of our physical bodies? Little to none?

For the coup de gras, one can apply the lessons of teletransportation to address our pointless fear of death. What is death, after all, but a journey through the teleporter without a known terminus?

The Conscious Mind by David J Chalmers – Just as the 4th century BCE saw a flourishing of philosophy and science in Greece, or the 16th century saw an explosion of literary invention in England, in the 1990’s Australia become the home of the most innovative works on the Philosophy of Mind in the world. Out of that period of wild genius David J Chalmers came out against the general trend driven by Daniel Dennett and Paul Churchland that denied the reality of consciousness. Chalmers, on the other hand, made the case through exacting arguments that consciousness is not only real, but is a fundamental property of the universe, alongside spatiality and temporality.

Hegel and the Metaphysics of Absolute Negativity by Brady Bowman – Since Dale Carnegie’s important work reforming the habits of white collar labor, positive thinking has been the ethos of professional life. The Marxian threat of alienated labor is eliminated by refusing to acknowledge the possibility of alienation in the corporate managerial class. Just as movie Galadriel tells us that “history became legend, legend became myth”, the power of positive thinking became a tenet of faith, then a method of prosperous Biblical exegesis, and finally a secret.

Do you ever get tired of mindless positivism? What if the underlying engine of the universe turns out not to be positive thinking but absolute negativity? And what if this can be proven through Hegel’s advanced dialectical logic? How much would you pay for a secret like that?

The Emperor’s New Mind by Roger Penrose – Penrose was a brilliant mathematical physicist who unleashed his learned background to the problem of human consciousness. Do physics and quantum physics in particular confirm or reject our theories about the human soul? I’ve always loved this book because Penrose comes up with a solution to human consciousness in a somewhat unphilosophical way – which made many philosophers nervous. The crux of his argument for the place of mind in a quantum universe is the size horizon of some features of the human brain. Ultimately, I think, Penrose provides a way to reconcile Kantian metaphysics with modern cutting edge physics and biology in a way that works – or that at least is consistent and the ground for the possibility of Kantianism.

Honorable mention: Darkside by Tom Stoppard – if you have been watching The Good Place then you should be familiar with The Trolley Problem, a thought experiment used to tease our ethical intuitions and commitments. What could make The Trolley Problem even better? What if it is incorporated into a radio play by one of our greatest living English dramatists, performed to the tracks of Pink Floyd’s Dark Side of the Moon, and acted out by Bill Nighy (The Hitchhiker’s Guide to the Galaxy), Rufus Sewell (Dark City) and Iwan Rheon (Game of Thrones).

Multi-modal User Input for Spatial Computing

August 31, 2020March 23, 2021 James Ashley1 Comment

One of the fights that I thought we had put behind us is over the question ‘which interface is better?’ For instance, this question the was frequently brought up in comparisons of the mouse to the keyboard, its putative precursor. The same disputes came along again with the rise of natural user interfaces (NUI) when people began to ask if touch would put the mouse out of business. Always the answer has been no. Instead, we use all of these input modes side-by-side.

As Bill Buxton famously said, every technology is the best at something and the worst at something else. We use the interface best adapted to the goal we have in mind. In the case of data entry, the keyboard has always been the best tool. For password entry, on the other hand, while we have many options, including face and speech recognition, it is remarkable how often we turn to the standard keyboard or keypad.

Yet I’ve found myself sucked into arguments about which is the best interaction model, the HoloLens v1’s simple gestures, the Magic Leap One’s magnetic 6DOF controller, or the HoloLens v2’s direct manipulation (albeit w/o haptics) with hand tracking.

Ideally we would use them all. A controller can’t be beat for precision control. Direct hand manipulation is intuitive and fun. To each of these I can add a blue tooth XBox controller for additional freedom. And the best replacement for a keyboard turns out to be a keyboard (this is known as the Qwerty’s universal constant).

It was over two years ago at the Magic Leap conference that James Powderly, a spatial computing UX guru, set us on the direction of figuring out ways to use multiple input modalities at the same time. Instead of thinking of the XOR scenario (this or that but not both) we started considering the AND scenario for inputs. We had a project at the time, VIM – an architectural visualization and data reporting tool for spatial computing –, to try it out with. Our main rule in doing this was that it couldn’t be forced. We wanted to find a natural way to do multi-modal that made sense and hopefully would also be intuitive.

We found a good opportunity as we attempted to refine the ability to move building models around on a table-top. This is a fairly universal UX issue in spatial computing, which made it even more fascinating to us. There are usually a combination of transformations that can be performed on a 3D object at the same time for ease of interaction: translation (moving from position x1 to position x2), scaling the size of the object, and rotating the object. A common solution is to make each of these a different interaction mode triggered by clicking on a virtual button or something.

But we went a different way. As you move a model in space by pointing the Magic Leap controller in different directions like a laser pointer with the building hanging off the end, you can also push it away by pressing on the top of the touch pad or rotate it by spinning your thumb around the edge of the touch pad.

This works great for accomplishing many tasks at once. A side effect, though, is that while users rotated a 3D building with their thumbs, they also had a tendency to shake the controller wildly so that it seemed to get tossed around the room. It took an amazing amount of dexterity and practice to rotate the model while keeping it in one spot.

To fix this, we added a hand gesture to hold the model in place while the user rotated it. We called this the “halt” gesture because it just required the user to put up their off hand with the palm facing out. (Luke Hamilton, our Head of Design, also called this the “stop in the name of love” gesture.)

But we were on a gesture inventing roll and didn’t want to stop. We started thinking about how the keyboard is more accurate and faster than a mouse in data entry scenarios, while the mouse is much more accurate than a game controller or hand tracking for pointing and selecting.

We had a similar situation here where the rotation gesture on the Magic Leap controller was intended to make it easy to spin the model in a 360 degree circle, but consequently was not so good for very slight rotations (for instance the kind of rotation needed to correctly orient a life-size digital twin of a building).

We got on the phone with Brian Schwab and Jay Juneau at Magic Leap and they suggested that we try to use the controller in a different way. Rather than simply using the thumb pad, we could instead rotate the controller on its Z-axis (a bit like a screwdriver) as an alternative rotational gesture. Which is what we did, making this a secondary rotation method for fine-tuning.

And of course we combined the “halt / stop in the name of love” gesture with this “screwdrive” gesture, too. Because we could but more importantly because it made sense and most importantly because it allows the user to accomplish her goals with the least amount of friction.