Oculus Rift – The Imaginative Universal

Streaming and Recording VR Tracking Data

August 11, 2017August 11, 2017 James Ashley

Jasper Brekelmans is already pretty famous for creating Brekel Pro Tools, which turns the sensor streams from various 3D cameras, including the Kinect, into useful 3D data that can be used for 3D animations, visual effects and general research. OpenVR Recorder expands on this capability by also ingesting high quality data used for tracking virtual reality devices like the Oculus Rift and the HTC Vive.

The original Kinect, introduced in 2010, created a small revolution in the 3D sensor industry. It helped to drive down the price of 3D tracking and inspired new research into time-of-flight, structured light pattern and marker-based techniques that dramatically improved the accuracy with which we capture and understand 3D space. In turn, this has driven innovation in the area of interpreting 3D spaces using 2D cameras – which goes into solutions like Apple’s ARToolkit.

Though it may not be immediately evident, the current developments in VR and AR are built on top of these leaps and bounds in 3D sensor technology. The HoloLens uses it for position tracking and spatial mapping. The HoloLens solution is going into the SLAM-based position tracking tech for the Mixed Reality Headsets from HP and Acer. At the same time, VR tech from Oculus and HTC are constantly streaming 3D data in order to provide accurate tracking of both the user and handheld controllers.

Since this data is already flowing through the air around you when you are in VR space, there must be a way to capture it, right? That’s what Jasper Brekelmans has done with his new tool, OpenVR Recorder.

If you’re already working with the Kinect for visual effects, motion capture, or research, then this new software should be included as an essential part of your tool chest. There’s even a free trial available.

Rethinking VR/AR Launch Dates

April 10, 2016February 28, 2017 James Ashley

The HTC Vive, Oculus Rift and Microsoft HoloLens all opened for pre-orders in 2016 with plans to ship in early April (or late March in the case of the Oculus). All have run into fulfillment problems creating general confusion for their most ardent fans.

I won’t try to go into all the details of what each company originally promised and then what each company has done to explain their delays. I honestly barely understand it. Oculus says there were component shortages and is contacting people through email to update them. Oculus also refunded some shipping costs for some purchasers as compensation. HTC had issues with their day one ordering process and is using its blog for updates. Microsoft hasn’t acknowledged a problem but is using its developer forum to clarify the shipping timeline.

Maybe it’s time to acknowledge that spinning up production for expensive devices in relatively small batches is really, really hard. Early promises from 2015 followed by CES in January 2016 and then GDC in March probably created an artificial timeline that was difficult to hit.

On top of this, internal corporate pressure has probably also driven each product group to hype to the point that it is difficult to meet production goals. HTC probably has the most experience with international production lines for high tech gear and even they stumbled a bit.

Maybe it’s also time to stop blaming each of these companies as they reach out for the future. All that’s happened is that some early adopters aren’t getting to be as early as they want to be (including me, admittedly).

As William Gibson said, “The future is already here — it’s just not very evenly distributed.”

HoloLens App Development with Unity

June 25, 2015 James Ashley

A few months ago I wrote a speculative piece about how HoloLens might work with XAML frameworks based on the sample applications Microsoft had been showing.

Even though Microsoft has still released scant information about integration with 3D platforms, I believe I can provide a fairly accurate walkthrough of how HoloLens development will occur for Unity3D. In fact, assuming I am correct, you can begin developing games and applications today and be in a position to release a HoloLens experience shortly after the hardware becomes available.

To be clear, though, this is just speculative and I have no insider information about the final product that I can talk about. This is just what makes sense based on publicly available information regarding HoloLens.

Unity3D integration with third party tools such as Kinect and Oculus Rift occurs through plugins. The Kinect 2 plugin can be somewhat complex as it introduces components that are unique to the Kinect’s capabilities.

The eventual HoloLens plugin, on the other hand, will likely be relatively simple since it will almost certainly be based on a pre-existing component called the FPSController (in Unity 5.1 which is currently the latest).

To prepare for HoloLens, you should start by building your experience with Unity 5.1 and the FPSController component. Here’s a quick rundown of how to do this.

Start by installing the totally free Unity 5.1 tools: http://unity3d.com/get-unity/download?ref=personal

Next, create a new project and select 3D for the project type.

Click the button for adding asset packages and select Characters. This will give you access to the FPSController. Click done and continue. The IDE will now open with an practically empty project.

At this point, a good Unity3D tutorial will typically show you how to create an environment. We’re going to take a shortcut, however, and just get a free one from the Asset Store. Hit Ctrl+9 to open the Asset Store from inside your IDE. You may need to sign in with your Unity account. Select the 3D Models | Environments menu option on the right and pick a pre-built environment to download. There are plenty of great free ones to choose from. For this walkthrough, I’m going to use the Japanese Otaku City by Zenrin Co, Ltd.

After downloading is complete, you will be presented with an import dialog box. By default, all assets are selected. Click on Import.

Now that the environment you selected has been imported, go the the scenes folder in your project window and select a sample scene from the downloaded environment. This will open up the city or dungeon or forest or whatever environment you chose. It will also make all the different assets and components associated with the scene show up in your Hierarchy window. At this point, we want to add the first-person shooter controller into the scene. You do this by selecting the FPSController from the project window under Assets/Standard Assets/Characters/FirstPersonCharacter/Prefabs and dragging the FPSController into your Hierarchy pane.

This puts a visual representation of the FPS controller into your scene. Select the controller with your mouse and hit “F” to zoom in on it. You can see from the visual representation that the FPS controller is basically a collision field that can be moved with a keyboard or gamepad that additionally has a directional camera component and a sound component attached. The direction the camera faces ultimately become the view that players see when you start the game.

Here is another scene that uses the Decrepit Dungeon environment package by Prodigious Creations and the FPS controller. The top pane shows a design view while the bottom pane shows the gamer’s first-person view.

You can even start walking through the scene inside the IDE by simply selecting the blue play button at the top center of the IDE.

The way I imagine the HoloLens integration to work is that another version of FPS controller will be provided that replaces mouse controller input with gyroscope/magnetometer input as the player rotates her head. Additionally, the single camera view will be replaced with a two camera rig that sends two different, side-by-side feeds back to the HoloLens device. Finally, you should be able to see how all of this works directly in the IDE like so:

There is very good evidence that the HoloLens plugin will work something like I have outlined and will be approximately this easy. The training sessions at the Holographic Academy during /Build pretty much demonstrated this sort of toolchain. Moreover, this is how Unity3D currently integrates with virtual reality devices like Gear VR and Oculus Rift. In fact, the screen cap of the Unity IDE above is from an Oculus game I’ve been working on.

So what are you waiting for? You pretty much have everything you already need to start building complex HoloLens experiences. The integration itself, when it is ready, should be fairly trivial and much of the difficult programming will be taken care of for you.

I’m looking forward to seeing all the amazing experiences people are building for the HoloLens launch day. Together, we’ll change the future of personal computing!

Minecraft in Virtual Reality and Augmented Reality

June 17, 2015 James Ashley

Microsoft recently created possibly the best demo they have ever done on stage for E3. Microsoft employees played Minecraft in a way no one has ever seen it before, on a table top as if it was a set of legos. Many people speculated on social media that this may be the killer app that HoloLens has been looking for.

What is particularly exciting about the way the demo captured people’s imaginations is that they can start envisioning what AR might actually be used for. People are even getting a firm grip on the differences between Virtual Reality, which creates an immersive experience, and augmented reality which creates a mixed experience overlapping digital objects with real world objects.

Nevertheless, there is still a tendency to see virtual reality exemplified by the Oculus Rift and augmented reality exemplified by HoloLens and Magic Leap as competing solutions. In fact they are complementary solutions. They don’t compete with one another any more than your mouse and your keyboard do.

Bill Buxton has famously said that everything is best for something and worst for something else. By contrasting the Minecraft experience for Oculus and HoloLens, we can better see what each technology is best at.

The Virtual Reality experience for Oculus is made possible by a free hacking effort called Minecrift. It highlights the core UX flavor of almost all VR experiences – they are first person, with the player fully present in a 3D virtual world. VR is great for playing Minecraft in adventure or survival mode.

Adventure mode with HoloLens is roughly equivalent to the adventure mode we get today on a PC or XBOX console with the added benefit that the display can be projected on any wall. It isn’t actually 3D, though, as far as we can tell from the demo, despite the capability of displaying stereoscopic scenes with HoloLens.

What does work well, however, is Minecraft in creation mode. This is basically the god view we have become familiar with from various strategy and resource games over the years.

God View vs First Person View

In a fairly straightforward way, it makes sense to say that AR is best for a god-centric view while VR is best for a first-person view. For instance, if we wanted to create a simulation that allows users to fly a drone or manipulate an undersea robot, virtual reality seems like the best tool for the job. When we need to create a synoptic view of a building or even a city, on the other hand, then augmented reality may be the best UX. Would it be fair to say that all new UX experiences fall into one of these two categories?

Most of our metaphors for building software, building businesses and building every other kind of buildable thing, after all, are based on the lego building block and it’s precursors the Lincoln log and erector sets. We play games as children in order, in part, to prepare ourselves for thinking as adults. Minecraft was built similarly on the idea of creating a simulation of a lego block world that we could not only build but also virtually play in on the computer.

lego_astronaut

The playful world of Lego blocks is built on two things: the blocks themselves formed into buildings and scenes and the characters that we identify with who live inside the world of blocks. In other words the god-view and the first-person view.

coffee prince

It should come as no surprise, then, that these two core modes of our imaginative lives should stay with us through our childhoods and into our adult approaches to the world. We have both an interpersonal side and an abstract, calculating side. The best leaders have a bit of both.

You apparently didn’t put one of the new coversheets on your TPS report

The god-view in business tends to be the synoptic view demanded by corporate executives and provided in the form of dashboards or crystal reports. It would be a shame if AR ended up falling into that use-case when it can provide so much more and in more interesting ways. As both VR and AR mature over the next five years, we all have a responsibility to keep them anchored in the games of our childhood and avoid letting them become the faults and misdemeanors of the corporate adult world.

Update 6/20

A recent arstechnica article indicates that the wall-projected HoloLens version of Minecraft in adventure mode can be played in true 3D:

One other impressive feature of the HoloLens-powered virtual screen was the ability to activate a three-dimensional image, so that the scene seemed to recede into the wall like a window box. Unlike a standard 3D monitor, this 3D image actually changed perspective based on the viewing angle. If I went up near the wall and looked at the screen from the left, I could see parts of the world that would usually be behind the right side of the wall, as if the screen was simply a window into another world.