What tech stack will be used to develop applications for HoloLens, Microsoft’s innovative new augmented reality platform?
Keeping in mind that Alex Kipman is the visionary behind both HoloLens and the Kinect sensor, some clues can be gleaned from the current Kinect SDK. The Kinect SDK initially supported development in WPF and C++ with DirectX. Over time, however, and in line with Microsoft’s internal shift to be more open and embrace the tech stacks of non-Microsoft communities, the Kinect SDK has grown to include support for Unity3D, Cinder, OpenFrameworks, OpenCV and even MATLAB.
Legos vs Play-Doh
This is a two-pronged approach to tooling that Microsoft luminary Rick Barraza has famously characterized as the distinction between building with legos and sculpting with play-doh. Legos represent what Microsoft has traditionally been extremely good at: creating reusable components. Reusable components abstract the underlying technology layers so even beginning developers can accomplish difficult tasks without needing to understand the intricacies of networking, graphics cards, or memory management. As long as all you want to do is the 95% of tasks that Microsoft components support, legos may be all that you ever need. Microsoft has almost single-handedly created the modern enterprise software developer community based on component building, drag-and-drop IDEs, and copy /paste.
But what if you are not interested in building applications that look like everyone else’s? What if you want to do the 5% of things that reusable components do not allow you to do. This has typically been difficult. Not only does it require having a fine grain understanding of the underlying operating system but also a desire to hack around Microsoft’s safeguards. Because Microsoft had for so long embraced the component model of software development, it internally saw its role as one of safeguarding applications from attempts to follow any other model of software development. API classes are typically sealed to prevent extension and if one were to ask to have them unsealed, the inevitable reply was always “what would you use that for”?
Now lets consider a more playful software world in which it makes sense for classes to be unsealed by default so we can just start squishing them around in our hands to see what comes of it. This is software as exploration and in fact there is a whole community, frequently styled “creative coders”, who work in this way. The processing software programming platform created by Casey Reas and Ben Fry is the epitome of this movement. It was originally created as a better way to teach computer programming since it is built around drawing shapes rather than displaying text. From this simple and divergent starting point, all “hello world” applications are dramatically different. Even more profound, however, it became clear that through simple loops and seed values, vastly different effects could be generated using only a few lines of code. Playing with processing feels like sculpting with play-doh. Other homologous tool chains were eventually created that shared processing’s emphasis on the visual rather than the textual: OpenFrameworks, Arduino, Cinder and Unity3D.
Which is a long-about way of saying that for premium experiences, these creative coding tool chains will likely be the tools of choice. If you want to get a jump start on HoloLens development, go learn these platforms:
HologramFramwork APIs
I mentioned above that Microsoft has a two-pronged approach to tooling software developers. So far I have only mentioned the play-doh side. As we come closer to the Windows 10 release, however, there has been increased activity in the long dormant WPF platform team – enough to suggest that some sort of Holographic support might be released with a new version of WPF. WPF, after all, is Microsoft’s premier platform for component-based development. If the marketing ideal for HoloLens is to make as many HoloLens supported applications as possible straight out of the gate (as it should be), then an easy to use platform for building new HoloLens applications as well as porting old ones to the new paradigm is an obvious pre-requisite.
The image above is from enterprising colleagues who inspected the Windows 10 symbol packages to find out what kind of holographic support would be natively built into the new OS. The initial impression is that low-hanging integration will be possible by using some sort of texture mapping model. For example, Silverlight provided a component called the VideoBrush that allowed any control that supported brushes to use a video rather than a solid or gradient texture as a background image. This included even complex 3D shapes or skewed geometries.
HoloLens Grids
To me, this suggests a grid-based programming model for quickly painting applications as textures onto valid surfaces identified by the HoloLens’s built-in depth sensors. The depth sensors will use computer vision algorithms to identify and tag surfaces in a room that can be used to project digital content. The user’s movements and any desirable digital-realistic skewing will be taken care of by the underlying holographic framework. For now, let’s assume that interactions will also be taken care of automatically and will follow a mouse-like hover/press idiom for convenience.
XAML-based languages like WPF have a unique layout component called a Grid. Unlike tables in html, XAML grids define the layout and the content separately. The layout is specified in ColumnDefinitions and RowDefinitions – for instance one may specify a grid that is 3 X 3, or 2 X 1 (as above), and so on. Content is written out (or dragged) below the column and row definitions. Their placement in the layout is then defined by attaching positional directives on the content as shown below.
In this code snippet, I haven’t defined a row so there is just one row by default. I have defined two columns which are a zero based array. Finally, I’ve specified on the green panel that I want it to be positioned in the first column by writing Grid.Column=”0” . The red panel, in turn, is placed in column 1, the second column in the series. The resulting WPF application is shown below:
In this case, the application is not particularly impressive. We can imagine the same code being written against the Holographic Framework with the following results, however:
And here is what the code for your first “Hello, World” application might look like:
It is, I hope, not difficult to see how we can then go from a standard XAML Grid like the one above to a HoloLens-enabled Grid like the one below:
At this point, given the dearth of information currently available (I’m dumpster diving through Windows 10 symbol packages, after all) this is obviously just a wild guess. I believe it is a plausible programming model, however, and would provide a royal road to quickly generate applications for Microsoft’s newest and brightest technology innovation.