James Ashley

The Mvp Program REORG Explained Through Gamification

October 7, 2015February 28, 2017 James Ashley

Today chief Microsoft evangelist Steve Guggenheimer announced dramatic changes to the MVP program on his blog.

In case you are unfamiliar with the MVP program, it is a recognition Microsoft gives to members of the developer community and is generally understood as a mark of expertise in a particular Microsoft technology (e.g. Windows Phone, Outlook, Kinect for Windows). In truth, though, there are many ways to get the MVP recognition without necessarily being an expert in any particular technology and running user groups or helping at coding events are common ways. The origins of the program go back to the days when it was given out for answering forum questions about Microsoft technologies. This is a good way to understand the program – it is a reward of sorts for people who are basically helping their communities out as well as helping Microsoft out. Besides the status conferred by the award, the MVP includes an annual subscription to MSDN and an annual invitation to the Redmond campus for the MVP summit. Depending on the discipline as well as marketing cycles, you may also have access to regular calls with particular product teams. MVPs also have to renew every year by explaining what they’ve done in the prior 12 months to help the developer, IT or consumer community.

As with any sort of status thingy that confers a sense of self-worth and may even affect income, it is occasionally a source of turmoil, stress and drama for people. Like soccer mom levels of drama. For instance, occasionally a product category like Silverlight will just disappear and that particular discipline has to be scrapped. The people who are Silverlight MVPs will typically feel hurt by this and understandably feel slighted. They didn’t become suddenly unworthy, after all, simply because the product they had poured so much energy into isn’t around anymore.

Some products are hot and some are not, while others start off hot then become not. If you were one of those Silverlight MVPs, you probably would like to point out that you are in fact worthy and know lots of other things but had been ignoring other technical interests in order to promote just Silverlight. You probably would feel that it is unjust to be punished for overinvesting in one technology.

In response to situations such as this, the Microsoft MVP program is undergoing a re-organization.

I’ll quote the synoptic statement from Steve’s post:

Moving forward, the MVP Award structure will shift to encompass the broad array of community contributions across technologies. For our Developer and IT Pro oriented MVPs, we’re moving from 36 areas of technical expertise to a set of 10 broader categories that encompass a combined set of 90 technology areas—including open source technologies.

The ~~best~~ most fun way to understand this is in terms of Dungeons & Dragons. To do so it is important that I first try to explain the difference between class-based role playing games and skills-based RPGs. Diablo is a great example of a class-based RPG. You choose from a handful of classes like barbarian, demon hunter or monk, and based on that your skills are pretty much picked out for you. At the opposite extreme is a game like Fallout where you have full control over how to upgrade your abilities; the game doesn’t prescribe how you should play at all. In the middle are RPGs like World of Warcraft which has cross-class skills but also provides a boost to certain skills depending on what class you initially choose. Certain class/skill combinations are advisable, but none are proscribed. You have freedom to play the game the way you want – for instance as an elf hunter with mutton chops and a musket. Totally do-able.

Dungeons & Dragons is a game that changes its rules every so often and causes lots of consternation whenever it does so. One of the corrections happened between D&D 3.0 and D&D 3.5 when the game went from a simplified class-based system to a more open skills based system. This allowed players a lot more freedom in how they customized their characters who could now gain skills that aren’t traditionally tied to their class.

The MVP program is undergoing the same sort of correction, moving from a class-based gaming system to a skills-based gaming system. Instead of just being a Silverlight MVP, you can now be an fifth-level druid with Javascript and handle animals skills, or a third-level Data Platform MVP with interests in IoT, Azure machine learning, alchemy, light armor and open lock. You can customize the MVP program to fit your style of play rather than letting the program prescribe what sort of tech things you need to be working on.

This, I believe, will help meliorate the problem of people basing their self-worth on a fixed idea of what their MVP-ness means or the bigger problem of comparing their MVP-ness to the MVP-nesses of others. Going forward, one’s MVP-ness is whatever one makes of it. And that’s a good thing.

The Next Big Thing in Depth Sensors

October 1, 2015February 28, 2017 James Ashley

Today Orbbec3D, my employer, announced a new depth sensor called the Orbbec Persee. We are calling it a camera-computer because instead of attaching a sensor to a computer, we’ve taken the much more reasonable approach of putting the computer inside our sensor. It’s basically a 3D camera that doesn’t require a separate computer hooked up to it because it has its own onboard CPU while maintaining a small physical footprint. This is the same sort of move being made by products like the HoloLens.

Unlike the Oculus Rift which requires an additional computer or Google Glass which needs a CPU on a nearby smartphone, the Persee falls into a category of devices that perform their own computing and that you can program as well as load pre-built software on.

For retail scenarios like advanced proximity detection or face recognition, this means compact installations. One of the major difficulties with traditional “peripheral” cameras is placing computers in such a way that they stay hidden while also allowing for air circulation and appropriate heat dissipation. Having done this multiple times, I can confirm that this is a very tricky problem and typically introduces multiple points of failure. The Persee removes all those obstacles and allows for sleek fabricated installs at a great price.

What has me truly excited about the Persee is Orbbec’s efforts to cater to the creative coding community and the way that the creative community has taken to it. These people are my heroes and having them give our product the nod means the world to me. People like Golan Levin, Phoenix Perry, Kyle McDonald, James George, Greg Borenstein, and Elliot Woods.

The device is OpenNI compatible but also provides it’s own middleware to add new capabilities and fill both creative and commercial needs (<– this is the part I’m working on).

Is it a replacement for Kinect? In my opinion, not really because they do different things. The Kinect will always be my first love and is the full package, offering high-rez video, depth and a 3D mic. It is primarily a gaming peripheral. The Orbbec Persee fills a very different niche and competes with devices like the Asus Xtion and Intel RealSense as realtime collectors of volumetric data – in the way your thermometer collects thermal data. What distinguishes the Persee from its competitors is that it is an intelligent device and not just a mere peripheral. It is a first class citizen in the Internet of Things — to invoke magical marketing thinking – where each device in the web of intelligent objects not only reports its status but also reflects, processes and adjusts its status. It’s the extra kick that makes the Internet of Things not just a buzzword, but also a step along the path toward non-Skynet hard AI. It’s the next big thing.

Ceci n’est pas une pipe bombe

September 23, 2015 James Ashley

This is the picture of the homemade clock Ahmed Mohamed brought to his Irving, Texas high school. Apparently no one ever mistook it for a bomb, but they did suspect that it was made to look like a bomb and so they dragged the hapless boy off in handcuffs and suspended him for three days.

This is a strange case of perception versus reality in which the virtual bomb was never mistaken for a real bomb. Instead, what was identified was the fact that it was, in fact, only a bomb virtually and, as with all things virtual, therefore required some sort of explanation.

The common sympathetic explanation is that this isn’t a picture of a virtual bomb at all but rather a picture of a homemade clock. Ahmed recounts that he made the clock, in maker fashion, in order to show an engineering teacher because he had done robotics in middle school and wanted to get into a similar program in high school. Homemade clocks, of course, don’t require an explanation since they aren’t virtually anything other than themselves.

It turns out, however, that the picture at the top does not show a homemade maker clock. Various engineering types have examined the images and determined that it is in fact a disassembled clock from the 80’s.

The telling aspect is the DC power cord which doesn’t actually get used in homemade projects. Instead, anyone working with arduino projects typically (pretty much always) uses AA batteries. The clock components have also been tracked back to their original source, however, so the evidence seems pretty solid.

The photo at the top shows not a virtual bomb nor a homemade clock but, in fact, a virtual homemade clock. That is, it was made to look like a homemade clock but was mistakenly believed to be something made to look like a homemade bomb.

[As a disclaimer about intentions, which is necessary because getting on the wrong side of this gets people in trouble, I don’t know Ahmed’s intentions and while I’m a fan of free speech I can’t say I actually believe in free speech having worked in marketing and I think Ahmed Mohammed looks absolutely adorable in his NASA t-shirt and I have no desire to be placed in company with those other assholes who have shown that this is not a real homemade clock but rather a reassembled 80’s clock and therefore question Ahmed’s motives whereas I refuse to try to get into a high schooler’s head, having two of my own and knowing what a scary place that can be … something, something, something … and while I can’t wholeheartedly support every tweet made by Richard Dawkins and have at times even felt in mild disagreement with things he and others have tweeted on twitter I will say that I find his book The Selfish Gene a really good read … etc, etc, … and for good measure fuck you FoxNews.]

micronta

The salient thing for me is that we all implicitly know that a real bomb isn’t supposed to look like a bomb. The authorities at Ahmed’s high school knew that immediately. Bombs are supposed to look like shoes or harmless tourist knickknacks. If you think it looks like a bomb, it obviously isn’t. So what does it mean to look like a bomb (to be virtually a bomb) but not be an actual bomb?

I covered similar territory once before in a virtual exhibit called les fruits dangereux and at the time concluded that virtual objects, like post-modern novels, involve bricolage and the combining of disparate elements in unexpected ways. For instance combining phones, electrical tape and fruit or combining clock parts and pencil cases. Disrupting categorical thinking at a very basic level makes people – especially authority people – suspicious and unhappy.

Which gets us back to racism which is apparently what has happened to Ahmed Mohammed who was led out of school in handcuffs in front of his peers – and we’re talking high school! and he wasn’t asking to be called “McLovin.” It’s pretty cruel stuff. The fear of racial mixing (socially or biologically) always raises it’s head and comes from the same desire to categorize people and things into bento box compartments. The great fear is that we start to acknowledge that we live in a continuum of types rather than distinct categories of people, races and objects. In the modern age, mass production makes all consumer objects uniform in a way that artisanal objects never were while census forms do the same for people.

Virtual reality will start by copying real world objects in a safe way. As with digital design, it will start with isomorphism to make people feel safe and comfortable. As people become comfortable, bricolage will take hold simply because, in a digital world rather than a commoditized/commodified world, mashups are easy. Irony and a bit of subversiveness will lead to bricolage with purpose as we find people’s fantasies lead them to combine digital elements in new and unexpected ways.

We can all predict augmented and virtual ways to press a digital button or flick through a digital menu projected in front of us in order to get a virtual weather forecast. Those are the sorts of experiences that just make people bored with augmented reality vision statements.

The true promise of virtual reality and augmented reality is that they will break down our racial, social and commodity thinking. Mixed-reality has the potential to drastically change our social reality. How do social experiences change when the color of a person’s avatar tells you nothing real about them, when our social affordances no longer provide clues or shortcuts to understanding other people? In a virtual world, accents and the shoes people wear no longer tell us anything about their educational background or social status. Instead of a hierarchical system of discrete social values, we’ll live in a digital continuum.

That’s the sort of augmented reality future I’m looking forward to.

The important point in the Ahmed Mohammed case, of course, is that you shouldn’t arrest a teenager for not making a bomb.

The Problem with Comparing Depth Camera Resolutions

September 15, 2015 James Ashley

We all want to have an easy way to compare different depth cameras to one another. Where we often stumble in comparing depth cameras, however, is in making the mistake of thinking of them in the same way we think of color cameras or color displays.

When we go to buy a color television or computer monitor, for instance, we look to the pixel density in order to determine the best value. A display that supports 1920 by 1080 has roughly 2.5 times the pixel density of a 1280 by 720 display. The first is considered high definition resolution while the second is commonly thought of as standard definition. From this, we have a rule of thumb that HD is 2.5 times denser than SD. With digital cameras, we similarly look to pixel density in order to compare value. A 4 megapixel camera is roughly twice as good as a 2 megapixel camera, while an 8 MP camera is four times as good. There are always other factors involved, but for quick evaluations the pixel density trick seems to work. My phone happens to have a 41 MP camera and I don’t know what to do with all those extra megapixels – all I know is that it is over 20 times as good as that 2 megapixel camera I used to have and that makes me happy.

When Microsoft’s Kinect 2 sensor came out, it was tempting to compare it against the Kinect v1 in a similar way: by using pixel density. The Kinect v1 depth camera had a resolution of 320 by 240 depth pixels. The Kinect 2 depth camera, on the other hand, had an increased resolution of 512 b 424 depth pixels. Comparing the total depth pixels provided by the Kinect v1 to the total provided by the Kinect 2: 76,800 vs 2, 217,088, many people arrived at the conclusion that the Kinect 2’s depth cameras was roughly three times better than the Kinect v1’s.

Another feature of the Kinect 2 is a greater field of view for the depth camera. Where the Kinect v1 has a field of view of 57 degrees by 43 degrees, the Kinect 2 has a 70 by 60 degree field of view. The new Intel RealSense 3D F200 camera, in turn, advertises an improved depth resolution of 480 by 360 degrees with an increased field of view of roughly 90 degrees by 72 degrees.

What often gets lost in these feature comparisons is that our two different depth camera attributes, resolution and field of view, can actually affect each other. Increased pixel resolution is only really meaningful if the field of view stays the same between different cameras. If we increase the field of view, however, we are in effect diluting the resolution of each pixel by trying to stuff more of the real world into the pixels we already have.

It turns out that 3D math works slightly differently from regular 2D math. To understand this better, imagine a sheet of cardboard held a meter out in front of each of our two Kinect sensors. How much of each sheet is actually caught by the Kinect v1 and the Kinect 2?

To derive the area of the inner rectangle captured by the Kinect v1 in the diagram above, we will use a bit of trigonometry. The field of view of the Kinect v1 is 58.5 degrees horizontal by 46.6 vertical. To get good angles to work with, however, we will need to bisect these angles. For instance, half of 46.6 is 23.3. The tangent of 21.5 degrees times the 1 meter hypotenuse (since the cardboard sheet is 1 M away) gives us an opposite side of .39 meters. Since this is only half of that rectangle’s side (because we bisected the angle) we multiply by two to get the full vertical side which is .78 meters. Using the same technique for the horizontal field of view, we capture a horizontal side of 1.09 meters.

Using the same method for the sheet of cardboard in front of the Kinect 2, we discover that the Kinect 2 captures a rectangular surface that is 1.4 meters by 1.14 meters. If we now calculate the area on the cardboard sheets in front of each camera and divide by each camera’s resolution, we discover that far from being three times better than the Kinect v1, each pixel caught by the Kinect 2 depth camera holds 1.5 times as much of the real world as each pixel of the Kinect v1. It is still a better camera, but not what one would think by comparing resolutions alone.

This was actually a lot of math in order to make a simple and mundane point: it all depends. Depth pixel resolutions do not tell us everything we need to know when comparing different depth cameras. I invite the reader to compare the true density of the RealSense 3D camera to the Kinect 2 or Xtion Pro Live camera if she would like.

On the other hand, it might be worth considering the range of these different cameras. The RealSense F200 cuts off at about a meter whereas the Kinect cameras only start performing really well at about that distance. Another factor is, of course, the accuracy of the depth information each camera provides. A third factor is whether one can improve the performance of a camera by throwing on more hardware. Because the Kinect 2 is GPU bound, it will actually work better if you simply add a better graphics card.

For me, personally, the most important question will always be how good the SDK is and how strong the community around the device is. With good language and community support, even a low quality depth camera can be made to do amazing things. An extremely high resolution depth camera with a weak SDK, alternatively, might in turn make a better paperweight than a feature forward technology solution.

[I’d like to express my gratitude to Kinect for Windows MVPs Matteo Valoriani and Vincent Guigui for introducing me to this geometric bagatelle.]

HoloLens Fashionista

September 10, 2015 James Ashley

microsoft-executives-testing-the-hololens

While Google Glass certainly had its problems as an augmented reality device – among other things not really being an augmented reality device as GA Tech professor Blair MacIntyre pointed out – it did demonstrate two remarkable things. First, that people are willing to shell out $1500 for new technology. In the debates over the next year concerning the correct price point for VR and AR head mounted displays, this number will play a large role. Second, it demonstrated the importance of a sense of style when designing technology. Google glass, for many reasons, was a brilliant fashion accessory.

If a lesson can be drawn from these two data points, it might be that new — even Project Glass-level iffy — technology can charge a lot if it manages to be fashionable as well as functional.

When you look at the actual HoloLens device, you may, like me, be thinking “I don’t know if I’d wear that out in public.” In that regard, I’d like to nudge your intuitions a bit.

Obviously there is time to do some tweaking with the HL design. I recently found some nostalgic pictures online that made me start to think that with modifications, I could rock this look.

It all revolves around one of the first animes imported to the United States in the 70s called Battle of the Planets. It sounded like this:

Battle of the Planets! G-Force! Princess! Tiny! Keyop! Mark! Jason! And watching over them from Center Neptune, their computerized coordinator, 7-Zark-7! Watching, warning against surprise attacks by alien galaxies beyond space. G-Force! Fearless young orphans, protecting Earth’s entire galaxy. Always five, acting as one. Dedicated! Inseparable! Invincible!

And it looked AMAZING. I think this look could work for HoloLens. I think I could pull it off. The capes and tights, of course, are purely optional.

Why Augmented Reality is harder than Virtual Reality

September 9, 2015 James Ashley

hololensx519_0

At first blush, it seems like augmented reality should be easier than virtual reality. Whereas virtual reality involves the generation of full stereoscopic digital environments as well as interactive objects to place in those environments, augmented reality is simply adding digital content to our view of the real world. Virtual reality would seem to be doing more heavy lifting.

perspective

In actual fact, both technologies are creating illusions to fool the human eye and the human brain. In this effort, virtual reality has an easier task because it can shut out points of reference that would otherwise belie the illusion. Augmented reality experiences, by contrast, must contend with real world visual cues that draw attention to the false nature of the mixed reality content being added to a user’s field of view.

In this post, I will cover some of the additional challenges that make augmented reality much more difficult to get right. In the process, I hope to also provide clues as to why augmented reality HMDs like HoloLens and Magic Leap are taking much longer to bring to market than AR devices like the Oculus Rift, HTC Vive and Sony Project Morpheus.

terminator vision

But first, it is necessary to distinguish between two different kinds of augmented reality experience. One is informatics based and is supported by most smart phones with cameras. The ideal example of this type of AR is the Terminator-vision from James Cameron’s 1984 film “The Terminator.” It is relatively easy to to do and is the most common kind of AR people encounter today.

star wars chess

The second, and more interesting, kind of AR requires inserting illusory 3D digital objects (rather than informatics) into the world. The battle chess game from 1977’s “Star Wars” epitomizes this second category of augmented reality experience. This is extremely difficult to do.

The Microsoft HoloLens and Magic Leap (as well as any possible HMDs Apple and others might be working on) are attempts to bring both the easy type and the hard type of AR experience to consumers.

Here are a few things that make this difficult to get right. We’ll put aside stereoscopy which has already been solved effectively in all the VR devices we will see coming out in early 2016.

cloaked predator

1. Occlusion The human brain is constantly picking up clues from the world in order to determine the relative positions of objects such as shading, relative size and perspective. Occlusion is one that is somewhat tricky to solve. Occlusion is an effect that is so obvious that it’s hard to realize it is a visual cue. When one body is in our line of sight and is positioned in front of another body, that other body is partially hidden from our view.

In the case where a real world object is in front of a digital object, we can clip the digital object with an outline of the object in front to prevent bleed through. When we try to create the illusion that a digital object is positioned in front of a real world object, however, we encounter a problem inherent to AR.

In a typical AR HMD we see the real world through a transparent screen upon which digital content is either projected or, alternatively, illuminated as with LED displays. An obvious characteristic of this is that digital objects on a transparent display are themselves semi-transparent. Getting around this issue would seem to require being able to make certain portions of the transparent display more opaque than others as needed in order to make sure our AR objects look substantial and not ghostly.

citizen kane

2. Accommodation It turns out that stereoscopy is not the only way our eyes recognize distance. The image above is from a scene in Orson Welles’s “Citizen Kane” in which a technique called “deep focus” is used extensively. Deep focus maintains clarity in the frame whether the actors and props are in the foreground, background or middle ground. Nothing is out of focus. The technique is startling both because it is counter to the way movies are generally shot but also because it is counter to how our eyes work.

If you cover one eye and use the other to look at one of your fingers, then move the finger toward and away from you, you should notice yourself refocusing on the finger as it moves while other objects around the finger become blurry. The shape of the cornea actually becomes more rounded when objects are close in order to cause light to refract more in order to reach the retina. For further away objects, the cornea flattens out because less refraction is needed. As we become older, the ability to bow the cornea lessens and we lose some of our ability to focus on near objects – for instance when we read. In AR, we are attempting to make a digital object that is really only centimeters from our eyes appear to be much further away.

Depending on how the light from the display passes through the eye, we may end up with the digital object appearing clear while the real world objects supposedly next to it and at the same distance appear blurred.

vergence_accomodation

3. Vergence-Accommodation Mismatch The accommodation problem is one aspect of yet another VR/AR difficulty. The term vergence describes the convergence and divergence of the two eyes from one another as objects move closer or further away. An interesting aspect of stereoscopy – which is used both for virtual reality as well as augmented reality to create the illusion of depth – is that the distance at which the two eyes coordinate to see an object is generally different from the focal distance from the eyes to the display screen(s). This consequently sends two mismatched signals to the brain concerning how far away the digital object is supposed to be. Is it the focal length or the vergence length? Among other causes, vergence-accommodation mismatch is believed to be a contributing factor to VR sickness. Should the accommodation problem above be resolved for a given AR device, it is safe to assume that the vergence-accommodation mismatch will also be solved.

4. Tetherless Battery Life Smart phones have changed our lives among other reasons because they are tetherless devices. While the current slate of VR devices all leverage powerful computers to which they are attached, since VR experiences are all currently somewhat stationary (the HTC Vive being the odd bird), AR needs to be portable. This naturally puts a strain on the battery, which needs to be relatively light since it will be attached to the head-mounted-display, but also long-lived as it will be powering occasionally intensive graphics, especially for games.

5. Tetherless GPU Another strain on the system is the capability of the GPU. Virtual reality devices can be fairly intense since they require the user to purchase a reasonably powerful and somewhat expensive graphics card. AR devices can be expected to have similar graphics requirements as VR with much less to work with since the GPU needs to be onboard. We can probably expect a streamlined graphics pipeline dedicated to and optimized for AR experiences will help offset lower GPU capabilities.

6. Applications Not even talking about killer apps, here. Just apps. Microsoft has released videos of several impressive demos including Minecraft for HoloLens. Magic Leap up to this point has only shown post-prod, heavily produced illustrative videos. The truth is that everyone is still trying to get their heads around designing for AR. There aren’t really any guidelines for how to do it or even what interactions will work. Other than the most trivial experiences (e.g. weather and clock widgets projected on a wall) this will take a while as we develop best practices while also learning from our mistakes.

Conclusion

With the exception of V-AM, these are all problems that VR does not have to deal with. Is it any wonder, then, that while we are being led to believe that consumer models of the Oculus Rift, HTC Vive and Sony Project Morpheus will come to market in the first quarter of 2016, news about HoloLens and Magic Leap has been much more muted. There is simply much more to get right before a general rollout. One can hope, however, that dev units will start going out soon from the major AR players in order to mitigate challenge #6 while further tuning continues, if needed, on challenges #1-#5.

Come hear me speak about Mixed Reality at Dragon Con 2015

September 1, 2015 James Ashley

I’ve been invited by the Robotics and Maker Track to speak about near future technologies at Dragon Con this year. While the title of the talk is “Microsoft Kinect and HoloLens,” I’ll actually be talking more broadly about 3D sensors like Kinect and the Orbbec Astra, Virtual Reality with the Oculus Rift and HTC Vive as well as Augmented Reality with HoloLens and Magic Leap. I will cover how these technologies will shape our lives and potentially change our world over the next five years.

I am honored to have been asked to be a panelist at Dragon Con on technology I am passionate about and that has been a large part of my life and work over the past several years.

I should add that being a panelist at Dragon Con is a nerd and fan’s freakin’ dream come true for me. Insanely so. Hopefully I’ll be able to stay cool enough to get through all the material I have on our collective sci fi future.

I will cover each technology and the devices coming out in the areas of 3D sensors, virtual reality and augmented reality. I’ll discuss their potential impact as well as some of their history. I’ll delve into some of the underlying technical and commercial challenges that face each. I’ll bring lots of Kinect and Oculus demos (not allowed to show HoloLens for now, unfortunately) and will also provide practical advice on how to experience these technologies as a consumer as well as a developer in 2016.

My panel is on Sunday, Sept 6 at 2:30 in Savannah rooms 1, 2 and 3 in the Sheraton. Please come say hi!

The HoloCoder’s Resume

July 12, 2015June 24, 2017 James Ashley

In an ideal world, the resume is an advertisement for our capabilities and the interview process is an audit of those claims. Many factors have contributed to complicating what should be a simple process.

The first is the rise of professional IT recruiters and the automation of the resume process. Recruiters bring a lot to the game, offering a wider selection of IT job candidates to hiring companies, on the one hand, and providing a wider selection of jobs to job hunters, on the other. Automation requires standardization, however, and this has led to an overuse of key search terms when matching candidates to positions. The process begins with job specs from the hiring company — which parenthetically often have little to do with the actual job itself and highlights the frequent disconnect between IT departments and HR departments. A naive job hunter would try to describe their actual experience, which typically will not match the job spec as written by HR. At this point the recruiter helps the job hunter modify the details of her resume to match the template provided by the hiring company by injecting and prioritizing key buzzwords into the resume. “I’m sorry but Lolita, Inc will never hire you unless you have synesthesia listed in your job history. You do have experience with synesthesia, don’t you?”

All of this gerrymandering is required in order to get to the next step, the job interview. Unfortunately, the people doing the job interview have little confidence in the resume as a vehicle for accurately describing a candidate’s actually abilities. First of all, they know that recruiters have already gone over it to eliminate useful information and replace it with keywords instead. Next, the interviewers typically haven’t actually seen the HR job specs and do not understand what kind of role they are hiring for. Finally, none of the interviewers have any particular training in doing job interviews or any particular skill in ascertaining what a candidate knows. In short, the interviewer doesn’t know what he’s looking for and wouldn’t know how to get it if he did.

A savvy interviewer will probably realize that he is looking for the sort of generalist that Joel Spolsky describes as “smart and gets things done,” but how do you interview for that? The tools the interviewer is provided with are not generic but instead highly specific technology skills. At some point, this impedance mismatch between technology specific interview questions on the one had and a desire to hire generalists on the other (technology, after all, simply changes too quickly to look for only one skillset) lead to an increased reliance on behavioral questions and eventually Google-style language games. Neither of these, it turns, out, particularly help in hiring good candidates.

Once we historically severed any attempt to match interview questions to actual skills, the IT interview process was allowed to become a free floating hermeneutic exercise. Abstruse but non-specific questions involving principles and design patterns have taken over the process. This has led to two strange outcomes. On the one hand, job applicants are now required to be fluent in technical information they will never actually use in their jobs. Literary awareness of ten year old blog posts by Martin Fowler are more important than actually knowing how to get things done. And if the job interviewer exhibits any self-awareness when he turns down a candidate for not being clear on the justified uses of the CQRS pattern (there are none), it will not be because the candidate didn’t know something important for the job but rather because the candidate was unwilling to play the software architecture language game, and anyone unwilling to play the game is likely going to be a poor cultural fit.

The other consequence of an increased reliance on abstruse and non-essential IT knowledge has been the rise of the Architect across the industry. The IT industry has created a class of software developers who cannot actually develop software but instead specialize in telling other people what is wrong with their code. The architect is a specialization that probably indicates a deviant phase in the software industry – but at the same time it is a natural outcome of our IT job spec – resume – interview process. The skills of a modern software architect – knowledge of abstruse information and jargon often combined with an inability to get things done – is what we currently look for through our IT cargo cult hiring rituals.

This distinction between the ritual of IT hiring and the actual goals of IT hiring become most apparent when we look for specific as opposed to generalist skills. We hire generalists to be on staff over a long period. We hire specialists to perform difficult but real tasks that can eventually be handed over to our generalists – when we need to get something specific done.

Which gets us to the point of this post. What are the skills we should look for when hiring for a HoloLens developer? And what are the skills a HoloLens developer should be highlighting on her resume?

At this point in time, when there is still no SDK generally available for the HoloLens and all HoloLens coders are working for Microsoft and under various NDAs, it is hard to say. Fortunately, important clues have been provided by the recent announcement of the first consulting agency dedicated to the HoloLens and co-founded by someone who has been working on HoloLens applications for Microsoft over the past year. The company Object Theory was just started by Michael Hoffman and Raven Zachary and they threw up a website to advertise this new venture.

Among the tasks involved in creating this sort of extremely specialized website is explaining what capabilities you offer. First, they offer experience since Hoffman has worked on several of the demos that Microsoft has been exhibiting at conferences and in promotional videos. But is this enough of a differentiator? What skills do they have to offer to a company looking to build a HoloLens application?

This is part of the fascination of their “Work” page. It cannot describe any actual work since the company just started and hasn’t technically done any technical work. Instead, it provides a list of capabilities that look amazingly like resume keywords – but different from any keywords you may have come across:

- - - - Entirely new Natural User Interfaces (NUI)
      - Surface reconstruction and object persistence
      - 3D Spatial HRTF audio
      - Mesh reduction, culling and optimization
      - Baked shadows and ambient occlusion
      - UV mapping
      - Optimized render shaders
      - Efficient WiFi connectivity to back-end services
      - Unity and DirectX
      - Windows 10 APIs

These, in fact, are probably the sorts of skills you should be putting on your resume – or learning about in order to put on your resume – if getting a job programming HoloLens is your goal.

The verso side of this coin is that the list can also be turned into a great set of interview questions for someone thinking of hiring for HoloLens development, for instance:

Explain the concept of NUI to me.

Tell me about your experience with surface reconstruction and object persistence.

What is 3D spatial HRTF audio and why is it important for engineering HoloLens apps?

What are mesh reduction, mesh culling and mesh optimization?

Do you know anything about baked shadows and ambient occlusion?

Describe how you would go about performing UV mapping.

What are optimized render shaders and when would you need them?

How does the HoloLens communicate with external services such as a database?

What are the advantages and disadvantages of developing in Unity vs DirectX?

Describe the Windows 10 APIs that are used in HoloLens application development.

Then again, maybe these questions are a bit too abstruse?

HoloLens App Development with Unity

June 25, 2015 James Ashley

A few months ago I wrote a speculative piece about how HoloLens might work with XAML frameworks based on the sample applications Microsoft had been showing.

Even though Microsoft has still released scant information about integration with 3D platforms, I believe I can provide a fairly accurate walkthrough of how HoloLens development will occur for Unity3D. In fact, assuming I am correct, you can begin developing games and applications today and be in a position to release a HoloLens experience shortly after the hardware becomes available.

To be clear, though, this is just speculative and I have no insider information about the final product that I can talk about. This is just what makes sense based on publicly available information regarding HoloLens.

Unity3D integration with third party tools such as Kinect and Oculus Rift occurs through plugins. The Kinect 2 plugin can be somewhat complex as it introduces components that are unique to the Kinect’s capabilities.

The eventual HoloLens plugin, on the other hand, will likely be relatively simple since it will almost certainly be based on a pre-existing component called the FPSController (in Unity 5.1 which is currently the latest).

To prepare for HoloLens, you should start by building your experience with Unity 5.1 and the FPSController component. Here’s a quick rundown of how to do this.

Start by installing the totally free Unity 5.1 tools: http://unity3d.com/get-unity/download?ref=personal

Next, create a new project and select 3D for the project type.

Click the button for adding asset packages and select Characters. This will give you access to the FPSController. Click done and continue. The IDE will now open with an practically empty project.

At this point, a good Unity3D tutorial will typically show you how to create an environment. We’re going to take a shortcut, however, and just get a free one from the Asset Store. Hit Ctrl+9 to open the Asset Store from inside your IDE. You may need to sign in with your Unity account. Select the 3D Models | Environments menu option on the right and pick a pre-built environment to download. There are plenty of great free ones to choose from. For this walkthrough, I’m going to use the Japanese Otaku City by Zenrin Co, Ltd.

After downloading is complete, you will be presented with an import dialog box. By default, all assets are selected. Click on Import.

Now that the environment you selected has been imported, go the the scenes folder in your project window and select a sample scene from the downloaded environment. This will open up the city or dungeon or forest or whatever environment you chose. It will also make all the different assets and components associated with the scene show up in your Hierarchy window. At this point, we want to add the first-person shooter controller into the scene. You do this by selecting the FPSController from the project window under Assets/Standard Assets/Characters/FirstPersonCharacter/Prefabs and dragging the FPSController into your Hierarchy pane.

This puts a visual representation of the FPS controller into your scene. Select the controller with your mouse and hit “F” to zoom in on it. You can see from the visual representation that the FPS controller is basically a collision field that can be moved with a keyboard or gamepad that additionally has a directional camera component and a sound component attached. The direction the camera faces ultimately become the view that players see when you start the game.

Here is another scene that uses the Decrepit Dungeon environment package by Prodigious Creations and the FPS controller. The top pane shows a design view while the bottom pane shows the gamer’s first-person view.

You can even start walking through the scene inside the IDE by simply selecting the blue play button at the top center of the IDE.

The way I imagine the HoloLens integration to work is that another version of FPS controller will be provided that replaces mouse controller input with gyroscope/magnetometer input as the player rotates her head. Additionally, the single camera view will be replaced with a two camera rig that sends two different, side-by-side feeds back to the HoloLens device. Finally, you should be able to see how all of this works directly in the IDE like so:

There is very good evidence that the HoloLens plugin will work something like I have outlined and will be approximately this easy. The training sessions at the Holographic Academy during /Build pretty much demonstrated this sort of toolchain. Moreover, this is how Unity3D currently integrates with virtual reality devices like Gear VR and Oculus Rift. In fact, the screen cap of the Unity IDE above is from an Oculus game I’ve been working on.

So what are you waiting for? You pretty much have everything you already need to start building complex HoloLens experiences. The integration itself, when it is ready, should be fairly trivial and much of the difficult programming will be taken care of for you.

I’m looking forward to seeing all the amazing experiences people are building for the HoloLens launch day. Together, we’ll change the future of personal computing!

Marshall McLuhan and Understanding Digital Reality

June 23, 2015 James Ashley

Understanding McLuhan

While slumming on the internet looking for new content about digital media I came across this promising article entitled Virtual Reality, Augmented Reality and Application Development. I was feeling hopeful about it until I came across this peculiar statement:

“Of the two technologies, augmented reality has so far been seen as the more viable choice…”

What a strange thing to write. Would we ever ask whether the keyboard or the mouse is the more viable choice? The knife or the fork? Paper or plastic? It should be clear by now that this is a false choice and not a case of having your cake or eating it, too. We all know that the cake is a lie.

But this corporate blog post was admittedly not unique in creating a false choice between virtual reality and augmented reality. I’ve come across this before and it occurred to me that this might be an instance of a category mistake. A category mistake is itself a category of philosophical error identified by the philosopher Gilbert Ryle to tackle the sticky problem of Cartesian dualism. He pointed out that even though it is generally accepted in the modern world that mind is not truly a separate substance from mind but is in fact a formation that emerges in some fashion out of the structure of our brains, we nevertheless continue to divide the things of the world, almost as if by accident, into two categories: mental stuff and material stuff.

sony betamax

There are certainly cases of competing technologies where one eventually dies off. The most commonly cited example is the Betamax and VHS. Of course, they both ultimately died off and it is meaningless today to claim that either one really succeeded. There are many many more examples of apparently technological duels in which neither party ultimately falls or concedes defeat. PC versus Mac. IE vs Chrome. NHibernate vs EF. etc.

The rare case is when one technology completely dominates a product category. The few cases where this has happened, however, have so captured our imaginations that we forget it is the exception and not the rule. This is the case with category busters like the iPhone and the iPad – brands that are so powerful it has taken years for competitors to even come up with viable alternatives.

What this highlights is that, typically, technology is not a zero sum game. The norm in technology is that competition is good and leads to improvements across the board. Competition can grow an entire product category. The underlying lie, however, is perhaps that each competitor tells itself that they are in a fight to the death and that they are the next iPhone. This is rarely the case. The lie beneath that lie is that each competitor is hoping to be bought out by another larger company for billions of dollars and has to look competitive up until that happens. A case of having your cake and eating it, too.

stop_staring

There is, however, a category in which one set of products regularly displace another set of products. This happens in the fashion world.

Each season, each year, we change out our cuts, our colors and accessories. We put away last year’s fashions and wouldn’t be caught dead in them. We don’t understand how these fashion changes occur or what rules they obey but the fashion houses all seem to conform to these unwritten rules of the season and bring us similar new things at the proper time.

This is the category mistake that people make when they ask things such as which is more viable: augmented reality or virtual reality? Such questions belong to the category of fashion (which is in season: earth tones or pastels?) and not to technology. In the few unusual cases where this does happen, then the category mistake is clearly in the opposite direction. The iPhone and iPad are not technologies: they are fashion statements.

Virtual reality and augmented reality are not fashion statements. They aren’t even technologies in the way we commonly talk about technology today – they are not software platforms (though they require SDKs), they are not hardware (though they are useless without hardware), they are not development tools (you need 3D modeling tools and game engines for this). In fact, they have more in common with books, radio, movies and television than they do to software. They are new media.

A medium, etymologically speaking, is the thing in the middle. It is a conduit from a source to a receiver – from one world to another. A medium lets us see or hear things we would otherwise not have access to. Books allow us to hear the words of people long dead. Radio transmits words over vast distances. Movies and television let us see things that other people want us to see and we pay for the right to see those things. Augmented reality and virtual reality, similarly, are conduits for new content. They allow us to see and hear things in ways we haven’t experienced content before.

The moment we cross over from talking about technology and realize we are talking about media, we automatically invoke the spirit of Marshall McLuhan, the author of Understanding Media: The Extensions of Man. McLuhan thought deeply about the function of media in culture and many of his ideas and aphorisms, such as “the medium is the message,” have become mainstays of contemporary discourse. Other concepts that were central to McLuhan’s thought still elude us and continue to be debated. Among these are his two media categories: hot and cold.

understanding media

McLuhan claimed that any media is either hot or cold, warm or cool. Cool mostly means what we think it means metaphorically; for instance, James Dean is cool in exactly the way McLuhan meant. Hot media, in turn, is in most ways what you would think it is: kinetic with a tendency to overwhelm the senses. To illustrate what he meant by hot and cold, McLuhan often provides contrasting examples. Movies are a hot medium. Television is a cold medium. Jazz is a hot medium. The twist is a cool medium. Cool media leave gaps that the observer must fill in. It is highly participatory. Hot media is a wall of sensation that does not require any filling in: McLuhan characterizes it as “high definition.”

I think it is pretty clear, between virtual reality and augmented reality, which falls into the category of a cool medium and which a hot one.

To help you come to your own conclusions about how to categorize augmented reality glasses and the virtual reality goggles, though, I’ll provide a few clues from Understanding Media:

“In terms of the theme of media hot and cold, backward countries are cool, and we are hot. The ‘city slicker’ is hot, and the rustic is cool. But in terms of the reversal of procedures and values in the electric age, the past mechanical time was hot, and we of the the TV age are cool. The waltz was hot, fast mechanical dance suited to the industrial time in its moods of pomp and circumstance.”

“Any hot medium allows of less participation than a cool one, as a lecture makes for less participation than a seminar, and a book for less than dialogue. With print many earlier forms were excluded from life and art, and many were given strange new intensity. But our own time is crowded with examples of the principle that the hot form excludes, and the cool one includes.”

“The principle that distinguishes hot and cold media is perfectly embodied in the folk wisdom: ‘Men seldom make passes at girls who wear glasses.’ Glasses intensify the outward-going vision, and fill in the feminine image exceedingly, Marion the Librarian notwithstanding. Dark glasses, on the other hand, create the inscrutable and inaccessible image that invites a great deal of participation and completion.”

audrey hepburn glasses