Certain people are bellwethers for creative technology and you want to check in on what they are up to every 3 to 6 months to find out where the zeitgeist of the coding world is headed. I’m thinking of people like Kyle McDonald, James George and Phoenix Perry – folks who, per Jean Cocteau’s maxim, manage to stay on the avant-garde even when everyone else has caught up to what had been the avant-garde half a year earlier.
Phoenix is currently teaching physical computing in London. She has spoken and led workshops at most of the leading conferences devoted to emerging technology. You can (and should) keep up with her adventures on her website, phoenixperry.com, and on twitter.
What movie has left the most lasting impression on you? 2046.
What is the earliest video game you remember playing? Pong.
Who is the person who has most influenced the way you think? The women of Code Liberation. Over the duration of the organization have radically shifted how I think and who I am. Mentoring younger women in tech has changed who I am. The conversations we have are inspired and open up my mind to a deeper, more compassionate way to live.
When was the last time you changed your mind about something? This week. The river is different every time you step into it.
What’s a programming skill people assume you have but that you are terrible at? I think it’s more a skill level. People assume I’m some super expert but the truth is I’m often relearning my skill set constantly because it’s so broad. For example, every single time I look at javascript, it’s brand new all over. I’ll go delve in an area deeply and the other spaces will move forward and I’m a novice all over again.
What inspires you to learn? Humility at how little I know.
What do you need to believe in order to get through the day? I need to believe the people around me value my work and contributions.
What’s a view that you hold but can’t defend? I hate Opera. Have no real reason why other than it sounds so annoying to my ears. What will the future killer Mixed Reality app do?
One that allows me to interact with one experience fluidly across contexts.
With the Mixed Reality Toolkit 2017.1.2, the Keyword Manager was finally retired, after being “obsolete” for the past several toolkit iterations.
As the toolkit matures, many key components are being refactored to make them more flexible and architecturally proper. The downside to this – and the source of much frustration – is that these refactors tend to upend what developers are used to doing. The Keyword Manager is a great example of this. It was one of the best Unity style interfaces for HoloLens / MR because it encapsulated a lot of complex code in a easy to use drag-and-drop style visual.
The challenge for those working on the toolkit – Stephen Hodgson, Neeraj Wadhwa, and all the others – is to refactor without too badly breaking the interface abstractions we’ve all gotten used to. For the KeywordManager refactor, this was accomplished by breaking the original component into two parts, a SpeechInputSource and a SpeechInputHandler.
The SpeechInputSource lets you determine if speech recognition starts up automatically and the recognition confidence level (you want this higher if you are using ambiguous or short phrases like “Start” and “Stop”). The Persistent Keywords field lets you keep the same speech recognition phrases between different scenes in your app. Most important, though, is the Keywords list. This lets you add to a list of phrases you want to be recognized in your app.
The SpeechInputHandler is the component that lets you determine what happens when a phrase is recognized (the response). You click on the plus icon to add a response, select the phrase that will be handled in the Keyword field, and then can drag and drop gameobjects into your response and select the script and method that is called.
The one thing you need to remember to do is the check off the Is Global Listener field if you want behavior similar to the old KeywordManager. This will listen for all speech commands all the time. If Is Global Listener is not selected, then only the SpeechInputHandler the user is gazing at will receive commands. This is really useful if you have multiple copies of the same object and only want to apply commands to a particular instance at a time.
Roland Smeenk belongs to a species of coders variously known as creative technologists, new media developers, creative designers, etc. As with human civilization, there came a time when coders were expected to settle down in one place and learn to cultivate their code repositories while also effectively implementing division of labor and managerial techniques in order to get us where we are today.
But there are always the hunter-gatherers like Roland who insist on traveling unworn paths, who resist division of labor and want to learn everything for themselves. They stand athwart history, it is true, but they are also the innovators who determine what history will become.
Roland wrote and published the HoloLens Shader Pack to the Unity Asset Store. This lets you have a range of configurable scanning effects like the ones you’ve probably seen in Microsoft’s official HoloLens demo reels – plus a lot more. It’s one of the earliest HoloLens related tools in the Asset Store and an essential component for anyone working on professional MR applications.
What movie has left the most lasting impression on you? Minority Report, because it envisioned future digital interfaces that are already technically feasible these days. From a human factors standpoint most of them are not what you would want in a system that requires efficient manipulation of data. However from an esthetics standpoint they provide a great inspiration for making interactive installations.
What is the earliest video game you remember playing? I remember playing Pong on my uncle’s television back in the seventies, but that was only one afternoon. It lasted until the eighties before my father bought an Apple IIe personal computer. I played only a few games on that system with it’s monochrome screen. One game that impressed me was “Dung Beetles”, because of the audio sample “We’ve gotcha” that was coming from a speaker that normally only produced simple beeps.
Who is the person who has most influenced the way you think? I believe there’s a mix of people and experiences that influence your way of thinking in different areas. So what comes to mind are my parents, wife, friends, former colleagues, people I follow online and authors of books I read, all in different areas.
When was the last time you changed your mind about something? Three questions ago.
What’s a programming skill people assume you have but that you are terrible at? I don’t know what people are assuming about my programming skills. They are probably better than my mechanical engineering skills that I earned my master’s degree for many moons ago.
What inspires you to learn? There’s a lot of ground to be explored in mixing the digital and the real world in interesting new ways. And this goes well beyond the use of HoloLens. For interactive experiences it often starts by drawing inspiration from sci-fi movies, music videos, games, artworks and new technology, but I also like a good story as a starting point. This eventually leads to what I need to learn. Lately these are often areas where no software is involved.
What do you need to believe in order to get through the day? That the things I create will touch people in a positive way.
What’s a view that you hold but can’t defend? That there is no life after death. At least this reminds me to live life to the fullest.
What will the future killer Mixed Reality app do? There will not be one killer app, but instead it will be a ubiquitous ecosystem that understands us and helps us do all kinds of everyday tasks. This will give us more free time to spend with family and friends and frees our minds for other things.
My friend Dennis Vroegop and I have recorded a course for LinkedIn Learning (Lynda.com) called Developing Apps for the Microsoft HoloLens. We wanted this to be the get up and go course covering the HoloLens that takes you from zero to 60, while also providing a larger understanding of the device itself, it’s place in history, and what the current market for the HoloLens is.
As we say in the course, this is what we wished we had had when we were starting off with HoloLens a year ago. All in all, I think we succeeded beyond our wildest dreams. LinkedIn Learning is offering a free initial month to their video courses, so you can watch the whole thing and start programming on the emulator, if you want, with free software from Microsoft and Unity. All it takes is a will. This is the way.
Our producer on the shoot, Dennis Meyer, has written his insights on the whole process and what he now things about AR and VR after going through this adventure with me and Dennis V.
I hope you enjoy the course. Please let me know what you think in the comments.
Sky Zhou is a HoloLens developer who has spent the past year exploring some of the most difficult MR development problems such as storing and retrieving spatial maps and the ins-and-outs of the locatable camera coordinate system. He recently tested out his Room Scanner with help from members of the HoloDevelopers slack group. You can signup to try the public beta now.
If his name sounds familiar, you may be remembering his earlier work on a Pokemon fighting app for the HoloLens.
What movie has left the most lasting impression on you? I have two, The Matrix and Inception; both pose philosophical questions about the nature of existence and reality.
In addition, I like The Matrix for its vision of the future, in terms of the possibility of simulating a world with computer programs and its depiction of the power and danger of AI. I like Inception for its crafty metaphoric story about story telling itself as an art form. HoloLens brought us a giant step closer to the visions of simulated 3D world and alternate realities like dreams. Mixed reality will be a great art form for storytelling.
What is the earliest video game you remember playing? I didn’t have much play time on video games when I was little. I can remember glympses of super mario, street fighters, and ninja turtles. However, I remember very well the first computer game I played. It was an RPG based on the Romance of Three Kingdoms. I enjoyed the game and the story so much. I wish I can build something like that in mixed reality with HoloLens.
Who is the person who has most influenced the way you think? Laozi, an ancient Chinese philosopher from 600 BC. It’s hard to explain all his deep thinking here and my interpretation might be flawed. However, if I have to narrow down the idea: things are observed and defined by us, and they can be observed and defined in many other ways. Therefore, nothing has to be absolute and we need not attach ourselves to extremes defined by us. Once you see that, you can always reach a balance, a compromise, or a feasible solution. It really helps me keep an open mind to this world. The concept of mixed reality could help us understand this fine point. Many different observations and representations can be made in the same physical space, depending on what apps you are running and seeing through the HoloLens, so we can all agree that we don’t always get the same perspective and the same information, and therefore we end up forming different conclusions and decisions.
When was the last time you changed your mind about something? In general, I change my mind based on additional information and evidence. For example, I used to think Microsoft was a lazy non-innovative company that puts no effort into making another great product and is just milking profit from its Windows and Office software. However, the first public demo of HoloLens in January 2015 blew my mind and completely changed how I see Microsoft.
What’s a programming skill people assume you have but that you are terrible at? People assume I have some fixed programming skills in some languages, but what I really have is the ability to solve problems, by doing research and experimenting. In this regard, I owe many thanks to the awesome developer community around HoloLens and windows mixed reality.
What inspires you to learn? 1. Problems! I love to apply knowledge to solve problems. It pushes me to learn new things and often lead to even more new knowledge. For example, the frame rate was poor on my first demo of photo-textured room scanning on HoloLens. In order to improve performance on this app, I learned how to use compute shaders.
2. Community. With a great community around HoloLens, you feel you are not alone in this crazy endeavor, and there are people who value what you do. Other developers’ awesome demos also inspire me to do more myself. The feedback I get is the most valuable for alerting me to problems and pushing me to make my apps better. For example, when I saw 3 comments about poor frame rate on my Youtube channel, I knew I had to figure out a way to address it.
What do you need to believe in order to get through the day? There is another problem to solve and a new thing to learn! Plus my HoloLens is still working!
What’s a view that you hold but can’t defend? I hold a view that Microsoft got it right this time, with their innovative approach to mixed reality. I can see consumer and business adoption at a massive scale because this technology truly brings amazing utility never seen before, once certain limitations are worked out (such as price, field of view, weight, full gesture recognition, etc.). Windows Mixed Reality is like an extension of Windows with all the familiarities from the existing flat desktop, but in 3D space. This familiarity in non-gaming uses of PCs makes it much more attractive to consumers and businesses.
What will the future killer Mixed Reality app do? It depends on what pain it kills. Different people experience different pains. For example, a common pain is a long commute time. One possible solution is that company offices can be significantly reduced in size and most people work from home or anywhere they choose. Many large companies are currently heading in this direction already. Besides using mixed reality tools for creative 3D art work, an immersive virtual home office could improve focus, and shared experiences with the holo-presence of colleagues could improve teamwork and collaborations. Thus, the effectiveness and efficiency of working from home can be improved.
The ultimate dream of mixed reality is that you have 3D digital information and holograms overlay on top of a real physical environment no matter where you go and users can interact with them in shared experiences. Think of Pokémon Go plus Google street view, but with useful and relevant information wherever and whenever you need them. This requires much finer mapping—finer than Google street view currently provides—to your desk, your fridge, your car, and even yourself. This idea is not original to me: Ori Inbar wrote a great article recently about it and termed it “the AR Cloud.” When that happens, in essence we will have also created a massive interactive virtual world like the Matrix.
Stephen is an immersive experiences developer at Valorem. He is also a moderator and the most prolific contributor to the HoloToolkit (now the Mixed Reality Toolkit). If there has been a change to the toolkit over the past year that improved your developer experience, Stephen probably made it. Moreover, he does this in his free time.
I spent an evening online with him about two weeks ago fixing an issue with automatic builds for Visual Studio 2017 and spent much of that time wondering how one person could maintain so much knowledge about such a large and complex code base. He is the centurion at the gate, finding satisfaction when our Mixed Reality projects go well and happy when he can provide tools to make our development easier. The next time you successfully deliver a project, please consider sending Stephen a thank you.
What movie has left the most lasting impression on you? Arrival was pretty interesting. I really find time travel and their paradoxes quite fascinating. The looping nature of the alien language and the connection to the movies main theme really helped drive the action. I also loved how the movie started at both the beginning and end. I enjoy many movies that range across different genres, from sci-fi to action.
What is the earliest video game you remember playing? Chrono Cross by Square Enix is definitely the one I remember most vividly, but I’m pretty sure the first video game I ever played was the original Mario game on the Nintendo.
Who is the person who has most influenced the way you think? This one is pretty difficult to answer. I’m not sure if any one person had a huge impact on why I think the way I do. I’ve always been off in my own thoughts thinking about how light, electromagnetism, and matter work together—oh and also the paradoxes of time travel, haha.
When was the last time you changed your mind about something? I feel like I change my mind a lot. Generally, I like to approach things from many different angles and evaluate them to see whichever option is the best, and even then I’m likely to make an adjustment down the line. When it matters, I try to corner myself into making a decision that I can’t back out of.
What’s a programming skill people assume you have but that you are terrible at? Professionally I like to stay pretty honest about my abilities and accept work that fits my skill level while still challenging me to grow. I definitely feel like I need to get stronger in networking and understanding the lower level languages a bit more, although the need for memory management is no longer a requirement, it’s nice to know exactly why a function or class works the way it does.
What inspires you to learn? Leaning is a part of life, and is essential to survival. On that note, I’ve always liked the notion that the only way to learn is to make mistakes. I’ve always found it frustrating when people can’t admit they’re wrong about something, because they are admitting they don’t make mistakes, which in turn leads me to believe that they’re incapable of learning new ideas.
What do you need to believe in order to get through the day? Tomorrow never comes. I usually don’t think about the end of the day until it arrives, and by then I’m already planning tomorrow.
What’s a view that you hold but can’t defend? Not sure how to answer this one. I usually don’t hold a view that can’t stand on its own.
What will the future killer Mixed Reality app do? The whole idea of mixed reality apps is to bring the virtual into the real world. There’s been a big push recently about how the internet will dramatically change after the association with a physical space and it’s virtual space—like a webpage or other virtual element—can be brought together to create a meaningful productive space.
What book have you recommended the most? I haven’t read an actual book in quite a while, but growing up I was definitely a bookworm. Now I read code all day. If I had to recommend reading, it’s the new APIs and change logs from Unity, Windows 10 SDK, and the Mixed Reality Toolkit for Unity.
Nate is a multimedia engineer currently freelancing in New York City. He is also a good friend going back to our time together at Razorfish Emerging Experiences where we worked side-by-side pushing the envelope on interactive installation projects with Surface tables, IoT, Kinect, tablets, phones, VR, and several devices we hacked together ourselves.
Over a year ago, while most people playing with HoloLens were still finding their way around the Unity IDE, Nate was exploring how to build artistic effects for the HoloLens using DirectX and seeing how hard he could push the HoloLens GPU. Since then, he’s been working on various HoloLens and HTC Vive projects to earn his bread while pursuing his ultimate goal of creating cool interactive experiences involving music, lighting, technology, dance and people. He’s someone I’ve always admired as much for the clarity of his artistic vision as for his technical prowess.
What movie has left the most lasting impression on you? I’ve never been able to sit still long enough to gain a real appreciation for movies. I only watch movies on airplanes these days. Fight Club had a pretty big impact on me, though. I first saw it at a very impressionable age and there always seems to be relevant scenes popping into my head.
What is the earliest video game you remember playing? It’s hard for me to place… I grew up with an engineer for a Dad and surrounded by techy folks. I remember playing tennis on VirtualBoy. We had some early atari system as well. I also had a Macintosh SE as my first computer (older than me) that I used to play this weird 2.5D pong game called Shufflepuck Café.
Who is the person who has most influenced the way you think? I’m influenced by many people in my life. I believe you should try to glean something from everyone you admire and respect. I’ve had many great mentors along the way (including James).
My parents, of course, taught me a ton about how to live and reach my goals. My Dad taught me everything I know about computers. When I was young we’d go on long drives together and he’d be telling me about how CPU caches work. It was wild. My Mom has been an artist for many years and she’s always encouraged me to keep my artistic side alive.
For one I’ve never met, I’d pick Aldous Huxley. I discovered his writing at a time when I needed some direction in my life pretty badly. He has a way of capturing the intangible nature of the human experience with words better than any other writer I know, and it’s helped shape the way I consider my place in the world more than anyone else.
When was the last time you changed your mind about something? I’m a Gemini, I do it constantly. Recently I changed my opinion of mobile AR. I thought it didn’t have much of a future. Gotta give it up to Apple and all the cool things people are making with ARKit. Pretty excited now.
What’s a skill people assume you have but that you are terrible at? I’m not very good at math, and I don’t really enjoy it. I failed Calculus II. Almost twice. I like making stuff with math though. Right now I’m working with chaotic systems to create natural movement and that is fun, but the analytical stuff isn’t for me.
What inspires you to learn? I think it’s really essential to what it means to be human. Plus I get bored easily. I always need to feel like I’m pushing some edge in one aspect or another. I feel very blessed that I’ve essentially made a career out of learning new things.
What do you need to believe in order to get through the day? This too shall pass.
What’s a view that you hold but can’t defend? Brunch culture was systemically created as a means of controlling the populace.
What will the future killer Mixed Reality app do? It will make talking about reality in terms of virtual, mixed. Our language will gradually phase out making any distinction between real and virtual. It’s funny how much time people spend debating what is AR/MR/VR. We need to move beyond that kind of thinking.
What book have you recommended the most? I recommend different books to different people. I don’t think I have a one size fits all answer, so I’m going for the audience here.
I’ve gotta say Neuromancer. It’s a must read for anyone in this field. Gibson literally wrote the book on our present and future in regards to the internet, mixed reality, and artificial intelligence. Oh, and he did it on a typewriter in the 80s.
[Update 10/17/17: I’m currently doing HoloLens development with VS 2017 v5.3.3, Unity 2017.1.2f1, MRTK 2017.1.2, and W10 17010. (This is one version of Windows 10 past what is recommended.) Unity 2017.2 for WMR development. There’s a new release of Visual Studio 2017 5.4 that I haven’t installed, yet. I am using Unity 2017.3.0 beta 5 for WebGL work.]
[Update 9/8/17: I’m currently having success in HoloLens development with Visual Studio 2017 (15.3.3), Unity 2017.1.0p5 (x64), SDK 14393, and HTK v1.2017.1.1 . This allows me to deploy Release builds, Unity C# Projects, and lets me debug remotely (my breakpoints are getting hit) from the regular File | Build Settings window. From the HoloToolkit Build and Deploy window, checking Unity C# Project throws an error. (To work around this, I am using two different deployment folders, one for debugging and one for non-debugging deployments.) I don’t think it matters, but I’m on Windows 10 insider build 16281. Will update if anything changes.]
[Update 9/7/17: Unity 2017.2.0b10 is definitely no go for HoloLens development. It just tries to run from the tile window and then goes black screen. Probably necessary for Immersive MR development, though.]
Sometimes it’s hard to know when to move forward and when to go backwards. If you are developing with Microsoft Mixed Reality products today (by which I mean on or around September 5th, 2017) you may need to do both.
Because common tools consisting of Visual Studio, Unity 3D and the open source Mixed Reality Toolkit (formerly the HoloToolkit) are used for both, and because we are at a point where multiple tools are being released at a rapid rate, conflicts are bound to happen, unfortunately.
For HoloLens development, it is best to roll back to the last working version of the tools you had. If you aren’t sure what those were, then I’d recommend using Visual Studio 2015 update 3, Unity 3D 5.6.3 patch 2 and HoloToolkit 1.5.8.0.
For immersive mixed reality, you want the latest of everything. As of today, that means Unity 2017.2.0b10 (and then b11 when that comes out). You will need Visual Studio 2017 update 3 (15.3) and finally you will need to be on the latest Windows Insider Build of Windows 10 from the fast ring (for instance build 16281, which has some very nice updates to the MR Portal).
These are the versions I could get working consistently. Other combinations, for HoloLens development, could potentially raise the following issues (though your mileage may vary – if so, please let me know in the comments):
unable to create Master builds, dev or release okay
apps deploy successfully but crash before the splash screen
unable to use remote debugging on your deployed app – get a dll missing error or “Please ensure that target device has developer mode enabled” error
IL weaving issues (this occurs for Visual Studio update 3 (15.3.3) and is fixed in patch 2017.1.0p5 as well as 5.6.3.0p2 (if you are using Visual Studio 2015 update 3)
Unable to create Unity C# Project builds (for debugging)
MR development with the wrong tool combination has raised some of these issues, such as the C# Project builds and IL weaving. In addition, there may be clipping issues if you don’t have the latest tools.
Note: You don’t really need Visual Studio 2017 for HoloLens development. Use VS 2015, instead. You only require VS 2017 if you are developing for Windows SDK 10.0.15063 or above. The HoloLens device, however, still runs on windows build 14393. When you export your solution from Unity, just be sure to change the UWP SDK property in Build Settings from its default value “Latest installed” to “10.0.14393.0”.
Note 2: You cannot currently publish your immersive MR app to the Windows Store. Just hold tight for now. It’s being worked out.
Quick recap: for now, you need different tool versions for HoloLens and Immersive MR Headset development. Use older tools for HoloLens but the latest and greatest for MR Headsets.
I will try to keep this updated as facts on the ground change.
Kevin is a Director at Microsoft who builds proofs of concepts for their most innovative technologies such as the HoloLens. He guided the HoloLens / Lowe’s collaboration which provides a much copied template for retail solutions using AR.
For those of you familiar with Bill Buxton’s book Sketching User Experiences – especially the sections about the design of the Active Desk – there is an easy analogy to draw here. Kevin’s role is to sketch and prototype the future before the future arrives.
What movie has left the most lasting impression on you? The Matrix.
What is the earliest video game you remember playing? Asteroids.
Who is the person who has most influenced the way you think? My father.
When was the last time you changed your mind about something? Last week.
What’s a skill people assume you have but that you are terrible at? How about dislike doing? Project management.
What inspires you to learn? The challenge.
What do you need to believe in order to get through the day? #HowNotIf .
What’s a view that you hold but can’t defend? Christianity.
What will the future killer Mixed Reality app do? Personal assistant – everything you need without having to find it.
Michael Washington, a decade-long Microsoft MVP out of Southern California, created one of the first blogs devoted to HoloLens development in 2016 called, conveniently, hololenshelpwebsite. In those first few months after the HoloLens was released, he addressed many of the scenarios we are still struggling with today such as capturing room meshes and communicating between devices. I still don’t completely understand how he figured so much out about the inner workings of the HoloLens development platform in such a short period of time.
Besides his incredible work, though, what I find most fascinating about Michael’s relationship with the HoloLens is that he has moved on. While many of the rest of us early adopters are waiting to see how augmented reality, virtual reality, mixed reality, and x-reality all shake out – and amid conflicting and confusing news stories from companies like Meta and Magic Leap – Michael actually sold his HoloLens to help pay for a new car and is patiently waiting for the HoloLens 2 to come out. In the meantime, he has written a book, Azure Machine Learning for the Non-Data Scientist, and is digging deep into AI.
While most of the profiles in this series are about people currently doing development work around the HoloLens, I wanted to profile Michael because he also represents a significant portion of our ecosystem. He hasn’t abandoned the technology, but he has put a pin in it for now despite having invested a significant amount of time in it at one point. It’s worth remembering that the HoloLens device is pretty expensive (about the price of a family vacation or a used car), and the barrier to entry was substantial for developers who just wanted to get their hands on it to learn how to make holographic apps.
What movie has left the most lasting impression on you? Unforgiven (Clint Eastwood) – It is a simple movie but actually deep and complex. They set off to do this simple thing to make some money. Along the way they run into “people” who have fears and motivations. They have to make hard decisions. It is an allegory for every software project I have ever worked on
What is the earliest video game you remember playing? Asteroids on a table top in a hotel lobby.
Who is the person who has most influenced the way you think? The Dalai Lama. Want to be happy, let go of attachments and expectations and have acceptance.
When was the last time you changed your mind about something? I didn’t think it was important to have transgenders in the military until they threatened to take that away. Now I think it is important .
What’s a skill people assume you have but that you are terrible at? SQL queries. I still use the designer even to do simple joins.
What inspires you to learn? Knowing that I can use that knowledge to make people’s lives better.
What do you need to believe in order to get through the day? That having acceptance of all things at all times will relieve all suffering.
What’s a view that you hold but can’t defend? That having acceptance of all things at all times will relieve all suffering
What will the future killer Mixed Reality app do? It will allow me to answer my emails while driving.
What book have you recommended the most? Super Powereds. This guy is one of the best writers alive.
