March 2015 – The Imaginative Universal

Unity 5 and Kinect 2 Integration

March 27, 2015 James Ashley7 Comments

pointcloud

Until just this month one of the best Kinect 2 integration tools was hidden, like Rappuccini’s daughter, inside a walled garden. Microsoft released a Unity3D plugin for the Kinect 2 in 2014. Unfortunately, Unity 4 only supported plugins (bridges to non-Unity technology) if you owned a Unity Pro license which typically cost over a thousand dollars per year.

On March 3rd, Unity released Unity 5 which includes plugin support in their free Personal edition – making it suddenly very easy to start building otherwise complex experiences like point cloud simulations that would otherwise require a decent knowledge of C++. In this post, I’ll show you how to get started with the plugin and start running a Kinect 2 application in about 15 minutes.

(As an aside, I always have trouble keeping this straight: Unity has plugins, openFrameworks as add-ins, while Cinder has bricks. Visual Studio has extensions and add-ins as well as NuGet packages after a confusing few years of rebranding efforts. There may be a difference between them but I can’t tell.)

1. First you are going to need a Kinect 2 and the Unity 5 software. If you already have a Kinect 2 attached to your XBox One, then this part is easy. You’ll just need to buy a Kinect Adapter Kit from the Microsoft store. This will allow you to plug your XBox One Kinect into your PC. The Kinect for Windows 2 SDK is available from the K4W2 website, though everything you need should automatically install when you first plug your Kinect into your computer. You don’t even need Visual Studio for this. Finally, you can download Unity 5 from the Unity website.

2. The Kinect 2 plugin for Unity is a bit hard to find. You can go to this Kinect documentation page and scroll half-way down to find the link called Unity Pro Packages. Aternatively, here is a direct link to the most current version of the plugin as of this writing.

3. After you finish downloading the zip file (currently called KinectForWindows_UnityPro_2.0.1410.zip), extract it to a known location. I like to use $\Documents\Unity. Inside you will find three plugins as well as two sample scenes. The three Kinect plugins are the basic one, a face recognition plugin, and a gesture builder plugin, each wrapping functionality from the Kinect 2 SDK.

4. Fire up Unity 5 and create a new project in your known folder. In my case, I’m creating a project called “KinectUnityProject” in the $\Documents\Unity folder where I extracted the Kinect plugins and related assets.

5. Now we will add the Kinect plugin into our new project. When the Unity IDE opens, select Assets from the top menu and then select Import Package | Custom Package …

6. Navigate to the folder where you extracted the KinectforWindows_Unity components and select the Kinect2.0.xxxxx.unitypackage file. That’s our plugin along with all the scripts needed to build a Kinect-enabled Unity 5 application. After clicking on “Open”, an additional dialog window will open up in the Unity IDE called “Importing Package” with lots of files checked off. Just click on the “Import” button at the lower right corner of the dialog to finish the import process. Two new folders will now be added to your Project window under the Assets folder called Plugins and Standard Assets. This is the baseline configuration for any Kinect project in Unity.

7. Now we’ll get a Kinect with Unity project quickly going by simply copying one of the sample projects provided by the Microsoft Kinect team. Go into file explorer and copy the folder called “KinectView” out of the KinectforWindows_Unity folder where you extracted the plugins and paste it into the Assets directory in your project folder. Then return to the Unity 5 IDE. A warning message will pop up letting you know that there are compatibility issues between the plugin and the newest version of Unity and that files will automatically be updated. Go ahead and lie to the Unity IDE. Click on “I Made a Backup.”

8. A new folder has been added to your Project window under Assets called KinectView. Select KinectView and then double click on the MainScene scene contained inside it. This should open up your Kinect-enabled scene inside the game window. Click on the single arrow near the top center of the IDE to see your application in action. The Kinect will automatically turn on and you should see a color image, an infrared image, a rendering of any bodies in the scene and finally a point cloud simulation.

9. To build the app, select File | Build & Run from the top menu. Select Windows as your target platform in the next dialog and click the Build & Run button at the lower right corner. Another dialog appears asking you to select a location for your executable and a name. After selecting an executable name, click on Save in order to reach the final dialog window. Just accept the default configuration options for now and click on “Play!”. Congratulations. You’ve just built your first Kinect-enabled Unity 5 application!

les fruits dangereux

March 21, 2015 James Ashley

potato_clock

What ever happened to the potato clock? In a revived period of do-it-yourselfers, arduino artists and 3d printing presses – a bright new age of artisanal software and hardware development – what has happened to the epitome of nerdish home engineering? At one point in time you couldn’t escape your teenage years without an awareness of the potato clock – stashed somewhere between the x-ray glasses and the sea monkeys – and then poof, suddenly they vanish from the collective consciousness.

Over lunch with my good friends Joel and Nate, I raised this question and Nate was quick to identify the exact date on which the potato clock disappeared. It happened in 2001. On September 11th, to be precise. On that day, many seemingly innocent things were recognized for the danger they are. The test of this perceptual shift occurred on January 31, 2007 when a marketing campaign for the cartoon series Aqua Teen Hunger Force turned into a bomb scare. Whereas levity typically brings with it perspective and reestablishes old norms, in this case – five years after 9/11 – the guerrilla campaign involving led lights was seen as offensive, in poor taste and led to the resignation of Jim Samples, the man who built up Cartoon Network from nothing. Which led us to ask, over our fruit salads, what other common objects – objects like the lowly pototo — might be revealed to have a sinister aspect.

Children have always known the threatening qualities of fruits and vegetables. Adults, on the other hand, have always been somewhat oblivious to this implicit threat that children are aware of in their bones, and have been known to eat kale and Brussels sprouts with reckless abandon. We considered what it would take to capture the danger implicit in fruits and vegetables and realized all it required was a bit of tape and a phone.

Consider the banana. At first appearance, it is a pleasant and unassuming fruit with a waxy peel and a soft, sweet interior. Attach a phone to it with some electrical tape and leave it in the lobby at the airport and its true nature reveals itself. Drop it in the mailbox at the post office and see just how innocent it really is.

Leave this at the entrance to a police station and see who laughs.

Context obviously matters. A banana in a fruit basket suggests a certain functional role. A banana with electronics taped to it sets up a different sort of context. Through experimentation, we found that even the type of tape used can shift the context in subtle but meaningful ways. Silvery duct tape, for instance, is much more threatening than glossy black electrical tape when attached to fruit.

Which begs the question, what is the most threatening fruit? Here is an apple attached to an AT&T Pantech cell phone – one of the last free-with-contract AT&T phones that did not require a data plan and the automatic $25 data fee charged to your account whether you are using data or not – why can’t I just use it for phone calls and wifi, AT&T? In that scenario, the AT&T cellular contract is obviously the most frightening thing.

But what happens when we exchange the Pantech for an HTC Windows Phone?

Or an iPhone for that matter? Which is more intimidating: an iPhone 5 or an iPhone 6?

Here is a Microsoft Band wrapped around an apple. There’s a metaphor in there somewhere.

iPhone 5 and a kiwi.

Samsung Galaxy 4 with onion and duct tape.

I normally associate electrical tape with bombs and duct tape with kidnapping – which makes duct tape more viscerally terrifying for me. Some people claim they associate duct tape with ducks but I think that’s a canard. Onions make me want to cry.

One of life’s riddles: is a coconut a fruit or a vegetable?

No matter what I do, I can’t make strawberries look threatening.

Carrots, on the other hand, are Nature’s terrorists.

These carrots are organic, by the way. The extra cost is worth it for the additional fear factor.

Rubber bands can be intimidating in the right context. Especially when that context is celery. The safety pin is overkill, maybe?

This is a Nokia Lumia Windows Phone 7 developer unit with both front-facing and rear-facing cameras. It is attached to a red plum.

Kindle White meets cantaloupe.

And of course, potatoes: fear incarnate.

The Next Book

March 3, 2015 James Ashley

min_lib

The development community deserves a great book on the Kinect 2 sensor. Sadly, I no longer feel I am the person to write that book. Instead, I am abandoning the Kinect book project I’ve been working on and off over the past year in order to devote myself to a book on the Microsoft holographic computing platform and HoloLens SDK. I will be reworking the material I’ve so far collected for the Kinect book as blog posts over the next couple of months.

As anyone who follows this blog will know, my imagination has of late been captivated and ensorcelled by augmented reality scenarios. The book I intend to write is not just a how-to guide, however. While I recognize the folly of this, my intention is to write something that is part technical manual and part design guide, part math tutorial, part travel guide and part cookbook. While working on the Kinect book I came to realize that it is impossible to talk about gestural computing without entering into a dialog with Maurice Merleau-Ponty’s Phenomenology of Perception and Umberto Eco’s A Theory of Semiotics. At the same time, a good book on future technologies should also cover the renaissance in theories of consciousness that occurred in the mid-90’s and which culminated with David Chalmers’ masterwork The Conscious Mind. Descartes, Bergson, Deleuze, Guattari and Baudrillard obviously cannot be overlooked either in a book dealing with the topic of the virtual, though I can perhaps elide a bit.

A contemporary book on technology can no longer stay within the narrow limits of a single technology as was common 10 or so years ago. Things move at too fast a pace and there are too many different ways to accomplish a given task that choosing between them depends not only on that old saw ‘the right tool for the job’ but also on taste, extended community and prior knowledge. To write a book on augmented reality technology, even when sticking to one device like the HoloLens, will require covering and uncovering to the uninitiated such wonderful platforms as openFrameworks, Cinder, Arduino, Unity, the Unreal Engine and WPF. It will have to cover C#, since that is by and large the preferred language in the Microsoft world, but also help C# developers to overcome their fear of modern C++ and provide a roadmap from one to the other. It will also need to expose the underlying mathematics that developers need to grasp in order to work in a 3D world – and astonishingly, software developers know very little math.

Finally, as holographic computing is a wide new world and the developers who take to it will be taking up a completely new role in the workforce, the book will have to find its way to the right sort of people who will have the aptitude and desire to take up this mantle. This requires a discussion of non-obvious skills such as a taste for cooking and travel, an eye for the visual, a grounding in architecture and an understanding of how empty spaces are constructed, a general knowledge of literary and social theory. The people who create the next world, the augmented world, cannot be mere engineers. They will also need to be poets and madmen.

I want to write a book for them.

Screens, Sensors and Engines

March 3, 2015 James Ashley

Valve’s recent announcement about their new Vive headset for virtual reality as well as Epic’s announcement that the Unreal Engine is now free made me realize that it is time to once again catalog the current set of future technologies vying for our attention. Just as pre-NUI computer users need the keyboard and mouse, the post-NUI user needs sensors and just as the pre-NUI user required a monitor to see what she was doing, the post-NUI user needs a headset. Here is the list for 2015 from which, you will notice, Google Glass is now absent:

Virtual Reality	Augmented Reality	Sensors	Development Platforms

Oculus Rift	Microsoft HoloLens	Microsoft Kinect 2	Unity 3D
Samsung Gear VR	Magic Leap	Leap Motion	Unreal Engine
Google Cardboard	castAR	Myo	WPF
Valve HTC Vive	Epson Moverio	Intel RealSense	Cinder
Sony Project Morpheus		Orbbec	openFrameworks
OSVR Razer		Eye Tribe Tracker
Zeiss VR One