Category: Kinect

Daydreaming about Kinect Fun Labs

James Ashley

cruise-minority-report

While at the CodeStock conference in Knoxville, I had the pleasure of talking with several indie XNA developers about the potential of the Kinect.  They were discouraged, though, at the lack of opportunities for indie developers to create games for the Kinect.  All I knew how to do was create XNA games and put them on a PC with a Kinect hooked up to it – which wasn’t quite what they were looking for.

Then last week at E3 Microsoft announced Kinect Fun Labs.  Currently Fun Labs is a free download for the XBOX with what are basically a few concept apps.  It includes Kinect Me, which scans the player and turns her image into a 3D model (though not, as has been reported, an XBOX avatar that can be used in the rest of the system).  The Kinect then tracks the player and manipulates the model to follow the player’s movements.

Kinect Googly Eyes does basically the same thing with any rectangular object.

And then there are a few additional one-offs that can be downloaded from within the Kinect Fun Labs app.

What is interesting in all this are hints on the official website that we can upload our own apps at some point.  This may be rumor only, of course, and it probably won’t fully satisfy indie developers who would like to make some money off of their games, but once we get past all the ifs and buts …

It would be very cool to have a way to share our Kinect apps – watching demos on Youtube only get us so far.  If the XBOX also supports Silverlight at some point, we will have additionally have a development platform on which to quickly develop Kinect apps.  It’s the stuff dreams are made of.

The WPF Renaissance

James Ashley

mask

The ‘Mysterious Stranger’ is a stock device in literature.  Poe’s main character in The Masque of the Red Death is one instance of the theme.  Mark Twain left us an unfinished novel built around another.  An unexpected guest is central to one of the earliest works in English literature, Sir Gawain and the Green Knight.  In Plato’s late dialogues, Socrates is replaced by the more enigmatic Xenon or Stranger. The old testament is populated by mysterious guests who turn out to be angels.  The Venetian Carnival always potentially allowed commoners to mix with royalty since everyone was in disguise.  Middle Eastern literature is filled with Tufaylîs or ‘unexpected guests’ who would provide additional entertainment to any party should they be allowed over the threshold.  Medieval troubadours appear to have filled the same function in the West.

At the recent MIX 2011 keynotes in Las Vegas, amidst announcements about HTML5, IE9, ASP.NET MVC 3.x, Silverlight 5, Windows Phone Mango and the Kinect SDK, WPF made such an appearance.  It was never called out, as such, but as each Kinect demo was presented on stage, WPF aficionados were beaming because they recognized their old friend behind the Kinect mask.

Why WPF?  Because WPF is the natural workhorse for NUI development.  From Kinect hacks to multi-touch displays to Surface technology, it is again and again the technology of choice providing both native integration with low level touch messages on the host OS as well as a powerful programming idiom.  For many of us, it was the technology we learned to do MVVM on.  We get to use the full resources of the .NET Framework with it as well as the ease of development provided by Visual Studio and Blend.

And if we are at the beginning of a NUI renaissance, as my friend Corey Schuman suggests in his MIX presentation, it follows that we are also in the midst of a WPF Renaissance as, for now, NUI and WPF are closely bound together.

To add sauce to the goose, over the weekend Rob Relyea from the WPF team tweeted the following elliptical comment:

“Yesterday wpf team had champagne/sparkling cider at our all hands meeting. Reached significant milestone.”

As pleased as I was with the Kinect demos at MIX, this piece of messaging took me completely by surprise.  There are still WPF milestones?  The Microsoft Tufaylîs are indeed gifted at entertaining and delighting.

“K” is for Hacking

James Ashley3 Comments

K

At the Atlanta .NET User Group presentation on hacking the Kinect last night, I promised to provide instructions on installing the necessary components for hacking the Kinect with the OpenNI stack.  I already feel some trepidation about doing this as figuring out these minimal first steps has always been a bit of an initiation rite for the various members of the OpenNI forums.  Strangely enough, there are few straight-forward walkthroughs on installing basic components.

The few that can be found on the web typically leave out key steps or provide the steps out of order — almost as if there is a general conspiracy to keep the details of these essential steps secret.

You are going to install three pieces: 1. SensorKinect which is the drivers that allow communication between the Kinect device and your PC 2. OpenNI which is an API that sits on top of the drivers and 3. NITE which is a higher level API that sits on top of OpenNI.  4. Additionally, you are going to copy some necessary config files onto your machine.

1. First, download the OpenNI component.  It can be found here.  You can choose to download from either the stable or the unstable branch.  The latter will provide more functionality while the former will be … well … more stable.

It may seem strange that you will be installing the higher level API before you install the drivers they sit on, but that’s the way it has to be done.  To do this in any other order risks opening a rift in space-time and ending the world as we know it.

Install OPENNI-Win32-1.0.0.25.exe (or whatever version happens to be current at the time of this reading).

2. Next you want SensorKinect.  You will need to create a github account to download it from here. 

Install SensorKinect-Win32-5.0.0.exe (or whatever version happens to be current at the time of this reading).

3. Download the NITE middleware which, like OpenNI, comes in both stable and unstable flavors. 

Install NITE-Bin-Win32-v1.3.0.18.exe (or whatever version happens to be current at the time of this reading).  Use the following key: 0KOIk2JeIBYClPWVnMoRKn5cdY4= when prompted.

4. Download and extract the KinectXMLs.zip file: http://box5921.temp.domains/~imagipi2/codesamples/kinectxmls.zip

Copy KinectXMLs\OpenNI\SamplesConfig.xml to C:\Program Files (x86)\OpenNI\Data\

Copy NITE\Sample-Scene.xml, NITE\Sample-Tracking.xml and NITE\Sample-User.xml to C:\Program Files (x86)\Prime Sense\NITE\Data\

These instructions were passed on to me by my colleague Alex Nichols who keeps them in a safe deposit box.  He, in turn, copied these instructions by candlelight from Steven Dawson, a shadowy and rarely seen uber-developer known for writing kinect hacks like this: http://www.engadget.com/2010/12/05/razorfish-ports-davinci-interface-to-kinect-makes-physics-cool/

You know you have installed everything correctly if a portal to another dimension does not open up above your Kinect.  Just to make sure, though, browse to C:\Program Files (x86)\OpenNI\Samples\Bin\Release (“Program Files” if you are on a 32-bit OS).  Plug your Kinect into a USB port and run NiViewer.exe.  If you can see yourself in the screen that pops up, then the OpenNI layer is working correctly.

Next, browse to C:\Program Files (x86)\Prime Sense\NITE\Samples\Bin\Release.  Try running Sample-PointViewer.exe.  You should quickly know if your NITE middleware is installed correctly.

In my experience, depending on which code branch you install, various samples may not work.  That’s okay.  As long as some of them work everything is fine.

Now you’ll want to try programming something.  Kinect hacker Vangos Pterneas provides several excellent examples you can peruse:

http://www.studentguru.gr/blogs/vangos/archive/2011/01/28/kinect-and-wpf-getting-the-raw-and-depth-image-using-openni.aspx

http://www.studentguru.gr/blogs/vangos/archive/2011/02/12/kinect-and-wpf-painting-with-kinect-using-openni.aspx

http://www.studentguru.gr/blogs/vangos/archive/2011/03/15/kinect-and-wpf-complete-body-tracking.aspx

Again, sometimes these work and sometimes they don’t.  If they aren’t working for you don’t worry too much about it.  Just move on.

Next, you’ll want to play with this Visual Studio 2010 C++ project on github: https://github.com/OpenNI/SampleAppSinbad  It is a 3D model of a piratical ogre that will track your movements.  You have to do some configuration if you want to run it from VS2010.  You can, however, just run the executable in the bin directory.  Reach behind your head with both hands to play with the swords.

I also showed several videos last night providing a visual history of gestural interfaces from laboratories, to the movies, to the mass-market, and finally to the hacking community.  If you are a fan of fantasy-UX, then all of these should already be familiar to you:

Star Trek Voyage Home (1986): http://www.youtube.com/watch?v=v9kTVZiJ3Uc&feature=related

Put That There (1979): http://www.youtube.com/watch?v=RyBEUyEtxQo

DreamSpace (1998): http://www.research.ibm.com/natural/dreamspace/#example

Minority Report (2002): http://www.youtube.com/watch?v=NwVBzx0LMNQ

Iron Man (2008): http://www.youtube.com/watch?v=PYjlMflhysc

Project Natal: http://www.youtube.com/watch?v=Mf44bWQr3jc

12 Best Kinect Hacks: http://www.youtube.com/watch?v=ho8KVOe_y08

Again, to everyone who came out for last night’s presentation — you were extremely patient and I can’t thank you enough for your time and your enthusiasm.

The Kinect’s Past

James Ashley2 Comments

shortcuts

— Outlook Shortcuts, 2005

“Humans communicate using speech, gesture, and body motion, yet today’s computers do not use this valuable information.  Instead, computers force users to sit at a typewriter keyboard, stare at a TV-like display, and learn an endless set of arcane commands, all of which often leads to frustration, inefficiencies and disuse.”

— Mark Lucente, Aug 1999

Mark Lucente wrote this for a panel organized in 1999 in Los Angeles that included Bill Buxton (then with Wavefront, Inc.) and others.  At the time, Mark was working on a project called Dreamspace, pictured below. 

dreamscape

Does Dreamspace look familiar?  It uses a combination of gestures and voice commands, just like this fantasy UX from 2002 you may have seen:

minority-report-02-800-75

Instead of inspiring us to find new ways to do astronomy or police work, of course, in 2010 these sparks of inspiration gave rise to this instead:

kinect1

Which is alright.  First we hook the kids, then we conquer the future.

I found Mark’s synopsis of the UX issues of 1999 in the archives of the MIT Media Lab Tangible Media Group.  I’m currently working my way through these papers and am overwhelmed by how smart the authors are as well as how far behind I am in trying to understand UX and what can be accomplished with a bit of vision.