Kinect v2 – The Imaginative Universal

One Kinect to rule them all: Kinect 2 for XBox One

April 5, 2015 James Ashley

Yes. That’s a bit of a confusing title, but it seems best to lay out the complexity upfront. So far there have been two generations of the Kinect sensor which combine a color camera, a depth sensing camera, an infrared emitter (basically used for the depth sensing camera) and a microphone array which works as a virtual directional shotgun microphone. Additional software called the Kinect SDK then allows you to write programs that read these data feeds as well as interpolating them into 3D animated bodies that are representations of people’s movements.

Microsoft has just announced that they will stop producing separate versions of the Kinect v2, one for windows and one for the XBox One, but will instead encourage developers to purchase the Kinect for Windows Adapter instead to plug their Kinects for XBox One into a PC. In fact, the adapter has been available since last year, but this just makes it official. All in all this is a good thing. With the promise that Universal Windows Apps will be portable to XBox, it makes much more sense if the sensors – and more importantly the firmware installed on them – are exactly the same whether you are on a PC running Windows 8/10 or an XBox running XBox OS.

This announcement also vastly simplifies the overall Kinect hardware story. Up to this point, there weren’t just two generations of Kinect hardware but also two versions of the current Kinect v2 hardware, one for the Xbox and one for Windows (for a total of four different devices). The Kinect hardware, both in 2010 and in 2013, has always been built first as a gaming device. In each case, it was then adapted to be used on Windows machines, in 2012 and 2014 respectively.

The now discontinued Kinect for Windows v2 differed from the Kinect for the Xbox One in both hardware and software. To work with Windows machines, the Kinect for Windows v2 device uses the specialized power adapter to pump additional power to the hardware (there is a splitter in the adapter that attaches the hardware to both a USB port as well as a wall plug). The Xbox One, being proprietary hardware, is able to pump enough juice to its Kinect sensor without needing special adapter. Additionally, the firmware for the original Kinect for Windows v1 sensor diverged over time from the Kinect for Xbox’s firmware – which led to differences in how the two versions of the hardware performed. It is now clear that this will not happen with Kinect v2.

Besides the four hardware devices and their respective firmware, the loose term “Kinect” can also refer to the software APIs used to incorporate Kinect functionality into a software program. Prior to this, there was a Kinect for Windows SDK 1.0 through 1.8 that was used to program against the original Kinect for Windows sensor. For the Kinect for XBox One with the Kinect for Windows Adapter, you will want to use the Kinect for Windows SDK 2.0 (“for Windows” is still part of the title for now, even though you will be using it with a Kinect for XBox One, though of course you can still use it with the Kinect for Windows v2 sensor if you happen to have bought one of those prior to their discontinuation). There are also other SDKs floating around such as OpenNI and Libfreenect.

[Much gratitude to Kinect MVP Bronwen Zande for helping me get the details correct.]

Projecting Augmented Reality Worlds

November 7, 2014 James Ashley

In my last post, I discussed the incredible work being done with augmented reality by Magic Leap. This week I want to talk about implementing augmented reality with projection rather than with glasses.

To be more accurate, varieties of AR experiences are often projection based. The technical differences depend on which surface is being projected on. Google glass projects on a surface centimeters from the eye. Magic Leap is reported to project directly on the retina (virtual retinal display technology).

AR experiences being developed at Microsoft Research, which I had the pleasure of visiting this past week during the MVP Summit, are projected onto pre-existing rooms without the need to rearrange the room itself. Using fairly common projection mapping techniques combined with very cool technology such as the Kinect and Kinect v2, the room is scanned and appropriate distortions are created to make projected objects look “correct” to the observer.

An important thing to bear in mind as you look through the AR examples below is that they are not built using esoteric research technology. These experiences are all built using consumer-grade projectors, Kinect sensors and Unity 3D. If you are focused and have a sufficiently strong desire to create magic, these experiences are within your reach.

The most recent work created by this group (led by Andy Wilson and Hrvoje Benko) is a special version of RoomAlive they created for Halloween called The Other Resident. Just to prove I was actually there, here are some pictures of the lab along with the Kinect MVPs amazed that we were being allowed to film everything given that most of the MVP Summit involves NDA content we are not allowed to repeat or comment on.

IllumiRoom is a precursor to the more recent RoomAlive project. The basic concept is to extend the visual experience on the gaming display or television with extended content that responds dynamically to what is seen onscreen. If you think it looks cool in the video, please know that it is even cooler in person. And if you like it and want it in your living room, then comment on this thread or on the youtube video itself to let them know it is definitely an M viable product for the XBox One, as the big catz say.

The RoomAlive experience is the crown jewel at the moment, however. RoomAlive uses multiple projectors and Kinect sensors to scan a room and then use it as a projection surface for interactive, procedural games: in other words, augmented reality.

A fascinating aspect of the RoomAlive experience is how it handles appearance preserving point-of-view dependent visualizations: the way objects need to be distorted in order to appear correct to the observer. In the Halloween experience at the top, you’ll notice that the animation of the old crone looks like it is positioned in front of the chair she is sitting on even the the projection surface is actually partially extended in front of the chair back and at the same time extended several feet behind the chair back for the shoulders and head. In the RoomAlive video just above you’ll see the view dependent visualization distortion occurring with the running soldier changing planes at about 2:32”.

You would think that these appearance preserving PDV techniques will fall apart anytime you have more than one person in the room. To address this problem, Hrvoje and Andy worked on another project that plays with perception and physical interactions to integrate two overlapping experiences in a Wizard Battle scenario called Mano-a-Mano or, more technically, Dyadic Projected Spatial Augmented Reality. The globe at visualization at 2:46” is particularly impressive.

My head is actually still spinning following these demos and I’m still in a bit of a fugue state. I’ve had the opportunity to see lots of cool 3D modeling, scanning, virtual experiences, and augmented reality experiences over the past several years and felt like I was on top of it, but what MSR is doing took me by surprise, especially when it was laid out sequentially as it was for us. A tenth of the work they have been doing over the past two years could easily be the seed of an idea for any number of tech startups.

In the middle of the demos, I leaned over to one of the other MVPs and whispered in his ear that I felt like Steve Jobs at Xerox PARC seeing the graphical user interface and mouse for the first time. He just stroked his beard and nodded. It was a magic moment.

Playing with Toasters

August 11, 2014 James Ashley

Every parent at some point faces the dilemma of what to tell her children. There’s a general and possibly mistaken notion that if you provide education about S-E-X in schools, you will encourage young ‘uns to turn words into deeds. Along the same lines, we can’t resist telling our children not to put forks in the toaster, even though we know that a child told not to do something will likely do it within five minutes. No more dangerous words were ever spoken than “don’t touch that!”

On a recent conference call I was on, someone asked if it would be dangerous to take an Xbox One Kinect and plug it into your computer. Although I waited more than five minutes, I eventually had to give into my impulse to find out.

I have several versions of the Kinect. I have both of the older model Kinect for Xbox 360 and Kinect for Windows v1. I also have a Kinect for Xbox One, Kinect for Windows v2 developer preview and Kinect for Windows v2 consumer (shown above).

The common opinion is that most of the differences between versions of the Kinect v2 are purely cosmetic. Kinect for Windows has a “Kinect” logo where the Kinect for Xbox One has a metallic “XBOX” logo. The preview K4Wv2 hardware is generally assumed to be a Kinect for Xbox One with razzmatazz stickers all over it. There is a chance, however, that the Kinect for Windows hardware lacks the IR blaster included with the Xbox One’s Kinect. The blaster is used to change channels on your TV from the Kinect, which “blasts” an IR signal over your room which the TV’s IR receiver picks up the reflection of.

	Kinect for Xbox One	K4Wv2 Preview	Kinect for Windows v2
SDK Color Sample	yes	yes	yes
SDK Audio Sample	yes	yes	yes
SDK Coord Map	yes	yes	yes
Xbox Fitness	yes	yes	no
Xbox Commands	yes	yes	no
Xbox IR Blaster	yes	yes	no

This was slightly scary, of course. I didn’t want to brick a $150 device. Then again, I reasoned it was being done for science – or at least for a blog post – so needs must.

I began by running the preview hardware against the latest SDK 2.0 preview. I plugged the preview hardware into the new power/usb adapter that comes with the final hardware. I then ran the color camera sample WPF project that comes with the SDK 2.0 preview. It took about 30 to 60 seconds for the Kinect to be recognized as the firmware was automatically updated. The sample then ran correctly. I did the same with the Audio sample and the Coordinate Mapper, both of which ran correctly.

Next, I tried the same thing with the Kinect for Xbox One. I plugged it into the Kinect for Windows v2 adapter and waited for it to be recognized. I was, of course, concerned that even if I succeeded in getting the device to run, I might hose the Kinect for use back on my Xbox. As things turned out, though, after a brief wait, the Kinect for Xbox One ran fine on a PC and with applications build on the SDK 2.0.

I think plugged my Kinect for Xbox One back into my Xbox One. The only application I have that responds to the player’s body is the fitness app. I fired that up and it recognized depth just fine. I also tried speech commands such as “Xbox Go Home” and “Xbox Watch TV”. I tested the IR blaster by shouting out “Xbox Watch PBS”. Apparently my Kinect for Xbox was not damaged.

I then performed the same actions using the Kinect for Windows preview hardware and, I think, confirmed the notion that it is simply a Kinect for Xbox. Everything I could do with the Xbox device could also be done using the Kinect for Windows preview hardware.

Finally I plugged in the Kinect for Windows final hardware and nothing happened. The IR emitters never lighted up. Either the hardware is just different enough or there is no Xbox compatible firmware installed on it.

There was no smoke and no one was harmed in the making of this blog post.