Kinect Application Project Template

Over the past year, every time I start a new Kinect for Windows project, I’ve basically just copied the infrastructure code from a previous project.  The starting point was the code my friend Jarrett Webb and I wrote for our book Beginning Kinect Programming with the Microsoft Kinect SDK, but I’ve made incremental improvements to this code as needed and based on pointers I’ve found in various places.  I finally realized that I’d made enough changes and it was time to just turn this base code into a project template for myself and my colleagues at work.  Realizing that there wasn’t a Kinect Application project template available yet on the visual studio gallery, I uploaded it there, also.

The cool thing about templates uploaded to the gallery is that anyone with visual studio can now install it from the IDE.  If you select Tools | Extension Manager … and then search for “Kinect” under the Online Gallery, you should see something like this.  From here you can install the Kinect Application project template to your computer.

Kinect Application Project Template

If you then create a new project and look under C# | Windows, you will be able to build a Kinect WPF application with a bit of a headstart.  Here are some key features:

1. Initialization Code

All the initialization code and Kinect stream event handlers are stubbed out in the InitSensor method.  All you need to do is uncomment the streams you want to use.  Additionally, the event handler code is also stubbed out with the proper pattern for opening and disposing of frame objects.  Whatever you need to do with the image, depth and skeleton frames can be done inside those using statements.  This code also uses the latest agreed upon best practices for efficiently managing streamed data as of the 1.7 SDK.

void sensor_ColorFrameReady(object sender
    , ColorImageFrameReadyEventArgs e)
{
    using (ColorImageFrame frame = e.OpenColorImageFrame())
    {
        if (frame == null)
            return;

        if (_colorBits == null) _colorBits = 
            new byte[frame.PixelDataLength];
        frame.CopyPixelDataTo(_colorBits);

        throw new NotImplementedException();
    }
}

2. Disposal Code

Whatever you enable in the InitSensor method you will need to disable and dispose of in the DeInitSensor method.  Again, this just requires uncommenting the appropriate lines.  The DeInitSensor also implements a disposal pattern that is somewhat popular now.  The sensor is actually shut down on a background thread rather than on the main thread.  I’m not sure if this is a best practice as such, but it resolves a problem many C# developers were running into in shutting down their Kinect-enabled applications.

3. Status Changed Code

The Kinect can actually be disconnected in mid-process or simply not be on when you first run an application.  It is also surprisingly common to forget to plug the Kinect’s power supply in.  Generally, your application will just crash in such situations.  If you properly handle the KinectSensors.StatusChanged event, however, your application will just start up again when you get the sensor plugged back in.  A pattern for doing this was first introduced in the KinectChooser component in the Developer Toolkit.  A lightweight version of this pattern is included in the Kinect Application Project Template.

void KinectSensors_StatusChanged(object sender
    , StatusChangedEventArgs e)
{
    if (e.Status == KinectStatus.Disconnected)
    {
        if (_sensor != null)
        {
            DeInitSensor(_sensor);
        }
    }

    if (e.Status == KinectStatus.Connected)
    {
        _sensor = e.Sensor;
        InitSensor(_sensor);
    }
}
 

4. Extension Methods

While most people were working on controls for the Kinect 4 Windows SDK, Clint Rutkas and the Coding4Fun guys brilliantly came up with the idea of developing extension methods for handling the various Kinect streams.

The extension methods included with this template provide lots of conversions from bitmaps byte arrays to BitmapSource types (useful for WPF image controls) and vice-versa.  This allows you to do something easy like display a color stream which otherwise can be rather hairy.  The snippet below assumes there is an image control in the MainWindow named canvas.

using (ColorImageFrame frame = e.OpenColorImageFrame())
{
    if (frame == null)
        return;

    if (_colorBits == null) _colorBits = 
        new byte[frame.PixelDataLength];
    frame.CopyPixelDataTo(_colorBits);

    // new line
    this.canvas.Source = 
        _colorBits.ToBitmapSource(PixelFormats.Bgr32, 640, 480);
}

More in line with the original Coding4Fun Toolkit, the extension methods also make some very difficult scenarios trivial – for instance background subtraction (also known as green screening), skeleton drawing, player masking.  These methods should make it easier for quickly mock up a demo or even show off the power of the Kinect in the middle of a presentation using just a few lines of code.

private void InitSensor(KinectSensor sensor)
{
    if (sensor == null)
        return;

    sensor.ColorStream.Enable();
    sensor.DepthStream.Enable();
    sensor.SkeletonStream.Enable();
    sensor.Start();
    this.canvas.Source = sensor.RenderActivePlayer();
}

Again, this code assumes there is an image control in MainWindow named canvas.  You’ll want to put the following code in the InitSensor method to ensure that the code is called again if your Kinect sensor accidentally gets dislodged.  To create a simple background subtraction image, enable the color, depth and skeleton streams and then call the RenderActivePlayer extension method.  By stacking another image beneath the canvas image, I create an effect like this:

me on tatooine

Here are some overloads of the RenderActivePlayer method and the effects they create.  I’ve removed Tatooine from the background in the following samples.

 

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Blue);

blue_man

 

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Blue
                , System.Drawing.Color.Fuchsia);

blue_fuschia

 

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Transparent
                , System.Drawing.Color.Fuchsia);

trasnparent_fuschia

 

And so on.  There’s also this one:

canvas.Source = sensor.RenderPredatorView();

predator

 

… as well as this oldie but goodie:

canvas.Source = sensor.RenderPlayerSkeleton();
skeleton_view

The base method uses the colors (and quite honestly most of the code) from the Kinect Toolkit that goes with the SDK.  As with the RenderActivePlayer extension method, however, there are lots of overrides so you can change all the colors if you wish to.

canvas.Source = sensor.RenderPlayerSkeleton(System.Drawing.Color.Turquoise
    , System.Drawing.Color.Indigo
    , System.Drawing.Color.IndianRed
    , trackedBoneThickness: 1
    , jointThickness: 10);

balls

 

Finally, you can also layer all these different effects:

canvas.Source = sensor.RenderActivePlayer();
canvas2.Source = sensor.RenderPlayerSkeleton(System.Drawing.Color.Transparent);

everything

PRISM, Xbox One Kinect, Privacy and Semantics

loose lips

It’s interesting that at one time getting people to keep quiet was a priority for the government.  During World War II the government promoted a major advertising campaign to remind people that “loose lips sink ships.”  During war time (back when wars were temporary affairs), it was standard practice to suppress the flow of information and censor personal letters to ensure that useful information would not fall into enemy hands.  In a sense, privacy and national security were one.

Recent leaks about the NSA’s PRISM program suggest that things have dramatically changed.  We’ve realized for several years now that our cell phone service providers, our social networks, and our search engines are constantly tracking our physical and digital movements and mining that data for marketing.  We basically have traded our privacy for convenience in the same way that we accept ads on TV and on the Internet in exchange for free content. 

The dark side of all this is when all of this information is being passed along to third parties we didn’t even know about until we start getting junk mail in our inboxes for products we have no interest in.

What we only suspected, until now, was that the infrastructure that has been built to support these transactions of personal information for services were also of interest to our government and that we are sharing our identifying information not only with content providers, service providers, spammers and junk mailers but also with the United States security apparatus.  Now that all that information has been collected, the government wants to mine it also.

We don’t live in a police state today.  I don’t belong to either the far right wing nor the far left wing – I’m neither an occupier nor a tea partay kind of guy – so I also don’t believe we are even close to slipping into a police state in the near future.  I’m not concerned that the government will or ever will use this information to track me down and I am pretty confident that all this data mining will mainly be used only to track down terrorists and to send me unwanted emails.  And yet, it bugs me on a visceral level that people are going through my stuff, whatever that ethereal stuff actually is.

Gears of War

The main argument against this cooties feeling about my privacy is that only metadata is being inspected and not actual content.  Unfortunately, this seems like a porous boundary to me.  To paraphrase Hegel’s overarching criticism of Kant, whenever we draw a line we also necessarily have to cross over it at the same time.  From everything I know about software, the only way to gather metadata is to inspect the content in order to generate metadata about it.  For instance, when a government computer system listens to phone traffic in order to pick out key words and constellations of words, it still has to listen to all the other words first in order to pick out what it is interested in. 

Moreover, according to Slate, the data mining being done by PRISM is incredibly broad:

It appears the National Security Agency’s sweeping surveillance is not something only Verizon customers should be concerned about. The agency has also reportedly obtained access to the central servers of major U.S. Internet companies as part of a secret program that involves the monitoring of emails, file transfers, photos, videos, chats, and even live surveillance of search terms.

The semantics of Privacy today, as defined under the regime of the NSA, doesn’t mean no one is listening to what you are saying – it just means no one cares.  The best way to protect one’s privacy today is to simply be boring.

At the same time that all these revelations about PRISM were coming out (in fact on the very same day), Microsoft released a brief about privacy concerns around the new Xbox One’s Kinect peripheral.  Here’s an attempted explanation of the brief on Windows Phone Central I found particularly fascinating:

A lot of people feared that the Kinect would be able to listen to you when the Xbox One was off. Apparently, when off, the Xbox One is only listening for one command in its low-power state: “Xbox On”. It’s nice to know that you’re in control when the Kinect is on, off or paused. Some games though will require Kinect functionality (again, at the discretion of the game developers/publisher). That’s up to you to play or not play those games.

 

The author’s reassurance is based on a semantic sleight-of-hand.  The Kinect is not listening to you, according to the author, because it “is only listening for one command.”  This is an honest mistake, but a dangerous one.  In fact, in order to listen for one command, the Kinect has to have that microphone turned on and listening to everything anyone is saying.  What it is actually doing is only acting on one command – and hopefully throwing away everything else.  Additionally I do have a bit of experience with Microsoft’s speech recognition technology both on the Kinect and on the PC, and the “low-power state” modifier doesn’t particularly make sense.  It takes a similar amount of effort to identify insignificant data as it does to identify significant data, AFAIK. (There’s always the possibility that the Xbox Kinect has an on-board language processor just to listen for this one command that is separate from the rest of its speech recognition processing chain – but I haven’t heard about anything like that so far.)

Halo IV

The original Microsoft brief called Privacy by Design, upon which I assume the Windows Phone Central post is based, doesn’t play this particular semantic game – though it plays another.  At the same time, it also seems particularly and intentionally vague about certain points.

The semantic game in Microsoft’s Privacy post is around the term ‘design’.  Does design  here refer to the hardware design, the software architecture, the usability design or the marketing campaign?  These are all things that are encompassed by the term design and, in the linked article, privacy could be referring to any of them.  If it refers to the marketing campaign and UX, as it probably does, this doesn’t actually provide me any guarantees of privacy.  All it tells me is that Microsoft doesn’t initially intend to use the new Kinect sitting in my living room to collect random conversations.  ‘Design’ may refer to the initial software architecture, but this doesn’t provide us with any particular guarantees since any post-release software update can change the way the software works.

To put this another, way, the article describes Microsoft’s intent but doesn’t provide any guarantees.  Is there anything in the hardware that will prevent speech data from being mined in the future?  Probably not.  In that case, is there anything in the licensing that prevents Microsoft from mining this data?  Microsoft’s privacy brief doesn’t even touch on this.

So should you be concerned?  Totally – and here’s why.  In its pursuit of security, the NSA has instituted an infrastructure that performs better and better the more information it is fed.  Do terrorists play Xbox?  I have no idea.  Would the NSA want all that data anyways? 

Call of Duty

Hypothetically, the new Xbox One and the Kinect can collect this information on us.  Here’s how.  According to recent Microsoft announcements, the Xbox One must be connected to the Internet once every 24 hours in order to play games on it.  The new Kinect is designed to always be on and I am obligated to have it (I can’t buy a Kinect One without it).  Even when my Xbox One is off, my Kinect is still on listening for a command to turn it on.  The infrastructure is there and the NSA’s PRISM project is a monster that is hungry for it.

To be clear, I don’t think Microsoft is particularly interested in collecting this data.  Microsoft has no particular use for the typically rather boring conversations I have in my living room.  They won’t be gleaning any particularly useful marketing information from my conversations either. 

Nevertheless, I think it would be extremely forward looking of Microsoft to explain what they have put in place to prevent the government from ever issuing a request for this data and getting it the way they have already gotten other data, so far, from Verizon, AT&T, Microsoft, Yahoo, Google, Facebook, AOL, Skype, YouTube, and Apple.

Has Microsoft designed a mechanism, either through hardware or through a customer agreement they won’t/can’t rescind in the future, that will future proof my privacy?

What Game of Thrones Can Teach Us About Terrorism

distrubance 

"I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror …”

Last night’s airing of Game of Thrones season 3 episode 9, The Rains of Castamere, was in many ways the culmination of the “A Song of Ice and Fire” experience.  In the books by G.R.R. Martin, the Red Wedding occurs half way through the third book (there are currently five).  The RW is the primary reason people get their friends to read the book.  According to the producers of the HBO series, it is the episode they felt they had to get to.

In going through the social media related to the Red Wedding, there seemed to be mainly two reactions.  One was the sense of shock, grief and eventually numbness from people who didn’t know it was coming. I well recognize this mental state from the time I read the RW scene almost ten years ago.  The second was the strange elation of people who had already read the books in response to the reaction of the people who hadn’t.

black_frey

I wish I could find a word for this second, reflective emotion.  It isn’t exactly schadenfreude, that amazing German word for the the pleasure we take in other people’s misfortune.  Schadenfreude always has an element of ressentiment in it and seems generally directed to people who are better off than us.  The object of our schadenfreude thinks he is an innocent while in our minds, the misfortune is in some way deserved — though perhaps excessive.  Schadenfreude is the emotion Walder Frey feels as he watches the Starks and their bannermen being cut down.

In my bedroom wall, there is a hole made by a very heavy paperback tome. It marks the place where my wife threw her copy of A Storm of Swords against the wall after the Red Wedding scene – and for those more in the know, specifically the scene involving Arya and the Hound’s axe. I hadn’t read it yet and it was at that point my wife made me start with the first book, A Game of Thrones, so I could catch up and find out why there was a hole in our bedroom wall.

walder

There was a serious angst (‘nother awesome German word but still not the one we want) to her mood and it wouldn’t go away until I’d gotten to the emotional place she wanted me. I wanted to throw the book at the wall, too, but it seemed pointless by then.  The important thing though was she would finally talk to me again and we were on the same page, so to speak.  Oddly enough, we talked about what a great movie these books would make. 

The reflective emotion online was partly a weird glee but also a solicitousness towards those who were experiencing the RW psychic shock for the first time.  It’s as if for those who had already gone through this trauma, the trauma itself presented a barrier between themselves and everyone who was going about their lives in ignorance of the fact that a horrible thing happens in the middle of the third book of this series of books they probably are never going to read because adults don’t read Proust-length fantasy novels.  And then, thanks to the HBO series, now that trauma has been shared with the rest of the world.

bolton

I think the emotional word I’m looking for might be terrorism.  Isn’t this what terrorists do to people who don’t understand or sympathize with their plight?  They find a way to share their trauma with others in order to externalize their angst?

With terrorism, though, we never get to the point where people say, hey, thanks for the bombing, now I see where you’re coming from and everything’s going to be okay.

readthebook

Following the airing of The Rains of Castamere, on the other hand, all of us are now on the same page emotionally, are ready for healing, and can move on to the next thing, whether that next thing is the new season of True Blood or possibly a new Gene Wolfe novel.  On the other hand, if you are just interested in connecting with more people who have gone through what you just went through, you can try the online Song of Ice and Fire community at http://asoiaf.westeros.org/

It can be thought of as the largest and longest lasting group therapy session ever created. While I haven’t been back for a while, my wife and I joined it shortly after we created that hole in our bedroom wall and it was the source of much comfort and consolation to us.  It was the place, strangely enough, where some of the casting for the HBO series occurred as well as the best place to learn how to decipher one of the great hidden secret of the series: R+L=J.