To build 3D movies with Kinect, you first have to import all of your depth data into a point cloud. A point cloud is basically what it sounds like: a cloud of points in 3D space. Because the Kinect v2 has roughly 3 times the depth data provided by the Kinect v1, the cloud density is much richer using it.
The next step in building up a 3D movie is to color in the pixels of the point cloud. Kinect v1 used an SD camera for color images. For many people, this resolution was too low, so they came up with various ways to sync the data from an DSLR camera with the depth data. This required precision alignment to make sure the color images lined up with and then scaled to the depth pixels. This alignment also tended to be done in post-production rather than in realtime. One of the most impressive tools created for this purpose is called the RGBD Toolkit, which was used to make the movie Clouds by James George and Jonathan Minard. The images in this post, however, come from an application I wrote over Memorial Day weekend.
Unlike its predecessor, Kinect for Windows v2 is equipped with an HD video camera. The Kinect for Windows v2 SDK also has facilities to map this color data to the depth positions in realtime, allowing me to record in 3D and view that recording at the same time. I can even rotate and scale the 3D video live.
You’ll also notice some distortion in these images. I actually ran this 3D video capture on a standard laptop computer. One of the nicest features of the Kinect v2 is that it takes advantage of the GPU for calculations. If I don’t like the quality of the images I’m getting, I can always switch to a more powerful machine.
The next step, of course, is to use multiple Kinects to record 3D video. While I can rotate the current images, there are shadows and distortions which become more evident when the image is rotated to orientations not covered by a single camera. Two cameras, on the other hand, might allow me to do a live “bullet time” effect.
I don’t really know what this would be used for – for now it’s just a toy I’m fiddling with–, but I think it would at least be an interesting way to tape my daughter’s next high school musical. On the farther end of the spectrum, it might be an amazing way to do a video chat or to take the corporate video presentation to the next level.