I’m currently experimenting a bit with the Kinect depth camera, which I intend to use for a prototype. I’ve played around with the device before on Windows using the Microsoft Kinect SDK. Setting up libfreenect on Windows is actually quite a pain, which was the reason why I started with the Microsoft SDK. As I wanted to code in Python, I’ve had a look at PyKinect and the Python Tools for Visual Studio (see also this PyCon US 2012 talk about building a Kinect game with Python). Unfortunately, the PyKinect samples seem to be outdated, and don’t work with the latest version of the Kinect SDK (version 1.0, released in February).
So I decided to see how far I would get with the Kinect and libfreenect on Ubuntu (12.04). Step 1: install the libfreenect demos to test if it works (this will also install libfreenect and its dependencies).
$ sudo apt-get install libfreenect-demos
So far so good. Unfortunately, the demo application wouldn’t start:
$ freenect-glview Kinect camera test Number of devices found: 1 Could not claim interface on camera: -6 Could not open device
Strange.. I did another check to see if my user was in the plugdev group (which
was the case):
$ groups jo jo : jo adm cdrom sudo dip plugdev lpadmin sambashare
Then I noticed that the Kinect’s LED kept blinking continously, which usually means there’s some sort of connection problem. It was correctly recognized, though:
$ lsusb ... Bus 002 Device 007: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor Bus 002 Device 008: ID 045e:02ad Microsoft Corp. Xbox NUI Audio Bus 002 Device 009: ID 045e:02ae Microsoft Corp. Xbox NUI Camera
After a quick web search for the specific libfreenect-glview error message, I learned that recent versions of the Linux kernel prevent libfreenect from claiming the Kinect as they now include a driver to support the device as a regular webcam (see kinect.c), which is actually quite cool. This also means there should be a specific video device for the Kinect (/dev/video1 in my case).
The kernel modules are indeed loaded:
$ lsmod | grep -i gspca gspca_kinect 12936 0 gspca_main 28366 1 gspca_kinect videodev 98259 2 gspca_main,uvcvideo
Let’s try playing the camera stream using GStreamer and Video4Linux:
$ gst-launch-0.10 v4l2src device=/dev/video1 ! video/x-raw-yuv ! ffmpegcolorspace ! xvimagesink
That seems to work!
The Kinect actually has two image sensors: an RGB camera and a depth sensor, which consists of an infrared laser projector and a monochrome CMOS sensor. A depth map is created by projecting structured infrared light and capturing the resulting image from the monochrome sensor. As I wasn’t sure how to grab an image from the monochrome sensor, I contacted the author of the kernel module (Antonio Ospite). He told me it’s possible to get a monochrome image by specifying an image size of of 640×488 pixels (instead of the usual 640×480). Note that the kernel module currently only supports video streams from the monochrome sensor as unprocessed output.
If we pass that specific width and height to GStreamer, we get this:
It’s also possible to get the full-size (1280×1024) monochrome image. However, most webcam apps will just show the video stream from the RGB sensor for that resolution as you can’t specify that you want the specific Y10B format. To do that, you can use a separate program like qv4l2, which can be installed as follows:
$ sudo apt-get install qv4l2
To get the full-size monochrome image, run qv4l2, open the /dev/video1 device in raw mode (File > Open raw device), select 1280×1024 for the frame size, and “Y10B – Y10B” for the capture format, and click the red capture button:
OK, back to running the libfreenect demo. Someone over at superuser.com suggested to remove the modules (so that the user-mode libfreenect driver could
take over) and run libfreenect-glview again.
And indeed, after removing both gspca modules, the freenect-glview demo did work:
$ sudo modprobe -r gspca_kinect $ sudo modprobe -r gspca_main $ freenect-glview Kinect camera test Number of devices found: 1 GL thread Write Reg 0x0006 <= 0x00 Write Reg 0x0012 <= 0x03 Write Reg 0x0013 <= 0x01 Write Reg 0x0014 <= 0x1e Write Reg 0x0006 <= 0x02 Write Reg 0x0005 <= 0x00 Write Reg 0x000c <= 0x00 Write Reg 0x000d <= 0x01 Write Reg 0x000e <= 0x1e Write Reg 0x0005 <= 0x01 ...
The left side of the window shows a colored depth image, while the right side shows the RGB image from the camera. Of course, it would be quite cumbersome to remove the modules again every time you want to use libfreenect. One option is to blacklist the gspca module as follows:
$ echo "blacklist gspca_kinect" >> /etc/modprobe.d/blacklist.conf
Now that we've got that sorted out, I'll try out the libfreenect Python wrapper (or perhaps SimpleCV).
Update: Antonio Ospite pointed out in the comments that recent versions of libfreenect (0.1.2) can automatically detach the kernel driver. There's a PPA available by Florian Echtler with updated libfreenect packages for Ubuntu 12.04.
