OpenNI Application "Bouncing Ball"

Introduction

I implemented an application which

1. generates a point cloud from a depth data captured by the Xtion Pro Live,
2. calculates a normal vector for each point in the point cloud using the Point Cloud Library(PCL),
3. introduces virtual balls into an environment on the viewer,
4. and simulates collisions between the balls and the objects in the environment.

The application does not retrieve normal vectors from each frame, but from a certain frame just one time. The simulation is executed in real time under those normal vectors. I used the OpenGL library to draw frames.

Development Environment

1. Mac OS X 10.7.4
2. Processor：3.06 GHz Intel Core 2 Duo
3. Memory：4GB
4. Xcode4.3.3

Primary Libraries

1. Boost-1.50.0
2. PCL-1.6.0
3. OpenNI-Bin-Dev-MacOSX-v1.5.4.0
4. Sensor-Bin-MacOSX-v5.1.2.1

Source Codes

Here is my source codes. I confirmed that the application works well on my machine.

Compile Options

Header Search Path:

Path	Purpose
/usr/include/ni/	to include the OpenNI headers
/opt/local/include/	to include the Boost headers
/usr/local/include/pcl-1.6/	to include the PCL headers
/opt/local/include/eigen3/	to include the Eigen headers on which the PCL depends
/opt/local/include/vtk-5.10/	to include the VTK headers on which the PCL depends

Library Search Path:

Path	Purpose
/opt/local/lib	to link with the Boost libraries
/usr/local/lib	to link with the PCL libraries

In addition, to use the OpenGL, I set "-framework OpenGL." More detailed information is in my source codes.

Usage

The executable file name is OpenNiViewer. Running it without any arguments, it prints the following statements. Running it with setting the path to a file "SamplesConfig.xml" to --config, it opens a viewer for a video captured by the Xtion Pro Live. The file "SamplesConfig.xml" is enclosed in my source codes. I don't verify the behavior of the application with values other than default ones for --width and --height. Typing "p" on the keyboard under the situation where the application focuses on the viewer, the derivation of normal vectors begins. After seeing "Normals has been extracted" on the standard output, typing "i" yields virtual balls in the captured scenes, and the interactions between the balls and the objects in the environment are drawn in real time. To quit the application, type the escape key.

Brief Overview of Source Codes

View Point of OpenGL

The view point of the OpenGL is as follows: A unit of length in the application is mm. You need to change the argument in the 8th line according to your environment.

Transforming Depth into Point Cloud

A depth data is transformed into a point cloud in the method OpenNiDevice::create_cloud. I generate two point clouds, one (cloud_) is for displaying the color video and another (cloud_xyz_) is for calculating normal vectors. RGB values are set in the 26th, 27th, and 28th lines. The depth data is transformed into the point cloud in the 33th line. The point cloud for calculating normal vectors is created in the 34th line.

Drawing Color Image

The point cloud used for the color video, cloud_, is passed to the OpenGL method glInterleavedArrays as

Derivation of Normal Vectors

The cloud used for deriving normal vectors (cloud_xyz_) is processed in the method OpenNiDevice::calculate_normals. The method initialize in the 6th line is as follows: The region of the local nearest search is set in the 9th line. This argument varies depending on your environment. The plane is calculated using points inside the region. The normal vector of the plane is what we want. The normal vectors is derived with each typing "p" on the keyboard. If you change position of the Xtion Pro Live, you have to type "p" again.

Configuration of Balls

The virtual balls is registered in the method OpenNiViewer::register_balls. The radius of the ball is set in the 25th line. The initial position, the initial velocity, and the acceleration are set in the 31th, 32th, and 33th lines, respectively. These values also vary depending on your environment.

Configuration of Walls

A wall is placed at the front of the viewer to avoid the balls disappearing from view.

Search of Normal Vectors

For each frame, the normal vectors located adjacent to the balls are detected by means of the kd-tree. kdtree_ in the 8th line is the object of the class pcl::KdTreeFLANN<pcl::PointXYZ>. Because the object returns nan if it fails to detect neighbors, the returned value is checked in the 12th line.