Developing your own software based on the nestk library

Nestk is the core library used by the demo programs within RGB Demo?. It is designed to be easy to integrate into existing cmake-based software and provide access the Kinect features quickly.

The library is built on top of OpenCV and QT for the graphical parts. Parts of it also depends on PCL. It includes or look for all the other libraries you need. In particular, libfreenect is included.

It can be used as a classical external library installed on the system, but the easiest way is to include it to use it is designed to be included as a subdirectory of your project.

Support

Please send your questions, patches, … to rgbdemo@googlegroups.com .

Tutorial to get started

Here is a simple tutorial to get started. You can get its full code on github. Suppose you want to create a new project called mysuperdemo. Then just put the nestk directory from github or extract it from a release of rgbdemo:

mkdir mysuperdemo
cd mysuperdemo
git clone https://github.com/rgbdemo/nestk.git

Note that if your project is using git, you might want to add it as a submodule.

Then just create a C Make Lists?.txt file as follows:

cmake_minimum_required(VERSION 2.6)

PROJECT(mysuperdemo)

SET(EXECUTABLE_OUTPUT_PATH ${CMAKE_BINARY_DIR}/bin)
SET(LIBRARY_OUTPUT_PATH ${CMAKE_BINARY_DIR}/lib)

# Default values
SET(NESTK_USE_FREENECT 1)
SET(NESTK_USE_OPENNI 1)

ADD_SUBDIRECTORY(nestk)
## nestk/UseNestk.cmake.in defines variable to use nestk.
## It itself includes nestk/deps/cmake/UseEmbeddedNestkDeps.cmake that
## contains most of the cmake find packages commands.
INCLUDE("${nestk_BINARY_DIR}/UseEmbeddedNestk.cmake")

ADD_EXECUTABLE(mysuperdemo mysuperdemo.cpp)
TARGET_LINK_LIBRARIES(mysuperdemo nestk)

and finally enter your program in mysuperdemo.cpp:

#include <ntk/camera/freenect_grabber.h>
#include <ntk/camera/rgbd_processor.h>
#include <ntk/utils/opencv_utils.h>

using namespace ntk;
using namespace cv;

int main()
{
  FreenectGrabber grabber;
  grabber.initialize();
  // Set camera tilt.
  grabber.setTiltAngle(15);
  grabber.start();

  // Postprocess raw kinect data.
  // Tell the processor to transform raw depth into meters using baseline-offset technique.
  FreenectRGBDProcessor processor;
  processor.setFilterFlag(RGBDProcessor::ComputeKinectDepthBaseline, true);

  // OpenCV windows.
  namedWindow("color");
  namedWindow("depth");
  namedWindow("depth_as_color");

  // Current image. An RGBDImage stores rgb and depth data.
  RGBDImage current_frame;
  while (true)
  {
    grabber.waitForNextFrame();
    grabber.copyImageTo(current_frame);
    processor.processImage(current_frame);

    // Display the color image
    imshow("color", current_frame.rgb());

    // Show the depth image as normalized gray scale
    imshow_normalized("depth", current_frame.depth());

    // Enable switching to InfraRead mode.
    unsigned char c = cv::waitKey(10) & 0xff;
    if (c == 'q')
      exit(0);
  }

  return 0;
}

To compile it (in release mode with debug information), do the following:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
make
cd bin
./mysuperdemo

Documentation

There is no documentation as such yet. The code is partially doxygenated though, and you can find a bunch of examples in nestk/samples and tests in nestk/tests that showcase most features. If you want to enable sample and tests building, you need to enable respectively the NESTK_BUILD_SAMPLES and NESTK_BUILD_TESTS cmake variables.

Here is a list of samples.

Using the library with an external install of nestk

If you don’t want to embed the library in your program, you can have a look at the following example: example_external

You will need to compile nestk separately and run make install then.

Read images using libfreenect backend

libfreenect_backend.cpp

Read images using OpenNI backend

openni_backend.cpp

Grab a snapshot mesh of the current scene

grab_mesh.cpp

Hand tracking

handpoint_tracking.cpp

Skeleton tracking

skeleton_tracking.cpp

Acquiring 3D models of the objects lying on a table

table_top_object.cpp

Grabbing from multiple Kinect

multiple_kinects.cpp

Grabbing synchronously from multiple Kinect

synchronized_multiple_kinects.cpp

Interfacing with PCL

pcl_compatibility.cpp

Using fast ICP from PCL

icp_pcl.cpp