Neat OpenCV smoothing trick when Kineacking (Kinect Hacking) [w/ code]

I found a nice little trick to ease the work with the very noisy depth image the Kinect is giving out. The image is filled with these "blank" values that basically note where the data is unreadable. The secret is to use inpainting to cover these areas and get a cleaner image. And as always, no need to dig deep - OpenCV has it all included.

Start from a simple Kinect frames feed from here:

int main(int argc, char **argv) {
	bool die(false);

	Mat depthMat(Size(640,480),CV_16UC1);
	Mat depthf  (Size(640,480),CV_8UC1);
	Mat rgbMat(Size(640,480),CV_8UC3,Scalar(0));
	Mat ownMat(Size(640,480),CV_8UC3,Scalar(0));

        Freenect::Freenect freenect;
        MyFreenectDevice& device = freenect.createDevice<MyFreenectDevice>(0);


    while (!die) {
    	depthMat.convertTo(depthf, CV_8UC1, 255.0/2048.0);
		char k = cvWaitKey(5);
		if( k == 27 ){

	return 0;

Now let's stretch the signal a little bit and add the inpainting:

		//interpolation & inpainting
			Mat _tmp,_tmp1; //minimum observed value is ~440. so shift a bit
			Mat(depthMat - 400.0).convertTo(_tmp1,CV_64FC1);
			Point minLoc; double minval,maxval;
			minMaxLoc(_tmp1, &minval, &maxval, NULL, NULL);
			_tmp1.convertTo(depthf, CV_8UC1, 255.0/maxval);  //linear interpolation
                       //use a smaller version of the image
			Mat small_depthf; resize(depthf,small_depthf,Size(),0.2,0.2);
                        //inpaint only the "unknown" pixels
			cv::inpaint(small_depthf,(small_depthf == 255),_tmp1,5.0,INPAINT_TELEA);
			resize(_tmp1, _tmp, depthf.size());
			_tmp.copyTo(depthf, (depthf == 255));  //add the original signal back over the inpaint

Note that I'm using a small copy of the image, because inpainting is a heavy computation, and it works best on low frequencies. I copy back the original signal over the up-sized inpainted one to retain high frequencies.

It works pretty well!


  • Roopa

    Hi Rov

    Thanks for posting this useful information. I am facing a similar problem, and I used your approach but it didn't do so well as I expected it to be. If you please explain the algorithm in detail, it will be highly appreciated.

    Hope to hear from you soon.

    Thank you

  • louis

    hi Roy,

    you example is great. Do you have any idea to make the same thing by flash action script3 ?
    please feel free to contact me if you interest about this job.

    Thankyou !

  • Tamar

    This is amazing, works like a charm, slows down processing a tad but has so much potential, thanks!

  • vaibhav

    can you please explain the inpainting algo and tell how its implemented on kinect?

  • Thanks for the algorithm, works quite well indeed. My results:

    Here's a cleaned up version:

    Mat depthMat(height, width, CV_16UC1, depth); // from kinect
    Mat depthf(height, width, CV_8UC1);

    depthMat.convertTo(depthf, CV_8UC1, 255.0/2048.0);
    imshow("original-depth", depthf);

    const unsigned char noDepth = 0; // change to 255, if values no depth uses max value
    Mat temp, temp2;

    // 1 step - downsize for performance, use a smaller version of depth image
    Mat small_depthf;
    resize(depthf, small_depthf, Size(), 0.2, 0.2);

    // 2 step - inpaint only the masked "unknown" pixels
    cv::inpaint(small_depthf, (small_depthf == noDepth), temp, 5.0, INPAINT_TELEA);

    // 3 step - upscale to original size and replace inpainted regions in original depth image
    resize(temp, temp2, depthf.size());
    temp2.copyTo(depthf, (depthf == noDepth)); // add to the original signal

    imshow("depth-inpaint", depthf); // show results