3d code graphics opencv programming video vision

Neat OpenCV smoothing trick when Kineacking (Kinect Hacking) [w/ code]

I found a nice little trick to ease the work with the very noisy depth image the Kinect is giving out. The image is filled with these “blank” values that basically note where the data is unreadable. The secret is to use inpainting to cover these areas and get a cleaner image. And as always, no need to dig deep – OpenCV has it all included.

Start from a simple Kinect frames feed from here:

int main(int argc, char **argv) {
	bool die(false);
	Mat depthMat(Size(640,480),CV_16UC1);
	Mat depthf  (Size(640,480),CV_8UC1);
	Mat rgbMat(Size(640,480),CV_8UC3,Scalar(0));
	Mat ownMat(Size(640,480),CV_8UC3,Scalar(0));
        Freenect::Freenect freenect;
        MyFreenectDevice& device = freenect.createDevice<MyFreenectDevice>(0);
    while (!die) {
    	depthMat.convertTo(depthf, CV_8UC1, 255.0/2048.0);
		char k = cvWaitKey(5);
		if( k == 27 ){
	return 0;

Now let’s stretch the signal a little bit and add the inpainting:

		//interpolation & inpainting
			Mat _tmp,_tmp1; //minimum observed value is ~440. so shift a bit
			Mat(depthMat - 400.0).convertTo(_tmp1,CV_64FC1);
			Point minLoc; double minval,maxval;
			minMaxLoc(_tmp1, &minval, &maxval, NULL, NULL);
			_tmp1.convertTo(depthf, CV_8UC1, 255.0/maxval);  //linear interpolation
                       //use a smaller version of the image
			Mat small_depthf; resize(depthf,small_depthf,Size(),0.2,0.2);
                        //inpaint only the "unknown" pixels
			cv::inpaint(small_depthf,(small_depthf == 255),_tmp1,5.0,INPAINT_TELEA);
			resize(_tmp1, _tmp, depthf.size());
			_tmp.copyTo(depthf, (depthf == 255));  //add the original signal back over the inpaint

Note that I’m using a small copy of the image, because inpainting is a heavy computation, and it works best on low frequencies. I copy back the original signal over the up-sized inpainted one to retain high frequencies.
It works pretty well!


5 replies on “Neat OpenCV smoothing trick when Kineacking (Kinect Hacking) [w/ code]”

Hi Rov
Thanks for posting this useful information. I am facing a similar problem, and I used your approach but it didn’t do so well as I expected it to be. If you please explain the algorithm in detail, it will be highly appreciated.
Hope to hear from you soon.
Thank you

hi Roy,
you example is great. Do you have any idea to make the same thing by flash action script3 ?
please feel free to contact me if you interest about this job.
Thankyou !

This is amazing, works like a charm, slows down processing a tad but has so much potential, thanks!

Thanks for the algorithm, works quite well indeed. My results:
Here’s a cleaned up version:
Mat depthMat(height, width, CV_16UC1, depth); // from kinect
Mat depthf(height, width, CV_8UC1);
depthMat.convertTo(depthf, CV_8UC1, 255.0/2048.0);
imshow(“original-depth”, depthf);
const unsigned char noDepth = 0; // change to 255, if values no depth uses max value
Mat temp, temp2;
// 1 step – downsize for performance, use a smaller version of depth image
Mat small_depthf;
resize(depthf, small_depthf, Size(), 0.2, 0.2);
// 2 step – inpaint only the masked “unknown” pixels
cv::inpaint(small_depthf, (small_depthf == noDepth), temp, 5.0, INPAINT_TELEA);
// 3 step – upscale to original size and replace inpainted regions in original depth image
resize(temp, temp2, depthf.size());
temp2.copyTo(depthf, (depthf == noDepth)); // add to the original signal
imshow(“depth-inpaint”, depthf); // show results

Leave a Reply

Your email address will not be published. Required fields are marked *