Let’s start!

Some mathematical background

Don’t worry, it wont hurt. much.

So Spherical Harmonics, were invented to numerically express a whole bunch of things in physics like gravity and magnetic fields. But they also became very useful for computer graphics as they are perfect for modelling light falling on a spherical body.

But what ARE those mysterious spherical harmonics?

The way I see it, they are a series of “modes” or “eigenvectors” or “orthogonal components” of a base that spans the surface of a sphere.
To put it simple, they describe the surface of a sphere in increasing finer grained portions. Much like a Fourier decomposition does to a function, there is the base and there are coefficients that when multiplied with the base they recover the function.

How is that good for graphics?

People have used spherical harmonics mostly to model lighting of spherical objects. When you know the coefficients that describe the lighting, you can change them to Re-light an object, or De-light, or transfer the lighting conditions of one scene to another. Very useful!

Some good researchers, Basri and Jacobs, back in 2001 have formulated the first 9 harmonics as a function of the surface normal. On this page Basri references all his work on the subject: http://www.wisdom.weizmann.ac.il/~ronen/index_files/harmonic.html

But I like to reference a work that’s easier to process than Basri’s, that is the work of Wang et al from 2007. These guys made the steps to use spherical harmonics easier to follow: http://research.microsoft.com/en-us/um/people/zliu/cvpr2007.pdf.
But their algorithm is quite advanced, as it solves not only for the harmonics’ coefficients but also for the normals of the object in the image. They use some fancy optimization of an energy function over a graph, that I’m not going to discuss.
But they did make the process of finding the spherical harmonics’ coefficient very clear.

The bottom line

We should solve for a vector of 9 coefficients that describes the “lighting of the object” (a face in our case).
Each coefficient will tell us how much that specific harmonic is strong or weak, or in other words how lit is that certain area of the object.

Wang and Basri show a very simple method of using simultaneous linear equations to solve for the lighting coefficients, it depends only on knowing the normal of the object’s surface at each pixel in the image.

Getting the normals of a canonical face

So to get the normals, I thought the best way is to use a canonical model of a face (some king of an average face), instead of trying to recover the normals from the image pixels.
For that end, I used Rhino3D to model (very roughly) a shape that resembles a human face, starting from an elongated sphere.
Now all that’s left is to align the model with the face to relight, and that will supply the normals.

Cool. Then I built a small app that allows the user to move the model around until it’s aligned with the face image. I used FLTK 3.0 to do it since they have a simple interface with OpenGL, they are cross platform, and lightweight.
So I set up a scene where I have the image as the background, and the model is floating above it, half transparent so the user can find the right spot. I added functions for rotating the model, and extra stuff like turning the model opaque.

To get the normal map I used a very simple GLSL shader, that simply colors the pixel with the value of the normal nX,nY,nZ -> R,G,B.
This way the result image OpenGL renders is simply the normal map of the face model. I just grab it using glReadPixels.

Estimating spherical harmonics

So, after the model is aligned, we can assume we have the normals ready for us for each pixel in the image, and the intensity in each pixel is also known.
The first step that Wang suggests, without knowledge of the real face albedo (the real color of every pixel without any lighting effect), is to get an approximation of the 9-vector of lighting coefficients by setting a constant albedo. Easy enough, we can set the albedo to the average color in the face.
Then we can simply build a huge set of linear equations (huge as the number of pixels in the image), and solve an overdetermined system to get the 9 coefficients.

		Scalar albedo_constant = mean(face_img_hsv, smallFaceMask);

		//setup linear equation system, lighting coefficients (l) is unknown
		//I = p00 * Ht * l
		float p00 = (float)albedo_constant[2] / 255.0f;

		cout << "Build Ht("<<n<<",9)...";
		cout << "Build I("<<n<<",1)...";
		//build Ht and I
		Mat_<float> Ht(n,9);
		Mat_<float> I(n,1);
		int pos = 0;
		vector<Mat_<uchar> > face_img_chnls; split(face_img_hsv, face_img_chnls);
		for (int i=0; i<normalMapFlat.rows; i++) {
			if (smallFaceMask(i) == 0) { //is this pixel on the face?
				continue;
			}
			Ht.row(pos) = p00 * calculateSphericalHarmonicsForNormal(normalMapFlat(i));
			I(pos,0) = face_img_chnls[2](i) / 255.0f; //get V from HSV of pixel [0,1]
			pos ++;
		}
		cout << "DONE"  << endl;

		cout << "Solve" <<endl;
		solve(Ht, I, l, DECOMP_SVD);

		cout << "initial lighting coeffs: ";
		for (int i=0; i<l.rows; i++) {
			cout<<l.at<float>(i)<<",";
		}

Booyah! lighting coefficients.

But this is only the first step. Now we can get an approximation of the albedo as well, using the coefficients:

		Mat_<Vec3b> face_img_v3b = face_img;

		#pragma omp parallel for schedule(dynamic)
		for (int y=0; y<face_img.rows; y++) {
			for (int x=0; x<face_img.cols; x++) {
				if (face_mask(y,x) == 0) {
					albedo(y,x) = 0;
					continue;
				}
				Mat sph = calculateSphericalHarmonicsForNormal(normalMap(y,x));
				Mat_<float> sph_l = sph * l;
				float fsph_l = sph_l(0);

				for (int cn = 0; cn<3; cn++) {
					float fimg = face_img_v3b(y,x)[cn] / 255.0f;
					albedo(y,x)[cn] = (fimg / fsph_l);
				}
			}
		}

Done.
Now that we have an initial albedo, Wang suggests we compute the coefficients again to get a better approximation, and then the albedo again.
I however ran into some problems trying to do the second iteration, and the results always came out too dark… But even with the first iteration you can see a very nice change.
Look at the video from before, you can see the right side of the face, which is over-lit, was darkened and the left side was lit up.

Code

The code for spherical harmonics analysis of images is part of a bigger project I have been working on for some time. I also spoke of it in a previous post.
Anyway it’s up in GitHub: https://github.com/royshil/HeadReplacement/tree/master/HeadReplacement
You’re looking for 4 files:

• SpharmonicsUI.cpp
• SpharmonicsUI.h
• spherical_harmonics_analysis.cpp
• spherical_harmonics_analysis.h

You can use the CMakeLists.txt to compile, but here’s a CMakeLists.txt that should take you there in one piece (fingers crossed):

find_package(OpenCV REQUIRED)
find_package(OpenGL REQUIRED)
find_package(OpenMP REQUIRED)

######## Find and add GLEE ########
file(GLOB_RECURSE GLEE_PATH "${CMAKE_SOURCE_DIR}/GLee.c")
if(GLEE_PATH STREQUAL GLEE_PATH-NOTFOUND)
	message(STATUS "GLEE was not found")
else()
	list(LENGTH GLEE_PATH GLEE_PATH_LEN)
	if(GLEE_PATH_LEN GREATER 1)
		list(GET GLEE_PATH 1 GLEE_PATH)
	endif()
	file(RELATIVE_PATH GLEE_PATH ${CMAKE_SOURCE_DIR} ${GLEE_PATH})
	get_filename_component(GLEE_PATH ${GLEE_PATH} REALPATH)
	get_filename_component(GLEE_PATH ${GLEE_PATH} PATH)
	message(STATUS "Found GLEE at ${GLEE_PATH}")
	add_library(GLEE ${GLEE_PATH}/GLee.c)
endif()

############ Find FLTK ############
if(NOT DEFINED FLTK_PATH)
	file(GLOB_RECURSE FLTK_PATH "${CMAKE_SOURCE_DIR}/Widget.h")
	if(FLTK_PATH STREQUAL FLTK_PATH-NOTFOUND   OR   FLTK_PATH STREQUAL "")
		message(STATUS "FLTK was not found !!!!!")
	else()
		list(LENGTH FLTK_PATH FLTK_PATH_LEN)
		if(FLTK_PATH_LEN GREATER 1)
			list(GET FLTK_PATH 1 FLTK_PATH)
		endif()
		file(RELATIVE_PATH FLTK_PATH ${CMAKE_SOURCE_DIR} ${FLTK_PATH})
		get_filename_component(FLTK_PATH ${FLTK_PATH} REALPATH)
		get_filename_component(FLTK_PATH ${FLTK_PATH} PATH)
		message(STATUS "Found FLTK at ${FLTK_PATH}")
	endif()
else()
	get_filename_component(FLTK_PATH ${FLTK_PATH} REALPATH)
	message(STATUS "FLTK path set to ${FLTK_PATH}")
endif()
set(FLTK_INCLUDE_DIR ${FLTK_PATH}/include)
set(FLTK_LIB_DIR ${FLTK_PATH}/lib)

######## Relighting #######
include_directories(${FLTK_INCLUDE_DIR})
include_directories(${OpenGL_INCLUDE_DIRS})
include_directories(${GLEE_PATH})
add_library(VirtualSurgeon_Relighting
	../HeadReplacement/glm.cpp
	../HeadReplacement/spherical_harmonics_analysis.cpp
	../HeadReplacement/LaplacianBlending.cpp
	../HeadReplacement/SpharmonicsUI.cpp
	../HeadReplacement/OGL_OCV_common.cpp
	)

Note that I had to resort to some very dark magic to recover the location of FLTK and GLEE… But it’s a jungle out there.

The source of the photograph is: http://www.flickr.com/photos/roel1943/309048020/
It is released under Creative Commons 2.0 ShareAlike-Attribution. So all the results here are also CC-2.0-SA-A…

Enjoy,
Roy.

From a template FireBreath plugin to an OpenGL plugin

FireBreath is kind of an amazing project. They aim to be able to write a single source that will create plugins for all browsers and all operating systems. A daunting feat by my book. But building a MacOS Safari/Firefox plugin using their framework proved very simple…
So I started here: http://www.firebreath.org/display/documentation/Mac+Video+Tutorial
It’s a video tutorial of how to create a plugin from template, build it, install it and run it. Follow their instructions and you’ll have your plugin ready in 10 minutes.
The next step will be to make our plugin display an OpenGL scene, which is what OpenNI/NITE use to display their depth map. This was also easy, borrowing code from the FireBreath OpenGL example.
However I ended up with a smaller source since I threw away most of the stuff…

class tutorialpluginMac : public tutorialplugin {
public:
    tutorialpluginMac();
	~tutorialpluginMac();

    BEGIN_PLUGIN_EVENT_MAP()
	EVENTTYPE_CASE(FB::AttachedEvent, onWindowAttached, FB::PluginWindowMac)
	EVENTTYPE_CASE(FB::DetachedEvent, onWindowDetached, FB::PluginWindowMac)
	PLUGIN_EVENT_MAP_CASCADE(tutorialplugin)
    END_PLUGIN_EVENT_MAP()

    virtual bool onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindowMac*);
    virtual bool onWindowDetached(FB::DetachedEvent *evt, FB::PluginWindowMac*);
protected:

private:
    void* m_layer;

};

void glutDisplay (void); //this is implemented in the NITE code

@interface MyCAOpenGLLayer : CAOpenGLLayer {
    GLfloat m_angle;
}
@end

@implementation MyCAOpenGLLayer

- (id) init {
    if ([super init]) {
        m_angle = 0;
    }
    return self;
}

- (void)drawInCGLContext:(CGLContextObj)ctx pixelFormat:(CGLPixelFormatObj)pf forLayerTime:(CFTimeInterval)t displayTime:(const CVTimeStamp *)ts {
    //m_angle += 1;
    GLsizei width = CGRectGetWidth([self bounds]), height = CGRectGetHeight([self bounds]);
    GLfloat halfWidth = width / 2, halfHeight = height / 2;

    glViewport(0, 0, width, height);

	glutDisplay(); //let NITE draw it's stuff

    [super drawInCGLContext:ctx pixelFormat:pf forLayerTime:t displayTime:ts];
}

@end

tutorialpluginMac::tutorialpluginMac() : m_layer(NULL) {}

tutorialpluginMac::~tutorialpluginMac()
{
    if (m_layer) {
        [(CALayer*)m_layer removeFromSuperlayer];
        [(CALayer*)m_layer release];
        m_layer = NULL;
    }
}

bool tutorialpluginMac::onWindowAttached(FB::AttachedEvent* evt, FB::PluginWindowMac* wnd)
{
	cout << "tutorialpluginMac::onWindowAttached" << endl;
    if (FB::PluginWindowMac::DrawingModelCoreAnimation == wnd->getDrawingModel() ||
		FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel())
	{
        cout << " Setup CAOpenGL drawing. "<<endl;
        MyCAOpenGLLayer* layer = [MyCAOpenGLLayer new];
        layer.asynchronous = (FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel()) ? NO : YES;
        layer.autoresizingMask = kCALayerWidthSizable | kCALayerHeightSizable;
        layer.needsDisplayOnBoundsChange = YES;
        m_layer = layer;
        if (FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel())
            wnd->StartAutoInvalidate(1.0/30.0);
        [(CALayer*) wnd->getDrawingPrimitive() addSublayer:layer];
    }
    return tutorialplugin::onWindowAttached(evt,wnd);
}

bool tutorialpluginMac::onWindowDetached(FB::DetachedEvent* evt, FB::PluginWindowMac* wnd)
{
    return tutorialplugin::onWindowDetached(evt,wnd);
}

(You guys will have to fill in the gaps… includes, etc.)

This goes in a new file, a new subclass of the generic plugin, only for Mac. For windows, you should subclass again and create the OpenGL context using WIN32 API or equivalent.

CMakeLists.txt files are also affected. Check out the repo.

NITE OpenGL rendering

Now that the plugin will just draw whatever NITE is drawing, half the battle is done. So for the drawing code I took the simple NiPointViewer example from the NITE library (get it here).
But, since we have need no windows management in the OpenNI, again we can make everything more simple. I took the code exactly as it is, and changed really just a small bit.
I added

#undef USE_GLUT
#undef USE_GLES
, which pretty much makes that code compile to a very lean code (without window management etc.).
And I rescued the glOrtho call in glutDisplay()

//#ifdef USE_GLUT
glOrtho(0, mode.nXRes, mode.nYRes, 0, -1.0, 1.0);
#if defined(USE_GLES)
glOrthof(0, mode.nXRes, mode.nYRes, 0, -1.0, 1.0);
#endif


But the rest is pretty much identical.

One more thing, we should start the Kinect driver and OpenNI stack from somewhere in the plugin loading steps. In the main.cpp file from NITE I changed the main() function to kinect_main().
I did that by adding it here, in the generic plugin (not the Mac subclass because it should be called from all OSs):

bool tutorialplugin::onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindow *)
{
    // The window is attached; act appropriately
	kinect_main(0, 0);
	cout << "tutorialplugin::onWindowAttached" << endl;
    return true;
}

It now will fire when a window is attached to the plugin. The OpenGL calls will start running after the OGL context is up and starts rendering in a loop.

Source and stuff

Get the source for the Kinect-FireBreath plugin at GitHub: https://github.com/royshil/KinectPlugin

This is how it looks:

Cool.
Roy

I would like to present something I have been working on recently, a work that immensely affect what I wrote in the blog in the past two years…

To use it:
Go on this page,
Watch the short instruction video,
download the application (MacOSX-Intel-x64 Win32)
and make yourself a model!
It takes just a couple of minutes and it’s very simple…

This work is an academic research project, Please please, take the time to fill out the survey! It is very short..
The results of the survey (the survey alone, no photos of your work) will possibly be published in an academic paper.

Note: No information is sent anywhere in any way outside of your machine (you may even unplug the network). All results are saved locally on your computer, and no inputs are recorded or transmitted. The application contains no malware. The source is available here.

Note II: All stock photos of models used in the application are released under Creative Commons By-NC-SA 2.0 license. Creator: http://www.flickr.com/photos/kk/. If you wish to distribute your results, they should also be released under a CC-By-NC-SA 2.0 license.

Thank you!
Roy.

OK let’s get down to business.

I followed a very simple technique described in this paper. I know, you say, “A Paper? Really? I’m not gonna read that technical boring stuff, give the bottom line! man.. geez.” Well, you are right, except that this paper IS the bottom line, it’s dead simple. It’s almost a tutorial. It is also referenced by the OpenCV documentation.

The method is simple:
- Extract features of choice from training set that contains all classes.
- Create a vocabulary of features by clustering the features (kNN, etc). Let’s say 1000 features long.
- Train your classifiers (SVMs, Naive-Bayes, boosting, etc) on training set again (preferably a different one), this time check the features in the image for their closest clusters in the vocabulary. Create a histogram of responses for each image to words in the vocabulary, it will be a 1000-entries long vector. Create a sample-label dataset for the training.
- When you get an image you havn’t seen – run the classifier and it should, god willing, give you the right class.

Turns out, those crafty guys in WillowGarage have done pretty much all the heavy lifting, so it’s up for us to pick the fruit of their hard work. OpenCV 2.3 comes packed with a set of classes, whose names start with BOW for Bag Of Words, that help a lot with implementing this method.

Starting with the first step:

Mat training_descriptors(1,extractor->descriptorSize(),extractor->descriptorType());

SurfFeatureDetector detector(400);
vector keypoints;

// computing descriptors
Ptr extractor(
   new OpponentColorDescriptorExtractor(
      Ptr(new SurfDescriptorExtractor())
   )
);

while(..loop a directory? a file?..) {
   Mat img = imread(filepath);
   detector.detect(img, keypoints);
   extractor->compute(img, keypoints, descriptors);
   training_descriptors.push_back(descriptors);
}

Simple!
Let’s go create a vocabulary then. Luckily, OpenCV has taken care of that, and provide a simple API:

BOWKMeansTrainer bowtrainer(1000); //num clusters
bowtrainer.add(training_descriptors);
Mat vocabulary = bowtrainer.cluster();

Boom. Vocabulary.
Now, let’s train us some SVM classifiers!
We’re gonna train a 2-class SVM, in a 1-vs-all kind of way. Meaning we train an SVM that can say “yes” or “no” when choosing between one class and the rest of the classes, hence 1-vs-all.
But first, we need to scour the training set for our histograms (the responses to the vocabulary, remember?):

vector<KeyPoint> keypoints;
Mat response_hist;
Mat img;
string filepath;
map<string,Mat> classes_training_data;

Ptr<FeatureDetector > detector(new SurfFeatureDetector());
Ptr<DescriptorMatcher > matcher(new BruteForceMatcher<L2<float> >());
Ptr<DescriptorExtractor > extractor(new OpponentColorDescriptorExtractor(Ptr<DescriptorExtractor>(new SurfDescriptorExtractor())));
Ptr<BOWImgDescriptorExtractor> bowide(new BOWImgDescriptorExtractor(extractor,matcher));
bowide->setVocabulary(vocabulary);

#pragma omp parallel for schedule(dynamic,3)
for(..loop a directory?..) {
   img = imread(filepath);
   detector->detect(img,keypoints);
   bowide.compute(img, keypoints, response_hist);

   #pragma omp critical
   {
      if(classes_training_data.count(class_) == 0) { //not yet created...
         classes_training_data[class_].create(0,response_hist.cols,response_hist.type());
         classes_names.push_back(class_);
      }
      classes_training_data[class_].push_back(response_hist);
   }
   total_samples++;
}

Now, two things:
First notice I’m keeping the training data for each class separately, this is because we will need this for later creating the 1-vs-all samples-labels matrices.
Second, I use OpenMP multi(-threading)processing to make the calculation parallel, and hence faster, on multi-core machines (like the one I used). Time is sliced by a whole lot. OpenMP is a gem, use it more. Just a couple of #pragma directives and you’re multi-threading.

Alright, data gotten, let’s get training:

#pragma omp parallel for schedule(dynamic)
for (int i=0;i<classes_names.size();i++) {
   string class_ = classes_names[i];
   cout << omp_get_thread_num() << " training class: " << class_ << ".." << endl;

   Mat samples(0,response_cols,response_type);
   Mat labels(0,1,CV_32FC1);

   //copy class samples and label
   cout << "adding " << classes_training_data[class_].rows << " positive" << endl;
   samples.push_back(classes_training_data[class_]);
   Mat class_label = Mat::ones(classes_training_data[class_].rows, 1, CV_32FC1);
   labels.push_back(class_label);

   //copy rest samples and label
   for (map<string,Mat>::iterator it1 = classes_training_data.begin(); it1 != classes_training_data.end(); ++it1) {
      string not_class_ = (*it1).first;
      if(not_class_.compare(class_)==0) continue; //skip class itself
      samples.push_back(classes_training_data[not_class_]);
      class_label = Mat::zeros(classes_training_data[not_class_].rows, 1, CV_32FC1);
      labels.push_back(class_label);
   }

   cout << "Train.." << endl;
   Mat samples_32f; samples.convertTo(samples_32f, CV_32F);
   if(samples.rows == 0) continue; //phantom class?!
   CvSVM classifier;
   classifier.train(samples_32f,labels);

   //do something with the classifier, like saving it to file
}

Again, I parallelize, although the process is not too slow.
Note how I build the samples and the labels, where each time I put in the positive samples and mark the labels ’1′, and then I put the rest of the samples and label them ’0′.

Moving on to …. testing the classifiers!
Nothing seems to me like more fun than creating a confusion matrix! Not really, but let’s see how it’s done:

map<string,map<string,int> > confusion_matrix; // confusionMatrix[classA][classB] = number_of_times_A_voted_for_B;
map<string,CvSVM> classes_classifiers; //This we created earlier

vector<string> files; //load up with images
vector<string> classes; //load up with the respective classes

for(..loop over a directory?..) {
   Mat img = imread(files[i]),resposne_hist;

   vector<KeyPoint> keypoints;
   detector->detect(img,keypoints);
   bowide->compute(img, keypoints, response_hist);

   float minf = FLT_MAX; string minclass;
   for (map<string,CvSVM>::iterator it = classes_classifiers.begin(); it != classes_classifiers.end(); ++it) {
      float res = (*it).second.predict(response_hist,true);
      if (res < minf) {
         minf = res;
         minclass = (*it).first;
      }
   }
   confusion_matrix[minclass][classes[i]]++;
}

When you take a look in my files, you will find a much complicated way of doing this. But this is the core idea – look in the image for the response histogram to the vocabulary of features (rather, feature-cluster-ceneters), run it by all the classifiers and take the one with the best score. Simple.
Consider making this parallel as well. No reason for it to be serial.

That’s about covers it.

Code

Lately I’m pushing stuff in Github.com using git rather than SVN on googlecode. Donno why, it’s just like that.
Get the whole thing at:
https://github.com/royshil/FoodcamClassifier

Follow the build instructions, they’re a breeze, and then follow the runnning instructions. It’s basically a series of command-line programs you run to get through each step, and in the end you have like a “predictor” service that takes an image and produces a prediction.

OK guys, have fun classifying stuff!
Roy.

The only things I was interested were the head position and hands positions so I created a struct to hold these, plus some things OpenNI need to get the positions. I did this so it could be run in a different thread, and this struct can be the shared memory:

struct openni_stuff {
	xn::DepthGenerator* dg;
	xn::UserGenerator* ug;
	xn::Context* ctx;
	XnSkeletonJointPosition* Head;
	XnSkeletonJointPosition* rh;
	XnSkeletonJointPosition* lh;
};

All these must be populated in the main before starting the thread

xn::Context g_Context;
xn::DepthGenerator g_DepthGenerator;
xn::UserGenerator g_UserGenerator;
XnSkeletonJointPosition Head;
XnSkeletonJointPosition lHand;
XnSkeletonJointPosition rHand;

int main(..) {
..
g_Context.Init();
g_Context.FindExistingNode(XN_NODE_TYPE_DEPTH, g_DepthGenerator);
g_Context.FindExistingNode(XN_NODE_TYPE_USER, g_UserGenerator);
..
g_UserGenerator.Create(g_Context);
..
g_Context.StartGeneratingAll();
..
DWORD threadid;
struct openni_stuff s;
s.ctx = &g_Context;
s.dg = &g_DepthGenerator;
s.ug = &g_UserGenerator;
s.Head = &Head;
s.rh = &rHand;
s.lh = &lHand;
CreateThread(
            NULL,                   // default security attributes
            0,                      // use default stack size
			MyThreadFunction,       // thread function name
            (LPVOID)(&s),          // argument to thread function
            0,                      // use default creation flags
            &threadid);   // returns the thread identifie

glutMainLoop();

This code is very abstracted, there are more things to do in order for it to work, you can see them in the code repo.
But basically the new thread is the one getting the information off the OpenNI framework and keeps the head position and hands positions vectors updated.

DWORD WINAPI MyThreadFunction( LPVOID lpParam ) {
       //Unpack the struct, don't care for shallow copy since it's all pointers anyway
	struct openni_stuff s = *((struct openni_stuff*)lpParam);
	for(;;) {
		getOpenNIData(s);
		Sleep(30);
	}
	return 0;
}

void getOpenNIData (struct openni_stuff s)
{
	xn::SceneMetaData sceneMD;
	xn::DepthMetaData depthMD;
	s.dg->GetMetaData(depthMD);

	if (!g_bPause)
	{
		// Read next available data
		s.ctx->WaitAndUpdateAll();
	}

	// Process the data
	g_DepthGenerator.GetMetaData(depthMD);
	s.dg->GetMetaData(depthMD);
	rHand.position.X = NULL;
	s.ug->GetUserPixels(0, sceneMD);
	DrawDepthMap(depthMD, sceneMD, *s.Head, *s.rh, *s.lh);
}

I thought this will give a performance boost as the WaitAndUpdateAll() call usually takes a little while, but it didn’t matter much…

The OpenGL (GLUT) runs on the main thread, and just looks at these updated vectors for the current position.

Off-Axis projection

The concept of off-axis projection is very important for this project. This very good article explains everything about generalized perspective projections, it also includes C code!, I recommend reading it. But basically off-axis projection is when the viewing eye is not perpendicular the projection surface, nor it needs to be centered in relation to it. It’s what goes on in our human binocular vision, each eye looks at the same point but they are not perpendicular to the virtual projection surface (they are angled to it), and they both have an offset from the center. Just read that little paper….

Anyway, cutting to the chase, we need to project the rendered objects in the scene onto the projection table, assuming the user is not looking at it perpendicularly (like they would with a normal screen). Thanks to the code in the aforementioned article – this is a breeze.

void subtract(float u[3], float v[3], float n[3]) {
	u[0] = v[0] - n[0];
	u[1] = v[1] - n[1];
	u[2] = v[2] - n[2];
}

void projection( float *pa,
				float *pb,
				float *pc,
				float *pe, float n, float f)
{
	float va[3], vb[3], vc[3];
	float vr[3], vu[3], vn[3];
	float l, r, b, t, d, M[16];

	// Compute an orthonormal basis for the screen.
	subtract(vr, pb, pa);
	subtract(vu, pc, pa);

	glmNormalize(vr);
	glmNormalize(vu);
	glmCross(vr, vu, vn);
	glmNormalize(vn);

	// Compute the screen corner vectors.
	subtract(va, pa, pe);
	subtract(vb, pb, pe);
	subtract(vc, pc, pe);

	// Find the distance from the eye to screen plane.
	d = -glmDot(va, vn);

	// Find the extent of the perpendicular projection.
	l = glmDot(vr, va) * n / d;
	r = glmDot(vr, vb) * n / d;
	b = glmDot(vu, va) * n / d;
	t = glmDot(vu, vc) * n / d;
	// Load the perpendicular projection.
	glMatrixMode(GL_PROJECTION);
	glPushMatrix();
	glLoadIdentity();
	glFrustum(l, r, b, t, n, f);
	// Rotate the projection to be non-perpendicular.
	memset(M, 0, 16 * sizeof (float));
	M[0] = vr[0]; M[4] = vr[1]; M[ 8] = vr[2];
	M[1] = vu[0]; M[5] = vu[1]; M[ 9] = vu[2];
	M[2] = vn[0]; M[6] = vn[1]; M[10] = vn[2];
	M[15] = 1.0f;
	glMultMatrixf(M);
	// Move the apex of the frustum to the origin.
	glTranslatef(-pe[0], -pe[1], -pe[2]);
	glMatrixMode(GL_MODELVIEW);
	glPushMatrix();
}

I am using glm.h & glm.c from Nate Robbins to do some basic lin-algebra. I just didn’t feel like re-writing the code, and I’m already using it to load Wavefront OBJ models. The only missing function is subtract which is included.

Loading the OBJ models is super easy with glm.h:

	   objmodel_ptr = glmReadOBJ("../bunny1.obj");
	   if (!objmodel_ptr)
		   exit(0);

	   glmUnitize(objmodel_ptr);
	   glmFacetNormals(objmodel_ptr);
	   glmVertexNormals(objmodel_ptr, 90.0);

Now that we can create off-axis views (this can be reused for other projects, such as projects with VR glasses!), I draw the scene after applying this projection:

GLfloat eye[4] = {0,200,1050,0}; //position of eye
double kinectHeight = 300;  //the Kinect is by the table, at a certain height (measured)
GLdouble tlv[3] = {-530, -kinectHeight, 90},   //top-left point of table in Kinect coordinates (millimeters)
		trv[3] = {530, -kinectHeight, 90}, //top-right
		brv[3] = {530, -kinectHeight, 955}, //bottom-right
		blv[3] = {-530, -kinectHeight, 955}; //bottom-left
GLdouble obj[3] = {-200, tlv[1], 522.5}; //the virtual object's real-world position (mm)

static void display(GLenum mode)
{
       //set the eye position
	if(Head.position.X != 0.0f || Head.position.Y != 0.0f || Head.position.Z != 0.0f)
	{
		eye[0] = Head.position.X;
		eye[1] = Head.position.Y;
		eye[2] = Head.position.Z;
	}

	glClear(GL_DEPTH_BUFFER_BIT | GL_COLOR_BUFFER_BIT);
	offAxisView();
}

void offAxisView() {
	projection(blvf, brvf, tlvf, eye, 1.0f, 10000.0f);

	glLightfv(GL_LIGHT0, GL_POSITION, lightp);
	drawScene();

	glPopMatrix();
	glMatrixMode(GL_PROJECTION);
	glPopMatrix();
	glMatrixMode(GL_MODELVIEW);
}

void drawScene() {
	//Just draw an object..
	glPushMatrix();
	glTranslated(obj[0]-10,obj[1]+80,obj[2]); //translating to accomodate for obj size
	glColor4f(1.0, 0.0, 0.0, 1.0);
	glScaled(80,80,80);
	glmDraw(objmodel_ptr,GLM_SMOOTH);
	glPopMatrix();
}

You can see that I measured the position of the table in respect to the Kinect sensor’s center, we assume that it is the origin, and these are used for the off-axis projection w.r.t the eye.

That’s pretty much it… the program runs, you have to stand in the silly “Psi” position for the OpenNI framework to calibrate, and then the graphics will be rendered according to your head position.

To create your own setup, just put in the right position of the table in respect to the Kinect sensor in real-world coordinates (mm).

Code

Can be downloaded from SVN as usual:
svn co https://morethantechnical.googlecode.com/svn/trunk/kinect_motion_parallax/main.cpp

Video

Enjoy
Roy.

So first you’ll need to export your loved tracks from Last.fm in tab separated format – a mere button press.

The “tsv” (tab separated values) file has a simple format: <song name> <artist> <Last.fm url>

And now for the script, first, the loved tracks file is tab separated, so we use AWK to get the 2 first fields which are song-name and song-artist.
Then we use a neat command-line tool to download YT movies: http://rg3.github.com/youtube-dl/documentation.html.

mkdir mylovedtracks
cd mylovedtracks
awk -F\t '{print "../youtube-dl.py -f 18 -t \"ytsearch:" $1 " " $2 "\""}' ../my_lovedtracks.tsv | csh

The single-liner will download all the loved tracks from the tsv file into the current directory, given that youtube-dl.py & my_lovedtracks.tsv exist in the parent directory. -f 18 says it will download only MP4s and ytsearch says it will try to search YT for the term “song-name song-artist” and download the 1st result. The | csh says it will send this command AWK formatted into a new shell process.

The saved MP4 will be named after the name of the video, with addition of the YT hash string.

All the mp4s have been downloaded, so let’s batch convert them to mp3s:

mkdir sound
for f in *.mp4 ; do n=`echo $f | cut -d '.' -f1`; if [ ! -e sound/$n.mp3 ]; then `mplayer $n.mp4 -vc dummy -vo null -ao pcm:file=sound/temp.wav; lame -V2 sound/temp.wav sound/$n.mp3; rm sound/temp.wav`; fi ; done

This single-liner will extract audio from the mp4 into a PCM temp.wav file using MEncoder, and then convert to VBR MP3 using Lame.
You can run this command many times, as it checks if the file has not been converted yet. So you’re impatient (like me) on converting some of the MP4s before everything was downloaded – just run it, and later run it again.

Congrats, all your loved tracks were downloaded.

A few limitation to this method:
* Sometimes downloaded songs are not exactly what you wanted, especially specific versions. The search is arbitrary, and can’t be controlled too much.
* ID3 tags are non existent, although something can probably be done about that in the Lame encoding phase.
* Very high potential for parallelization that is unexploited. Mostly in the YT download phase, where YT pushes the first ~15% of the video very fast (I saw 1200Kb/s even), and then maintains a steady d/l rate to get the video downloaded by ~1:00 minute (may be as low as 50Kb/s). Downloading many videos at once could help.
* Still not a true single-liner, it is a two-step thing. But that can be done by modifying the 2nd step a bit and putting into the AWK print of the 1st step.
* MP3′s volume normalization – very important! else every songs sounds different and you must do vol-up vol-down all the time…

Still, did a nice quick job for me…

Roy.

Start from a simple Kinect frames feed from here:

int main(int argc, char **argv) {
	bool die(false);

	Mat depthMat(Size(640,480),CV_16UC1);
	Mat depthf  (Size(640,480),CV_8UC1);
	Mat rgbMat(Size(640,480),CV_8UC3,Scalar(0));
	Mat ownMat(Size(640,480),CV_8UC3,Scalar(0));

        Freenect::Freenect freenect;
        MyFreenectDevice& device = freenect.createDevice<MyFreenectDevice>(0);

	device.startVideo();
	device.startDepth();

    while (!die) {
    	device.getVideo(rgbMat);
    	device.getDepth(depthMat);
    	depthMat.convertTo(depthf, CV_8UC1, 255.0/2048.0);
        cv::imshow("depth",depthf);
		char k = cvWaitKey(5);
		if( k == 27 ){
			break;
		}
    }

   	device.stopVideo();
	device.stopDepth();
	return 0;
}

Now let’s stretch the signal a little bit and add the inpainting:

		//interpolation & inpainting
		{
			Mat _tmp,_tmp1; //minimum observed value is ~440. so shift a bit
			Mat(depthMat - 400.0).convertTo(_tmp1,CV_64FC1);

			Point minLoc; double minval,maxval;
			minMaxLoc(_tmp1, &minval, &maxval, NULL, NULL);
			_tmp1.convertTo(depthf, CV_8UC1, 255.0/maxval);  //linear interpolation

                       //use a smaller version of the image
			Mat small_depthf; resize(depthf,small_depthf,Size(),0.2,0.2);
                        //inpaint only the "unknown" pixels
			cv::inpaint(small_depthf,(small_depthf == 255),_tmp1,5.0,INPAINT_TELEA);

			resize(_tmp1, _tmp, depthf.size());
			_tmp.copyTo(depthf, (depthf == 255));  //add the original signal back over the inpaint
		}

Note that I’m using a small copy of the image, because inpainting is a heavy computation, and it works best on low frequencies. I copy back the original signal over the up-sized inpainted one to retain high frequencies.

It works pretty well!

Enjoy
Roy.

Using Android’s Frame Animation API

The first attempt I took at frame animation was using Android’s own AnimationDrawable. I thought it would give me the best solution, as it’s the closest to the native OS and probably optimized. I was wrong.
This API is highly suspect to OutOfMemory exceptions, either when loading an animation of more than ~50 frames, an when loading more than one animation. On top of that, it is not doing a great job at displaying the frames, and produces a lot of jitter.

So using it is very very simple, and I wrote about it in a previous post, when I was still trying to make it work.

Basically, all you need is an ImageView to display your animation. Prepare some AnimationDrawable in XML. Then you can either pre-load an animation, or just fire an animation regularly, which is only setting the Drawable for the ImageView.

If you fire the animation more than once, remember to reset it.

This is by far the simplest way to go, and best if you have a simple animation. But it dies very quickly of memory issues if you push it too hard.

Using HTML and animated GIFs

This seemed like a classical solution for frame animation. The browser should have absolutely no problems playing it – so I thought. Turns out Android 2.1′s web browser doesn’t play animated GIFs! So when I was working on my development phone, a 2.2er, there was no problem, but when I switched to my deployment phone, a 2.1er , the screen just goes black.
If you’re on 2.2 – this a very nice way to frame animate. It’s clean and works at high frame rates.

First you would need to obtain animated GIFs that are compatible with Android’s web browser. I did that using ImageMagick’s (IM) animation toolbox.

convert myanim_split_*.png myanim.gif

But… there are some details to attend to. First, you probably would like to have control over looping the animation, that can be done using the -loop parameter of convert, setting it to 1 will play the animation once, 0 will loop forever.
How about transparent background? this is very important for character animation, that usually live in a “world” and should not occlude the background. Well, if your animation’s PNGs have the background transparet, that would reflect in the animated GIF. But, just creating a GIF out of your PNGs will give you something like this:

The background is not clearing frame-to-frame. So add in the -set dispose background (or “dispose previous” to the IM line:

convert myanim_split_*.png -set dispose background myanim.gif

Now it looks better, and also has a transparent background:

There’s still the issue of showing it up as part of the layout, and that will be done with WebView. The tricky part is controlling when the animation will fire. There are several ways to go about it, such as loading a single HTML file and using webview.loadUrl with a “javascript:…” URL to replace the image, or using a simpler method of webview.loadData with HTML code that just displays the image with <img src="..">. There is no “right way” as this is the “hack way” anyway.

WebView wb = (WebView) findViewById(R.id.webview);
wb.loadDataWithBaseURL(
		"fake://lala",
		"<div style=\"text-align: center;\"><IMG id=\"myanim\" SRC=\"file:///android_asset/myanim.gif\" style=\"height: 100%\" /></div>",
		"text/html",
		"UTF-8",
		"fake://lala");

Notice that the GIF file must reside in the “assets” directory of the android project, and the “file:///android_asset/” URL goes right to there. You may as well skip the whole HTML thing and just have WebView loadURL the GIF file right away.

Now this will work on Android 2.2 and up, but will fail for Android 2.1 as the web browser doesn’t implement animated GIFs yet.

Using MediaPlayer and animated GIFs

I saw somewhere that even though the WebView in 2.1 cannot play GIFs, the MediaPlayer sure can, so it’s worth mentioning that.

So I set up a surface in the layout

<SurfaceView android:id="@+id/mysurfaceview"
	android:layout_width="fill_parent"
	android:layout_height="0dip"
	android:layout_weight="1"
	/>

Got the holder in code, and instantiated a MediaPlayer

mCharPreview = (SurfaceView) findViewById(R.id.mysurfaceview);
holder = mCharPreview.getHolder();
holder.addCallback(this);
extras = getIntent().getExtras();

mMediaPlayer = new MediaPlayer();
mMediaPlayer.setDisplay(holder);
mMediaPlayer.setOnCompletionListener(this);
mMediaPlayer.setOnPreparedListener(this);
mMediaPlayer.setOnBufferingUpdateListener(this);
mMediaPlayer.setOnVideoSizeChangedListener(this);
mMediaPlayer.setAudioStreamType(AudioManager.);

And then listen on surfaceCreated to prepare the player

public void surfaceCreated(SurfaceHolder holder) {
        try {
            AssetFileDescriptor openFd = getAssets().openFd("myanim.gif");
		mMediaPlayer.setDataSource(openFd.getFileDescriptor());
            mMediaPlayer.prepare();
	} catch (Exception e) {
		e.printStackTrace();
		(new AlertDialog.Builder(this)).setTitle("Exception").setMessage(e.getClass().getName() + ":" + e.getLocalizedMessage()).create().show();
	}
}

Then listen on prepared to start the animation

public void onPrepared(MediaPlayer mediaplayer) {
        mIsVideoReadyToBePlayed = true;
        if (mIsVideoReadyToBePlayed) {
             holder.setFixedSize(200, 300);
             mMediaPlayer.start();
        }
}

But – this code failed for me in the mMediaPlayer.prepare() with an exception like "Prepare failed.: status=0xFFFFFFF". I’m guessing that the player fails because the format is not recognized.

Using HTML and Javascript

So, after being exhausted with having some engine play the animation, I decided to go back to using WebView with Javascript code to flip the images one after the other.

Well this was pretty straight forward, I created a small HTML code:

<html>
 <script>
 //---------------------------------------------------------------
//just a function to pad numbers with 0s, good for animation frames in sequential files...
 function FormatNumberLength(num, length) {
    var r = "" + num;
    while (r.length < length) {
        r = "0" + r;
    }
    return r;
}
//---------------------------------------------------------------

  var state=0;
  var myinterval = -1; //JS interval handler

//---------------------------------------------------------------
//parse querystring - that's where I get the image to show, and other parameters
  var qs = (function(a) {
    if (a == "") return {};
    var b = {};
    for (var i = 0; i < a.length; ++i)
    {
        var p=a[i].split('=');
        b[p[0]] = decodeURIComponent(p[1].replace(/\+/g, " "));
    }
    return b;
  })(window.location.search.substr(1).split('&'));
//---------------------------------------------------------------

  var firstFrame = parseInt(qs["first"]);   //the index of the first frame
  var lastFrame = parseInt(qs["last"]);    //the index of the last frame

//---------------------------------------------------------------
  // intializes animation timer
  function ini() {
  	if(firstFrame >= 0) { //animation
    	setTimeout("myinterval = setInterval(\"periodic()\",100);",100);
    } else {				//static
    	document.getElementById("myimg").src="file:///android_asset/"+ qs["anim_file"] +".png";
    }
  }
//---------------------------------------------------------------

  // called regularly to perform animation
  function periodic() {
    state+=2;  //jump by 2 frames? can change this to 1 if you like...
    animState = firstFrame + state;

    if (animState <= lastFrame) {
		document.getElementById("myimg").src="file:///android_asset/"+ qs["anim_file"] + FormatNumberLength(animState,4) +".png";
    }
    else {
    	if(qs["loop"] == "false")  //'loop' will say if we keep playing the animation.. duh
    		clearInterval(myinterval);
    	else
    		state = 0;
    }
  }

 </script>
 <style>
 #wrapper {
     text-align: center;
     background-color: black; /* make sure to use this, since sometimes there are ghosts */
 }
 #im {
     background-color: black;
 }
 </style>
 <body onLoad="ini()">
  <div id="wrapper">
     <img id="myimg" src=""/> <!--   this is where the image goes -->
  </div>
 </body>
</html>

Put this file, as usual, in the “assets” directory where it can be found easily with the "file:///android_assets/..." URL.

So, this HTML is loaded into the WebView with some parameters on the URL that will tell it what to show, like so:

       //This class holds all we need for an animation: filename, number of frames, etc.
	private class MyAnim {
		String filename;
		int start;
		int end;
		boolean loop;
		public MyAnim(String filename, int start, int end) {
			super();
			this.filename = filename;
			this.start = start;
			this.end = end;
			this.loop = false;
		}
		public MyAnim(String filename, int start, int end, boolean loop) {
			super();
			this.filename = filename;
			this.start = start;
			this.end = end;
			this.loop = loop;
		}
	}

        //this function will "fire" an animation, essentially load the HTML code with the proper parameters
	private void fireAnimation(final MyAnim myAnim, final boolean shouldTurn) {
		findViewById(R.id.webview).post(new Runnable() {
			@Override
			public void run() {
				long now = (new Date()).getTime();
				if((now - anim_start_ts) < 2000) return; //let other animations finish man! geez...

				WebView wb = (WebView) findViewById(R.id.webview);

				//supply the "base" filename, the first frame number, last frame and should the animation repeat
				wb.loadUrl("file:///android_asset/animate.html?anim_file="+myAnim.filename+"&first="+myAnim.start+"&last="+myAnim.end+"&loop="+myAnim.loop);
				wb.invalidate();

				anim_start_ts = now;
			}
		});
	}

The filenames should be sequential, like: “myanim0001.jpg”, “myanim0002.jpg”, ….
And then an animation may be from the file “mayanim0023.jpg” to “myanim0046.jpg”. The parameters for the HTML code will be then: anim_file="myanim"&first=23&last=46.

But! We are not done.
Because stupid Android 2.1 does not clear the first image that loads into the <img ... >!
So you get these weird “ghosting” effects, where the new images of the animation are shown superimposed on the first image…
I couldn’t get past this, so I found another way of doing it….

Using HTML, JS and innerHTML

This is what I ended up using.
Basically everything stays the same, except for the fact that now we are not leaving the <img ... > in there to ghost stuff up, we’re completely replacing it with new HTML code using innerHTML.

The change is slight, in the JS I used:

    	document.getElementById("wrapper").innerHTML=
	    	"<div style=\"-webkit-transform: scaleX("  +  ((qs["flip"]=="0")?"-1":"1")  +  ");\">"+
	    	"<img id=\"im\" style=\"height:100%\" src=\"file:///android_asset/"+ qs["anim_file"]+"\" />"+
	    	"</div>";

Note that now I also introduced a new parameter: “flip”, that uses -webkit-transform to flip the displayed image.

Phew, that was a battle. I won. And now we learned a few things about Android and options for frame animation.
Some other options to explore: flipping OpenGLES textures, encoding animations to mp4s?

Please share your experience with Android frame animation!

Roy.

Portions of this page are modifications based on work created and shared by Google and used according to terms described in the Creative Commons 3.0 Attribution License.

I’d like to share a quick tip on rotating video files.

I’m always frustrated with taking videos with my phone. Single handedly it’s easiest to do it when the phone is upright and not in landscape mode. But the files are always saved in landscape mode, which makes them rotated when you watch.
Although there are plenty of GUI software to do it, using the command line is faster and can also be batched!

Using FFmpeg

This is the basic syntax
./ffmpeg -vf transpose=0 -i input.mp4 output.avi

Just using -vf transpose=0 to rotate 90 deg clockwise. If you get the “Unrecognized option ‘vf’” error, you need to configure & build ffmpeg with --enable-filters (or at least without --disable-filters), and check with ffmpeg -filters that you get them to show up.

Also, I always get very lousy video quality when using the plain vanilla settings. It turns out the problem is with the frame rate. If you leave it as-is the (default) mpeg compression makes a lot of I and P frames, and too few B frames. So set it up explicitly.

I ended up with
ffmpeg -vf transpose=0 -i input.mp4 -r 30 output.avi

Easy.

Update [3/1/11]: Actually just using -vf transpose=0 will flip the video horizontally as well, which is undesirable in some cases. To counter that I use: -vf "transpose=0,hflip=0", and it resolves the problem.

Using MEncoder

Again, pretty simple thing to do:
mencoder input.mp4 -nosound -o characters-resize-turn.avi -vf rotate=0 -ovc lavc -lavcopts vcodec=mpeg4 -ofps 30

Note that I kill the audio with -nosound, and again set the frame rate-ofps 30 or else I get the “duplicate frames” problem.

Enjoy
Roy.

Just wanted to share a thing I made – a simple 2D hand pose estimator, using a skeleton model fitting. Basically there has been a crap load of work on hand pose estimation, but I was inspired by this ancient work. The problem is setting out to find a good solution, and everything is very hard to understand and implement. In such cases I like to be inspired by a method, and just set out with my own implementation. This way, I understand whats going on, simplify it, and share it with you!

Anyway, let’s get down to business.

A bit about energy minimization problems

A dear friend revealed before me the wonders of energy minimization problems a while back, and ever since I have trying to find uses for that method. Basically, it is trying to find a global minimum for a complicated energy function (usually with many parameters), by following the function’s gradient. Such methods are often called Gradient Descent, and used mostly for non-linear systems that can’t be solved easily using a least-squares variant.

A lot of work in computer vision was done using energy functions (I believe the most seminal was Snakes, over 10,000 citations), usually having two terms: Internal energy and External energy. The equilibrium between the two terms should result in a low-energy system – our optimal result. So we would like to formulate the terms in our system such that when they are 0 – they describe the system as we want it.

Following the works with active contours, I believe the external energy function should have to do with how the hand model fits to the hand blob, and the internal energy will have to do with how “comfortable” the hand is with this configuration.

The hand model

Let’s see how a 2D model of a hand might look like

Kinda looks like a rake… huh?

There are some parts that practically can’t change much, i.e the palm (orange), and some that might change drastically, i.e the fingers (red). Each finger has joints (blue circle), and a tip (bigger blue circle).

typedef struct finger_data {
	Point2d origin_offset;		//base or finger relative to center hand
	double a;					//angle
	vector<double> joints_a;	//angles of joints
	vector<double> joints_d;	//bone length
} FINGER_DATA;

typedef struct hand_data {
	FINGER_DATA fingers[5];		//fingers
	double a;					//angle of whole hand
	Point2d origin;				//center of palm
	Point2d origin_offset;		//offset from center for optimization
	double size;				//relative size of hand = length of a finger
} HAND_DATA;

At first I thought, since I’m only interested in the tips of the fingers, to use Inverse Kinematics to guide the tips to a certain point and let the joints find their own minimal energy position, following this article. But I abandoned this method because of complications.

I also had to simplify this model, for real-time estimation and also better results. So in the end I ended up with a very rigid model, that allows only on joint per finger and no angular movement.

Using tnc.c

tnc.c is a “library”, essentially one c file, that implements a line search algorithm that is able to find the minimum point of a multi-variate function. I’m not certain of the algorithm details, and it’s not so important as it can be replaced with any other similar library. But, tnc.c has a great advantage – it is dead simple. One function will start the gradient decent, calling-back a function to calculate the gradients.

So basically I had to write just one very short function:

static int my_f(double x[], double *f, double g[], void *state) {
	DATA_FOR_TNC* d_ptr = (DATA_FOR_TNC*)state;
	DATA_FOR_TNC new_data = *d_ptr;

	mapVecToData(x,new_data.hand);

	*f = calc_Energy(new_data,*d_ptr);

	//calc gradients
	{
		double _x[SIZE_OF_HAND_DATA];

		for(int i=0;i<SIZE_OF_HAND_DATA;i++) {
			memcpy(_x, x, sizeof(double)*SIZE_OF_HAND_DATA); //reset variables
			_x[i] = _x[i] + EPSILON; //change only one variable
			mapVecToData(_x, new_data.hand);
			double E_epsilon = calc_Energy(new_data,*d_ptr);
			g[i] = ((E_epsilon - *f) / EPSILON); //calc the gradient for this variable change
		}
	}

	return 0;
}

This function is called by tnc.c on every iteration of the search, the double x[] is the state of variables the search is now examining, double* f is the energy for this state, double g[] are the gradients (same size as x[]), and voide* state is a user-defined variable that can be carried along the process.

So what I did is simply changed the value of each parameter in turn, to test how it effects the energy in the system. I get a measure of the energy, then I subtract it from the “natural” setup (without any changes to parameters) energy measure, and I get the gradient for this parameter.

The energy function came out a bit different in the end:

static double calc_Energy(DATA_FOR_TNC& d, DATA_FOR_TNC& orig_d) {
	double _sum = 0.0;

	//external energy: how close are the joints to the hand blob? (how well do they fit to it)
	vector<Point2d> joints;
	Mat tips(5,1,CV_64FC2);

	for (int j=0; j<5; j++) {
		joints.clear();
		FINGER_DATA f = d.hand.fingers[j];
		Point2d _newTip = newTip(f,d.hand,joints); //get joints for this finger

		for (int i=0; i<tmp.size(); i++) { //for each joint find how far it is from the blob
			double ds = pointPolygonTest(d.contour, tmp[i]+getHandOrigin(d.hand), true);
			ds += 5;
			ds = 1 * ((ds < 0) ? -1 : 1) * (ds*ds) ;
			_sum -= (ds > 0) ? 0 : 100*ds;
		}

		tips.at<Point2d>(j,0) = _newTip;
	}

	//lazyness of fingers - joints should strive to be as they were in the natural pose
	vector<double> _angles;
//	for (int j=0; j<5; j++) {
//		FINGER_DATA f = d.hand.fingers[j];
//		FINGER_DATA of = orig_d.hand.fingers[j];
////		_angles.push_back(f.a - of.a);
//		for (int i=0; i<f.joints_d.size(); i++) {
////			_angles.push_back(f.joints_a[i] - of.joints_a[i]);
//			_angles.push_back(f.joints_d[i] - of.joints_d[i]);
//		}
//	}
	_angles.push_back(d.hand.a-orig_d.hand.a); //the angle of the hand should be as it was before
	_sum  += 10000*norm(Mat(_angles));

	if(_sum < 0) return 0;
	return _sum;
}

You’ll notice the commented out section. The “laziness of fingers” turned out not to give good results… A different metric is needed! I have not found it yet, maybe you have a good idea?

Starting tnc.c is very simple: Allocating the vectors for X and gradients, initializing the model from the blob, and calling the simple_tnc convenience method. simple_tnc starts tnc with some default parameters that don’t affect the outcome (at least in my tries).

void estimateHand(Mat& mymask) {
	double _x[SIZE_OF_HAND_DATA] = {0};
	Mat X(1,SIZE_OF_HAND_DATA,CV_64FC1,_x);
	double f;
	Mat gradients(Size(SIZE_OF_HAND_DATA,1),CV_64FC1,Scalar(0));

	namedWindow("state");

	initialize_hand_data(d, mymask);

	mapDataToVec((double*)X.data, d.hand);

	simple_tnc(SIZE_OF_HAND_DATA, (double*)X.data, &f, (double*)gradients.data, my_f, (void*)&d, 1, 0);

	mapVecToData((double*)X.data, d.hand);
	showstate(d,1);

	d.hand.origin = getHandOrigin(d.hand); //move to new position
}

Results and Discussion

Here are my results so far:

It’s not perfect, but it’s a start. Tracking and estimating open hand is pretty good, with some orientation change as well. But when the fingers are closed… that’s where problems start.

Sometimes the joints “hover” over the black area to “land” in a white area so they “fit”, but they should not do that. One easy thing to do to counter this is to measure the distance of the whole bone, and not just the joint.

The model right now doesn’t use all the joints possible, because it is too heavy computationally. Plus the energy does not depend (or change) the angle of the fingers. So this is a very very simple model of a hand…

But, it is a good start! All the other stuff I have seen online is just basic high-curvature points counting and color-based or feature-based segmentation and tracking… My model actually tries to fit an articulate and precise model of a hand to the image.

How did you get such nice blobs?!

You ask. They are beautiful aren’t they… nice and clean, easy for tracking and model fitting. It’s no magic though…
Well, I took part of a project in the Media Lab, called DepthJS, that uses the MS Kinect to control web pages. I wrote the computer-vision part. So all the code is there, you can grab it, I just plugged it into this little project. Basing off this very simple example of using OpenCV2.X and libfreenect.

Wow, this was a longie.. I hope you learned something and got inspired. I got to do a second overview of the project, and I’m inspired. Inspiration all around!

Code is obviously yours for the taking:
https://github.com/royshil/OpenHPE

Please contribute your own views, thoughts, code, rants in the comments and github page.

Enjoy
Roy.