More Than Technical » gui

Combining OpenCV 2.x and Microsoft WPF [w/ code]

Roy — Wed, 21 Jul 2010 21:30:59 +0000

Hi,
Just a quicky about OpenCV and Windows Presentation Framework interoperability. It’s really easy with a simple Managed C++ wrapper. What I’ll show is how to transfer an OpenCV cv::Mat into WPF’s BitmapSource. Let’s see how it’s done.

First some framework. I’ll define a managed C++ (ref) class that holds everything WPF needs to make a BitmapSource:

public ref class OpenCVImageWrapper {
	public:
		int width;
		int height;
		cli::array^ buf;
		int buf_size;
		int row_stride;
		int channels;
	};

That’s it, now let’s build a BitmapSource out of an instance of this class in C#:

OpenCVImageWrapper im = getFromOpenCV();
BitmapSource bs = BitmapSource.Create(
                im.width, im.height, 96, 96,
                System.Windows.Media.PixelFormats.Bgr24, //need to look out here: this is for BGR 24-bits images, you might need Gray8, Gray32Float or one of others...
                null, im.buf, im.row_stride);
imageOnScreen.Source = bs;

(imageOnScreen is an Image object I dragged and dropped into a window in the visual GUI builder)

Now we need a bit more managed C++ wizardry to fill up the wrapper, here it is:

OpenCVImageWrapper^ getFromOpenCV() {
Mat im = imread("someimage.jpg");

OpenCVImageWrapper^ i = gcnew OpenCVImageWrapper();

i->buf = gcnew cli::array(im.rows * im.step);
System::Runtime::InteropServices::Marshal::Copy(System::IntPtr::IntPtr(im.data),i->buf,0,im.rows * im.step);

i->width = im.cols;
i->height = im.rows;
i->row_stride = im.step;
i->channels = im.channels();
i->buf_size = im.rows * im.step;

return i;
}

We’re pretty much done. Basic stuff!

It’s not memory optimized as I’m copying the buffer, but it gets the job done.

Thanks
Roy.

Image Recoloring using Gaussian Mixture Model and Expectation Maximization [OpenCV, w/Code]

Roy — Thu, 24 Jun 2010 15:34:59 +0000

Hi,
I’ll present a quick and simple implementation of image recoloring, in fact more like color transfer between images, using OpenCV in C++ environment. The basis of the algorithm is learning the source color distribution with a GMM using EM, and then applying changes to the target color distribution. It’s fairly easy to implement with OpenCV, as all the “tools” are built in.

I was inspired by Lior Shapira’s work that was presented in Eurographics 09 about image appearance manipulation, and a work about recoloring for the colorblind by Huang et al presented at ICASSP 09. Both works deal with color manipulation using Gaussian Mixture Models.

Let’s see how it’s done!

A little theory

I won’t bore you with the math, but just to get a hang of the idea, what we would like to do is learn how the colors in the source and target images are distributed. Naturally we can assume, like many other things in nature, the colors in a picture have a normal (Gaussian) distribution, but we can go further by saying the distribution might have a few Gaussians describing it. This is called a mixture distribution, and it’s a very handy statistical tool. Mixtures can be estimated using a powerful and ubiquitous tool called Expectation Maximization, which I have previously covered. EM essentially tries to recover the mean (mu) and variance (sigma) of the Gaussians in the mixture, by iteratively checking the current hypothesis against the data until finally converging at an extremum.

Learning the color model

For the learning process we must set up the sample data. So we create a binary mask saying where in the image the model can find the colors to learn.
–images–
Then we scan the mask and for each foreground pixel we add it’s value (here, RGB, but basically can be anything) to the sample data. Finally we train the CvEM object, which contains the GMM parameters.

void TrainGMM(CvEM& source_model, Mat& source, Mat& source_mask) {
		int src_samples_size = countNonZero(source_mask);
		Mat source_samples(src_samples_size,3,CV_32FC1);

		Mat source_32f;
		source_32f = source;

		int sample_count = 0;
		for(int y=0;y(y);  //pointer to pixel data in the row
			uchar* mask_row = source_mask.ptr(y); //pointer to binary mask
			for(int x=0;x 0) {
					source_samples.at(sample_count++,0) = row[x];
				}
			}
		}

		source_model.clear();
		CvEMParams ps(3/* = number of gaussians*/);
		source_model.train(source_samples,Mat(),ps,NULL);
}

What we have are three 3-dimensional (R,G,B) Gaussians, describing the colors in the selected area.

Matching Gaussians

Now we have a couple of GMMs – one for the target and one for the source. The idea is to take a pixel in the target, see how the 3 target Gaussians describe it, and shift it to use the 3 source Gaussians. This will, hopefully, cause its color to change from target to source. But we need to know which Gaussian in the target corresponds to which Gaussian in the source. I made a quick selection algorithm that greedily chooses a Gaussian for each Gaussian. I permutate the order of selection for the greedy algorithm, and pick the best permutation to get closer to the optimal selection.

vector Recoloring::MatchGaussians(CvEM& source_model, CvEM& target_model) {
		int num_g = source_model.get_nclusters();
		Mat sMu(source_model.get_means());
		Mat tMu(target_model.get_means());
		const CvMat** target_covs = target_model.get_covs();
		const CvMat** source_covs = source_model.get_covs();

		double best_dist = std::numeric_limits::max();
		vector best_res(num_g);
		vector prmt(num_g); 

		for(int itr = 0; itr < 10; itr++) {
			for(int i=0;i res(num_g);
			vector taken(num_g);
			for(int sg = 0; sg < num_g; sg++) {
				double min_dist = std::numeric_limits::max();
				int minv = -1;
				for(int tg = 0; tg < num_g; tg++) {
					if(taken[tg]) continue;

					//TODO: can save on re-calculation of pairs - calculate affinity matrix ahead
					//double d = norm(sMu(Range(prmt[sg],prmt[sg]+1),Range(0,3)),	tMu(Range(tg,tg+1),Range(0,3)));

					//symmetric kullback-leibler
					Mat diff = Mat(sMu(Range(prmt[sg],prmt[sg]+1),Range(0,3)) - tMu(Range(tg,tg+1),Range(0,3)));
					Mat d = diff * Mat(Mat(source_covs[prmt[sg]]).inv() + Mat(target_covs[tg]).inv()) * diff.t();
					Scalar tr = trace(Mat(
						Mat(Mat(source_covs[prmt[sg]])*Mat(target_covs[tg])) +
						Mat(Mat(target_covs[tg])*Mat(source_covs[prmt[sg]]).inv()) +
						Mat(Mat::eye(3,3,CV_64FC1)*2)
						));
					double kl_dist = ((double*)d.data)[0] + tr[0];
					if(kl_dist
I used Symmetric Kullback-Leibler for the distance between Gaussians, as suggested by Huang et al.
Applying the color
Now all we have to do is use the method Shapira suggested in his work to transform a pixel’s color, from the Gaussians describing it.

I’m only putting the essence of the code, the rest is in the file.
		Mat pr; Mat samp(1,3,CV_32FC1);
		for(int y=0;y(y);
			uchar* mask_row = target_mask.ptr(y);
			for(int x=0;x 0) {
                                        //take pixel data
					memcpy(samp.data,&(row[x][0]),3*sizeof(float)); 

                                        //Use the GMM to predict how close this pixel is to each gaussian
					float res = target_model.predict(samp,&pr);

					Mat samp_64f; samp.convertTo(samp_64f,CV_64F);

                                        //Move the pixel to the new Gaussians
					//From Shapira09: Xnew = Sum_i { pr(i) * Sigma_source_i * (Sigma_target_i)^-1 * (x - mu_target) + mu_source }
					Mat Xnew(1,3,CV_64FC1,Scalar(0));
					for(int i=0;i
You might notice I skip the part of multiplying by the covariances matrices, as Shapira did. I found it produces better results, but it’s probably caused by a bug.
Results



Code and stuff
Source code is available in SVN repo:
svn checkout http://morethantechnical.googlecode.com/svn/trunk/GMM_Recoloring recoloring

Images from Flickr (Creative Commons):

http://www.flickr.com/photos/wwworks/2956622857/sizes/s/
http://www.flickr.com/photos/violentz/3199292482/sizes/s/
http://www.flickr.com/photos/davidw/164670455/sizes/m/
http://www.flickr.com/photos/djania/252225693/sizes/m/

Now go recolor the world!
Thanks for tuning in..

Roy.

Awesome pictures fusing with a GIMP plugin [w/ code]

Roy — Mon, 21 Sep 2009 12:58:58 +0000

Switching, merging or swapping, call it what you like – it’s a pain to pull off. You need to spend a lot of time tuning the colors, blending the edges and smudging to get a decent result. So I wrote a plugin for the wonderful GIMP program that helps this process. The merge is done using a blending algorithm that blends in the colous from the original image into the pasted image.

I’ll write a little bit about coding GIMP plugins, which is very simple, and some about the algorithm and its sources.

Let’s see how it’s done

Writing a GIMP plugin

As I mentioned, writing a plugin for the GIMP is not very hard. I used the excellent (though outdated) guide for writing a plugin from GIMP.org.
There are 2 kinds of plugins: scripts, and “native” plugins. Script plugins are written in script-fu, the scripting language for GIMP, they are not compiled. On the opposite side, the “native” plugins are written either in C and compiled to executable, or in Python or Perl. I wrote my plugin in C based roughly on the plugin template, and compiled both on Linux and windows.

From the ground up

The first step I did was getting my hands on the pixel data of the selected layer. This is covered pretty nicely in the tutorial.
Once you get a GimpDrawble you just use gimp_pixel_rgn_init, and then gimp_pixel_rgn_get_row to get a whole row of pixel data (interleaved RGB/RGBA).

To access elements in the row of bytes you need to know how many channels are in the interleave.

gint         channels, x1, y1, x2, y2, x_off, y_off, m, n;
GimpPixelRgn rgn_in;
guchar       *row;

gimp_drawable_mask_bounds (drawable->drawable_id,
	                                   &x1, &y1,
	                                   &x2, &y2);
gimp_drawable_offsets(drawable->drawable_id,&x_off,&y_off);
m = y2 - y1; //height of relevant region
n = x2 - x1; //width of relevant region

channels = gimp_drawable_bpp (drawable->drawable_id);
gimp_pixel_rgn_init (&rgn_in,
			 drawable,
			 x1, y1,
			 n, m,
			 FALSE, FALSE);
row = g_new (guchar, channels * n);
for (int y = 0; y < m; y++) {
gimp_pixel_rgn_get_row (&rgn_in, 		row, 		0, y, n);

//manipulate pixels...
}

Working with layers masks is also simple. Masks are GimpDrawables as well.

gint32 mask_id = gimp_layer_get_mask(drawable->drawable_id);
GimpDrawable* mask_drawable = gimp_drawable_get(mask_id);

Progress reporting

During the run of a lengthy operation it’s recommended to notify the user of the progress. The easiest way to do it in a GIMP plugin is use the built-in progress bar. Very easy to use:

gimp_progress_init ("Clone...");
....
gimp_progress_set_text_printf("Create matrix %dx%d",mn,mn);
gimp_progress_update(0.33);
....
gimp_progress_update(1.0);

User inputs

User input for plugins can be easily done using GTK dialogs. The gimp plugin development tutorial is covering this briefly, but its fairly simple for someone who is already fimiliar with GTK. I copied off the tutorial, and changed the spinbutton to a checkbutton for my needs.

One tip: in the tutorial code the gimp_dialog_new function will not accept just an int for the 4th parameter (needs GtkDialogFlags), so I passed in GtkDialogFlags::GTK_DIALOG_MODAL.

Now that we got the bases covered, let’s move on to the algorithm.

The blending algorithm

The algorithm is based on a paper published in 03′ by Microsoft Research. The article presents a way to merge two images by inspecting their gradients, and using this information to “bleed” in colors from the background image into the pasted image. More generally, the colors from the outer image (the background) will seep into the pasted image, and they will do it faster if the pasted image is more smooth. This creates a very nice blending effect.

There’s some mathematical mumbo-jumbo about estimating and discretising the calculation of the gradient, and in the end it boils down to a simple sparse matrix multiplication. I used GMM++ library to do this in C environment. The library compiles from source (only .h files), so you only need to #include it and set the project paths correctly.

How to use the plugin

OK, so all this BS about GIMP, C and math gave me a headache, and we’re here to have some fun after all. So without further ado, let’s see how to use this plugin to do some nice tricks. Originally I used this plugin to merge faces together, but it actually can be used to merge anything.

For example, a polar bear in the desert:

This is how to do it:

get a background picture, preferably one with a uniform color part on which you’ll want to blend in the external object

get an image of an object, preferably surrounded by a uniform color “buffer” so it will blend nicely into the background

if needed, cut out a bounding rectangle of the object with a good margin on all sides, and place it where you want on the background

create a mask, and use lasso to create a tighter mask around the object

select the object layer, and fire the plugin

Some other stuff I did:

Video time…

Source of course

Source is as usual in our Google Code SVN repo.

Executable (win32) is available here.

Thank you for tuning in!
Roy.

Advanced topics in 3D game building [w/ code, video]

Roy — Mon, 27 Jul 2009 12:17:27 +0000

The graphics course I took at TAU really expanded my knowledge of 3D rendering, and specifically using OpenGL to do so. The final task of the course, aside from the exam, was to write a 3D game. We were given 3 choices for types of games: worms-like, xonix-like and lightcycle-like. We chose to write our version of Worms in 3D.

I’ll try to take you through some of the problems we encountered, the decisions we made, and show as much code as possible. I’m not, however, gonna take you through the simple (yet grueling) work of actually showing meshes to the screen or moving them around, these subjects are covered extensively online.

The whole game is implemented in Java using JOGL and SWT for 3D rendering. The code is of course available entirely online.

In the begining there was mesh

Our game is a turn-based shooter, Worms-like. The players are supposed to walk on the face of a world mesh (a 3d structure), and shoot missiles at each other.
When it comes to positioning and aligning 3D objects on top of other 3D objects, I must say I was overwhelmed by the complexity. Its really not that easy to do, but once you get the idea its easy.
So you have an arbitrary 3D object, and you want another object to “walk” over it. Few things you need to know:

the normal of the mesh at any given point, so your player can have an “up” direction (away from the mesh)
the global orientation of your player object, so when you align it up it will actually face up

Now, to align the object you have to make it be oriented upwards from the mesh and point in the forward direction. So before drawing the player object make the model-view matrix face the right direction. For this we ported some CPP code that does exactly that:

public static void multLookAt (/*float eyex, float eyey, float eyez,
            float atx, float aty, float atz,
            float upx, float upy, float upz,*/
			Vector3D origin,
			Vector3D forward,
			Vector3D up,
            GL gl)
	{
		float m[] = new float[16];

		// Compute our new look at vector, which will be
		//   the new negative Z axis of our transformed object.
		forward = forward.getNormalized();

		// Cross product of the new look at vector and the current
		//   up vector will produce a vector which is the new
		//   positive X axis of our transformed object.
		Vector3D xaxis = forward.crossProduct(up).getNormalized();
		m[0] = xaxis.x;
		m[1] = xaxis.y;
		m[2] = xaxis.z;

		// Calculate the new up vector, which will be the
		//   positive Y axis of our transformed object. Note
		//   that it will lie in the same plane as the new
		//   look at vector and the old up vector.
		up = xaxis.crossProduct(forward);
		m[4] = up.x;
		m[5] = up.y;
		m[6] = up.z;

		// Account for the fact that the geometry will be defined to
		//   point along the negative Z axis.
		forward = forward.multiply(-1f);
		m[8] = forward.x;
		m[9] = forward.y;
		m[10] = forward.z;

		// Fill out the rest of the 4x4 matrix
		m[3] = 0.f;     // xaxis is m[0..2]
		m[7] = 0.f;     // up is m[4..6]
		m[11] = 0.f;    // -at is m[8..10]
		m[12] = origin.x;
		m[13] = origin.y;
		m[14] = origin.z;
		m[15] = 1.f;

		// Multiply onto current matrix stack.
		gl.glMultMatrixf(FloatBuffer.wrap(m));
	}

Walk the mesh

Another key point in our game is making the player character walk smoothly on the mesh. The instructions for this exercise were that we have to make the player either walk from vertex to vertex, from edge to edge or the most complicated option – an arbitrary point on the mesh to another. We chose to make the player walk from edge to edge.
Say that the player object is on an edge. To make it walk to another edge we needed to know what are the possible edges that it could reach. For that we created a half-edge structure of the world mesh. Then when we query the structure we can get in constant time the neighboring edges to the player (typically the one-ring of both vertices of the current edge he’s standing on). We only have to choose from the list of edges which edge the player should go to, and where on that edge he will “land”. This is done by choosing the edge with the closest point to the approximation of the players next location.
Let me explain:

The player has a certain speed, meaning it can cover a certain distance in a given time.
The approximate next location of the player is the distance it should cover in minimal time. We took a constant value.
Once you have a approximate location, find the edge (and a point on that edge) that best conforms with this location

In the image you see the player object (snail), the ring of edges around the player that we check (green), the approximate next location (yellow), and the calculated next location (purple).

To find a best location for the object we calculated the distance between the player and the approximate. We ported another piece of C code to measure distance between segments to do that:

/**
	 * find the distance between 2 line segments
	 * @param S1P1 1st point of segment 1
	 * @param S1P0 2nd pt of seg 1
	 * @param S2P1 1st pt of seg 2
	 * @param S2P0 2nd pt of seg 2
	 * @return
	 */
public static Vector3D dist3D_Segment_to_Segment( Vector3D S1P1, Vector3D S1P0, Vector3D S2P1, Vector3D S2P0)
	{
	    Vector3D   u = S1P1.minus(S1P0);
	    Vector3D   v = S2P1.minus(S2P0);
	    Vector3D   w = S1P0.minus(S2P0);
	    float    a = u.innerProduct(u);        // always >= 0
	    float    b = u.innerProduct(v);
	    float    c = v.innerProduct(v);        // always >= 0
	    float    d = u.innerProduct(w);
	    float    e = v.innerProduct(w);
	    float    D = a*c - b*b;       // always >= 0
	    float    sc, sN, sD = D;      // sc = sN / sD, default sD = D >= 0

	    // compute the line parameters of the two closest points
	        sN = (b*e - c*d);
	        if (sN < 0f) {       // sc < 0 => the s=0 edge is visible
	            return null;
	        }
	        else if (sN > sD) {  // sc > 1 => the s=1 edge is visible
	            return null;
	        }

	    // finally do the division to get sc
	    sc = (Math.abs(sN) < Vector3D.EPSILON ? 0f : sN / sD);

	    return S1P0.add(u.multiply(sc)); // return the intersection point
	}

We actually check what is the intersection point of the approximate edge with the line that goes from the player and in the forward direction. This is the point the player should be if it walked from his current position to that edge.

Orchestrating:

	/**
	 * find the location where the player should be in the next frame
	 * @param distanceToMoveInThisInterval the distance the player object should (try to) move
	 * @return
	 */
	public EdgeAndIntersectionPointAndDistance findNextLocation(float distanceToMoveInThisInterval) {
		//find all adjacent edges using half-edge struct
		Edge currentEdge = model.getCurrentEdge();
		Set allEdges = currentEdge.b.get2RingOfEdges();
		allEdges.addAll(currentEdge.a.get2RingOfEdges());

//A point that lies a little bit along the line from the player and in the forward direction
		Vector3D locationAndSome = location.add(dirForward.multiply(0.05f));

		//find the edge that the player direction is intersecting
		allEdges.remove(currentEdge);
		ArrayList intersectingEdges = new ArrayList();
		for (Edge e : allEdges) {
			//skip the opposite edge too.
			if((e.a == currentEdge.a && e.b == currentEdge.b) ||
					(e.b == currentEdge.a && e.a == currentEdge.b))
				continue;

			Vector3D intPt = Utils.dist3D_Segment_to_Segment(e.a.getVector3D(), e.b.getVector3D(), model.getLocation(), locationAndSome);
			if(intPt != null) {
				float dist = intPt.minus(locationAndSome).getNorm();
				intersectingEdges.add(
					new EdgeAndIntersectionPointAndDistance(e,dist,intPt)
					);
			}
		}

		EdgeAndIntersectionPointAndDistance bestEdge = null;
		float bestDistance = Float.MAX_VALUE;
		for (EdgeAndIntersectionPointAndDistance ead : intersectingEdges) {
			float absD = ead.distance;
			if(absD < bestDistance) {
				bestDistance = absD;
				bestEdge = ead;
			}
		}

		//find the location on that edge where the player should be
		return bestEdge;
	}

Of course when the player gets to his next location, he needs to get the correct up-direction and forward direction. This can be taken from the next edge’s vertices that hold normals.

There are some additional issues to traversing the mesh:

Avoiding hitting other players. Done by not allowing 2 player to inhabit the same edge.
Collecting bonuses that are on the ground by checking if the player had walked over (or close to) an edge that has a bonus on it
Up keeping the half-edge structure by updating the edges if they have players/bonuses on them

Make a dent in the world

One more interesting issue is the missiles impact on the world. Naturally, we’d like the missiles that hit the ground to make a dent in the world. Our solution was to take the 1-ring of the hit vertex, and lower all the vertices on the ring in the opposite direction to their normals.

	public static void makeADentInTheMesh(Vertex vtx, float amount, IWorldMeshHandler worldMeshHandler) {
		//Make a dent in the mesh...
		Set es = vtx.get2RingOfEdges();
		for (Edge ed : es) {
			Vector3D v = new Vector3D(ed.a.x,ed.a.y,ed.a.z);
			v = v.minus(ed.a.getNormal().multiply(amount));
			ed.a.x = v.x; ed.a.y = v.y; ed.a.z = v.z;
			v = new Vector3D(ed.b.x,ed.b.y,ed.b.z);
			v = v.minus(ed.b.getNormal().multiply(amount));
			ed.b.x = v.x; ed.b.y = v.y; ed.b.z = v.z;

			//If there's stuff on the edge, move it accordingly
			if(ed.playerOnEdge != null) {
				ed.playerOnEdge.getModel().fixLocation();
			}
			if(ed.packageOnEdge != null) {
				Utils.fixObjLocationToEdge(ed.packageOnEdge.getModel(), ed);
			}
			if(ed.treeOnEdge != null) {
				Utils.fixObjLocationToEdge(ed.treeOnEdge.getModel(), ed);
			}
		}

		//Recalculate normals - the positions have changed, creating new "up" directions
		for (Edge ed : es) {
			ed.a.calcNormal();
			ed.b.calcNormal();
		}

		//reset the display list of the world mesh since the vertices and faces have changed
		worldMeshHandler.getWorld().getRenderer().setResetList();
	}

if pigs (missiles) could fly…

So far I’ve covered mesh oriented movement. Missiles, however, are not mesh-bound – they fly around “freely” above the mesh. To imitate gravity and a “steep course of flight” for the missiles, we use Bezier curves of either 3 or 4 keypoints.
To calc a point on the curve all you need to know is the current time of flight.

	protected Vector3D[] mBezierMultiplyV0_V3 = null;

	protected Vector3D V0;
	protected Vector3D V1;
	protected Vector3D V2;
	protected Vector3D V3;

	protected void calcExpectedFlyTime() {
		float distToMove = V3.minus(V0).getNorm(); //total distance to cover
		expectedFlyTime = distToMove / flySpeed;  //approximate time of flight
	}

	protected Vector3D getCurrentCurveLocation(float u) {
		Vector3D out = null;
		out = mBezierMultiplyV0_V3[0].multiply(u * u * u);
		out = out.add(mBezierMultiplyV0_V3[1].multiply(u * u));
		out = out.add(mBezierMultiplyV0_V3[2].multiply(u));
		out = out.add(mBezierMultiplyV0_V3[3]);

		return out;
	}

	private void advanceMissile() {
		float u = timeFromLaunch / expectedFlyTime;
		Vector3D newLocation = this.getCurrentCurveLocation(u);
		this.location = newLocation;

		//calc the change of angle
		this.dirForward = newLocation.minus(this.prevLocation).getNormalized();
		this.dirUp = ((this.dirForward).crossProduct(this.dirLeft)).getNormalized(); //cross-product of left and forward is the new up direction

		this.prevLocation = newLocation;
	}

Get ready for impact!

I’ve talked about what happens when a missile hits, and about the missile’s course, but how do we know when the missile hits the ground?
We have implemented this using a KD-tree over all the vertices in the world mesh, to check what is the closest vertex to the missile. When the missile gets close enough we check the dot product of the missile’s location and the normal of the surface – when the sign flips the missile hit the ground.

	protected void checkForHitWithMesh() {
		Vector3D tipLocation = location.add(dirForward.multiply(distToTip));
		double[] key = {tipLocation.x, tipLocation.y, tipLocation.z};

		//check with KD Tree to get the nearest vertex
		Vertex closestVetrex = null;
		try {
			closestVetrex = VertexKdTree.getVertexKdTree().nearest(key);
		} catch (KeySizeException e) {
			e.printStackTrace();
			return;
		}

		//check if missile is lost in space...
		Vector3D closestVertLocationWorld = aimer.getTransformToMeshLocation().transform(closestVetrex.getVector3D());
		if (tipLocation.minus(closestVertLocationWorld).getNorm() > 1) {
			projectileHandler.setCurrentProjectile(null);
			flightFinished = true;
			objNModeHandler.nextMode();
			return;
		}

		//Check if missile is inside ground
		if ((closestVetrex.getNormal()).innerProduct(tipLocation.minus(closestVertLocationWorld)) < 0) {
			flightFinished = true;
			hitMeshAtVertex(closestVetrex);
			projectileHandler.setCurrentProjectile(null);
		}
	}

Particle Shmarticle

Particle systems are a decent way to simulate smoke, fire, water splashes and anything “particley”. We ripped some C code from the net (I can’t remember where, but it was GPLed), again, and ported it to Java.
The only problem is that OpenGL can’t really display particles, you need something that has some kind of surface, or a line. Other options are drawing GL_POINTs, or GLU spheres, but both options either don’t look pleasing or are very costly in terms of performance. So we used GL_TRIANGLE_STRIP to draw small rectangles of random sizes as the particles.

	public void drawParticles(GL gl) {
		gl.glEnable(GL.GL_BLEND);
		gl.glBlendFunc(GL.GL_SRC_ALPHA, GL.GL_ONE_MINUS_SRC_ALPHA);
		gl.glDisable(GL.GL_LIGHTING);

		for (int i = 0; i < MAX_PARTICLES; i++) {
			// Each particle is handled differently depending on whether it's
			// alive or not.
			Particle particle = m_aParticles[i];
			if (particle.isAlive()) {
				// This particular particle is alive.
				handleLiveParticle(gl, particle);
			} else {
				// This particular particle is dead.
				handleDeadParticle(gl, particle);
			}
		}
		gl.glDisable(GL.GL_BLEND);
		gl.glEnable(GL.GL_LIGHTING);
	}

	private void handleLiveParticle(GL gl, Particle particle) {
		// The current location of the particle; Need to account for the
		// zoom
		// distance so user can zoom in and out the particles.
		float x = particle.getXLocation();
		float y = particle.getYLocation();
		float z = particle.getZLocation();

		// Set the color to draw this particle. The particle's life value
		// will act as the alpha.
		gl.glColor4f(particle.getRed(), particle.getGreen(),
				particle.getBlue(), particle.getLife());
		// Draw the particle using triangle strips.
		gl.glBegin(GL.GL_TRIANGLE_STRIP);
		// Map the texture and create the vertices for the particle.
		float pSize = (float) Math.random() * particle.getLife();
		float r = (float) Math.random() - pSize;
		gl.glVertex3f(x + pSize, y + pSize, z + r);
		gl.glVertex3f(x - pSize, y + pSize, z + r);
		gl.glVertex3f(x + pSize, y - pSize, z - r);
		gl.glVertex3f(x - pSize, y - pSize, z - r);
		gl.glEnd();

		// Update the particles' properties.
		updateParticle(particle);
	}

Note that we make sure the “particles” are blended into the buffer, as particles’ color is fading as they near death.

Sum up

OK, I’ve tried to share most of the interesting points in the making of the game. The code is downloadable from the Google code SVN repo. And here’s a short video explaining some aspects of playing the game:

Thanks for tuning in!
Roy.

Combining Java’s BufferedImage and OpenCV’s IplImage

Roy — Thu, 14 May 2009 10:51:44 +0000

I recently did a small project combining a Java web service with a OpenCV processing. I tried to transfer the picture from Java environment (as BufferedImage) to OpenCV (IplImage) as seamlessly as possible. This proved a but tricky, especially the Java part where you need to create your own buffer for the image, but it worked out nicely.

Let me show you how I did it

First up was creating a 3-component RGB BufferedImage in Java, with a custom memory buffer that will fit nicely with IplImages:


private BufferedImage createOpenCVCompatibleBufferedImage(int w, int h)
{
ComponentColorModel cm = new ComponentColorModel(
ColorSpace.getInstance(ColorSpace.CS_sRGB),
false,  //no alpha channel
false,  //not premultiplied
ColorModel.OPAQUE,
DataBuffer.TYPE_BYTE); //important - data in the buffer is saved by the byte

SampleModel sm = cm.createCompatibleSampleModel(w, h);
DataBufferByte db = new DataBufferByte(w*h*3); //3 channels buffer
WritableRaster r = WritableRaster.createWritableRaster(sm, db, new Point(0,0));
BufferedImage bm = new BufferedImage(cm,r,false,null);
return bm;
}

This took some trial and error, but I ended up with a working thing.

Now, the C++/OpenCV side of things – a JNI function to read the buffer and do some OpenCV stuff:


JNIEXPORT jint JNICALL Java_test_OpenCVShowImage
(JNIEnv *env, jobject jo, jbyteArray pic, jint w, jint h, jint bpp, jint bpr) {
IplImage* img;
jint len;
unsigned char* result;
int n1;

img = cvCreateImageHeader(cvSize(w,h),8,bpp/8); //create the "shell"

len = (*env)->GetArrayLength(env, pic);
result = (unsigned char *)malloc(len + 1);
if (result == 0) {
fatal_error("out of memory");
(*env)->DeleteLocalRef(env, pic);
return 0;
}
(*env)->GetByteArrayRegion(env, pic, 0, len,(jbyte *)result);

cvSetData(img,result,bpr);    //set the buffer

cvNamedWindow("window");
cvShowImage("window",img);
cvWaitKey(0);
cvDestroyWindow("window");

free(result);
cvReleaseImage(&img);
return 1;
}

One last thing -the Java call to the JNI function:

BufferedImage tmp = ImageIO.read(new File("bird.png"));

BufferedImage bi1 = createOpenCVCompatibleBufferedImage(tmp.getWidth(), tmp.getHeight());

//paint the image with a little transform...
Graphics2D biDestG2D = bi1.createGraphics();
biDestG2D.setColor(Color.white);
biDestG2D.fillRect(0, 0, w, h);
AffineTransform transform = AffineTransform.getShearInstance(0.2, 0.1);
transform.scale(0.5, 0.5);
biDestG2D.drawImage(tmp,transform,null);
biDestG2D.dispose();

byte[] bytes = ((DataBufferByte)bi1.getRaster().getDataBuffer()).getData();

OpenCVShowImage(bytes, w, h, 24, w*3);

That’s all folks!

Qt & OpenCV combined for face detecting QWidgets

Roy — Thu, 05 Mar 2009 12:19:37 +0000

As my search for the best platform to roll-out my new face detection concept continues, I decided to give ol’ Qt framework a go.

I like Qt. It’s cross-platform, a clear a nice API, straightforward, and remindes me somewhat of Apple’s Cocoa.

My intention is to get some serious face detection going on mobile devices. So that means either the iPhone, which so far did a crummy job performance-wise, or some other mobile device, preferably linux-based.
This led me to the decision to go with Qt. I believe you can get it to work on any linux-ish platform (limo, moblin, android), and since Nokia baught Trolltech – it’s gonna work on Nokia phones soon, awesome!

Lets get to the details, shall we?

First thing’s first: face detection.

I ripped OpenCV’s facedetect.c sample and extracted only the detect_and_draw() function. Originally the function detects the faces and draws a circle over them, but I needed only the face detection and the result bounding rectangle. So in the end I was left with this:


CvRect detect_and_draw( IplImage* img, CvMemStorage* storage, CvHaarClassifierCascade* cascade )
{
IplImage *gray, *small_img;
int i = 0;

gray = cvCreateImage( cvSize(img->width,img->height), 8, 1 );
small_img = cvCreateImage( cvSize( cvRound (img->width/scale),
cvRound (img->height/scale)), 8, 1 );

cvCvtColor( img, gray, CV_RGB2GRAY );
cvResize( gray, small_img, CV_INTER_LINEAR );
cvEqualizeHist( small_img, small_img );
cvClearMemStorage( storage );

CvRect* r = NULL;

if( cascade )
{
double t = (double)cvGetTickCount();
CvSeq* faces = cvHaarDetectObjects( small_img, cascade, storage,
1.1, 2, 0
|CV_HAAR_FIND_BIGGEST_OBJECT
//|CV_HAAR_DO_ROUGH_SEARCH
//|CV_HAAR_DO_CANNY_PRUNING
//|CV_HAAR_SCALE_IMAGE
,
cvSize(30, 30) );
t = (double)cvGetTickCount() - t;

printf( "detection time = %gms\n", t/((double)cvGetTickFrequency()*1000.) );

r = (CvRect*)cvGetSeqElem( faces, i );

cvReleaseImage( &amp;amp;amp;amp;gray );
cvReleaseImage( &amp;amp;amp;amp;small_img );

if(r) {
return cvRect(r->x,r->y,r->width,r->height);
} else {
return cvRect(-1,-1,0,0);
}

}

This can go anywhere in the code base, as it’s totally independant (as long as you train the cascade and allocate a MemStorage).Note that I am assuming only one face in the input image, and also that it will be the largest detected object. This bring my benchmark to about 25ms per frame, using the original general detection approach of facedetect.c benchmarked at about 160ms per frame.

OK, done with pure OpenCV, on to Qt.

I subclassed a QWidget, who’s sole purpose is to show the input video with the detected face. For starters, I needed to have a QImage and an IplImage instances as members, they can also share the same buffer (how awesome is that..). I also need a CvCapture, CvMemStorage and a CvHaarCalssifierCascade:


class FaceRecognizer : public QWidget
{
Q_OBJECT

public:
FaceRecognizer(QWidget *parent = 0);
~FaceRecognizer();

private:
Ui::FaceRecognizerClass ui;

QImage m_i;

QRect faceLoc;

CvMemStorage* storage;
CvHaarClassifierCascade* cascade;
CvCapture* capture;
IplImage* m_opencvimg;

QTimer* m_timer;

void paintEvent(QPaintEvent* e);

public slots:
void queryFrame();
};

You can see that I’m gonna use a QTimer to query the frames from the CvCapture and also I override paintEvent to paint the frame onto the canvas. In fact in my QWidget I have a QFrame, that the image will painted over it. UI was generated in Qt Designer.

First, some initialization in the constructor:


FaceRecognizer::FaceRecognizer(QWidget *parent)
: QWidget(parent)
{
ui.setupUi(this);

capture = cvCaptureFromAVI( "/home/user/Desktop/video.avi" );
//grab one frame to get width and height

IplImage* frame = cvQueryFrame( capture );

m_i = QImage(QSize(frame->width,frame->height),QImage::Format_RGB888);
ui.frame->setMinimumSize(m_i.width(),m_i.height());
ui.frame->setMaximumSize(ui.frame->minimumSize());
//create only the header, as the data buffer is shared, and was allocated by QImage

m_opencvimg = cvCreateImageHeader(cvSize(m_i.width(),m_i.height()),8,3);
m_opencvimg->imageData = (char*)m_i.bits(); // share buffers

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );

//images from cvQueryFrame come in BGR form and not what Qt expects - RGB

//and since the buffers are shared - format should be consistent
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

//we need memstorage and a cascade
storage = cvCreateMemStorage(0);
cascade = (CvHaarClassifierCascade*)cvLoad( CASCADE_NAME, 0, 0, 0 );

//set timer for 50ms intervals

m_timer = new QTimer(this);
connect(m_timer,SIGNAL(timeout()),this,SLOT(queryFrame()));
m_timer->start(50);
}

And now, querying the frame: query CvCapture, convert BGR to RGB, detect faces and update faceLoc QRect.


void FaceRecognizer::queryFrame() {
IplImage* frame = cvQueryFrame( capture );

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

CvRect r = detect_and_draw(m_opencvimg,storage,cascade);
faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));

this->update();
}

Finally - painting, which is easy:

void FaceRecognizer::paintEvent(QPaintEvent* e) {
QPainter painter(this);

painter.drawImage(QPoint(ui.frame->x(),ui.frame->y()),m_i);

if(faceLoc.x() > 0 &amp;amp;amp;&amp;amp;amp; faceLoc.y() > 0) {
painter.setBrush(Qt::NoBrush);
painter.setPen(QColor(255,0,0));
painter.drawRect(QRect(faceLoc.x()+ui.frame->x(),faceLoc.y()+ui.frame->y(),faceLoc.width(),faceLoc.height()));
}
}

Looks like it’s all done… Here’s a video:

Showing video with Qt toolbox and ffmpeg libraries

Roy — Sun, 02 Nov 2008 08:12:00 +0000

I recently had to build a demo client that shows short video messages for Ubuntu environment.
After checking out GTK+ I decided to go with the more natively OOP Qt toolbox (GTKmm didn’t look right to me), and I think i made the right choice.

So anyway, I have my video files encoded in some unknown format and I need my program to show them in a some widget. I went around looking for an exiting example, but i couldn’t find anything concrete, except for a good tip here that led me here for an example of using ffmpeg’s libavformat and libavcodec, but no end-to-end example including the Qt code.

The ffmpeg example was simple enough to just copy-paste into my project, but the whole painting over the widget’s canvas was not covered. Turns out painting video is not as simple as overriding paintEvent()…

Firstly, you need a separate thread for grabbing frames from the video file, because you won’t let the GUI event thread do that.
That makes sense, but when the frame-grabbing thread (I called VideoThread) actually grabbed a frame and inserted it somewhere in the memory, I needed to tell the GUI thread to take that buffered pixels and paint them over the widget’s canvas.

This is the moment where I praise Qt’s excellent Signals/Slots mechanism. So I’ll have my VideoThread emit a signal notifying some external entity that a new frame is in the buffer.
Here’s a little code:

void VideoThread::run() {
/*
... Initialize libavformat &amp; libavcodec data structures.
You can see it in the example i referred to before
*/
// Open video file
if(av_open_input_file(&amp;pFormatCtx,
"lala.avi",
NULL, 0, NULL)!=0)
return -1; // Couldn't open file

// Retrieve stream information
if(av_find_stream_info(pFormatCtx)&lt;0)
return -1; // Couldn't find stream information

// Find the first video stream ...

// Get a pointer to the codec context for the video
// stream...

// Find the decoder for the video stream...

// Open codec...

// Allocate video frame
pFrame=avcodec_alloc_frame();

// Allocate an AVFrame structure
pFrameRGB=avcodec_alloc_frame();
if(pFrameRGB==NULL)
return -1;

int dst_fmt = PIX_FMT_RGB24;
int dst_w = 160;
int dst_h = 120;

// Determine required buffer size and allocate buffer
numBytes = avpicture_get_size(dst_fmt, dst_w, dst_h);
buffer = new uint8_t[numBytes + 64];

//put a PPM header on the buffer
int headerlen = sprintf((char *) buffer,
"P6n%d %dn255n",
dst_w, dst_h);

_v-&gt;buf = (uchar*)buffer;
_v-&gt;len = avpicture_get_size(dst_fmt,dst_w,dst_h) +
headerlen;

// Assign appropriate parts of buffer to image planes
// in pFrameRGB...

// I use libswscale to scale the frames to the required
// size.
// Setup the scaling context:
SwsContext *img_convert_ctx;
img_convert_ctx = sws_getContext(
pCodecCtx-&gt;width, pCodecCtx-&gt;height,
pCodecCtx-&gt;pix_fmt,
dst_w, dst_h, dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);

// Read frames and notify
i=0;
while(av_read_frame(pFormatCtx, &amp;packet)&gt;=0)
{
// Is this a packet from the video stream?
if(packet.stream_index==videoStream)
{
// Decode video frame
avcodec_decode_video(pCodecCtx,
pFrame,
&amp;frameFinished,
packet.data,
packet.size);

// Did we get a video frame?
if(frameFinished)
{
// Convert the image to RGB
sws_scale(img_convert_ctx,
pFrame-&gt;data,
pFrame-&gt;linesize,
0,
pCodecCtx-&gt;height,
pFrameRGB-&gt;data,
pFrameRGB-&gt;linesize);

emit frameReady();

//My video is 5FPS so sleep for 200ms.
this-&gt;msleep(200);
}
}

// Free the packet that was allocated by
// av_read_frame
av_free_packet(&amp;packet);
}

// Free the RGB image
delete [] buffer;
av_free(pFrameRGB);

// Free the YUV frame
av_free(pFrame);

// Close the codec...

// Close the video file...
} //end VideoThread::run

Ok so I have a frame-grabber that emits a frameReady signal everytime the buffer is full and ready for painting.

A couple of things to notice:

I convert the image format to PIX_FMT_RGB24 (avcodec.h), which is required by Qt’s QImage::fromData() method.
I scale the image using ffmpeg’s libswscale. All conversion/scaling methods inside libavcodev are deprecated now.
But it’s fairly simple, here‘s a good example. Just remember you need a sws_getContext and then sws_scale.
I totally disregard actual frame rate here, I just sleep for 200ms because i know my file is 5FPS. For a (far-) more sophisticated way to get the FPS, very important if this is not a constant frame-rate video, you can find here.
I don’t cover audio in this example, although the mechanism to extract it from the file exists… you just need to grabe the audio stream’s frame. For playing audio you also need some Qt-external library. In a different project I used SDL very easily, here‘s an example online.

Now, for painting over the widget.
This is fairly easy:

void VideoWidget::paintEvent(QPaintEvent * e) {
QPainter painter(this);
if(buf) {
QImage i = QImage::fromData(buf,len,"PPM");
painter.drawImage(QPoint(0,0),i);
}
}

Two things to note:

The widget needs to be given the pointer to the video frame buffer (buf).
The frame buffer needs to be in a PPM format. That means it needs to get a PPM header, which looks something like this: “P6n320 240n255n”, and then all the pixels in 3-byte per-pixel format (RGB24). You can see that i take care of that in the previous code block.

Finally we need to orchestrate this whole mess.
So in my GUI-screen class I do:

....
vt = new VideoThread();
connect(vt,SIGNAL(frameReady()),this,SLOT(updateVideoWidget()));
vt-&gt;start();
....

And:

void playMessage::updateVideoWidget() {
videoWidget-&gt;repaint(); //or update().
}

This will make the widget repaint on each frame ready.

Note:

In this example I don’t take care of multi-threading issues. Since the GUI and the ffmpeg decoder threads share a memory buffer, I should probably have a mutex to protect it. It’s a classic producer-consumer problem.
Performance wise, Qt’s paint mechanism is by far the worst way to go when displaying video… but it’s great for a quick-and-dirty solution (I only needed 5fps). A more performance favorable solution will probably be using an overlay block and frame-serving with SDL.

Enjoy!
Roy.