Jun 28 2009

Augmented Reality with NyARToolkit, OpenCV & OpenGL

Published by at 2:33 pm under 3d,graphics,opengl,programming,video

arHi

I have been playing around with NyARToolkit's CPP implementation in the last week, and I got some nice results. I tried to keep it as "casual" as I could and not get into the crevices of every library, instead, I wanted to get results and fast.

First, NyARToolkit is a derivative of the wonderful ARToolkit by the talented people @ HIT Lab NZ & HIT Lab Uni of Washington. NyARToolkit however was ported to many other different platforms, like Java, C# and even Flash (Papervision3D?), and in the process making it object oriented, instead of ARToolkit procedural approach. NyARToolkit have made a great job, so I decided to build from there.

NyART don't provide any video capturing, and no 3D rendering in their CPP implementation (they do in the other ports), so I set out to build it on my own. OpenCV is like a second language to me, so I decided to take its video grabbing mechanism wrapper for Win32. For 3D rendering I used the straightforward GLUT library which does an excellent job ridding the programmer from all the Win#@$#@ API mumbo-jumbo-CreateWindowEx crap.

So let's dive in....

Couple of steps needed to be done:

  1. [Preprocessing - calibrating camera...]
  2. Initialize stuff & grab video.
  3. Use NyART to detect marker and get exrinsic camera parameters.
  4. "Calibrate" 3D world to fit camera.
  5. Draw 3D scene on-top of video input.

Initialization: easy...

I used cvCapture & cvGrabFrame, this is extremely easy. However, the frames in Win environment are always grabbed up side down. So we need to take care of image->origin.

capture = cvCaptureFromCAM('0');
cvNamedWindow("input");

	frame = cvQueryFrame( capture );
	if( frame ) {
		if(ra == NULL) {
			ra = new NyARRgbRaster_BGRA(frame->width, frame->height);
			ap.changeScreenSize(frame->width, frame->height);
			ar = new NyARSingleDetectMarker(ap, code, 80.0);
			code=NULL;
			ar->setContinueMode(false);

			arglCameraFrustumRH(ap,1.0,100.0,camera_proj);
		}

		if( !image )
		{
			CvSize s = cvGetSize(frame);
			image = cvCreateImage(s , 8, 3 );
			image->origin = frame->origin;
			bgra_img = cvCreateImage(s,8,4);
			bgra_img->origin = frame->origin;
			gray = cvCreateImage(s,8,1);
			gray->origin = frame->origin;
			flipped = cvCreateImage(s,8,3);
			ra->setBuffer((NyARToolkitCPP::NyAR_BYTE_t*)bgra_img->imageData);
		}
		win_w = frame->width;
		win_h = frame->height;
	}

I use 4 IplImages:

  1. frame buffer (3 channels),
  2. flipped (3 channels), which is the properly aligned frame to be used as the background for the OpenGL scene
  3. grayscale (1 channel), which provides beter results for marker detecting,
  4. And - bgra image (4 channels), to align with NyART's BGRA raster

Note that I can only initialize NyARRgbRaster_BGRA & NyARSingleDetectMarker after grabbing the first frame, as I need the width & height of the video stream. You can bypass this by querying the CvCapture object directly.

Also, note the shared buffer between the RGBA raster and bgra_image. I say that shared buffers are very helpful, sparing all the useless bit shuffling... you should try and use them if you can.

Another thing to notice is the loading of the camera matrix by using ARToolkit's arglCameraFrustumRH function. NyART also has this function, but I used the more native version by ARTk as it uses float[] and not float*, which is easier to handle and debug.

Finally, some OpenGL initialization code:

	mesh = loadFromFile("c:/downloads/apple.obj");

	glShadeModel(GL_FLAT);
	glClearColor(0.0f, 0.0f, 0.0f, 0.5f);
	glClearDepth(1.0f);
	glDepthFunc(GL_LEQUAL);
	glHint(GL_PERSPECTIVE_CORRECTION_HINT, GL_NICEST);

This came flying out of NeHe's number 2 lesson for GLUT. Except for loadFromFile, which is a small framework I wrote to load .obj files - I'll speak of it later as well.

OK, done with initialization, on to...

Detecting the marker:

Well, NyARTk are taking care of all that:

	bool doMult = false;

        frame = cvQueryFrame( capture );
		if( frame ) {
			cvCopy( frame, image);
			cvCvtColor(image,gray,CV_RGB2GRAY);
			cvCvtColor(gray,bgra_img,CV_GRAY2BGRA);	
			cvCvtColor(frame,flipped,CV_RGB2BGR);
			cvFlip(flipped);

			cvShowImage("input",gray);

			if(ar->detectMarkerLite(*ra, 100)) {
				ar->getTransmationMatrix(result_mat);
			//	printf("Marker confidence\n cf=%f,direction=%d\n",ar->getConfidence(),ar->getDirection());
				//printf("Transform Matrix\n");
				//printf(
				//	"% 4.8f,% 4.8f,% 4.8f,% 4.8f\n"
				//	"% 4.8f,% 4.8f,% 4.8f,% 4.8f\n"
				//	"% 4.8f,% 4.8f,% 4.8f,% 4.8f\n",
				//	result_mat.m00,result_mat.m01,result_mat.m02,result_mat.m03,
				//	result_mat.m10,result_mat.m11,result_mat.m12,result_mat.m13,
				//	result_mat.m20,result_mat.m21,result_mat.m22,result_mat.m23);
				doMult = true;
			}
		}

This is pretty straightforward... I grab the frame, convert it to grayscale and then to BGRA (and via the shared buffer into the NyART BGRA raster), and use detectMarkerLite on the raster.

As you remember, the marker detection is giving us the extrinsic camera properties, which means location in the world and rotation. Now this has to be input into OpenGL scene rendering to match the virtual camera with the real life camera.

Draw the feed from the camera as the background

But first, we need to draw the video frame as the background for the 3D scene:

	glMatrixMode(GL_PROJECTION);
	glPushMatrix();
	glLoadIdentity();
	gluOrtho2D(0.0,win_w, 0.0,win_h);
	glMatrixMode(GL_MODELVIEW);
	glPushMatrix();
	glLoadIdentity();

	glDrawPixels(win_w,win_h,GL_RGB,GL_UNSIGNED_BYTE,flipped->imageData);

	glMatrixMode(GL_PROJECTION);
	glPopMatrix();
	glMatrixMode(GL_MODELVIEW);
	glPopMatrix();

	glEnable(GL_DEPTH_TEST);
    glDepthFunc(GL_LEQUAL);

This is done by going to orthographic mode using gluOrtho2D, and drawing the pixels using glDrawPixels with flipped's buffer.

Set the OpenGL virtual camera to align with the real life camera

Next, we need to alter the camera to match the scene:

	if(doMult) {
		glMatrixMode(GL_PROJECTION);
		glLoadMatrixd(camera_proj);
		glMatrixMode(GL_MODELVIEW);
		glLoadIdentity();
		float m[16] = {0.0f}; m[0] = 1.0f; m[5] = 1.0f; m[10] = 1.0f; m[15] = 1.0f;
		toCameraViewRH(result_mat,m);
		glLoadMatrixf(m);
	} else {
		glMatrixMode(GL_MODELVIEW);
		glLoadIdentity();
	}

The result from NyARTk is in the global result_mat, and the function toCameraViewRH only takes it and puts in the float[] m matrix.
Note that before that I load the camera projection matrix, this can probably be done in init() just the same as this matrix doesn't change.

Some 3D rendering...

Then I do some OpenGL drawing...

	glPushMatrix();
	glScalef(.02f,.02f,.02f);
	glRotatef(xrot,0.0f,0.0f,1.0f);
	xrot+=5.0f;
	glRotated(90,1,0,0);

	glutSolidCube(50.0);

	glPopMatrix();

A simple rotating GLU cube...

It's about time for a video

Framework to load .obj files

The .obj format is a very simple, text-based format of saving 3D models. You can see the spec here. The implementation is included in the code, in objloader.cpp.
I should be completely honest, I ripped a small portion of the code from a forum online, and I can't remember or make out where I took it from. But since I took it I completely refaced the code, and added pretty much everything you see there.

The code

Check out the source from google code: http://code.google.com/p/morethantechnical/source/checkout
Under trunk/NyARToolkit-CPP

Enjoy!
Roy.

Share

19 responses so far

19 Responses to “Augmented Reality with NyARToolkit, OpenCV & OpenGL”

  1. Fugitiveon 06 Aug 2009 at 4:44 am

    Hi,

    Nice tutorial!

    I noticed you glossed over the line:
    glLoadMatrixd(camera_proj);

    without explaining how you managed to get the real-world camera projection matrix to conform to opengl.

    I know that using the opencv projection matrix (http://opencv.willowgarage.com/documentation/camera_calibration_and_3d_reconstruction.html) directly doesn't work. Would you be so kind as to explain a bit?

    Full reference of question:
    http://www.opengl.org/discussion_boards/ubbthreads.php?ubb=showflat&topic=50767&gonew=1

  2. Royon 07 Aug 2009 at 9:07 pm

    Hi Fugitive
    I just used the function
    void arglCameraFrustumRH(const NyARParam& cparam, const double focalmin, const double focalmax, GLdouble m_projection[16])

    with these parameters:
    arglCameraFrustumRH(ap,1.0,100.0,camera_proj);

    Take a look in the void init ( GLvoid ) function in main.cpp.

    This creates a projection matrix for OpenGL, with z-near = 1.0 and z-far = 1000.0 (plenty 3D space to draw in). The projection is based on the calibration inside the NyARParam (loaded from file) which I did earlier using the calib tools from ARToolkit.

    Good Luck
    Roy.

  3. Jimmyon 13 Aug 2009 at 7:21 pm

    Hi Roy,

    I just started figuring out how to use NyARToolkit (C++ version). I'm still in the stage of setting everything up but your tutorial has been incredibly helpful so far.

    I'm not sure if I have a later version or not, but a quick thing: the NyARSingleDetectMarker() function needed a fourth parameter. Something like this should work although I'm not sure if it's correct since I'm still finding my way around:

    ar = new NyARSingleDetectMarker(&ap,code, 80.0,ra->getBufferReader().getBufferType());

    As of right now ar->detectMarkerLite(*ra, 100) is always returning true, so I must be doing something else wrong somewhere.

    Jimmy

  4. Romanoon 08 Sep 2009 at 7:17 pm

    Hello,

    how do you calibrate the camera? Using ARToolkit?

    Thanks,
    Romano

  5. Royon 09 Sep 2009 at 8:54 am

    Hi Romano

    Yes, I did the calibrations with the tools from ARToolkit.
    I heard these calibration tools are outdated and there are better tools to do it (I know OpenCV has calib. tools), but there are compatibility issues with ARToolkit.

    Roy.

  6. Yoelon 25 Nov 2009 at 2:01 am

    Hi Roy!
    Thanks a lot for the tutorial, i found it very usefull =)

    For anyone trying to build it with latest NyARToolKit on visual studio 2008 there are some minor changes you need to do.
    To save you the trouble you can take a look here:
    http://paste.cdtag.de/view.php?id=3460

    This is what you should expect:
    http://paste.cdtag.de/uploads/2209/marker-4.JPG

    (I took the liberty of fixing the lighting a bit and switching to teapot =D)

    I had to rebuild NyARToolKit and do the following change:
    http://paste.cdtag.de/view.php?id=3461

    (Pretty brute, but works...)

    Just wanted to save some other people the time it took me to figure this out =)

    Thanks again Roy for this tutorial, it was great :)

  7. Yoelon 25 Nov 2009 at 2:07 am

    Oh forgot one more thing - minor modification on util.cpp
    http://paste.cdtag.de/view.php?id=3462

  8. Royon 25 Nov 2009 at 10:36 pm

    Thank you! Yoel, for your contribution
    כן ירבו, תודה

    Roy.

  9. Gplannedon 05 Apr 2010 at 7:06 pm

    Hi there, I am making an app using a iPhone 3G, just wondered if I needed a 3gs or if it would still work on 3g?
    Thanks for your time
    :-)

  10. Pabloon 20 May 2010 at 8:10 pm

    Hi! I'm getting several errors on VS 2008, even with the suggested modifications.

    The one that's stopping me now is:

    1>e:\nyartoolkit\forwindows\nyartoolkitcpp-rawtest\main.cpp(86) : error C2248: 'NyARToolkitCPP::NyARRgbRaster_BGRA::NyARRgbRaster_BGRA' : cannot access private member declared in class 'NyARToolkitCPP::NyARRgbRaster_BGRA'...

    Any ideas?

  11. vodreon 26 Jul 2010 at 7:14 pm

    hey!
    im compiling with the gcc compiler, and i want to know if i could use codeblocks to link the libraries...

    can i use more than 1 marker?

    sorry for my english, im mexican!

  12. Michaelon 27 Aug 2010 at 10:39 pm

    As the cpp port of the nyartoolkit is not available in english, i am a little confused as to how i need to setup visual studio 2008 for using the toolkit. I have already downloaded, what i believe, to be the necessary files for the toolkit at "http://sourceforge.jp/projects/nyartoolkit/downloads/47617/NyARToolkitCPP-2.5.4.zip/" ; however, i see that this is entirely sourcecode for different platforms. I was thinking all i needed to do was link to the dll's like i have done with opengl and sdl. But i am at a lost as to how i would compile this source code into a dll, if that is what i am supposed to do. If i could possibly get a point in the direction of how i should setup the nyartoolkit, i should be merrily on my way with making some interesting programns? Any help is much appreciated(help in setting up nyartoolkit-cpp in visual studio 2008 that is).

    ~thank you for your time

  13. Ashleyon 28 Oct 2011 at 8:36 am

    hello,

    This project is to be created on VS C++ , console project? is any library of OpenCV Required? Require help.
    Or anyone have the codes to share for me to refer?

    Contact me @ manchesterlim@hotmail.com

  14. Arnauon 16 Nov 2011 at 7:54 pm

    Hi!! I’m a student and I’m working with NyArtoolkit. I have two doubts with my project. I have to use diferents markers and models thats came in Nyartoolkit by default. Do you know any web to download some markers or any way to create my own markers ??? The same think with the 3D models. Thank for your time!!!!!!

  15. sampieon 26 Nov 2011 at 2:56 pm

    hi Roy!
    it's great tutorail!! thanks for sharing!:)
    i was asking if NyARtoolkit has a doc for C++ because i didn't find any thing on the net!

  16. Royon 01 Dec 2011 at 12:53 am

    @sample
    These days I would actually fall back on ARToolkit.. or use some library. NyARToolkit seems kind of dead..

  17. KeenBeginneron 29 Dec 2011 at 3:21 pm

    Hi Roy

    This is great work.
    I am a beginner and currently working on a project. I am to track a human face and then display 3d objects (hat/mask/beak on the mouth) on the tracked faces and fix those 3d objects on the face even when the face is rotated and moved. I have managed to detect faces and extract coordinates of certain keypoints of the face. I am using OpenCV for that. And as you suggested above, I am going to transfer the grabbed frame and display it in OpenGL.
    From my understanding, you acquire your projection matrix on the marker using the augmented reality toolkit. I am not using that and only have fixed coordinates of the keypoints as mentioned before so I am quite confused on how to proceed. I'd really appreciate if you could provide me some direction as to how I may be able to solve my problem.

    Many thanks

  18. Royon 31 Dec 2011 at 9:28 am

    @KeenBeginner
    I gave your problem some thought, and it seems like you can easily achieve this with ARToolkit! Take a look at the API:
    http://artoolkit.sourceforge.net/apidoc/ar_8h.html

    All you need to do is populate an ARMarkerInfo struct
    Fill it up with info of the face, like the corners can be the corners you get from CascadeClassifier::detectMultiScale, or your more accurate info from the facial features

    Then use arGetTransMat (and the like) to get the proper transformtion matrix for OpenGL... then display some graphics as you like.

    This will be a very initial solution! but it will definitely get you results right away, because everything exists and ready for you.

    I can also highly recommend Jason Saragih's 3D face detection libary:
    http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html

  19. giselon 06 Sep 2012 at 4:11 pm

    Hello
    It seems I have some linkage problem with NYARToolkit, does anybody encountered this problem? I cant find any lib to include in external projects, does it consist of only source files?

Trackback URI | Comments RSS

Leave a Reply