Categories
gui linux programming qt Uncategorized video

Qt & OpenCV combined for face detecting QWidgets

As my search for the best platform to roll-out my new face detection concept continues, I decided to give ol’ Qt framework a go.
I like Qt. It’s cross-platform, a clear a nice API, straightforward, and remindes me somewhat of Apple’s Cocoa.
My intention is to get some serious face detection going on mobile devices. So that means either the iPhone, which so far did a crummy job performance-wise, or some other mobile device, preferably linux-based.
This led me to the decision to go with Qt. I believe you can get it to work on any linux-ish platform (limo, moblin, android), and since Nokia baught Trolltech – it’s gonna work on Nokia phones soon, awesome!
Lets get to the details, shall we?

First thing’s first: face detection.
I ripped OpenCV’s facedetect.c sample and extracted only the detect_and_draw() function. Originally the function detects the faces and draws a circle over them, but I needed only the face detection and the result bounding rectangle. So in the end I was left with this:

CvRect detect_and_draw( IplImage* img, CvMemStorage* storage, CvHaarClassifierCascade* cascade )
{
IplImage *gray, *small_img;
int i = 0;
gray = cvCreateImage( cvSize(img->width,img->height), 8, 1 );
small_img = cvCreateImage( cvSize( cvRound (img->width/scale),
cvRound (img->height/scale)), 8, 1 );
cvCvtColor( img, gray, CV_RGB2GRAY );
cvResize( gray, small_img, CV_INTER_LINEAR );
cvEqualizeHist( small_img, small_img );
cvClearMemStorage( storage );
CvRect* r = NULL;
if( cascade )
{
double t = (double)cvGetTickCount();
CvSeq* faces = cvHaarDetectObjects( small_img, cascade, storage,
1.1, 2, 0
|CV_HAAR_FIND_BIGGEST_OBJECT
//|CV_HAAR_DO_ROUGH_SEARCH
//|CV_HAAR_DO_CANNY_PRUNING
//|CV_HAAR_SCALE_IMAGE
,
cvSize(30, 30) );
t = (double)cvGetTickCount() - t;
printf( "detection time = %gms\n", t/((double)cvGetTickFrequency()*1000.) );
r = (CvRect*)cvGetSeqElem( faces, i );
cvReleaseImage( &gray );
cvReleaseImage( &small_img );
if(r) {
return cvRect(r->x,r->y,r->width,r->height);
} else {
return cvRect(-1,-1,0,0);
}
}

This can go anywhere in the code base, as it’s totally independant (as long as you train the cascade and allocate a MemStorage).Note that I am assuming only one face in the input image, and also that it will be the largest detected object. This bring my benchmark to about 25ms per frame, using the original general detection approach of facedetect.c benchmarked at about 160ms per frame.
OK, done with pure OpenCV, on to Qt.
I subclassed a QWidget, who’s sole purpose is to show the input video with the detected face. For starters, I needed to have a QImage and an IplImage instances as members, they can also share the same buffer (how awesome is that..). I also need a CvCapture, CvMemStorage and a CvHaarCalssifierCascade:

class FaceRecognizer : public QWidget
{
Q_OBJECT
public:
FaceRecognizer(QWidget *parent = 0);
~FaceRecognizer();
private:
Ui::FaceRecognizerClass ui;
QImage m_i;
QRect faceLoc;
CvMemStorage* storage;
CvHaarClassifierCascade* cascade;
CvCapture* capture;
IplImage* m_opencvimg;
QTimer* m_timer;
void paintEvent(QPaintEvent* e);
public slots:
void queryFrame();
};

You can see that I’m gonna use a QTimer to query the frames from the CvCapture and also I override paintEvent to paint the frame onto the canvas. In fact in my QWidget I have a QFrame, that the image will painted over it. UI was generated in Qt Designer.
First, some initialization in the constructor:

FaceRecognizer::FaceRecognizer(QWidget *parent)
: QWidget(parent)
{
ui.setupUi(this);
capture = cvCaptureFromAVI( "/home/user/Desktop/video.avi" );
//grab one frame to get width and height
IplImage* frame = cvQueryFrame( capture );
m_i = QImage(QSize(frame->width,frame->height),QImage::Format_RGB888);
ui.frame->setMinimumSize(m_i.width(),m_i.height());
ui.frame->setMaximumSize(ui.frame->minimumSize());
//create only the header, as the data buffer is shared, and was allocated by QImage
m_opencvimg = cvCreateImageHeader(cvSize(m_i.width(),m_i.height()),8,3);
m_opencvimg->imageData = (char*)m_i.bits(); // share buffers
if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
//images from cvQueryFrame come in BGR form and not what Qt expects - RGB
//and since the buffers are shared - format should be consistent
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);
//we need memstorage and a cascade
storage = cvCreateMemStorage(0);
cascade = (CvHaarClassifierCascade*)cvLoad( CASCADE_NAME, 0, 0, 0 );
//set timer for 50ms intervals
m_timer = new QTimer(this);
connect(m_timer,SIGNAL(timeout()),this,SLOT(queryFrame()));
m_timer->start(50);
}

And now, querying the frame: query CvCapture, convert BGR to RGB, detect faces and update faceLoc QRect.

void FaceRecognizer::queryFrame() {
IplImage* frame = cvQueryFrame( capture );
if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);
CvRect r = detect_and_draw(m_opencvimg,storage,cascade);
faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));
this->update();
}
Finally - painting, which is easy:
void FaceRecognizer::paintEvent(QPaintEvent* e) {
QPainter painter(this);
painter.drawImage(QPoint(ui.frame->x(),ui.frame->y()),m_i);
if(faceLoc.x() > 0 && faceLoc.y() > 0) {
painter.setBrush(Qt::NoBrush);
painter.setPen(QColor(255,0,0));
painter.drawRect(QRect(faceLoc.x()+ui.frame->x(),faceLoc.y()+ui.frame->y(),faceLoc.width(),faceLoc.height()));
}
}

Looks like it’s all done… Here’s a video:

25 replies on “Qt & OpenCV combined for face detecting QWidgets”

Very nice tutorial and code on how to get Qt to work with OpenCV (especially on the IplImage and QImage buffer sharing using Format888. You might want to release your code on qt-apps.org with LGPL. Either that or you can modify the existing project on there that also have QtWidget for OpenGL face tracking (GPL).
One side note, the code section looks weird on Firefox on Ubuntu.

nice program
i am also trying with java code
plz send me the detail instruion of opencv & use of it.
i am trying on windows OS plz help me

hi,
I would like to know if you locate the PixMap into a QLabel. I am willing to get the mouse events over the opencv image or qt. Do you have any experience in that?
Regards,

I am painting the frame from camera myself with QPainter:
void FaceDetectorQWidget::paintEvent(QPaintEvent* e) {
if(!m_i.isNull()) {
QPainter painter(this);
painter.drawImage(ui.frame->geometry(),m_i,m_i.rect());
}
}
(m_i is private QImage member)
I’m not using a QLabel, although it might be a good idea.
For capturing events you can override this function:
void FaceDetectorQWidget::mouseReleaseEvent(QMouseEvent* e) {}

Hi,
I am confused with the following:
private:
Ui::FaceRecognizerClass ui;
Can you please clear the above issue, I am having compilation problems.
With Best Regards,
Asif Sardar.

This is made by Qt Designer, it is the GUI for this Qwigdet.
It includes the frame to paint on, buttons, etc.
You can remove this line, but then you should take care or the UI yourself. The designer makes it easier.
Roy.

Hi,
Now I have compile time problem with ui.frame, the error is
15: error: request for member ‘frame’ in ‘((FaceReconizer*)this)->FaceReconizer::ui’, which is of non-class type ‘Ui::FaceReconizer*’
If someone can help out this issue.
With Best Regards,
Asif Sardar.

Hi,
I sorted out the above error, don’t use ui.frame->setminimumsize etc. Just use it like
setminimumsize() …
Then the compiler does not create problem. We don’t need to reference the frame when we setupUi for the widget.
With Best Regards,
Asif Sardar.

hi,
I am having problem to build the above code in Qt 4.5.1 editor.
undefined reference to `cvCreateImage’
The above message is mentioned in the editor. The opencv and Qt variables have been detected but the funcitons couldn’t be detected because somehow the libraries should be linked.
If anyone could help in this regard.
Moreover what editor and libraries had been used in the above code.
With Best Regards,
Asif Sardar.

Hello Asif, if you are using ms visual studio, go to Project tab -> your project properties
(at the bottom). Chose Configuration Properties -> Linker -> Input.
Here at the Additional Dependencies you should have the following : qtmaind.lib QtCored4.lib QtGuid4.lib (!!SUPPOSING you have correctly installed QT and qt plugin for visual studio). Next: add 3 more libraries: cv.lib cxcore.lib highgui.lib -> these are from OpenCV.
This will only work if you previously defined the path to OpenCV’s include, source directories in Visual Studio tool bar -> Tools -> Options -> Projects and Solutions -> VC++ Directories. In this place you define the path to QT’s directories too.
Search Google for “installing OpenCV/ QT with Visual Studio”.
Good luck!

Hi Roy,
This is brilliant. I agree with Dat Chu that you should consider to share the application on gt-apps.org because I found it will be easier for a beginner like me to start from it.

Hi
My method, I believe, is better since I’m not shuffling bits around. The memory buffer is shared between OpenCV IplImage and Qt’s QImage.
When the byte-order (888 RGB interleave) is the same, there’s no problem in sharing buffers.
My method doesn’t require copying the IplImage->imageData, as that guy does in “void QOpenCVWidget::putImage(IplImage *cvimage)”. He also uses a less optimized way of using QImage.setPixel for each pixel, this can be done faster (memcpy for each row?).
Roy.

Hello Roy and all, thank you for your prompt response.
It is really a very good idea this “shared buffer”.
I only saw that on your blog.
I am a composer and I am currently developing software to make music with a webcam: NoizeKam
You can find here: http://blog.oliviergest.net/noizekam/
I am using OpenCV and STK, but now I use only PortAudio and I started to integrate QT for the interface and OpenGL.
I had already tested QT, but it slows down the execution.
I will try your method of shared buffer because I think it should speed up my soft and eat less CPU cycles.
Again thank you for sharing your knowledge (everyone does not, that’s a shame) and your help is so valuable in this area.
PS: translated from French into English by my friend Google

Hi,
thanks for the tutorial. I need to find out if i can convert QImage to IplImage. My requirement is to get the pixel co-ordinates when i click on some location of the image. and then convert it to IplImage and process.
Thank u

i seem to not be able to load the haar clasifier cascade from my files, can anyone help me?

Hello Roy,
I am a beginner in QT, and I like this example a lot.
Can you send me the whole project of this program?
Thanks in advance.

@Johnson: Sadly, I don’t have the original sources for this project anymore… It must be reconstructed from the sources in the post itself. Sorry.

Nice job, very helpful! Quick question:
Couldn’t the the following lines:
if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg, m_opencvimg, CV_BGR2RGB);
Be replaced by:
if( frame->origin == IPL_ORIGIN_TL )
cvCvtColor( frame, m_opencvimg, CV_BGR2RGB);
else
{
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);
}
to avoid a copy operation in the most common case when a flip is not required? I’ve tried it and it seems to work.

void FaceRecognizer::queryFrame() {
IplImage* frame = cvQueryFrame( capture );
if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);
CvRect r= detect_and_draw(m_opencvimg,storage,cascade);
faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));
this->update();
}
Can anyone help me ?
/root/Desktop/face_detect/detect_face/facerecognizer.cpp:46: undefined reference to `FaceRecognizer::detect_and_draw(_IplImage*, CvMemStorage*, CvHaarClassifierCascade*)’

I am using my hand to move the cursor instead of the mouse. Can you pls help me with the following doubt:
I want to perform an action when the cursor comes over a push button. How do I do it?
is Qt’s setFocus function appropriate for this???
Thanks in advance.

Leave a Reply

Your email address will not be published. Required fields are marked *