Mar 05 2009

Qt & OpenCV combined for face detecting QWidgets

Published by at 2:19 pm under gui,linux,programming,qt,Uncategorized,video

As my search for the best platform to roll-out my new face detection concept continues, I decided to give ol' Qt framework a go.

I like Qt. It's cross-platform, a clear a nice API, straightforward, and remindes me somewhat of Apple's Cocoa.

My intention is to get some serious face detection going on mobile devices. So that means either the iPhone, which so far did a crummy job performance-wise, or some other mobile device, preferably linux-based.
This led me to the decision to go with Qt. I believe you can get it to work on any linux-ish platform (limo, moblin, android), and since Nokia baught Trolltech - it's gonna work on Nokia phones soon, awesome!

Lets get to the details, shall we?

First thing's first: face detection.

I ripped OpenCV's facedetect.c sample and extracted only the detect_and_draw() function. Originally the function detects the faces and draws a circle over them, but I needed only the face detection and the result bounding rectangle. So in the end I was left with this:


CvRect detect_and_draw( IplImage* img, CvMemStorage* storage, CvHaarClassifierCascade* cascade )
{
IplImage *gray, *small_img;
int i = 0;

gray = cvCreateImage( cvSize(img->width,img->height), 8, 1 );
small_img = cvCreateImage( cvSize( cvRound (img->width/scale),
cvRound (img->height/scale)), 8, 1 );

cvCvtColor( img, gray, CV_RGB2GRAY );
cvResize( gray, small_img, CV_INTER_LINEAR );
cvEqualizeHist( small_img, small_img );
cvClearMemStorage( storage );

CvRect* r = NULL;

if( cascade )
{
double t = (double)cvGetTickCount();
CvSeq* faces = cvHaarDetectObjects( small_img, cascade, storage,
1.1, 2, 0
|CV_HAAR_FIND_BIGGEST_OBJECT
//|CV_HAAR_DO_ROUGH_SEARCH
//|CV_HAAR_DO_CANNY_PRUNING
//|CV_HAAR_SCALE_IMAGE
,
cvSize(30, 30) );
t = (double)cvGetTickCount() - t;

printf( "detection time = %gms\n", t/((double)cvGetTickFrequency()*1000.) );

r = (CvRect*)cvGetSeqElem( faces, i );

cvReleaseImage( &gray );
cvReleaseImage( &small_img );

if(r) {
return cvRect(r->x,r->y,r->width,r->height);
} else {
return cvRect(-1,-1,0,0);
}

}

This can go anywhere in the code base, as it's totally independant (as long as you train the cascade and allocate a MemStorage).Note that I am assuming only one face in the input image, and also that it will be the largest detected object. This bring my benchmark to about 25ms per frame, using the original general detection approach of facedetect.c benchmarked at about 160ms per frame.

OK, done with pure OpenCV, on to Qt.

I subclassed a QWidget, who's sole purpose is to show the input video with the detected face. For starters, I needed to have a QImage and an IplImage instances as members, they can also share the same buffer (how awesome is that..). I also need a CvCapture, CvMemStorage and a CvHaarCalssifierCascade:


class FaceRecognizer : public QWidget
{
Q_OBJECT

public:
FaceRecognizer(QWidget *parent = 0);
~FaceRecognizer();

private:
Ui::FaceRecognizerClass ui;

QImage m_i;

QRect faceLoc;

CvMemStorage* storage;
CvHaarClassifierCascade* cascade;
CvCapture* capture;
IplImage* m_opencvimg;

QTimer* m_timer;

void paintEvent(QPaintEvent* e);

public slots:
void queryFrame();
};

You can see that I'm gonna use a QTimer to query the frames from the CvCapture and also I override paintEvent to paint the frame onto the canvas. In fact in my QWidget I have a QFrame, that the image will painted over it. UI was generated in Qt Designer.

First, some initialization in the constructor:


FaceRecognizer::FaceRecognizer(QWidget *parent)
: QWidget(parent)
{
ui.setupUi(this);

capture = cvCaptureFromAVI( "/home/user/Desktop/video.avi" );
//grab one frame to get width and height

IplImage* frame = cvQueryFrame( capture );

m_i = QImage(QSize(frame->width,frame->height),QImage::Format_RGB888);
ui.frame->setMinimumSize(m_i.width(),m_i.height());
ui.frame->setMaximumSize(ui.frame->minimumSize());
//create only the header, as the data buffer is shared, and was allocated by QImage

m_opencvimg = cvCreateImageHeader(cvSize(m_i.width(),m_i.height()),8,3);
m_opencvimg->imageData = (char*)m_i.bits(); // share buffers

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );

//images from cvQueryFrame come in BGR form and not what Qt expects - RGB

//and since the buffers are shared - format should be consistent
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

//we need memstorage and a cascade
storage = cvCreateMemStorage(0);
cascade = (CvHaarClassifierCascade*)cvLoad( CASCADE_NAME, 0, 0, 0 );

//set timer for 50ms intervals

m_timer = new QTimer(this);
connect(m_timer,SIGNAL(timeout()),this,SLOT(queryFrame()));
m_timer->start(50);
}

And now, querying the frame: query CvCapture, convert BGR to RGB, detect faces and update faceLoc QRect.


void FaceRecognizer::queryFrame() {
IplImage* frame = cvQueryFrame( capture );

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

CvRect r = detect_and_draw(m_opencvimg,storage,cascade);
faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));

this->update();
}

Finally - painting, which is easy:

void FaceRecognizer::paintEvent(QPaintEvent* e) {
QPainter painter(this);

painter.drawImage(QPoint(ui.frame->x(),ui.frame->y()),m_i);

if(faceLoc.x() > 0 && faceLoc.y() > 0) {
painter.setBrush(Qt::NoBrush);
painter.setPen(QColor(255,0,0));
painter.drawRect(QRect(faceLoc.x()+ui.frame->x(),faceLoc.y()+ui.frame->y(),faceLoc.width(),faceLoc.height()));
}
}

Looks like it's all done... Here's a video:

Share

25 responses so far

25 Responses to “Qt & OpenCV combined for face detecting QWidgets”

  1. Dat Chuon 21 Mar 2009 at 10:39 pm

    Very nice tutorial and code on how to get Qt to work with OpenCV (especially on the IplImage and QImage buffer sharing using Format888. You might want to release your code on qt-apps.org with LGPL. Either that or you can modify the existing project on there that also have QtWidget for OpenGL face tracking (GPL).

    One side note, the code section looks weird on Firefox on Ubuntu.

  2. sachinon 16 May 2009 at 12:16 pm

    nice program
    i am also trying with java code
    plz send me the detail instruion of opencv & use of it.
    i am trying on windows OS plz help me

  3. Renatoon 28 May 2009 at 10:48 am

    hi,

    I would like to know if you locate the PixMap into a QLabel. I am willing to get the mouse events over the opencv image or qt. Do you have any experience in that?

    Regards,

  4. Royon 01 Jun 2009 at 9:29 am

    I am painting the frame from camera myself with QPainter:
    void FaceDetectorQWidget::paintEvent(QPaintEvent* e) {
    if(!m_i.isNull()) {
    QPainter painter(this);
    painter.drawImage(ui.frame->geometry(),m_i,m_i.rect());
    }
    }

    (m_i is private QImage member)

    I'm not using a QLabel, although it might be a good idea.

    For capturing events you can override this function:
    void FaceDetectorQWidget::mouseReleaseEvent(QMouseEvent* e) {}

  5. Asif Sardaron 08 Jun 2009 at 4:03 pm

    Hi,

    I am confused with the following:

    private:
    Ui::FaceRecognizerClass ui;

    Can you please clear the above issue, I am having compilation problems.

    With Best Regards,
    Asif Sardar.

  6. Royon 08 Jun 2009 at 6:03 pm

    This is made by Qt Designer, it is the GUI for this Qwigdet.
    It includes the frame to paint on, buttons, etc.

    You can remove this line, but then you should take care or the UI yourself. The designer makes it easier.

    Roy.

  7. Asif Sardaron 09 Jun 2009 at 9:27 am

    Hi,

    Now I have compile time problem with ui.frame, the error is

    15: error: request for member ‘frame’ in ‘((FaceReconizer*)this)->FaceReconizer::ui’, which is of non-class type ‘Ui::FaceReconizer*’

    If someone can help out this issue.

    With Best Regards,
    Asif Sardar.

  8. Asif Sardaron 09 Jun 2009 at 1:31 pm

    Hi,

    I sorted out the above error, don't use ui.frame->setminimumsize etc. Just use it like

    setminimumsize() ...

    Then the compiler does not create problem. We don't need to reference the frame when we setupUi for the widget.

    With Best Regards,
    Asif Sardar.

  9. Asif Sardaron 09 Jun 2009 at 3:11 pm

    hi,

    I am having problem to build the above code in Qt 4.5.1 editor.

    undefined reference to `cvCreateImage'

    The above message is mentioned in the editor. The opencv and Qt variables have been detected but the funcitons couldn't be detected because somehow the libraries should be linked.

    If anyone could help in this regard.

    Moreover what editor and libraries had been used in the above code.

    With Best Regards,
    Asif Sardar.

  10. Dragoson 04 Jul 2009 at 9:47 pm

    Hello Asif, if you are using ms visual studio, go to Project tab -> your project properties
    (at the bottom). Chose Configuration Properties -> Linker -> Input.

    Here at the Additional Dependencies you should have the following : qtmaind.lib QtCored4.lib QtGuid4.lib (!!SUPPOSING you have correctly installed QT and qt plugin for visual studio). Next: add 3 more libraries: cv.lib cxcore.lib highgui.lib -> these are from OpenCV.

    This will only work if you previously defined the path to OpenCV's include, source directories in Visual Studio tool bar -> Tools -> Options -> Projects and Solutions -> VC++ Directories. In this place you define the path to QT's directories too.

    Search Google for "installing OpenCV/ QT with Visual Studio".

    Good luck!

  11. mclauon 16 Jul 2009 at 6:52 am

    Hi Roy,

    This is brilliant. I agree with Dat Chu that you should consider to share the application on gt-apps.org because I found it will be easier for a beginner like me to start from it.

  12. Spaïon 16 Aug 2009 at 11:24 pm

    Hi, nice work !!

    Is this method (create only the header, as the data buffer is shared, and was allocated by QImage) faster than the method on qt-apps : http://www.qt-apps.org/content/show.php/Qt+Opencv+webcam+viewer?content=89995 ??

    (sorry for my poor english ;) )

    thx in advance

  13. Royon 17 Aug 2009 at 9:09 am

    Hi

    My method, I believe, is better since I'm not shuffling bits around. The memory buffer is shared between OpenCV IplImage and Qt's QImage.
    When the byte-order (888 RGB interleave) is the same, there's no problem in sharing buffers.

    My method doesn't require copying the IplImage->imageData, as that guy does in "void QOpenCVWidget::putImage(IplImage *cvimage)". He also uses a less optimized way of using QImage.setPixel for each pixel, this can be done faster (memcpy for each row?).

    Roy.

  14. Spaïon 17 Aug 2009 at 12:12 pm

    Hello Roy and all, thank you for your prompt response.

    It is really a very good idea this "shared buffer".
    I only saw that on your blog.

    I am a composer and I am currently developing software to make music with a webcam: NoizeKam
    You can find here: http://blog.oliviergest.net/noizekam/

    I am using OpenCV and STK, but now I use only PortAudio and I started to integrate QT for the interface and OpenGL.
    I had already tested QT, but it slows down the execution.
    I will try your method of shared buffer because I think it should speed up my soft and eat less CPU cycles.

    Again thank you for sharing your knowledge (everyone does not, that's a shame) and your help is so valuable in this area.

    PS: translated from French into English by my friend Google

  15. Azaron 03 Nov 2009 at 6:01 pm

    Hi,
    thanks for the tutorial. I need to find out if i can convert QImage to IplImage. My requirement is to get the pixel co-ordinates when i click on some location of the image. and then convert it to IplImage and process.

    Thank u

  16. Sevickon 20 Nov 2009 at 4:09 am

    Idea is good, but on my PC it is still too slow comparing with OpenCV ShowImage...

    [Edit: one link is enough dude..]

  17. Kronenon 19 Dec 2009 at 7:49 am

    I get a segmentation fault on cvCvtColor(img, gray, CV_RGB2GRAY);

    Can anyone help me?

  18. giuseppeon 18 May 2010 at 5:16 am

    Me too. On Windows, I get a run-time seg fault. But I cannot trace it.
    G

  19. stevenon 14 Jun 2010 at 10:33 am

    i seem to not be able to load the haar clasifier cascade from my files, can anyone help me?

  20. irawanon 23 Jul 2010 at 5:23 pm

    thank for idea, it's work...

  21. Johnsonon 29 Jul 2010 at 11:14 am

    Hello Roy,

    I am a beginner in QT, and I like this example a lot.
    Can you send me the whole project of this program?
    Thanks in advance.

  22. Royon 30 Jul 2010 at 1:17 am

    @Johnson: Sadly, I don't have the original sources for this project anymore... It must be reconstructed from the sources in the post itself. Sorry.

  23. Bluesmanon 21 Feb 2011 at 5:40 am

    Nice job, very helpful! Quick question:

    Couldn't the the following lines:

    if( frame->origin == IPL_ORIGIN_TL )
    cvCopy( frame, m_opencvimg, 0 );
    else
    cvFlip( frame, m_opencvimg, 0 );

    cvCvtColor(m_opencvimg, m_opencvimg, CV_BGR2RGB);

    Be replaced by:

    if( frame->origin == IPL_ORIGIN_TL )
    cvCvtColor( frame, m_opencvimg, CV_BGR2RGB);
    else
    {
    cvFlip( frame, m_opencvimg, 0 );
    cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);
    }

    to avoid a copy operation in the most common case when a flip is not required? I've tried it and it seems to work.

  24. mrbongdemon 12 Aug 2012 at 8:31 pm

    void FaceRecognizer::queryFrame() {
    IplImage* frame = cvQueryFrame( capture );

    if( frame->origin == IPL_ORIGIN_TL )
    cvCopy( frame, m_opencvimg, 0 );
    else
    cvFlip( frame, m_opencvimg, 0 );
    cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

    CvRect r= detect_and_draw(m_opencvimg,storage,cascade);
    faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));

    this->update();
    }
    Can anyone help me ?
    /root/Desktop/face_detect/detect_face/facerecognizer.cpp:46: undefined reference to `FaceRecognizer::detect_and_draw(_IplImage*, CvMemStorage*, CvHaarClassifierCascade*)'

  25. kevinon 04 Feb 2013 at 8:09 am

    I am using my hand to move the cursor instead of the mouse. Can you pls help me with the following doubt:
    I want to perform an action when the cursor comes over a push button. How do I do it?
    is Qt's setFocus function appropriate for this???
    Thanks in advance.

Trackback URI | Comments RSS

Leave a Reply