I like Qt. It’s cross-platform, a clear a nice API, straightforward, and remindes me somewhat of Apple’s Cocoa.

My intention is to get some serious face detection going on mobile devices. So that means either the iPhone, which so far did a crummy job performance-wise, or some other mobile device, preferably linux-based.
This led me to the decision to go with Qt. I believe you can get it to work on any linux-ish platform (limo, moblin, android), and since Nokia baught Trolltech – it’s gonna work on Nokia phones soon, awesome!

Lets get to the details, shall we?

First thing’s first: face detection.

I ripped OpenCV’s facedetect.c sample and extracted only the detect_and_draw() function. Originally the function detects the faces and draws a circle over them, but I needed only the face detection and the result bounding rectangle. So in the end I was left with this:

CvRect detect_and_draw( IplImage* img, CvMemStorage* storage, CvHaarClassifierCascade* cascade )
{
IplImage *gray, *small_img;
int i = 0;

gray = cvCreateImage( cvSize(img->width,img->height), 8, 1 );
small_img = cvCreateImage( cvSize( cvRound (img->width/scale),
cvRound (img->height/scale)), 8, 1 );

cvCvtColor( img, gray, CV_RGB2GRAY );
cvResize( gray, small_img, CV_INTER_LINEAR );
cvEqualizeHist( small_img, small_img );
cvClearMemStorage( storage );

CvRect* r = NULL;

if( cascade )
{
double t = (double)cvGetTickCount();
CvSeq* faces = cvHaarDetectObjects( small_img, cascade, storage,
1.1, 2, 0
|CV_HAAR_FIND_BIGGEST_OBJECT
//|CV_HAAR_DO_ROUGH_SEARCH
//|CV_HAAR_DO_CANNY_PRUNING
//|CV_HAAR_SCALE_IMAGE
,
cvSize(30, 30) );
t = (double)cvGetTickCount() - t;

printf( "detection time = %gms\n", t/((double)cvGetTickFrequency()*1000.) );

r = (CvRect*)cvGetSeqElem( faces, i );

cvReleaseImage( &gray );
cvReleaseImage( &small_img );

if(r) {
return cvRect(r->x,r->y,r->width,r->height);
} else {
return cvRect(-1,-1,0,0);
}

}

This can go anywhere in the code base, as it’s totally independant (as long as you train the cascade and allocate a MemStorage).Note that I am assuming only one face in the input image, and also that it will be the largest detected object. This bring my benchmark to about 25ms per frame, using the original general detection approach of facedetect.c benchmarked at about 160ms per frame.

OK, done with pure OpenCV, on to Qt.

I subclassed a QWidget, who’s sole purpose is to show the input video with the detected face. For starters, I needed to have a QImage and an IplImage instances as members, they can also share the same buffer (how awesome is that..). I also need a CvCapture, CvMemStorage and a CvHaarCalssifierCascade:

class FaceRecognizer : public QWidget
{
Q_OBJECT

public:
FaceRecognizer(QWidget *parent = 0);
~FaceRecognizer();

private:
Ui::FaceRecognizerClass ui;

QImage m_i;

QRect faceLoc;

CvMemStorage* storage;
CvHaarClassifierCascade* cascade;
CvCapture* capture;
IplImage* m_opencvimg;

QTimer* m_timer;

void paintEvent(QPaintEvent* e);

public slots:
void queryFrame();
};

You can see that I’m gonna use a QTimer to query the frames from the CvCapture and also I override paintEvent to paint the frame onto the canvas. In fact in my QWidget I have a QFrame, that the image will painted over it. UI was generated in Qt Designer.

First, some initialization in the constructor:

FaceRecognizer::FaceRecognizer(QWidget *parent)
: QWidget(parent)
{
ui.setupUi(this);

capture = cvCaptureFromAVI( "/home/user/Desktop/video.avi" );
//grab one frame to get width and height

IplImage* frame = cvQueryFrame( capture );

m_i = QImage(QSize(frame->width,frame->height),QImage::Format_RGB888);
ui.frame->setMinimumSize(m_i.width(),m_i.height());
ui.frame->setMaximumSize(ui.frame->minimumSize());
//create only the header, as the data buffer is shared, and was allocated by QImage

m_opencvimg = cvCreateImageHeader(cvSize(m_i.width(),m_i.height()),8,3);
m_opencvimg->imageData = (char*)m_i.bits(); // share buffers

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );

//images from cvQueryFrame come in BGR form and not what Qt expects - RGB

//and since the buffers are shared - format should be consistent
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

//we need memstorage and a cascade
storage = cvCreateMemStorage(0);
cascade = (CvHaarClassifierCascade*)cvLoad( CASCADE_NAME, 0, 0, 0 );

//set timer for 50ms intervals

m_timer = new QTimer(this);
connect(m_timer,SIGNAL(timeout()),this,SLOT(queryFrame()));
m_timer->start(50);
}

And now, querying the frame: query CvCapture, convert BGR to RGB, detect faces and update faceLoc QRect.

void FaceRecognizer::queryFrame() {
IplImage* frame = cvQueryFrame( capture );

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

CvRect r = detect_and_draw(m_opencvimg,storage,cascade);
faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));

this->update();
}

Finally - painting, which is easy:

void FaceRecognizer::paintEvent(QPaintEvent* e) {
QPainter painter(this);

painter.drawImage(QPoint(ui.frame->x(),ui.frame->y()),m_i);

if(faceLoc.x() > 0 && faceLoc.y() > 0) {
painter.setBrush(Qt::NoBrush);
painter.setPen(QColor(255,0,0));
painter.drawRect(QRect(faceLoc.x()+ui.frame->x(),faceLoc.y()+ui.frame->y(),faceLoc.width(),faceLoc.height()));
}
}

Looks like it’s all done… Here’s a video:

So anyway, I have my video files encoded in some unknown format and I need my program to show them in a some widget. I went around looking for an exiting example, but i couldn’t find anything concrete, except for a good tip here that led me here for an example of using ffmpeg’s libavformat and libavcodec, but no end-to-end example including the Qt code.

The ffmpeg example was simple enough to just copy-paste into my project, but the whole painting over the widget’s canvas was not covered. Turns out painting video is not as simple as overriding paintEvent()…

Firstly, you need a separate thread for grabbing frames from the video file, because you won’t let the GUI event thread do that.
That makes sense, but when the frame-grabbing thread (I called VideoThread) actually grabbed a frame and inserted it somewhere in the memory, I needed to tell the GUI thread to take that buffered pixels and paint them over the widget’s canvas.

This is the moment where I praise Qt’s excellent Signals/Slots mechanism. So I’ll have my VideoThread emit a signal notifying some external entity that a new frame is in the buffer.
Here’s a little code:

void VideoThread::run() {
/*
... Initialize libavformat & libavcodec data structures.
You can see it in the example i referred to before
*/
// Open video file
if(av_open_input_file(&pFormatCtx,
"lala.avi",
NULL, 0, NULL)!=0)
return -1; // Couldn't open file

// Retrieve stream information
if(av_find_stream_info(pFormatCtx)<0)
return -1; // Couldn't find stream information

// Find the first video stream ...

// Get a pointer to the codec context for the video
// stream...

// Find the decoder for the video stream...

// Open codec...

// Allocate video frame
pFrame=avcodec_alloc_frame();

// Allocate an AVFrame structure
pFrameRGB=avcodec_alloc_frame();
if(pFrameRGB==NULL)
return -1;

int dst_fmt = PIX_FMT_RGB24;
int dst_w = 160;
int dst_h = 120;

// Determine required buffer size and allocate buffer
numBytes = avpicture_get_size(dst_fmt, dst_w, dst_h);
buffer = new uint8_t[numBytes + 64];

//put a PPM header on the buffer
int headerlen = sprintf((char *) buffer,
"P6n%d %dn255n",
dst_w, dst_h);

_v->buf = (uchar*)buffer;
_v->len = avpicture_get_size(dst_fmt,dst_w,dst_h) +
headerlen;

// Assign appropriate parts of buffer to image planes
// in pFrameRGB...

// I use libswscale to scale the frames to the required
// size.
// Setup the scaling context:
SwsContext *img_convert_ctx;
img_convert_ctx = sws_getContext(
pCodecCtx->width, pCodecCtx->height,
pCodecCtx->pix_fmt,
dst_w, dst_h, dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);

// Read frames and notify
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0)
{
// Is this a packet from the video stream?
if(packet.stream_index==videoStream)
{
// Decode video frame
avcodec_decode_video(pCodecCtx,
pFrame,
&frameFinished,
packet.data,
packet.size);

// Did we get a video frame?
if(frameFinished)
{
// Convert the image to RGB
sws_scale(img_convert_ctx,
pFrame->data,
pFrame->linesize,
0,
pCodecCtx->height,
pFrameRGB->data,
pFrameRGB->linesize);

emit frameReady();

//My video is 5FPS so sleep for 200ms.
this->msleep(200);
}
}

// Free the packet that was allocated by
// av_read_frame
av_free_packet(&packet);
}

// Free the RGB image
delete [] buffer;
av_free(pFrameRGB);

// Free the YUV frame
av_free(pFrame);

// Close the codec...

// Close the video file...
} //end VideoThread::run

Ok so I have a frame-grabber that emits a frameReady signal everytime the buffer is full and ready for painting.

A couple of things to notice:

• I convert the image format to PIX_FMT_RGB24 (avcodec.h), which is required by Qt’s QImage::fromData() method.
• I scale the image using ffmpeg’s libswscale. All conversion/scaling methods inside libavcodev are deprecated now.
  But it’s fairly simple, here‘s a good example. Just remember you need a sws_getContext and then sws_scale.
• I totally disregard actual frame rate here, I just sleep for 200ms because i know my file is 5FPS. For a (far-) more sophisticated way to get the FPS, very important if this is not a constant frame-rate video, you can find here.
• I don’t cover audio in this example, although the mechanism to extract it from the file exists… you just need to grabe the audio stream’s frame. For playing audio you also need some Qt-external library. In a different project I used SDL very easily, here‘s an example online.

Now, for painting over the widget.
This is fairly easy:

void VideoWidget::paintEvent(QPaintEvent * e) {
QPainter painter(this);
if(buf) {
QImage i = QImage::fromData(buf,len,"PPM");
painter.drawImage(QPoint(0,0),i);
}
}

Two things to note:

• The widget needs to be given the pointer to the video frame buffer (buf).
• The frame buffer needs to be in a PPM format. That means it needs to get a PPM header, which looks something like this: “P6n320 240n255n”, and then all the pixels in 3-byte per-pixel format (RGB24). You can see that i take care of that in the previous code block.

Finally we need to orchestrate this whole mess.
So in my GUI-screen class I do:

....
vt = new VideoThread();
connect(vt,SIGNAL(frameReady()),this,SLOT(updateVideoWidget()));
vt->start();
....

And:

void playMessage::updateVideoWidget() {
videoWidget->repaint(); //or update().
}

This will make the widget repaint on each frame ready.

Note:

• In this example I don’t take care of multi-threading issues. Since the GUI and the ffmpeg decoder threads share a memory buffer, I should probably have a mutex to protect it. It’s a classic producer-consumer problem.
• Performance wise, Qt’s paint mechanism is by far the worst way to go when displaying video… but it’s great for a quick-and-dirty solution (I only needed 5fps). A more performance favorable solution will probably be using an overlay block and frame-serving with SDL.

Enjoy!
Roy.