So first you’ll need to export your loved tracks from Last.fm in tab separated format – a mere button press.

The “tsv” (tab separated values) file has a simple format: <song name> <artist> <Last.fm url>

And now for the script, first, the loved tracks file is tab separated, so we use AWK to get the 2 first fields which are song-name and song-artist.
Then we use a neat command-line tool to download YT movies: http://rg3.github.com/youtube-dl/documentation.html.

mkdir mylovedtracks
cd mylovedtracks
awk -F\t '{print "../youtube-dl.py -f 18 -t \"ytsearch:" $1 " " $2 "\""}' ../my_lovedtracks.tsv | csh

The single-liner will download all the loved tracks from the tsv file into the current directory, given that youtube-dl.py & my_lovedtracks.tsv exist in the parent directory. -f 18 says it will download only MP4s and ytsearch says it will try to search YT for the term “song-name song-artist” and download the 1st result. The | csh says it will send this command AWK formatted into a new shell process.

The saved MP4 will be named after the name of the video, with addition of the YT hash string.

All the mp4s have been downloaded, so let’s batch convert them to mp3s:

mkdir sound
for f in *.mp4 ; do n=`echo $f | cut -d '.' -f1`; if [ ! -e sound/$n.mp3 ]; then `mplayer $n.mp4 -vc dummy -vo null -ao pcm:file=sound/temp.wav; lame -V2 sound/temp.wav sound/$n.mp3; rm sound/temp.wav`; fi ; done

This single-liner will extract audio from the mp4 into a PCM temp.wav file using MEncoder, and then convert to VBR MP3 using Lame.
You can run this command many times, as it checks if the file has not been converted yet. So you’re impatient (like me) on converting some of the MP4s before everything was downloaded – just run it, and later run it again.

Congrats, all your loved tracks were downloaded.

A few limitation to this method:
* Sometimes downloaded songs are not exactly what you wanted, especially specific versions. The search is arbitrary, and can’t be controlled too much.
* ID3 tags are non existent, although something can probably be done about that in the Lame encoding phase.
* Very high potential for parallelization that is unexploited. Mostly in the YT download phase, where YT pushes the first ~15% of the video very fast (I saw 1200Kb/s even), and then maintains a steady d/l rate to get the video downloaded by ~1:00 minute (may be as low as 50Kb/s). Downloading many videos at once could help.
* Still not a true single-liner, it is a two-step thing. But that can be done by modifying the 2nd step a bit and putting into the AWK print of the 1st step.
* MP3′s volume normalization – very important! else every songs sounds different and you must do vol-up vol-down all the time…

Still, did a nice quick job for me…

Roy.

I’d like to share a quick tip on rotating video files.

I’m always frustrated with taking videos with my phone. Single handedly it’s easiest to do it when the phone is upright and not in landscape mode. But the files are always saved in landscape mode, which makes them rotated when you watch.
Although there are plenty of GUI software to do it, using the command line is faster and can also be batched!

Using FFmpeg

This is the basic syntax
./ffmpeg -vf transpose=0 -i input.mp4 output.avi

Just using -vf transpose=0 to rotate 90 deg clockwise. If you get the “Unrecognized option ‘vf’” error, you need to configure & build ffmpeg with --enable-filters (or at least without --disable-filters), and check with ffmpeg -filters that you get them to show up.

Also, I always get very lousy video quality when using the plain vanilla settings. It turns out the problem is with the frame rate. If you leave it as-is the (default) mpeg compression makes a lot of I and P frames, and too few B frames. So set it up explicitly.

I ended up with
ffmpeg -vf transpose=0 -i input.mp4 -r 30 output.avi

Easy.

Update [3/1/11]: Actually just using -vf transpose=0 will flip the video horizontally as well, which is undesirable in some cases. To counter that I use: -vf "transpose=0,hflip=0", and it resolves the problem.

Using MEncoder

Again, pretty simple thing to do:
mencoder input.mp4 -nosound -o characters-resize-turn.avi -vf rotate=0 -ovc lavc -lavcopts vcodec=mpeg4 -ofps 30

Note that I kill the audio with -nosound, and again set the frame rate-ofps 30 or else I get the “duplicate frames” problem.

Enjoy
Roy.

Just a quick share of lessons learned about Android’s Frame-by-Frame animations. Some of the functionality is poorly documented, as many people point out, so the web is the only place for answers. Having looked for some answers to these questions and couldn’t find any – here’s what I found out myself.

Update [2/3/11]: A new post on this topic gives a more broad view of my experience.

Resetting Frame Animation

The API is a bit weird here, because the restart function is inside the setVisible function of AnimationDrawable. If you don’t restart and just do another start() on an already finished animation – it will just jump to the last frame. You must reset the animation before starting it.

However if you do setVisible(true,true) your animation will run twice!, so you must do setVisible(false,true);. This will reset the animation for a start() operation.

Preloading Animations

I got to preloading animations after understating that trying to fire an animation “right off the XML”, has a lag of over a second. This must be due to the inflation process. And I was looking for snappy smooth animations right away, without waiting for them to inflate, so naturally I needed to preload them. But how?

So I loaded the frame animation into an ImageView with android:visibility="gone", and “hijacked” its drawable when I needed to fire it:

The XML, just put it somewhere in your main xml so it will be inflated when the app starts.

<!-- Preload animations -->
<ImageView android:id="@+id/anim_preloaded_imgview"
	android:layout_width="wrap_content"
	android:layout_height="wrap_content"
	android:src="@drawable/myanim"
	android:visibility="gone" />

The Java:

ImageView animateHere = (ImageView) findViewById(R.id.animate_here);
animateHere.setImageDrawable(((ImageView)findViewById(R.id.anim_preloaded_imgview).getDrawable());
AnimationDrawable anim = (AnimationDrawable) animateHere.getDrawable();
anim.setVisible(false, true); //reset! see previous section
anim.start(); //now good to start, even if already fired before

Voila.

OutOfMemory expection? srsly?

So at the beginning I was getting a lot of OutOfMemory exceptions on the loading of the animations. Probably due to loading many big bitmaps into the memory. So I reduced the size of my PNGs to ~80,000 pixels (images are ~200×400), and then I was able to load as many as I wanted! Strange…
I didn’t really research what’s the limit, but using ~140,000 pixels did not fly.

In these cases mogrify is your friend. This will resize all images beginning with “myanim” to 200xWhatever keeping aspect ratio:

mogrify -resize 200 myanim*

That’s it for now
But tomorrow I will learn more. Android is surprising..

Please comment if you [dis]agree, and share your own experience!
Roy.

Here are the steps we took to bring this project to life.

The Idea

Our vision is to create a home monitoring and controlling system, that will enable tracking different sensors around the house and also control switches.The system should be centralized by a master controller, and also wireless so it will not need cumbersome wiring throughout the house. We would also like the system to be easily controlled by a PC, so visual information could be displayed to the user, as well as allow manual control of the electronic switches.

Such a system will be able to automatically:

• Turn off the garden lighting when the light outside is bright enough,
• Control the lawn watering system,
• Control air conditioning in the house according to the temperature,
• etc.

The Hardware

As I mentioned, we were given an LPC2148 education board [link], implemented by Embedded Artists, that boasts a 12Mhz ARM CPU, and many peripheral subsystems. Among the systems are: LCD screen, numerous LEDs, a LED matrix, a fan, analog dials, USB and UART I/O and more. The boards also has a port for connecting a ZigBee module to enable RF communication, but it doesn’t contain the actual module. Since we needed remote communication between our stations, we bought 3 XBee modules from Maxstream [link].

The Software

The LPC boards can be programmed very easily using tools provided by Embedded Artists, such as GCC with Newlib for compiling (both Win and Linux), and Flashmagic (for windows) or lpc21isp for loading the compiled program. We used these tools for programing the “embedded” part of the application, for the PC client we used Java with SWT and serial port connectivity (RxTx for Windows and Linux).

Embedded program

The embedded program on the LPC board has two parts: A master and A client. The system has only one master, and up to 3 clients. The master gathers information from the clients, and controls their behavior according to the user requests. For communication between the master and clients we created a communication protocol that has only a few simple messages:

• INIT – The master sends this request to initialize the connection between itself and a client.
• ACK – This is the response for every request.
• POLL – The clients answers this request with the data from all it’s sensors.
• OPERATE – The master commands the client to switch something on or off.
• ERROR – A general error response.

All commands / responses have a unified structure described here:

General struct:
 ___________________________________________ _____
|    OP   | To | From | <----- DATA ------> | EOM
|_________|____|______|_____________________|_____

Data:

POLL (response)
_ ____________ ______ ______ ________
 | Temperature| ADC0 | ADC1 | Button |
_|____________|______|______|________|

OPERATE
_ _________
 | BITMASK |
_|_________|

This way we had a very easy implementation of the communication module, since we always had to look for only 14 bytes on the wire.

Communication with ZigBee module

To jumpstart our implementation we used the examples provided by Embedded Artists for operating the ZigBee module [link]. The communication with the ZigBee module is on the UART1 port of the MCU. The example code takes care of opening the correct GPIO pins, setting the IRQ masks to enable interrupts and provides a very simple API for transmitting and receiving characters with the XBee over UART  (described in uart.h and uart.c files).

We took the XBee example code and expanded it to be able to recieve and send data between two stations. That means setting up each station’s module with it’s ID, address, channel and target address by AT commands [spec, see page. 28]: ATID, ATCH, ATMY and ATDL. Two other key features are putting the module into command mode (rather than transmit mode) to set the mentioned parameters, this is done by ‘+++’ to enter command mode and ATCN to exit. Once we had a decent framework to communicate between the stations, we started to build the logic.

The master station logic is like so:

1. Initialize XBee module.
2. INIT all client stations, and see which station answers – these will be our “up” stations.
3. Loop:
   1. POLL each “up” station in a loop.
   2. Take care of any OPERATE requests from the PC client.

This way, the client’s logic boils down to just looping, testing for any request and taking care of it. Both client and master share most of the code, so only the main process code is essentially different. The example code uses a framework called “Preemtive OS” to allow multitasking / processes (code was bundled in the examples).

Communication with PC

Communication between the master and PC client also required some lightweight “protocol”. The communication is again over UART (0 this time, 1 is used by XBee), only now the PC is doing a UART-over-USB with the board. The protocol we ended up with supports these features:

• Commands the PC wants the master to perform look like “m=<command>=<parameters>”, such commands can be:
  • “poll=<i>”, send a POLL to the station indexed i
  • “test=<i>”, send an INIT to the station indexed i
  • “toggle=<i>_<sw>_<onoff>”, set the switch sw on the station indexed i to onoff status
• Notifications the master would like to share with the user (PC) look like “pc=<notification>”, such notifications can be:
  • “up=<i>”, the station indexed i is up
  • “master_online”,
  • “temp=<i> <temp>”, the station indexed i has a temperature reading of temp
  • “e=<error message>”, error message from the master

This is an incomplete set of features, but it’s representative of the idea we want to present.

PC client program

The PC client we built using Java, so it would (and should) be portable between OSs. The GUI was built with the SWT, using the eclipse Visual Editor plug-in which simplified the process. Communication with the LPC board is done over the serial port of the board, as I mentioned the PC creates a virtual COM port using a USB-to-UART driver (FTDI, bundled with windows) [link].

To test the GUI we created a “simulator”, that mimics the operation of a master station, that is shown in the video above.

The high-level design of the program is described in this inheritance UML diagram:

Our PC client uses serial-port connectivity based on the old javax.serial APIs. Though these APIs have been abandoned by Sun, and there is no official Win32 implementation bundled with the JDK (only for *NIXs/Solaris), a project named RxTx is upkeeping an implementation of this API for windows [link].

The Code

We are releasing the code under the BSD license, for everyone to use, enjoy, learn and expand.

It is available via the blog’s SVN repo:

PC Client (requires RxTx serial port impl and SWT)

svn checkout http://morethantechnical.googlecode.com/svn/trunk/SmartHomePCClient

Embedded program (includes all dependencies)

svn checkout http://morethantechnical.googlecode.com/svn/trunk/smarthome_embedded/final project

To compile the master program “make” in the master directory, and you’ll get an “xbee_master.hex” file, same goes for the client in the client directory (these directories don’t contain code, only a makefile). Then you have to upload the hex into the board, this is done either by “make deploy” (if you have lpc21isp), or FlashMagic on Win.

Java is compiled as usual… just remember the dependencies.

We would like to thank Sivan Toledo for the guidance, loaned hardware and inspiration.

And thanks all for listening!
Roy S. & Gil Ramon

I like Qt. It’s cross-platform, a clear a nice API, straightforward, and remindes me somewhat of Apple’s Cocoa.

My intention is to get some serious face detection going on mobile devices. So that means either the iPhone, which so far did a crummy job performance-wise, or some other mobile device, preferably linux-based.
This led me to the decision to go with Qt. I believe you can get it to work on any linux-ish platform (limo, moblin, android), and since Nokia baught Trolltech – it’s gonna work on Nokia phones soon, awesome!

Lets get to the details, shall we?

First thing’s first: face detection.

I ripped OpenCV’s facedetect.c sample and extracted only the detect_and_draw() function. Originally the function detects the faces and draws a circle over them, but I needed only the face detection and the result bounding rectangle. So in the end I was left with this:

CvRect detect_and_draw( IplImage* img, CvMemStorage* storage, CvHaarClassifierCascade* cascade )
{
IplImage *gray, *small_img;
int i = 0;

gray = cvCreateImage( cvSize(img->width,img->height), 8, 1 );
small_img = cvCreateImage( cvSize( cvRound (img->width/scale),
cvRound (img->height/scale)), 8, 1 );

cvCvtColor( img, gray, CV_RGB2GRAY );
cvResize( gray, small_img, CV_INTER_LINEAR );
cvEqualizeHist( small_img, small_img );
cvClearMemStorage( storage );

CvRect* r = NULL;

if( cascade )
{
double t = (double)cvGetTickCount();
CvSeq* faces = cvHaarDetectObjects( small_img, cascade, storage,
1.1, 2, 0
|CV_HAAR_FIND_BIGGEST_OBJECT
//|CV_HAAR_DO_ROUGH_SEARCH
//|CV_HAAR_DO_CANNY_PRUNING
//|CV_HAAR_SCALE_IMAGE
,
cvSize(30, 30) );
t = (double)cvGetTickCount() - t;

printf( "detection time = %gms\n", t/((double)cvGetTickFrequency()*1000.) );

r = (CvRect*)cvGetSeqElem( faces, i );

cvReleaseImage( &gray );
cvReleaseImage( &small_img );

if(r) {
return cvRect(r->x,r->y,r->width,r->height);
} else {
return cvRect(-1,-1,0,0);
}

}

This can go anywhere in the code base, as it’s totally independant (as long as you train the cascade and allocate a MemStorage).Note that I am assuming only one face in the input image, and also that it will be the largest detected object. This bring my benchmark to about 25ms per frame, using the original general detection approach of facedetect.c benchmarked at about 160ms per frame.

OK, done with pure OpenCV, on to Qt.

I subclassed a QWidget, who’s sole purpose is to show the input video with the detected face. For starters, I needed to have a QImage and an IplImage instances as members, they can also share the same buffer (how awesome is that..). I also need a CvCapture, CvMemStorage and a CvHaarCalssifierCascade:

class FaceRecognizer : public QWidget
{
Q_OBJECT

public:
FaceRecognizer(QWidget *parent = 0);
~FaceRecognizer();

private:
Ui::FaceRecognizerClass ui;

QImage m_i;

QRect faceLoc;

CvMemStorage* storage;
CvHaarClassifierCascade* cascade;
CvCapture* capture;
IplImage* m_opencvimg;

QTimer* m_timer;

void paintEvent(QPaintEvent* e);

public slots:
void queryFrame();
};

You can see that I’m gonna use a QTimer to query the frames from the CvCapture and also I override paintEvent to paint the frame onto the canvas. In fact in my QWidget I have a QFrame, that the image will painted over it. UI was generated in Qt Designer.

First, some initialization in the constructor:

FaceRecognizer::FaceRecognizer(QWidget *parent)
: QWidget(parent)
{
ui.setupUi(this);

capture = cvCaptureFromAVI( "/home/user/Desktop/video.avi" );
//grab one frame to get width and height

IplImage* frame = cvQueryFrame( capture );

m_i = QImage(QSize(frame->width,frame->height),QImage::Format_RGB888);
ui.frame->setMinimumSize(m_i.width(),m_i.height());
ui.frame->setMaximumSize(ui.frame->minimumSize());
//create only the header, as the data buffer is shared, and was allocated by QImage

m_opencvimg = cvCreateImageHeader(cvSize(m_i.width(),m_i.height()),8,3);
m_opencvimg->imageData = (char*)m_i.bits(); // share buffers

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );

//images from cvQueryFrame come in BGR form and not what Qt expects - RGB

//and since the buffers are shared - format should be consistent
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

//we need memstorage and a cascade
storage = cvCreateMemStorage(0);
cascade = (CvHaarClassifierCascade*)cvLoad( CASCADE_NAME, 0, 0, 0 );

//set timer for 50ms intervals

m_timer = new QTimer(this);
connect(m_timer,SIGNAL(timeout()),this,SLOT(queryFrame()));
m_timer->start(50);
}

And now, querying the frame: query CvCapture, convert BGR to RGB, detect faces and update faceLoc QRect.

void FaceRecognizer::queryFrame() {
IplImage* frame = cvQueryFrame( capture );

if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, m_opencvimg, 0 );
else
cvFlip( frame, m_opencvimg, 0 );
cvCvtColor(m_opencvimg,m_opencvimg,CV_BGR2RGB);

CvRect r = detect_and_draw(m_opencvimg,storage,cascade);
faceLoc = QRect(QPoint(r.x,r.y),QSize(r.width,r.height));

this->update();
}

Finally - painting, which is easy:

void FaceRecognizer::paintEvent(QPaintEvent* e) {
QPainter painter(this);

painter.drawImage(QPoint(ui.frame->x(),ui.frame->y()),m_i);

if(faceLoc.x() > 0 && faceLoc.y() > 0) {
painter.setBrush(Qt::NoBrush);
painter.setPen(QColor(255,0,0));
painter.drawRect(QRect(faceLoc.x()+ui.frame->x(),faceLoc.y()+ui.frame->y(),faceLoc.width(),faceLoc.height()));
}
}

Looks like it’s all done… Here’s a video:

So anyway, I have my video files encoded in some unknown format and I need my program to show them in a some widget. I went around looking for an exiting example, but i couldn’t find anything concrete, except for a good tip here that led me here for an example of using ffmpeg’s libavformat and libavcodec, but no end-to-end example including the Qt code.

The ffmpeg example was simple enough to just copy-paste into my project, but the whole painting over the widget’s canvas was not covered. Turns out painting video is not as simple as overriding paintEvent()…

Firstly, you need a separate thread for grabbing frames from the video file, because you won’t let the GUI event thread do that.
That makes sense, but when the frame-grabbing thread (I called VideoThread) actually grabbed a frame and inserted it somewhere in the memory, I needed to tell the GUI thread to take that buffered pixels and paint them over the widget’s canvas.

This is the moment where I praise Qt’s excellent Signals/Slots mechanism. So I’ll have my VideoThread emit a signal notifying some external entity that a new frame is in the buffer.
Here’s a little code:

void VideoThread::run() {
/*
... Initialize libavformat & libavcodec data structures.
You can see it in the example i referred to before
*/
// Open video file
if(av_open_input_file(&pFormatCtx,
"lala.avi",
NULL, 0, NULL)!=0)
return -1; // Couldn't open file

// Retrieve stream information
if(av_find_stream_info(pFormatCtx)<0)
return -1; // Couldn't find stream information

// Find the first video stream ...

// Get a pointer to the codec context for the video
// stream...

// Find the decoder for the video stream...

// Open codec...

// Allocate video frame
pFrame=avcodec_alloc_frame();

// Allocate an AVFrame structure
pFrameRGB=avcodec_alloc_frame();
if(pFrameRGB==NULL)
return -1;

int dst_fmt = PIX_FMT_RGB24;
int dst_w = 160;
int dst_h = 120;

// Determine required buffer size and allocate buffer
numBytes = avpicture_get_size(dst_fmt, dst_w, dst_h);
buffer = new uint8_t[numBytes + 64];

//put a PPM header on the buffer
int headerlen = sprintf((char *) buffer,
"P6n%d %dn255n",
dst_w, dst_h);

_v->buf = (uchar*)buffer;
_v->len = avpicture_get_size(dst_fmt,dst_w,dst_h) +
headerlen;

// Assign appropriate parts of buffer to image planes
// in pFrameRGB...

// I use libswscale to scale the frames to the required
// size.
// Setup the scaling context:
SwsContext *img_convert_ctx;
img_convert_ctx = sws_getContext(
pCodecCtx->width, pCodecCtx->height,
pCodecCtx->pix_fmt,
dst_w, dst_h, dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);

// Read frames and notify
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0)
{
// Is this a packet from the video stream?
if(packet.stream_index==videoStream)
{
// Decode video frame
avcodec_decode_video(pCodecCtx,
pFrame,
&frameFinished,
packet.data,
packet.size);

// Did we get a video frame?
if(frameFinished)
{
// Convert the image to RGB
sws_scale(img_convert_ctx,
pFrame->data,
pFrame->linesize,
0,
pCodecCtx->height,
pFrameRGB->data,
pFrameRGB->linesize);

emit frameReady();

//My video is 5FPS so sleep for 200ms.
this->msleep(200);
}
}

// Free the packet that was allocated by
// av_read_frame
av_free_packet(&packet);
}

// Free the RGB image
delete [] buffer;
av_free(pFrameRGB);

// Free the YUV frame
av_free(pFrame);

// Close the codec...

// Close the video file...
} //end VideoThread::run

Ok so I have a frame-grabber that emits a frameReady signal everytime the buffer is full and ready for painting.

A couple of things to notice:

• I convert the image format to PIX_FMT_RGB24 (avcodec.h), which is required by Qt’s QImage::fromData() method.
• I scale the image using ffmpeg’s libswscale. All conversion/scaling methods inside libavcodev are deprecated now.
  But it’s fairly simple, here‘s a good example. Just remember you need a sws_getContext and then sws_scale.
• I totally disregard actual frame rate here, I just sleep for 200ms because i know my file is 5FPS. For a (far-) more sophisticated way to get the FPS, very important if this is not a constant frame-rate video, you can find here.
• I don’t cover audio in this example, although the mechanism to extract it from the file exists… you just need to grabe the audio stream’s frame. For playing audio you also need some Qt-external library. In a different project I used SDL very easily, here‘s an example online.

Now, for painting over the widget.
This is fairly easy:

void VideoWidget::paintEvent(QPaintEvent * e) {
QPainter painter(this);
if(buf) {
QImage i = QImage::fromData(buf,len,"PPM");
painter.drawImage(QPoint(0,0),i);
}
}

Two things to note:

• The widget needs to be given the pointer to the video frame buffer (buf).
• The frame buffer needs to be in a PPM format. That means it needs to get a PPM header, which looks something like this: “P6n320 240n255n”, and then all the pixels in 3-byte per-pixel format (RGB24). You can see that i take care of that in the previous code block.

Finally we need to orchestrate this whole mess.
So in my GUI-screen class I do:

....
vt = new VideoThread();
connect(vt,SIGNAL(frameReady()),this,SLOT(updateVideoWidget()));
vt->start();
....

And:

void playMessage::updateVideoWidget() {
videoWidget->repaint(); //or update().
}

This will make the widget repaint on each frame ready.

Note:

• In this example I don’t take care of multi-threading issues. Since the GUI and the ffmpeg decoder threads share a memory buffer, I should probably have a mutex to protect it. It’s a classic producer-consumer problem.
• Performance wise, Qt’s paint mechanism is by far the worst way to go when displaying video… but it’s great for a quick-and-dirty solution (I only needed 5fps). A more performance favorable solution will probably be using an overlay block and frame-serving with SDL.

Enjoy!
Roy.