Implementing PTAM: stereo, tracking and pose estimation for AR with OpenCV [w/ code]

Hi

Been working hard at a project for school the past month, implementing one of the more interesting works I've seen in the AR arena: Parallel Tracking and Mapping (PTAM) [PDF]. This is a work by George Klein [homepage] and David Murray from Oxford university, presented in ISMAR 2007.

When I first saw it on youtube [link] I immediately saw the immense potential - mobile markerless augmented reality. I thought I should get to know this work a bit more closely, so I chose to implement it as a part of advanced computer vision course, given by Dr. Lior Wolf [link] at TAU.

The work is very extensive, and clearly is a result of deep research in the field, so I set to achieve a few selected features: Stereo initialization, Tracking, and small map upkeeping. I chose not to implement relocalization and full map handling.

This post is kind of a tutorial for 3D reconstruction with OpenCV 2.0. I will show practical use of the functions in cvtriangulation.cpp, which are not documented and in fact incomplete. Furthermore I'll show how to easily combine OpenCV and OpenGL for 3D augmentations, a thing which is only briefly described in the docs or online.

Here are the step I took and things I learned in the process of implementing the work.

Update: A nice patch by yazor fixes the video mismatching - thanks! and also a nice application by Zentium called "iKat" is doing some kick-ass mobile markerless augmented reality.
Continue reading "Implementing PTAM: stereo, tracking and pose estimation for AR with OpenCV [w/ code]"

Share

iPhone OS 3.x Raw data of camera frames

Hi All

It looks like it's finally here - a way to grab the raw data of the camera frames on the iPhone OS 3.x.

Update: Apple officially supports this in iOS 4.x using AVFoundation, here's sample code from Apple developer.

A gifted hacker named John DeWeese was nice enough to comment on a post from May 09' with his method of hacking the APIs to get the frames. Though cumbersome, it looks like it should work, but I haven't tried it yet. I promise to try it soon and share my results.

Way to go John!
Some code would be awesome...

Roy.

Share

Near realtime face detection on the iPhone w/ OpenCV port [w/code,video]

iphone + opencv = winHi
OpenCV is by far my favorite CV/Image processing library. When I found an OpenCV port to the iPhone, and even someone tried to get it to do face detection, I just had to try it for myself.

In this post I'll try to run through the steps I took in order to get OpenCV running on the iPhone, and then how to get OpenCV's face detection play nice with iPhoneOS's image buffers and video feed (not yet OS 3.0!). Then i'll talk a little about optimization

Update: Apple officially supports camera video pixel buffers in iOS 4.x using AVFoundation, here's sample code from Apple developer.
Update: I do not have the xcodeproj file for this project, please don't ask for it. Please see here for compiling OpenCV for the iPhone SDK 4.3.

Let's begin
Continue reading "Near realtime face detection on the iPhone w/ OpenCV port [w/code,video]"

Share

Porting Rob Hess's SIFT impl. to Java

beavers_siftThis is a Java port of Rob Hess' implementation of SIFT that I did for a project @ work.

However, I couldn't port the actual extraction of SIFT descriptors from images as it relies very heavily on OpenCV. So actually all that I ported to native Java is the KD-Tree features matching part, and the rest is in JNI calls to Rob's code.

I wrote this more as a tutorial to Rob's work, with an easy JNI interface to Java.

You can find the sources here: http://www.morethantechnical.com/extupload/code/JavaSIFT.zip

Here's how to use it:
Continue reading "Porting Rob Hess's SIFT impl. to Java"

Share

Combining Java's BufferedImage and OpenCV's IplImage

java_opencv_imgHi

I recently did a small project combining a Java web service with a OpenCV processing. I tried to transfer the picture from Java environment (as BufferedImage) to OpenCV (IplImage) as seamlessly as possible. This proved a but tricky, especially the Java part where you need to create your own buffer for the image, but it worked out nicely.

Let me show you how I did it

Continue reading "Combining Java's BufferedImage and OpenCV's IplImage"

Share