Packing Better Montages than ImageMagick with Python Rect Packer

ImageMagick has a built in Montage creating tool. It's good enough for casual montaging, but it's definitely suboptimal for packing varying size images.

All photos from: https://unsplash.com/collections/1199299/fun-with-fall-(thanksgiving%2C-autumn)

Simply using ImageMagick's montage it looks something the following. First the script that I run:

TEMP_DIRECTORY=$(mktemp -d /tmp/montageXXXXXX)
/usr/local/bin/mogrify -path ${TEMP_DIRECTORY}/ -geometry 480x480\> "$@"
/usr/local/bin/montage ${TEMP_DIRECTORY}/* -geometry +2+2 "$( dirname "$1" )"/montage.jpg

First I rescale all the images to "up-to 480x480" keeping aspect ratio, and then run the montage with a 2x2 pixel border.

Original images (just scaled down)

This looks pretty bad. Mostly because montage will not pack the rectangles more densely.

We could first resize all the images so that their height is e.g. 480px:

for f in "$@"
do
	/usr/local/bin/convert "$f" -geometry x480 "${f%.*}_480h.jpg"
done

And then running montage, to get this:

Images resized to height=480px

Already looking much better, but we have little control over the resulting size of the montage, ImageMagick just does its best job at packing everything. With similar heights - it's an easy job. However we can still see a lot of annoying whitespace on the right. What if there's a better way to pack the images?

Enter, rectpack: https://github.com/secnot/rectpack

This is a Python package implementing a few algorithms for rectangle packing, a concrete spatial instance of the classic knapsack problem (NP complete!) from computer science: https://en.wikipedia.org/wiki/Knapsack_problem

Here's my script:

import cv2
import rpack
import os
import glob
from rectpack import newPacker
import pickle
import numpy as np
import argparse

parser = argparse.ArgumentParser(description='Montage creator with rectpack')
parser.add_argument('--width', help='Output image width', default=5200, type=int)
parser.add_argument('--aspect', help='Output image aspect ratio, \
    e.g. height = <width> * <aspect>', default=1.0, type=float)
parser.add_argument('--output', help='Output image name', default='output.png')
parser.add_argument('--input_dir', help='Input directory with images', default='./')
parser.add_argument('--debug', help='Draw "debug" info', default=False, type=bool)
parser.add_argument('--border', help='Border around images in px', default=2, type=int)
args = parser.parse_args()

files = sum([glob.glob(os.path.join(args.input_dir, '*.' + e)) for e in ['jpg', 'jpeg', 'png']], [])
print('found %d files in %s' % (len(files), args.input_dir))

print('getting images sizes...')
sizes = [(im_file, cv2.imread(im_file).shape) for im_file in files]

# NOTE: you could pick a different packing algo by setting pack_algo=..., e.g. pack_algo=rectpack.SkylineBlWm
packer = newPacker(rotation=False)
for i, r in enumerate(sizes):
    packer.add_rect(r[1][1] + args.border * 2, r[1][0] + args.border * 2, rid=i)

out_w = args.width
aspect_ratio_wh = args.aspect
out_h = int(out_w * aspect_ratio_wh)

packer.add_bin(out_w, out_h)

print('packing...')
packer.pack()

output_im = np.full((out_h, out_w, 3), 255, np.uint8)

used = []

for rect in packer.rect_list():
    b, x, y, w, h, rid = rect

    used += [rid]

    orig_file_name = sizes[rid][0]
    im = cv2.imread(orig_file_name, cv2.IMREAD_COLOR)
    output_im[out_h - y - h + args.border : out_h - y - args.border, x + args.border:x+w - args.border] = im
    if args.debug:
        cv2.rectangle(output_im, (x,out_h - y - h), (x+w,out_h - y), (255,0,0), 3)
        cv2.putText(output_im, "%d"%rid, (x, out_h - y), cv2.FONT_HERSHEY_PLAIN, 3.0, (0,0,255), 2)

print('used %d of %d images' % (len(used), len(files)))

print('writing image output %s:...' % args.output)
cv2.imwrite(args.output, output_im)

print('done.')

Running it like so:

$ python3 pack.py --input_dir ~/Downloads/montage/resize480/ --width 2200 --border 10 --debug True

Resulted in this:

Montage with rectpack

That doesn't look the best, but it's definitely nice it tries to tile things together.

There are some options to consider:

$ python3 pack.py --help
usage: pack.py [-h] [--width WIDTH] [--aspect ASPECT] [--output OUTPUT]
               [--input_dir INPUT_DIR] [--debug DEBUG] [--border BORDER]

Montage creator with rectpack

optional arguments:
  -h, --help            show this help message and exit
  --width WIDTH         Output image width
  --aspect ASPECT       Output image aspect ratio, e.g. height = <width> *
                        <aspect>
  --output OUTPUT       Output image name
  --input_dir INPUT_DIR
                        Input directory with images
  --debug DEBUG         Draw "debug" info
  --border BORDER       Border around images in px

Running over the fixed height images:

$ python3 pack.py --input_dir ~/Downloads/montage/h480/ --width 4800 --aspect 0.5 --border 5 --debug True

Or:

$ python3 pack.py --input_dir ~/Downloads/montage/h480/ --width 2500 --aspect 1.2 --border 5

This gives us more control of the montage.

Enjoy!
Roy.

Share

Mastering OpenCV 4 - my new book!

mastering opencv4
I'm very excited to announce the publication of my latest Mastering OpenCV book!

With many new chapters and all the others re-written practically from scratch, this edition is by far the best ever.

The excellent David Millán Escrivá and I go deep and wide across the range of capabilities of OpenCV, explaining the theory and implementing recent real-world vision tasks from the ground up.
It's been baking for many months in the oven, rising slowly, and finally ready for consumption... yum!

The sources are free to grab: https://github.com/PacktPublishing/Mastering-OpenCV-4-Third-Edition

And copies are available on

Amazon: https://amzn.to/2Ff1mmE

Packt: https://www.packtpub.com/application-development/mastering-opencv-4-third-edition?utm_source=github&utm_medium=repository&utm_campaign=9781789533576

Enjoy reading!

Share

Cylindrical image warping for panorama stitching

Hey-o
Just sharing a code snippet to warp images to cylindrical coordinates, in case you're stitching panoramas in Python OpenCV...

This is an improved version from what I had in class some time ago... http://hi.cs.stonybrook.edu/cse-527
It runs VERY fast. No loops involved, all matrix operations. In C++ this code would look gnarly.. Thanks Numpy!

Enjoy!
Roy

Share

Aligning faces with py opencv-dlib combo

Face alignment with Dlib and OpenCV

This is my first trial at using Jupyter notebook to write a post, hope it makes sense.

I've recently taught a class on generative models: http://hi.cs.stonybrook.edu/teaching/cdt450

In class we've manipulated face images with neural networks.

One important thing I found that helped is to align the images so the facial features overlap.
It helps the nets learn the variance in faces better, rather than waste their "representation power" on the shift between faces.

The following is some code to align face images using the excellent Dlib (python bindings) http://dlib.net. First I'm just using a standard face detector, and then using the facial fatures extractor I'm using that information for a complete alignment of the face.

After the alignment - I'm just having fun with the aligned dataset 🙂
Continue reading "Aligning faces with py opencv-dlib combo"

Share

Projector-Camera Calibration - the "easy" way

First let me open by saying projector-camera calibration is NOT EASY. But it's technically not complicated too.

It is however, an amalgamation of optimizations that accrue and accumulate error with each step, so that the end product is not far from a random guess.
So 3D reconstructions I was able to get from my calibrated pro-cam were just a distorted mess of points.

Nevertheless, here come the deets.
Continue reading "Projector-Camera Calibration - the "easy" way"

Share

Revisiting graph-cut segmentation with SLIC and color histograms [w/Python]

As part of the computer vision class I'm teaching at SBU I asked students to implement a segmentation method based on SLIC superpixels. Here is my boilerplate implementation.

This follows the work I've done a very long time ago (2010) on the same subject.

For graph-cut I've used PyMaxflow: https://github.com/pmneila/PyMaxflow, which is very easily installed by just pip install PyMaxflow

The method is simple:

  • Calculate SLIC superpixels (the SKImage implementation)
  • Use markings to determine the foreground and background color histograms (from the superpixels under the markings)
  • Setup a graph with a straightforward energy model: Smoothness term = K-L-Div between superpix histogram and neighbor superpix histogram, and Match term = inf if marked as BG or FG, or K-L-Div between SuperPix histogram and FG and BG.
  • To find neighbors I've used Delaunay tessellation (from scipy.spatial), for simplicity. But a full neighbor finding could be implemented by looking at all the neighbors on the superpix's boundary.
  • Color histograms are 2D over H-S (from the HSV)

Result

Share

OMG CMake/OpenCV3 can you be more difficult? Linking order problems with OpenNI2...

So I just spent 1.5 hours figuring this out.
Compiling an example on Ubuntu 16.04 with OpenCV built from scratch with OpenNI2 support.
(OpenNI2 is from Orbbec, but that doesn't make any difference: https://orbbec3d.com/develop/)

When using this straightforward CMake script for compilation - it doesn't work:

cmake_minimum_required(VERSION 3.2)
project(MyApp)

find_package(OpenCV 3 REQUIRED)

set(OPENNI2_LIBS "OpenNI2")
link_directories("/home/user/Downloads/2-Linux/OpenNI-Linux-x64-2.3/Redist")

add_executable(myapp main.cpp)
target_link_libraries(myapp ${OpenCV_LIBS} ${OPENNI2_LIBS})

Complains of undefined references:

/usr/bin/c++   -g   CMakeFiles/myapp.dir/main.cpp.o  -o myapp  -L/home/user/Downloads/2-Linux/OpenNI-Linux-x64-2.3/Redist -rdynamic -lOpenNI2 /usr/local/lib/libopencv_shape.so.3.2.0 /usr/local/lib/libopencv_stitching.so.3.2.0 /usr/local/lib/libopencv_superres.so.3.2.0 /usr/local/lib/libopencv_videostab.so.3.2.0 /usr/local/lib/libopencv_objdetect.so.3.2.0 /usr/local/lib/libopencv_calib3d.so.3.2.0 /usr/local/lib/libopencv_features2d.so.3.2.0 /usr/local/lib/libopencv_flann.so.3.2.0 /usr/local/lib/libopencv_highgui.so.3.2.0 /usr/local/lib/libopencv_ml.so.3.2.0 /usr/local/lib/libopencv_photo.so.3.2.0 /usr/local/lib/libopencv_video.so.3.2.0 /usr/local/lib/libopencv_videoio.so.3.2.0 /usr/local/lib/libopencv_imgcodecs.so.3.2.0 /usr/local/lib/libopencv_imgproc.so.3.2.0 /usr/local/lib/libopencv_core.so.3.2.0 -Wl,-rpath,/home/user/Downloads/2-Linux/OpenNI-Linux-x64-2.3/Redist:/usr/local/lib 
/usr/local/lib/libopencv_videoio.so.3.2.0: undefined reference to `oniStreamGetProperty'
/usr/local/lib/libopencv_videoio.so.3.2.0: undefined reference to `oniRecorderDestroy'
/usr/local/lib/libopencv_videoio.so.3.2.0: undefined reference to `oniDeviceIsCommandSupported'
/usr/local/lib/libopencv_videoio.so.3.2.0: undefined reference to `oniDeviceSetProperty'

You'll notice that -lOpenNI2 does indeed appear for correct linking.
The linker doesn't complain that lib was not found - it just misses the references.
This lead me to understand it's a linking order problem (after ~45 minutes of banging my head vs. the keyboard and swearing profusely).

Some more swearing and head banging got me to understand that CMake is messing around with the link order.
So even if try:

target_link_libraries(myapp ${OpenCV_LIBS} ${OPENNI2_LIBS} ${OpenCV_LIBS} ${OPENNI2_LIBS})

i.e. making the order effectively meaningless -- it still doesn't work!

More swearing and head banging, another ~40 minutes passed, and I figured out a solution.
The real solution is to slap someone in CMake in the face with a trout, but here's a solution to my problem:

find_package(OpenCV 3 REQUIRED core highgui videoio) # ORDER MATTERS!!! videoio must be last!
set(OpenCV_LIBS "${OpenCV_LIBS};OpenNI2") #add openni2 at the end (although cmake doesn't keep order anyway)
target_link_libraries(myapp ${OpenCV_LIBS})

Now it compiles.

And look at the make VERBOSE=1:

/usr/bin/c++   -g   CMakeFiles/myapp.dir/main.cpp.o  -o myapp  -L/home/user/Downloads/2-Linux/OpenNI-Linux-x64-2.3/Redist -rdynamic /usr/local/lib/libopencv_highgui.so.3.2.0 /usr/local/lib/libopencv_videoio.so.3.2.0 -lOpenNI2 /usr/local/lib/libopencv_core.so.3.2.0 -Wl,-rpath,/home/user/Downloads/2-Linux/OpenNI-Linux-x64-2.3/Redist:/usr/local/lib -Wl,-rpath-link,/usr/local/lib 

Can you see how highgui and videoio are before OpenNI2, and core is after?
Why? Whhhhhhy?

The key is to get OpenNI to be linked in order after videoio.

OMG CMake, OMG OpenCV, OMG you gaiz, W-T-F?

Update:
This method breaks down as soon as more OpenCV components are added. The order goes haywire again, and OpenNI2 comes before videoio, which breaks the link.
As of now the way I can compile it is like so:

set(LINK_LIBS /usr/local/lib/libopencv_core.so.3.2
              /usr/local/lib/libopencv_highgui.so.3.2
              /usr/local/lib/libopencv_videoio.so.3.2
              /usr/local/lib/libopencv_imgproc.so.3.2
              /usr/local/lib/libopencv_calib3d.so.3.2
              OpenNI2)
Share

New edition to the Mastering OpenCV book - now with OpenCV3!

Mastering OpenCV 3
I'm happy to announce that the new edition of Mastering OpenCV is out!
You can get it on Amazon: Mastering OpenCV 3

It brings up most of the older OpenCV2 book projects to OpenCV3, including my Toy-SfM (or "Exploring SfM") project.

A lot has happened in the OpenCV3 APIs with respect to Structure from Motion.
It got much easier!
The book chapter on SfM is a gentle introduction to the subject, that focuses on coding and the core concepts, while abstracting on the math.

Thanks for listening!
Roy

Share