Nov 13 2011

Just a simple Laplacian pyramid blender using OpenCV [w/code]

Published by at 9:39 am under code,graphics,opencv,programming

I want to share a small piece of code to do Laplacian Blending using OpenCV. It's one of the most basic and canonical methods of image blending, and is a must exercise for any computer graphics student.

Well basically it's a matter of creating two Laplacian pyramids of both images, and a Gaussian pyramid of the mask.
Then we blend the pyramids into one, and collapse the resulting pyramid into the blended image.

#include "opencv2/opencv.hpp"

using namespace cv;

class LaplacianBlending {
private:
	Mat_<Vec3f> left;
	Mat_<Vec3f> right;
	Mat_<float> blendMask;
	
	vector<Mat_<Vec3f> > leftLapPyr,rightLapPyr,resultLapPyr;
	Mat leftSmallestLevel, rightSmallestLevel, resultSmallestLevel;
	vector<Mat_<Vec3f> > maskGaussianPyramid; //masks are 3-channels for easier multiplication with RGB
	
	int levels;
	
	
	void buildPyramids() {
		buildLaplacianPyramid(left,leftLapPyr,leftSmallestLevel);
		buildLaplacianPyramid(right,rightLapPyr,rightSmallestLevel);
		buildGaussianPyramid();
	}
	
	void buildGaussianPyramid() {
		assert(leftLapPyr.size()>0);
		
		maskGaussianPyramid.clear();
		Mat currentImg;
		cvtColor(blendMask, currentImg, CV_GRAY2BGR);
		maskGaussianPyramid.push_back(currentImg); //highest level
		
		currentImg = blendMask;
		for (int l=1; l<levels+1; l++) {
			Mat _down; 
			if (leftLapPyr.size() > l) {
				pyrDown(currentImg, _down, leftLapPyr[l].size());
			} else {
				pyrDown(currentImg, _down, leftSmallestLevel.size()); //smallest level
			}

			Mat down; 
			cvtColor(_down, down, CV_GRAY2BGR);
			maskGaussianPyramid.push_back(down);
			currentImg = _down;
		}
	}
	
	void buildLaplacianPyramid(const Mat& img, vector<Mat_<Vec3f> >& lapPyr, Mat& smallestLevel) {
		lapPyr.clear();
		Mat currentImg = img;
		for (int l=0; l<levels; l++) {
			Mat down,up;
			pyrDown(currentImg, down);
			pyrUp(down, up, currentImg.size());
			Mat lap = currentImg - up;
			lapPyr.push_back(lap);
			currentImg = down;
		}
		currentImg.copyTo(smallestLevel);
	}
	
	Mat_<Vec3f> reconstructImgFromLapPyramid() {
		Mat currentImg = resultSmallestLevel;
		for (int l=levels-1; l>=0; l--) {
			Mat up;
						
			pyrUp(currentImg, up, resultLapPyr[l].size());
			currentImg = up + resultLapPyr[l];
		}
		return currentImg;
	}
	
	void blendLapPyrs() {
		resultSmallestLevel = leftSmallestLevel.mul(maskGaussianPyramid.back()) + 
									rightSmallestLevel.mul(Scalar(1.0,1.0,1.0) - maskGaussianPyramid.back());
		for (int l=0; l<levels; l++) {
			Mat A = leftLapPyr[l].mul(maskGaussianPyramid[l]);
			Mat antiMask = Scalar(1.0,1.0,1.0) - maskGaussianPyramid[l];
			Mat B = rightLapPyr[l].mul(antiMask);
			Mat_<Vec3f> blendedLevel = A + B;
			
			resultLapPyr.push_back(blendedLevel);
		}
	}

public:
	LaplacianBlending(const Mat_<Vec3f>& _left, const Mat_<Vec3f>& _right, const Mat_<float>& _blendMask, int _levels):
	left(_left),right(_right),blendMask(_blendMask),levels(_levels) 
	{
		assert(_left.size() == _right.size());
		assert(_left.size() == _blendMask.size());
		buildPyramids();
		blendLapPyrs();
	};
	
	Mat_<Vec3f> blend() {
		return reconstructImgFromLapPyramid();
	}	
};

Mat_<Vec3f> LaplacianBlend(const Mat_<Vec3f>& l, const Mat_<Vec3f>& r, const Mat_<float>& m) {
	LaplacianBlending lb(l,r,m,4);
	return lb.blend();
}

int main(int argc, char** argv) {
   Mat l8u = imread("left.png");
   Mat r8u = imread("right.png");
   Mat_<Vec3f> l; l8u.convertTo(l,CV_32F,1.0/255.0);
   Mat_<Vec3f> r; r8u.convertTo(r,CV_32F,1.0/255.0);

   Mat_<float> m(l.rows,l.cols,0.0);
   m(Range::all(),Range(0,m.cols/2)) = 1.0;

   Mat_<Vec3f> blend = LaplacianBlend(l, r, m);
   imshow("blended",blend);
   waitKey(0);
}

To use, simply call the function LaplacianBlend with your two images and your mask, and the result will be returned.
Here's something I did with it:

Enjoy
Roy.

Share

13 responses so far

13 Responses to “Just a simple Laplacian pyramid blender using OpenCV [w/code]”

  1. D3vilJinon 04 Dec 2011 at 9:47 am

    Hi,
    I tried using the code,
    and i get an exception on this line:
    cvtColor(blendMask, currentImg, CV_GRAY2BGR);

    any idea what could cause the problem?
    i'm kinda new to open cv

  2. Royon 04 Dec 2011 at 5:20 pm

    @D3vilJin
    I can't help without knowing what is exception..

  3. Paulo Almeidaon 30 Dec 2011 at 10:14 am

    Hi,
    I'm currently experimenting with your code and I have a few questions if you could answer:

    does your class accept different kinds of images like CV_8UC3 and CV_32FC3;
    and if not, what are the relevant portions in the code where the image being a certain type is essential?

    I will be grateful if you can help me with this.

  4. Royon 31 Dec 2011 at 9:37 am

    @Paulo Almeida
    It actually only likes Mat_ which is CV_32FC3
    You can make a simple conversion from CV_8UC3 to CV_32FC3 in the main function LaplacianBlend, or simply create another LaplacianBlend function that accepts Mat_ (CV_8UC3)

    It would be nice if the whole thing will be templated, and then accept any kind of image without conversion

  5. niko cavon 10 Feb 2012 at 7:22 pm

    can you show me how I can call this in the main function? thanks.. I'am getting assertion failure.. So i was wondering, maybe there is a mistake on what I am doing.. Thanks and regards.

  6. jackon 30 Apr 2012 at 10:00 pm

    can you show how to use this in main, im getting assertion failure!
    hope to hear something soon! thank you!

  7. Royon 30 Apr 2012 at 10:52 pm

    @jack
    I've added a small main to the code excerpt.

  8. jackon 02 May 2012 at 11:40 pm

    Thank you! when i run the program the command line window pops up when the code is at pyrDown(currentImg, down), i guess it just doesn't do anything with it and just exits.. could that be because I am using opencv 2.2 ?

  9. k_deeon 30 May 2012 at 4:50 pm

    Even i'am facing the same problem as jack is facing,i'am also using open CV 2.2.
    I would be highly thankful if you could help me out with it.

  10. Royon 31 May 2012 at 4:57 am

    @k_dee @jack
    To get a better idea of the problem I would need some debug info.. like an exception thrown, etc.
    Anyway I recommend updating to OpenCV 2.3+ (2.4 is latest at the moment)

    R.

  11. ReinMon 11 Jul 2012 at 5:43 pm

    Hi,
    I've been using your code and it works perfectly. I have a doubt, when I have an image where the part of interest is sorrounded by black pixels I got a frame of strong Laplacian lines that I wouldn't like to have (blending doesn't look well). How do I manage this situation ?. Thanks for your help.

  12. Dirhemon 17 Apr 2013 at 9:43 pm

    I get this error, can you tell me how to fix it?

    here is the line of your code cause the error

    Mat_ l; l8u.convertTo(l,CV_32F,1.0/255.0);

    Here is the error message:

    OpenCV Error: Assertion failed (!fixedType() || ((Mat*)obj)->type() == mtype) in unknown function.

    PS: I am using Visual Studio 2012 and OpenCV 2.4.4

  13. panovron 17 Nov 2013 at 1:33 pm

    Hi,
    thanks for code sharing!
    May you also supply the three images for experimenting?
    Thanks!

Trackback URI | Comments RSS

Leave a Reply