3d Augmented Reality code graphics opencv python video vision

Projector-Camera Calibration – the "easy" way

Projector-camera calibration with OpenCV-Python

First let me open by saying projector-camera calibration is NOT EASY. But it’s technically not complicated too.
It is however, an amalgamation of optimizations that accrue and accumulate error with each step, so that the end product is not far from a random guess.
So 3D reconstructions I was able to get from my calibrated pro-cam were just a distorted mess of points.
Nevertheless, here come the deets.

I based my method on something I saw randomly on YouTube:
And actually the code they provide checks out: and is not much dissimilar to mine.
Although I’ve only seen the code after I wrote mine.
There’s also the 2009 SIGGRAPH course by Doug Lanman:
First off, we need a calibration board that has a ChARUco board pasted on it.

One half of it will be used for the camera and the other half for the projector to project on.
As you can see, we’re being scrappy and simply used a cut up corrugated-cardboard box. It’s just something to reflect the projector’s pattern.
Then we display a circles pattern image with the projector, and take a video from the camera moving the board around.

Save the video to a file.
The worst thing about computer vision work with a camera is to actually work with a live feed. Always save your data to a file and process it offline. This way you can do it at 4am, too. Works out when you have kids.
First step is to calibrate the camera, easily done with ARUco, which is now baked into OpenCV 3.3+ as a contrib module.

        # --------- detect ChAruco board -----------
        corners, ids, rejected = cv2.aruco.detectMarkers(frame, cb.dictionary)
        corners, ids, rejected, recovered = cv2.aruco.refineDetectedMarkers(frame, cb, corners, ids, rejected, cameraMatrix=K, distCoeffs=dist_coef)
        if corners == None or len(corners) == 0:
        ret, charucoCorners, charucoIds = cv2.aruco.interpolateCornersCharuco(corners, ids, frame, cb)
        charucoCornersAccum += [charucoCorners]
        charucoIdsAccum += [charucoIds]
        if number_charuco_views == 40:
            print("calibrate camera")
            print("camera calib mat before\n%s"%K)
            # calibrate camera
            ret, K, dist_coef, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco(charucoCornersAccum,
                                                                               (w, h),
                                                                               flags = cv2.CALIB_USE_INTRINSIC_GUESS)
            print("camera calib mat after\n%s"%K)
            print("camera dist_coef %s"%dist_coef.T)
            print("calibration reproj err %s"%ret)

An that looks like this:

Sometimes it misses out on the board, but it’s far more robust than the regular chessboard…
After 40 frames we calibrate the camera.
Moving on to the projector, we detect the circles in the camera and ray-plane intersect for the 3D positions using the ChAruco board transform.

        # --------- detect circles -----------
        ret, circles = cv2.findCirclesGrid(gray, circles_grid_size, flags=cv2.CALIB_CB_SYMMETRIC_GRID)
        img = cv2.drawChessboardCorners(img, circles_grid_size, circles, ret)
        # ray-plane intersection: circle-center to chessboard-plane
        circles3D = intersectCirclesRaysToBoard(circles, rvec, tvec, K, dist_coef)
        # re-project on camera for verification
        circles3D_reprojected, _ = cv2.projectPoints(circles3D, (0,0,0), (0,0,0), K, dist_coef)
        for c in circles3D_reprojected:
  , tuple(c.astype(np.int32)[0]), 3, (255,255,0), cv2.FILLED)

It looks like so:

Here’s the ray-plane intersect:

def intersectCirclesRaysToBoard(circles, rvec, t, K, dist_coef):
    circles_normalized = cv2.convertPointsToHomogeneous(cv2.undistortPoints(circles, K, dist_coef))
    if not rvec.size:
        return None
    R, _ = cv2.Rodrigues(rvec)
    plane_normal = R[2,:] # last row of plane rotation matrix is normal to plane
    plane_point = t.T     # t is a point on the plane
    epsilon = 1e-06
    circles_3d = np.zeros((0,3), dtype=np.float32)
    for p in circles_normalized:
        ray_direction = p / np.linalg.norm(p)
        ray_point = p
        ndotu =
        if abs(ndotu) < epsilon:
            print ("no intersection or line is within plane")
        w = ray_point - plane_point
        si = / ndotu
        Psi = w + si * ray_direction + plane_point
        circles_3d = np.append(circles_3d, Psi, axis = 0)
    return circles_3d

Once we have 3D points from the ray-plane intersection, we have what we need for stereo calibration: 2D camera points, 2D projector points and 3D points.

            # calibrate projector
            print("calibrate projector")
            print("proj calib mat before\n%s"%K_proj)
            ret, K_proj, dist_coef_proj, rvecs, tvecs = cv2.calibrateCamera(objectPointsAccum,
                                                                            (w_proj, h_proj),
                                                                            flags = cv2.CALIB_USE_INTRINSIC_GUESS)
            print("proj calib mat after\n%s"%K_proj)
            print("proj dist_coef %s"%dist_coef_proj.T)
            print("calibration reproj err %s"%ret)
            print("stereo calibration")
            ret, K, dist_coef, K_proj, dist_coef_proj, proj_R, proj_T, _, _ = cv2.stereoCalibrate(
                    flags = cv2.CALIB_USE_INTRINSIC_GUESS
            proj_rvec, _ = cv2.Rodrigues(proj_R)
            print("R \n%s"%proj_R)
            print("T %s"%proj_T.T)
            print("proj calib mat after\n%s"%K_proj)
            print("proj dist_coef %s"       %dist_coef_proj.T)
            print("cam calib mat after\n%s" %K)
            print("cam dist_coef %s"        %dist_coef.T)
            print("reproj err %f"%ret)

Here’s what we get:

On the right we see the reprojection of the 3D points on the projector image plane (cyan dots). See how they sometimes misalign because the calibration isn’t perfect.
Now we can do some fun stuff like 3D scanning with binary patterns:

Which results in something like this:

That’s what I got!

One reply on “Projector-Camera Calibration – the "easy" way”

Leave a Reply

Your email address will not be published. Required fields are marked *