Calibrating & Undistorting with OpenCV in C++ (Oh yeah)

I’ve already talked about camera distortions and calibrating a camera. Now we’ll actually implement it. And we’ll do it in C++. Why? Because it’s a lot more easier and make much more sense. No more stupid CV_MAT_ELEM macros. And things will just work. But, I won’t talk about how the C++ is working. Figure it out yourself ;)

The setup

The first thing we need for this is the latest version of OpenCV. If you’re using 1.0 or 1.1pre or any of those, you need to get the latest version. It has the C++ interface. Previous version simply do not have it. Go download the most recent version at sourceforge.

Once you have it, follow these instructions if you use Visual Studio. If not, check around the OpenCV wiki, and you should see where you can find instructions for your IDE.

Onto the project

Once you have your IDE or whatever environment setup, start by creating a new project. Include the standard OpenCV headers:

#include <cv.h>
#include <highgui.h>

We’ll include the OpenCV namespace so we can use its functions directly (without a cv:: everytime):

using namespace cv;

Now the main function. We create some variables. The number of boards you want to capture, the number of internal corners horizontally and the number of internal corners vertically (That’s just how the algorithm works).

int main()
{
    int numBoards = 0;
    int numCornersHor;
    int numCornersVer;

Then, we get these values from the user:

    printf("Enter number of corners along width: ");
    scanf("%d", &numCornersHor);
 
    printf("Enter number of corners along height: ");
    scanf("%d", &numCornersVer);
 
    printf("Enter number of boards: ");
    scanf("%d", &numBoards);

We also create some additional variables that we’ll be using later on.

    int numSquares = numCornersHor * numCornersVer;
    Size board_sz = Size(numCornersHor, numCornersVer);

See the Size? That’s OpenCV in C++. Next, we create a camera capture. We want live feed for out calibration!

    VideoCapture capture = VideoCapture(0);

Next, we’ll create a list of objectpoints and imagepoints.

    vector<vector<Point3f>> object_points;
    vector<vector<Point2f>> image_points;

What do these mean? For those unfamiliar with C++, a “vector” is a list. This list contains items of the type mentioned within the angular brackets < > (it’s called generic programming). So, we’re creating a list of list of 3D points (Point3f) and a list of list of 2D points (Point2f).

object_points is the physical position of the corners (in 3D space). This has to be measured by us. [write relationg between each list item and list's eh you get the point]

image_points is the location of the corners on in the image (in 2 dimensions). Once the program has actual physical locations and locations on the image, it can calculate the relation between the two.

And because we’ll use a chessboard, these points have a definite relations between them (they lie on straight lines and on squares). So the “expected” – “actual” relation can be used to correct the distortions in the image.

Next, we create a list of corners. This will temporarily hold the current snapshot’s chessboard corners. We also declare a variable that will keep a track of successfully capturing a chessboard and saving it into the lists we declared above.

    vector<Point2f> corners;
    int successes=0;

Then we create two images and get the first snapshot from the camera:

    Mat image;
    Mat gray_image;
    capture >> image;

The >> is the C++ interface at work again!

Next, we do a little hack with object_points. Ideally, it should contain the physical position of each corner. The most intuitive way would be to measure distances “from” the camera lens. That is, the camera is the origin and the chessboard has been displaced.

Chessboards displaced around a camera in calibration

Usually, it’s done the other way round. The chessboard is considered the origin of the world. So, it is the camera that is moving around, taking different shots of the camera. So, you can set the chessboard on some place (like the XY plane, of ir you like, the XZ plane).

Camera being displaced around the chessboard

Mathematically, it makes no difference which convention you choose. But it’s easier for us and computationally faster in the second case. We just assign a constant position to each vertex.

And we do that next:

    vector<Point3f> obj;
    for(int j=0;j<numSquares;j++)
        obj.push_back(Point3f(j/numCornersHor, j%numCornersHor, 0.0f));

This creates a list of coordinates (0,0,0), (0,1,0), (0,2,0)…(1,4,0)… so on. Each corresponds to a particular vertex.

An important point here is that you’re essentially setting up the units of calibration. Suppose the squares in your chessboards were 30mm in size, and you supplied these coordinates as (0,0,0), (0, 30, 0), etc, you’d get all unknowns in millimeters.

We’re not really concerned with the physical dimensions, so we used a random unit system.

Now, for the loop. As long as the number of successful entries has been less than the number required, we keep looping:

    while(successes<numBoards)
    {

Next, we convert the image into a grayscale image:

        cvtColor(image, gray_image, CV_BGR2GRAY);

And we’re here. The key functions:

        bool found = findChessboardCorners(image, board_sz, corners, CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_FILTER_QUADS);
 
        if(found)
        {
            cornerSubPix(gray_image, corners, Size(11, 11), Size(-1, -1), TermCriteria(CV_TERMCRIT_EPS | CV_TERMCRIT_ITER, 30, 0.1));
            drawChessboardCorners(gray_image, board_sz, corners, found);
        }

The findChessboardCorners does exactly what it says. It looks for board_sz sized corners in image. If it detects such a pattern, their pixel locations are stored in corners and found becomes non-zero. The flags in the last parameter are used to improve the chances of detecting the corners. Check the OpenCV documentation for details about the three flags that can be used.

If corners are detected, they are further refined. Subpixel corners are calculated from the grayscale image. This is an iterative process, so you need to provide a termination criteria (number of iterations, amount of error allowed, etc).

Also, if corners are detected, they’re drawn onto the screen using the handy drawChessboardCorners function!

Next we update the display the images and grab another frame. We also try to capture a key:

        imshow("win1", image);
        imshow("win2", gray_image);
 
        capture >> image;
 
        int key = waitKey(1);

If escape is pressed, we quit. No questions asked. If corners were found and space bar was pressed, we store the results into the lists. And if we reach the required number of snaps, we break the while loop too:

        if(key==27)
            return 0;
 
        if(key==' ' && found!=0)
        {
            image_points.push_back(corners);
            object_points.push_back(obj);
            printf("Snap stored!\n");
 
            successes++;
 
            if(successes>=numBoards)
                break;
        }
    }

Next, we get ready to do the calibration. We declare variables that will hold the unknowns:

    Mat intrinsic = Mat(3, 3, CV_32FC1);
    Mat distCoeffs;
    vector<Mat> rvecs;
    vector<Mat> tvecs;

We modify the intrinsic matrix with whatever we do know. The camera’s aspect ratio is 1 (that’s usually the case… If not, change it as required).

    intrinsic.ptr<float>(0)[0] = 1;
    intrinsic.ptr<float>(1)[1] = 1;

Elements (0,0) and (1,1) are the focal lengths along the X and Y axis.

And finally, the calibration:

    calibrateCamera(object_points, image_points, image.size(), intrinsic, distCoeffs, rvecs, tvecs);

After this statement, you’ll have the intrinsic matrix, distortion coefficients and the rotation+translation vectors. The intrinsic matrix and distortion coefficients are a property of the camera and lens. So as long as you use the same lens (ie you don’t change it, or change its focal length, like in zoom lenses etc) you can reuse them. In fact, you can save them to a file if you want and skip the entire chessboard circus!

Note: The calibrateCamera function converts all matrices into 64F format even if you initialize it to 32F. Thanks to Michael Koval!

Now that we have the distortion coefficients, we can undistort the images. Here’s a small loop that will do this:

    Mat imageUndistorted;
    while(1)
    {
        capture >> image;
        undistort(image, imageUndistorted, intrinsic, distCoeffs);
 
        imshow("win1", image);
        imshow("win2", imageUndistorted);
 
        waitKey(1);
    }

And finally we’ll release the camera and quit!

    capture.release();
 
    return 0;
}

My results

I ran this program on a low quality webcam. I used a hand-made chessboard pattern and used 20 chessboard positions to calibrate. Here’s an undistort I did:

A result I got from undistortion

Make your own chessboard!

If you’re not working at some university, its very likely you don’t have a chessboard pattern that will work perfectly. You need an asymmetric chessboard: 5×6 or a 7×8 or27x3.

So make one yourself. Take a piece of paper and draw on it with a marker. Paste it on some cardboard. I made mine from a small notebook page. It’s a 5×4 chessboard. Not very big, but it works. Here’s what it looks like:

My chessboard pattern

You can even see the lines from the notebook. :| But the inner corners are detected pretty well. You’ll definitely want a better one if you work with higher resolutions.

If you’re looking for precision, get it printed. Here’s a picture that you can print on an A4 size paper at 300dpi (its PNG and around 35kb in size).

(click for a full size version: A4 at 300dpi)

Bad calibration

A bad calibration is very much possible. Here’s what I got in one of my attempts:

Bad calibration example

Yes the image on the left is the original, and the one on the right is “undistorted”.

Hopefully you won’t get such results. The key is calibrate with the chessboard everywhere on the screen. It should not be “biased” toward some corner or region.

Summary

Hope you’ve learned how to calibrate your cameras with OpenCV and how to undistort images taken from them. With OpenCV, you don’t need to know what goes on underneath while being able to fully utilize the calibration and undistortion. Got questions or suggestions? Leave a comment!

Issues? Suggestions? Visit the Github issue tracker for AI Shack

Back to top

37 Comments

  1. Joseph Lee
    Posted October 1, 2010 at 8:11 pm | Permalink

    Dear Sir,

    I have tried the program but only find value in [0,1] of intrinsic matrix. I don’t if any installation problem or any thing wrong. Any ideas?

    Thanks a lot for your help in advance.

    Joseph

    • Posted October 2, 2010 at 10:48 am | Permalink

      Hi Joseph! Either those values could be correct, or your calibration pattern isn’t working too well. Did you try undistorting the image and see if it looks good enough?

  2. yahia
    Posted October 15, 2010 at 3:59 am | Permalink

    Hi,
    when I complie the algorithm an error appear on the line
    calibrateCamera(object_points, image_points,image.size(),intrinsic,distCoeffs, rvecs, tvecs);

    The error is: invalid initialization of reference of type ‘std:: vector <cv:: Mat, std:: allocator > &’ from expression of type ‘resume:: Vector ‘ |

    Thank you

    • Posted October 17, 2010 at 10:55 am | Permalink

      Did you include the std namespace and vector.h?

      • yahia
        Posted October 17, 2010 at 3:18 pm | Permalink

        Hi,
        Of course, I think that the problem lie in this argument image.size().

        as you can see in cv.h the function findChessboardCorners, written as follows:

        findChessboardCorners( const Mat& image, Size patternSize,
        vector& corners,
        int flags=CV_CALIB_CB_ADAPTIVE_THRESH+
        CV_CALIB_CB_NORMALIZE_IMAGE );

      • yahia
        Posted October 17, 2010 at 3:19 pm | Permalink

        Hi,
        Of course, I think that the problem lie in this argument image.size().

        as you can see in cv.h the function findChessboardCorners, written as follows:

        findChessboardCorners( const Mat& image, Size patternSize,
        vector& corners,
        int flags=CV_CALIB_CB_ADAPTIVE_THRESH+
        CV_CALIB_CB_NORMALIZE_IMAGE );

        and I’m using code::block but I think this is not the problem .

  3. yahia
    Posted October 17, 2010 at 5:14 pm | Permalink

    Hi, the problem is that you most initialise this 2 references rvecs, tvecs, but how I do it, thank you

  4. Posted October 18, 2010 at 1:46 pm | Permalink

    Hello Utkarsh

    Nice site.
    I am doing a project on laser line geometry capture obvious the objects with square
    corners when scanned need to be corrected before x,y coordinates is processes.
    I have compiled and used the sample from opencv that is a bit complex for “me”
    so I found your site and the code look simple enough to start with some experiment.

    error C2664 cv::DrawChessboardCorners cannot convert parameter 3 from
    ‘std::vector to const cv::Mat

    Thank you for your information it already put me on a better track.

    Best Regards

    Awie Spengler

    • Posted October 19, 2010 at 10:52 am | Permalink

      Glad you’re back on track! All the best! :)

  5. Muhammad Humza
    Posted October 18, 2010 at 8:40 pm | Permalink

    This site is cool …. keep up the good work man :)

  6. Emdad
    Posted October 28, 2010 at 1:24 pm | Permalink

    Hi Utkarash,

    Good tutorial but I don’t understand how did u get the output from this. Because, there is an error coming out from the source code U presented here. the error is “error C2664: ‘cv::drawChessboardCorners’ : cannot convert parameter 3 from ‘std::vector’ to ‘const cv::Mat &’ “. the 3rd parameter needs correction. Please check and fix it, so that everybody can try it and appreciate U as well… Thanks anyway for such good work..

    • Posted October 31, 2010 at 4:22 am | Permalink

      Weird. It works on my computer. What version of OpenCV are you using?

  7. atomicity
    Posted November 4, 2010 at 8:30 pm | Permalink

    This happens with OpenCV 2.1 to me too – in addition, findChessboardCorners crashes all the time, and I have NO idea why.

    • Chang
      Posted February 16, 2011 at 8:15 pm | Permalink

      It happens to me two. “findChessboardCorners” crashes all the time, don’t know why…

  8. Posted November 10, 2010 at 1:07 pm | Permalink

    Note that cv::calibrateCamera() will convert the camera and distortion matrices to be of type CV_64F even if you initialize them as CV_32F. Just as in your example code, I spent hours debugging my program only to realize that I was printing the values as if they were floats instead of doubles.

    Regardless, thank you for the great line-by-line walkthrough of camera calibration: it’s much more digestible than the sample code included with OpenCV.

    • Posted November 12, 2010 at 1:17 am | Permalink

      Oh. Thanks for pointing out! I’ll add that to the article.

  9. weiquata
    Posted November 14, 2010 at 9:32 pm | Permalink

    i’m using VC++2008 & OpenCV2.1
    I also facing the problem ” C2664: ‘cv::drawChessboardCorners’ : cannot convert parameter 3 from ‘std::vector’ to ‘const cv::Mat &’ “.

    i search for the OpenCV Wiki:
    void drawChessboardCorners(Mat& image, Size patternSize, const Mat& corners, bool patternWasFound)

    bool findChessboardCorners(const Mat& image, Size patternSize, vector& corners, int flags=CV_CALIB_CB_ADAPTIVE_THRESH+ CV_CALIB_CB_NORMALIZE_IMAGE)

    [vector& corners] for findChessboardCorners
    [const Mat& corners] for drawChessboardCorners
    is that the problem here?

  10. Rouge
    Posted November 22, 2010 at 12:12 am | Permalink

    Thank you for this tutorial ! It helps me a lot !

    Best regards.

    PS : I change this : drawChessboardCorners(gray_image, board_sz, (cv::Mat) corners, found); and it worked ! using OpenCV 2.1 with Code::Blocks on Ubuntu 10.04

    • test_tube_baby
      Posted August 24, 2011 at 10:43 pm | Permalink

      Thanks! that did the job! fixed the error :-)

    • test_tube_baby
      Posted August 24, 2011 at 11:41 pm | Permalink

      It works! your change eradicates the error! thank you :-)

  11. Posted December 1, 2010 at 7:27 am | Permalink

    I too am getting this error:

    error C2664: ‘cv::drawChessboardCorners’ : cannot convert parameter 3 from ‘std::vector’ to ‘const cv::Mat &’

    It seems as though the issue is still unresolved.
    Cant seem to figure out what is causing it and I’m really excited to see this lens correction working :)

    Really like the site, lots of great information on here and really clear to understand!

    • Raingo
      Posted January 15, 2011 at 6:05 pm | Permalink
      drawChessboardCorners(gray_image, board_sz, Mat(corners), found);

      works!

  12. Koki
    Posted April 12, 2011 at 11:35 am | Permalink

    Hi
    Thank you for this tutorial !

    I have a question.

    What application you use to create 3D graph which describe camera positoin?

    • Posted April 29, 2011 at 8:04 pm | Permalink

      I found them on the internet somewhere.

      • Alessandro
        Posted January 11, 2012 at 9:29 pm | Permalink

        It’s an output of the Matlab calibration toolbox. =)

  13. Posted May 2, 2011 at 12:05 am | Permalink

    Just wanted to encourage you to keep up the great work with all these tutorials!

    Also, I’m amazed at the hand-drawn chessboard on notebook paper. That’s so hardcore.

  14. Sylvain
    Posted May 27, 2011 at 10:00 pm | Permalink

    Hi,

    Firstly, thank you very much for this tutorial, very interesting !

    Me, I have also a problem with the vector corner because it’s too long:
    “Vector too long”
    So, there is a problem when I try to update image_points.

    I tried to clear the vector corner at the beginning of the loop:
    if(!corner.empty()){
    corner.clear();
    }
    But clear produces an error in that case ! So, I don’t know what to do…
    I use Visual Studio 2010, OpenCV 2.2 and an old XP (no enough virtual memory maybe ?)
    I thank you by advance for your advice.

  15. Olu
    Posted June 6, 2011 at 7:58 pm | Permalink

    Hello
    Could you please provide the source code file?
    Thanks

  16. Kishor
    Posted June 11, 2011 at 10:58 pm | Permalink

    An Excellent tutorial. Keep up the good work.
    Thanx

  17. Sebastian
    Posted January 17, 2012 at 1:17 am | Permalink

    Hi! congratulations for your page! Do you know how can i save the position of coners in pixel in a file?

    Thanks in advance

    • Posted January 20, 2012 at 12:45 am | Permalink

      You could write code for that yourself. You simply write numbers into a file and read them back. A simple ‘format’ would be to have one number in each line – alternating x/y coordinates.

  18. Antonio
    Posted July 19, 2013 at 12:47 am | Permalink

    Hi,
     
    I am getting the following error when I try use calibrateCamera:
    Opencv Error: Assertion failed (ni >= 0) in unknown function. file ..\..\..\src\opencv\module\calib3d\calibration.cpp line 3173.
     
    Any idea about what is happen here?
    Thanks for you help! I have done an excellent work. 

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>