# Fix camera distortions in an easy way.

Fernando Souza

a year ago | 11 min read

When a camera takes a photograph, we often see the image not quite the same as we see it in our brain. This is caused by the camera lens and it happens more than we think.

This alteration of the image is what we call distortion.

Generally speaking, distortion is when a straight lines appear bent or curvy in an image.

There are different types of distortion, depending essentially on the model of the camera lens you used.

Although you sometimes want to create a nice effect on your images, a distortion can be bad for computer vision systems. Since the coordinates of the image are deviated from its original position, you might create some errors or fail to detect an object.

When you calibrate a camera, you discover some specific parameters that will fix these distortions.

# Camera Model

Source: here.

A camera model describes a relationship between a point in a 3D space and its projection into a 2D space (an image). The pinhole camera model is often used in computer vision as a reasonable approximation of a camera.

Duration the calibration process, some parameters are discovered to fix most of the distortions caused by a camera:

• Intrinsic parameters: focal length, optical center, and skew coefficients. Specific to each camera.
• Extrinsic parameters: rotation and translation vectors that translate a 3D scene into a 2D coordinate.
• Radial distortion coefficients: it models the radial distortion, that causes straight line to appear curved. It occurs when light rays bend more near the edges of a lens than they do at its optical center.
• Tangential distortion coefficients: it models the tangential distortion, that occurs when the lens and the image plane are not parallel.

To find these parameters, we need some images that contains a well defined pattern. The algorithm then find some specific points, whose coordinates we know from a real object. With the coordinates in the image, we then can solve the equations for the coefficients that we want.

There are different objects that we can use to calibrate that we are going to see bellow.

# OpenCV

First of all, let’s create our environment. To calibrate our camera and get the coefficients, we are going to use the OpenCV library.

OpenCV is an open-source library that contains a lot of computer vision algorithms, from image processing to object detection. It is a powerful library that can be used on different platforms.

To install it:

pip install opencv-python
pip install opencv-python-contrib

Note: in this tutorial it was used OpenCV version 4.5.1 and Python 3.8.7, in a Windows 10 PC.

# Chessboard

We can use a chessboard as the reference object. Since it is well defined and we know all the coordinates, it is a great object to use. In fact, it is the standard method to calibrate a camera.

First, let’s take some pictures of a chessboard using the camera we want to calibrate. You can use a chessboard image or use a real one. At least 10 images are necessary for better results.

Note: try to variate the position of the chessboard to get better results. Remember to put the chessboard on a flat surface.

To find the chessboard inside an image, we are going to use the function findChessboardCorners, passing the image and the pattern (number of rows and columns of the chessboard).

Once we find the corners, we improve their accuracy using the function cornerSubPix.

We do the same for every image we have, recording the corners we have found. After that, we call the function calibrateCamera that will return the camera matrix, distortion coefficients, rotation and translation vectors.

import cv2
import numpy as np
import pathlib

def calibrate_chessboard(dir_path, image_format, square_size, width, height):
'''Calibrate a camera using chessboard images.'''
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(8,6,0)
objp = np.zeros((height*width, 3), np.float32)
objp[:, :2] = np.mgrid[0:width, 0:height].T.reshape(-1, 2)

objp = objp * square_size

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.

images = pathlib.Path(dir_path).glob(f'*.{image_format}')
# Iterate through all images
for fname in images:
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (width, height), None)

# If found, add object points, image points (after refining them)
if ret:
objpoints.append(objp)

corners2 = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
imgpoints.append(corners2)

# Calibrate camera
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)

return [ret, mtx, dist, rvecs, tvecs]

Parameters used in the function:

• dir_path: path to the directory where the chessboard images are stored.
• image_format: extension of the images to be used.
• square_size: size, in centimeter, of each square of the real chessboard. Use a ruler and try to be as accurate as possible.
• width, height: how many squares there are in the chessboard (in my case, 6 x 9.

After we get the camera matrix and the coefficients, we then save them in a file so we can use it later without having to go through this process every time.

import cv2

def save_coefficients(mtx, dist, path):
'''Save the camera matrix and the distortion coefficients to given path/file.'''
cv_file = cv2.FileStorage(path, cv2.FILE_STORAGE_WRITE)
cv_file.write('K', mtx)
cv_file.write('D', dist)
# note you *release* you don't close() a FileStorage object
cv_file.release()

'''Loads camera matrix and distortion coefficients.'''

# note we also have to specify the type to retrieve other wise we only get a
# FileNode object back instead of a matrix
camera_matrix = cv_file.getNode('K').mat()
dist_matrix = cv_file.getNode('D').mat()

cv_file.release()
return [camera_matrix, dist_matrix]

The complete process is as follow:

import cv2
dst = cv2.undistort (original, mtx, dist, None, None)
cv2.imwrite ('undist.jpg', dst)

## Undistort

With the parameters, we then can undistort the image that we receive from the camera.

We use the function cv2.undistort that will return to us a new image.

import cv2
dst = cv2.undistort (original, mtx, dist, None, None)
cv2.imwrite ('undist.jpg', dst)

Original (left) and corrected (right) images. By author.

# ArUco boards

The ArUco fiducial marker can also be used to calibrate a camera. A fiducial marker is a landmark added to a scene to facilitate locating points.

An ArUco marker is a synthetic square marker composed by a wide black border and an inner binary matrix which determines its identifiier.

Calibrating using ArUco is much more versatile than using traditional chessboard patterns, since you don’t need to include the complete board view.

The first step of the ArUco process is to create the board. For that, we use the function GridBoard_create, indicating the dimensions (how many markers in horizontal and vertical), the marker length, the marker separation, and the ArUco dictionary to be used.

aruco_dict = aruco.Dictionary_get(aruco.DICT_6X6_1000)
# Dimensions in cm
marker_length = 2.25
marker_separation = 0.3
arucoParams = aruco.DetectorParameters_create()
board = aruco.GridBoard_create(5, 7, marker_length, marker_separation, aruco_dict)

Note: you can save this board and print it instead of getting the image from the internet. But remember to use with the real dimensions you printed when calibrating the images.

Then, we need to create a list with all the markers and ids we have found in the calibration images we’ve taken. We use the detectMarkers function to get ids within each image.

After that, we pass the markers, ids, and the board to the function calibrateCameraAruco. This function will then return the camera matrix and the distortion coefficients so we can use them later.

import numpy as np
import cv2
import cv2.aruco as aruco
import pathlib

def calibrate_aruco(dirpath, image_format, marker_length, marker_separation):
'''Apply camera calibration using aruco.
The dimensions are in cm.
'''
aruco_dict = aruco.Dictionary_get(aruco.DICT_6X6_1000)
arucoParams = aruco.DetectorParameters_create()
board = aruco.GridBoard_create(5, 7, marker_length, marker_separation, aruco_dict)

counter, corners_list, id_list = [], [], []
img_dir = pathlib.Path(dirpath)
first = 0
# Find the ArUco markers inside each image
for img in img_dir.glob(f'*.{image_format}'):
img_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
corners, ids, rejected = aruco.detectMarkers(
img_gray,
aruco_dict,
parameters=arucoParams
)

if first == 0:
corners_list = corners
id_list = ids
else:
corners_list = np.vstack((corners_list, corners))
id_list = np.vstack((id_list,ids))
first = first + 1
counter.append(len(ids))

counter = np.array(counter)
# Actual calibration
ret, mtx, dist, rvecs, tvecs = aruco.calibrateCameraAruco(
corners_list,
id_list,
counter,
board,
img_gray.shape,
None,
None
)
return [ret, mtx, dist, rvecs, tvecs]

And we follow the same process as with the chessboard.

from aruco import calibrate_aruco
import cv2

# Parameters
IMAGES_DIR = 'path_to_images'
IMAGES_FORMAT = '.jpg'
# Dimensions in cm
MARKER_LENGTH = 3
MARKER_SEPARATION = 0.25

# Calibrate
ret, mtx, dist, rvecs, tvecs = calibrate_aruco(
IMAGES_DIR,
IMAGES_FORMAT,
MARKER_LENGTH,
MARKER_SEPARATION
)
# Save coefficients into a file
save_coefficients(mtx, dist, "calibration_aruco.yml")

dst = cv2.undistort(img, mtx, dist, None, None)
cv2.imwrite('undist.jpg', dst)

# ChArUco

One of the advantages of the ArUco markers and boards is that they are very fast to detect. One the contrary, the detection of the corners of a chessboard can be more accurate than the ones of the ArUco, since each corner is surrounded by two black squares.

A ChArUco board combines these two approaches:

Source: here.

The first step is, of course, to create the ChArUco board. For that, we will need:

• Number of chessboard squares in X direction.
• Number of chessboard squares in Y direction.
• Length of square side.
• Length of marker side.
• The ArUco dictionary to be used.
aruco_dict = aruco.Dictionary_get(aruco.DICT_6X6_1000)
board = aruco.CharucoBoard_create(5, 7, square_length, marker_length, aruco_dict)

When you read an image, you need to detect the corners inside the image with the function aruco.detectMarkers and interpolate the corners with the function aruco.interpolateCornersCharuco.

Do these steps with every image you have taken to calibrate the camera. With all the corners captured, we call the function aruco.calibrateCameraCharuco. This function will return to us the camera matrix parameters as well as the coefficients.

import numpy as np
import cv2
import cv2.aruco as aruco
import pathlib

def calibrate_charuco(dirpath, image_format, marker_length, square_length):
'''Apply camera calibration using aruco.
The dimensions are in cm.
'''
aruco_dict = aruco.Dictionary_get(aruco.DICT_6X6_1000)
board = aruco.CharucoBoard_create(5, 7, square_length, marker_length, aruco_dict)
arucoParams = aruco.DetectorParameters_create()

counter, corners_list, id_list = [], [], []
img_dir = pathlib.Path(dirpath)
first = 0
# Find the ArUco markers inside each image
for img in img_dir.glob(f'*{image_format}'):
print(f'using image {img}')
img_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
corners, ids, rejected = aruco.detectMarkers(
img_gray,
aruco_dict,
parameters=arucoParams
)

resp, charuco_corners, charuco_ids = aruco.interpolateCornersCharuco(
markerCorners=corners,
markerIds=ids,
image=img_gray,
board=board
)
# If a Charuco board was found, let's collect image/corner points
# Requiring at least 20 squares
if resp > 20:
# Add these corners and ids to our calibration arrays
corners_list.append(charuco_corners)
id_list.append(charuco_ids)

# Actual calibration
ret, mtx, dist, rvecs, tvecs = aruco.calibrateCameraCharuco(
charucoCorners=corners_list,
charucoIds=id_list,
board=board,
imageSize=img_gray.shape,
cameraMatrix=None,
distCoeffs=None)

return [ret, mtx, dist, rvecs, tvecs]

And we follow the same process as before:

from charuco import calibrate_charuco
import cv2

# Parameters
IMAGES_DIR = 'path_to_images'
IMAGES_FORMAT = 'jpg'
# Dimensions in cm
MARKER_LENGTH = 2.7
SQUARE_LENGTH = 3.2

# Calibrate
ret, mtx, dist, rvecs, tvecs = calibrate_charuco(
IMAGES_DIR,
IMAGES_FORMAT,
MARKER_LENGTH,
SQUARE_LENGTH
)
# Save coefficients into a file
save_coefficients(mtx, dist, "calibration_charuco.yml")

dst = cv2.undistort(original, mtx, dist, None, mtx)
cv2.imwrite('undist_charuco.jpg', dst)

Original (left) and corrected (right) images.

# Final Thoughts

A camera calibration is an important process for a computer vision application. It will correct the distortions that a camera inserts into an image. If you need to know distances or orientation, it is a must-do process.

In this article, we’ve seen three ways to calibrate a camera. You can use any one of them.

If you have some problems, here are some tips that might help:

• Use a flat surface to put your chessboard image. You want to get the distortions caused by the camera, not the object.
• Take many pictures as necessary. I’ve found that 20 is a good number to start.
• Take pictures with different angles to improve the quality of the calibration.
• Use an environment with good illumination.

Hope you have enjoyed the reading.

Upvote

Created by

Fernando Souza

Post

Upvote

Downvote

Comment

Bookmark

Share

Related Articles