Understanding Depth Perception in Computer Vision

Depth perception is the visual ability to perceive the world in three dimensions (3D) and the ability to estimate the distance/depth of an object from the source. The world that we observe is three dimensional, but the image formed on the human retina is two dimensional, which means the input to our brain is in two dimensions (2D). But we are still able to perceive the world in 3D. It is the ability of our brain to perform depth perception which is the result of human evolution. It tells us about the depth of every object or we can say the relative distance of every object from our eyes. It is crucial to our everyday life and prevents us from bumping into things. It also helps us to determine the relative speed of an object.

In technology, there are many applications of depth perception, including self-driving cars. Here LiDAR is one of the many methods used for depth perception. It uses laser beams to measure the relative distance of an object by illuminating it with the laser light and then measuring the reflections using sensors.

Depth perception in humans

We know that artificial intelligence is based on the assumption that the process of human thoughts and abilities can be mechanized. So to understand how depth perception is used for computer vision, it is better to understand how we, humans perform depth perception.

Depth Cues

The details in the environment that allow us to perceive depth are called Depth Cues. In humans, depth perception is ascertained through both Monocular and Binocular cues. Here monocular means ‘with one eye’ and binocular means ‘with both the eyes’. Because of this, in spite of having no depth information in a 2D image, we can still interpret the depth effortlessly.

a. Relative size

b. Interposition

When an object overlaps the other, the object, which is partially hidden, is perceived as being farther away. It provides the depth of the object relative to one another.

c. Aerial Perspective

It provides information about the depth of objects by the way light that falls on them and gets reflected or the shadow the object cast.

g. Texture gradient

We can see the fine details of an object that is closer whereas it is not possible with the distant objects. For example, in a grassy field, the texture becomes less and less apparent the farther it goes into the distance.

Binocular Cues

Binocular cues provide the depth information when viewing a scene with both the eyes. Binocular cues allow us to gain a 3-dimensional interpretation of the world and allow us to navigate through it effortlessly. Some of the binocular cues are mentioned below.

a. Stereopsis

It explains how the image of the same scene obtained from slightly different angles can help us to judge the depth. The larger is the disparity, the closer is the object. It happens because of the horizontal separation of our eyes, we get two images of the same object but from slightly different angles. 3D movies are the best examples of stereopsis or retinal disparity. In such movies, the scenes are filmed with cameras at slightly different angles.

b. Convergence

We focus on an object from both the eyes, in doing so they converge. Our eye muscles have to contract and relax to focus on objects at different positions. Our muscle movements provide information to perceive the depth of an object. To observe convergence, we can hold our finger in front of our face and focus on its tip. Then slowly bring it closer. We could feel the stress and observe the image becoming two images and getting blurred.

Depth Perception in Computer Vision

In the 21st century, computer vision has become one of the most dominating sectors of Artificial Intelligence. Computer vision deals with how computers can gain a high-level understanding of images and videos to perform various tasks. The images are taken using electronic cameras which are usually in the visible spectrum. Cameras capture a 2D picture which is a projection of the 3D world on a medium (or film or sensor or screen) much like the human eyes, thereby losing the depth dimension. Click here to know more about depth sensors in computer vision applications.

In many of the applications, 2D image understanding is sufficient but some applications like autonomous vehicles need 3D scene understanding which is a challenge in itself because now we have to estimate the depth dimension from 2D images only.

Monocular cues are primarily used to achieve information about the depth of the objects in the imagery. This is because most of the time we use a single camera that captures monocular images. The monocular depth estimation has gained popularity and attracted many researchers. Most research activities used geometrical cues to estimate depth.

Implementing depth estimation

Machine Learning and Deep Learning techniques are heavily used in Computer Vision because they are able to efficiently mimic human-like pattern recognition.

Below are some videos for a quick starter on Machine Learning. Feel free to skip them, in case we are already familiar.

There are many techniques, architectures, algorithms used for depth estimation/perception. Let us look at some of these.

This approach is used to predict the depth of dynamic objects. This method uses motion parallax cues from the static areas of the scenes to guide the depth prediction. It uses videos in which people imitate mannequin (freeze in elaborate, natural poses, while a hand-held camera tours the scenes).

The input of the neural network includes a reference image, a binary mask of human-region, a depth map estimated from motion parallax, a confidence map C, and an optional human keypoint map K. The network is trained to fit the Multi-View Stereo (MVS) depth map, which is the output of the network.

Unsupervised Learning

As we have seen how supervised depth estimation works. It is quite challenging to obtain depth datasets of high quality that account for all possible background conditions in an environment. Enhancing the performance of supervised methods is difficult due to the lack of accurate data. Since unsupervised (and semi-supervised) methods do not require ground truth depth at training time, hence they are not limited by this constraint. Click here for the reference research paper.

The below image shows the result of an unsupervised depth estimation approach.

During the training of unsupervised depth estimation models, the relative pose of multiple cameras is used to predict the appearances of a held-out nearby image. So this approach enables a CNN to learn to perform single image depth estimation in the absence of ground truth depth data.

These algorithms compute the similarity between each pixel in the first image and every other pixel in the second image. This approach presents the depth estimation as an image reconstruction problem.

The intuition of this approach is that, given a calibrated pair of binocular cameras, we can learn a function that can reconstruct one image from the other.

Applications of Depth Perception

1. Augmented Reality reconstruction of an object

Augmented reality is one of the key applications of depth estimation. It helps us to visualize objects in three dimensions as well as view it from multiple angles and scales. Depth information is absolutely necessary for the AR/VR devices.

2. Robotics

Most of the industries are using fully automated robots in their production lines and for such applications, depth-estimation is one of the major factors to calculate the motion along the third dimension. The depth information of objects also plays a primary role in an autonomous vehicle to detect the distance of objects from the vehicle.

3. Cameras

We use the depth information for many computer vision-based applications such as facial recognition and image-based classification to increase efficiency and accuracy. The 3D face modeling also adds up more features to the face recognition model. Depth estimation is also necessary to adjust the focus of the cameras and for the portrait mode photography.

Conclusion

The depth is an important cue that is lost while capturing the image or video that limits the performance of computer vision. With the advancement in technology, where automation is taking over in all the sectors, estimating depth is very important for estimating the motion and relative speed. This article would have given us a brief idea about depth perception and its implementation in the field of computer vision. In the coming years, we are going to witness dramatic innovations in this field.