A Comprehensive Guide to Human Pose Estimation

What is Human Pose Estimation?

Human Pose Estimation (HPE) is a way of identifying and classifying the joints in the human body.

Essentially it is a way to capture a set of coordinates for each joint (arm, head, torso, etc.,) which is known as a key point that can describe a pose of a person. The connection between these points is known as a pair.

The connection formed between the points has to be significant, which means not all points can form a pair. From the outset, the aim of HPE is to form a skeleton-like representation of a human body and then process it further for task-specific applications.

However—

There are three types of approaches to model the human body:

1. Skeleton-based model

2. Contour-based model

3. Volume-based model
   
   

Classical vs. Deep Learning-based approaches

HPE approaches are primarily in the area of computer vision, and it is used to understand geometric and motion information of the human body, which can be very intricate.

This section explores the two approaches: the classical approach and the deep learning-based approach to HPE. We will also explain how classical approaches fail to capture the geometric and motion information of the human body, and how deep learning algorithms such as the CNNs excel at it.

Classical approaches to 2D Human Pose Estimation

Classical approaches usually refer to techniques and methods involving swallow machine learning algorithms.

For instance, the earlier work to estimate human pose included the implementation of random forest within a “pictorial structure framework”.  This was used to predict joints in the human body.

The pictorial structure framework (PSF) is commonly referred to as one of the traditional methods to estimate human pose. PSF contained two components:

1. Discriminator: It models the likelihood of a certain part present at a particular location. In other words, it identifies the body parts.

2. Prior: It is referred to as modeling the probability distribution over pose using the output from the discriminator; the modeled pose should be realistic.

In essence, the PSF objective is to represent the human body as a collection of coordinates for each body part in a given input image. PSF uses nonlinear joint regressors, ideally a two-layered random forest regressor.

These models work well when the input image has clear and visible limbs, however, they fail to capture and model limbs that are hidden or not visible from a certain angle.

To overcome these issues, feature building methods like histogram oriented gaussian (HOG), contours, histograms, etc.,  were used. In spite of using these methods, the classical model lacked accuracy, correlation, and generalization capabilities, so adopting a better approach was just a matter of time.

Deep Learning-based approaches to 2D Human Pose Estimation

Deep learning-based approaches are well defined by their ability to generalize any function (if a sufficient number of nodes are present in the given hidden layer).

When it comes to computer vision tasks, deep convolutional neural networks (CNN) surpass all other algorithms, and this is true in HPE as well.  

CNN has the ability to extract patterns and representations from the given input image with more precision and accuracy than any other algorithm; this makes CNN very useful for tasks such as classification, detection, and segmentation.

Unlike the classical approach, where the features were handcrafted; CNN can learn complex features when provided with enough training-validation-testing data.

Looking for quality datasets to train your models? Check out 65+ Best Free Datasets for Machine Learning.

Toshev et al in 2014 initially used the CNN to estimate human pose, switching from the classical-based approach to the deep learning-based approach, and they named it DeepPose: Human Pose Estimation via Deep Neural Networks.

In the paper that they had released, they defined the whole problem as a CNN-based regression problem towards body joints.

The authors also proposed an additional method where they implemented the cascade of such regressors in order to get more precise and consistent results. They argued that the proposed Deep Neural Network can model the given data in a holistic fashion, i.e. the network has the capability to model hidden poses, which was not true for the classical approach.

With strong and promising results shown by DeepPose, the HPE research naturally gravitated towards the deep learning-based approaches.

Human Pose Estimation using Deep Neural Networks

As the research and development started to take off in HPE, it brought forth new challenges.

One of them was to tackle the multi-person pose estimation.

DNNs are very proficient in estimating single human pose but when it comes to estimating multi-human they struggle because:

1. An image can contain multiple numbers of people in different positions.

2. As the number of people increases, the interaction between increases leads to computational complexities.

3. An increase in computational complexities often leads to an increase in inference time in real-time.

In order to tackle these problems, the researchers introduced two approaches:

1. Top-down: Localize the humans in the image or video and then estimate the parts followed by calculating the pose.

2. Bottom-up: Estimate the human body parts in the image followed by calculating the pose.

Now, let's have a look at deep learning models that are used for multi-human pose estimation.

Check out 12 Types of Neural Networks Activation Functions to learn more.

OpenPose

OpenPose was proposed by Zhe Cao et. al. in 2019.

It is a bottom-up approach where the network first detects the body parts or key points in the image, followed by mapping appropriate key points to form pairs.

OpenPose also uses CNN as its main architecture. It consists of a VGG-19 convolutional network that is used to extract patterns and representations from the given input. The output from the VGG-19 goes into two branches of convolutional networks.

The first network predicts a set of the confidence map for each body part while the second branch predicts a Part Affinity Fields (PAFs) which creates a degree of association between parts. It is also useful to prune the weaker links in the bipartite graphs.

The image above shows the architecture of OpenPose, which is a multi-stage CNN.  

Essentially the predictions from the two branches, along with the features, are concatenated for the next stage to form a human skeleton depending upon the number of humans present in the input. Successive stages of CNNs are used to refine the prediction.

The image above describes the overall pipeline of OpenPose.

AlphaPose (RMPE)

Regional Multi-person Pose Estimation (RMPE) or AlphaPose implements a top-down approach to HPE.

The top-down approach to HPE raises a lot of error in localization and inaccuracies during prediction and is, therefore, quite challenging.

For instance, the image above shows two bounding boxes, the red box represents ground truth while the yellow box represents the predicted bounding box.

Although, when it comes to classification, the yellow bounding box will be considered as a “correct” bounding box to classify a human. However, the human pose can not be estimated even with the “correct” bounding box.

The authors of AlphaPose tackled this issue of imperfect human detection with a two-step framework. In this framework, they introduced two networks:

1. Symmetric Spatial Transformer Network (SSTN): It ​​helps to crop out the appropriate region in the input, which subsequently simplifies the classification task leading to better performance.

2. Single Person Pose Estimator (SPPE): It is used to extract and estimate human pose.

The objective of AlphaPose is to extract a high-quality single-person region from an inaccurate bounding box by attaching SSTN to the SPPE. This method increases classification performances by tackling invariance while providing a stable framework to estimate human pose.

Read 9 Essential Features for a Bounding Box Annotation Tool to choose the right bounding box tool for your needs.

DeepCut

DeepCut was proposed by Leonid Pishchulin et. al. in 2016 with the objective of jointly solving the tasks of detection and pose estimation simultaneously.

It is a bottom-up approach to estimate human pose.

The idea was to detect all possible body parts in the given image, then label them such as a head, hands, legs, etc., followed by the process of separating the body parts belonging to each person.

The network uses Integral Linear Programming (ILP) modeling to implicitly group all the detected key points in the given input such that the resulting output resembles a skeleton representation of the human.

Mask R-CNN

Mask R-CNN is a very popular algorithm for instance segmentation.

The model has the capability to simultaneously localize and classify objects by creating a bounding box around the object and also by creating a segmentation mask.

The basic architecture can be easily extended for Human Pose Estimation tasks.

Fast R-CNN uses CNN to extract features and representation from the given input.

The extracted features are then used to propose where the object might be present through a Region Proposal Network (RPN).

Since the bounding box can be of various sizes like in the image above, a layer called RoIAlign is used to normalize the extracted features so that they are all of the uniform sizes.

The extracted features are passed into the parallel branches of the network to refine the proposed region of interest (RoI) to generate bounding boxes and the segmentation masks.

When it comes to human pose estimation, the mask segmentation output yielded by the network can be used to detect humans in the given input. Because mask segmentation is very precise in object detection, in this case—human detection, the human pose can be estimated quite easily.

This method resembles the top-down approach, where the person detection stage is performed in parallel to the part detection stage.

In other words, the keypoint detection stage and person detection stage are independent of each other.