The Beginner’s Guide to Semantic Segmentation

Ready to streamline AI product deployment right away? Check out:

• V7 Model Training

• V7 Workflows

• V7 Auto Annotation

• V7 Dataset Management

What is Semantic Segmentation?

Semantic Segmentation follows three steps:

1. Classifying: Classifying a certain object in the image.

2. Localizing: Finding the object and drawing a bounding box around it.

3. Segmentation: Grouping the pixels in a localized image by creating a segmentation mask.
   
   

Essentially, the task of Semantic Segmentation can be referred to as classifying a certain class of image and separating it from the rest of the image classes by overlaying it with a segmentation mask.

It can also be thought of as the classification of images at a pixel level.

Below is an example of how you can perform semantic segmentation using V7.

Semantic Segmentation in V7

The goal is simply to take an image and generate an output such that it contains a segmentation map where the pixel value (from 0 to 255) of the iput image is transformed into a class label value (0, 1, 2, … n).

An overview of the Semantic Image Segmentation process

Pro Tip: Looking for the perfect data annotation tool to perform Semantic Segmentation? See this list of 13 Best Image Annotation Tools.

Now, let’s explore deep learning methods for semantic segmentation.

Semantic Segmentation Deep Learning methods

Semantic Segmentation often requires the extraction of features and representations, which can derive meaningful correlation of the input image, essentially removing the noise.

Pro Tip: Read A Simple Guide to Data Preprocessing in Machine Learning to learn more about data preparation

The convolutional neural network (CNN) performs this task and is frequently used in most computer vision tasks.

The following section will explore the different semantic segmentation methods that use CNN as the core architecture. The architecture is sometimes modified by adding extra layers and features, or changing its architectural design altogether.

Fully Convolutional Network

CNN consists of a convolutional layer, a pooling layer, and a non-linear activation function.

Pro Tip: Check out 12 Types of Neural Network Activation Functions: How to Choose?

In most cases, CNN has a fully connected layer at the end in order to make class label predictions.

CNN with fully connected layer

But—

When it comes to semantic segmentation, we usually don’t require a fully connected layer at the end because our goal isn’t to predict the class label of the image.

In semantic segmentation, our aim is to extract features before using them to separate the image into multiple segments.

However, the issue with convolutional networks is that the size of the image is reduced as it passes through the network because of the max-pooling layers.

To efficiently separate the image into multiple segments, we need to upsample it using an interpolation technique, which is achieved using deconvolutional layers.

CNN with upsampling and deconvolutional layer

In general AI terminology, the convolutional network that is used to extract features is called an encoder. The encoder also downsamples the image, while the convolutional network that is used for upsampling is called a decoder.

Here’s a visual representation of convolutional encoder-decoder.

Convolutional encoder-decoder structure

The output yielded by the decoder is rough, because of the information lost at the final convolution layer i.e., the 1 X 1 convolutional network. This makes it very difficult for the network to do upsampling by using this little information.

To address this upsampling issue, the researchers proposed two architectures: FCN-16 and FCN-8.

In FCN-16, information from the previous pooling layer is used along with the final feature map to generate segmentation maps. FCN-8 tries to make it even better by including information from one more previous pooling layer.

Pro Tip: Learn more about data compression from our Beginner's Guide to Autoencoders.

U-net

The U-net was a modification of a fully convolutional network.

It was introduced by Olaf Ronneberger et al. in 2015 for medical purposes—primarily to find tumors in the lungs and brain.

The U-net has a similar design of an encoder and a decoder.

The former is used to extract features by downsampling, while the latter is used for upsampling the extracted features using the deconvolutional layers. The only difference between the FCN and U-net is that the FCN uses the final extracted features to upsample, while U-net uses something called a shortcut connection to do that.

U-net architecture

The shortcut connection in the U-Net is designed to tackle the information loss problem.

Pro tip: Ready to train your models? Have a look at Mean Average Precision (mAP) Explained: Everything You Need to Know

The U-net is designed in such a manner that there are blocks of encoder and decoder. These blocks of encoder send their extracted features to its corresponding blocks of decoder, forming a U-net design.

So far, we have learned that as the image travels through the convolutional networks and its size is reduced.

This is because it simultaneously max-pools layers, which means that information is lost in the process. This architecture enables the network to capture finer information and retain more information by concatenating high-level features with low-level ones.

This process of concatenating the information from various blocks enables U-net to yield finer and more accurate results.

Pyramid Scene Parsing Network (PSPNet)

Pyramid Scene Parsing Network (PSPNet) was designed to get a complete understanding of the scene.

Scene parsing is difficult because we are trying to create a Semantic Segmentation for all the objects in the given image.

Scene Parsing

The FCN doesn’t perform too well because of the information loss that we discussed earlier.

Some of this information is lost because of the spatial similarities between two different objects. A network can capture spatial similarities if it can exploit the global context information of the scene.

The comparison of FCN and PSPNet performance

PSPNet exploits the global context information of the scene by using a pyramid pooling module.

PSPNet with Pytaming Pooling Module

From the diagram above, we can see that the pyramid pooling module has four convolutional components.

The first component indicated in red yields a single bin output, while the other three separate the feature map into different sub-regions and form pooled representations for different locations.

These outputs are upsampled independently to the same size and then concatenated to form the final feature representation.

Because the filter size of the convolution network is varied (i.e., 1X1, 2X2, 3X3, and 6X6), the network can extract both local and global context information.

The concatenated upsampled result from the pyramid module is then passed through the CNN network to get a final prediction map.

DeepLab

Google developed DeepLab, which also uses CNN as its primary architecture.

Unlike U-net, which uses features from every convolutional block and then concatenates them with their corresponding deconvolutional block, DeepLab uses features yielded by the last convolutional block before upsampling it, similarly to CFN.

The Deeplab applies atrous convolution for upsampling.

Atrous convolution

Atrous convolution (or Dilated convolution) is a type of convolution with defined gaps.

To better understand it, let’s imagine k is the dilation rate.

If k=1, the convolution will be normal. But if we increase the value of k by one—for example, k=2—then we skip one pixel per input.

See the image below.

Dilated convolution

Dilation rate defines the spacing between the values in a convolution filter.

As shown in the image above, a 3x3 filter with a dilation rate of 2 will have the same field of view as a 5x5 filter while only using nine parameters.

The advantage of using an Atrous or Dilated convolution is that the computation cost is reduced while capturing more information.

In the image above, the bottom figure shows that Atrous convolution achieves a denser representation than the top figure.

DeepLab V1

Now you know that DeepLab’s core idea was to introduce Atrous convolution to achieve denser representation where it uses a modified version of FCN for the task of Semantic Segmentation.

This idea introduced DeepLab V1 that solves two problems.

1. Feature resolution

The issue with DCNN is multiple pooling and down-sampling, which causes a significant reduction in spatial resolution.

To achieve a higher sampling rate, the down-sampling operator was removed from the last few max-pooling layers, and instead, up-sample the filters (atrous) in subsequent convolutional layers were added.

2. Localization accuracy due to DCNN invariance

In order to capture fine details, data scientists employed a fully connected Conditional Random Field (CRF), which smoothens and maximizes label agreement between similar pixels.

The CRF also enables the mode to create global contextual relationships between object classes.

DeepLab V2

DeepLab V1 was further improved to represent the object in multiple scales.

In the previous section, we saw how PSPNet used a pyramid pooling module to achieve multiple Semantic Segmentation with greater accuracy.

Building on that theory, DeepLab V2 used Atrous Spatial Pyramid Pooling (ASPP).

The idea was to apply multiple atrous convolutions with different sampling rates and concatenate them together to get greater accuracy.

DeepLab V3

The aim of DeepLab V3 was to capture sharper object boundaries.

This was achieved by adopting the encoder-decoder architecture with atrous convolution.

Here are the advantages of using encoder-decoder architecture:

1. It can capture sharp object boundaries.

2. It can capture high semantic information because the encoders gradually reduce the input image while extracting vital spatial information; similarly, decoders gradually recover spatial information.

ParseNet

ParseNet was introduced by Wei Liu et al. in 2015.

The authors of this paper suggested that FCN cannot represent global context information.

Contextual representation of the data or image is known to be very useful for improving performance segmentation tasks. Because FCN lacks contextual representation, they are not able to classify the image accurately.

The comparison between FCN and ParseNet output

To acquire global context information or vector, the authors used a feature map that was pooled over the input image, i.e., global average pooling.

Once acquired, the global context vector was then appended to each of the features of the subsequent layers of the network.

It is worth noting that global context information can be extracted from any layer, including the last one.

The authors stated that:

‍The quality of Semantic Segmentation is greatly improved by adding the global feature to local feature map.

ParseNet contexture model

The image above represents the overview of the ParseNet contexture module.

As you can see, once the global context information is extracted from the feature map using global average pooling, L2 normalization is performed on them.

Similarly, L2 normalization is also performed directly on the feature map.

To combine the contextual features to the feature map, one needs to perform the unpooling operation. It ensures that both features are of the same size.

In conclusion, ParseNet performs better than FCN because of global contextual information.