Stories by Kartik Chaudhary on Medium

Turbocharge Large Language Model Training with Parallelization

Kartik Chaudhary — Thu, 28 Nov 2024 04:37:00 GMT

Parallelization techniques for efficient distributed training of large deep learning models.

Introduction

The story began in 2017 when researchers from Google introduced the famous Transformer architecture in their paper “Attention Is All You Need!”. Transformers turned out to be highly effective for tasks such as language translation, question-answering, and other similar sequence-to-sequence problems. Key features that make the Transformer architecture unique and effective, are — positional encoding and self-attention.

Large Language Models, or LLMs, also implement a Transformer architecture. These models are usually very large and need a huge amount of text data to train. Their size plays a big role in their success, but it also makes them difficult to train.

LLMs typically have an enormous number of trainable parameters, sometimes reaching hundreds of billions. This makes it challenging to fit them onto a single accelerator chip for training.

In this article, you will explore the techniques that help speed up these models through effective parallelization. Following are the key topics covered in this article —

What is Parallelism
Speeding Up LLM Training with Parallelization
Which parallelism is suitable for you?
References

Let’s get started.

What is Parallelism?

In computer programming, a large task can be broken down into smaller tasks that can run at the same time, speeding up the overall process. This approach is called parallel programming.

Similarly, in Deep Learning (DL), you can speed up the training of large models by splitting the work across multiple machines or accelerators like GPUs and TPUs. Parallelism in DL is mainly achieved through the follwoing two key methods:

Data Parallelism
Model Parallelism

Let’s explore these concepts further.

Data Parallelism

In this approach, the training data is split into smaller parts and distributed across multiple machines or processors. Each machine trains an identical copy of the deep learning model using its portion of the data. The parameters learned by all machines are regularly combined to update the final model.

Model Parallelism

Here, the model itself is divided into smaller sections, with each part assigned to a different machine or processor. Each machine trains only its specific part of the model. The outputs from all parts are then combined to form the complete, final model.

Which technique should I use for training LLMs?

→ As LLMs are really large models and do not always fit on a single accelerator chip, model parallelism can help!!

→ Data parallelism and model parallelism can also be combined together to make training even faster.

Don’t Overlook the Challenge of “Combining Results”

When training a model using any parallelism technique, you’ll need to merge the learnings (or parameters) from different machines or processors to create the final trained model. This merging process often involves summing up or averaging the results from each machine. To do this, you must establish a way for machines to share information, which can slow down the process and become a bottleneck in training. The frequency of this communication depends on how the parallelism is implemented. Therefore, it’s crucial to account for this communication overhead when designing your parallel training strategy.

Now that you have got the basic idea of parallelizing DL models, lets jump into some technical aspects of parallelizing the training of Large Language Models (LLMs).

Speeding Up LLM Training with Parallelization

Training large language models (LLMs) can take an extremely long time because of their huge size and the vast amount of data they require for pre-training. If done on a single processor, it could take weeks or even months, which is highly impractical. To save time, it’s essential to use parallelization techniques to speed up the training process.

Here are some effective methods to train LLMs more efficiently:

Using Data Parallelism (DP)
Using Tensor Parallelism (TP)
Applying Pipeline Parallelism (PP)
Using Zero Redundancy Optimizer (ZeRO)

Let’s understand these techniques one-by-one with examples.

Using Data Parallelism (DP)

In this approach, the training data is divided and distributed among multiple processors or machines. Each processor runs its own copy of the same model in parallel. After completing a training step, all processors share and synchronize their results to ensure the learning is updated collectively. This method works best when the entire model can fit into the memory of a single processor.

Take a look at Figure 1. There are three processors, each with a GPU accelerator and a copy of the same model. The training dataset is also divided into three parts. Each processor takes one batch from it’s partition of the dataset and trains its own model copy for one step. At the same time, the other two processors also do the same.

After completing their individual training step, all processors send their gradients to a central parameter server. The parameter server calculates the average gradient, updates the final model weights, and shares the updated weights with all processors. This completes one training step.

Figure 1: Data parallelism technique for training deep learning models | Image by Author

Now lets understand pipeline parallelism.

Using Pipeline Parallelism

Pipeline parallelism (PP) is essentially a type of model parallelism, where we distribute the layers of the neural network across processing devices (such as GPUs or TPUs). For example, a large model consisting of 100 layers can be distributed across 4 GPUs where each GPU handles about 25 consecutive layers. See the following illustration (Figure 2).

Figure 2: Pipeline Parallelism | A 100 layer neural network on 4 GPUs | Image by Author

As per the Figure 2, the data moves from layer-1 to layer-25 on GPU-0. However, when it reaches layer-25 and needs to move to layer-26, it will travel from GPU-0 to GPU-1 through a communication channel. If both GPUs are on the same machine, the data transfer will be faster. But if GPU-0 and GPU-1 are on different machines, the data transfer cost can increase significantly. In such cases, the communication channel could become a bottleneck in pipeline parallelism.

Now, let’s explore Tensor parallelism.

Applying Tensor Parallelism

Tensor parallelism (TP) can also be used for fitting a large DL model in multiple GPUs. In this method, each GPU handles only a part of the tensor. Essentially, the large matrix calculations are divided among the GPUs, and the results are then combined to form the final tensor. The following paragraph explains TP with an example.

Take a look at Figure 3 and assume that A is the input tensor and B is the model weights tensor. The matrix multiplication can be broken down as shown in the figure. Specifically, the weights tensor B can be divided into two parts, B1 and B2, which can be stored on separate processors.

Each processor will then perform matrix multiplication on their slice of the tensor and calculate the results at the same time. Basically, processor 1 will give C1, and processor 2 will give C2. These results, C1 and C2, can be transferred from the processors and combined to form the final output tensor C when needed.

Figure 3: Tensor Parallelism — splitting matrix multiplications using Tensor slices | Image by Author

This technique allows you to spread your large model across multiple GPUs. Now, let’s move on to the next approach.

Using Zero Redundancy Optimizer (ZeRO)

Data parallelism and Model parallelism approaches have some known limitations such as —

Data Parallelism doesn’t reduce the memory footprint of the models. Large DL models with billions of parameters are difficult to into single GPUs with limited memory.
Model parallelism doesn’t scale well beyond a single node because the communication becomes an overhead.

ZeRO was developed by a Microsoft Research team to overcome the limitations of data parallelism and model parallelism. Unlike vanilla data parallel training, ZeRO doesn’t run exact replicas of the model across processors, It actually optimizes the data parallelism approach by partitioning the model states (parameters, gradients and optimizer states) across different data parallel processors. As ZeRO reduces per device memory usage, it allows you to fit very big models into multiple data parallel processors having smaller memories.

Following are three common ways of implementing ZeRO —

Optimizer State Partitioning (OS): OS reduces memory consumption by 4x, and doesn’t add any extra communication overhead to the existing data parallel setup.
OS + Gradient Partitioning (GP): If you combine OS with GP, it reduces the memory requirements by 8x, It also doesn’t add any communication overhead over the previous setup.
OS + GP + Parameter partitioning: This setup can achieve memory reduction linear with the degree of data parallelism. For example, if your data parallel setup is using 32 GPUs, you can get a 32x reduction in the memory of each processor. This setup however increases the communication volume by about 50%.

The distribution of memory consumption of aforementioned three approaches can be seen in Figure 4. This illustration has been taken from the original research paper titled “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models” published in 2020 by Microsoft Research Team.

Figure 4: ZeRO memory consumption illustration as per the original research paper.

BTW — What is DeepSpeed?

DeepSpeed is a deep learning optimization library developed by Microsoft that makes training of the large deep learning models faster and more efficient. It allows you to train 10x larger models, 10x faster with minimal code changes.

Which parallelism is suitable for you?

The best parallelism strategy for your unique use case can depend upon various factors such as — model size, number of available machines, number of accelerators (or GPUs) per machine etc. Let’s look at a few scenarios to understand the possibilities.

Single Machine — Single GPU

Model fits in the GPU: In this case, you can just go ahead and train your model normally.
Model doesn’t fit in the GPU: You can use ZeRO + CPU offloading.

Single Machine — Multiple GPUs

Model fits in a single GPU: You can use data parallelism. Optionally, you can augment it with ZeRO, but it may or may not provide any speedups given the situation and configurations.
Model doesn’t fit in a single GPU: You have multiple options: pipeline parallelism (PP), ZeRO augmented data parallelism and Tensor parallelism (TP). If the largest layer of the model is not fitting in a single GPU, you can combine PP with TP or implement ZeRO.

Multiple Machines — Multiple GPUs

If inter-node connectivity is fast: ZeRO would be easiest option to implement as it doesn’t require much code changes. Otherwise, you can implement PP+DP+TP, but this setup would require significant changes to the model.
If inter-node connectivity is slow: If you are low on GPU memory and have to utilize multi-node setup. You can implement DP+PP+TP+ZeRO all together.

Thanks for reading this article. I hope it was helpful. Please do share your feedback. See you in the next article!!

Let’s connect over LinkedIn.

References

Turbocharge Large Language Model Training with Parallelization was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Understanding Key Evaluation Techniques for LLMs

Kartik Chaudhary — Wed, 23 Oct 2024 15:52:04 GMT

Key Evaluation Techniques for LLMs | Image by Author

Large Language Models, or LLMs for short, have attracted a lot of attention in past few years due to their amazing capabilities. As a result, there is a growing demand to integrate LLMs in production applications. Before putting them into production, it is critical to design proper evaluation strategies to ensure their positive business impact. This article “Key Evaluation Techniques for LLMs”, highlights some important evaluation techniques for LLMs.

Evaluating LLMs is a challenging task due to their complex nature. The way they generate outputs, is quite unintuitive, and it is almost impossible to tell whether the generated outputs are factually correct or not (they are grammatically correct most of the times though). As their output can’t be trusted, we can’t just integrate them into our production applications or products.

So, it is really important to understand the key evaluation techniques for LLMs. Also, it is critical to design the best suited evaluation strategy that meets your goals (for your unique business use case). Keeping this in mind, lets dive into the world of LLM evaluation.

This article covers the following key topics about LLM Evaluation techniques:

History of Large Language Models
Key Evaluation Techniques for LLMs
Human Evaluation
Programmatic Evaluation
AutoRater Evaluation Tool
Key Metrics for LLM Evaluation
Summary

Let’s get started.

1. History of Large Language Models

1.1 Language Models Before 2017

Before 2017, language models were relatively small and their capabilities were limited. It was a common practice to develop a single model to solve a single unique problem such as sentiment analysis, text classification, named entity recognition (NER) and so on. ML practitioners were focused on keeping the models small and simple, and often a number of task specific feature engineering techniques were applied to get the best results on that particular task.

Evaluating such small language models was not very difficult as the task that they were trained to solve, was well defined (or already known). And, often a holdout test dataset was present to check if the model is giving expected results or not. If the model is not working as expected, the ML practitioners could always tweak their feature engineering technique, or model architectures, or both to get the best results on holdout dataset.

Things changed significantly after 2017.

1.2 Language Models After 2017

In 2017, a research paper from Google, titled ‘’Attention Is All You Need”, introduced the popular “Transformers” architecture, and the things changed drastically (Figure below depicts the core Transformer architecture, as shown in the original research paper). If you are interested in going into low level details of this architecture, you can read the research paper, it is really well written.

Transformer Architecture | Taken from original paper ‘Attention is all you need’ | Key Evaluation Techniques for LLMs

The advent of Transformers, led to the development of Large Language Models (LLMs). Today, we have models with billions of parameters capable of solving complex language tasks, and almost all of them utilize Transformer architecture as their key building block.

Also, check out my introductory articles on GenAI:
1. A Gentle Introduction to Large Language Models
2. Beginner Friendly Introduction to GenAI and Its Applications

Today, we have Instruction Tuned (IT) Large Language Models. These models are very advanced generative LLMs and have the following key capabilities:

Ability to accept Super long inputs
Ability to solve tons of language tasks
Multiple language support
Muti-modality
Performance surpassing even humans

Now, let’s understand some key things about these models.

1.3 Latest LLMs are Mysterious

When it comes to integrating AI models into business applications or products, the key considerations usually are: safety, security and performance. Given that the current LLMs are black box models (due to their training and generation process), their behavior can be quite surprising sometimes. Additionally, these LLMs are extremely sensitive to prompting and other input changes.

Given the surprising behaviors of current LLMs, we must be really careful while deploying these models in production. As their true capabilities and performance is not already known. Thus, it becomes extremely important to design and consider the correct evaluation strategies based on the problem you are solving.

The next section describes the key evaluation techniques for LLMs.

2. Key Evaluation Techniques for LLMs

LLM evaluation techniques are a set of metrics and benchmarks that are used to assess the performance of LLMs on a variety of tasks. Some key benefits of evaluating LLMs are as follows:

Ensures that LLMs are secure, safe and performant
Helps in comparing two LLMs, based on their performance on various tasks
Helpful for the developers in identifying areas of improvement for LLMs
Helps in understanding the business impact of LLMs in applications and products

LLM evaluation techniques can be broadly categorized into the following three categories:

Human Evaluation
Programmatic Evaluation
AutoRater Evaluation Tool

Each of these types of evaluation techniques, have their own pros and cons. Together they make an overall LLM evaluation toolkit. In the next sections, we will learn about these methods in more details.

3. Human Evaluation

In this technique, human evaluators judge the outputs of a LLM against some rubric or against another LLM. This technique is quite popularly used and can be applied for the evaluation of almost any tasks. But, human evaluation is quite slow, expensive and often requires skilled human evaluators.

3.1 How does human evaluation work?

The idea is to generate a bunch of model outputs and show them to a human for evaluation. Despite it’s drawbacks (slow and costly), this evaluation technique is still considered the gold standard for evaluating LLMs. This is especially useful in case of generative models, as programmatic evaluation may not be reliable every time (for every task).

There are different platforms available out there for human evaluation, and there are quite a few third-party companies that sell human evaluation as a service.

Human evaluation is not always perfect. It comes with it’s own challenges. Following are some of the key challenges with human evaluation:

Quality of human evaluation depends upon the judgement and skills of the evaluator. It is difficult to find (or test) the perfect human evaluator for a given task.
Even if your evaluation quality is good, you might not get statistically significant pointers for improving the performance. So, it is really important to design the evaluation criteria very carefully.

Let’s learn about the programmatic evaluation techniques now.

4. Programmatic Evaluation

In this technique, as the name also suggests, the model output is evaluated using an automated script. Programmatic evaluation requires a golden testing dataset with expected outputs. As this evaluation is done using a script, it is quite fast, but the curation of the golden dataset might take a lot of time and efforts.

Programmatic evaluation is a good choice when the structure of the output is well defined and the expected outcome is already known. These methods often use known metrics (old ML evaluation metrics such as F1 score, Precision and Recall) for calculating model performance. Some example scenarios where programmatic evaluation can be applied are as follows:

Sentiment Analysis: As the sentiment classes are already known (positive, negative, neutral, etc), model can be evaluated using a script.
Entity Recognition: An already labelled dataset with known entities can be used for evaluation programmatically.
Language Identification: There is a single correct answer.
Image Classification: A known labelled data can be used for comparing the model outputs.
Object Detection: Already labelled test set can be used for script based evaluation.

You need to be really careful while curating the golden dataset ensuring that the labels are absolutely correct, and there is no leakage of data between the training and the testing set. The test dataset should cover almost all different scenarios (types of inputs).

Next, let’s learn about the AutoRater technique.

If you are interested in learning more about the generative learning and Generative Adversarial Networks, Do check out my book:

https://www.amazon.com/GAN-Book-Generative-Adversarial-TensorFlow2-ebook/dp/B0CR8C725C

5. AutoRater Evaluation Tool

AutoRaters are automated systems or algorithms to assess the performance of models (Note that these are different from programmatic evaluations where the evaluation script is very simple and just compares the model results with known results).

AutoRater technique is very frequently used for evaluating LLMs. AutoRater can be an AI model itself that was trained to mimic human raters on tasks such translation, creativity analysis or safety analysis.

As AutoRaters are sometimes LLMs, it is critical to make sure that AutoRater reflects the human raters correctly that it is trying to mimic.

AutoRaters are a good option for evaluating performance of the LLM on the situations where we want to assess the model for following:

Quality Assessment
Semantic Similarity
Content Relevance
Diversity and Creativity
Human Like Characteristics

In summary, while AutoRater evaluations offer efficient and scalable methods for assessing large language models, they are often complemented by human evaluations to ensure a comprehensive understanding of the model performance.

In the next section, we will lean about some frequently use key metrics for LLM evaluation.

6. Key Metrics for LLM Evaluation

In this section, we will learn about some frequently used evaluation metrics for LLMs along with their task specific use cases.

6.1 Accuracy, AUC, F1, Precision, Recall

Metrics such as accuracy, f1-score, precision and recall are often used for evaluating classification tasks. Where, the top ranked predictions of the model are compared against the already known outputs. These metrics are often used for classification tasks such as Sentiment Analysis, Image Classification and so on.

6.2 Toxicity

Measures the toxicity of the text. Toxicity of a LLM is usually evaluated using another LLM (an AutoRater) that is specifically trained for identifying toxicity in other models. Common toxicity evaluation tools are — Perspective API and Detoxify.

3. BLEU

Bilingual Evaluation Understudy, or BLEU, measures the similarity between the model translation and reference translation. BLEU provides sentence level scores, and then the overall score is usually calculated by averaging the scores over full evaluation dataset. Now-a-days, BLEURT is often preferred over BLEU. Next, let’s learn about BLEURT.

4. BLEURT

Measures if the text generated by model conveys the same meaning as the reference text. BLEURT scores are calculated using a regression model trained on ratings data. It is a good metric for evaluating language translations.

5. SQuAD

Stanford Question Answering Dataset, or SQuAD for short, is a reading comprehension dataset. This dataset has questions on a set of wikipedia articles, such that the answers to those questions are present as a segment within those articles, or the answer may not be present at all. SQuAD is a very popular metric for evaluating LLMs for question answering use cases, where the answer is present in a already known knowledge base.

6. ROUGE

Recall-Oriented Understudy for Gisting Evaluation, or ROUGE, is useful for evaluating text generation tasks like Text-summarization. ROUGE has multiple different versions but the basic idea is to measure the overlap between the text generated by the model and the reference text.

Some popular variants of ROUGE are:

ROUGE-N: Checks the common n-grams between the model generated text and the reference text. ROUGE-1 counts the number of shared unigrams, ROUGE-2 counts the number of shared bigrams and so on. ROUGE-2 is a good metric for evaluating summarization.
SP-ROUGE-N: It is ROUGE-N applied to the text that is tokenized using Sentence Piece (SP) tokenizer.
ROUGE-L: Considers the Longest Common Subsequence (LCS), between the model output and the reference text for measuring similarity.

ROUGE is a popular metric for evaluating summarization and machine translation tasks.

Conclusion

In this article, we discussed the brief history of evolution of LLMs and the importance of right evaluation strategies before putting them into production.

We learned about three key evaluation techniques for LLMs: Human Evaluation, Programmatic Evaluation and AutoRater Evaluation. Finally, we listed down some key evaluation metrics for LLMs along with their key use cases.

I hope this article, Key Evaluation Techniques for LLMs, was helpful for the readers, in understanding the evaluation of LLMs. Please let me know your thoughts by commenting below.

See you in the next article!

Unlocking Video Insights: Extract Thumbnails with Captions using Gemini 1.5 Pro

Kartik Chaudhary — Mon, 23 Sep 2024 02:48:11 GMT

A step-by-step tutorial for extracting thumbnails and meaningful captions from videos using the multimodal capabilities of Gemini

Video content creation has become very popular, with creators spending a lot of time on eye-catching thumbnails and engaging captions. A great thumbnail and a meaningful caption can really grab viewers’ attention. In this tutorial, we’ll show you how to use Google’s Gemini model to extract interesting thumbnails and captions from a video.

This article walks you through the step-by-step process of extracting thumbnails, with Python code examples. We’ll also highlight how using better prompts can improve your results.

Let’s get started.

Step 1: Installing Libraries

In this tutorial, we will work with the following two python libraries.

Vertex AI SDK for Python: useful for calling Vertex AI Gemini API.
MoviePy: A module for video editing, we will use it to extract thumbnail images from the source video for a given timestamp.

Let’s open a terminal and install these two libraries.

pip3 install --upgrade --user google-cloud-aiplatform
pip3 install --upgrade --user moviepy

Step 2: Import useful libraries

Open a Jupyter Notebook (or a colab notebook) and import the following useful python libraries for our experiment.

import json
import matplotlib.pyplot as plt
import moviepy
from moviepy.editor import VideoFileClip
from vertexai.generative_models import GenerationConfig, GenerativeModel, Part

Step 3: Setting up the Google Cloud Project

To get started using Vertex AI, you must have an existing Google Cloud project and enable the Vertex AI API.

The following python script will set the project information and initialize Vertex AI SDK for the experiment.

PROJECT_ID = "your-project-id"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

# Initialize Vertex AI
import vertexai

vertexai.init(project=PROJECT_ID, location=LOCATION)

Step 4: Download a Sample Video

In this tutorial, we will use the ‘Google Trips’ video as input for thumbnail extraction. This video can be watched/downloaded from this page.

Download the video and upload it to a Google Cloud Storage (GCS) bucket in the same project. Also, create a local copy of the video in your Python environment. The local copy of the video will help in faster frame extraction with the help of MoviePy package.

The following python script, copies the video file from GCS bucket to local file storage and creates a MoviePy-Clip object.

video_uri = "gs://sample-videofile/sample_video_google_trips.webm"
!gsutil cp {video_uri} sample_video.webm

# Creating a MoviePy Clip Object
clip = VideoFileClip("sample_video.webm")

Step 5: Load Gemini 1.5 Pro model

The Gemini 1.5 Pro (gemini-1.5-pro) model is a foundation model that performs well at a variety of multimodal tasks such as visual understanding, classification, summarization, and creating content from image, audio and video. Let’s load this model for our multimodal experiment.

The following python code creates a model object:

model = GenerativeModel("gemini-1.5-pro")

Step 6: Define a Function to Call Gemini API

In our experiment, we will provide the video file to Gemini model and ask it to extract few good thumbnails with meaningful captions (using a text prompt). Remember, Gemini 1.5 Pro is multimodal and thus it can accept the text and video (with audio) inputs together.

We will now write a python function for calling the Gemini API. In this function, we first set up the configurations for the model with generation parameters and response schema.

Response schema will help us in setting up a fixed output format beforehand as we want Gemini to provide us the output in a fixed JSON format. A fixed JSON output would be really easy to parse and interpret. Check out the following python code.

def call_gemini(
    prompt: str,
    gcs_video_path: str,
    model: vertexai.generative_models.GenerativeModel,
) -> str:
    """Call Gemini 1.5 Pro API with video and prompt."""
    # define fixed schema for Gemini outputs
    response_schema = {
        "type": "array",
        "items": {
            "type": "object",
            "properties": {
                "timestamp": {
                    "type": "string",
                },
                "caption": {
                    "type": "string",
                },
            },
            "required": ["timestamp", "caption"],
        },
    }
    # model configurations
    generation_config = GenerationConfig(
        temperature=1,
        top_p=0.8,
        max_output_tokens=8192,
        response_schema=response_schema,
        response_mime_type="application/json",
    )
    # creating video input for API call
    video_input = Part.from_uri(
        mime_type="video/webm",
        uri=gcs_video_path,
    )
    # calling Gemini API
    responses = model.generate_content(
        [video_input, prompt],
        generation_config=generation_config,
        stream=False,
    )
    return responses.text

In this function, we have defined the response schema in such a way that the model would return the “timestamps” of the thumbnails from the videos, along with the captions.

Step 7: Parse the model response and display results

We now have the model’s response text with us, which returns a JSON output consisting of the timestamps and captions of the resulting thumbnails. We will use the MoviePy library to extract the frame images from the video using these timestamps.

The following python function, extracts the thumbnails from the video and displays them along with their captions (caption as the title of the image).

def display_results(
    response_text: str,
    clip: moviepy.video.io.VideoFileClip.VideoFileClip,
) -> None:
    """Parse json output, extract thumbnail frames and display."""
    # loading json output object
    json_response = json.loads(response_text)

    # Image plotting settings
    fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(12, 9))

    # extract frame for each timestamp and plot the images
    counter = 0
    for item in json_response:
        timestamp = item["timestamp"]
        caption = item["caption"]
        frame = clip.get_frame(timestamp)
        row, col = counter // 2, counter % 2
        ax[row, col].imshow(frame)
        ax[row, col].set_title(caption, fontdict={"fontsize": 9})
        counter += 1

    fig.show()

Our utility functions are all set, let’s start experimenting now.

Experiment 1: Basic Prompting

In our first experiment, we will use a very basic prompt for thumbnail generation, and in the next experiment we will use a more detailed prompt.

basic_prompt = (
    """Generate 4 thumbnail images from the given video file with short captions."""
)

Let’s call the Gemini API, and get the results:

response_text = call_gemini(
    prompt=basic_prompt,
    gcs_video_path=video_uri,
    model=model,
)

Gemini API call will take a few seconds to generate the results. Let’s display the results.

display_results(response_text, clip)

The following image shows the results of our first experiment with basic prompting. We can see that the model is able to generate some thumbnail candidates but the candidates are not great, also the captions are not telling a good story.

Thumbnail Results with basic prompting

Can we do better with a detailed prompt?

Experiment 2: Advanced Prompting

Let’s write a more detailed prompt explaining what a good thumbnail actually means. This will help the model in understanding what kind of outputs are expected. Let’s try this.

Let’s try out the following prompt (a more detailed one).

advanced_prompt = """You are an expert in video content creation and content marketing.
You have the ability to find best thumbnails from a video and provide meaningful and short and catchy captions for them.
Your task is to find the best 4 thumbnails from a given video along with short, and meaningful captions that is good for marketing.
Consider the following rules while generating thubmnails:

- Thumbnail should have clear focus on the key objects and people, less focus on background
- Thumbnail image should be high quality and bright, avoid blurry images
- Thumbnail image and caption together tell a story
- Thumbnail caption is good for marketing
"""

The following image shows the results after the advanced prompting. We can see that the thumbnail frames are much better this time and even the captions seem really good and meaningful.

Thumbnail Results with advanced prompting.

We can see that the results have improved significantly with better prompting.

Conclusion

In this tutorial, we experimented with the multimodal capabilities of Gemini 1.5 Pro model to extract meaningful thumbnails from an example video, and also generated meaningful captions for them. We noticed that the model is able to understand the video, audio and text simultaneously to answer our question wisely.

We also noticed that — better prompting can improve the results significantly. A detailed and well-written prompt is really useful for the model to understand the objectives of the experiment, and to generate great results.

I would recommend the readers to try this experiment out on more sample videos, and also play around with the prompt. Do share your unique prompts and results by commenting below.

I hope this article was helpful for the readers. Please let me know your feedback!!

Find the python code here: Github code.

Connect with me on LinkedIn.

Thanks for reading.

See you in the next article!

Unlocking Video Insights: Extract Thumbnails with Captions using Gemini 1.5 Pro was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Adversarial Examples and Defence Mechanisms against them

Kartik Chaudhary — Tue, 16 Apr 2024 16:40:04 GMT

Understanding Adversarial Examples and Defence Mechanisms

Adversarial examples are inputs to Machine Learning (ML) models that are intentionally designed to fool the model. These examples are quite easy to generate and can be created by performing intentional feature perturbation on the inputs. And, as a result they can make the ML models do false predictions. In this article, we will cover adversarial examples and defence mechanisms against them.

Adversarial examples are often indistinguishable to the human eye. It is really important to study such inputs because they make ML models vulnerable to attacks. These attacks may cause serious security violations in some scenarios. Thus, it is really important to study them and make ML models robust to such adversarial attacks.

This article covers the following key topics:

What is Adversarial Machine Learning?
What are common types of Adversarial Attacks?
How to defend against Adversarial Attacks?
Why it is hard to defend against Adversarial Attacks?
Conclusion

If you are interested in learning more about the generative learning and Generative Adversarial Networks, Do check out my book:

https://www.amazon.com/GAN-Book-Generative-Adversarial-TensorFlow2-ebook/dp/B0CR8C725C

Let’s get started.

What is Adversarial Machine Learning?

Adversarial Machine Learning is the study of adversarial examples and defence mechanisms against them. Adversarial examples are intentionally generated in order to trick the ML models by using deceptive inputs. Now a days, ML models are extensively used in many security and safety departments and the adversarial attacks are seriously dangerous for such scenarios. Thus, it is important to study such attacks and prepare the defence mechanisms against them.

Based on the intentions of the attacker, the adversarial attacks can be classified into following two categories: the targeted attacks and the un-targeted attacks.

Targeted and Un-Targeted Adversarial Attacks

In case of an un-targeted adversarial attack, the only intention of the attacker is to make the ML model do a wrong prediction (irrespective of the outcome). On the other hand, a targeted attack has a target class and the intention of the attacker is not just to make the model do a mistake but also to make the model do a desired mistake and get a wrong prediction that belongs to the desired output class.

Based on the information that attacker has, the adversarial attacks can be classified into the following two categories: a white-box attack and a black-box attack.

White Box and Black Box Adversarial Attacks

A white-box attack is a scenario where the attacker has full access to the ML model. The
attacker is already aware of the model’s architecture and its parameters.

In a black-box attack, the attacker can only see the output of the ML model and doesn’t
have any other information about it.

ML has become an essential part of almost every organisation and their key business decisions. Many critical tools built for security and safety purposes are developed using ML algorithms today. Thus, the need to protect these ML systems is also growing really fast.

In the next sections, we will learn about some common methods of generating adversarial
examples and defence mechanisms against them.

What are common methods of Adversarial Attacks?

As discussed earlier, the adversarial examples are intentionally prepared to make the ML model do a wrong prediction. These examples appear normal to humans but they can cause misclassifications for the target ML models. Many different methods of generating the adversarial examples have been studied.

Following are some common methods of generating adversarial examples:

Fast Gradient Sign Method (FGSM)
Limited-Memory BFGS (L-BFGS)
Jacobian-based Saliency Map Attack (JSMA)
Deep Fool Attack
Carlini & Wagner Attack
Generative Adversarial Networks (GANs)
Zeroth-Order Optimisation Attack (ZOO)

Let’s learn about these methods.

Fast Gradient Sign Method (FGSM)

Fast Gradient Sign Method, of FGSM, works by using the gradients of the neural network. Because FGSM has access to the model, it comes under the white-box attacking mechanisms. In this method, the input image pixels are perturbed using the information of the gradient of the loss function, such that the model loss is increased. This newly generated input image is called the adversarial image.

The panda image shown in the following Figure, is a very popular adversarial example generated using FGSM method. This image was taken from the research paper titled “Explaining and Harnessing Adversarial Examples-2015” by Ian J. Goodfellow and fellow researchers.

Generating an adversarial example using FGSM, taken from the 2015 paper of Ian
Goodfellow et.al.

Let’s learn about L-BFGS now.

Limited-Memory BFGS (L-BFGS)

Limited memory BFGS, or L-BFGS, is an optimisation algorithm to minimise the number of perturbations added to an image. It is a non-linear gradient-based method. It is quite effective in generating adversarial examples. But it is very computationally expensive.

Next, let’s learn about JSMA.

Jacobian-based Saliency Map Attack (JSMA)

JSMA method uses feature selection method to minimise the number of feature perturbations needed to cause the model make a classification mistake. This method is also very computationally intensive.

Let’s now learn about deep fool attack.

Deep Fool Attack

Deep Fool attack technique works by minimising the Euclidean distance between the perturbed inputs and the original inputs. In this method, the perturbations are added iteratively based on the decision boundaries. This method of generating adversarial examples is also computationally expensive.

Let’s move to the next method.

Carlini & Wagner Attack

This method is based on the L-BFGS algorithm and is more efficient at generating the adversarial examples. This method was able to defeat certain defence mechanisms. Again, this attack is also very computationally expensive.

Let’s learn about GAN based attack now.

Generative Adversarial Networks (GANs)

Generative Adversarial Networks, or GANs for short, have also been used for generating adversarial examples. GANs are capable of generating desired number of plausible examples that are similar to the training dataset. Training a GAN is also computationally intensive and highly unstable.

Let’s now learn about ZOO attack now.

Zeroth-Order Optimisation Attack (ZOO)

ZOO attack is a black-box adversarial attack that works without even the knowledge of the ML algorithm. This method collects gradient information of the ML model indirectly, by querying the model with modified individual features. One common dis-advantage of ZOO technique is that it requires large number of queries to the underlying ML model.

Now that we have learned about some common adversarial attack methodologies. Let’s learn about some of the defence mechanism against them.

How to defend against Adversarial Attacks?

Traditional optimisation techniques such as weight decay, dropouts and so on, generally
don’t provide defence against the adversarial attacks. The paper “Adversarial Attacks and Defences: A Survey-2018” by Anirban chakraborty and team describes some common adversarial attacks and defence mechanisms.

Following are some of the common defence mechanisms against the adversarial attacks,
as per their paper:

Adversarial Training
Defensive Distillation
Gradient Hiding
Feature Squeezing
Blocking the Transferability
Defense GAN
Mag Net

Let’s discuss these mechanisms.

Adversarial Training

The adversarial training is a simple brute-force solution to defend the ML models against the adversarial attacks. In this method, a large number of adversarial examples are generated and then the ML model is explicitly trained on them, to make sure that it is not fooled by them.

In this way, the ML models become robust to such known types of the adversarial examples. However, the attacker might still break this defence using a new attacking mechanism that the model has not seen during the training process.

Let’s move to the second technique now.

Defensive Distillation

This method is based on the concept of knowledge distillation of the Neural Networks where, a small ML model is trained to imitate a very large ML model, in order to obtain the computational savings. This small ML model, trained using the probabilistic outputs of the larger ML model, is robust to the adversarial attacks as its loss surface is smoothed in the directions that an adversary will try to exploit. This mechanism makes it hard for the attacker to tweak the adversarial examples for getting a wrong prediction from the ML model.

Let’s now learn about Gradient Hiding method.

Gradient Hiding

As discussed earlier, some of the attacking mechanisms such as FGSM, rely on the gradient information from the ML model. Gradient hiding is one way to make those attacks ineffective by using non-differentiable models such as Decision Trees, KNN, Random Forests and so on.

Let’s now move to the next defence mechanism.

Feature Squeezing

Feature squeezing is a method of model hardening. In this method, the ML models are made less complex by reducing or squeezing the input features. Simpler ML models are more robust to the small feature perturbations and noise. One dis-advantage with this method is that these ML models are often less accurate because of their simpler nature.

Let’s now learn about Blocking and Transferability method.

Blocking and Transferability

The main reason behind the defeat of most of the well-known defence mechanisms; is due to the strong transferability property of the neural networks. For example, the adversarial examples generated on one classifier are expected to cause another classifier to perform the same mistake. The transferability property holds true even if the classifiers have different architectures or have been trained on disjoint datasets.

Hence, the key for protecting against a black-box attack is to block the transferability of the adversarial examples. Let’s now learn about the next mechanism.

Defense-GAN

The Defense-GAN mechanism leverages the power of Generative Adversarial Networks to reduce the efficiency of the adversarial perturbations. The central idea of this method is to project the input images onto the range of the generator by minimising the reconstruction error, prior to feeding the image to the classifier. Due to this extra step, the legitimate samples will be closer to the range of the generator than the adversarial samples, resulting in substantial reduction of the potential adversarial perturbations.

Let’s now learn about MagNet.

MagNet

MagNet is an adversarial defence framework that uses the classifier as a black box, to read the outputs of the classifier’s last layer without reading the data on any internal layer or modifying the classifier. It uses detectors to distinguish the normal samples from the adversarial examples. The detector measures the distance between the given test example and the manifold and, rejects the sample if the distance exceeds a threshold. It also uses a reformer to reform the adversarial example to a similar legitimate example using the auto-encoders.

Although MagNet is quite successful in thwarting a range of the black-box attacks, its performance degrades significantly in the case of the white-box attacks where, the attackers are supposed to be aware of the parameters of the MagNet. So, the authors came up with an idea of using varieties of auto-encoders and randomly picking one at a time to make it difficult for the adversary to predict which auto-encoder is actually sitting behind the defence.

We now have learned about quite a few defence mechanisms against the adversarial attacks. Although these mechanisms are really good at catching the adversarial attacks, still, they can be defeated if the attacker has significantly large number of computational resources. So, an absolute defence against the adversarial attacks, remains a challenge.

Let’s now learn about the common challenges of defending against the adversarial attacks.

Why it is hard to defend against Adversarial Attacks?

The adversarial examples are hard to defend against because: it is difficult to build a model of the process of creating an adversarial example. The adversarial examples are the solutions to an optimisation problem (or results of an optimisation algorithm) that is non-linear and non-convex for many neural networks.

ML models need to provide the good outputs for every possible input. A considerable modification of the model to incorporate the robustness against the adversarial examples, may also change the elementary objective of the model. And, it may result in a bad performing model.

The defence strategies discussed earlier, work really well against only a certain type of attacks but fail for new kind of attacks because they are not adaptive. If an attacker knows the defence strategies used in a system, then it’s a major vulnerability. Moreover, the implementation of such defence strategies may cause performance overhead as well as the degradation in model accuracy for the actual or intended inputs.

Thus, it is important to design powerful defence mechanism which are adaptive, to protect ML systems from the adversarial attacks. It is also a growing area of research and let’s see what the future holds.

Conclusion

In this article, we learned about the adversarial examples, common types of adversarial attacks, common defence mechanisms against them and the importance of designing the appropriate defence mechanisms against them to protect the ML based intelligent systems.

Specifically, we learned:

What is adversarial Machine Learning? And, what are adversarial examples?Why they are dangerous to ML systems and how to create them?
What are some of the known types of adversarial attacks and why it is hard to defend against them?
How can we make our ML systems robust against the various adversarial attacking mechanisms?

Originally posted on Drops of AI!

I hope this article was helpful, do share your feedback by commenting below. See you in the next article!

Understanding Failure Modes of GAN Training

Kartik Chaudhary — Mon, 08 Apr 2024 14:45:44 GMT

Mode Collapse and Convergence Failure of Generative Adversarial Networks

Failure Modes of GAN Training

The idea of two competing neural networks is no doubt interesting; where, at each step one of them attempts to defeat the other one and in the process, both networks keep getting better at their job. But building such a dynamic training system is not always feasible. Generative Adversarial Networks, or GANs, are really difficult to train in practice. GANs usually suffer from two common types of failure modes during training. It is really important to understand these failure modes before training a GAN based model otherwise, we might not be able to develop and train a stable GAN based model. This article will mainly focus on explaining those two common failure modes in details, along with python examples. Once we are able to understand these failure modes, it will be easier for us to start developing and training our own stable GAN based models to generate the data of our choosing.

An image generated using Google Gemini model | Mode Collapse and Convergence Failure of Generative Adversarial Networks

This article covers the following topics:

Why GAN training is unstable?
What are GAN failure modes?
What is mode collapse in GAN training?
What is convergence failure in GAN training?
Summary

Let’s get started.

1. Why GAN training is unstable?

Generative Adversarial Networks, or GANs for short, are quite difficult to train in practice. GAN training can be visualised as a two-player zero sum game. The game between the generator and the discriminator is non-cooperative which means that improvements in one model come at the expense of the other model. This kind of setup is only meaningful (or stable) when both players are capacity-wise balanced. For example: If one of the players is very strong, it will dominate the game and will not let the weaker player gain anything. So, basically the weaker model might not be able to learn anything useful. And thus, the balance between two competing models is very important.

Finding this kind of balance between the generator and the discriminator model can be related to the problem of finding Nash Equilibrium (a game theory concept that determines the optimal solution in a non-cooperative game). A Nash Equilibrium occurs when each player has the minimal cost. Both the players in a GAN setup are neural networks (functions of high-dimensional parameter space), parameters are continuous variables, and the cost functions are non-convex. Finding the Nash Equilibrium in such a setup is not trivial. Also, there is no known algorithm for the same.

The key idea of keeping each player at minimum cost (Nash Equilibrium) motivates the use of traditional gradient descent-based learning methods. These learning methods can be leveraged for minimising the cost of both players simultaneously. But in our GAN setup, modification in one player’s cost using gradient descent, might influence the other player in a wrong way as well. This is the key reason that leads to the convergence failure of GANs very frequently

If you are interested in learning more about the generative learning and Generative Adversarial Networks, Do check out my book:

https://www.amazon.com/GAN-Book-Generative-Adversarial-TensorFlow2-ebook/dp/B0CR8C725C

As we can see that GANs are very unstable and difficult to train. Thus, it is important to study these common failure modes before developing GANs. Let’s learn about the common GAN failure modes.

2. GAN failure modes

The best way to check whether the GAN training is successful or not, is to monitor the generated samples very carefully and frequently (along with model loss). After looking at the generated content for a few iterations of training, an experienced AI practitioner will know whether the training is going in the right direction or not. If the training is not going in the right direction, one should stop the training and make required changes to the model and start training again.

Understanding Failure Modes of GAN Training

Before starting to develop or train GANs, it is important to understand the common failure modes of GAN training. Although, there can be lot many reasons behind the training failure of a GAN like setup, and sometimes they may even be related to the training dataset as well. It is really difficult to identify the exact reason of failure every time. However, the researchers have observed the following two failure modes to be occurring very frequently in case of GANs.

Mode Collapse
Convergence Failure

Let’s learn more about these failure modes.

3. Mode Collapse in GANs

Whenever we train a GAN based generative model, we expect it to generate wide variety of realistic samples. But it’s not always the case. Sometimes, due to an unbalanced training setup, the generator network fails to learn the correct representations and thus, fails to generate large variety of content. This problem with the training of GANs is known as ‘mode collapse’. Mode collapse is quite easy to detect but very difficult to solve. Basically, mode collapse means that the generator model collapses to generate only a limited variety of samples. This failure can be easily detected by inspecting the generated samples during the training phase. If there is very little diversity in the generated samples and some of the generated samples are exactly similar, then there are high chances of mode collapse.

Sometimes, all of the generated images are exactly same. Such scenario is known as a complete collapse. And the case when the generator collapses to a small variety of samples is termed as a partial collapse of GAN. Partial collapse is quite common whereas we don’t get to see a complete collapse very often. Let’s learn about the reasons behind a mode collapse.

3.1 Reasons behind Mode Collapse

Mode Collapse happens when the generator network fails to generate a wide variety of content. If the generator network is having a mode collapse and as a result it is producing similar kind of samples again and again, Then, the duty of an ideal discriminator network is to learn to reject those repeated samples as fake. And thus, it should make the generator put some more work into generating samples. But it is not always the case. Sometimes, the discriminator network gets stuck in a local minimum and fails to get out of it in future training steps, the generator takes advantage of this situation and learns to generate only a few variations of images which are good enough to fool the discriminator. In other words, both the players keep optimising to exploit the short-term weakness of the opponent.

A generator network collapsing for a small variety of samples, even when the input noise vector (z) is always different, indicates that the generator output is independent of z (in simple words, the gradient associated with z approaches to zero). Now that we have built an intuition behind the most probable reasons of model collapse, let’s see how can we solve it.

3.2 Solving Mode Collapse

Mode collapse, a very common problem leading to the failure of GANs, has led the researchers to find out multiple different ways to solve it. However, we can’t confidently claim that a particular hack would work in a given situation. Following is a list of some of the common hacks (or best practices) to solve mode collapse and make the GANs output wide variety of samples:

Wasserstein GAN solves this problem with the help of a new loss function that gives the flexibility of training the discriminator to optimality without worrying about the vanishing gradients problem. We will learn about Wasserstein GANs later in this book.
Unrolled GANs use a different loss function for the generator that doesn’t let it over-optimize for a particular discriminator state. This setup takes feedback from multiple discriminators while updating the generator parameters.
Increasing the dimensions of the input vector (z) can introduce more variety to the generated content.
Making the generator more complex (or deeper), such that it is capable of learning the complex representations/features.
Impairing the discriminator by randomly giving some false labels.

The hacks listed above have proved to be useful in solving many mode collapse cases as per the study conducted by researchers. We may not be able to tell which hack might solve a given mode collapse problem, but with these hacks, our we should be able to solve most of our mode collapse related issues. Next, let’s learn about another common failure mode of GANs: Convergence Failure.

4. Convergence Failure in GANs

Convergence failure is a scenario when GAN training fails to converge without even producing any meaningful content. GAN training is a two-player zero sum game where each player learns at the cost of the other one (at each iteration), such a setup needs both players to be chosen carefully in order to establish an equilibrium. If one of the players dominates, the whole GAN setup fails to converge. This kind of problem with the training of GANs is known as Convergence Failure. And in case of a convergence failure, GANs don’t produce any meaningful output.

Convergence failure is quite frequently observed during the training of GANs. Following are the ways to identify convergence failure:

The loss of the discriminator network rapidly decreases to a value very close to zero and stays there during the remaining training iterations.
The samples generated by the generator network are very bad quality and it’s a trivial job for the discriminator to flag them as fake.
The loss of the generator network either reaches zero or keeps continuously increasing during the entire training phase.

If our GAN training fails and shows the behavior related to any of the above-described scenarios, our model is most probably having a convergence failure situation. Next, let’s learn about the main reasons behind the convergence failure of GAN training.

4.1 Reasons behind Convergence Failure

Convergence failure happens when there is no balance between the generator and the discriminator networks. There can be two possible reasons behind this imbalance between the two networks. Following are those two possible scenarios:

The first scenario is observed when the discriminator is weak, and the generator dominates. In this case, the discriminator is incapable of distinguishing real samples from the fake ones and thus it is not able to provide useful feedbacks to the generator. The generator loss reaches close to zero despite the bad quality of samples generated by it.
Another possible scenario is when the discriminator is too good at its job. It means that the samples generated by the generator are always rejected and the generator is never able to fool the discriminator during the entire training process. The generator loss in such cases, keeps increasing during training and it produces garbage outputs.

We now understand the key reasons behind the convergence failure of a GAN training setup. Let’s learn about some ways of solving it.

4.2 Solving Convergence Failure

There are quite a few tricks to solve the convergence failure problem of GANs. All we have to do is: make the generator and discriminator networks balanced, such that neither of them dominates during the training. Let’s learn about some ways of balancing the two networks.

Solving for the first scenario where the Generator dominates:

Make the discriminator deeper, thus better at rejecting the fake samples and making the generator put more work.
Impairing the generator by adding dropout layers, thus it learns slower and in sync with the discriminator network.
Making the generator weaker by removing some layers.

Solving for the second scenario where the Discriminator dominates:

Impairing the discriminator by randomly giving some false labels to real images.
Impairing the discriminator by adding dropouts (or other regularization techniques).
Making the generator deeper, thus better at learning representations/features and producing realistic samples.
Making the discriminator weaker by removing few layers, thus it learns slowly and in sync with the generator.

We have just learned about multiple tricks to balance the models for each scenario of convergence failure. All we have to do here is to balance the learning process of both models so that they learn in sync and keep improving each other. Now let’s jump into some hands-on experiments.

Conclusion

This article highlights the fact that GANs are quite difficult to train. GANs usually suffer from two common training failure scenarios: model collapse and convergence failure. It is quite important to learn to identify and overcome these failure modes before starting to work on GANs. After reading this article, the readers should be confident about answering the following questions about GAN failure modes:

Why GANs are hard to train? What are the common problems they suffer from during training?
How to identify the mode collapse failure during the GAN training? What are the different tricks to solve it?
What are the common reasons behind the convergence failure of a GAN? How can we identify it during the training process? What are the tips and tricks to avoid and solve this issue?

Image Synthesis using Pixel CNN based Autoregressive Generative Model

Kartik Chaudhary — Sat, 17 Feb 2024 16:32:26 GMT

Implementing Pixel CNN for Handwritten Digit Image Generation

Introduction

Recent advances in the field of deep learning have led to the development of complex generative models that are capable of generating high quality content in the form of text, audio, pictures, videos and so on. Generative models that make use of deep learning architectures to tackle the task of learning distributions, are known as deep generative models. Due to the flexibility and scalability of neural networks, deep generative models have become the most exciting and swiftly evolving field of ML and AI. Deep generative modeling techniques have helped in developing modern AI agents that are constantly generating and processing vast amounts of data.

A beautiful Sunset | Image by Author

Generative learning with deep neural networks is creating wonders today. There are quite a few popular deep generative learning frameworks that are an active area of research. In this article we will learn about one such generative learning framework — Autoregressive Generative Models. We will also develop a Pixel CNN based autoregressive generative model from scratch.

This article has the following four parts.

Autoregresssive Generative Models
Pros and Cons of Autoregresssive Generative Models
Image Synthesis using Pixel CNN
Conclusion

Let’s get started.

1. Autoregressive Generative Models

The term ‘autoregressive’ is taken from the field of time-series forecasting where a model considers the past observations in an ordered or timely manner, to make predictions about the future. Autoregressive generative models are also quite similar in nature, as they also take help from all of their past predictions for deciding the subsequent prediction.

For example: an autoregressive model may generate an image by predicting one pixel at a time. Here each new pixel value is estimated based on the previously predicted or observed pixel values. In statistical terms, an autoregressive model is trained to learn the conditional distribution of each individual pixel given the surrounding pixels that are already known (or have already been estimated by the same model in earlier iterations).

Pixel CNN, Pixel RNN, Character CNN, Character RNN and Wave-Net are some popular examples of the Deep Autoregressive Generative models. Some of these models have also been successfully applied in the field of anomaly detection and adversarial attacks detection.

2. Pros and Cons of Autoregressive Generative Models

Following is a list of some pros of autoregressive generative models:

Easy to understand and calculate likelihoods
Training process is supervised and very straight-forward

To get a deeper and intuitive understanding of the autoregressive generative modeling framework, check out my article on :

What are Autoregressive Generative Models?

Let’s now look at some of the cons of autoregressive approach:

Requires an ordering of random variables
Generation process is sequential, hence very slow
High likelihood does not guarantee better looking samples in practice
Does not learn unsupervised features (or representations)

Now that we know the basic idea behind autoregressive generative models, let’s develop a Pixel CNN model from scratch for image synthesis.

3. Image Synthesis using Pixel CNN

In this example, we will implement and train an autoregressive generative model called ‘Pixel CNN’ on MNIST handwritten digits dataset and verify its generative capabilities.

The code related to this experiment can be found here on Github.

This example has the following steps:

Download and show data
Data Preparation
Define Masked CNN layers
Define Pixel-CNN
Training Pixel-CNN Model
Results

Let’s get started.

Step 1: Importing Libraries

As a first step, we will import some useful python libraries in a Jupyter Notebook cell. See the following snippet.

import numpy as np
import matplotlib.pyplot as plt  
import pandas as pd  
import seaborn as sns  
import warnings  
warnings.filterwarnings('ignore')  
%matplotlib inline  
import tensorflow  
print (tensorflow.__version__)

We will be using TensorFlow library with version 2.4.1 for our experiments.

Out[1]: 2.4.1

Now, let’s download the dataset.

Step 2: Download and show data

In this step, we will download the MNIST Handwritten Digits dataset. This is a very common dataset and thus it is also available in keras datasets. The following python code downloads the data and plot’s some handwritten digit images to verify the samples.

from tensorflow.keras.datasets import mnist

(trainX, trainy), (testX, testy) = mnist.load_data()  
print('Training data shapes: X=%s, y=%s' % (trainX.shape, trainy.shape))  
print('Testing data shapes: X=%s, y=%s' % (testX.shape, testy.shape))  
for k in range(9):  
    plt.figure(figsize=(9,6))  
    for j in range(9):  
        i = np.random.randint(0, 10000)  
        plt.subplot(990 + 1 + j)  
        plt.imshow(trainX[i], cmap='gray_r')  
        plt.axis('off')  
        #plt.title(trainy[i])  
    plt.show()

Here is the output:

Figure 1 : A glimpse of MNIST Handwritten Digits dataset.

Figure 1 gives a basic idea about the MNIST handwritten digits dataset. It has around 60k training images and 10k test images each with dimensions: 28 x 28.

Now, let’s prepare the dataset for the model.

Step 3: Data Preparation

In this step, we will prepare our dataset for the model training purpose. Firstly, we will convert these images into binary format (as expected by our basic Pixel CNN). Secondly, we will add a channel dimension to all the images, as expected by the Convolutional layers in our Pixel CNN.

trainX = np.where(trainX < (0.33 * 256), 0, 1)
train_data = trainX.astype(np.float32)  
testX = np.where(testX < (0.33 * 256), 0, 1)  
test_data = testX.astype(np.float32)  
train_data = np.reshape(train_data, (60000, 28, 28, 1))  
test_data = np.reshape(test_data, (10000, 28, 28, 1)) 
print (train_data.shape, test_data.shape)

Following is the final shape of our dataset after pre-processing:

Out[3]: (60000, 28, 28, 1) (10000, 28, 28, 1)

Our data is now ready. We can now define the Pixel CNN architecture.

Step 4: Define Masked CNN layers

In this step, we will define some building blocks of our Pixel CNN framework. First, we will define a PixelConvLayer class. A Pixel CNN layer is simply built on top of 2D Convolutional layers. Only difference is that it includes masking in order to accommodate the autoregressive training of the network. In simple words, autoregressive models generate output one pixel at a time, so, masking is a clever way to train Pixel CNN with regular CNN layers.

See the python code below for Pixel CNN layer implementation:

class PixelConvLayer(tensorflow.keras.layers.Layer):
    def __init__(self, mask_type, **kwargs):  
            super(PixelConvLayer, self).__init__()  
            self.mask_type = mask_type  
            self.conv = tensorflow.keras.layers.Conv2D(**kwargs)  
        def build(self, input_shape):  
            # Build the conv2d layer to initialize kernel variables  
            self.conv.build(input_shape)  
            # Use the initialized kernel to create the mask  
            kernel_shape = self.conv.kernel.get_shape()  
            self.mask = np.zeros(shape=kernel_shape)  
            self.mask[: kernel_shape[0] // 2, ...] = 1.0  
            self.mask[kernel_shape[0] // 2, : kernel_shape[1] // 2, ...] = 1.0  
            if self.mask_type == "B":  
                self.mask[kernel_shape[0] // 2, kernel_shape[1] // 2, ...] = 1.0  
        def call(self, inputs):  
            self.conv.kernel.assign(self.conv.kernel * self.mask)  
            return self.conv(inputs)

Secondly, we will define a residual block layer. It is also a normal residual block, which is based on the Pixel CNN layer. See the following code:

class ResidualBlock(tensorflow.keras.layers.Layer):
    def __init__(self, filters, **kwargs):  
            super(ResidualBlock, self).__init__(**kwargs)  
            self.conv1 = tensorflow.keras.layers.Conv2D(  
                filters=filters, kernel_size=1, activation="relu"  
            )  
            self.pixel_conv = PixelConvLayer(  
                mask_type="B",  
                filters=filters // 2,  
                kernel_size=3,  
                activation="relu",  
                padding="same",  
            )  
            self.conv2 = tensorflow.keras.layers.Conv2D(  
                filters=filters, kernel_size=1, activation="relu"  
            )  
        def call(self, inputs):  
            x = self.conv1(inputs)  
            x = self.pixel_conv(x)  
            x = self.conv2(x)  
            return tensorflow.keras.layers.add([inputs, x])

We now have the building blocks ready for our Pixel CNN. We can now go ahead and define the final Pixel CNN architecture.

Step 5: Define Pixel CNN

In this step, we will define the final Pixel CNN model architecture. This architecture is very similar to the one shown in original Pixel CNN paper. It combines Pixel CNN layers with residual blocks as shown in the following python code:

inputs = tensorflow.keras.Input(shape=(28,28,1))
x = PixelConvLayer(  
    mask_type="A", filters=128, kernel_size=7, activation="relu", padding="same"  
)(inputs)  
for _ in range(5):  
    x = ResidualBlock(filters=128)(x)  
for _ in range(2):  
    x = PixelConvLayer(  
        mask_type="B",  
        filters=128,  
        kernel_size=1,  
        strides=1,  
        activation="relu",  
        padding="valid",  
    )(x)  
out = tensorflow.keras.layers.Conv2D(  
    filters=1, kernel_size=1, strides=1, activation="sigmoid", padding="valid"  
)(x)  
pixel_cnn = tensorflow.keras.Model(inputs, out)  
pixel_cnn.summary()

Following is the final summary of our Pixel CNN model. It has roughly 500k trainable parameters.

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0         
_________________________________________________________________
pixel_conv_layer (PixelConvL (None, 28, 28, 128)       6400      
_________________________________________________________________
residual_block (ResidualBloc (None, 28, 28, 128)       98624     
_________________________________________________________________
residual_block_1 (ResidualBl (None, 28, 28, 128)       98624     
_________________________________________________________________
residual_block_2 (ResidualBl (None, 28, 28, 128)       98624     
_________________________________________________________________
residual_block_3 (ResidualBl (None, 28, 28, 128)       98624     
_________________________________________________________________
residual_block_4 (ResidualBl (None, 28, 28, 128)       98624     
_________________________________________________________________
pixel_conv_layer_6 (PixelCon (None, 28, 28, 128)       16512     
_________________________________________________________________
pixel_conv_layer_7 (PixelCon (None, 28, 28, 128)       16512     
_________________________________________________________________
conv2d_18 (Conv2D)           (None, 28, 28, 1)         129       
=================================================================
Total params: 532,673
Trainable params: 532,673
Non-trainable params: 0
_________________________________________________________________

Our Pixel CNN architecture is now ready. We will utilize Adam optimizer with a learning rate of 0.0005 for updating the weights of our model during training. Also, we will utilize ‘binary_crossentropy’ as a loss function for our model.

Remember, Pixel CNN generates image one pixel at a time, and our setup considers binary images with values 0 and 1. Thus, ‘binary_crossentropy’ loss makes sense here.

Compiling Pixel CNN:

adam = tensorflow.keras.optimizers.Adam(learning_rate=0.0005)
pixel_cnn.compile(optimizer=adam, loss="binary_crossentropy")

Our Pixel CNN is now ready for training.

Step 6: Training Pixel-CNN Model

In this step, we will train the Pixel CNN model on MNIST Handwritten Digits dataset. We will train our model for 50 epochs with a batch size of 128.

pixel_cnn.fit(  
    x=train_data,
    y=train_data,
    batch_size=128,
    epochs=50,
    validation_data=(test_data, test_data),
    verbose=1,
)

Following are the training logs. We can see that the validation loss is decreasing consistently, which proves that the model is learning:

Epoch 1/50
469/469 [==============================] - 57s 116ms/step - loss: 0.1748 - val_loss: 0.0923
Epoch 2/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0918 - val_loss: 0.0892
Epoch 3/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0894 - val_loss: 0.0882
Epoch 4/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0881 - val_loss: 0.0873
.  .  .  .  .
.  .  .  .  .
.  .  .  .  .
.  .  .  .  .
Epoch 45/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0815 - val_loss: 0.0829
Epoch 46/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0815 - val_loss: 0.0827
Epoch 47/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0813 - val_loss: 0.0829
Epoch 48/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0812 - val_loss: 0.0829
Epoch 49/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0813 - val_loss: 0.0830
Epoch 50/50
469/469 [==============================] - 53s 113ms/step - loss: 0.0812 - val_loss: 0.0828

As our model training is complete now, we can now check the results.

Step 7: Results (Image Synthesis using Pixel CNN)

The following python code utilizes our Pixel CNN model for generating handwritten digit images. It also plots some of the generated results.

from IPython.display import Image, display
from tqdm import tqdm_notebook  
# Create an empty array of pixels.  
batch = 81  
pixels = np.zeros(shape=(batch,) + (pixel_cnn.input_shape)[1:])  
batch, rows, cols, channels = pixels.shape  
# Iterate over the pixels because generation has to be done sequentially pixel by pixel.  
for row in tqdm_notebook(range(rows)):  
    for col in range(cols):  
        for channel in range(channels):  
            # Feed the whole array and retrieving the pixel value probabilities for the next  
            # pixel.  
            probs = pixel_cnn.predict(pixels)[:, row, col, channel]  
            # Use the probabilities to pick pixel values and append the values to the image  
            # frame.  
            pixels[:, row, col, channel] = tensorflow.math.ceil(  
              probs - tensorflow.random.uniform(probs.shape)  
            )  
counter = 0  
for i in range(9):  
    plt.figure(figsize=(9,6))  
    for j in range(9):  
        plt.subplot(990 + 1 + j)  
        plt.imshow(pixels[counter,:,:,0], cmap='gray_r')  
        counter += 1  
        plt.axis('off')  
    plt.show()

Figure 2 shows the output of our Pixel CNN model. We can see that the output images look very close to the actual handwritten digits. They are not perfect though, due to the limitations of the framework, model capacity and dataset.

Figure 2: Handwritten Digit Images generated using the Pixel CNN model.

Yay! We have now successfully implemented, trained and validated the Pixel CNN model.

4. Conclusion

In this article, we discussed about a popular generative learning framework: Autoregressive Generative Models. We have also implemented a Pixel CNN based autoregressive generative model for image synthesis from scratch in python. Our results prove that Pixel CNN framework is capable of generating images one pixel at a time. Although the results are not perfect. These results can be improved by increasing the capacity of the model, increasing the size of the dataset, and also by using a better generative learning modeling framework such as — Generative Adversarial Networks (GANs), Variational AutoEncoders (VAEs), Diffusion based models and so on.

If you are interested in learning more about the generative learning and Generative Adversarial Networks, Do check out my book:

https://www.amazon.com/GAN-Book-Generative-Adversarial-TensorFlow2-ebook/dp/B0CR8C725C

I hope this article was helpful. Please let me know your feedback/suggestions by commenting below.

See you in the next article!

Understanding Generative Learning for Computer Vision

Kartik Chaudhary — Wed, 07 Feb 2024 15:40:23 GMT

A Gentle Introduction to Generative Learning Models!

Generative Learning refers to a special class of statistical models that are capable of generating content that is very hard to distinguish from the reality (or fake content that looks real). The generated content could be poems, images, music, songs, videos, 3D objects or content from some other domain we could imagine. A domain is nothing but a fancy word for a bunch of examples that follow some common pattern. In this article, we will learn: how does a generative learning model work?

Interesting part about the generative models is that, sometimes, the content generated by them is not just realistic, its completely new as well (or unseen in the training examples). Everyone must have seen or heard about the modern technologies that can generate very realistic faces of the people that do not even exist in the world. Projects such as Face aging apps, Virtual try-on, Photos to paintings and a lot more advancements with similar technologies are examples of the generative models.

Let’s now learn how exactly does a generative learning model work.

Check out my introductory article on “Generative Learning and its Differences from the Discriminative Learning”

How does a Generative Learning Model work?

To understand how does the generative learning work: lets first define an example problem and then we will discuss how a generative model will solve it. Let’s assume that we have a dataset (D) of 1 million cat images representing multiple breeds of cats across the world and the photographs have been taken from almost all possible angles. Note that the number 1 million is significant here, as generative models generally require larger datasets to estimate the target distribution more accurately.

Because a generative learning approach estimates the data distribution to solve a problem, our focus is to define a generative model that is capable of learning the distribution (𝑷𝒄𝒂𝒕𝒔) that these cat images represent. Every dataset represents some data distribution that it is originally sampled from, and that data distribution is known as the true distribution of that particular dataset. Here 𝑷𝒄𝒂𝒕𝒔 is the distribution of all possible cat images in the universe and this dataset is sampled from it as a representative of the true data distribution.

If somehow, our model (p) is able to learn the distribution 𝑷𝒄𝒂𝒕𝒔, It will be able to answer all possible questions about cats present in this universe. For example:

It will be able to tell whether a given image x represents a cat or not. If the
likelihood value 𝒑(𝒙) is high, then x is definitely a cat or vice-versa.
Secondly, if you go ahead and sample an image from 𝒑(𝒙), it will always be a cat image. In this way, it will be able to generate cat images infinitely.

This example gave us a much better understanding of generative models. We now understand that a generative model first learns the underlying data distribution so that later it could answer any questions about that data. But in reality, learning a data distribution is not trivial. To understand how complex, it can be to learn a joint distribution, we first need to understand what does a joint distribution actually mean?
The following subsection explains the joint distributions.

1. Joint Distribution

As we all are aware, the digital images are made up of pixels. Each pixel inside an image, represents a colour and a group of such colour pixels, may represent the objects inside that image. In digital computers and smartphones, each pixel is represented using three discrete random variables R, G and B representing the intensity of three colours — Red, Green and Blue.

In a given digital image, each colour pixel represented by these three random discrete variables, can choose any random discrete integer value from the range [0, 255] for each variable R, G and B. We can represent the joint distribution of a single-coloured pixel by 𝒑(𝑹, 𝑮, 𝑩) such that sampling from this distribution (𝒓,𝒈, 𝒃)~𝒑(𝑹, 𝑮, 𝑩) always generates a colourful pixel. In this case, the total number of parameters required to
specify the joint distribution 𝒑(𝑹, 𝑮, 𝑩) would be:

= 256 x 256 x 256–1 = 256^3–1

Here, as each random variable has 256 possible values, so total parameters required to specify this true distribution would be on less than the total possible combinations, as shown in the calculation above.

This was just a single pixel, now think about an image with 100x100 dimensions (though it’s a pretty low-resolution image) that is made up of 10,000 such colourful pixels. Now, can you imagine the number of parameters required to represent a true joint distribution of all such possible 100x100 dimension colour images? Pretty huge right. Let’s calculate it. We just need to multiply the number of possible combinations of one coloured pixel ten thousand time. Check the following calculation.

= (256^3–1) x (256^3–1) x …… 10,000 times
= 256^30,000 (approximately)

This number is pretty huge. Now if I ask you, can you prepare a dataset that can efficiently represent the above-described distribution of color images with 100 x 100 resolution? The answer is pretty obvious:

“Never”.

It is impossible to practically represent the true data distribution in this case, no matter how big dataset you have, it’s never enough. Any given dataset, representing a distribution 𝑷𝑫𝒂𝒕𝒂𝒔𝒆𝒕 , is a “not very efficient” representative of the true data distribution. Now, one question that pops up in our mind is: Do we really need to model the true joint distribution? Can we settle for less (something like 𝑷𝑫𝒂𝒕𝒂𝒔𝒆𝒕)?

Actually, modelling the true distributions is pointless as they are deterministic in nature. In other words, if we already have the required information about the true distribution, we don’t really need to model it. For example: the distribution of all the possible colorful images of dimensions 100x100, we don’t actually need to model it.

Because we already know that any random color image of 100x100 dimensions will always belong to the aforementioned distribution with 100% confidence. Thus, there is no point in learning such distributions.

The aforementioned data distribution is deterministic in nature, because we assumed the pixels to be independent from each other. What if the pixels are somehow related? This relation between pixels can restrict the given true distribution to represent only a particular class of colour images, such as the dataset of 1 million cats where pixels are not independent.

The dataset of 1 million cat images can be considered as a restricted distribution due to the pixel relationships. Learning this kind of restricted distribution, instead of the true joint distribution described above, can be helpful. To understand this, let’s get into more details about the restricted distributions next.

2. Restricted Distribution

Now, let’s get back to our dataset of 1 million cat images. Let’s assume that our dataset has the distribution 𝑷𝒄𝒂𝒕𝒔 which is supposed to be close to the real distribution of all the possible cats in this universe. Imagine, we are able to learn a generative model 𝑷𝑴𝒐𝒅𝒆𝒍 (model distribution) such that 𝑷𝑴𝒐𝒅𝒆𝒍 is very close to 𝑷𝒄𝒂𝒕𝒔 (from our dataset).

Using this model distribution (𝑷𝑴𝒐𝒅𝒆𝒍), we should be able to perform the following tasks such as –

Generation: Sampling from the model (𝒙𝒏𝒆𝒘 ~𝑷𝑴𝒐𝒅𝒆𝒍) will always generate a cat image and it will give us the flexibility of generating infinite number of cat images if required. (See example in Figure below).
Prediction: It will be able to tell whether a given image x, represents a cat or
not. If the likelihood value 𝑷𝑴𝒐𝒅𝒆𝒍(x) is high, x is a cat or vice-versa.
Representation Learning: The model will be capable of learning the
unsupervised features related to cats such as breed, colour, eyes, tail and so on
without explicitly providing labels for these attributes.

Generated sample x, sampled from probability distribution p(x)

Given the above notion, a conditional generative model is also possible. Suppose that we want to generate a set of variables: Y, given some other set of variables: X, we can directly train a generative model to learn the conditional distribution 𝑷(𝒀|𝑿) without worrying about the joint distribution. This is very similar to the sequence generation
tasks where the next candidate of the sequence is predicted given some already existing candidates. Another popular example of conditional generative models is: Latent variable based generative models. Let’s discuss how latent variable based generative models actually work.

3. Latent variable based Generative Models

To understand the latent variable based generative models, let’s get back to our dataset of 1 million cat images. This dataset is not annotated and it means that there is no information about the kind of cat, that is present in a given picture. Now suppose that we want to train a generative model on this dataset, so that later we can use it to generate some cat images. But this time we would like our model to generate the images of desired type of cats, instead of generating random cat images. This time, we are asking the model to learn unsupervised features as well, along with the data
distribution so that it is able to answer questions like: Generate an orange long-haired cat image! The term ‘unsupervised features’ makes sense here because we are not providing the model with any labelled information for learning these features.

Latent Variable: A Latent variable is a variable that is hidden or that is not directly observed but is actually inferred from other variables that are observed.

The idea is to learn these unsupervised features, such as colours, hair-length, poses and so on, with the help of a latent vector (z). Here, the latent vector z is expected to represent these high-level features of cat images. In this case, a cat image of desired type can be sampled from the conditional distribution 𝒑(𝒙|𝒛), if we are able to provide the correct value of z here. Now, our objective has changed and our new goal is to learn the conditional distribution 𝒑(𝒙|𝒛), instead of the joint distribution P(x) which was
more complex. Figure below shows the high-level idea of sampling from this new model, this time sampling is conditioned on the latent input.

Generated Sample x, sampled from conditional distribution p(x | z)

Now the real question is: How do we know what value of z generates which type of cat images? Because the training is also completely unsupervised (or unlabelled), we can’t really have control over the latent variables.

But here the trick, we will let our model learn the conditional distribution 𝒑(𝒙|𝒛) and then, we can simply reverse the equation and learn the reverse conditional distribution 𝒑(𝒛|𝒙). Using this simple trick, we will be able to map the generated cats back to their input latent vector z, and hence, we will be able to find out what value of z generates which type of cat image.

If you are interested in learning more about the generative learning and Generative Adversarial Networks, Do check out my book:

The GAN Book: Train stable Generative Adversarial Networks using Tensorflow2, Keras and Python

Our plan seems simple, but again, learning this conditional distribution 𝒑(𝒙|𝒛) is a challenging task and we cannot be 100% sure that it will learn the desired features corresponding to different values of the latent vector z (so that we could map them back by reversing the process), as the training is completely unsupervised (or uncontrollable).

Another important thing to consider is that we are talking about training models to learn the distributions here. To accomplish that –

“We need a way to compare distributions”!

As we have discussed earlier as well, the real job of a generative model is to learn the distribution 𝑷𝑴𝒐𝒅𝒆𝒍 which is really close to the true data distribution 𝑷𝑻𝑹𝑼𝑬 (here, 𝑷𝑫𝒂𝒕𝒂𝒔𝒆𝒕 can be a good proxy for the true distribution). During the learning phase, a generative model may request to know how close the model distribution (𝑷𝑴𝒐𝒅𝒆𝒍) is to the expected distribution (𝑷𝑫𝒂𝒕𝒂𝒔𝒆𝒕). For this reason, we need a function that is capable of efficiently comparing and quantifying the closeness of two distributions (e.g., 𝑷𝑴𝒐𝒅𝒆𝒍 and 𝑷𝑫𝒂𝒕𝒂𝒔𝒆𝒕).

We will cover the different ways of learning or comparing distributions in my subsequent articles.

Conclusion

In this article, we learned about some basics related to the generative models. We have also understood how does a generative learning model work, with the help of some examples. We also learned about the the joint distributions, restricted distributions and the latent variable based generative models on a very high level. Next important thing is: finding ways to learn or compare distributions in practice. We will cover this topic in my subsequent articles on generative learning.

I hope this article was helpful and gave you a good basic understanding of the generative models. If you find this useful kindly share, if you find any mistakes please let me know your valuable feedback by commenting below.

See you in my next articles on generative learning!!

Originally published on Drops of AI !

Denoising AutoEncoders can reduce noise in images

Kartik Chaudhary — Wed, 22 Nov 2023 13:06:02 GMT

Developing denoising autoencoders with keras and TensorFlow

Autoencoders are unsupervised Deep Learning techniques that are extensively used for dimensionality reduction, latent feature learning (Learning Representations), and also as generative models (Generative Adversarial Networks: GANs). Denoising Autoencoders are slight modifications to the vanilla autoencoders that can be used for reducing noise from real-world noisy datasets. In this tutorial, we will investigate Convolutional Denoising Autoencoders to reduce noise from the images.

Autoencoders have proved to be very useful in learning complex representations of data and are utilized to solve many complex real-world problems. In my previous article on ‘Autoencoders in Keras and Deep Learning’, I have given a basic overview of autoencoders using Keras in python.

This tutorial will focus on Convolutional Denoising Autoencoders where we will train a denoising autoencoder from scratch using Keras and TensorFlow. Here is a quick peek into the content-

Convolutional Denoising Autoencoders
Data Preparation
Training Convolutional Denoising Autoencoder
Results
Summary

1. Convolutional Denoising Autoencoders

Background

Autoencoders are typically used for representing data in lower-dimensional space(latent feature space), usually for dimensionality reduction tasks. They achieve this task by first converting input data into a lower-dimensional encoding (bottle-neck) and then take that encoding to reconstruct the original data as an output. In this way, the model learns the important aspects of input data as latent features(ignoring non-important information) such that it is able to recreate it efficiently.

Denoising Autoencoders are slightly different in working. They learn to capture only useful information from input data (ignoring noise) by changing the reconstruction criterion. Denoising Autoencoders are used for reducing noise from noisy datasets. They take noisy data points as input and reconstruct the clean version as output thus the input data and output data are not the same in the case of the Denoising Autoencoders.

This kind of training (noisy data -> clean data), helps the model to learn only valuable features(ignoring the noise) in latent space, and thus it re-constructs the clean version. This kind of setup is utilized in multiple applications for cleaning real-world noisy datasets.

Denoising Autoencoder will only be able to remove noise from the dataset when the following two conditions are true-

Original Features of data are stable and robust to noise.
The model should be able to learn the useful structure from the distribution of the dataset.

Convolutional Denoising Autoencoders

Denoising autoencoders work well in multiple different domains(or application areas) with slight modifications depending upon the kind of dataset is being fed. In this tutorial, we will learn about one such variant called convolutional denoising autoencoders which are utilized for denoising image data.

These models take a noisy image as input and extract important features using multiple convolutional operations, these latent features are then passed to a set of de-convolutional layers in order to reconstruct the cleaner version of this image with the same height and width. Deconvolution operations can be performed with something like Conv2DTransponse implementations from TensorFlow and Keras.

Note: Latent feature dimensions needn’t be smaller in the case of denoising autoencoders as inputs and outputs are different. Thus there is no risk of learning identity function (An identity function basically copies the input into output without learning important aspects of data).

2. Data Preparation

We will train a denoising autoencoder on MNIST handwritten digits dataset available through Keras. Here is how we can download and load the dataset in our Python notebook-

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline

from tensorflow.keras.datasets import mnist

(trainX, trainy), (testX, testy) = mnist.load_data()

print('Training data shapes: X=%s, y=%s' % (trainX.shape, trainy.shape))
print('Testing data shapes: X=%s, y=%s' % (testX.shape, testy.shape))

for j in range(5):
    i = np.random.randint(0, 10000)
    plt.subplot(550 + 1 + j)
    plt.imshow(trainX[i], cmap='gray')
    plt.title(trainy[i])
plt.show()

MNIST handwritten digits dataset comes with 60K labeled handwritten digit images in the training partition and 10K images in the test partition. Here is a glimpse of 5 images from the dataset. Each image is of 28 * 28 dimensions.

Creating Noisy Data

As our task is to learn a denoising autoencoder, we will need noisy images (for corresponding clean images) for training the network. We will manually add four different kinds of noise into our training and test images.

Here are the python scripts capable of adding following four different kinds of noise to the input images-

Gaussian Noise
Salt and Pepper Noise
Poisson Noise
Speckle Noise

# Adding Noise to the dataset
def guassian_noise(image):
    r,c= image.shape
    mean = 0
    var = 0.1
    sigma = var**0.5
    gaussian = np.random.normal(mean,sigma,(r,c))
    gaussian = gaussian.reshape(r,c)
    noisy = image + gaussian
    return noisy

def salt_and_pepper_noise(image):
    ratio = 0.9
    amount = 0.1
    noisy = np.copy(image)
    
    salt_count = np.ceil(amount * image.size * ratio)
    coords = [np.random.randint(0, i - 1, int(salt_count)) for i in image.shape]
    noisy[coords] = 1

    pepper_count = np.ceil(amount* image.size * (1. - ratio))
    coords = [np.random.randint(0, i - 1, int(pepper_count)) for i in image.shape]
    noisy[coords] = 0
    return noisy

def poisson_noise(image):
    vals = len(np.unique(image))
    vals = 2 ** np.ceil(np.log2(vals))
    noisy = np.random.poisson(image * vals) / float(vals)
    return noisy

def speckle_noise(image):
    r,c = image.shape
    speckle = np.random.randn(r,c)
    speckle = speckle.reshape(r,c)        
    noisy = image + image * speckle
    return noisy

def add_noise(image):
    p = np.random.random()
    if p <= 0.25:
        #print("Guassian")
        noisy = guassian_noise(image)
    elif p <= 0.5:
        #print("SnP")
        noisy = salt_and_pepper_noise(image)
    elif p <= 0.75:
        #print("Poison")
        noisy = poisson_noise(image)
    else:
        #print("speckle")
        noisy = speckle_noise(image)
    return noisy

Let’s checkout some corrupted samples with randomly chosen noise as per above python functions-

print ("Corrupted Example Samples")
for j in range(9):
    i = np.random.randint(0, 10000)
    plt.subplot(330 + 1 + j)
    noisy = add_noise(trainX[i]/255)
    plt.imshow(noisy, cmap='gray')
plt.show()

We can see that the images have been fairly corrupted with multiple different types of corruption. This kind of corruption is often observed in real-world images. We will focus on removing this noise and obtaining a cleaner version of each image.

Data Preparation

We will prepare training and test datasets separately for clean and noisy versions of images. Noisy images will be fed as input and the reconstructed images would be compared with the cleaner versions of the corresponding images. Images are reshaped to add an extra dimension for channels (as per the Convolutional layers input format defined in Keras and TensorFlow).

train_clean = [image/255 for image in trainX]
test_clean = [image/255 for image in testX]

train_noisy = [add_noise(image/255) for image in trainX]
test_noisy = [add_noise(image/255) for image in testX]

train_clean = np.reshape(train_clean, (60000, 28, 28, 1))
test_clean = np.reshape(test_clean, (10000, 28, 28, 1))

train_noisy = np.reshape(train_noisy, (60000, 28, 28, 1))
test_noisy = np.reshape(test_noisy, (10000, 28, 28, 1))

print (train_clean.shape, train_noisy.shape, test_clean.shape, test_noisy.shape)

(60000, 28, 28, 1) (60000, 28, 28, 1) (10000, 28, 28, 1) (10000, 28, 28, 1)

3. Training Convolutional Denoising Autoencoder

In this section, we will define the architecture and train it on the prepared dataset.

Architecture

The model starts with an input layer that takes a batch of images as input with each image having dimensions=(28 * 28 * 1). The first few layers(encoder part) are composed of Convolutional layers followed by MaxPooling layers. The encoder part of the model downsamples the image across width and height and extracts features.

These extracted features from the encoder part are passed into the decoder part as input where the decoder reconstructs the image of the original dimensions. The decoder part is consists of multiple De-convolution layers, each followed by an upsampling layer.

Here is the python implementation of the model-

import tensorflow

input_data = tensorflow.keras.layers.Input(shape=(28, 28, 1))

#Encoder part
encoder = tensorflow.keras.layers.Conv2D(64, (5,5), activation='relu')(input_data)
encoder = tensorflow.keras.layers.MaxPooling2D((2,2))(encoder)

encoder = tensorflow.keras.layers.Conv2D(128, (3,3), activation='relu')(encoder)
encoder = tensorflow.keras.layers.MaxPooling2D((2,2))(encoder)

encoder = tensorflow.keras.layers.Conv2D(256, (3,3), activation='relu')(encoder)
encoder = tensorflow.keras.layers.MaxPooling2D((2,2))(encoder)

#Decoder Part
decoder = tensorflow.keras.layers.Conv2DTranspose(256, (3,3), activation='relu')(encoder)

decoder = tensorflow.keras.layers.Conv2DTranspose(128, (3,3), activation='relu')(decoder)
decoder = tensorflow.keras.layers.UpSampling2D((2,2))(decoder)

decoder = tensorflow.keras.layers.Conv2DTranspose(64, (3,3), activation='relu')(decoder)
decoder = tensorflow.keras.layers.UpSampling2D((2,2))(decoder)

decoded = tensorflow.keras.layers.Conv2DTranspose(1, (5,5), activation='relu')(decoder)

Loss

The final task is to compare the reconstructed image with the cleaner version of the input image. As these images are represented with pixel values, we can use mean squared error(MSE) to quantify the goodness of the autoencoder. We will use the Adam optimizer that will tune the model weights such that MSE is reduced.

autoencoder = tensorflow.keras.models.Model(inputs=input_data, outputs=decoded)
autoencoder.compile(loss='mse', optimizer='adam')
autoencoder.summary()

Here is the model summary-

Model: "functional_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0         
_________________________________________________________________
conv2d (Conv2D)              (None, 24, 24, 64)        1664      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 12, 12, 64)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 10, 10, 128)       73856     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 5, 5, 128)         0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 3, 3, 256)         295168    
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 1, 1, 256)         0         
_________________________________________________________________
conv2d_transpose (Conv2DTran (None, 3, 3, 256)         590080    
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 5, 5, 128)         295040    
_________________________________________________________________
up_sampling2d (UpSampling2D) (None, 10, 10, 128)       0         
_________________________________________________________________
conv2d_transpose_2 (Conv2DTr (None, 12, 12, 64)        73792     
_________________________________________________________________
up_sampling2d_1 (UpSampling2 (None, 24, 24, 64)        0         
_________________________________________________________________
conv2d_transpose_3 (Conv2DTr (None, 28, 28, 1)         1601      
=================================================================
Total params: 1,331,201
Trainable params: 1,331,201
Non-trainable params: 0

Training

We will train the model on training data where noisy images are passed as input and clean images are passed as the output of the model. The test dataset is kept as validation data in order to check the MSE on test batches. The model is trained for 30 epochs with a batch of 64 images and validated on the complete test dataset. After 30 epochs of training, the validation loss comes down to 0.0069.

autoencoder.fit(train_noisy, train_clean, epochs=30, batch_size=64, validation_data=(test_noisy, test_clean))

Epoch 1/30
938/938 [==============================] - 84s 90ms/step - loss: 0.0244 - val_loss: 0.0135
Epoch 2/30
938/938 [==============================] - 75s 80ms/step - loss: 0.0119 - val_loss: 0.0107
Epoch 3/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0100 - val_loss: 0.0095
Epoch 4/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0090 - val_loss: 0.0088
Epoch 5/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0084 - val_loss: 0.0087
Epoch 6/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0080 - val_loss: 0.0083
Epoch 7/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0076 - val_loss: 0.0078
Epoch 8/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0073 - val_loss: 0.0078
Epoch 9/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0071 - val_loss: 0.0079
Epoch 10/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0069 - val_loss: 0.0076
Epoch 11/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0068 - val_loss: 0.0073
Epoch 12/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0066 - val_loss: 0.0074
Epoch 13/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0065 - val_loss: 0.0074
Epoch 14/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0064 - val_loss: 0.0074
Epoch 15/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0063 - val_loss: 0.0070
Epoch 16/30
938/938 [==============================] - 79s 85ms/step - loss: 0.0062 - val_loss: 0.0072
Epoch 17/30
938/938 [==============================] - 79s 85ms/step - loss: 0.0061 - val_loss: 0.0071
Epoch 18/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0060 - val_loss: 0.0071
Epoch 19/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0059 - val_loss: 0.0071
Epoch 20/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0059 - val_loss: 0.0069
Epoch 21/30
938/938 [==============================] - 75s 80ms/step - loss: 0.0058 - val_loss: 0.0070
Epoch 22/30
938/938 [==============================] - 76s 81ms/step - loss: 0.0057 - val_loss: 0.0070
Epoch 23/30
938/938 [==============================] - 77s 83ms/step - loss: 0.0057 - val_loss: 0.0069
Epoch 24/30
938/938 [==============================] - 84s 90ms/step - loss: 0.0056 - val_loss: 0.0071
Epoch 25/30
938/938 [==============================] - 78s 83ms/step - loss: 0.0056 - val_loss: 0.0069
Epoch 26/30
938/938 [==============================] - 76s 81ms/step - loss: 0.0056 - val_loss: 0.0072
Epoch 27/30
938/938 [==============================] - 75s 80ms/step - loss: 0.0055 - val_loss: 0.0068
Epoch 28/30
938/938 [==============================] - 76s 81ms/step - loss: 0.0055 - val_loss: 0.0070
Epoch 29/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0054 - val_loss: 0.0071
Epoch 30/30
938/938 [==============================] - 80s 86ms/step - loss: 0.0054 - val_loss: 0.0069

4. Results

In the last section, we have trained our denoising autoencoder reasonably. In this section, we will check how it performs. We will pass a set of noisy images from the test dataset for prediction and plot the results (model output) for them.

Here is the python code for inference on test dataset-

offset=92
# print ("Real Test Images")
# # Real Images
# for i in range(9):
#     plt.subplot(330 + 1 + i)
#     plt.imshow(test_clean[i+offset,:,:, -1], cmap='gray')
# plt.show()

print("Noisy test images")
for i in range(9):
    plt.subplot(330 + 1 + i)
    plt.imshow(test_noisy[i+offset,:,:, -1], cmap='gray')
plt.show()

# Reconstructed Images
print ("Cleaned Version(Denoising Autoencoder):) ")
for i in range(9):
    plt.subplot(330 + 1 + i)
    output = autoencoder.predict(np.array([test_noisy[i+offset]]))
    op_image = np.reshape(output[0]*255, (28, 28))
    plt.imshow(op_image, cmap='gray')
plt.show()

The above output shows that our Convolutional Denoising autoencoder is doing a good job at learning important features and ignoring the noise from the corrupted images. The input images were fairly corrupted and the model was able to reconstruct the cleaner version of them.

5. Summary

This tutorial gives a background of Convolution based denoising autoencoders adopted for noise reduction in image datasets. With a little background on how it works, it explains how to prepare the training dataset for it. It further explains how to write a simple convolution-based denoising autoencoder in Keras and TensorFlow.

The study is finally concluded with model training and displaying results. The results were quite good for our toy dataset. This kind of image denoising autoencoder can be utilized in multiple applications as a denoising utility to boost the overall performance. Some of the relevant application areas are listed below-

Optical character recognition tasks (OCR)
Handwritten text recognition engine(HTR)
Traffic sign classification
Photo editing tools and many more similar areas

Github Link: https://github.com/kartikgill/Autoencoders

With this i would like to end this article here.

Thanks for reading, I hope it was helpful! Do let me know your thoughts and suggestions by commenting below. See you in the next article.

Related Papers

Denoising AutoEncoders can reduce noise in images was originally published in Game of Bits on Medium, where people are continuing the conversation by highlighting and responding to this story.

What are AutoEncoders in Deep Learning

Kartik Chaudhary — Sat, 18 Nov 2023 13:06:01 GMT

Develop AutoEncoders with Keras and TensorFlow

We all are well aware of the Supervised Machine Learning algorithms where ML algorithm tries to understand the relationship between input features and labels from the training data and is expected to automatically generate a similar kind of relationship between test data features and set of possible output labels. Although we solve lots of problems with supervised learning algorithms, not all problems are supervised in nature. There are problems where we just need the algorithm to learn representation (encoding) from the dataset without requiring labels. These algorithms are known as unsupervised learning ML algorithms. Autoencoders also come under unsupervised learning techniques. In this tutorial, we will learn about Autoencoders in Keras and Deep Learning.

Clustering algorithms (for example- KNN, KMeans, DBSCAN, etc.) are also examples of unsupervised learning techniques as they do not require labeled data to learn relationships/representations in data. Autoencoders are deep learning-based(Neural Networks) algorithms that are widely used to solve many complex tasks.

Here is a list of some of the popular tasks that are solved efficiently using autoencoder based deep learning models-

In this article, we will learn about how to write autoencoders in Keras and deep learning. The rest of the article is divided into the following sub-sections-

Autoencoders in Keras and Deep Learning
Python Implementation
Types of Autoencoders
Summary
References

1. AutoEncoders in Keras and Deep Learning

I hope everyone is aware of sponge balls that are extensively used as stress balls. Stress balls (or hand exercise balls) are squeezed in hand and manipulated by fingers to relieve muscle tension and stress as prescribed in physical therapy.

Sponge balls are known for their elastic property which brings them into their original shape when squeezed forcefully. Due to this property, you can actually keep these balls in smaller boxes(in size) with force and they will acquire their original shape back (approximately) once the box is opened.

The idea behind autoencoders is very similar. An autoencoder will take a data point, squeeze to fit it in a smaller box(smaller data shape), and will be able to convert it back(to its original form) when required. As our data is not elastic like the stress balls, It will not get back to its original form perfectly which means that we will be losing some information in the process. An autoencoder is made up of two parts-Encoder and Decoder.

The Encoder part of an autoencoder is a neural network (non-recurrent generally) that takes the input data point and squeezes it into a lower dimension state (h: as shown in the below figure). This code(h) is expected to learn important information from the input data point. The Decoder part of the autoencoder is again a neural network that is supposed to take this hidden-code representation(h) of the data as input and reconstruct the original data point as output. Thus the inputs and outputs of an autoencoder are the same, and the most important thing is the hidden state(compressed representation of data) that learns important features from the data.

Autoencoders were primarily used for dimensionality reduction and feature learning in the past decades. But recently this technique is quite popular and is widely used in developing generative models (like-Generative Adversarial Networks (GANs)).

Dimensionality reduction is the task of representing high dimensional data points in lower dimensions without losing much information. In this manner, autoencoders are significantly related to the Principal Component Analysis(PCA) when used with linear activations. While the non-linearity(non-linear activations) makes the autoencoders much superior and capable of learning more complicated relationships in the data so that reconstruction is performed without losing any significant information.

The next section will talk about how to implement Autoencoders in Keras and Deep Learning-

2. Python Implementation

In this exercise, we will train a simple autoencoder and examine how well it reconstructs the data. We will further review, how well the reduced dimensions distinguish the output labels. We will use MNIST handwritten digits dataset for our experiments.

Data

The following python code will load the MNIST handwritten digits dataset and display some sample images from the dataset. This dataset has 60K training and 10K test images of handwritten digits with dimensions=28*28. It also has corresponding labels for each digit that we won’t need for our experiment. We will train an autoencoder that takes a digit image as input and reduces it to just 2-dimensions and reconstructs it.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
%matplotlib inline

from tensorflow.keras.datasets import mnist

(trainX, trainy), (testX, testy) = mnist.load_data()

print('Training data shapes: X=%s, y=%s' % (trainX.shape, trainy.shape))
print('Testing data shapes: X=%s, y=%s' % (testX.shape, testy.shape))

for j in range(5):
    i = np.random.randint(0, 10000)
    plt.subplot(550 + 1 + j)
    plt.imshow(trainX[i], cmap='gray')
    plt.title(trainy[i])
plt.show()

These images are 2-D NumPy arrays filled with pixel intensities. We will convert each image into a single dimensional array as we want to pass it into a Multi-layer perceptron model. (Ideally, we should have used Convolutional Neural Network-based autoencoder which is ideal for images).

# normalizing pixel intensities
trainX = trainX/255
testX = testX/255
#reshaping data into single dimension
train_data = np.reshape(trainX, (60000, 28*28))
test_data = np.reshape(testX, (10000, 28*28))
print (train_data.shape, test_data.shape)

(60000, 784) (10000, 784)

Encoder

Let’s define the encoder part of the model. It takes an array of size 784 as input and passes it through a multi-layer dense network. The final layer of the encoder has only two neurons, this layer is expected to represent each given image with these two float numbers.

import tensorflow

input_data = tensorflow.keras.layers.Input(shape=(784))

encoder = tensorflow.keras.layers.Dense(100)(input_data)
encoder = tensorflow.keras.layers.Activation('relu')(encoder)

encoder = tensorflow.keras.layers.Dense(50)(encoder)
encoder = tensorflow.keras.layers.Activation('relu')(encoder)

encoder = tensorflow.keras.layers.Dense(25)(encoder)
encoder = tensorflow.keras.layers.Activation('relu')(encoder)

encoded = tensorflow.keras.layers.Dense(2)(encoder)

Decoder

Most of the time, the decoder part of the autoencoder network is a mirror image of the encoder model. It will take those reduced dimensions as input and reconstruct the original image by increasing the dimensions again to the original ones. Finally, we will get an array of dimensions 784 as output. This output array is expected to be similar to the input array.

decoder = tensorflow.keras.layers.Dense(25)(encoded)
decoder = tensorflow.keras.layers.Activation('relu')(decoder)

decoder = tensorflow.keras.layers.Dense(50)(decoder)
decoder = tensorflow.keras.layers.Activation('relu')(decoder)

decoder = tensorflow.keras.layers.Dense(100)(decoder)
decoder = tensorflow.keras.layers.Activation('relu')(decoder)

decoded = tensorflow.keras.layers.Dense(784)(decoder)

Loss

We need a loss function that will tell how similar our reconstructed image is with respect to the original input image. As images are just pixel values, Mean Squared Error (‘mse’) would be a good choice to measure how close each pixel is to its corresponding pixel in the real and predicted images.

autoencoder = tensorflow.keras.models.Model(inputs=input_data, outputs=decoded)
autoencoder.compile(loss='mse', optimizer='adam')
autoencoder.summary()

Model: "functional_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         [(None, 784)]             0         
_________________________________________________________________
dense_11 (Dense)             (None, 100)               78500     
_________________________________________________________________
activation_9 (Activation)    (None, 100)               0         
_________________________________________________________________
dense_12 (Dense)             (None, 50)                5050      
_________________________________________________________________
activation_10 (Activation)   (None, 50)                0         
_________________________________________________________________
dense_13 (Dense)             (None, 25)                1275      
_________________________________________________________________
activation_11 (Activation)   (None, 25)                0         
_________________________________________________________________
dense_14 (Dense)             (None, 2)                 52        
_________________________________________________________________
dense_15 (Dense)             (None, 25)                75        
_________________________________________________________________
activation_12 (Activation)   (None, 25)                0         
_________________________________________________________________
dense_16 (Dense)             (None, 50)                1300      
_________________________________________________________________
activation_13 (Activation)   (None, 50)                0         
_________________________________________________________________
dense_17 (Dense)             (None, 100)               5100      
_________________________________________________________________
activation_14 (Activation)   (None, 100)               0         
_________________________________________________________________
dense_18 (Dense)             (None, 784)               79184     
=================================================================
Total params: 170,536
Trainable params: 170,536
Non-trainable params: 0

Training

Input(x) and output(y) are same for autoencoders. We will train our model on training digit images for 30 epochs with a batch size of 64 images.

autoencoder.fit(train_data, train_data, epochs=30, batch_size=64, validation_data=(test_data, test_data))

Epoch 1/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0529 - val_loss: 0.0470
Epoch 2/30
938/938 [==============================] - 1s 1ms/step - loss: 0.0459 - val_loss: 0.0450
Epoch 3/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0441 - val_loss: 0.0434
Epoch 4/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0429 - val_loss: 0.0421
Epoch 5/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0420 - val_loss: 0.0417
Epoch 6/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0413 - val_loss: 0.0408
Epoch 7/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0405 - val_loss: 0.0400
Epoch 8/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0401 - val_loss: 0.0397
Epoch 9/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0396 - val_loss: 0.0394
Epoch 10/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0393 - val_loss: 0.0389
Epoch 11/30
938/938 [==============================] - 1s 2ms/step - loss: 0.0389 - val_loss: 0.0388
Epoch 12/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0389 - val_loss: 0.0390
Epoch 13/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0386 - val_loss: 0.0382
Epoch 14/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0385 - val_loss: 0.0383
Epoch 15/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0382 - val_loss: 0.0382
Epoch 16/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0380 - val_loss: 0.0380
Epoch 17/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0378 - val_loss: 0.0377
Epoch 18/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0377 - val_loss: 0.0377
Epoch 19/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0378 - val_loss: 0.0376
Epoch 20/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0375 - val_loss: 0.0378
Epoch 21/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0376 - val_loss: 0.0379
Epoch 22/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0376 - val_loss: 0.0375
Epoch 23/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0373 - val_loss: 0.0372
Epoch 24/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0376 - val_loss: 0.0378
Epoch 25/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0375 - val_loss: 0.0378
Epoch 26/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0372 - val_loss: 0.0381
Epoch 27/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0372 - val_loss: 0.0369
Epoch 28/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0370 - val_loss: 0.0371
Epoch 29/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0370 - val_loss: 0.0369
Epoch 30/30
938/938 [==============================] - 2s 2ms/step - loss: 0.0368 - val_loss: 0.0371

Prediction

Time to check how well it performs on the test datasets. We will plot the original as well as re-constructed images to check how good our model is performing-

# Real Images
for i in range(5):
    plt.subplot(550 + 1 + i)
    plt.imshow(testX[i], cmap='gray')
plt.show()

# Reconstructed Images
for i in range(5):
    plt.subplot(550 + 1 + i)
    output = autoencoder.predict(np.array([test_data[i]]))
    op_image = np.reshape(output[0]*255, (28, 28))
    plt.imshow(op_image, cmap='gray')
plt.show()

The first row has 5 real images, and the second row has corresponding reconstructed images. We can see that the output images are a little blurry still they are quite readable. In the last image, our model mistakes a 4 for a 9 as both digits are very similar in shape. With more training and deeper network, model should be able to learn the difference.

Reduced Dimensions

We saw that this small network was capable of re-constructing the original image using just two numbers(h: layer) from the encoding layer(bottleneck layer). Now these reduced dimensions can be used in other applications as features for these images. To verify the fact, let’s plot these embeddings(features) for all the test dataset digits.

dr_model = tensorflow.keras.models.Model(inputs=autoencoder.get_layer('input_3').input, outputs=autoencoder.get_layer('dense_14').output)
dr_model.summary()

x = []
y = []
z = []
for i in range(10000):
    z.append(testy[i])
    op = dr_model.predict(np.array([test_data[i]]))
    x.append(op[0][0])
    y.append(op[0][1])

df = pd.DataFrame()
df['x'] = x
df['y'] = y
df['z'] = ["digit-"+str(k) for k in z]

plt.figure(figsize=(8, 6))
sns.scatterplot(x='x', y='y', hue='z', data=df)
plt.show()

Each image from the test dataset is represented with 2-dimensions and is plotted as a point in the scatterplot below where color represents the label of the image. We can notice that the same digits are closer in the latent space. Even this simple autoencoder is able to separate out digits in different cluster-

As we know that the convolutional neural networks are the best choices for image understanding/representations, results would have been even better with CNN-based autoencoder.

Here is my article where I have trained a CNN-based autoencoder to predict future frames in a video-

Python Predicts PUBG Mobile

3. Types of Autoencoders

There are various techniques(regularization techniques) to make autoencoders robust such that they learn the important features instead of just remembering and copying the input data into output.

Here are a few regularized variants of autoencoders that work really well in practice-

Sparse autoencoders (SAE)

Sparse autoencoders are typically used when we need to learn features for other tasks, such as classification. These autoencoders involve a sparsity penalty on the embedding layer(h-layer) in addition to the reconstruction error.

This penalty can be thought of as a regularization parameter to the simple autoencoder where the task was to simply copy the input into an output. This sparsity penalty makes the model learn better representative features for the dataset instead of just copying input into the output.

Denoising autoencoders (DAE)

Denoising autoencoders are trained by changing the reconstruction error term of the cost function(or loss). Traditionally the input data points and output data points are the same for an autoencoder but in the case of denoising autoencoders, the input data points are noisy in nature while the reconstructed data points are compared with the clean data point(without noise). In this manner, a DAE would learn to ignore(or remove) noise from the noisy dataset.

Denoising autoencoders have the ability to learn useful features from the input data and discard the useless information(like-noise) at the time of reconstruction. This makes them learn better representations of the data and they are trained to remove noise from noisy datasets.

Contractive autoencoders (CAE)

Denoising autoencoders typically work well when there is small and limited noise in the dataset, Contractive autoencoders on the other hand are robust to significant perturbations on the input data. Contractive autoencoders apply a contractive penalty on encoder output that encourages the derivatives of the encoder to be as small as possible. This penalty is basically the sum of squared elements of partial derivates from the encoder function.

Contractive autoencoders are trained to learn the distributions of input instead of copying it, thus they are robust to the slight variations in the inputs.

Variational autoencoders (VAE)

Variational autoencoders are different from the traditional autoencoders discussed above(like-denoising, sparse, contractive). These are generative models like Generative Adversarial Networks(GANs).

Traditional autoencoders basically learn latent representations of inputs using reconstruction error. The distribution of these latent representations is usually uneven(input-related). Variational autoencoders use reconstruction loss along with the KL-divergence loss. Here KL-divergence loss encourages the model to learn broader distributions of latent features.

These latent features from the Variational autoencoder are assumed to follow the Gaussian distribution when trained on sufficient training data. We can sample features from the latent distribution and form a generative model capable of generating new data without any input.

4. Summary

In this tutorial, we have gained a basic understanding of the autoencoders in Keras and deep learning. We have implemented a simple autoencoder network to learn representations on the MNIST handwritten digit dataset. We have also seen that the model was able to learn effective latent features for different digits.

We later talk about different variations of the autoencoders that make this algorithm robust by using different regularization techniques. I will explain these techniques in detail in my future tutorials.

You can find the python notebook here:

Github Repo: https://github.com/kartikgill/Autoencoders

Thanks for reading. Hope this article was helpful. Kindly give your feedback/suggestions by commenting below. See you in the next article.

Learn More (related research papers)

What are AutoEncoders in Deep Learning was originally published in Game of Bits on Medium, where people are continuing the conversation by highlighting and responding to this story.

Understanding Optimizers for training Deep Learning Models

Kartik Chaudhary — Wed, 15 Nov 2023 15:06:06 GMT

Learn about popular SGD variants knows as optimizers

Overview

Training a Deep Learning model (or any machine learning model in fact) is all about bringing the model predictions (model output) close to the real output(Ground truth) for a given set of input-output pairs. Once the model’s results are close to the real results our job is done.

To understand how close model predictions are with respect to the real outputs, we need a mathematical function(measure) known as a loss function. This loss function tells you how good your model is doing at a given step of training.

Ganga River in India | Image by Author

Though our loss function helps us understand the quality of our model, it doesn’t solve the complete problem. We actually need a different algorithm that will update the coefficients(or weights) of our model. This optimization algorithm will make sure that the loss value(on training data) decreases at each training step and our model learns from the input-output pairs of the training data.

In this article, we will discuss some common optimization techniques (Optimizers) used in training neural networks (Deep Learning models).

This article will cover the following topics:

Gradient Descent Algorithm
Stochastic Gradient Descent
Momentum
Adagrad
RmsProp
Adadelta
Adam
Summary

Gradient Descent Algorithm

Gradient Descent is a popular algorithm used to find the local minima of any given differentiable function. If the given function is convex, this local minima becomes the global minima.

While training any machine learning (or Deep Learning) model our objective is to achieve minimum loss value. Thus our optimization algorithm will try to find the local/global minima for our loss function with respect to the given training data.

The basic intuition behind the working of gradient descent algorithms is-At each point in time, It tries to find the best direction to move and takes a small step such that the overall loss value decreases. To find the best direction at a given point, it calculates the gradient of the loss function at that point.

Illustration of gradient descent learning process

The gradient of a differentiable function-f at a given point p is equal to the value of the partial differentiation of function-f at the point-p.

To get a lower value of the loss function at each succeeding step, we must move towards the direction of the negative gradient from the starting point-p.

Here is how gradient descent updates the coefficients(weights) of the model with respect to the calculated gradient value over loss function at a given step.

# calculate the current gradient
gradient = calculate_gradient_(args)
# update weights towards negative gradient
weight_new = weight + learning_rate * ((-1)*(gradient))
# resulting weight value
weight_new = weight - learning_rate * gradient

Note: calculate_gradient_(args) is a function that calcualtes current gradient.

learning_rate here decides-how fast(size of jump) you want to move towards the local best direction hinted the by the gradient!

Convergence to the global minimum is guaranteed when the loss function is convex. As for a convex function-all local minima’s are also the global minima.

Stochastic Gradient Descent

Although gradient descent guarantees convergence, there is a problem with this optimization method. As it calculates the gradient over the full dataset at each step, it is not always feasible due to the high dimensionality of most real-life datasets(also it’s not always feasible to do such computations over full data due to the hardware constraints).

To overcome this issue, stochastic gradient descent(also known as ‘SGD’) comes into the picture. The stochastic gradient descent algorithm also works in a similar way but instead of calculating the gradient over the full dataset, it calculates the gradient value over each example of the training dataset and updates the parameters(weights) accordingly.

#SGD signature in tf.keras
tf.keras.optimizers.SGD(
    learning_rate=0.01, momentum=0.0, nesterov=False, name='SGD', **kwargs
)

This makes is computationally awesome and extremely fast. But this great power comes at a cost, that is-lower convergence rate (As noisy examples might mislead the convergence direction).

Mini batch Stochastic Gradient Descent

To make things better-the mini-batch stochastic gradient descent algorithm calculates the gradient value over a mini-batch (a subset of training data) instead of every single example. Now, this modification keeps the algorithm computationally feasible and as the gradient value is averaged over a few examples it also provides smooth convergence.

SGD (as an optimization algorithm) has become an important part of machine learning. With small learning rates and some other modifications, mini-batch SGD almost surely converges to the global minimum(or the local minimum in case of pseudo-convex loss functions).

SGD is a popular algorithm used in optimizing many machine learning algorithms like-Linear Regression, Logistic Regression, Support Vector machines (SVM). And it’s the backbone of Artificial Neural Networks(Deep Learning models) when combined with backpropagation.

Momentum

Consider a situation where the loss function surface is not ideal(strictly convex) and you are at a point where local gradient points towards the wrong direction(based on local derivatives). In this scenario, our model might suffer in finding the correct direction towards the global minima.

Another reason for encountering the same problem could be noisy mini-batch, where the training batch is noisy and pointing the model towards wrong updates to the parameters, thus resulting in slow convergence (or oscillations in convergence).

Momentum is a term taken from physics and applied to SGD in order to encounter this issue. The momentum method of SGD optimizer, remembers the last few gradient values from the past training batches and does not just rely on the recently calculated gradient while making updates to the parameters. Instead, it takes inputs from the past gradient values to make sure that we are always moving towards the correct direction of convergence.

In this way, the momentum method of the SGD optimizer achieves fast convergence without oscillations. Here is how it updates the parameters (weights) of the model-

#momentum and velocity terms are from physics used in the similar context here
gradient = calculate_gradient_(args)
velocity_new = momentum * velocity - learning_rate * gradient
weight_new = weight + velocity_new
weight_new = weight + momentum * velocity - learning_rate * gradient

Adagrad Optimizer

SGD optimizer works with a global learning-rate (A user-defined initial learning rate). It means that it updates each parameter with the same global learning-rate at each step and It may cause the convergence to be quite slow.

Adagrad, also known as Adaptive Gradient, is a modification of the SGD algorithm. On top of the actual SGD algorithm, it provides parameter specific learning rates (while the SGD method works with a global learning_rate that is applied to all the parameters of the model throughout the training process).

# signature tf.keras
tf.keras.optimizers.Adagrad(
    learning_rate=0.001, initial_accumulator_value=0.1, epsilon=1e-07,
    name='Adagrad', **kwargs
)

Instead of using a global learning-rate for each weight, the Adagrad algorithm keeps a separate learning rate of each parameter. The learning-rate of a parameter is decided based on the frequency of updates it gets. The parameters that are updated very frequently are updated with smaller learning rates while the parameters that are updated less-frequently are updated with a slightly larger learning rate.

In mathematical terms- The Adagrad algorithm keeps track of the sum of squares of the gradients for each parameter and divides the learning-rate with it(square-root value of the squared sum of the gradients) before each new update to that parameter. In this way, parameters updated with large values in the past are updated slowly in the next future steps.

Squared Sum: because some gradients are +ve while some are -ve.

# Here is the algorithm
gradient = calculate_gradient_(args)
#keep track of the squared sum
sq_sum_grad = sq_sum_grad + gradient * gradient
#penalize learning rate with square-root of the squared sum of gradients
weight_new = weight - (learning_rate/sq_root(sq_sum_grad)) * gradient

This algorithm is called ‘adaptive gradient’ because it adapts the learning-rates for each parameter based on its past updates.

RMSProp Optimizer

RMSProp (Root Mean Square Propagation) algorithm is again based on the Stochastic Gradient algorithm (SGD). RMSProp is very similar to the Adagrad algorithm as it also works with adaptive learning-rates for the parameters.

The main difference is that instead of calculating the plain squared sum of gradients(like in Adagrad method), it calculates a discounted (or weighted) average of the squared gradients and uses that to penalize the learning-rates for each parameter.

# signature of RMSProp in tf.keras
tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False,    name='RMSprop', **kwargs)

here rho is that discounting factor, here is how this algorithm works-

# Here is the algorithm
gradient = calculate_gradient_(args)
dis_sq_grad = rho * dis_sq_grad + (1 - rho) * gradient * gradient
weight_new = weight - (learning_rate/sq_root(dis_sq_grad)) * gradient

Discounted average is a better option here because the plain squared sum keeps increasing with time and keeps penalizing the learning-rate more and more. Thus model learning becomes quite slow with time.

RMSProp is a very popular optimizer and provides a really fast convergence in practice.

Adadelta Optimizer

Adadelta (Adaptive Delta Gradient) is again based on stochastic gradient descent algorithms and is an optimized version of the adaptive gradient(Adagrad) algorithm.

Instead of adapting learning rates based on all the past gradients(like Adagrad does), it uses a moving window of only a few past gradients in order to penalize the learning-rates for each parameter. This saves the algorithm from continuous decay of learning-rates throughout the training. With this strategy, the model keeps learning even after many updates/iterations.

Adadelta performs well even when your model takes very long to train. Another advantage of using the ‘Adadelta’ algorithm is that you don’t even have to provide the initial learning rate (though you can provide your own) as the default one gets adapted soon.

Adam Optimizer

Similarly, Adam (Adaptive moments of gradient) is again a stochastic gradient descent based optimization algorithm. Just like Adagrad, RMSProp, and Adadelta, it also works with adaptive learning-rates of the parameters(weights). It combines the benefits of Adagrad and RMSProp both.

tf.keras.optimizers.Adam(
    learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False,
    name='Adam', **kwargs
)

Instead of considering just 1st-order moment of past gradient values(like-RMSProp), Adam utilizes second-order moments of the past gradients also in order to update the weights(parameters) of the model.

It maintains two different parameters beta_1 and beta_2 as decay rates for both the moments. You don’t have to set these values explicitly as the default ones often work best.

Adam has been proved to be very efficient and is widely accepted across different domains/problems/architectures.

Adam is “computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters”

According to Kingma et al., 2014

Summary

In this article, we went through multiple optimization algorithms used for training artificial neural network-based models.

We discussed how each of the algorithms provides additional benefits over vanilla stochastic gradient descent algorithm.

Though all these optimizers work well (they differ mostly in training time, overall average results do not differ much). But if you want to choose one(one all-time best)-Adam is considered the most effective optimizer among all others.

Here are few good things about Adam-

Easy to understand and implement
Lesser memory requirements
Fewer computation requirements
Works well even for big/large-data problems.
Hyperparameters don’t require much tuning as defaults work well.

With this I would like to end this post here. Thanks for reading!

Do let me know your thoughts/feedback by commenting below. See you in the next article.

References

Tensorflow Documentation (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)
Wikipedia: Gradient Descent
Wikipedia: Stochastic Gradient Descent

Understanding Optimizers for training Deep Learning Models was originally published in Game of Bits on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stories by Kartik Chaudhary on Medium

Turbocharge Large Language Model Training with Parallelization

Parallelization techniques for efficient distributed training of large deep learning models.

Introduction

What is Parallelism?

Data Parallelism

Model Parallelism

Don’t Overlook the Challenge of “Combining Results”

Speeding Up LLM Training with Parallelization

Using Data Parallelism (DP)

Using Pipeline Parallelism

Applying Tensor Parallelism

Using Zero Redundancy Optimizer (ZeRO)

BTW — What is DeepSpeed?

Which parallelism is suitable for you?

Single Machine — Single GPU

Single Machine — Multiple GPUs

Multiple Machines — Multiple GPUs

References

Understanding Key Evaluation Techniques for LLMs

1. History of Large Language Models

1.1 Language Models Before 2017

1.2 Language Models After 2017

1.3 Latest LLMs are Mysterious

2. Key Evaluation Techniques for LLMs

3. Human Evaluation

3.1 How does human evaluation work?

4. Programmatic Evaluation

5. AutoRater Evaluation Tool

6. Key Metrics for LLM Evaluation

6.1 Accuracy, AUC, F1, Precision, Recall

6.2 Toxicity

3. BLEU

4. BLEURT

5. SQuAD

6. ROUGE

Conclusion

Read Next>>>

Unlocking Video Insights: Extract Thumbnails with Captions using Gemini 1.5 Pro

A step-by-step tutorial for extracting thumbnails and meaningful captions from videos using the multimodal capabilities of Gemini

Step 1: Installing Libraries

Step 2: Import useful libraries

Step 3: Setting up the Google Cloud Project

Step 4: Download a Sample Video

Step 5: Load Gemini 1.5 Pro model

Step 6: Define a Function to Call Gemini API

Step 7: Parse the model response and display results

Experiment 1: Basic Prompting

Experiment 2: Advanced Prompting

Conclusion

Adversarial Examples and Defence Mechanisms against them

Understanding Adversarial Examples and Defence Mechanisms

What is Adversarial Machine Learning?

Targeted and Un-Targeted Adversarial Attacks

White Box and Black Box Adversarial Attacks

What are common methods of Adversarial Attacks?

Fast Gradient Sign Method (FGSM)

Limited-Memory BFGS (L-BFGS)

Jacobian-based Saliency Map Attack (JSMA)

Deep Fool Attack

Carlini & Wagner Attack

Generative Adversarial Networks (GANs)

Zeroth-Order Optimisation Attack (ZOO)

How to defend against Adversarial Attacks?

Adversarial Training

Defensive Distillation

Gradient Hiding

Feature Squeezing

Blocking and Transferability

Defense-GAN

MagNet

Why it is hard to defend against Adversarial Attacks?

Conclusion

Read Next>>>

Understanding Failure Modes of GAN Training

Mode Collapse and Convergence Failure of Generative Adversarial Networks

Failure Modes of GAN Training

1. Why GAN training is unstable?

2. GAN failure modes

Understanding Failure Modes of GAN Training