Are you familiar with gradient descent and have practised some form of machine learning at some point in your life? If yes, congrats! You can easily understand the basic building blocks of ChatGPT.

This article is an attempt at simplifying the explanation of the ChatGPT paper and keeping jargon to a minimum.

The methodology behind ChatGPT is exactly the same as one of OpenAI’s previous models- instructGPT: https://arxiv.org/pdf/2203.02155.pdf

The training comprises 3 phases:

1. Supervised Fine Tune (SFT) a pretrained GPT3 model
2. Train a Reward Model (RM)
3. Train a Reinforcement Learning Model (RL)

Let’s look at each step in detail.

1. Supervised Fine Tune (SFT)

The goal of this step is to fine tune a pretrained GPT3 model using human generated responses.

OpenAI hired a bunch of humans to write high quality prompts and responses, threw the prompt and responses at a pretrained GPT3 model in order to finetune it and voilà, the step was completed!

Model

A pretrained GPT3 model is used.

Dataset

OpenAI collected prompts not just from humans, but also from their API. The responses to each of these were written exclusively by humans. The prompts were taken from a variety of task types- brainstorming, classification, Q&A, summarization, chat, extraction, etc. Some example of the prompts:

1. List five ideas for how to regain enthusiasm for my career
2. {java code} What language is the code above written in?
3. Write a short story where a bear goes to the beach, makes friends with a seal, and then returns home.
4. Who built the statue of liberty?

The paper doesn’t mention the responses to the prompts but rest assured, some human wrote a response to each prompt. (In fact the examples above are just a mimicry of the actual prompts; authors didn’t disclose the real data).

Training regime

If you’re familiar with sequence modelling (specifically ‘Next Token Prediction’), feel free to skip this section. If not, read on.

Since this is a supervised learning setup, one has to have access to the input and its corresponding target at the time of training. This is a GPT model- the task is to predict the next token in the sequence given all the past tokens. Let’s take a simple example:

Prompt- How are you doing?

Response- I am good

First step is to concatenate the prompt and the response-

How are you doing? I am good

Assuming one word is one token (which is not the case since GPT typically uses Byte Pair Encoding, but it simplifies our explanation without loss of generality), we have the following tokens-

‘How’, ‘are’, ’you’, ‘doing’, ‘?’, ‘I’, ‘am’, ‘good’

- this gives us 8 tokens (’?’ is an individual token).

Now, at each timestep, the output of the model comprises a probability distribution over the entire vocabulary of the dataset. Eg: if there are a total of 50,000 different tokens the model is allowed to produce, these 50k tokens would make up the vocabulary of the model. Our 8 tokens are 8 entries of this vocabulary. So, at each time step, a probability distribution over 50,000 tokens is produced. These 50k values denote the probability of each of the 50k tokens to be the next predicted token in the sequence. Since here we already know the whole sequence, we know exactly what we expect the probability distribution to be at each time step- a 1.0 for the token we know exists at the timestep and 0.0 for all other tokens in the vocab. In our example, for each timestep, 49,999 entries would be 0.0, only one entry would be 1.0 — this would be the expected training distribution.

The training process depicted in the above diagram for the first 4 time steps can be broken down as below:

Given our sequence of 8 tokens- How are you doing? I am good

Let’s look at the first timestep-

At t=0,

expected output distribution- P(‘how’) = 1.0 given a beginning-of-sentence (<bos>) token to start generation, all other tokens P(token)=0.

Since the model is untrained, P(‘how’) would not be 1.0 (=0.85 in the diagram). Similarly, all the 49,999 probabilities would also not be 0.0. Hence, we can calculate a loss between the expected and actual output. We use this loss to update the model.

At t=1,

expected output distribution- P(‘are’ | ‘how’) = 1.0, all other tokens P(token | ‘how’)=0.

We already know the first token in the sequence(= ‘how’) and would now want to get the second token(= ‘are’). Again, an untrained model produces probabilities that are different from expected. So, we incur a loss that we backpropagate and update the model.

The above procedure goes on for all tokens in the sequence and through multiple sequences (thousands).

When the SFT is trained, one can trigger a generation process by adding a prompt and let the model produce one token per timestep (one token can be selected from the the probability distribution produced for that timestep by, for example, selecting the highest prob token in the distribution).

We end up with a Supervised Fine Tuned model (SFT) at the end of this procedure.

SFT model is where the story usually ends for a majority of ML practitioners at the moment. The model trained using usecase specific data is now ready to start generating. However, the model is still likely to produce responses that are not preferable to humans. As an example, let’s say we have a prompt- How are you doing?

3 different runs of the model produce 3 different responses:

1. I am good
2. Doing good. How about you?
3. Apple is a nice fruit.

It is easy to see that 3rd response makes no sense for the given prompt, while it could be argued that 2nd response is marginally better than the first one. So, not all responses are created equal! What if one could train a model that would produce responses that are more aligned with human preference? This is precisely the goal of the next 2 phases.

2. Train a Reward Model (RM)

One could take the responses produced by the SFT model and ask a human to rate those responses on the basis of how close those are to the responses a human would write for that prompt. Once such a rating is available, it could be used to update the model to align the generation with human-preference.

Simulating human preference is the goal of this second step- training a so-called Reward Model. Here the term ‘reward’ refers to the rating mentioned above- a higher reward means that the human is more likely to prefer that response; a lower reward indicates the opposite.

Let’s now look at the model used in this phase and the corresponding data used to train it.

Model

1. The SFT trained in phase 1 is copied.
2. The last linear layer of SFT is cut-off (referred to as ‘unembedding’ layer in the paper).
3. A randomly initialized linear layer is slapped on top of the model that outputs a scalar value.

So, we basically end up with a regressor!

Dataset

1. Take the SFT model from phase 1.

2. Throw a prompt multiple times (4–9 times) at it to generate multiple responses (4–9 responses).

3. Create pairs of responses from a given prompt. C(4,2) to C(9,2) pairs per prompt.

4. Ask a human to look at each pair of responses and rank individual response within each pair. Essentially, for a given pair of responses to a prompt, indicate which of the response in the pair is better.

33k prompts were used in this step. Since 4–9 responses were used for each prompt, the total number of samples generated here were between [33k X C(4, 2)] and [33k X C(9, 2)].

Training regime

1. Take a response pair- this pair has already been ranked as explained before.
2. Concatenate the prompt with each of the responses in the pair. As a result, there would be one winning prompt-response and one losing prompt-response.

3. Pass the winning prompt-response to the reward model and generate a winning reward.

4. Pass the losing prompt-response to the same reward model and generate a losing reward.

5. Take the difference between the two rewards and update the model so as to maximize this difference.

The loss for training the reward model is:

The loss is a difference in log-odds of a high reward and low reward response. Intuitively, the reward model is learning to maximize the difference between a winning and a losing response.

Once the model is trained, one can simply concatenate the prompt with a response and get a scalar reward that indicates the degree to which a human will prefer that response for the given prompt.

So, we now have a way to simulate a human’s preference for a given prompt-response combination.

3. Train a Reinforcement Learning Model (RL)

Until now, we have trained 2 models:

1. An SFT model trained in phase 1 that is capable of generating a response given a prompt- but it still generates responses that are not aligned with human preferences.
2. A reward model that gives out a scalar value but is incapable of actually generating any text.

So, now we have to come up with a model that utilizes the above 2 models to produce another model that is capable of generating text that aligns well with human preferences- this is the goal of the 3rd phase- training a reinforcement learning model.

Aligning generation with human preference boils down to generation of responses that will yield a high reward for a given prompt. So, while training the RL model, the aim of the model is to maximize reward. Fortunately, one can formulate this aim mathematically as an objective maximization problem:

where,

One can take the gradient of this objective function and update the parameters of the model like a regular machine learning objective function, the only difference from the usual case being the use of gradient ascent instead of gradient descent since this is a maximization problem:

With some simple mathematical aerobics, the expression for the above gradient reduces to:

where D refers to all the prompt-response pairs, x denotes token at different timesteps and the gradient-log term has a tractable value.

Let’s look at the model, the dataset and the training regime for this step.

Model

The SFT trained in phase 1 is copied.

That’s it! No modifications, just copy the SFT model.

Dataset

31k prompts- no responses are available; it’s just a collection of 31k prompts.

Training regime

1. Throw a prompt at the model.
2. The model generates a response.
3. Concatenate the prompt-response and pass it to the reward model to get a reward score for the pair.
4. Plug the values for reward and probabilitites in the gradient equation above.
5. Use this gradient to update the model parameters.

That’s it!

BUT, as it turns out, if you use the regime above, the updates are unstable- they’re too large and generic, which makes the model prone to generating gibberish. To drive the update in a more deterministic direction, one can ensure that the updates are done such that the resulting model parameters are not too far away from the SFT trained in phase 1.

This notion of distance is applied through the use of KL divergence (in the paper at least)- which measures a proxy of distance between the SFT and the model under training (RL model). Too large a divergence indicates that the model parameters are too far away from SFT, too low indicate the opposite. KL divergence is essentially a way to compare two distributions. Remember from phase 1 that at each time step, the generative model emits a probability distribution over the whole vocabulary. These are the distributions that can be compared.

More importantly, having access to this divergence value, one can directly use it as a part of the objective function. This is accomplished as follows:

1. For a prompt, response is generated by the RL model.
2. At each time step of generation, a probability distribution over the vocab is generated.
3. The same prompt-response is fed to the SFT model and a probability distribution over the same vocab for each timestep is obtained.
4. Distributions from the above 2 steps are used to calculate KL divergence between SFT and RL model.
5. The divergence value is subtracted from the reward generated for a prompt-response pair.

For a low divergence value, the reward has more impact on the update than for a high divergence. A hyperparameter beta is added to control the effect of the divergence term on the objective.

However, when the authors trained the model with the above cost function, they found that the model performance dropped on some datasets compared to the performance of a pretrained GPT3 on those datasets. So, if the authors hadn’t gone through all this trouble, they would still obtain a better performance on some datasets using a naive pretrained GPT3. A bummer! So, they decided to do something about it- add another term in the cost function to ensure that the RL model isn’t allowed to go very far away from the pretrained model either (this is the model that was the base for finetuning in the first phase to obtain the SFT model). The cost function now becomes:

The intuition behind the term is that the RL model should also maximize the likelihood of observing the sequences seen during the pretraining of GPT3. GPT3 was pretrained using a bunch of pretraining sequences and the objective was to maximize the likelihood of observing these pretraining sequences. If the RL model also maximizes the likelihood of observing the same sequences, one could argue that it is not very different from the pretrained GPT3. A hyperparameter gamma is added to control the effect of the pretraining term on the objective.

In conclusion, the model is optimized to:

1. Maximize reward obtained using the reward model.
2. Minimize divergence from SFT model.
3. Maximize likelihood of observing sequences seen during pretraining.

After updating the RL model for some steps, one could initialize a new reward model from it, ask humans to generate another dataset following the procedure described in Phase 2 and conduct another training session for the reward model. The intuition here is that the updated RL model is now better at capturing human preference than the SFT model and hence, would make for a better initialization point for the reward model.

Is it a guarantee that the model will now generate responses that a human would prefer? No, but it’s merely more likely to produce human-preferred responses. We have just managed to reduce this gap, not close it.

And this concludes our explanation of the instructGPT paper!

Give yourself a cookie for making it till the end!

ChatGPT: A no-jargon explanation was originally published in Axel Springer Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

The first task aligned with my profession as a Machine Learning Engineer. imagededup, the open source python package I built with my team at work, could use a functionality that would allow user to find duplicates in an image collection at scale. I would refer to this task as mlTask henceforth.

The second task was a more mundane one: I had managed to rip my winter jacket in several places and was in urgent need of buying a new one during chilly January Berlin winter. In non-covid times, this task would comprise a hasty visit to 1 or 2 stores of my choice, where I would try on a few jackets and be done with the whole thing while consuming minimal mental bandwidth (minimal patience for shopping). In the lockdown period however, this bliss was snatched away from me, leaving me no choice but to get the darn jacket online. (which is a massive chore for me since I can’t just feel how the product is, but need to put in time to understand what I’m buying and I hate trying on and returning articles in case they don’t work for me). I will refer to this task as jacketTask from here on.

Phase 1: Basic requirements

First, I needed to have a basic idea of what I’m looking for.

mlTask: Try to find one algorithm that would find duplicates in an image collection comprising a million images in a short amount of time with low memory consumption while making minimal sacrifices to the deduplication accuracy.

jacketTask: Get a Jack Wolfskin winter jacket( with a limit on my budget). Why Jack Wolfskin (JW)?- Because I was curious to use one. Having used H&M and Jack & Jones, I now wanted to go a little more premium (premium for me at least).

Phase 2.1: Ask around

Talk to humans you know might know whatever might be worth knowing about the thing you are interested in knowing about. You get the drift ;)

mlTask: Several users of the package(i.e., imagededup) had pointed to interesting algos/python packages that could be used here. This told me that firstly, there are already alternatives out there and secondly, there are alternatives that the community seems to be leaning towards. Had these alternatives not come up with my own research, I would have gone back and specifically looked at them.

jacketTask: A simple message in a whatsapp group comprising friends living in Germany. Got the first suggestion as a JW with a slight exposition of the specifics of the jacket. Good start! And some validation of my decision to buy a JW. Additionally, I looked at my current jacket and noticed what I liked/disliked about it.

Phase 2.2: Preliminary googling/Optimistic googling

As is usually the case, google is the know-it-all we all go to for all our questions concerning life, its meaning and everything beyond. But, the crucial deal at this precarious moment is this: What shall the search term be? I like to be optimistic at this point and try to look for the most complete solution available.

mlTask: The first search term was on the lines of- ’Large scale image deduplication’. Could I be so lucky that somehow someone has tried doing exactly what I’m doing and has been kind enough to condense all that info into a great, working piece of code and a corresponding document? This is common enough a problem, but this search term returned results that turned out to be a little bland or seemed to miss the point.

jacketTask: Looking at the basic requirements, it must surely be easy to find what I was after and hence, a straightforward search term- ’Jack Wolfskin winter jacket’. An immediate autosuggestion led me to append the words ‘for men’ to the search term. This opened up an army of google shopping, amazon, zalando links listing pieces from about a 100 euros to several times my budget. The shopaphobic in me immediately felt disoriented. I just didn’t know what to try next and had a strong urge to just use a thick rag for the remainder of the winter.

Phase 3: Better search term

The optimistic googling didn’t help much (the usual deal), so it was time to try and get specific.

mlTask: A slight deliberation narrowed the issue down: it’s the similarity search component that needed to be optimised. So, a modified search term became- ‘Largescale similarity search’. Going through several reddit discussions on largescale similarity vector search and perusing the Readmes of several github repos (both pythonic and otherwise), certain algorithms seemed to be talked about more than others: Local Sensitive Hashing (LSH), Graph-based proximity methods, specialised tree structures such as vp-tree. Within these choices, Graph-based proximity methods seemed to be more promising as per the community. Another common term that came up is: approximate nearest neighbour. Along with the algorithms themselves, several open source, ready-to-use python packages were available as implementations. Some more talked-about were: Spotify’s annoy, facebook’s faiss and nmslib. Some of these algorithms/packages were already known to me (Phase 2.1 and otherwise), but I wanted this to be a clean research attempt lest I miss something more suited to my required functionality. Next, I needed to look for a way to compare these algos/packages. So, the better search term became: ‘Benchmarks annoy faiss nmslib’.

jacketTask: No amount of deliberation got me any closer to improving my search term, so I had no choice but to click what seemed like a gazillion shopping links from the preliminary search term. After reading the descriptions, certain terms emerged: 3-in-1 jackets, something about polyester, about breathability (it’s a stupid jacket, not an Asthma-medicine-dispenser), about fur-hoods, about ass-coverability, outer-shell, inner-covering, etc. Feeling megastupid, I decided to google what a 3-in-1 jacket is: an outer shell with a detachable inner jacket (usually a fleece jacket), combinations of which allow one to wear it in 3 different ways. This is what I had found fascinating about JW when I was looking for my first winter jacket 8 years ago (didn’t buy a JW then #budget #studentlife). So, at least I knew that I wanted a 3-in-1 jacket now instead of other variants. This gave me a better search term: ’3-in-1 Jack Wolfskin winter jacket men’.

Phase 4 : Drill down/ Noise reduction

Now that I had better search terms at my disposal, I was ready to get my hands dirtier.

mlTask: The better search term helped me find a veritable jackpot; a repo that compared many different largescale similarity search algos/packages on several open source datasets having different properties. It seemed like a rare stroke of good luck and I’m grateful to the author for putting in the effort that helped reduce my own. Anyway, I went about examining the detailed recall graphs in the repo and ended up deciding to check 4 python packages that seemed to be doing decently well across most datasets: annoy, nmslib (Hierarchical Navigable Small World Graph variant), faiss and pynndescent. (nmslib additionally performed best on a 960 dimensional dataset, which was fairly close to the 1024 dimensions that CNN method of imagededup produced). I was now in a position to set up an experiment to deduplicate another dataset using all 4 packages in order to narrow it down further.

jacketTask: What seemed like a win in the last phase quickly resembled a case of counting chickens before hatching.

It turns out that there are many different models of 3-in-1 jackets manufactured by JW: Arland, Iceland, Gotland. There are probably more variants, but these 3 came up regularly. Next, I tried to find some benchmarks that could compare these variations. The search terms moved in the vicinity of: ‘Gotland vs Iceland’, ‘Gotland vs Arland’, ‘Iceland vs Arland vs Gotland’. Some links turned up, but nothing promising came up that would offer a comprehensive comparison. Additionally, most blogs kept mentioning ‘The North face’ as a comparison leading me to question my JW decision. I decided not to pay heed to the noise and stayed with JW. Having been unable to find a good comparison, I started re-reading the specifics: the outer-shell material, breathability, stretchability and so-on.

I had never thought of Jacket-making as anything mega-technical. I guess my primitive ideas about jackets could be summed up as- “Thick covering=you stay alive in cold, thin covering=you die”. Ignoring my ego, I googled up: ‘What to look for in a winter jacket’. (Bought several winter jackets over last years, for me and others, so wasn’t expecting to go back to absolute basics in this manner). Several verbose albeit wonderful explanatory blogs later(reddit-you beauty), I felt like a caveman had just discovered electricity (not fire, but electricity). Now, I knew that simply looking for a ‘winter jacket’ doesn’t have much meaning without the intended purpose: Are you going to do any type of sport while wearing it? Maybe go for more stretchability/ breathability. Are you looking for extreme hikes? Go for more expensive outer shell materials like Gore-tex (JW has Texapore as a Gore-tex implementation). Are you looking to hang about in the city? Then don’t worry too much about stretchability. Another basic difference I had looked over was the one between water/wind-proofing and water/wind-resistance. Certain materials were more resistant to water and wind than others, which is kind of important in the precipitation-prone and windy Germany climate. This gave me some ammo to critique these 3-in-1 variants based on these properties.

Phase 5: Decision

At this point, some kind of final decision needed to be reached using the last bits of deciding factors.

mlTask: Knowing the 4 packages, I ran a quick experiment: Deduplicating a dataset of 10,000 images. While all of the methods did better than the current similarity search implemented in imagededup, 2 packages: nmslib and faiss vastly outperformed the others. Additionally, faiss doesn’t support windows, which would make it unsuitable for being an imagededup dependency. Hence, nmslib emerged as a clear winner. I’m currently in the process of running an experiment that attempts to deduplicate a dataset comprising 1 million images using nmslib (in addition to reading the paper). If the results are encouraging and there are no conflicting dependency issues with imagededup, nmslib would be the ultimate winner and conclude my attempt at finding something for largescale similarity search. For this task, a lot more needs to be done, but due to this exercise, the initial hassle of coming up with a promising candidate algo/implementation has been tackled to a good degree.

jacketTask: Ass-coverability was an important consideration for me and several amazon reviews had mentioned that Arland failed at this task. So, Arland was out. The choice came down to Iceland vs Gotland which seemed too similar when it came to these properties. So, I just picked up Gotland in the end. This concluded my jacket buying antics.

Phase 3 and 4 usually go on in a loop: You find a better search term, then you drill down and find a still better search term and so on until you feel you have enough information to go onto phase 5.

Phase 6: The crippling doubt

Could I have done better? Maybe spend more time researching specifics? Maybe my decision isn’t a good one after all? Did I screw up in a massive way somehow? All these worries are legit, but tasks can not stay in research phase forever. Time-to-completion vs quality is a real trade-off that needs to be respected. What good is buying a winter jacket in May because you wasted all winter coming up with more info? What good is forever second-guessing and never going through with code implementations in the fear of having barfed somewhere?

The fact that these 2 completely unrelated tasks ended up sharing a similar research approach interested me. It told me that in today’s world, it’s not just the scientists, the technical workers or the journalists that need to engage in research, it’s literally anybody trying to make a well informed decision. Therefore, developing a robust research approach is of paramount importance for everyone on the planet and must be treated as a high-priority addition to everyone’s to-do list.

Do share your approach to research!

The Machine learning world currently sees Data Scientists (DS) performing one or both of the following 2 prominent roles:

1. Where a DS receives a data dump, applies some Machine learning algo on the data and reports back the results in the form of some presentation or report.
2. Where the DS creates a usable piece of software for the stakeholders to consume the machine learning models.

In this blog post, I’m attempting to display an example approach to the second aspect of a DS’s job i.e., creating some software that can be used by the stakeholders. Specifically, we would create a web-service that can be queried to obtain the predictions from a machine learning model. The post is mostly intended for machine learning practitioners who would like to go beyond only developing models.

Tech-stack: Python, Flask, Docker, AWS ec2

The workflow can be broken down into following basic steps:

1. Training a machine learning model on a local system.
2. Wrapping the inference logic into a flask application.
3. Using docker to containerize the flask application.
4. Hosting the docker container on an AWS ec2 instance and consuming the web-service.

DISCLAIMER: The system presented here is light years away from what a commercial production system should look like. The key takeaways from this blogpost should be the development workflow, acquaintance with the tech stack and getting the first taste of building an ML production system.

Let’s start with the first step.

Training a machine learning model on a local system

We need some machine learning model that we can wrap in a web-service. For demo purpose, I chose a logistic regression model to do multiclass classification on iris dataset (Yep, super easy! #LazinessFTW). The model was trained on a local system using python 3.6.

Using the familiar scikit-learn, the above mentioned model can be trained quickly. For model development, refer the notebook ‘Model_training.ipynb’ in the github repo for this blog. There are only 2 important aspects of model development that I would like to highlight:

1. The model file generated after training is stored as a pickle file which is a serialized format for storing objects. (In the repo, the file is named ‘iris_trained_model.pkl’)
2. The inference call (.predict()) call requires 4 features per test sample in the form of a numpy array.

Wrapping the inference logic into a flask web service

Now that we have the trained model file, we are ready to query the model to get a class label for a test sample. The inference is as simple as calling a predict() function on the trained model with the test data. However, we would like to build the inference as a web-service. For this purpose, we would use Flask.

Flask is a powerful python microwebserver framework that allows us to build REST API based web-services quickly with minimum configuration hassle. Let’s dive into the code:

a. First, let’s define a simple function to load the trained model file.

Here, we define a global variable called ‘model’ and populate it within the load_model() function. The purpose of using a global variable will become clear shortly.

b. Next, we instantiate a Flask object called ‘app’:

c. Now, we define a home endpoint, which when hit, returns a ‘Hello World!’ message.

Notice the use of app.route decorator.

d. Now, we define a ‘predict’ endpoint. The endpoint accepts a ‘POST’ request wherein the test data on which we wish to get a prediction is received by the endpoint. Keeping things simple, the function works only when a single test sample needs to be predicted (won’t work if multiple samples need to be predicted in a single call to the endpoint).

Notice the direct call to the predict function through the ‘model’ variable.

e. Finally, declare the main function:

Here, a call to the load_model() function ensures that the variable ‘model’ is populated with the trained model attributes (and hence the need for a global model variable). So, there is no need to load the model repeatedly with every call to the predict endpoint. This allows the web-service to be quick. The response is returned as a string which is the predicted class label.

The complete flask specific code is as below:

At this point, the web-service is ready to be run locally. Let’s test this.

Execute the command python app.py from the terminal. Go to the browser and hit the url 0.0.0.0:80 to get a message Hello World! displayed. This corresponds to the home endpoint return message.

NOTE: A permission error may be received at this point. In this case, change the port number to 5000 in app.run() command in app.py. (Port 80 is a privileged port, so change it to some port that isn't, eg: 5000)

Next, let’s test if we can get predictions using this web-service using the following curl post request on the terminal:

curl -X POST \
   0.0.0.0:80/predict \
   -H 'Content-Type: application/json' \
   -d '[5.9,3.0,5.1,1.8]'

The curl request posts one test sample [5.9,3.0,5.1,1.8]to our web-server and returns a single class label.

Using docker to containerize the flask service

Up to this point, we have a web-service that runs locally. Our ultimate intention is to be able to run this piece of code on a cloud virtual machine.

In the Software Development world, there is a famous justification given by a developer whose code was found to be broken by a tester: ‘But it worked on my machine!’. The problem portrayed here can usually be attributed to a lack of consistent environment that runs the software across different machines. Ideally, our code itself should be independent of the underlying machine/OS that runs it. Containerization allows developers to provide such isolation.

How is it important here?

Our intention is to run our web-service on a cloud VM. The cloud VM itself may run any OS. Containerization of our web-server allows us to avoid the trouble of running into environment related issues. If the containerized code works on one machine, it will surely run on another irrespective of the characteristics of the machine. Docker is the most famous containerized technology out there at this point and we will be using the same here. For a quick tutorial on docker, check this link.

Let’s dive into the Dockerfile that comprises a set of instructions for docker daemon to build the docker image.

We pull the base docker image from python dockerhub repo on which our specific build instructions are executed. The COPY commands are simply taking specific files from the current folder and copying them over to a folder called ‘deploy’ within the docker image we are trying to build. In addition to app.py and model file, we also need a requirements file that lists specific versions of python packages we use to run our code. The WORKDIR command changes the working directory to ‘deploy/’ within the image. We then issue a RUN command to install specific python packages using the requirements file. The EXPOSE command makes the port 80 accessible to the outside world (our flask service runs on port 80; we need this port inside the container to be accessible outside the container).

Issue the build command to end up with a docker image:

docker build -t app-iris .

(Don’t forget the period at the end of the command).

Use command ‘docker images’ to see a docker image with a docker repository named ‘app-iris’ created. (Another repository named python will also be seen since it is the base image on top of which we build our custom image.)

Now, the image is built and ready to be run. We can do this using the command:

docker run -p 80:80 app-iris .

The above commands uses -p flag to map port 80 of the local system to the port 80 of the docker container for the redirection of traffic on local HTTP port 80 to port 80 of the container. (If you are using local port 5000 instead of port 80, change the port mapping part of the command to 5000:80).

Let’s test if this works by hitting the URL: http://0.0.0.0:80 on the browser which should display ‘Hello World!’ which is the home endpoint output message (If port 5000 is used, modify the http port to 5000 in the url). Also, use the curl request mentioned earlier to check if the predicted class label is returned.

Hosting the docker container on an AWS ec2 instance

We already have a containerized application that works on our local system. Now, what if someone else wishes to consume the service? What happens if we need to build an architectural ecosystem around the service that needs to be available, automated and scalable? It’s easy to see that having a web-service running locally would be a very bad idea. So, we wish to host the web-service somewhere on the internet to fulfil the requirements we listed. For this blog, we choose to host our service on an AWS ec2 instance.

As a prerequisite, one needs to have an AWS account for using the ec2 instance. For new users, there are several AWS resources that are available for free for a period of 1 year (usually up to some limit). In this blog, I would be using a ‘t2.micro’ ec2 instance type which is free tier eligible. For users who have exhausted their AWS free-tier period, this instance costs around 1 cent(USD) per hour at the time of writing this blog; a super negligible amount to pay.

Let’s start with the process.

Log into the AWS management console and search for ec2 in the search bar to navigate to EC2 dashboard.

Look for the below pane, select ‘Key Pairs’ and create one.

This will download a ‘.pem’ file that is the key. Save this file somewhere safely. Now navigate to the location of this file on your system and issue the below command with key file name replaced by yours:

chmod 400 key-file-name.pem

This commands changes permissions on your key pair file to private. The use of key pairs will be explained later.

Next, click ‘Launch Instance’ on the EC2 dashboard:

Choose the Amazon Machine Instance (AMI) from the list of options. An AMI determines the OS that the VM will be running (plus some other stuff we don’t care about at this point). For this blog, I chose ‘Amazon Linux 2 AMI’ which was the default selection.

The next screen allows you to select the instance type. This is where the hardware part of the VM can be selected. As mentioned previously, we will work with ‘t2.micro’ instance.

You can select ‘Review and Launch’ that takes you to ‘Step 7: Review Instance Launch’ screen. Here, you need to click the ‘Edit Security Groups’ link:

You now have to modify the security group to allow HTTP traffic on port 80 of your instance to be accessible by the outside world. This can be done by creating a rule. At the end, you should end up with such a screen:

In the absence of this rule, your web-service will never be reachable. For more on security groups and configuration, refer AWS documentation. Clicking on the ‘Launch’ icon will lead to a pop up seeking a confirmation on having a key-pair. Use the name of the key pair that was generated earlier and launch the VM.

You would be redirected to a Launch screen:

Use the ‘View Instance’ button to navigate to a screen that displays the ec2 instance being launched. When the instance state turns to ‘running’, then it is ready to be used.

We will now ssh into the ec2 machine from our local system terminal using the command with the field public-dns-name replaced with your ec2 instance name (of the form: ec2–x–x–x–x.compute-1.amazonaws.com) and the path of the key pair pem file you saved earlier.

ssh -i /path/my-key-pair.pem ec2-user@public-dns-name

This will get us into the prompt of our instance where we’ll first install docker. This is required for our workflow since we will build the docker image within the ec2 instance (There are better, but slightly complicated alternatives to this step). For the AMI we selected, the following bunch of commands can be used:

sudo amazon-linux-extras install docker
sudo yum install docker
sudo service docker start
sudo usermod -a -G docker ec2-user

For an explanation of the commands, check the documentation.

Log out of the ec2 instance using the ‘exit’ command and log back in again using the ssh command. Check if docker works by issuing the ‘docker info’ command. Log out again or open another terminal window.

Now let’s copy the files we need to build the docker image within the ec2 instance. Issue the command from your local terminal (not from within ec2):

scp -i /path/my-key-pair.pem file-to-copy ec2-user@public-dns-name:/home/ec2-user

We would need to copy requirements.txt, app.py, trained model file and Dockerfile to build the docker image as was done earlier. Log back into the ec2 instance and issue ‘ls’ command to see if the copied files exist. Next, build and run the docker image using the exact same commands that were used in the local system (Use port 80 at all locations in the code/commands this time).

Hit the home endpoint from your browser using the public dns name to see the familiar ‘Hello World!’ message:

Now send a curl request to your web-service from local terminal with your test sample data after replacing the public-dns-name by yours:

curl -X POST \
public-dns-name:80/predict \
-H 'Content-Type: application/json' \
-d '[5.9,3.0,5.1,1.8]'

This should get you the same predicted class label as the one you got locally.

And you are done! You can now share this curl request with anyone who wishes to consume your web-service with their test samples.

When you no longer need the web-service, do not forget to stop or terminate the ec2 instance:

Some additional thoughts

This is a super basic workflow intended for ML practitioners itching to go beyond model development. A huge number of things need to be changed to make this system into one that is more suited to a real production system. Some suggestions (far from complete):

1. Use a Web Server Gateway Interface (WSGI) such as gunicorn in the flask app. Bonus points for using nginx reverse proxy and async workers.
2. Improve security of the ec2 instance: The service is currently open to the entire world. Suggestions: Restrict access to a set of IPs.
3. Write test cases for the app: Software without testing = Shooting yourself in the leg and then throwing yourself in a cage full of hungry lions all the while being pelted with stones. (Moral: Do not float a production software without thoroughly testing first)

A lot more could be added to the above list, but maybe that’s something for another blog post.

Github repo: https://github.com/tanujjain/deploy-ml-model

It would be great to hear your feedback!

Simple way to deploy machine learning models to cloud was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the biggest, ever-present banes of a data scientist’s life is the constant wait for the data processing code to finish executing. The problem with slow code affects almost every step of a typical data science pipeline: data collection, data pre-processing/parsing, feature engineering, etc. Many times, the gigantic execution times even end up making the project infeasible and more often than desired, forces a data scientist to work with only a subset of data, depriving him/her of the insights and performance improvements that could be obtained with a larger dataset.

One of the tools that could mitigate this problem and speed up data science pipelines is parallelisation through multiprocessing or multithreading.
In this blogpost, I am sharing my experience of solving a data collection problem by using an approach that utilizes multiprocessing with python. The main achievement of this exercise was the development of a general approach for parallelisation wherein a code template was created and used. Using the approach, I was able to reduce the execution time from several days to under an hour for a practical problem discussed in this blog. The same code template has been tested for other use cases and has proven to be effective.

The Vanilla approach to parallelisation

This is the usual way parallelisation is applied in python.

1. Get a list of all items for which the same task needs to be performed independent of other items.
2. Workers execute in parallel and each worker applies the task logic to a single item in this list and obtains the result for this specific item.
3. After a worker finishes the task on a single item, it picks up another item from the global list of items and applies the task logic to it.
4. The process goes on until the list of items gets exhausted.

The Splitting approach to parallelisation (New)

This is the new parallelisation approach.

1. Get a list of all items for which the same task needs to be performed independent of other items.
2. Divide the list of items into sublists.
3. Pass one sublist to each worker/thread so that each worker/thread iterates over its own sublist in a serial manner and performs the required task.
   The workers themselves execute in parallel. Thus, even though each worker is performing the task in a serial manner, the workers are working simultaneously on their own sublist of tasks and hence, parallelisation is achieved as the net effect. Therefore, the approach is actually a mix of parallel and serial processing.

In contrast, the Serial approach works as follows:

1. Get a list of all items for which the same task needs to be performed.
2. Pick up an item in the list and execute the task for this item.
3. Pick up the next item in the list. The process goes on until the list gets exhausted.

Problem Description

For one of my recent projects, I was faced with the problem of downloading a huge number of emails from a mail server (~3 million). An attempt to download the mails serially resulted in 100k emails being downloaded after 5 days of run time. The email download time clearly needed to be reduced for which parallelisation seemed like an obvious candidate.

For the purpose of code demonstration, I am downloading emails from a public imap (created a fake email id on yandex and filled it with 1200 emails) using Multiprocessing with the splitting approach. Later on, the same task is going to be performed with all the variants described above as well (code not shown for these in the post). My machine is a MacBook Pro with 2.8 GHz i7 quad core processor.

Multiprocessed splitting

1. Get a list of all mail ids present in the mail box after establishing connection with the imap server.
2. Select the number of multiprocessing workers (I generally set this to twice the number of logical cores on my machine).
3. Divide the list of mail-ids into sublists.
4. Launch multiple processes where each process corresponds to a worker and has a sublist of items it must operate upon. Each process establishes an independent connection with the imap server and calls a function that performs the download.
5. Each worker downloads the mails serially using the sublist of mail-ids it possesses.

Let us develop the logic for each of these steps:

1. Get a list of all mail ids present in the mail box after establishing a single connection with the imap server.

2. Select the number of multiprocessing workers (I generally set this to twice the number of logical cores on my machine).

3. Divide the list of mail-ids into sublists.

4. Launch multiple processes where each process corresponds to a worker and has a sublist of items it must operate upon. Each process establishes an independent connection with the imap server and calls a function that performs the download.

The get_mails function launches a pool of workers and passes a list of sublists to mp_process function such that each worker gets one sublist of mail-ids to process. Virtually, each worker is executing its own copy of the mp_process function with its own specific sublist of mail-ids that it needs to download. All the workers execute in a parallel fashion with no dependence on each other.

5. Each worker downloads the mails serially using the sublist of mail-ids it possesses.

This function iterates over the sublist of mail-ids it receives from its calling process and begins downloading the sublist serially.

The complete code can be found in the appendix.

The general template that can be reused and adapted to various tasks is:

EXPERIMENTS

Several methods were compared in terms of execution time:

1. Serial download (Serial).
2. Multiprocessing download using the vanilla approach (MP-V).
3. Multiprocessing download using the splitting approach (code shown above) (MP-Sp).
4. Multithreaded download using the vanilla approach (Mth-V).
5. Multithreaded download using the splitting approach (Mth-Sp).

The number of threads/workers were also varied across different runs of multithreading/multiprocessing approaches (X-axis of the graph below).

RESULTS

The following conclusions can be drawn from the experiments:

1. The splitting approach always outperforms the vanilla approach and the serial approach regardless of the number of workers/threads.
2. The vanilla approach performs worse than the serial approach if the number of threads/workers are less than 8.
3. There is no real difference between the performance of multithreading and multiprocessing.
4. The vanilla approach tends to give better gains in terms of execution time as the number of workers/threads increase compared to the splitting approach. (Further experiments were carried out so that the vanilla approach uses more workers/threads, but the splitting approach was always better even when it used fewer workers/threads.)

Therefore, using a small number of workers/threads with the splitting approach ensures the quick completion of a task while only occupying a small amount of resources.

SUMMARY

An alternative approach of splitting item list into sublists for achieving better parallelisation in python was introduced and shown to be better than the usual approach wherein each item in the list is picked and processed by workers/threads. A practical example was used to demonstrate the performance improvement with the use of the splitting approach. Furthermore, a template to use this approach was developed which makes it possible to apply it regardless of the nature of the underlying task.

APPENDIX

ADDITIONAL QUIRKS

1. The experiments were carried out on a rather small set of data (only 1200 emails with 2KB per email). It is possible that vanilla approach could outperform the splitting approach for much larger datasets (Splitting approach seemed promising with the initial experimentation, so I ended up choosing it for my final task which involved downloading hundreds of Gigabytes of data across 3 million mails).
2. Multiprocessing vs. Multithreading: As can be seen from the results, these two approaches performed similarly. However, as far as python is concerned, Global Interpreter Lock (GIL) usually ends up hampering the multithreading performance and it is usually difficult to ascertain the usability of multithreading for a given use case in python. Multiprocessing, on the other hand, suffers from no such drawbacks and hence, I tend to prefer multiprocessing over multithreading in most cases. But then again, multiprocessing eats up a lot more resources than multithreading, so there’s usually a trade-off between multiprocessing and multithreading when it comes to python. (I chose multiprocessing for my final task).
3. Several runs were done for each experiment. During one of the runs, a strange behaviour was observed with the vanilla approach (for both multithreading and multiprocessing) — About 95% of tasks got executed rather quickly, but the last 5% took an extremely long time to finish. This issue wasn’t reproduced.

The results are largely empirical in nature and a satisfactory explanation of why splitting approach outperforms the vanilla approach still evades me. I would be happy to hear any ideas on this.

Parallelisation In Python — An Alternative Approach was originally published in idealo Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.