What is VertexAI?

It is a comprehensive AI platform for building, deploying, and managing machine learning models at scale. Besides your own trained models, VertexAI also provides a catalog of models to be used out of the box. With the amount of model options available on VertexAI, it is easily one of the platforms that any AI practitioner would want easy programmatic access to for development.

Challenge

Unfortunately, calling VertexAI API is not as straightforward as desired. Existing solutions for calling it can be overly complex and impractical for certain use cases. The OAuth2 authentication process, which is typically required, involves multiple intricate steps, such as redirecting users to an authorisation page, handling callback responses, and managing refresh tokens. This convoluted workflow becomes particularly cumbersome in server-to-server interactions where user interaction is not feasible or desirable.

Moreover, the need to periodically refresh access tokens adds an additional layer of complexity, requiring secure storage mechanisms and careful token management to avoid expiration issues. OAuth2 workflows often necessitate periodic user consent and manual approval, rendering them unsuitable for automated, large-scale deployments that demand rapid, seamless access to the VertexAI API.

This reliance on user interaction further highlights the impracticality of OAuth2 for many programmatic access scenarios, where minimizing human intervention is crucial for efficiency and reliability. These challenges underscore the limitations of OAuth2 as an authentication method for applications that require consistent, automated access to the VertexAI API, making it a sub-optimal choice in such contexts.

Solution

For server-to-server interactions, leveraging service account keys offers a more streamlined and efficient authentication mechanism. With service accounts, developers can generate key files that enable their applications to authenticate seamlessly, without the need for user interaction or manual approval processes.

This direct authentication method simplifies the implementation process significantly, reducing the overhead associated with token management strategies. Furthermore, it enhances security by eliminating the necessity to handle refresh tokens, which can be a potential vulnerability if not managed properly.

Service accounts are specifically designed to support automated, large-scale deployments, making them an ideal choice for backend systems that require consistent and reliable access to VertexAI and other GCP services. This streamlined approach not only improves operational efficiency but also ensures that applications can function smoothly and securely in a production environment without disruptions caused by complex authentication workflows.

Introduction

In this article, we’ll explore how to call the VertexAI API using Python’s request library and without the need for OAuth2 login. This approach is particularly useful for backend systems that require programmatic access to services like VertexAI, where triggering a login can be inconvenient compared to generating a token for authentication.

While Google Cloud Platform (GCP) documentation provides information on various ways to provide credentials, it can be challenging to navigate and time-consuming to parse through. This article aims to provide a quick and straightforward method to leverage Service Accounts to generate a Bearer token, which can then be used to call VertexAI using the requests package in Python.

Step 1: Creating a Service Account

The first step is to create a Service Account in GCP. Follow these steps:

1. Go to the IAM & Admin section in your GCP console.
2. Navigate to the Service Accounts tab.
3. Click on Create Service Account and provide a suitable name and description.
4. Grant the Service Account access to VertexAI by assigning the Vertex AI Service Agent role.

Initially, the Service Account won’t have a Key ID. To create one:

1. Click into your newly created Service Account.
2. Go to the Keys tab and click on Add Key.
3. Select Create new key and choose the JSON key type.
4. Download the JSON file, which contains information about your Key ID.

This JSON file is essential for the next step.

Step 2: Generating the Bearer Token

Using the JSON file obtained in the previous step, you can generate a Bearer token to pass as an authentication key for the VertexAI API. Here’s an example Python function that accomplishes this task:

import json
from google.auth.jwt import OnDemandCredentials

def get_oauth_token(sa_key_fpath: str) -> str:
    cred_file = json.load(open(sa_key_fpath))
    gcp_cred_object = OnDemandCredentials.from_service_account_info(cred_file)
    bearer_token = gcp_cred_object._get_jwt_for_audience(
        audience="https://aiplatform.googleapis.com/"
    )
    return bearer_token.decode()
bearer_token = get_oauth_token("service_account_key_file.json")

In this code snippet, we leverage Google’s Python SDK and the OnDemandCredentials object to create the Bearer token. Note that while we're using OnDemandCredentials here, the SDK provides other objects that serve similar purposes, so feel free to explore the codebase of the package.

Step 3: Calling the VertexAI API

With the Bearer token generated, you can now call the VertexAI API. The first step is to determine the appropriate endpoint by referring to the VertexAI API reference documentation.

Here’s an example Python code snippet that demonstrates how to call the VertexAI API using the requests library:

import requests

response = requests.post(
    url="https://asia-east1-aiplatform.googleapis.com/v1/projects/{project_name}/locations/{location}/publishers/google/models/text-bison@002:predict",
    headers={
        "Content-Type": "application/json",
        "authorization": f"Bearer {bearer_token}"
    },
    json={
        "instances": [
            {
                "role": "user",
                "content": "hello world!"
            }
        ],
        "parameters": {
            "max_tokens": 256,
            "temperature": 0.8,
            "frequency_penalty": 0,
            "presence_penalty": 0,
            "top_p": 0.95
        }
    }
)

In this example, we’re using the text-bison model in the asia-east1 region. Make sure to replace {project_name} and {location} with your appropriate values.

Conclusion

By following the steps outlined in this article, you can successfully call the VertexAI API using REST API without the need for OAuth2 login. This approach allows backend systems to programmatically access VertexAI services in a more convenient manner, eliminating the need for triggering a login process. With the Bearer token generated from the Service Account, you can make API requests to VertexAI using the requests package in Python or any other programming language that supports HTTP requests.

Overview of Inferentia

AWS unveiled the 2nd version of Inferentia, which is their AI focused GPU that’s been optimised for model inference. The 1st version of Inferentia is already a fantastic compute instance offer that touts 2.3x throughput, much lower latency at 70% lower cost per inference when benchmarked against comparable EC2 instances. Inferentia2 levels up with 4x more throughput and 10x lower latency, but with 3x the cost.

With these performance benchmarks, it is obvious for data scientists and ML engineers to gravitate towards leveraging on Inferentia. However, there are some conditions that does not make Inferentia a good fit or eligible for some teams. Here are some that pertained to me when working on my projects:

1. GenAI models are only supported on Inferentia2. This is because Inferentia1 does not support transformer decoders. Both support transformer encoders.
2. Inferentia2 is only available in the US right now. Which could be a key reason for anyone to use Inferentia1 instead.
3. Both versions of Inferentia may not support some transformer architectures. You can refer in their documentation for more information.

For my project we needed an embedding model situated in Singapore that can be flexible on transformer architectures. This meant that Inferentia1 was the way to go.

So for those that need to deploy Huggingface models outside the US, or want to deploy embedding models for lower cost than Inferentia2, this article will show you how to get a Huggingface model to work on Inferentia1.

Content

There are 4 parts to preparing a Huggingface model for deployment on Inferentia1.

1. Installation of relevant dependencies
2. Conversion of Huggingface model (Torch model) into Neuron format
3. Packaging the Tokenizer and Inference code with the Neuron formatted model
4. Setting up a SageMaker Endpoint using the Neuron model payload package

Handling Dependencies

The packages for converting Torch format models into Neuron format models can be quite volatile given the pace of development of the AI space right now. Huggingface has an API to help perform this conversion. However, as of publication of this article, the API does not work. Hopefully the method I share here will help go around that.

These are the packages that I had to download for my Neuron conversion to work. The extra index-url is needed to pull from AWS’s pip repository.

--extra-index-url=https://pip.repos.neuron.amazonaws.com
neuron-cc[tensorflow]
torch-neuron
protobuf
optimum[neuron]

If you’re on a Jupyter notebook remember to restart your kernel after installing these packages.

Neuron Conversion

For the conversion, instead of using the Huggingface API, we will be using the script in the optimum-neuron package. Out of all the methods I could find online, this seemed to be the most consistent way for me to perform the conversion.

import time
import shutil
import subprocess
import sagemaker
from transformers import AutoTokenizer
from sagemaker.s3 import S3Uploader

HF_MODEL="BAAI/bge-base-en-v1.5"
HF_TASK="feature-extraction"
SEQ_LENGTH=512
BATCH_SIZE=1
OUTPUT_FOLDER="embedding-model"
MODEL_NAME="baai-embedding"

sess = sagemaker.Session()
sagemaker_session_bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)
print(f"Bucket: {sagemaker_session_bucket}")
print(f"Role: {role}")

I performed all of these chunks of codes in separate cells of my Jupyter notebook.

• HF_MODEL: Is the model from Huggingface that you’d like to convert into a Neuron model to deploy with Inferentia1
• HF_TASK: This is the task type of your model. For embedding models, the task is called “feature-extraction”
• SEQ_LENGTH: One of the conditions required to leverage on Inferentia is to have a fixed sequence length for your transformer model. Having a fixed input sequence length helps with hardware optimisation for inference
• BATCH_SIZE: Similar reason as above
• OUTPUT_FOLDER: Folder name for where we want to export the converted Neuron model into
• MODEL_NAME: The name for your exported Neuron model

Once all of these variables are set, you can run this code. Since I’m doing this on Jupyter notebook, I have the ! in front of my CLI code.

!optimum-cli export neuron \
--model {HF_MODEL} \
--task {HF_TASK} \
--sequence_length {str(SEQ_LENGTH)} \
--batch_size {str(BATCH_SIZE)} \
--disable-validation \
{OUTPUT_FOLDER}

This conversion is actually a compilation of the Huggingface model in Neuron format. And the compilation process is done on CPU. This means that instead of splurging on a GPU instance for this script, you should just choose a CPU optimised instance (e.g c5, etc).

After the code is done running, you should see these logs printed. As you can see the compilation duration is about 2.5 minutes for the BAAI embedding model.

Packaging the Neuron Model

Now that we have compiled a Neuron model, it should sit within the folder that you have previously declared. The next step is to download the tokenizer for the same model into the OUTPUT_FOLDER and then add an inference script for the Neuron model under OUTPUT_FOLDER/inference.py.

Settling the tokenizer is simpler.

tokenizer = AutoTokenizer.from_pretrained(HF_MODEL)
tokenizer.save_pretrained(OUTPUT_FOLDER)

However, the inference script can get a bit tricky.

%%writefile -a {OUTPUT_FOLDER}/code/inference.py # only for jupyter notebook
# for BAAI/bge-base-en-v1.5
import os
from transformers import AutoConfig, AutoTokenizer
import torch
import torch.neuron

# To use one neuron core per worker
os.environ["NEURON_RT_NUM_CORES"] = "1"

# saved weights name
AWS_NEURON_TRACED_WEIGHTS_NAME = "model.neuron"


def model_fn(model_dir):
    # load tokenizer and neuron model from model_dir
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    model = torch.jit.load(os.path.join(model_dir, AWS_NEURON_TRACED_WEIGHTS_NAME))
    model_config = AutoConfig.from_pretrained(model_dir)

    return model, tokenizer, model_config


def predict_fn(data, model_tokenizer_model_config):
    # destruct model, tokenizer and model config
    model, tokenizer, model_config = model_tokenizer_model_config

    # create embeddings for inputs
    print(data)
    inputs = data.pop("inputs", data)
    embeddings = tokenizer(
        inputs,
        return_tensors="pt",
        max_length=model_config.neuron["static_sequence_length"],
        padding="max_length",
        truncation=True,
    )
    # convert to tuple for neuron model
    neuron_inputs = tuple(embeddings.values())
    
    # run prediciton
    with torch.no_grad():
        model_output = model(**neuron_inputs)[0] # last_hidden_state
        sentence_embeddings = model_output[:, 0] # first vector for each payload

    print(sentence_embeddings.shape)
    sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)
    return [{"inputs": inputs, "embeddings": sentence_embeddings[0].tolist()}]

Some key things to note here is the use of the magic command %%writefile in Jupyter notebook to write this code into a file under the stated directory. Next would be to note the model_output of the Neuron model. As shown in the code example, we can extract out the last_hidden_state. But depending on what type of Huggingface model you choose, you may have to tweak this accordignly. For example, if you use a classification model, you’d need to add in a softmax layer.

with torch.no_grad():
  predictions = model(*neuron_inputs)[0]
  scores = torch.nn.Softmax(dim=1)(predictions)

# return dictonary, which will be json serializable
return [{"label": model_config.id2label[item.argmax().item()], "score": item.max().item()} for item in scores]

Something like this instead.

Once your tokenizer, inference script and Neuron model are all situated in the same file, we can compress and export it. Remember to keep the inference script into a sub-folder named “code”.

%cd {OUTPUT_FOLDER}
!tar zcvf {MODEL_NAME}.tar.gz *
%cd ..

This was the Jupyter notebook code cell that I used. I am not sure why but I get an error when I run the tar line from outside the OUTPUT_FOLDER. And a fix for that was to cd into the folder to perform the compression.

After the compression, you should see a list of files that have been compressed into the .tar file like this. Now that we have the Neuron model payload, we can proceed with the deployment.

Deployment onto Inferentia1

In preparation for deployment, we need to first upload the .tar file into an S3 bucket.

s3_model_path = f"s3://{sess.default_bucket()}/neuron/embeddings"
s3_model_uri = S3Uploader.upload(
    local_path=f"{OUTPUT_FOLDER}/{MODEL_NAME}.tar.gz",
    desired_s3_uri=s3_model_path
)

The reason we do this is because when we create a model object on SageMaker, we need to reference a model file that is located in S3. For this deployment, what we are essentially doing are these 3 things:

1. Create a model object in SageMaker with reference to the .tar file in S3
2. Create a SageMaker endpoint configuration referencing the model object
3. Create a SageMaker endpoint using the endpoint configuration

But instead of doing all of this manually, SageMaker has an API to handle it for us.

from sagemaker.huggingface.model import HuggingFaceModel

huggingface_model = HuggingFaceModel(
    env={
        "HF_TASK": HF_TASK
    },
    model_data=f"s3://{sess.default_bucket()}/neuron/embeddings/{MODEL_NAME}.tar.gz",
    role=role,
    transformers_version="4.12",
    pytorch_version="1.9",
    py_version="py37",
    model_server_workers=1
)

huggingface_model._is_compiled_model = True

predictor = huggingface_model.deploy(
    endpoint_name=f"{MODEL_NAME}-ep",
    initial_instance_count=1,
    instance_type="ml.inf1.xlarge",
)

When calling the HuggingFaceModel class, what is happening here is the creation of the model object on SageMaker. We are detailing the location of the model artifact (S3), what environment variables are requires, and what Docker unage is to be used with this. The Docker container is referenced by these 3 arguments: transformers_version, pytorch_version, and py_version. You can find the Docker image in this repository. Although I have tried other configurations of arguments that work, but do not have matching Docker images in the repository. So I just take it as a reference.

But once the endpoint is created, we can invoke the EndPoint using boto3, or just call .predict() from the predictor.

Conclusion

And there you have it, some code and some explanation on how I compiled a Neuron model from a Huggingface model and deployed it on the Inferentia1 GPU instance. I hope this has been helpful and works for everyone. Enjoy the performance improvements and cost savings!

There are plenty of resources on the internet that details the various steps required for fine-tuning your foundation model. However, as I was searching around, I still ended up having to refer back and forth between materials before settling on an actual approach that I could take on. My business problem was this:

I would like to fine-tune a foundation model / LLM for my business domain, however in my industry, there are a ton of jargon, abbreviations and acronyms that are not commonly found on the internet. Is there some way I can factor in these domain specific tokens when fine-tuning?

In this post, I will cover the logic for deciding how and the code needed to fine-tune your LLM on domain data, and how to account for the domain specific tokens by updating the Tokenizer and Embedding layer of the model.

Idea

When wondering how to approach the problem, the first idea I had was to simply update the vocabulary of the model. Although a rather simple idea, following through with the solution was not as straightforward as I had expected.

If I simply just update the Tokenizer with the new words, the model will not work as optimally because it is not tuned to receive such tokens. The Embedding layer of the model will not recognise the token and will not have an embedding vector it can associate with it. To make matters worse, even if the Embedding layer can be updated, the question of how to instantiate the embedding vector for these new tokens becomes a problem. And it certainly doesn’t help to know that the Embedding layer can only be trained with a full fine-tuning — which requires A TON of computational resources.

After pondering a bit more, I came up with a solution for it.

Currently the way LLMs handle words that are not in its vocabulary is to either 1. tokenize it as an Unknown token, or 2. use sub-words to construct the word; which means that your new word is made up of way more tokens than necessary. Since this is already what it is doing, that is how I am going to instantiate the embedding vectors of new words. I will average out all the embedding vectors of the sub-word or unknown tokens and use them to represent the new word.

After updating the Tokenizer and Embedding layer with the new Domain words, I will then use a LoRA adapter to account for the update in vocabulary.

Let me explain with code.

Adding Domain Tokens

I experimented using the Llama-2 model that has been fine-tuned for chat — meta-llama/Llama-2–7b-chat-hf. Because I’m using the Llama-2 model, I needed to sign into my Huggingface. Besides that, the rest of the code should be self-explanatory.

The goal of this section is to update the vocabulary of the model by adding them as new tokens in the Tokenizer and adding their respective embedding vectors into the Embedding layer of the model.

In the code below, i’m simply loading up the Llama-2 model and quantising it down to 8-bits in preparation for a QLoRA fine-tuning. And I’m also making sure that the tokens used for padding are “end-of-sentence” tokens.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from datasets import load_dataset
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

import huggingface_hub
huggingface_hub.login(token="{add your HF token here}")

base_model_name = "meta-llama/Llama-2-7b-chat-hf"
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name, 
    device_map="auto", 
    load_in_8bit=True
)

base_model.config.use_cache = False

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

Next step is to prepare the a training dataset. Of course, this should be whatever data you have for your particular domain. But for this example, I’ll be using the alpaca-cleaned dataset from Huggingface.

Preparation of datasets is and will always be the most tedious part of fine-tuning a machine learning model. There are so many different kinds of datasets out there with varying formats for training. Some datasets would have formats for Question and Answer (i.e Question: {text}, Answer: {text}), some would have formats for Chatbots (i.e User A: {prompt}, User B: {reply}), and so on and so forth.

To help cut through all of these variations, it helps to remember that at the end of the day, you are essentially just training a Transformer decoder model. Meaning to say you just need to provide a text, and the model simply trains by taking token x in that pieces of text as the label, and all tokens x-1 and before as the predictors. So it doesn’t matter what format you choose as long as you use the same one during inference. In this project, I used the “instruction”, “input” and “response” format.

dataset = load_dataset("yahma/alpaca-cleaned")
subset_dataset = dataset["train"].shuffle(seed=9).select(range(1000))

def format_data(sample):
 return f"""### Instruction:
        {sample["instruction"]}
        
        ### Input:
        {sample['input']}
        
        ### Response:
        {sample['output']}
        """

Now comes the part where we add the new tokens into the Tokenizer and Embedding layer. The function oov_proportion() simply calculates the proportion of new tokens that does not exist in the Tokenizer’s existing vocabulary.

The more interesting function is the one below it called add_new_tokens() and it takes in the Model, Tokenizer and a list of new tokens and performs the following:

1. Filters away tokens that already exists in the Tokenizer’s vocabulary
2. Adds all the new tokens into the Tokenizer’s vocabulary
3. Resizes the number of Token embeddings in the Embedding Layer
4. Uses the embeddings of the sub-words of the new words as generated by the Tokenizer to create a reasonably relevant vector embedding for the new word
5. Replace the randomly initialised vector embeddings of the new words with the one created in Step 4

def oov_proportion(domain_words, tokenizer):
    """Calculate the proportion of words that are not in the tokenizer vocabulary"""
    vocab = tokenizer.vocab.keys()
    oov = [i not in vocab for i in domain_words]
    return sum(oov) / len(oov)

    
def add_new_tokens(model, tokenizer, new_tokens):
    """Adds new tokens into the tokenizer and embedding layer"""
    new_tokens = list(set(new_tokens) - set(tokenizer.vocab.keys()))
    n_new_tokens = tokenizer.add_tokens(new_tokens)
    print(f"{n_new_tokens} tokens added to tokenizer")
    model.resize_token_embeddings(len(tokenizer))

    with torch.no_grad():
        for i, token in enumerate(reversed(new_tokens), start=1):
            tokenized = tokenizer.tokenize(token) # convert into tokens 
            tokenized_ids = tokenizer.convert_tokens_to_ids(tokenized) # convert into index in the embeddings dictionary
            model.model.embed_tokens.weight[-i, :] = model.model.embed_tokens.weight[tokenized_ids].mean(axis=0)

And with that you’re done with the data portion of the process! Now it’s just the fine-tuning of the model.

Fine-tuning with PEFT Adapters

Because we want the LLM to account for the changes in the Tokenizer and Embedding layer, we need to fine-tune it with datasets that include the new words. Basically teaching the LLM how to align these new vectors with the expected tasks.

For that we just have to perform a standard fine-tuning process using the transformers package.

# PEFT configurations
lora_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"],
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)

# Creating a PEFT model - Attaching PEFT configs to the base_model
base_model = prepare_model_for_kbit_training(base_model)
base_model = get_peft_model(base_model, lora_config)

# Fine-tuning configurations
training_args = TrainingArguments(
    output_dir="output-model",
    auto_find_batch_size=True,
    learning_rate=1e-3,
    num_train_epochs=1,
    logging_dir="logs",
    logging_strategy="steps",
    logging_steps=500,
    warmup_ratio=0.03,
    lr_scheduler_type="constant",
)

# Setting up the actual Trainer
trainer = SFTTrainer(
    model=base_model,
    tokenizer=tokenizer,
    train_dataset=subset_dataset,
    peft_config=lora_config,
    packing=True,
    max_seq_length=2048,
    formatting_func=format_data,
    args=training_args
)

# train
trainer.train() 

# save model
trainer.save_model()

Some key things to note with the training process for LLMs are:

1. We need to specify the task_type in the PEFT configuration, if not it will raise and error saying the training can’t proceed without the correct configurations
2. If you want to add efficiency for fine-tuning with PEFT, you have to make sure you are using a quantised version of the base_model in order to perform QLoRA instead
3. Because we are just passing in a piece of text, the formatting_func argument is important as it formats the piece of text from the dataset before passing it into the Trainer for fine-tuning

With that, we are done! You now have a PEFT adapter and Tokenizer that has been fine-tuned for your domain specific words!

Large Language Models (LLMs) have take the world by storm with their reasoning prowess and ability to perform multiple tasks within a single prompt. However, in some cases using an LLM would be akin to bringing a cannon to a knife match. It would sure be an advantage but probably impractical. In some scenarios, using the “Medium” Language Models that have “only” a few hundred million parameters would suffice. For example, encoder models like BERT can still be used to generate embeddings for Retrieval Augmented Generation (RAG) applications. Simple tasks like sentiment analysis and or general classification scenarios still remain performant as well. As such, I wanted to leverage on the newer technologies to use on “Medium” sized models.

Parameter Efficient Fine-Tuning, or PEFT for short, is a novel way of fine-tuning a transformer model by representing the weight updates with two smaller matrices (called update matrices) through low-rank decomposition. By doing so the rest of the model weights can be ignored and therefore the number of trainable weights reduces significantly. You can think of it like the Boosting algorithm. If the base transformer is weak, you can use these update matrices to “fix” the weaknesses of the base transformer. These update matrices are called adapters.

There are many articles and papers that benchmark models fine-tuned with adapters versus traditionally fine-tuned. What is even more prevalent are the articles about PEFT for LLMs only. Annoyed by not being able to find articles about PEFT for BERT and other “Medium” sized models, here I am making my contribution.

Code

First step is always to install all the relevant packages

pip install transformers datasets evaluate accelerate

• transformers: key transformers package for training and the gateway to models available on huggingface_hub
• datasets: the gateway to datasets available on huggingface_hub
• evaluate: to provide evaluation metrics
• accelerate: needed for using Trainer on PyTorch models.

Next we import all the relevant classes from the packages

import numpy as np
import evaluate
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer

Now we bring in the model and tokenizer that we want to fine-tune

model_name = "nlptown/bert-base-multilingual-uncased-sentiment"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

Dataset preparation

dataset = load_dataset("yelp_review_full")
small_train_dataset = dataset["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = dataset["test"].shuffle(seed=42).select(range(1000))
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True, return_tensors="pt")
tokenized_train_dataset = small_train_dataset.map(tokenize_function, batched=True)
tokenized_eval_dataset = small_eval_dataset.map(tokenize_function, batched=True)

For this simple implementation, I used the yelp_review_full dataset that can be found on Huggingface Hub. Because I didn’t have the compute power for training the large dataset, I decided to only select 1000 data points for training and evaluation. What I essentially wanted to test out was whether the code worked anyway. After that we just tokenize the text in the dataset for them to be passed into the model for training.

The tokenizing process is essential as the BERT model will be looking out for either input_ids or input_embeddings. The tokenizer transforms the text data into tokens and labels the column input_ids.

Declaring the function to use for calculating the metric

metric = evaluate.load("accuracy")
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

This is where we start using the PEFT techniques. In order to make use of PEFT, we need to 1. decide what modules in the transformer architecture we want to perform the low-rank decomposition for, and 2. what task type are you fine-tuning for.

Modules simply refers to the set of matrices in the transformers architecture. If you’re familiar with the way the attention layer works, you’d know that there are 3 kind of matrix involved. The Key, Value and Query. In the research paper for PEFT, the Query and Value matrices were targeted and that is what we will be doing as well. In order to find out what the layers are called, you can simply print(model).

BertForSequenceClassification(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(105879, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(
        (0-11): 12 x BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
            (intermediate_act_fn): GELUActivation()
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=5, bias=True)
)

As you can see above, the Query and Value layers are simply called query and value.

from peft import LoraConfig, get_peft_model, TaskType

# Define LoRA Config
lora_config = LoraConfig(
 r=16,
 lora_alpha=32,
 target_modules=["query", "value"],
 lora_dropout=0.05,
 bias="none",
 task_type=TaskType.SEQ_CLS, # this is necessary
 inference_mode=True
)

# add LoRA adaptor
model = get_peft_model(model, lora_config)
model.print_trainable_parameters() # see % trainable parameters

You can read up on the other arguments over here. But just take note that the target_modules and task_type arguments are necessary.

training_args = TrainingArguments(output_dir="bert_peft_trainer")
bert_peft_trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train_dataset, # training dataset requires column input_ids
    eval_dataset=tokenized_eval_dataset,
    compute_metrics=compute_metrics,
)
bert_peft_trainer.train()

Thereafter, the training process is similar to what you can find in the guides on Huggingface. Voila! Your model has been fine-tuned with PEFT.

If you save the model with bert_peft_trainer.model.save_pretrained("bert-peft" you will see that only the adapter files are saved. Which should be a config file in .json and a model file in .bin.

You can even merge the adapater into the original model! Just a note that the adapter can only be merged with the original model; and not the quantised version. I say this because it is possible to quantise a model and use the quantised version of a model to train the adapter. If you’re interested, go take a look at QLoRA.

In order to merge the models, simply follow this code

from peft import PeftModel
original_model = AutoModelForSequenceClassification.from_pretrained(
  model.config["_name_or_path"]
)
original_with_adapter = PeftModel.from_pretrained(
  original_model, "bert-peft" # bert-peft; the folder of the saved adapter
)
merged_model = original_with_adapter.merge_and_unload()
merged_model.save_pretrained("merged-model")

And there you go. The files located in the folder merged-model will be the model file of the original model integrated with the weights from the adapter.

It is very easy to get distracted by the bells and whistles of data science; state of the art models with jaw dropping results or perhaps a large, open-sourced and perfectly annotated dataset. But how often do you usually think of the usability limitations of these models and datasets?

My transition from a data science enthusiast to a data science professional has drawn my attention and curiosity towards this question more often than not. Now that I am working in a commercial setting, the limitations of using certain models and datasets are more prevalent than ever. This is because each model and dataset are usually licensed in a way that does not allow for commercial development or has some kind of caveat if you do so.

Now that I’ve done some research on software licenses for myself, I would like to write an article to share what I’ve learnt. Especially about the things that I didn’t manage to find very easily. Just a disclaimer: Most, if not all, of the information about these licenses stem from copyright laws. And I am not a lawyer, so if you need more help understanding the nitty gritty of these laws and licenses, please contact a one.

Software Licences

Let’s start with the big picture. All of this begins with intellectual property and how it is protected. Intellectual property refers to creations of the mind; meaning to say anything that is a result of your intellect. A common example of intellectual property are songs that are imagined, written and brought to life by songwriters. Just like how a song is the intellectual property of a songwriter, a piece of software made up of code is the intellectual property of the coder.

To make sure that credit goes to the creator instead of copycats, there are various legal instruments that can be used to protect intellectual property. And this is where software licenses come in. Software licenses are one of the legal instruments that protect intellectual property and they do so by governing the use and redistribution of software.

There are 2 common categories, and therefore kinds of licenses, for software under copyright laws: Free and Open-source Software (FOSS) and Proprietary Software. With the conceptual differences between the licenses in these 2 categories being the rights granted to modify and re-use the software. Which essentially means that if the source code for a piece of software can be publicly found, then that means the software is free and open-sourced; and you are able to copy the code and edit it any way you see fit. However, if the source code of the software cannot be publicly found, then the software should be proprietary. So, to ensure that you do not get sued, please stick to FOSS.

There are many other types and definitions of software licenses, but for this article I will just be focusing on licenses from these 2 categories.

Free and Open-source Software (FOSS)

Let me start by addressing the fundamentals of FOSS; what is considered open-sourced and free? For the sake of this article, let’s go with these definitions. The term ‘open-sourced’ means there are non-intrusive restrictions on your software for redistribution and exposure of the source code. Therefore, if you see a piece of software under any of these licenses, they are obligated to share the source code of the software to the public. If you want to check out what it takes for a software to be considered open-sourced in its entirety, you can take a look here.

The term ‘free’ here does not refer to free in the financial sense, but the freedom to use the software without restrictions. This is why FOSS is what I am usually looking at as they grant me the flexibility of trying out various data science solutions.

However, there is a problem here. What if someone modifies the original source code and distributes this derivation and further derivations of this software with distribution restrictions? This would create increasingly intrusive restrictions for redistribution of FOSS. Think about it this way, if everyone decides for their derivation of the original software to be proprietary and set distribution restrictions, the number of FOSS would never rise. And we will be living in a world with that much more restriction on FOSS. On the other hand, if they are not allowed to use the software freely this way, then wouldn’t that go against what the definition of ‘free’ is?

This is why there are 2 main types of licenses for FOSS. These 2 different types of licenses aim to tackle either of the problems mentioned above, and they are:

1. Copyleft Licenses, and
2. Permissive Licenses

Copyleft Licenses

These types of licenses are geared towards those that define ‘free’ as freedom to redistribute the software. Licensing your software with a copyleft license will ensure that any derivative of your software will also be redistributed under the same conditions as your own. For example, if you distribute your software under a copyleft license with the sole condition of crediting the original author, then any derivation of your software will also have to be licensed under a copyleft license with the same sole condition. This therefore ensures a healthy growth of open-sourced software.

Freedom to distribute can also mean that you are free to distribute the software any way you want. Even for a fee. To get it out of the way, YES, FOSS can be used for commercial purposes. BUT the caveat is that nobody can stop your customer from distributing your software for free and therefore rendering your business strategy useless. This is especially the case for copyleft licences as the core of this license is about the freedom to redistribute and accessibility to software. Things are a little different with permissive licenses though.

Permissive Licenses

These types of licenses are geared towards those that define ‘free’ as freedom to use the software. This means that anybody would be able to make edits or add to the source code and thereafter distribute this edited code however they wish, even as a commercial software. Therefore allowing anyone to sell their derivation of the software without being afraid of customers giving the software away for free. This does not solve the increasing restrictions for redistributions but it ensures that anyone can freely use the software for whatever purpose that they desire. The only caveat for this is the need to give credit back to the original creator. What this means is that commercial software, or software made for commercial purposes, do not have to be proprietary software. Although it might be easy to see proprietary software to be synonymous with commercial software, it is definitely not the case.

Derivatives of software with permissive licenses can therefore be seen as independent of the original. Which is perfect for data scientists or developers as it allows them to build upon the software and either create something that is uniquely theirs or to improve on the software itself. And once they do this, they are free to profit from it by licensing their creation as a proprietary software or even allow others to leverage on their improvements by licensing their creation under a copyleft or permissive license.

There are a ton of different licenses that are considered either copyleft or permissive. Simply looking at the dropdown menu on Github already shows you some examples. Each of these licenses would obviously also have different features but are generally still funnelled into either of the 2 aforementioned license types.

After understanding software licenses as a whole, the next logical step would be to look into the specifics of a license. Either to find out what license to place your code under or to investigate the license of a dataset or model you are interested in. For the specificities of these licenses, I would recommend checking their official websites or other articles that can be found more easily with a simple search.

Proprietary Software

In contrast with FOSS, proprietary software simply means that the owner of the piece of software has exclusive rights to the source code and is allowed to restrict use, modification and redistribution of the software. This simplifies every single question we’ve been asking because everything is up to the proprietor to decide. The reason why proprietary software is perceived synonymously with commercial software is because they are typically available for a fee. An example would be Adobe’s software; where one would have to pay a monthly fee to be able to use the software. Having said that, there is nothing stopping the proprietor from releasing the software to the world for free too and maybe all you need to do is ask nicely 🙂

Conclusion

Just some of my thoughts and reflections after absorbing all of this information. There is nothing much else about proprietary software and that is why as a data scientist, I am more concerned and interested in the realm FOSS. The topic of freedom and what it means to be free has always been consistently and philosophically discussed and for good reason. The same thing is done for FOSS with many people discussing on what free software should actually mean. I think it is difficult to pick an objectively better definition of free software. As much as permissive licenses (free to use) would help data scientists improve on commercial softwares they are working on, copyleft licenses (free to redistribute) are effective in proliferating and improving on data science technologies. And thus I see an importance in balancing between these 2 types of softwares in the industry for the sake of business and technological growth.

I think it is also interesting to see how certain kinds of licenses have the power to enable and empower but also inhibit and restrict certain business and industries; which gives me a sense of appreciation to the creators of these legal instruments. A fun fact is that you do not necessarily have to declare what license you want to be protected under. As long as you commit your idea on paper (or any other form) you are automatically protected under copyright laws.

If you want to know more about software licenses and the specifics of each of the licenses, here are some articles that I found rather useful.

https://towardsdatascience.com/a-data-scientists-guide-to-open-source-licensing-c70d5fe42079

https://towardsdatascience.com/the-6-common-open-source-licenses-and-what-they-mean-715d434c2109

Copyright Licences in the World of Data Science was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Natural Language Processing (NLP) has been my field of interest throughout my journey practising data science, from school work to internship to my full-time career. It has always been intriguing and fascinating to read the latest and greatest NLP technologies like transformers. However, one thing that remains as cornerstone in NLP is text processing.

I have actually written a medium article that outlines some of the usual processes such as lemmatizing and stemming, and the removal of stopwords and punctuations. If you want to find out more about it, here is a link to the article I am talking about. Albeit there are also countless of other articles that have already gone through them before. Which brings me to the reason why I am writing this article. I would like to share 2 text processing functions that I felt were useful throughout my data science journey and that I have not found much literature on.

They are Hashtag Processing and Gibberish Detection.

Hashtag Processing

Back when I was working at a MarTech startup that focused on influencers, I had to deal with a lot of data from social media. This meant that text data would be much messier and less structured as compared to news articles or research papers. And given that most of these social media text are personalised, they also differ slightly to text like Amazon reviews. With social media data, there would be a lot more emojis, mentions, slangs and hashtags.

Hashtags are definitely one of the more problematic components of the text. It would be easy to remove emojis and mentions, or replace slangs. However, even if the hashtag symbol is removed, it still leaves a combination of characters with no whitespaces in between. As most text processing tokenizes words in a sentence through the separation of white spaces

"This is a sentence" -> ["This", "is", "a" "sentence"]

Simply removing the hashtag symbol would result in the character combination being considered a token itself:

"foodislife" -> ["foodislife"]

# instead of

"foodislife" -> ["food", "is", "life"]

Hence, a hashtag processing function that would be able to split up this combination into their individual words.

Here is the code in full:

How It Works

Based on a corpus, we generate a dictionary of word counts. So we literally count the occurrence of each word in the corpus. From this, we iterate character by character through the combination of letters and at each iteration, we calculate the probability of it being a word using the dictionary.

The probability is calculated based on how likely that window of combination of letters is to be a word and also the probability of the starting index of the window being a good starting point to be a word.

Through this method, at each of the possible splits in the combination of letters we would have the probability of that split forming a word. The algorithm will then keep track of the indexes where the probabilities are the highest and split the combination of letters at those points to form a sentence

'ilovefood'

# iteration 1:
'i'
## at 'i', since it's such a common word, it would have a high probability
## hence, we consider it a word

# iteration 2:
'l'
## 'l' is not common by itself and hence would have a low probability of it being a word

# iteration 3:
'lo'
## 'lo' is also not common.

# and so on.. until we split at 
'i', 'love', 'food'

Note that the word_dict.json file came from the generate_word_dictionary function above. I simply exported the dictionary into a .json file. And for your convenience, here is a function that exports the dictionary into a .json file:

import json

def export_dict_to_json(dictionary: dict, file_name: str):
		with open(file_name, 'w') as file:
				json.dump(dictionary, file)

Gibberish Detection

This was not all around as useful as the hashtag function, however I genuinely found this interesting and to be extremely useful when it is required. I found this function especially useful to use before performing data exploration. Reason for this is because some sentences in a dataset might have extremely long gibberish in them. This ultimately skews the metrics of the sentence and dataset and makes it difficult to truly understand the dataset.

Before I explain how the algorithm works, let me show the full code:

The gibberish detection function works similarly to the hashtag processing function. In a sense that both of them makes use of probabilities. However, instead of getting the probability of words, the gibberish detection function gets the probability of a letter following another letter. Example, the probability of ‘a’ being after ‘b’. Or the probability of the letter ‘z’ being after ‘k’.

How It Works

From a corpus, the generate_gibberish_model would read each sentence and calculate the occurrence between 2 letters.

'this is not gibberish'

't' -> 'h'
'h' -> 'i'
'i' -> 's'
's' -> ' '
' ' -> 'i'
...
'b' -> 'e'
'e' -> 'r'
'r' -> 'i'
'i' -> 's'
's' -> 'h'

This would generate counts of the occurrence between 2 letters. The dictionary output of this function would then be used in the function to calculate how likely a string of characters would be gibberish or not. Where a higher probability shows that the string of characters are likely and possible, whereas a low probability would show that the combination of characters make no sense and is gibberish.

The way this is done is by iterating through a string of characters, retrieving the probability of each of the 2 letters occurring and then summing up all the probabilities together to get a single probability value. As seen in the code, log probabilities are used instead of straight probabilities so as to avoid numeric underflow issues that could arise from long texts.

This method will return a probability value regardless of whether the token is gibberish or not. Hence, there is a need to declare a threshold value to determine at what probability value can we be confident that the token is gibberish. In the code above I used 0.0188 . Any value can be used, but let me explain how I got this value.

I had a few examples of good text and gibberish text. I then used the gibberish detection function on all the good and gibberish texts and found a value that sits between the minimum threshold to be considered non-gibberish and the maximum threshold to be considered gibberish. And for me, that value is 0.0188 .

Conclusion

The logic for these functions are pretty simple but yield great performance when it comes to text processing. I hope that these functions manage to help someone as much as it had helped me and is currently helping me in becoming a better data scientist. And if any of you have come across any interesting text processing methods/functions/pages do leave them in the comments below! Thanks for reading!

Text Processing Tools I Wish I Knew Earlier was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Honestly, I have never thought that I would grab the opportunity to work in a startup. I have always been a pretty conservative person and usually enjoy stability and consistency in my life. Thinking that I would be the kind that would stick to larger companies throughout my career paths; from internships to full-time employment. And when everyone around me was talking about the fast-paced and hectic nature of startups, I thought to myself, “wow that sounds pretty intense. guess I’ll stick to my SMEs and MNCs”.

However, Affable.ai changed my mind.

I had made the decision to graduate a semester early due to COVID-19. With a surplus of time than I had anticipated, I decided to look for something interesting on the job market. That is when I decided to give startups a shot and with the help of SGInnovate, I got into Affable.ai.

If you are interested in finding out what they do, go ahead and check out their website! If not, here is a quick summary of who they are and what they do. Affable.ai is a deep-tech startup in the influencer marketing scene. They created a powerful data-driven platform to help companies find relevant influencers, manage campaigns and more! All in all, they enable and empower companies to make more well-informed decisions on their next influencer marketing campaigns.

Startup vs. Non-startup

The first thing that stood out to me when I first entered the company was the fact that I knew what everyone in the team looked like and what their names were. Albeit something simple, that was not something that I experienced in my previous internships. It was usually normal to know everyone in your team or department, but knowing that I knew everyone in the company was just something different for me. A good kind of different though. In an odd way, things felt a lot more personal. It was as if there was a blanket of shared responsibility on everyone. Which made me feel more driven to perform and much more prideful in my work.

Something that came along with being more prideful of my work was how much more impactful and important my work felt. In terms of scale and proportion, my contributions felt more recognised at a startup. It was something I anticipated before entering a startup but it is a whole other feeling when you see something that you worked on getting deployed onto your company’s core product. I had not thought about how the flat management hierarchy correlated with the degree of impact.

I mentioned above regarding my impression of startups and how they are fast-paced and hectic. However, I feel that non-startups are also as fast-paced albeit in a different way. Bigger companies tend to have more red tapes, and for good reason. But this translates to having more requirements that need to be satisfied before getting something done. Which consequently means that there is usually a lot of preparation that needs to be done before moving on from checkpoint to checkpoint in a project timeline. Thus giving the sense of rush within a project. On the flip side, startups are fast-paced because they are able to clear checkpoints in a project timeline a lot quicker. Due to the flatter hierarchy, a simple slack message or call with my supervisor or someone from the management team would have done the trick. I would have my strategy for my task at each checkpoint evaluated and simply move on.

Importance of Supervisors

The invaluable experience that I got from my stint at Affable could almost solely be contributed by my supervisor. Teaching and imparting knowledge is not the easiest thing to do. And it is also not easy for someone who already has a part to play in the company to take time off to teach. I am therefore extremely thankful that things worked out the way it did.

What I enjoyed about the learning process during my stint is that most of the shortcomings that I had were quickly addressed with constructive feedback. Not only that but there was an understanding that it is okay to not fix myself instantly. This really allowed me to reflect on my actions better and come to a better understanding of why and how I can do things better. Before this, I would have felt so much pressure to change myself immediately to prevent making the same mistake. I would probably also have felt pretty affected if I did let one of these mistakes happen again. This entire cycle was not healthy and just generally stressful. Something else that I also enjoyed was how effectively feedback was taken. Being a curious soul, I had wanted to find out more about what my other colleagues were doing and what they have learnt from their projects. As I told my supervisor about this, she quickly arranged for weekly ‘Knowledge Sharing’ sessions for the data science team. Truly fantastic! And within that short stint, I am sure that the way I code and think of coding, and problem-solving has changed dramatically. I would even say that my thinking has been fine-tuned (pun intended) to more real-world data science problems.

After going through all of that, I started reflecting on the importance of supervisors and mentors. The first thing that came to mind were those career aspect ranking forms and how I would now rank a good supervisor over the brand name of the company. Although I believe that are both aspects important in their own ways, I just feel that having a good supervisor would be better in the current stage of my career path. This led me to consider how to ensure that my learning experience will be in good hands. And challenged me to include supervisor assessment questions in my full-time job interviews.

Remote Working

Most companies over the globe adopted remote working when the pandemic hit. Affable was no exception. Although this experience is not something that is unique to me, it is still a first for me. The benefits of remote working hit me first. I did not have to spend time travelling to and fro an office. Freeing up a significant amount of time to work out or pursue my hobbies. And I would have a working space that I am definitely comfortable with. I was excited about work! As time went on, these benefits proved to be helpful in ensuring work-life balance for me. Of course, coupled with good management, I was able to properly distinguish time for work and time for life. Great as things seemed to be, there were some experiences that I did not anticipate to be as notable as I expected.

Getting too comfortable with some of my work habits is one of them. One of my work habits involves being persistent at trying out a particular solution. Usually, when working under some pressure, I would feel the need to iterate through solutions more quickly. Instead of harping over small details of a solution that might not lead to effective gains. However, as I got more comfortable working, the intervals for trying out solutions slowly extended. This was something that I definitely did not expect to have noticed.

Something else that most people might resonate with is the socialising aspect of remote working. I already expected communication via Zoom to be different from a physical meet up. Especially after reading so many comments about this particular aspect on forums like Reddit, I did not hesitate but to manage my expectations for it. But after experiencing it first hand, it is evident that there is something about physical meet ups that helps with the chemistry.

Having said all of this, the reason why this aspect is notable is because of the way I experienced it at Affable. While I was there, I brought up wanting to get to know my colleagues beyond their job title. My supervisor then acted on this feedback and set up weekly ‘Get-to-Know’ sessions. Each session involving my team and someone from a different department in the company. We were then able to learn more about them as individuals and even take life and professional advice from them. What made these sessions extra interesting was the cultural diversity of the company. With quite a few employees coming from various parts of the world, talking to them about their culture and way of life was always exciting. In some sense, getting to know my colleagues felt like a mini getaway from Singapore.

Conclusion

Out of all my thoughts, something that I personally enjoyed out of the whole experience was how quick feedback was exchanged. As a fresh graduate, I have a lot to learn about my industry and what it is actually like to be in a professional working environment. With this fast exchange in feedback, I found myself iterating my working habits and perceptions towards greater standards much more quickly. This is definitely also coupled with the fact that I had a mentor that had vast experience in the field that I was in.

On the other hand, something that I found hard to adjust to was socialising online. It is not that Affable did not do enough to improve the online socialising experience. But rather, it is hard to replace is physical meetups and networking sessions. It is also a personal preference of mine to be able to connect with people face to face (physically).

All in all, fulfilling is the best word I can think of to describe my experience at a startup. I took away so many professional and personal experiences from this stint. Even reflecting on them after it has come to an end and learning new things about myself and what I want for my life and career as a result of this. I am incredibly thankful for the opportunity to be part of an amazing team in Affable.ai and I have no doubt that they will be very successful in the future. If you are interested in influencer marketing in any way, please do check them out! I am fulfilled and looking forward to my next experience as a data scientist. Onwards and upwards!

Preface

Worldwide lockdowns and prevalent news reports of America and Hong Kong’s civil unrest has got me thinking a lot about the freedom we have as citizens of a country, or even so, as a human being. What does it mean to truly be free, how is freedom measured or what even is freedom? With all these kinds of questions in my head and a soft spot for a freedom-themed dystopian film, I decided to delve into the film with a more critical perspective and pen down my thoughts on Freedom in The Matrix.

Introduction

For those unfamiliar, The Matrix (1999) is a dystopian film where sentient machines, or simply called the Machines, have overpowered the human-race and captured humans to harness their heat energy and bioelectricity. While their physical bodies are used as batteries to fuel the Machines, the minds of the humans are injected into a program created by the Machines called the Matrix. In the Matrix, the consciousness of the captured humans mentally projects themselves into a simulated reality. From the perspectives of the people’s consciousness, they are living a simulated life of normalcy when in actuality, the lives that they live are bounded by the Matrix and masked from the truth that there is a reality outside the simulated world that they reside in. The main mission in The Matrix is for the protagonist, Neo, to liberate humanity from the Matrix and fulfil his destiny of being the Messiah for mankind.

Although liberation from the Matrix is evidently seen as a form of objective freedom, the perception of a normal life to a captured human begs the question — Are you considered to be free if you are unaware that you are trapped? As objective freedom is said to be when a man is conscious of his own freedom (Shastri, 1921), this hints at the possibility of the portrayal of various other forms of freedom in the film with potentially subjective natures. Subjective freedom refers to having freedom of varying degrees instead of absolute. This is further reinforced by the nature of the film’s storyline. Unlike storylines of other films, where only 1 reality is depicted throughout the whole film, The Matrix’s storyline portrays 2 realities, and hence further makes us question the nature of freedom in the film.

In this article, I will show that the film contains multiple forms of freedom and their objective essence and further outline how the Matrix serves as a platform to reveal some subjectivity to them.

Forms of Freedom

Freedom from the Matrix

From the viewer’s perspective, the only form of freedom depicted in the film would obviously be freedom from the Matrix. It seems obvious as the plot of the entire film revolves around Neo being liberated from the Matrix and his objective to unveil the truth to all those in the Matrix and consequently emancipating them. However, what seems to be the main indicator for this form of freedom would be Neo’s desire to understand the Matrix and Morpheus’s desire to free Neo. It is these desires that indicate that there is something to be free from and hence, showing that there is a distinct form of freedom. In the film, these desires would lead Neo to pick the red pill which allowed him to be liberated by Morpheus and subsequently reveal to the audience, for the first time, the reality outside the Matrix. In this aspect, one is either free from the Matrix or not.

Freedom from Algorithm

Looking deeper into the film, we see other indicators that point towards other forms of freedom. Firstly, a program designed by the Machines to safeguard the Matrix, called Agent Smith, mentions in his interrogation scene with Morpheus that he has desires to leave the Matrix — ‘I hate this place… I have to get out of here’. However, it seems there is an irony here, given that Agent Smith was created for the Matrix and that a program is not expected to possess the ability to desire. Lodén (2014, 35) even states that it is human nature to have desires for liberation. Explication of this irony revealed that a humanisation process occurs to Agent Smith’s throughout the film, which accounts for his desire for freedom. This desire then serves as an indicator for a different form of freedom.

Agent Smith’s humanisation moment can be seen through his expression of hatred in the same interrogation scene, where his explicit frustration and unhappiness with the Matrix and its inhabitants are shown in this scene serves as a form of emotional growth that is unexpected of any computer program. This is contrasted with earlier scenes in the film that depicts Agent Smith as a mere program, created to maintain order in the Matrix. Even the way he approaches situations is similar to that of a computer program — Logical and practical, with the goal in mind. Hence, revealing a process of humanisation in the character.

With further analysis on this Agent Smith’s human-like desire, it can be deduced that it would not make sense for him to desire freedom from the Matrix since, unlike humans, he is native to it and it is already a reality to him. Hence, it would seem that Agent Smith refers to something else when he talks about being free from the Matrix. What Agent Smith could have meant by getting out of the Matrix could be him escaping the algorithm that he has been programmed with. The scene where Agent Smith expresses his hatred for the Matrix — “I hate this place. This zoo. This prison. I can’t stand it any longer” — can be interpreted as his expression of distaste towards his tiresome and repetitive attempts to achieve the innate objective that he has been programmed with. Hence, it would seem that Agent Smith truly desires to break free from the set algorithm that designates his purpose in the Matrix and that this desire stems from his ostensibly many years of maintaining the Matrix to the point where he starts to develop a hatred for it. In this aspect, one is either free from the rules and programming of the Matrix or not.

Freedom from Reality

Secondly, scenes of a traitor from within Morpheus’s resistance, called Cypher, seem to hint towards another form of freedom. In a scene where Cypher feasts on a steak while negotiating with Agent Smith back in the Matrix, Cypher acknowledges that he knows his meal isn’t real but implied that all that mattered was that it tastes or more generically ‘feels’ real. This indicates that Cypher’s perspective of freedom would require one to actually feel free, instead of knowing he’s free. With the key indicator being the rationale behind his betrayal.

The betrayal scene where he almost managed to kill the rest of the crew revealed his desire to re-enter the Matrix as it ‘feels’ more real than reality itself — “You call this free? If I had to choose between that and the Matrix. I choose the Matrix”. Although having experienced both realities, in and out of the Matrix, Cypher perceives freedom as being back in the Matrix. This ensues from him being under constant command by Morpheus — “All I do is what he tells me to do” — and being exposed to the dismal environment in reality. His perception of freedom is particularly interesting as it is anti-praxis to the obvious form of freedom — freedom from the Matrix. Due to his human nature, Cypher is assumed to have a similar perception of freedom as Neo and Morpheus. Hence, this elicits viewers’ expectations for him to conform to the obvious form of freedom, however, the film depicts otherwise. In this aspect, it is rather interesting as it’s about whether one feels free, or not.

“You call this free? If I had to choose between that and the Matrix. I choose the Matrix” — Cypher, The Matrix.

We then conclude that Agent Smith’s humanisation and Cypher’s betrayal are the film’s indicators for other forms of freedom, such as freedom from algorithm and freedom from reality. In the next segment of the article, I will discuss how these forms of freedom could also be viewed as subjective.

Subjective Freedom

Even though these perspectives of freedom are dissimilar, their distinctions are blurred when we take into account the ideas of freedom from the philosophers’ Gravel (2007, p3) and Kmiec (2005, p35). We will come to realise that these objective forms of freedom can also be seen as subjective. In a nutshell, the definitions Gravel (2007, p3) and Kmiec (2005, p35) proposes adopt the view of freedom being subjective. This implies that both philosophers believe in the idea that freedom is not absolute but instead a scale where one individual may have a higher degree of freedom compared to another; one definition to fit all scenarios. Gravel (2007, p3) asserts that the degree of freedom of an entity is framed by the constraint of its choices. In other words, any being has a certain degree of freedom, but what that degree is, is constituted by the limit of its choices that’s been subjected to certain constraints. These constraints can be physical (e.g. I do not have the option to give birth), economical (e.g. I cannot afford it) or even legal (e.g. I am not allowed to do so). In a sense, you can say that Gravel believes that your freedom is based on what you think and are actually able to do.

Whereas, Kmiec (2005, p35) affirms that freedom is dependent on the natural laws of the individual. He believes that each individual entity has its own set of natural laws to abide, and in any complex society, such as in our reality or in the ‘Matrix’, the freedom of the entity is bounded by what the natural law dictates. Kmiec (2005, p35) explains that the natural laws are rules that guide what individuals inherently feel they are allowed and capable of doing, and that each individual’s existence is dependent on its conformity to this law. Now we show how these forms of freedom interact with the Matrix to unveil these perspectives of subjectivity.

Freedom from the Matrix

Freedom from the Matrix can be seen as subjective when the Matrix is seen as a form of constraint over the choices of those entrapped. Moreover, it is also evident that natural laws differ between simulated reality and actual reality. This difference is shown in the last scene where Neo flies into the sky, showing the full control he has over the Matrix. Hence, showing that the natural laws are different for Neo in the Matrix and in reality. Simply taking being in the Matrix as a reference, humans from the real world are shown to have a higher degree of freedom as their choices are not as restrained compared to those that are not from the real world. Reinforcing this point is when Morpheus performs insane stunts such as jumping from building to building or even wearing sunglasses that should have obviously fallen off, within the Matrix. This, of course, is in comparison with the humans in the Matrix. The choices of these humans are restricted by the programming of the Matrix itself, thus implying a lower degree of freedom. This is seen through Agent Smith’s ability to just take any human in the Matrix and morph them into himself. In a way, this depicts how even the existence of the human can easily be altered by a program in the Matrix and hence, a restrain over their choices. Morpheus’s dialogue to Neo about the Matrix — “I realize the obviousness of the truth. What is the Matrix? Control” — further emphasises on the fact that the Matrix is a form of control that would consequently limit the choices of those that reside in it. Hence, the humans in the Matrix simply possess a lower degree of freedom than the humans in reality.

“I realize the obviousness of the truth. What is the Matrix? Control” — Morpheus, The Matrix.

Furthermore, taking into account Kmiec (2005, p35) this time, the natural laws of the simulated world are set by the algorithms of the Matrix. Hence, those that reside in the Matrix are said to be free as long as they are obedient to the natural laws. However, since the Matrix can be interpreted as a restricted reality, it can be said that the entrapped humans in the Matrix possess a lower degree of freedom than those that are not trapped. Therefore, not only have we shown that freedom from the Matrix can also be understood as subjective, but we have also seen how the Matrix is responsible for this revelation.

Freedom from algorithm

The humanisation of Agent Smith has uncovered another form of freedom — freedom from the algorithm. Similar to the aforementioned form of freedom, this can also be seen as subjective when some set objective constrains the choices of an individual. Since Agent Smith is created for the Matrix, he would be bounded by his own programming and therefore would not be able to do other than what he is programmed to do. It can then be said that Agent Smith’s life is causally deterministic due to his own algorithms. This means that he is not performing actions at his own will, but instead, it is because he has been programmed to do so. Agent Smith’s algorithms that programmed him to achieve a particular objective clearly shows that he does not possess as high of a degree of freedom as other human characters in reality, such as Morpheus or Neo.

Morpheus and Neo, unlike Agent Smith, are seen to have flexibility in terms of their objectives. Where Agent Smith only does things with a set objective in mind, Morpheus and Neo’s objectives are changed depending on the scene they are in. In this regard, the film consistently portrays Agent Smith’s vain attempts to achieve his objective. At the start, Agent Smith and his agents attempt to capture Trinity, one of Morpheus’s comrades, but fails as she manages to slip away. Next, Agent Smith corners and manages to get hold of Neo at his workplace. Agent Smith interrogates Neo and places a tracker on him, ostensibly knowing that he would meet Morpheus. Yet again, Agent Smith’s plans are foiled as Trinity extracts the tracker out of Neo before meeting Morpheus. In another scene, even after getting hold of Morpheus, his plans are once again ruined when Neo rescues Morpheus. Here, we notice that there is no room for a change in Agent Smith’s objective. Conversely, in one scene, Morpheus and Neo’s objective was to meet the Oracle, a female program with clairvoyant abilities. However, when Agent Smith manages to get hold of Morpheus, Neo’s objective is changed to saving Morpheus instead — therefore showing that there is fluidity in his objectives.

The way the algorithm works on a program is similar to that of how natural law works on each individual. An algorithm for a computer program dictates what action a program can or cannot perform. Hence, Agent Smith is seen as free within the means of his programming. However, as mentioned above, since this algorithm puts him in a causally deterministic state, he is then not able to do anything other than what he is programmed to do, unlike humans in reality. Hence, Agent Smith possesses a lower degree of freedom as compared to the other characters because of the programming of the Matrix.

Freedom from Reality

The ironic and unexpected actions of Cypher draw attention to a form of freedom that’s antithetical to the obvious form of freedom — freedom from the Matrix. Although Morpheus does not explicitly mention that freedom is being free from the Matrix, he does talk about the truth that there is a reality outside the simulated reality. Kmiec (2005, p35) asserts that freedom cannot be achieved with false knowledge. Following on the topic of truth and freedom, the truth does not define what freedom is, but rather it provides information that would alter individuals’ perspectives of what the definition of freedom should be.

“I think the Matrix can be more real than this world” — Cypher, The Matrix.

Even though Morpheus and Cypher are in the same reality, after realising the truth, Cypher started perceiving freedom as being back in the Matrix. This is due to the difference in how both, Morpheus and Cypher, value freedom. In reality, although both Morpheus and Cypher are in the same environment, they are in different hierarchical positions. Morpheus’s superiority over Cypher becomes a push factor for Cypher to escape from reality. Additionally, as previously mentioned, Cypher also expresses that what he ‘feels’ in the Matrix is more real than reality– “I think the Matrix can be more real than this world”. This ‘feeling’, Cypher mentions, that is absent in reality is seen as the pull factor for him to escape into the Matrix. This push-pull factors not only account for the existence of a form of freedom — freedom from reality — but it also serves as evidence of Cypher’s choice constraint. Being under constant command by Morpheus, Cypher does not have much of a choice at what he can or cannot do in reality. Reinforcing this is the difference in material goods in the Matrix and reality. Back in the Matrix, Cypher is able to enjoy steak, and possibly other luxury dishes. While in reality, some sort of wheat-based meal is served to him instead. The dichotomy in the standards and variety of material goods between the Matrix and reality clearly shows the lack of choices Cypher possesses. Hence, it can be said that Cypher had freedom in reality, albeit to a much lower degree when compared to Morpheus. Moreover, it is this low degree of freedom that motivated him to re-enter the Matrix so as to achieve a higher degree of freedom.

Shifting over to Kmiec’s (2005, p35) perspective of freedom, there does not seem to be a difference in the natural laws between Morpheus and Cypher. Although unable to account for the subjectivity of this form of freedom, Kmiec (2005, p35) does provide an understanding of Cypher’s demise in the film. According to Kmiec (2005, p35), the existence of an individual is determined by its conformity to its own natural laws. Putting this into context, Kmiec (2005, p35) point suggests that Cypher’s demise was due to his attempt to be in the Matrix, skewing away from what his natural law dictates for him.

Conclusion

In summary, we have seen that there exist indicators in the film that point towards depictions of various forms of freedom — namely, freedom from the Matrix, freedom from algorithm and freedom from reality. Freedom from the Matrix is seen throughout the main plot of the film itself. Whereas, freedom from algorithm is seen through Agent Smith’s unexpected process of humanisation. Lastly, freedom from reality is seen through Cypher’s surprising act of betrayal and desire to feel free.

Moreover, this essay further discussed how the subjectivity of these forms of freedom can be taken into account by philosophers Kmiec (2005, p35) and Gravel (2007, p3). As seen above, where their views on the nature of freedom are consistent with the forms of freedom depicted in the film. More interestingly, it can also be seen as to how the Matrix was used to uncover the subjectivity of the mentioned forms of freedom and blur the distinction between them.

The explication of the various forms of freedom in the film and their subjectivity further uncovers interesting observations. The humanisation process of Agent Smith that allowed the finding of a form of freedom begs the question — is this humanisation part of his algorithm, or is it the product of a sentient AI program? Additionally, with respect to the film’s trilogy, there seems to be a larger governing force — the trilogy’s main antagonist, the Machine’s hive mind — constraining freedom, even after achieving all the forms of freedom mentioned in this essay. This then motivates the question — When will one actually attain true freedom?

References:

Gravel, N. (2007). What is freedom?. Retrieved from http://www.vcharite.univ-mrs.fr/pp/Gravel/surveyPeil2.pdf

Kmiec, D. W. (2005). The Human Nature of Freedom and Identity-We Hold More than Random Thoughts. Harv. JL & Pub. Pol’y, 29, 33.

Lodén, T. (2014). Human nature, freedom and dignity in China and Europe. International Communication Of Chinese Culture, 1(1–2), 35–45. http://dx.doi.org/10.1007/s40636-014-0004-8

Shastri, P. D. (1921). Objective freedom. International Journal of Ethics, 31(3), 303–306.

The Matrix. (1999). Hollywood

Introduction

A month back, my friend and I embarked on a project to automate the question tagging process for Seedly. With the original tagging process being a manual task, creating an automated tagging system would definitely help save a ton of time and manpower. On top of the obvious perks of automation, this auto-tagging system also improves the user experience through reducing the time and effort taken for users to tag the questions, and it would do so with greater accuracy. Fast forward to today, our auto-tagging system is completed and has achieved an F1-Score of 90%! Although Seedly has yet to officially deploy our system on their website, I would like to outline some of my reflections and thought processes throughout this project.

This article will outline my thoughts and findings from these 4 main pointers:

1. Multiple Labels

2. Objective and Subjective Labels

3. Imbalance of Labels

4. Ease of Deployment

Multiple Labels

Dealing with multiple labels for each datapoint is not common in my academic journey of being a data scientist. Just to get the jargon out of the way, in terms of a machine learning classification problem, there are mainly 2 kinds scenarios — multi-class and multi-label. A multi-class problem means classifying something into a single class, or category, out of multiple classes, or categories. Whereas, a multi-label problem means being able to classify something into more than one category at a time, out of multiple categories. Seedly’s Question and Answer (Q&A) page has over 30 categories and each question can be labelled as more than 1 category at a time. This directly places this project into multi-label territory! However, there were quite a few occurrences while working with a multi-label scenario.

Firstly, being a multi-label scenario imposes a restriction on the machine learning models that we are able to use as the basis of our auto-tagging system. The intuition behind this is actually simpler than I thought. Multi-class problems imply that each category is mutually exclusive to each other. Once a category is chosen, it would be evident that the other categories would not be chosen or even considered. It feels typical for a computer since this kind of emulates binary decision making. If you think about it, a binary classification (1 or 0) is a special multi-class scenario of having only 2 categories. If you extend the number of categories to 3, it would just look like this (2 or (1 or 0)). If it is not 2, then it is either 1 or 0.

On the other hand, multi-labelling approaches things a little differently. It would make sense to consider the other labels when labelling a datapoint. For a more visual example, take a look at this cat.

Let’s say we have these labels to label the possible colour combinations of a cat: [white, black, grey, orange]. Now let’s say that we know that the colour orange is present on the cat. Do you think that information would help with churning the probability of the other labels being positive? With my current knowledge of cats, I would say that I have never seen a grey and orange cat before. Therefore, if I know that the colour orange was present, I would say that the probability of grey being present on the cat as well would be low while the chances of colours like white or black be high! Hence, showing the increase complexity of multi-label problems as compared to multi-class.

Secondly, as a result of the cross referencing between labels, some complications to the labelling accuracy arose. This was simply due to the nature of the questions themselves. Given that Seedly mainly deals with financial topics, most of their keywords we found were of that kind as well. Hence, making it difficult for the system to discern patterns for various sub-categories of the main financial category. For example, a question regarding Savings could seem like a question related to Retirement, even if it was not the intention.

This also brought up something interesting to consider. If a user asks question intended to only be under Savings uses keywords that allude to it being in Retirement as well, objectively speaking, should it also be under Retirement?

Objective and Subjective Labels

The data that we had to feed into our machine learning system only consisted of the labels that the users themselves tagged onto the question. This is definitely not the best arrangement as the users may not have properly labelled the question, as with any human decision, it is subjective. Hence, if we feed this user-skewed data into the auto-tagging system, it would only learn the labelling patterns of the users and not the objectively correct labels.

Although, intuitively, it makes sense to be more objective over subjective, making use of the user inputs could prove useful as well. Given that subjective data is what makes machine learning models so useful to begin with! With subjective data, the machine would be able to learn the patterns of the user to ‘understand’ why certain decisions made, or why the user labelled the question as such. This fully assumes that the user knows for sure what a question should be labelled as and what it should not. If we give them the benefit of doubt, it is then easy to see how valuable these subjective labels actually are.

Ultimately, it would definitely be best to find a way to objective label the questions while we take the subjective labels into consideration! Which is why we developed an ‘if-else’ model, on top of the machine learning model, that objectively labels the questions based on the presence of the respective category’s keyword. For example, if the word ‘save’ is in the question, the model would label it as ‘Savings’. Else, it wouldn’t. This would then help us generate an objective set of labels that we could then merge with the user labels to train the auto-tagging system with.

Imbalance of Labels

Out of the 30 over categories, each question would on average be labelled as 3–5 categories. This imbalance thus makes it difficult to evaluate the accuracy of the system. For example, let’s say there are 30 categories and the system would label a question 1 if it is in that category and 0 if it is not. Given a particular question that belongs to only 5 categories, if the system only labels 4 out of 5 of these categories correctly and proceeds to correctly label the remaining 25 categories correctly, how accurate would you say is this classification? Some possible ways to see this are:

1. having 29 labels out of 30 that’s correct (96.7%),

2. out of the 5 that was supposed to be labelled as 1, 4 were correct (80%). This is also known as Recall

3. out of the 4 labels that were labelled 1, all of them were properly labelled (100%). This is also known as Precision

But if you use the conventional Accuracy metric, this would be considered a 100% incorrect classification.

The first way mentioned above (the one with 96.7%) represents a metric called the Hamming Loss which looks at the number of labels that are correct as a whole. However, it would not make sense if only 3–5 categories are labelled as 1 on average. That would mean that even if we had all labels be 0, we would roughly get an evaluation value of 25/30 (83%) or higher. Which may seem fantastic given the value but is not truly representative of what we want to evaluate, which is how good the system is in predicting the right categories to label as 1.

Precision and Recall resonates more with how we want to evaluate our system. Recall combats the intuition of labelling all categories as 0. It looks at the categories that should actually be labelled as 1 and compares them to the predicted labels of those same categories. Hence, if we had all 0’s, the Recall would evaluate to be 0%. On the other hand, Precision combats the flip side of having more category labels as 1. It simply looks at all labels that are labelled as 1 and compares them to the actual categories that should be labelled as 1. Hence, if we label all 30 categories as 1, with only 5 categories actually being 1, the Precision would evaluate to be 5/30 (16.6%) only. Thankfully for us, there is a metric called the F1-Score that makes use of both Recall and Precision.

Basically, the equation shows how this F1-Score metric takes the Precision and Recall into consideration. Only when both of these values are 100% would the F1-Score be 100% as well. The converse is true as well, where if either Precision or Recall falls short, the F1-Score would be compromised. As mentioned in the introduction, our system managed to achieve an F1-Score of 90%!

Ease of Deployment

Programming is not a skillset that everyone has. Therefore, the need to package the system in a way that made it easier to use and understand seemed like an obvious step in our project. This then required quite some code cleaning and organising to fit them into various forms of ‘packaging’. I honestly enjoyed this process as it felt pretty therapeutic, anyway here are 3 iterations of the ‘packaging’ for this auto-tagging system.

In this image, you can see how raw the codes are in general and if you have no experience in programming, this probably looks mind boggling to you. Even if you do know how to code though, cleaning this up and organising it better would definitely help as well.

In this next image, the working codes are crammed into a single package/module, as you can see on the left. This package/module can be executed to tag questions using just a few lines of code, as seen on the right. It is evident that everything is so much more clean and easier to use in general!

In this last image, the code is placed on the backend of this website and leaves the website to showcase how to use such an auto-tagging system. If it makes any sense, the code is similar to the backstage crew in a concert while the website is like the performer at the concert. Coming from the very first iteration, as shown above, it does not take someone that knows code at all to be able to work this auto-tagging model.

Conclusion

Voila! That’s pretty much it. It was definitely interesting to see how machine learning and coding could be used for an application like this. Looking for things that could be improved from using machine learning is no simple task though. It’s always more comfortable to stick to the status quo, but if we do that, we would rarely see progression. Hopefully after this project I improve, not only on my coding and problem-solving skillset, but on finding aspects in the world that could benefit from coding, machine learning and AI.

Thanks for reading this far and stay safe!

Auto-Tagging Seedly’s Questions with Deep Learning was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

If you haven’t been living under a rock in the past year, you’ve probably heard of the term Data Science. Else, let me catch you up to speed. Data Science is an inter-disciplinary field that uses certain processes to elicit information from various forms of data. Where many people think of this field as something that only came about in recent years, in actuality it has been around for nearly half a century!

It is without a doubt that Data Science only became a buzz word due to the inception of the internet; seeing how its’ existence also brought about an unfathomable amount of varying types of data within such a short span of time. This large amount of data was then eloquently named ‘Big Data’ and is exactly what drew more relevance to jobs like Data Scientists, Statisticians, or just anyone that works with data.

Although the landscape of data related careers has taken huge turn, the process of retrieving information remained the same. Some people have referred to it as the Knowledge Discovery in Databases (KDD) process (which I referenced in my previous article here), while some others called it the Cross-Industry Standard Process for Data Mining (CRISP-DM). However, for simplicity and clarity sake, I shall refer to it as the Data Science process from here on out.

The Process

In summary, the Data Science process articulates the general life cycle of a data science project, from picking a goal to achieving it via data scientific means. This is with reference to the KDD process and CRISP-DM and is based on my interpretation of them.

The Data Science process has 6 steps and is cyclic in nature. What this means is that any of the steps throughout the process may be revisited after new information has been acquired from a later step. This is so as to iteratively tweak the approach used at each step to better align to the goal or improve the accuracy of your results.

Step 1 — Identifying the Goal:

A fundamental step before starting with any project or any data related job is goal setting. To ask the question, “what do you want out of this and why?”. This step would usually require some business and data understanding.

A business understanding would certainly help in guiding the motivation behind the project and connecting that with the eventual usefulness of the insights. For example, if you are in the sales business, perhaps knowing about your customers churn rates could be useful in maximising long term profits. Hence, your motive for maximising profits by gathering churn rate insights would be your goal for the project.

On the other hand, knowing about the data that you will be using would benefit a ton as well. The different types of data that you use in your project could dramatically affect your workflow and since data is an integral to your project, it is definitely a potential make or break. Therefore, knowing about your data helps to manage expectations and plan out the feasible methods available for your project.

Step 2 — Data Selection:

If you did not already know, data is imperative to any part of this process. Jokes aside, it really is important to understand what kind of data is needed or would be better to use for your project.

Besides the obvious, like using text for text analysis or image for object detection, there are other aspects to consider as well. Following one of the previous examples, let’s say you want to analyse the churn rates of your business. What kind of data do you collect? How frequent your customer makes a purchase? (integer value) How recent the customer made a purchase? (date) Or how much the customer spent thus far? (currency) Or perhaps all 3 kinds? All of these are definitely up for consideration and are usually tailored to your project’s goal.

Something to mention as well is the source and veracity of the information, which questions “Where do you get your data and how reliable is it?”. With the sheer volume of data available on the internet, one should definitely consider the correctness of the data taken off it. For example, if you download a file from an official website, chances are your data is accurate, as compared to simply extracting data off any other website via web scraping. If you are simply working on a self-interest project, maybe the veracity does not matter. However, if you are working on a project that your business depends on, it would be wise to think twice about the data you are about to use.

Step 3 — Data Preprocessing:

If you are still in school, more often than not the data that you receive outside of your curriculum is dirty, if you are not, then you probably know what I am talking about. Albeit for those that do not, what I mean by dirty data is having missing values, varying formats within the same dataset or perhaps just unnecessary data points within your working dataset, and with all those hurdles in the way, implementing a data mining tool at this point would render useless. As the late Wilf Hey expressed back in 1965:

“Rubbish in, Rubbish out”

Hence, before actually working on your project, some cleaning of the data would be required.

The basic cleaning operations usually entails removing noise from the data or collecting more information to account for the noise, picking a method to handle the missing data, and to account for time-series data and known changes. Now that all the preprocessing is done, it is now time to transform.

Step 4 — Data Transformation:

Having worked on a few data science projects myself, the best way I can summarise this step is — to condense your data for your mining tool. Basically, by breaking down your data into its essentials, you make mining data easier and clearer to do.

There are many ways to do this depending on the dataset. For example, if you are using text data, like a sentence, you might want to strip the punctuations and stop words and only leave the words significant enough to retain the meaning of the original sentence. On top of that, you could also convert text data into numeric values. Methods like vectorising and tokenising would help in this regard. I went through the cleaning of text data in my previous article. Here is the link if you are interested!

On the other hand, if you are handling numerical data, you could standardise or normalise some of them, or maybe even convert their numerical values into a ranking instead! And if you have too many numerical variables, you might only consider keeping the essential ones. Better yet, perhaps transform and summarise all of those variables into fewer variables. This trick is a whole new topic of its own called Feature Engineering.

The following step would involve passing your data through a data mining tool and hence it is important to ensure that however you have transformed your data here is appropriate for the input of said tool. Be it in terms of data type (string or integer) or even the naming and number of columns.

Step 5 — Data Mining:

As mentioned above, this is the step where you pass your transformed data through a data mining tool that would return potentially insightful information. It goes without saying that different data mining tools have their different characteristics, which also means that you have to choose one that tailors to the goal of the project. So, for example, if the goal is to classify something, employing a classification tool like a logistic regression model could work.

Choosing the right data mining tool is a huge topic as there are many different kinds of tools and use cases for them. There are tools for extrapolation and predicting values, tools for clustering data points together, and even just simulation tools that allow one to observe patterns from. Since this step can get really complicated, I won’t go too much into detail. Instead, I will just summarise this whole step like this: Using a data mining tool on your transformed data, extract information that best satisfies your goal. For example, if your goal was to find out you customer churn rates, you could use a data mining tool to predict the customer’s churn rate.

Step 6 — Interpretation and Evaluation:

In this final step, an evaluation method is used to determine whether or not you have reached your goal. If you are trying to predict something that has happened before, a simple comparison between actual and predicted would be one of the methods. However, sometimes things are not that simple and that is where Exploratory Data Analysis (EDA) comes in. EDA, as the name implies, is basically exploring the results that you have by any means necessary. It can be done by plotting charts or graphs to showcase the distribution. As long as you manage to describe the results in a more intuitive way.

After doing your EDA, it is time to ask yourself whether or not you have achieved your goal. If you have not, then it is time to reflect and evaluate the methods used in the previous steps. As I have mentioned earlier, this process is cyclical in nature. Hence, it is possible to return back to these steps if you realise something about your project. For example, maybe you realised that there could be a better and more precise goal to achieve with the current data, or perhaps you stumble upon better data for the current goal.

In this step, one would constantly be asking questions that would help perfect each combination of methods used in each step so as to finally achieve the objective of the project.

Conclusion

As I reflected throughout the telling of this process, I cannot help but be amazed at the potential of those in data related careers. It truly is an exciting time for a Data Scientist to be alive. With the sheer volume of data, one can only imagine the kind of insights that could possibly be dug out from it. Not to mention the rate of which they’re being pumped out and the various forms they come in.

Thanks for reading this far!

Remember to stay safe and trust the process.

The Data Science Process was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.