LLM applications are not the easiest applications to test. As my co-founder Jeffrey put it, it’s a lot like “playing whack-a-mole” when you are changing prompts/improving RAG applications and before DeepEval, it was difficult to write tests to ensure good performance while iterating on LLM applications.

Let’s first explore the typical developer workflow for LLM applications:

• Developer writes a bunch of spaghetti code to get LangChain working.
• They test a few prompts and then finds a good one that satisfies their initial set of use cases.
• They then play around with different Langchain settings and tutorials to realize that they can do a lot more.
• They then make a few changes — they add in a Cohere Reranker, bring in summarisation into their prompts, switch LLM providers, try out OpenAI Function Calls, introduce GuardRails
• They then realize their initial set of use cases no longer work. And now they have to start from square 1 and figure out some testing framework. They can see they can apply BertScore comparisons and then proceed to do just that.
• They experience a significant slowdown in development.

To summarise, what makes up their MVP looks a bit like this (optional benchmarks required to prove to their CTO/Leads about performance):

And that’s great! But as they explore more, they find their RAG applications can balloon up quite quickly:

The problem now is that each time you add a new tool, it invites more issues and errors because of a number of reasons:

• Natural language outputs can vary quite a bit and are stochastic.
• Agents can reason things differently with different conclusions — imagining Tree of Thought, Chain of Thought and other agent tools
• LLMs can hallucinate at various levels or reasons

If an agent/RAG application can’t get to the right conclusion — could it be because they didn’t reason properly to the right scenario? How can you test for that?

In traditional software engineering practices, these kind of tests are impossible to write. This is where DeepEval comes in as an testing/evaluation framework for these types of problems.

Enter DeepEval — Unit Testing For LLMs

Instead of the usual pattern of build, deploy, eyeball, DeepEval gives developers a framework to unit test these LLM applications.

DeepEval helps developers first create an evaluation dataset, build and test them immediately. The best part? This is all do-able with tooling they already know! CLI, Pytest, OpenAI completion protocols. No obscure SDK/API. Just a pure open-source testing framework designed to improve on prompts.

A quick overview diagram of our framework can be seen below (read more about it here).

DeepEval allows engineers and data scientists to focus on developing these LLM-based applications to confidently know when a new tool/component they added has a down-stream effect. For example — they can test for RAG hallucinations in just 1 line of code as shown below:

from deepeval.metrics.factual_consistency import assert_factual_consistency
assert_factual_consistency(
  "He left at 3PM", 
  "At 3PM, James stood up, looked at the gate and exited."
)

This enables developer flows to look more like this.

An example workflow with example tests developers can now write:

• We start off with a simple Question-Answering application and get it working.
• We then decide “What if we want to add summary as a use case” so we implement a guardrail to ensure for those use cases. To make sure that it doesn’t effect other queries, we add a DeepEval test case for conceptual similarity with previous reasons.
• We may then decide to switch this out for an LLM Agent instead of a QA application so we add an additional tool on top. We add a Deepeval overall test score to ensure this isn’t effected.
• We then discover Cohere embeddings to perform better on multilingual datasets and switch them out instead of OpenAI embeddings. So we add support for the multilingual test case to ensure it is similar to the others.
• We decide next week to fine-tune LLMs as well and then add a bias test to ensure we don’t accidentally add bias ot our fine-tuned LLMs

DeepEval allows these new LLM applications to be built with more robustness than before.

But what if you don’t like writing tests?

DeepEval gives users a way to easily start benchmarking their datasets.

from deepeval.dataset import create_evaluation_query_answer_pairs
dataset = create_evaluation_query_answer_pairs(
  """Python is a great language for mathematical expression and 
machine learning."""
)

Under the hood, we use LLMs to create synthetic queries and answers using ChatGPT (support for more models to come soon!).

And what if ChatGPT gives terrible synthetic queries and answers?

Users can review these queries and answers using our no-code dashboard. You can launch the dashboard in just 1 line of code. You can read more about that here.

dataset.review()

We have an ambitious roadmap to quickly build up the evaluation space with integrations with tools machine learning engineers love like Unstructured, Guardrails, LangChain, LlamaIndex, Streamlit.

We welcome checking out our repository and leaving any feedback!

• Our Github Repository: https://github.com/confident-ai/deepeval
• Join our Discord: https://discord.gg/a3K9c8GRGt
• To read more on how you can get started on this today, check out —

Thanks for reading!

Obviously much of DeepEval wouldn’t have been possible without readings/open-source efforts from various experts/packages/open-source communities.

• SentenceTransformers — https://www.sbert.net/index.html#
• HuggingFace — https://huggingface.co/
• Eugene Yan’s Writings on LLM Evaluations and the various papers linked
• RAGAS — https://github.com/explodinggradients/ragas

To understand retrieval-augmented generation, let’s break it down into its two main components: retrieval models and generative models.

1. Retrieval models: These models are designed to retrieve relevant information from a given set of documents or a knowledge base. They typically use techniques like information retrieval or semantic search techniques to identify the most relevant pieces of information based on a given query. Retrieval-based models excel at finding accurate and specific information but lack the ability to generate creative or novel content.
2. Generative models: Generative models, on the other hand, are designed to generate new content based on a given prompt or context. These LLMs (now explained all over the internet) use a large amount of training data to learn the patterns and structures of natural language. Generative models can generate creative and coherent text, but they may struggle with factual accuracy or relevance to a specific context.

Now, retrieval-augmented generation combines these two approaches to overcome their individual limitations. In this framework, a retrieval-based model is used to retrieve relevant information from a knowledge base or a set of documents based on a given query or context. The retrieved information is then used as input or additional context for the generative model.

By incorporating the retrieved information, the generative model can leverage the accuracy and specificity of the retrieval-based model to produce more relevant and accurate text. It helps the generative model to stay grounded in the available knowledge and generate text that aligns with the retrieved information.

About retrieval models

Retrieval models are a type of language model that focus on finding relevant information from a dataset, in response to a given query. These models can benefit from vast stores of knowledge and are usually trained to produce meaningful and context-specific results. The most common examples of retrieval models:

Retrieval models are generally designed to find and rank relevant pieces of information from a dataset in response to a query. Here are some examples of popular retrieval models and algorithms:

1. Neural Network Embeddings: Neural network embeddings (Such as OpenAI’s embeddings or Cohere’s embeddings) ranks documents based on their similarity in the vector space.
2. BM25 (Best Match 25): A widely used text retrieval model based on probabilistic information retrieval theory. It ranks documents based on term frequencies and inverse document frequencies, considering both the relevance and rarity of terms within a corpus.
3. TF-IDF (Term Frequency — Inverse Document Frequency): A classic information retrieval model that measures the importance of a term within a document relative to the whole corpus. It combines term frequency (how often a term appears in a document) and inverse document frequency (how rare the term is in a corpus) to rank documents in relevance.
4. Hybrid Search: a combination of the above methodologies with different weightings.
5. There are a few other methods but such as LDA but they’re not particularly powerful by themselves as of yet.

Applications

Retrieval-augmented generation has several applications. For example, in question-answering systems, the retrieval-based model can find relevant passages or documents containing the answer, and the generative model can then generate a concise and coherent response based on that information. In content generation tasks, such as summarization or story writing, the retrieval-based model can provide relevant facts or context, which the generative model can use to create more informative and engaging content.

In summary, retrieval-augmented generation combines the strengths of retrieval-based models and generative models to improve the quality and relevance of generated text. By leveraging the retrieval-based model’s ability to find accurate information and the generative model’s ability to produce creative text, this approach enables more robust and contextually grounded language generation systems.

Building your own RAG engine

There are a few solutions out there where you can test building your own RAG engine (I will be writing and sharing my experiences on these soon!).

• If you are interested in an interesting open-source solution, I recommend checking out haystack
• Langchain also offers this but their solution right now is quite inflexible and it’s not clear how results can be improved if they are bad

If you enjoyed this, you might enjoy my AI newsletter (The AI Hero) and join me in building AI — https://ai-hero.beehiiv.com/subscribe

So, you’re the person who finds all the good stuff in the data. You gather, organize, and analyze data using fancy tools to come up with real insights. People who use your insights then make important business decisions.

Here’s the good news: you don’t need to be an expert to be a data analyst. You can get started with a bachelor’s degree in math, statistics, economics or any other field related to math.

Also, you’re the person who can fit in anywhere! You can work in different teams or roles to help make data-driven decisions. You’re like the Swiss Army knife of data.

Data Engineer

Big data engineering is all the rage these days! You’re the person who prepares and manages big data that is then analyzed by data analysts and scientists. You’re responsible for designing, building, integrating, and maintaining data from several sources.

You’re not the person who analyzes the data you receive and store. Instead, you make sure that the data is available to the data analysts and scientists. If a company wants to use data to better their business, you’re the first person to come up with the plan. You’re also the person who regularly improves these plans to make sure the data stored for analysis is accurate and accessible.

You need advanced software development skills, which are not as essential for data analysts and data scientists. But you’re the person who brings it all together!

Data Scientist

You’re the person who makes sense of all the crazy data out there. You analyze and interpret raw data into business solutions using machine learning and algorithms. That’s some serious skills!

You do the same thing that a data analyst does, but you possess more advanced algorithms and statistics expertise. You know how to build, train, and use machine learning and deep learning models to understand data — skills that data analysts don’t possess.

These skills make you immensely valuable in interpreting answers from open-ended questions and also identifying hidden insights. You’re the person who can make sense of all the data that’s out there!

You can be an engineer who has strong business acumen and communication skills. However, you’re not usually in charge of developing or maintaining data architecture. Most data scientist jobs ask for a master’s degree in data science or a related field.

What is fuzzy c-means?

Fuzzy c-means (FCM) is a clustering algorithm used in data science and machine learning. It is a type of unsupervised learning, meaning that it is used to identify patterns and relationships in data without any predefined labels or categories.

FCM works by grouping similar data points into clusters based on their similarity to each other. Unlike traditional clustering algorithms, such as k-means, which assign each data point to a single cluster, FCM assigns each data point a membership value indicating the degree to which it belongs to each cluster.

The membership values are based on a distance metric, which measures the similarity between data points and cluster centroids. FCM aims to minimize the sum of the squared differences between each data point and its assigned centroid, while also taking into account the membership values.

The “fuzzy” aspect of FCM refers to the fact that it allows for overlapping clusters, where data points can belong to multiple clusters simultaneously, with varying degrees of membership.

FCM has a wide range of applications in fields such as image processing, bioinformatics, and market segmentation, where identifying clusters of similar data points can provide insights into patterns and trends in the data.

Okay but can you ELI5?

Fuzzy c-means is like when you have a lot of toys and you want to put them into groups based on how similar they are to each other.

For example, you might put all your cars in one group because they have wheels and can drive, and you might put all your stuffed animals in another group because they are soft and cuddly.

But with fuzzy c-means, each toy can belong to more than one group, like a car that is also soft and cuddly might belong a little bit to both groups. It’s like each toy gets a special sticker that shows how much it belongs to each group.

By putting toys into different groups, we can learn more about them and find out what they have in common. And that’s what fuzzy c-means does with data, it helps us find patterns and groups of similar things!

Imagine you have a big box of toys, and you want to organize them into different groups based on their colors or shapes. A self-organizing map is like a special machine that can help you do this!

You put all the toys into the machine, and it looks at each toy and decides which group it belongs in based on its color or shape. The machine creates a big map with different colors or shapes for each group, and it puts each toy into the group that matches its color or shape the most.

The cool thing about this machine is that it can learn by itself! If you put new toys into the machine that have different colors or shapes than the ones it has seen before, it can create new groups for them and add them to the map.

So, in short, a self-organizing map is a machine that can help you organize things into groups based on their similarities, and it can keep learning and creating new groups as you add more things to it.

Why use self-organizing maps?

Self-organizing maps (SOMs) are used for a variety of reasons, but some of the most common ones include:

1. Data visualization: SOMs can be used to visualize complex data sets in a simple and intuitive way. By mapping high-dimensional data onto a low-dimensional space, SOMs can help identify patterns and relationships that might be difficult to see in the original data.
2. Pattern recognition: SOMs can be used to identify patterns in data and group similar data points together. This can be useful for tasks such as image recognition, where SOMs can be trained to recognize different features in an image.
3. Clustering: Similar to pattern recognition, SOMs can be used to cluster data into groups based on their similarities. This can be useful for tasks such as customer segmentation, where SOMs can be used to identify groups of customers with similar characteristics or behaviors.
4. Data compression: By reducing the dimensionality of data, SOMs can be used to compress large data sets into a smaller, more manageable size. This can be useful for tasks such as storing and transmitting large amounts of data.

More content at PlainEnglish.io.

Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.

Interested in scaling your software startup? Check out Circuit.

ELI5: Self-Organizing Maps In Neural Networks was originally published in Artificial Intelligence in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Variations of fuzzy clustering

There are several fuzzy clustering techniques available, each with its own strengths and weaknesses. Here are some of the most commonly used fuzzy clustering techniques:

1. Fuzzy c-means (FCM): FCM is the most well-known and widely used fuzzy clustering technique. It is an iterative algorithm that minimizes the sum of the weighted squared distances between each data point and the centers of the clusters. The degree of membership of each data point to each cluster is calculated using a membership function, which assigns a probability value between 0 and 1 for each cluster.
2. Possibilistic c-means (PCM): PCM is similar to FCM but allows for the possibility that a data point may not belong to any cluster with a high degree of certainty. It uses a possibility function instead of a membership function, which assigns a value between 0 and 1 for each cluster, representing the degree of possibility that the data point belongs to that cluster.
3. Gustafson-Kessel (GK) algorithm: The GK algorithm is a fuzzy clustering technique that uses the Mahalanobis distance to measure the similarity between data points and cluster centers. It is particularly useful for data sets with irregular shapes and overlapping clusters.
4. Fuzzy clustering using self-organizing maps (SOM): SOM is a neural network-based technique that can be used for fuzzy clustering. The SOM algorithm creates a low-dimensional map of the data, and each node in the map represents a cluster. The degree of membership of each data point to each cluster is calculated using a Gaussian function centered on the node.
5. Fuzzy clustering based on entropy minimization: This technique minimizes the entropy of the membership function to achieve the optimal clustering. It is useful for data sets with a large number of clusters and noisy data.

These are just a few examples of the many fuzzy clustering techniques available. The choice of technique depends on the specific requirements of the problem, the characteristics of the data, and the computational resources available.

Variants of one-to-many clustering

Other techniques similar to fuzzy clustering that can perform one-to-many clustering are:

1. Mixture modeling: Mixture modeling is a probabilistic modeling technique that assumes that the data is generated from a mixture of underlying subpopulations or clusters. Each data point is assigned a probability of belonging to each subpopulation, and this probability is used to determine the degree of membership of each data point to each cluster.
2. Spectral clustering: Spectral clustering is a graph-based clustering technique that uses the eigenvalues and eigenvectors of a similarity matrix to identify clusters. Spectral clustering can handle non-convex clusters and is particularly useful for data sets with a complex structure.
3. Hierarchical clustering: Hierarchical clustering is a technique that creates a tree-like structure of clusters by recursively merging or splitting clusters. In one-to-many clustering, hierarchical clustering can be used to identify the overlapping clusters at different levels of the hierarchy.
4. Non-negative matrix factorization: Non-negative matrix factorization (NMF) is a technique that factorizes a non-negative matrix into two matrices, one representing the basis vectors and the other representing the coefficients. NMF can be used for one-to-many clustering by allowing the coefficients to sum to more than one, which allows each data point to belong to multiple clusters.

Industry applications of fuzzy clustering

1. Image segmentation: Fuzzy clustering can be used for image segmentation, where an image is partitioned into regions based on similar pixel characteristics such as color, texture, and intensity. Fuzzy clustering allows for soft boundaries between regions, which can be useful in cases where it is difficult to determine a clear boundary between different regions.
2. Customer segmentation: Fuzzy clustering can be used to group customers into different segments based on their purchasing behavior, demographics, and other characteristics. This can help businesses tailor their marketing strategies to specific customer segments.
3. Medical diagnosis: Fuzzy clustering can be used to assist in medical diagnosis by grouping patients with similar symptoms and medical histories. This can help doctors identify patterns and commonalities among patients and develop targeted treatment plans.
4. Document clustering: Fuzzy clustering can be used to cluster documents based on their content, such as keywords, topics, and themes. This can help in organizing and retrieving documents efficiently.

What is Fuzzy Clustering was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Face2Vec refers to the process of transforming faces to vectors, a process named vectorization that transforms objects/images/text into numerical representations (if you would like to learn more, read this).

Recently, I added Face2Vec to the VectorHub repository (an open-source repository I maintain to host and share ___2Vec models). Face2Vec is one of the most widely-used vectorizations in the world, forming the basis of a significant number of applications today, including social media filters, identification, and verification.

In this article, we explore FaceNet’s strengths and weaknesses — providing readers with an understanding of where the model performs well and not so well. From there, we go more technical into understanding FaceNet’s architecture and its implementation.

I recommend exploring this model on VectorHub and trying it out for yourself! You can try out the model in 3 lines of Python code and begin searching through your own dataset in less than 5 minutes.

Strengths & Weaknesses

Note to the reader: The strengths and weaknesses are comparing Face2Vec models using the VectorHub implementation. Results may vary from different implementations. A series of experiments were run using custom, random, and transformed images to test how this model performs practically. A lot of the initial faces were retrieved from a portion of the Celeb-A dataset (cited below — of approximately 5 000 images), which can be found in Tensorflow’s datasets modules. We use this as a baseline for exploring the effectiveness of the Face2Vec model that we implemented into VectorHub.

There are, however, a few weaknesses in our experimentation we would like to acknowledge. There are a number of areas that are still left to be explored (how differing races/genders can play a role in the facial image recognition and unless there are specific attributes in the dataset that allow us to explore these — identifying whether these are edge-cases for the Face2Vec implementation needs to be considered. The examples seen below are also hand-picked ones that best illustrate the concepts that were spoken but are reflective of a broader overall trend.

Strengths

• Able to identify similar faces through different facial expressions and angles

• Able to handle different lightings/transformations

• Able to identify the same faces despite having markings on the face

• Action shots where the faces can be highly different can still be matched! This means that photos taken during sports games where facial expressions can be significantly different from everyday expressions, the models are still able to identify similar individuals.

• Able to handle half of a face. Surprisingly — the model was able to even determine the individual correctly when half the face was cut off!

Weaknesses

• Unable to find good matches if the image is edited with words in front of it.

• If individuals are wearing eyewear, Face2Vec overfocused on similar eyewear.

• The model, however, could also be improved across different races and skin colors. Below is an example of the model performing subpar of one such instance.

Overall the model performed well in a number of areas and not so well in a few edge cases. However, I have no doubt that if the model was re-trained with a dataset that has more images with a larger variety of ethnicities, eyewear and distortions with larger representative capacity then it would perform better. If readers are aware of such models, feel free to let us know and we are happy to add these models to VectorHub (or submit your own pull request and we will review it!).

In the following sections, we explore details about how it is performed and the two-stage implementation process that can be found in our VectorHub repository.

Understanding FaceNet

The FaceNet architecture is simple (a standard CNN), but its novelty comes from its use of triplet loss.

Triplet loss today is nothing ground-breaking today and is now part of what is considered One-Shot Learning.

VectorHub Implementation

The VectorHub Implementation is a 2-stage process from photo to face extraction to vectorization.

The model is able to detect faces and also identify facial landmarks — see below. From this, we can use vector search to grab photos of individuals with facial landmarks in similar places, allowing us to search for other photos with half-faces/different angular orientations.

If you would like to see more of this type of analysis or prefer more in-depth analysis of ____2Vec models (apart from just Face2Vec), give this story a clap/leave a comment and we may introduce this as a series and go more into depth into them. Remember to check out VectorHub if you are interested in testing out the model!

Citations & Acknowledgement

The Face2Vec implementation could not have been possible without a lot of support from researchers and engineers who have built some incredible packages!

These include those of the Tensorflow Datasets team, FaceNet team[1], the MTCNN team (https://github.com/ipazc/mtcnn) and the Celeb A team[2]! Thank you for their hard work in allowing us access into the FaceNet2Vec model implementation for VectorHub!

Schroff, Florian and Kalenichenko, Dmitry and Philbin, James, FaceNet: A Unified Embedding for Facial Recognition and Clustering (2015), CVPR 2015

Liu, Ziwei and Luo, Ping and Wang, Xiaogang and Tang, Xiaoou, Deep Learning Face Attributes in the Wild, (2015), IEEE Computer Society.

The Strengths & Weaknesses of Face2Vec (FaceNet) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

For data scientists, ML Engineers, ML developers, search engine enthusiasts.

Search engines and Top-K rankings are difficult to compare and are often difficult to justify. Top-K rankings are usually evaluated with several common criteria:

• Top results are more important than bottom-ranked results.
• A drop from 2nd to 3rd is more significant than a drop from 5th to 6th and so on.
• A specific result not appearing in another list is more indicative of difference than a result dropping in another list as it suggests the ranking itself is greater than K.

In light of this, I found little literature that actually proposed a statistical comparison of Top-K comparisons. Common methodologies included Kendall-Tau and Spearman Rank correlation but these only capture finite lists and do not quite answer the problem. In other words, they are only able to compare lists they have ranked all n of their results. However, I stumbled upon a different statistical solution that solved the aforementioned problem — Rank Biased Overlap. It suggested an interesting way to compare ranks between results that I had not seen in other algorithms or explored in other blogs to its fullest extent.

The rest of this blog will be discussing rank-biased overlap with an illustration of the problem while also trying to illustrate how to maximize its use in comparing lists. Some of which are built into Confident AI as -testing tools for LLM applications. These are linked below that can be explored by users.

The formula, which has been nicely refactored in the paper, can be shown below:

The astute reader will immediately notice that the p is just a weighting factor is not immediately intuitive. The question is then why use something so complex?

Let us reconsider this from a different perspective by looking at the similarity between 2 lists such as

Now, to fulfil the criteria of rank-based importance, we want 2 simple things:
1) The weights to be of decreasing importance.

2) The sum of the weights to be finite-bound (convergent) to allow for interpretation. In other words — we want the scale for RBO to be from 0 to 1 so that we can interpret if results are good.

The simplest implementation of that is the geometric series. As we want the sum of weights to be 1, we multiply both sides with (1-p) and then set w to the values below.

This then leads us to RBO — which is a fantastic metric for comparing Top-K lists.

However, practitioners will then ask — what is a good value for p? The original paper used p = 0.9 for the weighting to occur but I personally believe it is much easier to interpret these when you have the weightings.

Based on the above formula, we get the following weights at each rank:

If we explore this behavior on the extremes of p, we notice that if p was close to 0, the weightings are biased very much to the top results:

Conversely, if close to 1, the weightings are biased towards evening the result weightings.

Furthermore — an added advantage of RBO is that it can be extended to compare uneven lists, which is not possible with other statistical methods and comparisons.

Let us consider another positive attribute of RBO — modeling how an item in position 2 in list 1 dropping down to position 6 in list 2 is captured differently to it dropping out of the list. We can immediately see that at position 6, the intersection between the 2 is once-again captured at lower weighting in the first instance but is not included in the second instance. This simple method of calculating the intersection provides a great way to compare search results at differing positions!

RBO provides an easy-to-calculate way of calculating these results. Through clever hash-table implementations that allow for quick calculation of intersections and a clever weighting scheme, we are able to use RBO as a great statistic for comparing Top-K ranks.

Comparing Top-K Rankings Statistically was originally published in Artificial Intelligence in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Table of Contents

1. NLP Pipeline Refresher
2. So What Exactly Does Sentencepiece Do?
3. Subword Regularisation
4. How Does This All Relate To Transformer Architectures?
5. How Does SentencePiece Stack Up Against Other Sub-word Generators?
6. My Thoughts On Future Research
7. Conclusion / Writing Philosophy

“Sentencepiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing” — SentencePiece Paper

This is really, really dense. For the reader, I assume that there is prior experience of natural language processing. To understand what the above quote means, we first revisit our understanding of how natural language processing works in general and the NLP pipeline. If you are already very familiar with the pipeline and are looking for a more thorough dive into SentencePiece paper, feel free to skip to “So What Does SentencePiece Do” .

NLP Pipeline Refresher

Let us consider firstly what an example pipeline looks like.

The purpose of this pipeline is to turn words into numbers to feed into our model. These numbers simply represent vectors and we have different vectors for different words. An intuitive understanding of vectorised representation of words is explored here by Christopher Olah. Please read this before continuing if you do not already grasp this concept. Now, let us discuss this pipeline to better understand where SentencePiece comes in.

Tokenizing refers to splitting up a text document. We can split up a text document in a number of different ways such as through white spaces, punctuation, spaces.

In the example above (the image), the tokeniser splits up on vowels unless it is the second letter of a word. (Strange, absurd rule I know but please bear with me as I explain why this is necessary.) Now, there is something important to note here: there are 2 #’s in a few of the ‘subwords’ but not in others. In this tokenisation example, having 2 #’s in front of the word means that it is a continuation of the previous sub-word. While this may seem to be an appropriate thing to do — no distinction is usually made to determine in what order these are done (for example, having multiple #’s to denote their position in the word).

From here, we map these separated words into numerical representations of words. This is done via a look-up table. Each sub-word or word has a vector representation that has been established from pre-training.

Once we vectorise these words (which refers to changing words to numerical vectors), we can plug them through some model that has been trained on the tokenisers.

The rest of the article will go on to explain the problems as this “simple” pipeline grows and SentencePiece’s vital role in this pipeline.

So what does SentencePiece do?

Now that we understand SentencePiece basically splits words into several parts called subwords (via the process of tokenisation), we now explore why the implementation is so popular that it is used across so many different models.

What I just said was perhaps slightly misleading — it doesn’t just ‘split’ words, it does a bit more than that and I will clarify what I mean. Imagine the following sentence:

“How does sentencepiece work at a fundamental level?”

What is the best way to split this question into different subwords? And how do we even quantify this approach?

To answer questions such as these — let us re-frame the question into a more understandable problem statement:

How do I capture the most frequent and diverse sub-words when I have a fixed vocabulary list?

But wait a minute — why does our word segmentation algorithm need a vocabulary size limit?

To answer this question as intuitively as possible, let us consider the inverse scenario where we do not set a vocabulary size limit. At some point, you will end up including very rare words only counted once or words where their meaning can be complicated. For example — the words “simple” and “simplify” are the same words but slightly altered to suit grammatical purpose. Is it really worth having an additional vector stored in memory? Furthermore, words like “Jentacular” (I googled unusual English words), we are not too interested in recording as it comes up very very rarely! Hence, we want to have a restriction on the vocabulary list. With SentencePiece, the limit is set by the number of words in the vocabulary list.

Now that the dimensions now intuitively make sense — we want to capture words that appear frequently enough to determine the importance of the word but also diverse enough between the sub-words to minimise re-capturing the same information and build up a useful, diverse sub-word vocab list.

There are a few different algorithms to resolve these issues. Such algorithms can include Byte-Pair Encoding (supported in SentencePiece), Unigram language model (also supported in Sentencepiece) or subword-nmt (not supported in SentencePiece).

I will not go too in-depth with these algorithms as I found very, very clear explanations in this blog which discusses NLP pre-processing quite in-depth although I found its examination of SentencePiece to be quite general and not too in-depth. It does, however, note the distinction between SentencePiece as a concept and as a piece of software. So naturally, the next question is — if these are the algorithms then what exactly is Sentencepiece and what do these algorithms have to do with this?

To answer this question — we explore the underlying components of SentencePiece.

SentencePiece itself comprises of 4 different components — which we will go into more detail — Normaliser, Encoder, Decoder and Trainer.

Let us go through the components of SentencePiece one by one:

• The normaliser does not refer to taking the mean and removing standard deviations from some digit. Except, in NLP, normalisation refers to standardising the words of the text such that they follow a suitable format. In this case, SentencePiece alter words/letters into equivalent NFKC Unicode (e.g. things starting with U+0026). This means that semantic equivalence is established by splitting/removing accents as outlined here. Note, however, as outlined in the normalization README, there is room to try different unicode methodologies. For those interested, the code implementation of the normaliser can be found here and is implemented in C++.
• The trainer uses the specified algorithm to build up a word vocabulary based on sub-word components. As previously mentioned, SentencePiece supports 2 main algorithms — BPE and unigram language model. Each of these are quite simple concepts that are very clearly explored in other blogs/Wikipedia articles so I will once again refrain from re-inventing the wheel.
• The encoder/decoder itself is quite self-explanatory. The encoder refers to the process of pre-processing and post-processing. This can be summarised neatly in the equation presented in the SentencePiece paper as below:
  
  Decode(Encode(Normalized(text))) = Normalized(text)
  
  The paper refers to the above equation as lossless tokenisation. This is the idea that no information is lost in encoding/decoding. There is a bit of novelty here in SentencePiece in how it deals with white spaces. It explains that it handles white spaces by replacing it with ‘_’ at the beginning of the word following the white space — allowing it to reconstruct this. However, upon decoding, it then simply replaces all ‘_’ with with an empty space. While not entirely lossless as you lose the value of ‘_’, I couldn’t think of any meaning interpretation of the underscore anyway and the preservation of white spaces is more important.

Subword Regularisation

There is, however, one small problem with the tokenisation process — which is what happens when you have multiple ways to split up the word based on the vocabulary list.

If we were to frame this into a definitive problem, it would be:

How do we best split a sentence to ensure that when words are used in the same context, they are matched to the same IDs?

Now that we have defined the problem, let us explore how SentencePiece aims to resolve this problem. This is largely explored in this paper but we will go a bit into how he solves it and the intuition behind them here as they are useful components to SentencePiece.

Subword regularization aims to “employ multiple subword segmentations to make the NMT model accurate and robust”. Here, NMT simply means “Neural Machine Translation”, referring to the use of neural networks in translating from one language to another in this context. Furthermore, “multiple subword segmentations” refer to our segmentation model considering the different ways in which a sentence can be split.

Subword regularisation in Sentencepiece comprises of 2 components:

1. A new training algorithm to “integrate multiple segmentation candidates”. This means that SentencePiece takes into consideration multiple ways to split the sentence and number of parameters. NMT training with on-the-fly subword sampling (wherein the paper utilises an algorithm that is not too computationally complex by reducing the number of words considered in the sample and the number of alternative samples we can consider based on probability). In this case ‘on-the-fly’ refers to when we are training and predicting for each word. Subword sampling refers to selecting a random subword based on particular probability distributions.
2. A new subword segmentation algorithm based on language models, which the paper calls ‘n-best decoding’. This simply refers to when given n-best segmentations, we choose the best to maximise a specific score. This score’s formula for those who are interested is:

The reader will notice that there is a penalisation parameter added in the denominator aimed at penalising shorter sentences. However, it is not immediately obvious why this is the case. As mentioned by reader Jmkernes, this is a concept that is quite common in NLP and we can understand this phenomenon by understanding how the probability is calculated in the first place.

Therefore a penalisation term is introduced to make shorter sentences (which have fewer multiplications and hence higher probabilities) as likely as the longer sentences. The lambda is stated in the paper to be optimized from the development data (I am unsure what the exact method is so if any readers in the future know, please inform me and I am happy to update this section of the article in the future).

So what does this all have to do with transformer architectures? How does this apply to models like T5/Reformer/XLNET?

This is all to do with how the model is trained in the first place. By firstly using SentencePiece to generate word IDs and then feeding them through the architecture, you can train the model and then decode to return words again.

However, we then need to think why can’t you use the same SentencePiece tokenizer for all the different transformer models? This is a result of different architectures being trained on different texts in the first place. The idea behind transfer learning is incorporating a lot of pre-training from massive text databases like Wikipedia and then trying to understand how to model against those. Hence, because we have different text sources and different sub-word vocabulary generation algorithms that we can implement, we end up with different words from vocabularies and IDs that match to these vocabularies. As a result, we end up with different tokenisers for different models.

As an example — BERT is a transformer architecture. It uses a tokenizer called ‘BertTokenizer’ which is based on the WordPiece tokeniser. Whilst this was Google’s own internal closed-source tokeniser, it has been re-created in the transformers library and can be used to extend existing vocabulary sources.

How SentencePiece Stacks Up Against Other Sub-word generators?

I found the above table to be a very concise summarisation of the benefits that SentencePiece offers in comparison to other implementations.

TLDR; SentencePiece isn’t too bad.

Where I Think Future Research Should Go

I think no discussion of concept is complete without outlining work for future contributors and where I think research can take place. Here, I discuss 2 potential improvements that I think need to be made.

1. I think tokenisation should be based on a minimal frequency criteria for BPE/minimal probability criteria for unigram language modelling rather than having fixed vocabulary size. While I understand this is to ensure that memory limitations are met (or at least I believe that is why we set a limit at 32K — please let me know if I am misinformed), I think we should change the criteria from a hard-set total count limit to a sub-word frequency limit instead.
2. I also think asian character language tokenisation should be further researched. At the time of writing this, I am not aware of any intra-character chinese tokenisation but I believe they are crucial to improving Chinese language modelling. For example, note the following words and their left parts: 说话 (speak), 谈论(talk), 告诉 (tell). One would expect that as these are similar in meaning and it would make sense to group them. I expect there to be research in this field already but perhaps in Chinese. If the reader knows any, please feel free to link me!
3. Improving our understanding of sub-word regularisation and the measures we need to take in providing robust sub-words.

If you enjoyed this, you might enjoy my AI newsletter (The AI Hero) and join me in building AI — https://ai-hero.beehiiv.com/subscribe

Writing/Teaching Philosophy

I share my writing philosophy in order to give exposure to teachers who have helped me improve clarity in the field and offer readers a chance to learn from them too.

My attempt at distiling information surrounding the paper and code behind transformers’ most popular sub-word vocabulary generator has largely been inspired by the works of Christopher Olah (known for his recurrent neural network blogs), Chris McCormick (his Youtube channel is great for understanding Bert at a deeper level) and Jeremy Howards (FastAI for all deep learning resources). The purpose behind writing detailed posts like these is to make the mountain easier to climb and to ensure future AI algorithms are built properly and with care. Please feel free to reach out to me via e-mail if I have helped you or connect with me on LinkedIn here.

So generally — what are managers’ looking for? I have also put the questions in order of importance because to me, the biggest difference between success and failure is how quickly you get to work on it. I am sure none of us want to wait 12 months before we can work on something we genuinely have a vision in.

Question 1 — Why is it urgent?

Understand urgency and the ideas of business. Expect your managers to have 20 different things that needed to be done so why should your idea be the one that they prioritise? If you believe something needs to be done — make it sound like it needed to be done yesterday. To ensure success pitching your business use case, here are a few dimensions you should consider when it comes to this:

• If competitors are doing it — you probably should as well if it is core functionality. For example — are your competitors using better fraud detection to boost sales by a million dollars? Are we doing that?
• Explore magnitude and importance — who will this business decision effect? What is the likely magnitude of your project? By improving your artificial intelligence powered search engine — will you end up improving search relevance and sales by a significant amount? Think revenue, cost optimisations (core to any business). Will your customers have a significantly improved shopping experience?
• Are there any recent events to fuel why something needs to be done? Perhaps a pandemic has caused everyone to stay inside and you need to build a tool for the team to work remotely? This will require proper discussion with the team about what would work best.

Question 2 — Why you/your team?

This can be difficult and varies depending on your organisation’s expertise. Ideally, you try and build as much out-of-work experience in this as possible and then suggest you can do the same for work to significantly improve the team’s work. Also consider what assets your team have — do you guys have a data tracking tool to allow you to examine success? Do you have knowledge about something that no one else has? Have you guys done similar projects in the past? Similar to building a competitive advantage in startups — what is something that will take other teams a long time to learn and understand that you can do straight away? Build a business use case that makes you guys sound like the perfect team to solve a problem!

Question 3 — How do you plan to implement this?

Self-explanatory question. Dimensions to think about here include:

• What is the expected length of the project? (Including testing)
• How much resources will be required?
• What kind of expertise will you need and in what areas?

If you are pursuing this because this is something you believe in — you need to be willing to put in extra hours to make your idea come to life. Nothing comes for free and so you should try and factor this in when making your decision. I don’t recommend pitching business decisions unless you are truly passionate about what you are building or doing for the team.

Question 4 — Is your presentation going to be understandable immediately?

For business presentations — a safe rule of thumb is to assume that your manager will look at your presentation only once and will not spend more than the time of your meeting to explore this. Therefore, have graphs that people can understand in less than 3 seconds. Have icons that are directly relevant to the problem at hand. If even 1% of your presentation is irrelevant, then it is going to be discarded. Furthermore, data scientists have a problem with being technical — please avoid jargon as much as possible. Do not mention you are using “Bert”, just say “model” unless you are discussing implementation.