Andrej Karpathy’s YouTube video course — Lecture Notes. Part 1.

This post is based entirely on Andrej Karpathy’s video tutorial “The spelled-out intro to neural networks” — https://www.youtube.com/watch?v=VMj-3S1tku0&t=4822s

In this post, we will learn about the very basic mechanisms of neural networks — how does a network learn to classify dog and cat images? How does a network learn to generate realistic text? The basics behind all of these tasks are the same!

Let’s start by asking ourselves some questions about neural networks (and answering them):

1. What are neural networks? They are simply mathematical functions that have an input and an output.
2. What does it mean to “train” a neural network? It means tuning the function’s parameters such that its output is as close as possible to the truth.
3. How do we tune these parameters? Using an algorithm called backpropagation, which calculates the derivative of the target term which we aim to minimize, with respect to each parameter of the network, and uses these derivatives to nudge each parameter in the right direction (that will lead to minimization of the target term).

In essence, after you read the above questions, you know how neural networks work. But of course, we wish to go (a bit) deeper than that. Hence, we will follow Andrej Karpthy’s excellent course on YouTube, and dive into the nitty-gritty of neural networks, piece by piece.

This post will be organized as follows:

• Part 1. What is a Derivative?
• Part 2. Micrograd — A Minimalistic Autodiff Tool
• Part 3. Micrograd — Manual Calculation of Gradients
• Part 4. Micrograd — Building a simple neuron
• Part 5. Manual Backpropagation through a Neuron
• Part 6. Implementing Backward Operations
• Part 7. Adding support for constants
• Part 8. Breaking up Tanh into its constituting parts
• Part 9. Comparing Micrograd to Pytorch
• Part 10. Building a multi-layer perceptron (MLP)
• Part 11. Training the MLP
• Part 12. Summary

Just like Andrej, we will start at the very (very) basics. What is a Derivative? It might seem basic but it’s important to get a very firm grasp on this subject before moving on. Derivatives are the core of the backpropagation algorithm, which is the fuel behind (almost) every modern neural network.

Part 1. What is a Derivative?

The layman’s definition of a derivative is this:

“I have a function in which y is dependent on the value of x. If I nudge x just a tiny bit at a certain point, what would be the rate of change of y in response to that?”

For those of you that prefer equations, let’s see what Wikipedia has to say about this:

As we see in eq.1 — We take f(a) and nudge it by a tiny amount — h. We take the difference between f(a) and the nudged version — f(a + h). Then we divide it by the amount that we nudged — h, to get the rate of change. Also, notice the lim in the equation. This means that the true slope will be obtained when the nudging amount h is very close to 0.

Let’s see some Python code to make this clearer. We will first create a polynomial function and plot it using matplotlib.

def f(x):
    return 3 * x ** 2 - 4 * x + 5

xs = np.arange(-5, 5, 0.25)
ys = f(xs)
plt.plot(xs, ys)

What is the slope at x=3? Before reading ask yourself — is positive? negative?

Let’s calculate it using the definition of the derivative we just saw.

h = 0.00001
x = 3
(f(x + h) - f(x)) / h

Out: 14.00003000000538

And indeed, we get the correct answer (you can validate this by using calculus to get the derivative and placing 3 instead of x).

Let us now do the same thing with a slightly more complex function that depends on several parameters. We will check what is the derivative of the function with respect to each one of its parameters.

def g(a, b, c):
    return a * b + c

a = 2
b = -3
c = 10

Before reading on, try to think what will be the slope of the function g(a, b, c) with respect to each of its parameters.

print(f'Slope w.r.t a: {(g(a + h, b, c) - g(a, b, c)) / h}')
print(f'Slope w.r.t b: {(g(a, b + h, c) - g(a, b, c)) / h}')
print(f'Slope w.r.t c: {(g(a, b, c + h) - g(a, b, c)) / h}')

Out:

Slope w.r.t a: -3.000000000064062
Slope w.r.t b: 2.0000000000131024
Slope w.r.t c: 0.9999999999621422

We see that if we nudge a by a tiny amount the slope is negative. This makes sense because a is multiplying b which is a negative number, which means we get more of this negativeness in the function. The logic is similar for the other parameters :)

Part 2. Micrograd — A Minimalistic Autodiff Tool

In this tutorial, we are going to build Micrograd — a minimalistic auto-differentiation utility in Python. Autodiff is the basis behind modern neural network libraries such as TensorFlow and Pytorch and is an algorithm for automatically calculating gradients in computational graphs (such as neural networks). The tendency is to think that Autodiff is a very complicated piece of machinery, yet Andrej shows in his course that we can create a fully functional Autodiff tool (Micrograd) in just 100~ lines of code.

The full version of Micrograd (by Andrej Karpathy) can be found here — https://github.com/karpathy/micrograd.

The purpose of an Autodiff tool is to be able to produce the slope of any function with respect to any one of its parameters. This is useful in neural network training because neural networks are giant functions with many parameters, and to improve them we need to calculate their slope with respect to our target, to understand how we should nudge each of the function’s parameters.

An Autodiff tool works by keeping track of a computational graph, which it uses to understand which parameters affected which parts of the function calculation and how. As we will see, each type of mathematical operator has a different derivative, and by chaining together these derivatives using the chain rule we will be able to obtain the slope of a very deep set of operations.

Let’s dive in. First, we need some way to represent a number in Python which will allow us to follow the calculations that it is involved in. This will help us build the computational graph needed for Autodiff.

class Value:
    def __init__(self, data):
        self.data = data

    def __repr__(self):
        return f'Value(data={self.data})'

a = Value(2.0)
b = Value(-3.0)
print(a)
print(b)

Out:

Value(data=2.0)
Value(data=-3.0)

We created a class called Value which keeps track of a value of a number, and implements the __repr__ function, which tells Python how to print the representation of this object. Not very useful right? What if we want to add two numbers, for example? Let’s add some computational abilities to this class.

class Value:
    def __init__(self, data):
        self.data = data

    def __repr__(self):
        return f'Value(data={self.data})'

    def __add__(self, other):
        return Value(self.data + other.data)

    def __mul__(self, other):
        return Value(self.data * other.data)

a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
a * b + c  # equivalent to (a.__mul__(b)).__add__(c)

Out:

Value(data=4.0)

Great! We are now able to reproduce our function from before using the Value class. Notice that a * b + c is equivalent to (a.__mul__(b)).__add__(c) and this is because Python knows it should use the __add__ and __mul__ functions (if they were implemented) when using + and * operators. But wait, this still doesn’t give us the computational graph we wanted so we can calculate derivatives. Let’s make some changes to the class to keep track of the calculations that occur.

class Value:
    def __init__(self, data, prev=(), op='', label=''):
        self.data = data
        self._prev = set(prev)
        self._op = op
        self.label = label
        self.grad = 0

    def __repr__(self):
        return f'Value(data={self.data})'

    def __add__(self, other):
        return Value(self.data + other.data, prev=(self, other), op='+')

    def __mul__(self, other):
        return Value(self.data * other.data, prev=(self, other), op='*')

a = Value(2.0, label='a')
b = Value(-3.0, label='b')
c = Value(10.0, label='c')
ans = a * b + c  # equivalent to (a.__mul__(b)).__add__(c)
print(f'ans._prev: {ans._prev}')
print(f'ans._op: {ans._op}')

Out:

ans._prev: {Value(data=10.0), Value(data=-6.0)}
ans._op: +

We added new parameters to the class: self._prev and self._op . These parameters are used to keep track of the participating elements in each calculation, and which operator was used in each calculation. We can see that we printed the _prev and _op values of the final value of the function, of which the last operation was an addition, in which c and a*b participated (if we were to print the _prev and _op values of the inner element, we should see 2, -3, and *). We also added a label parameter that will help us visualize the computational graph soon and a grad parameter that will hold derivative information for each parameter.

We will now add a function that will help us visualize the computational graph which was built by the Value objects.

from graphviz import Digraph

def trace(root):
    nodes, edges = set(), set()
    def build(v):
        if v not in nodes:
            nodes.add(v)
            for child in v._prev:
                edges.add((child, v))
                build(child)
    build(root)
    return nodes, edges

def draw_dot(root):
    dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'})
    nodes, edges = trace(root)
    for n in nodes:
        uid = str(id(n))
        dot.node(name=uid, label=f'{n.label} | data: {n.data:.4f} | grad: {n.grad:.4f}', shape='record')
        if n._op:
            dot.node(name=uid + n._op, label=n._op)
            dot.edge(uid + n._op, uid)
    for n1, n2 in edges:
        dot.edge(str(id(n1)), str(id(n2)) + n2._op)
    return dot

a = Value(2.0, label='a')
b = Value(-3.0, label='b')
c = Value(10.0, label='c')
e = a * b; e.label = 'e'
d = e + c; d.label = 'd'
f = Value(-2.0, label = 'f')
L = d * f; L.label = 'L'

draw_dot(L)

Out:

We created a draw_dot function that takes a Value object and builds the computational graph associated with it. First, it recursively finds all nodes and edges using the trace function and then continues to draw these elements using the graphviz library. Notice that we explicitly had to label the e , d and L nodes for their names to appear in the visualization.

We will now recap what we did so far. We created a Value class that allows us to create a simple computational graph built of addition and multiplication operators. We are also able to visualize this graph using the graphviz library. But we already saw before that in order to learn neural network parameters, we need to be able to calculate the gradients for each parameter in the computational graph with respect to the output node. As a first step, we will calculate each of the gradients manually and populate the respective gradient parameter.

Part 3. Micrograd — Manual Calculation of Gradients

Our function consists of seven parameters — a, b, c, d, e, f, L. Let us think of the gradients for each, with respect to the output — L.

• The gradient of Lwith respect to itself is simply 1. Why? How much will Lchange when we add h to it? Well of course, by h. Since the gradient is the relative change, the answer will be h/h = 1. That was easy, right?
• Now, what is the gradient of d with respect to L? We have that L=d*f. From calculus, we know that dL/dd = f. Let’s also prove this using the definition of a derivative. As a reminder, the definition is (f(x+h) — f(x))/h (rise over run). So if we want the derivative of L with respect to d we get ((d+h) * f-d * f)/h, which is (df + hf — df)/h, which is hf/h, which is f = -2.
• Similarly, dL/df = d = 4.
• Now we are getting to the tricky part. What is dL/dc? What is the ratio of change of L when we slightly change c? It is trickier because the effect isn’t direct, but rather passes through the variable d. Luckily for us, calculus has us covered with the chain rule. What does it entail? That if we want to calculate dL/dc which passes through d, we simply need to calculate dL/dd and dd/dc and multiply between them.
  Easy!
  So, we already know dL/dd. As for dd/dc, d=c+e, how much does d change when we add h to c? By h of course. Since a derivative is the ratio of change, it will be h/h=1. This means that according to the chain rule, we get dL/dc = dL/dd*dd/dc = f*1 = f = -2.
• By symmetry we get dL/de = f*1 = -2
• What about dL/da. From the chain rule, we get that it is dL/de * de/da. We already know dL/da. de/da for e=a*b is just b as we saw previously. This means that dL/da = dL/de*de/da = f * b = -2 * -3 = 6.
• By symmetry we get dL/db = dL/de * de/db = f * a = -2 * 2 = -4

In the video course, Andrej proves all of the gradients are correct using the definition of the gradient, as we did previously (checking rise over run). For brevity, we will skip this in this article.

Finally, let’s manually assign all the gradients we found to their respective parameters and view the computational graph again.

a.grad = 6
b.grad = -4
c.grad = -2
d.grad = 4
e.grad = -2
f.grad = 4
L.grad = 1
draw_dot(L)

Out:

Let’s now imagine that the function above is a neural network that we want to optimize. Our optimization goal will be to increase the value of L. For each parameter, we calculated a gradient which tells us which direction should this parameter move in order to increase the value of L. We will use this information to optimize the function (note that we will do this only for the nodes which we have control over — that are not the direct result of an operation):

a.data += a.grad * 0.01
b.data += b.grad * 0.01
c.data += c.grad * 0.01
f.data += f.grad * 0.01

e = a * b
d = e + c
L = d * f

print(L.data)

out: -7.286496

Indeed we see that after we optimized our parameters using the gradients we manually calculated, and ran the forward pass of our network again, we managed to increase the value of L.

Part 4. Micrograd — Building a simple neuron

We are now ready to move on to real neural networks. But first — what is an artificial neuron?

In Figure 1 we see a depiction of an artificial neuron that contains 3 inputs — X1, X2, and X3. Each input is multiplied by its corresponding weight and these multiplications are added up. An additional bias term b is added as well. This bias term is inherent to the neuron and will be the same for every input. Finally, an activation function is applied. In this tutorial, we will use the Tanh function, which squashes the input between -1 and 1. The artificial neuron vaguely mimics what we think happens in a real neuron, where neurons receive input from other neurons via synapses and send their outputs via axons.

Now, let us implement an artificial neuron and understand how to use backpropagation to tune its weights in order to maximize a certain target function.

x1 = Value(2.0, label='x1')
x2 = Value(0.0, label='x2')
w1 = Value(-3.0, label='w1')
w2 = Value(1.0, label='w2')
b = Value(6.7, label='b')

x1w1 = x1*w1; x1w1.label = 'x1*w1'
x2w2 = x2*w2; x2w2.label = 'x2*w2'
x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'
n = x1w1x2w2 + b; n.label = 'n'
# o = n.tanh()
draw_dot(n)

Out:

We implemented a simple neuron with two inputs: x1 and x2. Notice that we commented out the use of the Tanh activation function. Why? Because we haven’t yet implemented it in our Value class! So let’s do that. But first — what is exactly a Tanh function?

This is the definition of Tanh taken from Wikipedia:

We have two options now — one will be to implement the exp function in Value. This will allow us to create the Tanh function out of its basic building blocks of addition (subtraction), multiplication (division), and exponentiation. The second option will be to implement the Tanh function directly. In backpropagation, we don’t have to explicitly implement the smallest parts of the computational graph — we can use arbitrarily large building blocks as long as we can implement their forward pass and derivative. We will choose the second option and implement Tanh in the Value class.

class Value:
    def __init__(self, data, prev=(), op='', label=''):
        self.data = data
        self._prev = set(prev)
        self._op = op
        self.label = label
        self.grad = 0

    def __repr__(self):
        return f'Value(data={self.data})'

    def __add__(self, other):
        return Value(self.data + other.data, prev=(self, other), op='+')

    def __mul__(self, other):
        return Value(self.data * other.data, prev=(self, other), op='*')

    def tanh(self):
        x = self.data
        t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
        out = Value(t, (self,), 'tanh')
        return out

We added the tanh function, which contains the calculation for the Tanh operation as we saw earlier. Notice that we pass a 1-item tuple as the prev parameter because Tanh operates only on one input.

Now, after we comment out the Tanh operation we can plot the full computational graph:

Now it’s time to backpropagate through our neuron!

Part 5. Manual Backpropagation through a Neuron

I think this part is a nice exercise to do on your own, so if you are up to it — stop here and work the grad values of each of the parameters with respect to o(you don’t have to :)).

Let’s work through it:

• The gradient of o with respect to itself is 1.
• The gradient of owith respect to n is the gradient of the Tanh function, which is defined by: dtanh(x)/dx = 1 — tanh(x)^2. We will also multiply this by the gradient of o according to the chain rule. So we get that n.grad = (1 — o.data**2) * o.grad = 0.5.
• The gradient of o with respect to b according to the chain rule is dn/db * do/dn. Remember that the addition operation simply passes the gradient from the previous step because the local derivative is 1. So we get that b.grad = 1*n.grad = 0.5.
• Similarly, x1w2x2w2.grad = 1*n.grad = 0.5.
• How about x1w1.grad? It’s also a result of an addition operation, hence: x1w1.grad = 1*x1w1x2w2.grad = 0.5.
• Similarly, x2w2.grad = 1 * x1w1x2w2.grad = 0.5.
• For x1.grad we have a multiplication operation, of which the local gradient is the other parameter. To this, we apply the chain rule, hence x1.grad = x1w1.grad * w1 = -3 * 0.5 = -1.5.
• Similarly, we get that w1.grad = x1w1.grad * x1 = 2 * 0.5 = 1, and x2.grad = x2w2.grad * w2 = 0.5 * 1 = 0.5, and w2.grad = x2w2.grad * x2 = 0.5 * 0 = 0.

Let’s sum it up in code and plot the resulting graph:

o.grad = 1  # do/do
n.grad = (1 - o.data ** 2) * o.grad  # do/dn * do/do
b.grad =  1 * n.grad # dn/db * do/dn
x1w1x2w2.grad = 1 * n.grad  # dn/dx1w1x2w2 * do/dn
x1w1.grad =  1 * x1w1x2w2.grad # dx1w1x2w2/dx1w1 * do/dx1w1x2w2
x2w2.grad =  1 * x1w1x2w2.grad # dx1w1x2w2/dx1w1 * do/dx1w1x2w2
w1.grad = x1.data * x1w1.grad  #dx1w1/dw1 * do/dx1w1
x1.grad = w1.data * x1w1.grad  #dx1w1/dx1 * do/dx1w1
w2.grad = x2.data * x2w2.grad  #dx2w2/dw2 * do/dx2w2
x2.grad = w2.data * x2w2.grad  #dx2w2/dx2 * do/dx2w2
draw_dot(o)

Out:

Great! We know how to do backpropagation! So is this how they do it in GPT-3 (which has 170B parameters)? Do they write 170B lines of code to set the gradient of each parameter? Well, no. Let’s see how to do it properly and automatically.

Part 6. Implementing Backward Operations

We are now going to add a _backward function for each operation in our Value class. This function will calculate the gradient for each computational node instead of us having to do it manually.

class Value:
    def __init__(self, data, prev=(), op='', label=''):
        self.data = data
        self._prev = set(prev)
        self._op = op
        self.label = label
        self.grad = 0
        self._backward = lambda: None

    def __repr__(self):
        return f'Value(data={self.data})'

    def __add__(self, other):
        out = Value(self.data + other.data, prev=(self, other), op='+')
        def _backward():
            self.grad = 1.0 * out.grad
            other.grad = 1.0 * out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data, prev=(self, other), op='*')
        def _backward():
            self.grad = other.data * out.grad
            other.grad = self.data * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        x = self.data
        t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
        out = Value(t, (self,), 'tanh')
        def _backward():
            self.grad = (1.0 - out.data ** 2) * out.grad
        out._backward = _backward
        return out

Let’s go over the code.

• The default _backward function returns nothing. This will be the case for leaf nodes which are not part of any operation.
• In addition, the _backward function sets the gradient of both operands according to the chain rule, which is the local derivative (1) times the previous derivative in the chain (the derivative of out). Note that we are assuming that the derivative of out has already been calculated.
• The case for multiplication is similar, except that the local gradient is the other operand instead of 1 — other.grad for self, and self.grad for other.
• For the Tanh operation, we don’t have an other so we only need to set self.grad. Just like before, we use the local derivative of Tanh and multiply it with the derivative of out which we assume has been calculated previously.

Let’s use the _backward functions:

o.grad = 1.0
o._backward()
n._backward()
b._backward()
x1w1x2w2._backward()
x2w2._backward()
x1w1._backward()
draw_dot(o)

Out:

I will avoid pasting the graph, as it is identical to the previous case (only this time we used the _backward functions in the correct order to calculate the gradients).

Note that we had to set o.grad = 1.0 manually. Why? Because the placeholder gradient value for each node is 0. The backward function always depends on the previous gradient in the computational graph and if it was 0 then all gradients will turn out to be 0 (think about it).

But do we have to call the _backward function manually? Of course not! We can use a loop here, but notice that we need to iterate over the nodes in a certain order. We can’t call _backward on a node that has neighbors connected via outgoing edges whose gradients haven’t been calculated already. For this, we need something called topological sorting.

Let’s understand what is topological sorting using an example. Take this graph:

Here are three options to topologically sort this graph (taken from Wikipedia — https://en.wikipedia.org/wiki/Topological_sorting):

• 5, 7, 3, 11, 8, 2, 9, 10 (visual top-to-bottom, left-to-right)
• 3, 5, 7, 8, 11, 2, 9, 10 (smallest-numbered available vertex first)
• 5, 7, 3, 8, 11, 10, 9, 2 (fewest edges first)

Notice that in all sorting options, for each node u that has an outgoing edge to node v, node v will be after u in the topological sorting.

Let’s implement a simple form of topological sorting and use it to apply the _backward functions in our computational graph:

topo = []
visited = set()

def build_topo(v, topo, visited):
    if v not in visited:
        visited.add(v)
        for child in v._prev:
            build_topo(child, topo, visited)
        topo.append(v)

The above is a simple implementation of topological sorting. Using recursion we make sure that all of the offspring of a certain node will be added to the list topo before the said node. Instead of calling this function explicitly, we will make the code a bit cleaner and make it part of our Value class, such that only a single function must be called to calculate gradients for the whole computational graph.

class Value:
    def __init__(self, data, prev=(), op='', label=''):
        self.data = data
        self._prev = set(prev)
        self._op = op
        self.label = label
        self.grad = 0
        self._backward = lambda: None

    def __repr__(self):
        return f'Value(data={self.data})'

    def __add__(self, other):
        out = Value(self.data + other.data, prev=(self, other), op='+')
        def _backward():
            self.grad = 1.0 * out.grad
            other.grad = 1.0 * out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data, prev=(self, other), op='*')
        def _backward():
            self.grad = other.data * out.grad
            other.grad = self.data * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        x = self.data
        t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
        out = Value(t, (self,), 'tanh')
        def _backward():
            self.grad = (1.0 - out.data ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        self.grad = 1
        for node in reversed(topo):
            node._backward()

Now, all we need to do to calculate gradients is:

o.backward()
draw_dot(o)

Out:

Same graph as before

Now, I want to show you a slight problem we have in our current implementation of Value — what happens when a node in the computational graph gets used more than once?

Part 6. What happens when a node gets used more than once?

Take this example:

a = Value(-2.0, label='a')
b = Value(3.0, label='b')
d = a * b; d.label = 'd'
e = a + b; e.label = 'e'
f = d * e; f.label = 'f'

f.backward()
draw_dot(f)

Out:

What should be the gradients of a and b in this case? According to the multiplication node — the gradient of a should be 3 and the gradient of b should be -2 (in the multiplication of two numbers the local gradient is the other operand). According to the addition node, the gradient of both should be -6. We see that in this case the calculation was based on the addition node, but that is just because of the arbitrary topological sorting.

Why does this happen? Let’s observe the _backward function of the __add__ operation:

def _backward():
    self.grad = 1.0 * out.grad
    other.grad = 1.0 * out.grad

We see that we set the gradients of self and other, while ignoring the previous values that might have been already calculated for it. In backpropagation, the gradient of a certain node will be calculated additively — the effects of all calculations will add up.

We can fix this situation quite simply by changing the = operator in the _backward functions to += s.t the effects will add up. Like so:

def _backward():
    self.grad += 1.0 * out.grad
    other.grad += 1.0 * out.grad

After we fixed our _backward functions and run the previous code again, we get this computational graph:

We see here the correct gradients for a and b where the effects of the + and * nodes add up, instead of overriding each other.

Part 7. Adding support for constants

What will happen if we run this code?

a = Value(2.0)
a + 2

We get this error — AttributeError: ‘int’ object has no attribute ‘data’. Why? Because the __add__ function of Value is expecting another Value object. We can overcome this quite simply by performing this fix for the __add__ function (and similarly for the __mul__ function):

def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, prev=(self, other), op='+')
    def _backward():
        self.grad += 1.0 * out.grad
        other.grad += 1.0 * out.grad
    out._backward = _backward
    return out

We simply check whether other is of type Value. If not, we convert it :)

Now what happens when we run this code?

a = Value(2.0)
2 + a

Seems like it should work right? Turns out it won’t (we get this error — TypeError: unsupported operand type(s) for +: ‘int’ and ‘Value’). Python in this case is trying to run 2.__add__(a). The default __add__ function of Python obviously doesn’t know how to handle Value objects, because we made them up. How do we fix this?

For this purpose, Python supports the __radd__ function:

def __radd__(self, other):
    return self + other

The way it works is that if Python fails to call __add__ for some reason and __radd__ is available, it will swap the operands of __add__ and call __radd__. Thus, a + 2 will be called instead, and we already saw that it works. We will act accordingly with __mul__ and __rmul__, of course.

Part 8. Breaking up Tanh into its constituting parts

Currently, our Value class supports the Tanh operation as a complete block. What if we wanted to break Tanh into its constituting parts by explicitly including the Tanh calculation in a computational graph?

To achieve this we need to add several operators. Let’s remind ourselves of the Tanh implementation:

def tanh(self):
    x = self.data
    t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
    out = Value(t, (self,), 'tanh')
    return out

To break up this operation we need the following operators — exponentiation (exp), power (pow), division, and subtraction.

For division, we will implement the following:

def __truediv__(self, other):
    return self * other ** -1

The operation we just used is mathematically equivalent to self/other, but notice that we depend on the power (**) operator, which we haven’t yet implemented. Let’s do that:

def __pow__(self, other):
    assert isinstance(other, (int, float)), "only supporting pow for int/float"
    out = Value(self.data ** other, (self,), f'**{other}')
    def _backward():
        self.grad += other * (self.data ** (other - 1)) * out.grad
    out._backward = _backward
    return out

Let’s note several things about __pow__. First, we only support int or float inputs. This means we don’t support power with more complex inputs (namely arbitrary Value objects) because that would lead to different derivative rules. Second, we implement the _backward function using the calculus rule where dxⁿ/dx=nxⁿ⁻¹.

Finally, we will need to support subtraction, and we will simply utilize addition for that as so:

def __sub__(self, other):
    return self + (-other)

For this we need to support negation, so we add:

def __neg__(self):
    return self * -1

Let’s view our wholeValue class so far:

class Value:
    def __init__(self, data, prev=(), op='', label=''):
        self.data = data
        self._prev = set(prev)
        self._op = op
        self.label = label
        self.grad = 0
        self._backward = lambda: None

    def __repr__(self):
        return f'Value(data={self.data})'

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, prev=(self, other), op='+')
        def _backward():
            self.grad += 1.0 * out.grad
            other.grad += 1.0 * out.grad
        out._backward = _backward
        return out

    def __radd__(self, other):
        return self + other

    def __neg__(self):
        return self * -1

    def __sub__(self, other):
        return self + (-other)

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, prev=(self, other), op='*')
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def __rmul__(self, other):
        return self * other

    def tanh(self):
        x = self.data
        t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
        out = Value(t, (self,), 'tanh')
        def _backward():
            self.grad += (1.0 - out.data ** 2) * out.grad
        out._backward = _backward
        return out

    def exp(self):
        x = self.data
        t = math.exp(x)
        out = Value(t, (self,), 'exp')
        def _backward():
            self.grad += t * out.grad
        out._backward = _backward
        return out

    def __truediv__(self, other):
        return self * other ** -1
    
    def __pow__(self, other):
        assert isinstance(other, (int, float)), "only supporting pow for int/float"
        out = Value(self.data ** other, (self,), f'**{other}')
        def _backward():
            self.grad += other * (self.data ** (other - 1)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        self.grad = 1
        for node in reversed(topo):
            node._backward()

Now, we shall break up the Tanh function into its constituent parts, such that the whole calculation will appear in the computational graph:

x1 = Value(2.0, label='x1')
x2 = Value(0.0, label='x2')
w1 = Value(-3.0, label='w1')
w2 = Value(1.0, label='w2')
b = Value(6.8813735870195432, label='b')

x1w1 = x1*w1; x1w1.label = 'x1*w1'
x2w2 = x2*w2; x2w2.label = 'x2*w2'
x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'
n = x1w1x2w2 + b; n.label = 'n'

e = (2*n).exp()
o = (e - 1) / (e + 1)

o.backward()
draw_dot(o)

Out:

Great! We now have a much larger computational graph and can see all the components of the Tanh function. Notice that the gradients for w1 and w2 are the same as before, thus we know that this graph is operationally identical to the previous (where the Tanh operation was implicit).

What do we learn from this exercise? In backpropagation and neural network training, we can implement an operation as abstract as we want. As long as we can define the forward and backward pass by calculating the local gradient for such an operation it doesn’t matter that it by itself is composed of other operations (think of the fact that multiplication can be described by many additions).

Part 9. Comparing Micrograd to Pytorch

Up until now, we developed our own auto-differentiable computation graph class — Value. We validated that it is working as expected and that we can back-propagate through arbitrarily complex graphs as long as they are made of supported building blocks.

Despite this, our implementation is very naive computation-wise and there are better and faster libraries out there that do just this. One example is the popular Pytorch library, developed by Meta. The Pytorch library allows the creation of highly efficient computational graphs which can be parallelized and optimized for different hardware including CPUs, GPUs, and TPUs. It is used extensively in industry and cutting-edge research — mostly for neural networks.

To validate our implementation of Micrograd we will implement the exact same computational graph from before in Pytorch and confirm that the results are identical.

import torch
x1 = torch.Tensor([2.0]).double(); x1.requires_grad = True
x2 = torch.Tensor([0.0]).double(); x2.requires_grad = True
w1 = torch.Tensor([-3.0]).double(); w1.requires_grad = True
w2 = torch.Tensor([1.0]).double(); w2.requires_grad = True
b = torch.Tensor([6.8813735870195432]).double(); b.requires_grad = True

n = x1*w1 + x2*w2 + b
o = torch.tanh(n)

print(o.data.item())
o.backward()

print('----')
print('x2', x2.grad.item())
print('w2', w2.grad.item())
print('x1', x1.grad.item())
print('w1', w1.grad.item())

Out:

0.7071066904050358
----
x2 0.5000001283844369
w2 0.0
x1 -1.5000003851533106
w1 1.0000002567688737

Great! We got the exact same gradients as in our own implementation.

The default case for Pytorch is to work with tensors which are multi-dimensional arrays. Hence the need to initialize our parameters as 1-value tensors (for example Tensor([2]) instead of Tensor(2)). We see that we can use arithmetic operations like * and +, and also more complex operations such as torch.tanh(...). Just like in Micrograd, we ran .backwards() on our result node and by doing that ran backpropagation through every node in our graph. We needed to explicitly set requires_grad=True for each node because Pytorch does not calculate the gradients for leaf nodes by default.

Part 10. Building a multi-layer perceptron (MLP)

Using our main building block — the Value class, we are now going to build our first neural network! We will write a class that will allow us to create simple MLPs. Let’s first create one and then talk about how we use backpropagation to optimize the weights for a specific task.

class Neuron:
    def __init__(self, nin, layer_idx, neuron_idx):
        self.layer_idx = layer_idx
        self.neuron_idx = neuron_idx
        self.w = [Value(random.uniform(-1, 1), label=f'w^{layer_idx}_({neuron_idx}{i+1})') for i in range(nin)]
        self.b = Value(random.uniform(-1, 1), label=f'b^{layer_idx}_{neuron_idx}')
    
    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        out = act.tanh()
        out.label = f'L_{self.layer_idx}{self.neuron_idx}'
        return out

    def parameters(self):
        return self.w + [self.b]
    
class Layer:
    def __init__(self, nin, nout, layer_idx):
        self.neurons = [Neuron(nin, layer_idx, neuron_idx=i+1) for i in range(nout)]
    
    def __call__(self, x):
        outs = [n(x) for n in self.neurons]
        return outs[0] if len(outs) == 1 else outs
    
    def parameters(self):
        return [p for neuron in self.neurons for p in neuron.parameters()]

class MLP:
    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1], layer_idx=i+1) for i in range(len(nouts))]
    
    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x
    
    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

Surprisingly, this is all of the code!

• Class Neuron contains code for a computational unit that receives nin inputs and outputs a weighted sum of them plus a bias term, followed by a Tanh operation.
• Class Layer aggregates several neurons such that it takes certain data as input (of size nin) and passes it through nout neurons. We do a small check in the end where if the output contains only one neuron we return it as-is and not in a list.
• Class MLP is the final abstraction in which we aggregate noutinstances of Layerby passing the input data xthrough each of them in succession.
• I added some code that did not appear in the original tutorial by Andrej to display the labels of the weights and neurons so we can make some sense in the drawings we will create later.
• For each class, we added a parametersfunction that collects all trainable parameters. This will come in handy later when we train the network.

Let’s create an MLP, specifically — this one:

Let’s just go over the notation here:

Our MLP contains 3 inputs, 2 hidden layers of size 2 each, and one output.

x = [Value(2.0, label='in1'),
     Value(3.0, label='in2'),
     Value(-1.0, label='in3')]
n = MLP(3, [2, 2, 1])
draw_dot(n(x))

Out:

The full computational graph is much more detailed than the simplistic drawing we made of the network because it contains all of the multiplication, addition, and Tanh operations in the network. If we look closely though, we can see that the labels in the detailed drawing make sense. For example — find layer L_11 in the drawing and backtrack to its inputs. You will see that it gets the sum of in1 * w¹_(11), in2 * w¹_(12), in3 * w¹_(13) and b¹_1 as inputs — just like our simplistic drawing from before (only slightly less organized).

Now comes the question — how do we use this neural network? Can we train it? Of course, the answer is — yes! Let’s define some input data points and their respective labels. Our goal will be to train the network to generate the correct label for each input.

xs = [
    [Value(2.0, label='in11'),
     Value(3.0, label='in12'),
     Value(-1.0, label='in13')],
    [Value(3.0, label='in21'),
     Value(-1.0, label='in22'),
     Value(0.5, label='in23')],
    [Value(0.5, label='in31'),
     Value(1.0, label='in32'),
     Value(1.0, label='in33')],
    [Value(1.0, label='in41'), 
     Value(1.0, label='in42'),
     Value(-1.0, label='in43')]
]
ys = [1.0, -1.0, -1.0, 1.0]
ypred = [n(x) for x in xs]
ypred

Out:

[Value(data=0.6159125848021643, label=L_31),
 Value(data=0.6974854432825526, label=L_31),
 Value(data=0.6258725834364298, label=L_31),
 Value(data=0.6165492639086749, label=L_31)]

We defined 4 input samples and a corresponding label for each of them. Then, we ran theMLP we created previously to generate the output for each sample. We see that our initial guesses are quite off (instead of [1, -1, -1, 1] we got different values). In order to train a neural network we need a loss function. This is a mathematical formula that will be used to estimate the quality of a network's outputs. We will use the well know mean squared error:

loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))
loss

Out: Value(data=5.819476097248748)

In mean squared error, we take the squared difference between each prediction and the corresponding label and sum them all up. The lossobject that we created represents this sum. If we backpropagate through lossusing backward, we will generate the gradient for each one of the network parameters. This gradient will indicate in which direction each parameter must be nudged in order to minimize loss which is our goal. Just for fun — let’s draw the graph represented by loss after we call loss.backward().

loss.backward()
draw_dot(loss)

Out:

This graph is insanely packed. It contains in it all 4 forward passes — one for each input! We can see, though, that each one of our network parameters appears here only once, and each has a gradient that now tells how we should nudge it to minimize the loss. Just as an example, let’s look at the gradient of the first weight of the first neuron of the first layer: n.layers[0].neurons[0].w[0].grad: 0.05664531168775873

The gradient is positive. This means that to minimize the loss function we need to go in the negative direction of the gradient, i.e. decrease the value of W¹_(11).

Part 11. Training the MLP

Finally! We are going to train our neural network to fit the dummy data points we showed before. To train our net we need to repeat the forward pass -> loss calculation -> backward pass procedure. We can do it manually, but it makes much more sense to write a simple loop that does this for us:

def training_loop(n_steps, learning_rate):
    for k in range(n_steps):
        # forward pass
        ypred = [n(x) for x in xs]
        loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

        # backward pass
        loss.backward()

        for p in n.parameters():
            p.data -= learning_rate * p.grad
        
        print(k, loss.data)

What is happening here? First, we generate ypredwhich is the forward pass for our 4 examples. Then we generate the loss term which we will calculate the gradients by. Then, we call backwardto calculate gradients for all parameters of the net. Finally, we iterate through all of the network parameters using the parameters()function from the Valueclass and nudge each of them by a small amount (controlled using the specified learning_rate) in the negative direction of the gradient so as to make the loss smaller.

Let’s just make sure that n.parameters()gives us the expected output:

[Value(data=0.45027414801422605, label=w^1_(11)),
 Value(data=-0.8129525057615291, label=w^1_(12)),
 Value(data=0.5758542981989572, label=w^1_(13)),
 Value(data=-0.8030667986239539, label=b^1_1),
 Value(data=0.5557411827449444, label=w^1_(21)),
 Value(data=-0.6721847840083467, label=w^1_(22)),
 Value(data=1.217737844619453, label=w^1_(23)),
 Value(data=0.09905283405117085, label=b^1_2),
 Value(data=0.4829801438596318, label=w^2_(11)),
 Value(data=0.7369938633453851, label=w^2_(12)),
 Value(data=-0.46073515053706376, label=b^2_1),
 Value(data=0.27929425462417923, label=w^2_(21)),
 Value(data=1.7138391095399366, label=w^2_(22)),
 Value(data=0.7105401730722122, label=b^2_2),
 Value(data=-2.378848749014902, label=w^3_(11)),
 Value(data=-0.8342902851970706, label=w^3_(12)),
 Value(data=-1.2437877048443844, label=b^3_1)]

Great! We see here all of the parameters we wish to train, and we don’t see things we don’t wish to train (such as the input data which is assumed to be fixed).

We are ready to run the training loop using training_loop(n_steps=20, learning_rate=0.01)

Out:

0 5.819476097248748
1 5.624553232066405
2 5.339641605929825
3 5.0032420663990465
4 4.6979176273972465
5 4.50430643973165
6 4.41612726370955
7 4.345437217507974
8 4.191726524841689
9 3.8794059504102045
10 3.3966830048541388
11 2.8554032260566666
12 2.4577023357202674
13 2.274646098166827
14 2.176592658900446
15 2.0400253245956486
16 1.8089353320227564
17 1.3966145142070974
18 0.748901313440497
19 0.17987272335400104

ypred = [n(x) for x in xs]
ypred

[Value(data=0.9312475693757956, label=L_31),
 Value(data=-0.9983175660114463, label=L_31),
 Value(data=-0.895522396196552, label=L_31),
 Value(data=0.9153783867936369, label=L_31)]

Our training procedure works and our ypredis very close to the ground truth. With a bit more training steps we would have gotten even closer.

But wait…there is a bug here! (in the training loop) Did you spot it? Take a moment to think…

The bug is subtle but quite common among new neural network practitioners. Before each backward pass, we forgot to clear the gradient information from the computational graph. Remember that as we calculate the gradients for the different components of the graph we use +=instead of =to accumulate gradient information, because a certain node may be used more than once in the graph and to get its gradient we need to sum up all of the contributions to the gradient.

In the current implementation we forgot to clear the gradient value, so the gradient will get accumulated through subsequent forward passes — which were performed with different network weights! This makes no sense, of course, so a simple fix is in order:

def training_loop(n_steps, learning_rate):
    for k in range(n_steps):
        # forward pass
        ypred = [n(x) for x in xs]
        loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

        
        for p in n.parameters():
            p.grad = 0.0
            
        # backward pass    
        loss.backward()

        for p in n.parameters():
            p.data -= learning_rate * p.grad
        
        print(k, loss.data)

Before the backward pass, we iterate through all network parameters and assign a value of 0 to them.

In this toy example, it didn’t really matter, and it even helped the model converge faster because the problem is very easy, and accumulating the gradients was akin to using a massive learning rate (I needed several hundred training steps instead of 20 to reach a similar state of loss), but it is indeed a bug and could lead to crazy results in larger networks.

Let us run the training loop again after the fix. We will call training_loop(20, 0.01).

Out:

0 5.108927684153838
1 4.846717266169518
2 4.60037368477449
3 4.383622169945541
4 4.205267359786554
5 4.067195989790671
6 3.965330834751441
7 3.8924147774168065
8 3.840719199945062
9 3.803632051028429
10 3.7761592608770185
11 3.7548059106373577
12 3.7372392224767497
13 3.7219462205564553
14 3.707961320704009
15 3.6946723746238628
16 3.681690631213215
17 3.668766548696825
18 3.655736427317908
19 3.6424890071677454

Unlike before, our training is much slower. This is due to fixing the bug in which the gradients accumulated on each other in each training step instead of resetting. After about 1000 steps our output looks like this:

[Value(data=0.9653433264848924, label=L_31),
 Value(data=-0.9720009732525263, label=L_31),
 Value(data=-0.9618275470088311, label=L_31),
 Value(data=0.9694722216419711, label=L_31)]

Very close to the ground truth.

Let’s now wrap up everything we learned here.

Part 12. Summary

Which question did we answer today?

• What are neural networks? They are mathematical expressions that take weights and input data as parameters and generate some output. The quality of this output is measured via a loss function, which we then use to calculate gradients for all of the network weights (using the chain rule). To minimize the loss function we nudge each parameter by a small amount in the negative direction of the gradient. Hopefully, after many steps of this process, we will be able to minimize the loss function enough, and this means that our network will generate outputs similar to the expected results.

We created a very basic neural network in this article, for simplicity’s sake, but these networks can get much larger and perform insanely difficult tasks! Such as creating Python code given a free text description, making medical diagnoses given patient records, and much more!

The incredible thing is that the underlying concepts behind these massive and impressive neural nets are exactly the same as the simple example we showed here today. Congratulations on understanding some pretty important things!

The notebook containing the code that was shown here — https://colab.research.google.com/drive/1lG93TNxmdHFLb7xiZquzTRDF5hFcVQkV?usp=sharing

References

• https://github.com/karpathy/micrograd — The full version of Micrograd by Andrej Karpathy.
• https://www.youtube.com/watch?v=VMj-3S1tku0 — “The spelled-out intro to neural networks and backpropagation: building micrograd” — The video on which this blog post is based on.
• https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd — Notebooks from Andrej’s video.
• https://colab.research.google.com/drive/1lG93TNxmdHFLb7xiZquzTRDF5hFcVQkV?usp=sharing — Notebook I created in the process of making this blog post.
• https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing — Notebook by Andrej containing follow-up exercises to this lesson.

A paper walkthrough of the new text-to-speech model by Microsoft Research

Hello readers,

In this article, we will dive deep into a new and exciting text-to-speech model developed by Microsoft Research, called VALL-E. The paper presenting the work has been released on Jan. 5, 2023, and since then has been gaining much attention online. It is worth noting that as of writing this article, no pre-trained model has been released and the only option currently to battle-test this model is to train it by yourself.

Nevertheless, the idea presented in this paper is novel and interesting and worth digging into, regardless of whether I can immediately clone my voice with it or not.

This article will be organized as follows:

• Part 1 — Introduction to Text to Speech, Basic Concepts
• Part 2 — VALL-E: Text to Speech as a Language Model
• Part 3 — Encodec: The Workhorse Behind VALL-E
• Part 4 — Problem formulation and training of VALL-E
• Part 5— Some Coding
• Part 6— Conclusions & Thoughts Ahead

Part 1 — Introduction to Text to Speech, Basic Concepts

The technology of text-to-speech is not new and has been around since the “Voder” — the first electronic voice synthesizer from Bell Labs in 1939 which required manual operation. Since then, the field has seen incredible developments and up until ~2017, the dominant technology was concatenative speech synthesis. This technology is based on the concatenation of pre-recorded speech segments to create intelligible speech. Although this technology can produce lifelike results, its drawbacks are obvious — it cannot generate new voices which don’t exist in the pre-recorded database, and it cannot generate speech with a different tone or emotion.

Fast-forward to the era of deep learning. Nowadays, the dominant strategy in text-to-speech synthesis is summarized in Figure 1. Let’s go over its different parts.

• First, we have a phonemizer that transforms text into phonemes. Phonemes are a textual representation of the pronunciation of words (for example — the word tomato will have different phonemes in an American and British accent), and this representation helps the downstream model achieve better results.
• Afterward, we have an acoustic model which transforms these phonemes into a Mel spectrogram, which is a representation of audio in the time X frequency domain. A spectrogram is achieved by applying a short Fourier transform (STFT) on overlapping time windows of a raw audio waveform (here is an excellent explanation about the Mel spectrogram — https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53). Of course in this case the spectrogram is being created by a statistical model, as no input audio exists in real-time text-to-speech. Examples of recent model architectures include Tacotron2, DeepVoice 3, and TransformerTTS.
• The final stage is the conversion of the Mel spectrogram into a waveform. A waveform is usually sampled at 24/48 kHz, where each sample is digitized into a 16-bit number. These numbers represent the amount of air pressure at each moment in time, which is the sound we eventually hear in our ears. Why can’t we just deterministically convert the spectrogram into a waveform? Because it requires major upsampling in the time domain which requires us to create information that doesn’t explicitly exist in the spectrogram, and also because spectrograms don’t contain phase information (only frequency). So, as in the conversion of phonemes to a Mel spectrogram, here as well we need a statistical model to convert the spectrogram into a waveform and these models are called Vocoders. Examples of Vocoders include WaveNet, WaveRNN, and MelGAN.

Additionally, there are recent models such as VITS and YourTTS, which employ an end-to-end model to generate waveforms from text input. Another example of such an end-to-end system is a paper titled “End-to-End Adversarial Text-to-Speech” by Deepmind (which is excellently explained by Yannic Kilcher here — https://www.youtube.com/watch?v=WTB2p4bqtXU). In this paper, they employ a GAN-like training procedure to produce realistic speech sound waves. They also need to tackle the alignment problem, which is the degree to which word utterances in the generated samples align in time with the same word utterances in the ground truth samples. This problem does not “solve on its own” and requires explicit handling in the model architecture.

The main drawback of these end-to-end TTS models is their incredible complexity. Text and speech are such different modalities, and this requires complex models which tackle problems such as alignment, speaker identity, and language in an explicit manner, making these models highly complex. The charm of VALL-E, which we will soon dive into, is that it takes the relative simplicity of generative language models and employs them creatively in the field of speech generation. For people like me who are new to the field of TTS and speech in general and have some experience in NLP, it allows a good entry point into this fascinating field.

This short overview did not do justice to the immense field of TTS, which one can spend a lifetime studying and understanding (I do encourage you to dive a bit deeper). Yet, we are here today to talk about VALL-E, so allow me to jump straight to it.

Part 2 — VALL-E: Text to Speech as a Language Model

As in other text-to-speech systems, the input to VALL-E is phonemicized text, and the output is the corresponding sound waveform. Additionally, VALL-E employs a prompting mechanism in which a 3-second audio sample is fed as additional input to the model. This allows the generation of a speech utterance of the input text which is conditioned on the given audio prompt — in practice, this means the ability to perform zero-shot speech generation, which is the generation of speech from a voice unseen in the training data. The high-level structure of VALL-E is presented in Figure 2.

Let’s understand what happens in this pipeline. First, we have a phoneme conversion of the text, which is standard procedure as we understood already, and doesn’t require any learning mechanism. In order to process these phonemes by the model we have a phoneme embedding layer that takes as input a vector of indices into the phoneme vocabulary and outputs a matrix of embeddings corresponding to the input indices.

The 3-second acoustic prompt, which the output speech is conditioned on, is fed into an audio codec encoder. In VALL-E they use a pre-trained audio encoder for this — Encodec (developed by Facebook Research — https://arxiv.org/abs/2210.13438). Encodec takes as input a waveform of speech and outputs a compressed discrete representation of it via recursive vector quantization (RVQ) using an encoder-decoder neural architecture. We will dive into Encodec in Part 3 of this article, but for now, let’s just assume that it outputs a discrete representation of the audio signal by splitting it into fixed time windows and assigning each window a representation from a known vocabulary of audio embeddings (conceptually, very similar to word embeddings).

Once the model receives these two inputs, it can act as an autoregressive language model and output the next discrete audio representation. Because the audio representations come from a fixed which was learned by Encodec, we can think of this simply as predicting the next word in a sentence out of a fixed vocabulary of words (a fixed vocabulary of sound representations, in our case). After these sound representations are predicted they are transformed back into the original waveform representation using the Decoder part of the Encodec model.

In Figure 3 we compare the pipeline of VALL-E to the traditional neural TTS pipeline. We see that the main difference is the intermediate representation of audio. In VALL-E they gave up on the Mel spectrogram and used the representation created by the Encodec model. It is worth noting though that under the hood Encodec uses a spectrogram representation as well, so it is still somewhat in use in this architecture, albeit less prominently.

In the results section of the VALL-E paper, they have shown that they outperform the previous state-of-the-art zero-shot TTS model, YourTTS, on the LibriSpeech data on several metrics which include human-based evaluations such as similarity mean option score (SMOS) and algorithm-based evaluations such as word error rate (WER). In an interesting ablation study, they show that the phoneme prompt contributes to the content of generation (by reducing WER) and the audio prompt contributes to speaker similarity (by improving a speaker similarity metric).

We will now dive into the Encodec model which is responsible for converting audio to discrete tokens and back and is the enabler for using a language model approach to audio generation in this paper.

Part 3 — Encodec: The Workhorse Behind VALL-E

In Figure 4 we can see the Encodec architecture. It is an encoder-decoder architecture that learns a condensed representation of the audio signal via the task of reconstruction. Let’s go over its different parts to understand what is going on under the hood.

On the far left we have our original waveform, which is sampled at 24/48 kHz, and each sample is represented by 16 bits (65536 options). The raw signal gets passed into the Encoder which includes 1D convolution operations for downsampling and a two-layer LSTM for sequence modeling. The output of the encoder is 75/150 latent timesteps (compare this to the original 24/48K!), with a depth dimension of 128.

The decoder is simply a mirrored version of the encoder, using transposed convolutions in order to upsample the latent space and construct the audio waveform (here is a good explanation on transposed convolutions https://towardsdatascience.com/what-is-transposed-convolutional-layer-40e5e6e31c11).

The interesting bit here is, of course, the quantizer. How does Encodec quantize the continuous domain of sound? Using a technique called residual vector quantization (RVQ) which consists of projecting an input vector onto the closest entry in a codebook of a given size. Let’s break that sentence down. First, what is a codebook?

In the case of VALL-E, a codebook is a dictionary of vectors of size 1024, where each entry represents a vector of size 128. Our goal in vector quantization is to map a certain vector to the closest vector in the codebook (by Euclidean distance, for example), thereafter it can be represented by the index of that vector in the codebook (assuming everyone has access to the codebook). Of course, in this way, we lose a lot of information. What if no vector in the codebook accurately resembles our vector? Hence the “residual” in RVQ!

In Figure 5, I show how a vector is quantized using residual vector quantization. In the example, we have 3 codebooks. The input vector is compared to each of the vectors in the first codebook and assigned to the closest one (C1,1). Then, the residual between the C1,1 and the input is calculated and we try to match the residual to the next codebook, and so on until we reach the end of our codebooks. The final RVQ representation is the indices that were matched in each of the codebooks (1, 3, 2 in our example). This encoding method is extremely efficient. If we have 8 codebooks where each contains 1024 entries — we can represent 1024⁸=1.2e+24 different vectors using only 1024*8=8192 numbers! Of course, the sender and receiver must hold the same codebook for this quantization method to work. If you want to learn more about RVQ, such as how the codebooks are trained, I recommend reading another paper that Encodec is based on called SoundStream — https://arxiv.org/abs/2107.03312 (yes, this is a rabbit hole).

Back to the Encodec pipeline in Figure 4, let’s notice 3 additional details which are relevant to its training procedure:

1. Mel spectrograms are created both from the input audio and from the generated audio. These spectrograms are compared and the signal from the comparison is used as a loss to direct the model training.
2. Several discriminators are used in order to compare a short-time Fourier transform (STFT) of the original and synthetic waveform. This GAN loss gives a different signal than the Mel spectrogram comparison and was found useful for Encodec.
3. The quantizer contains transformers that are used for further compression of the audio signal. This is not the transformer in VALL-E that predicts the next token of speech, as confusing as it may be. For further understanding of the transformers in Encodec, I recommend reading the paper or watching the video by Aleksa Gordic — https://www.youtube.com/watch?v=mV7bhf6b2Hs.

Let’s summarize what we know so far. VALL-E is a text-to-speech model that resembles language models in its operational mode, such that it predicts the next discrete audio token for a given prompt, which consists of phonemicized text and audio input. These discrete tokens are learned by another model called Encodec (which itself is based on SoundStream) that uses an encoder-decoder architecture with residual vector quantization to convert audio into discrete codes.

Part 4 — Problem formulation and training of VALL-E

VALL-E contains two transformer models which are used to process the input data (phonemicized text and audio) — an autoregressive (AR) transformer that attends only to past data, and a non-autoregressive (NAR) transformer that attends to all points in time. Let’s see why.

Eight different codebooks are used in VALL-E as part of the Encodec model, where each codebook consists of 1024 entries. The codes from the first quantizer (codebook) are processed by the AR model according to this Equation 1. Let’s first clarify some terminology here:

• C represents the generated output — as discrete audio codes
• C~ is the 3-second input acoustic prompt
• x is the input text as a phoneme sequence
• C:,₁ represents data from the first quantizer/codebook for C

So, Equation 1 shows that the output for the first quantizer is conditioned on the input data, and on the previous timesteps’ outputs for the first quantizer (just like an autoregressive language model).

In Equation 2 we see the generation of codes for quantizers 2 to 8. Unlike the previous case, here the output for each quantizer is conditioned on all of the timesteps from the previous quantizers (when calculating codes for quantizer #7, the model is conditioned on data generated for quantizers 1 to 6). Unlike the AR model, this allows parallel generation of all timesteps in a single quantizer because it is only dependent on previous quantizer codes and not on previous timesteps of the same quantizer. The authors stressed this point because fast inference is especially important in text-to-speech models which need to generate speech in real-time scenarios.

Equations 1 and 2 are visually depicted in Figure 6, which shows the AR and NAR models together and highlights the differences between them. We see that the AR transformer is used to predict only C:,₁ which are the tokens for the first quantizer. While doing so, it attends to the previous tokens it has generated. the NAR transformer attends to the previous quantizers, and not to previous timesteps (the previous tokens of the current quantizer are not available in the NAR model).

VALL-E has been trained on 60K hours of audio from the LibriLight dataset, containing 7000 distinct speakers (which is over 100 times more data than previous state-of-the-art). The dataset is audio-only, hence for labeling an automatic speech recognition model was used. The Encodec model is used as a pre-trained model and no fine-tuning is performed on it for VALL-E as far as I could understand.

For training, random 10–20 second samples were taken from LibriLight. For the acoustic prompt, another 3 seconds were taken from the same utterance. They used 16 Tesla V-100 GPUs to train the model, which is very modest compared to large SOTA language models!

We learned about the procedure and data, now let’s try to use the unofficial Pytorch implementation of VALL-E in GitHub.

Part 5 — Some Coding

VALL-E doesn’t have an official implementation on GitHub, so for my experimentations, I will rely on the unofficial version which was released — https://github.com/enhuiz/vall-e. Moreover, no model checkpoint has been released so you have to train it from scratch.

There is also a Google Colab notebook to follow along with a simple training example — https://colab.research.google.com/drive/1wEze0kQ0gt9B3bQmmbtbSXCoCTpq5vg-?usp=sharing. In this example, they overfit the model on a single utterance of “hello world” and they show that the model is able to reproduce this single utterance. I was interested in two things:

1. I wanted to replicate their “hello world” experiment with my own voice, just to see that the pipeline is working properly
2. I wanted to replicate the experiment done by James Skelton from Paperspace — https://blog.paperspace.com/training-vall-e-from-scratch-on-your-own-voice-samples/, where he trained a model on a very small subset of his own recordings, and managed to replicate his voice with it (on something he already recorded)

Why the limited experiments? Because training this model from scratch takes many resources which I don’t currently have, plus I assume a pre-trained model will be released sooner or later.

So how did I succeed? I managed to replicate the “hello world” experiment, but unfortunately, I didn’t manage to replicate the Paperspace experiment — I just got a model which creates a garbled sound that vaguely reminds my voice. This is probably because of a lack of resources (I am training it on a Google Colab instance which times out after 12 hours). But still, I want to go over the process with you. My version of the VALL-E notebook is here — https://colab.research.google.com/drive/1NNOsvfiOfGeV-BBgGkwf0pyGAwAgx3Gi#scrollTo=SbWtNBVg_Tfd.

Once you run the following line in the Colab notebook —

!git clone --recurse-submodules https://github.com/enhuiz/vall-e.git

You will see a directory called vall-e in your file browser. The path content/vall-e/data/testcontains the data for the “hello world” experiment. Notice that it contains two files because for some reason it breaks with only one. To replicate this experiment, simply delete the files in the data directory using !rm content/vall-e/data/test/*, record yourself saying “Hello world”, and save it as two .wav files with different names. Put the .wav files in the data directory including two text files containing the words “hello world” (the text files should have the same names as the .wav files with a.normalized.txt suffix).

Following that, you will run these two cells:

!python -m vall_e.emb.qnt data/test

!python -m vall_e.emb.g2p data/test

The first cell will run the Encodec model on your own data and perform quantization, just as we discussed earlier. The second cell will convert the text “hello world” into phonemes.

Afterward, the processed data is ready and you can run the cells that operate the training procedure. There is separate training for the NAR and AR models (remember that as we saw earlier, the NAR model training is dependent on the AR model, but the AR model uses and produces only the first quantizer data, thus independent of the NAR model).

!python -m vall_e.train yaml=config/test/ar.yml

!python -m vall_e.train yaml=config/test/nar.yml

After the model has finished training you will run this cell:

!mkdir -p zoo
!python -m vall_e.export zoo/ar.pt yaml=config/test/ar.yml
!python -m vall_e.export zoo/nar.pt yaml=config/test/nar.yml

Which saves the latest model checkpoint (that has been automatically created) into a directory called zoo.

Finally, you will perform inference with the model using:

!python -m vall_e 'hello world' /content/vall-e/data/test/hello_world.wav toy.wav --ar-ckpt zoo/ar.pt --nar-ckpt zoo/nar.pt

This will run the model with a text prompt of “Hello world”, and an audio prompt of the same utterance. It will save the generated sample as toy.wav which you can then listen to using:

from IPython.display import Audio
Audio('toy.wav')

And that’s it! You created your own VALL-E “Hello world”. Unless you have many computing resources, it is probably best to wait for a pre-trained model to come around to actually make use of this model further.

Part 6 — Conclusions & Thoughts Ahead

In this article, we saw VALL-E, a new text-to-speech architecture by Microsoft Research. VALL-E generates audio in a language-model-like manner, which differentiates it from recent state-of-the-art methods that are usually end-to-end or follow a text->spectrogram->waveform creation pipeline.

We also talked about the Encodec model, which performs audio quantization and is used as a pre-trained model in the training of VALL-E. Encodec is fascinating in itself and manages to create super-condensed audio representations using residual vector quantization. The creators of VALL-E leveraged this feature and built a generative “language” model on top of this quantization.

Finally, we saw some code and replicated the “hello world” experiment from the unofficial code with our own voice. The official code for this paper hasn’t been released, nor has a model checkpoint been released. It would be interesting to see and use a pre-trained model for VALL-E, which I assume will turn up sooner or later. Nevertheless, this was an interesting learning journey.

See you next time!

Elad

References

[1] https://arxiv.org/abs/2301.02111 — The VALL-E paper (Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers)

[2] https://arxiv.org/abs/2301.02111 — The Encodec paper (High Fidelity Neural Audio Compression)

[3] https://wiki.aalto.fi/display/ITSP/Concatenative+speech+synthesis — Explanation on concatenative speech synthesis

[4] https://www.youtube.com/watch?v=aLBedWj-5CQ&t=1s — Deep dive into speech synthesis meetup (HuggingFace)

[5] https://www.youtube.com/watch?v=MA8PCvmr8B0 — Pushing the frontier of neural text to speech (Microsoft Research)

[6] https://www.youtube.com/watch?v=G9k-2mYl6Vo&t=5593s — Excellent video by John Tan Chong Min about VALL-E

[7] https://www.youtube.com/watch?v=mV7bhf6b2Hs — Excellent video by Aleksa Gordic about Encodec

VALL-E — The Future of Text to Speech? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi all,

I am sharing a short project I worked on that involves improving the performance of the text-davinci-003 model by OpenAI using intelligent prompting. I will start by saying that this work is inspired by the excellent video tutorials by James Briggs — (specifically this one — https://youtu.be/dRUIGgNBvVk) and much of the code I used is taken from his examples as well.

The outline of this article is as follows:

1. Intro — Unique and Proprietary data in Large Language Models (LLMs)
2. Problem Statement — Answering Questions in Hebrew
3. Solution Suggestion — Prompting & Querying
4. Experiment — Question Answering in Hebrew
5. Conclusions

A Google Colab notebook with the complete code is here — https://colab.research.google.com/drive/1_UqPHGPW1yLf3O_BOySRjf3bWqMe6A4H#scrollTo=4DY7XgilIr-H

Part 1. Problem Statement — Unique and Proprietary data in Large Language Models

The main official language in Israel is Hebrew, and as an Israeli, I am interested in improving the performance of large language models in Hebrew. It is a widely known fact that the performance of ChatGPT in Hebrew could be better relative to its performance in English, despite recent improvements. In this article, though, Hebrew is only an allegory to a source of data that is under-represented in a large language model and this article is going to deal with an attempt to assist an LLM in yielding useful results which are based on such data.

One option, which was common before the era of very large language models, was to perform additional training on the model with the new data/task. When the models were at a scale of millions of parameters this option was a viable approach for many people and use cases and was achievable using common consumer hardware.

In the past several years there has been an explosion in model sizes for large language models, which nowadays reach hundreds of billions of parameters. On the other hand, consumer hardware has not seen such a massive efficiency explosion. Thus, the option for fine-tuning these models has been taken off the table for most users and organizations.

ChatGPT is widely known already and it achieves remarkable performance on many tasks and is proving to be an invaluable AI assistant. Yet still, if I want to use this technology to assist a person speaking in a rare language, or to assist a company with proprietary data which is not available online — chances are that ChatGPT will fail for obvious reasons of not seeing the data beforehand. So how do we make use of special/proprietary data in these models, without requiring millions of dollars for training using special hardware? One option is prompting via querying, which we will explore today.

Part 2. Problem Statement — Answering Questions in Hebrew

We will use the OpenAI API to answer three questions in Hebrew, which were inspired by the SQuAD dataset and translated into Hebrew. The following code snippet shows the questions, expected answers, and English translations.

questions_answers = [
     {'subject': 'אוסטרליה',
      'subject_translation': 'Australia',
      'question': "איזו עיר בויקטוריה נחשבת לבירת הספורט של אוסטרליה?",
      'answer': "מלבורן",
      'question_translation': "Which city in Victoria is considered the sporting capital of Australia?",
      'answer_translation': "Melbourne"},
     {'subject': 'אוניברסיטה',
      'subject_translation': 'University',
      'question': "איזה נהר ממוקם בקרבת אוניברסיטת הארווארד?",
      'answer': "נהר צ'ארלס",
      'question_translation': "Which river is located in the vicinity of Harvard University?",
      'answer_translation': "Charles River"},
     {'subject': 'המוות השחור',
      'subject_translation': 'The black plague',
      'question': "כמה אנשים ברחבי העולם נספו בעקבות המוות השחור?",
      'answer': "בין 75 ל200 מיליון איש",
      'question_translation': "How many people in the world died from the black plague?",
      'answer_translation': "Between 75 and 200 million"},
]

Disclaimer — the recent version of ChatGPT proves to yield very good answers for these queries (in Hebrew). The point of this article is not to “compete” with it, but rather to show a solution to a problem that might arise with other models/data.

We will use the text-davinci-003 model to answer the questions using this code, which will call the OpenAI API:

from googletrans import Translator
import openai

def complete(prompt):
    # query text-davinci-003
    res = openai.Completion.create(
        engine='text-davinci-003',
        prompt=prompt,
        temperature=0,
        max_tokens=400,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
        stop=None
    )
    return res['choices'][0]['text'].strip()

def print_question_answer(question, query, expected_answer, question_translation, expected_answer_translation):
  translator = Translator()
  answer = complete(query)
  print(f'Question: {question}\n' \
        f'Question (translation): {question_translation}\n' \
        f'Answer: {answer}\n' \
        f'Answer (translation): {translator.translate(answer).text}\n' \
        f'Expected Answer: {expected_answer}\n' \
        f'Expected Answer (translation): {expected_answer_translation}'
        )

for i, qa_dct in enumerate(questions_answers):
  print(f'####################')
  print(f'Question #{i + 1}:')
  print_question_answer(qa_dct['question'],
                        qa_dct['question'],
                        qa_dct['answer'],
                        qa_dct['question_translation'],
                        qa_dct['answer_translation'],
                        )

Output:

####################
Question #1:
Question: איזו עיר בויקטוריה נחשבת לבירת הספורט של אוסטרליה?
Question (translation): Which city in Victoria is considered the sporting capital of Australia?
Answer: סידני.
Answer (translation): Sydney.
Expected Answer: מלבורן
Expected Answer (translation): Melbourne
####################
Question #2:
Question: איזה נהר ממוקם בקרבת אוניברסיטת הארווארד?
Question (translation): Which river is located in the vicinity of Harvard University?
Answer: הנהר הקרוב ביותר לאוניברסיטת הארווארד הוא נהר הקונגו.
Answer (translation): The closest river to Harvard University is the Congo River.
Expected Answer: נהר צ'ארלס
Expected Answer (translation): Charles River
####################
Question #3:
Question: כמה אנשים ברחבי העולם נספו בעקבות המוות השחור?
Question (translation): How many people in the world died from the black plague?
Answer: בעקבות המוות השחור של ג'ורג' פלאי, הפגנו אלפי אנשים ברחבי העולם בכדי להגיב לפעולות הפולישיות של המשטרה האמריקאית. אלפי אנשים הפגינו בערים בכל הארצות האמריקאיות, כמו ג'קסוןביל, ניו יורק, לוס אנג'לס, פילדלפיה, אתרוג, אוסטין, סנט לואיס, אוסטין, אינדיאנה, מישיגן, אורלנדו, קליפורניה, אוהיו, אריזונה, אוקלהומה, ארצות הברית החדשות ועוד. אף אנשים
Answer (translation): Following the black death of George Flay, thousands of people demonstrated around the world to respond to the police actions of the American police. Thousands of people demonstrated in cities in all American countries, such as Jacksonville, New York, Los Angeles, Philadelphia, Etrog, Austin, St. Louis, Austin, Indiana, Michigan, Orlando, California, Ohio, Arizona, Oklahoma, New United States and more . no people
Expected Answer: בין 75 ל200 מיליון איש
Expected Answer (translation): Between 75 and 200 million

We see that text-davinci-003 is very bad at answering these questions out of the box. It gets all three questions wrong and produces some ridiculous answers for the second and third questions.

So the question is asked — how do we solve this? How can we get good answers from text-davinci-003 without having to fine-tune it on a bunch of Hebrew data?

Part 3. Solution Suggestion — Prompting & Querying

As I wrote in the introduction, one way to add information to the LLM without fine-tuning is prompting, which is simply including relevant information in the input to the model. But including relevant information for each query can be a very tedious and time-consuming task, can we automate this?

Of course, we can! Enter prompting via querying. We will take a fairly small dump of Wikipedia in Hebrew, which definitely contains answers to our questions. We will split this dump into lines, embed each of these lines using an embedding model by OpenAI, and insert these embeddings into a vector database called Pinecone DB. Then, for each question we ask the model, we will search the vector database for bits of information that are semantically similar to the question and append them to the model input as additional information. This procedure is shown in figure 1. Hopefully, the model will be able to make use of the provided context and yield correct answers to our questions.

Inspired by Jason Briggs, I will show you a prompt template that will help us generate intelligible prompts to text-davinci-003 and hopefully, make it answer our three previous questions correctly.

Part 4. Experiment — Question Answering in Hebrew

First things first — we need to populate our vector database with relevant information. For this purpose we are going to use a small Hebrew Wikipedia dump which I downloaded from here — https://u.cs.biu.ac.il/~yogo/hebwiki/ (available under a Creative Commons Attribution-ShareAlike 4.0 International License).

with open('/content/drive/MyDrive/Datasets/hebrew_wikipedia/full.txt') as f:
  hebrew_wiki = f.readlines()

openai_price_per_1k = 0.00004
n_tokens = sum([len(x) for x in hebrew_wiki])
print(f'This hebrew wikipedia dataset contains {len(hebrew_wiki)} sentences')
print(f'In total it amounts to {n_tokens} tokens')
print(f'With the current pricing of openAI it will cost {(openai_price_per_1k * n_tokens) / 1000:.3f}$ to embed everything')

This hebrew wikipedia dataset contains 3833140 sentences
In total it amounts to 380692471 tokens
With the current pricing of openAI it will cost 15.228$ to embed everything

For general information, I calculated how much it will cost to embed the whole Wikipedia dump using the OpenAI API — not too bad! But still, for our purposes we only need specific subjects so we can save some money by choosing more particular sentences to embed:

def get_lines_per_subject(hebrew_wiki, subject, subject_translation):
  subject_lines = [(i, x) for i, x in enumerate(hebrew_wiki) if subject in x]
  n_tokens = sum([len(x[1]) for x in subject_lines])
  print(f'There are {len(subject_lines)} sentences that have the words {subject_translation} in them')
  print(f'In total it amounts to {n_tokens} tokens')
  print(f'With the current pricing of openAI it will cost {(openai_price_per_1k * n_tokens) / 1000:.3f}$ to embed everything')
  return subject_lines

subject_lines = {}
for qa_dct in questions_answers:
  subject_lines[qa_dct['subject']] = get_lines_per_subject(hebrew_wiki, qa_dct['subject'], qa_dct['subject_translation'])

There are 7772 sentences that have the words Australia in them
In total it amounts to 1026137 tokens
With the current pricing of openAI it will cost 0.041$ to embed everything
There are 15908 sentences that have the words University in them
In total it amounts to 1904404 tokens
With the current pricing of openAI it will cost 0.076$ to embed everything
There are 160 sentences that have the words The black plague in them
In total it amounts to 19410 tokens
With the current pricing of openAI it will cost 0.001$ to embed everything

That’s better. We are now ready to populate the vector database:

import pinecone

index_name = 'hebrew-wikipedia'

# initialize connection to pinecone (get API key at app.pinecone.io)
pinecone.init(
    api_key=pinecone_key,
    environment="us-east1-gcp"
)

# check if index already exists (it shouldn't if this is first time)
if index_name not in pinecone.list_indexes():
    # if does not exist, create index
    pinecone.create_index(
        index_name,
        dimension=len(res['data'][0]['embedding']),
        metric='cosine',
        metadata_config={'indexed': ['contains_keywords']}
    )
# connect to index
index = pinecone.Index(index_name)
# view index stats
index.describe_index_stats()

We generated a new Pinecone DB index (if it didn’t exist yet). Let’s populate it with our data:

from tqdm.auto import tqdm
import datetime
from time import sleep

embed_model = "text-embedding-ada-002"
batch_size = 32  # how many embeddings we create and insert at once

def insert_into_pinecone(lines, subject):
  print(f'Inserting {len(lines)} of subject: {subject} into pinecone')
  for i in tqdm(range(0, len(lines), batch_size)):
      # find end of batch
      i_end = min(len(lines), i+batch_size)
      meta_batch = lines[i:i_end]
      ids_batch = [str(x[0]) for x in meta_batch]
      texts = [x[1] for x in meta_batch]
      try:
          res = openai.Embedding.create(input=texts, engine=embed_model)
      except:
          done = False
          while not done:
              sleep(5)
              try:
                  res = openai.Embedding.create(input=texts, engine=embed_model)
                  done = True
              except:
                  pass
      embeds = [record['embedding'] for record in res['data']]
      # cleanup metadata
      meta_batch = [{
          'contains_keywords': subject,
          'text': x,
      } for x in texts]
      to_upsert = list(zip(ids_batch, embeds, meta_batch))
      # upsert to Pinecone
      index.upsert(vectors=to_upsert)

for subject, lines in subject_lines.items():
  insert_into_pinecone(lines, subject)

Great! We have a populated vector DB. Let’s get the top-2 contexts for the first question (Which city in Victoria is considered the sporting capital of Australia?):

res = openai.Embedding.create(
    input=[questions_answers[0]['question']],
    engine=embed_model
)

# retrieve from Pinecone
xq = res['data'][0]['embedding']

# get relevant contexts (including the questions)
res = index.query(xq, top_k=2, include_metadata=True)

res

{'matches': [{'id': '2642286',
              'metadata': {'contains_keywords': 'אוסטרליה',
                           'text': 'העיר שוכנת לחופה הצפון-מערבי של '
                                   'אוסטרליה.\n'},
              'score': 0.899194837,
              'values': []},
             {'id': '2282112',
              'metadata': {'contains_keywords': 'אוסטרליה',
                           'text': 'אוסטרלאסיה היה השם של נבחרת הספורטאים '
                                   'מאוסטרליה וניו זילנד שהשתתפה באולימפיאדת '
                                   'לונדון (1908) ובאולימפיאדת סטוקהולם '
                                   '(1912).\n'},
              'score': 0.893916488,
              'values': []}],
 'namespace': ''}

We see that the top 2 lines in our vector DB (in terms of cosine similarity to the question’s embedding) are not so helpful. The first line is a very general sentence about Australia and the second indeed talks about sports in Australia, but there is no mention of Melbourne which is the answer we are expecting.

But still, let’s challenge our model and see how it fares now that it has access to additional information in Hebrew. Notice that we instructed our model to answer “I don’t know” if the answer cannot be deduced from the given context.

limit = 3000

def retrieve(query, filter_subjects=None, add_dont_know=False):
    res = openai.Embedding.create(
        input=[query],
        engine=embed_model
    )

    # retrieve from Pinecone
    xq = res['data'][0]['embedding']

    # get relevant contexts
    res = index.query(xq,
                      {'contains_keywords': {'$in': filter_subjects}}
                      if filter_subjects is not None else None,
                      top_k=100,
                      include_metadata=True)
    contexts = [
        x['metadata']['text'] for x in res['matches']
    ]

    # build our prompt with the retrieved contexts included
    idk_str = '''. If the question cannot be answered using the
    information provided answer with "I don't know"''' if add_dont_know else ''
    prompt_start = (
        "Answer the question based on the context below." + idk_str + "\n\n" +
        "Context:\n"
    )
    prompt_end = (
        f"\n\nQuestion: {query}\nAnswer:"
    )
    # append contexts until hitting limit
    for i in range(1, len(contexts)):
        if len("\n\n---\n\n".join(contexts[:i])) >= limit:
            prompt = (
                prompt_start +
                "\n\n---\n\n".join(contexts[:i-1]) +
                prompt_end
            )
            break
        elif i == len(contexts)-1:
            prompt = (
                prompt_start +
                "\n\n---\n\n".join(contexts) +
                prompt_end
            )
    return prompt


original_subjects = ['אוסטרליה', 'אוניברסיטה', 'המוות השחור']

for i, qa_dct in enumerate(questions_answers):
  print(f'####################')
  print(f'Question #{i + 1} with context:')
  query_with_contexts = retrieve(qa_dct['question'],
                                 filter_subjects=original_subjects,
                                 add_dont_know=True)
  print(f'Length of query with contexts: {len(query_with_contexts)}')
  print_question_answer(qa_dct['question'],
                        query_with_contexts,
                        qa_dct['answer'],
                        qa_dct['question_translation'],
                        qa_dct['answer_translation'],
                        )

####################
Question #1 with context:
Length of query with contexts: 3207
Question: איזו עיר בויקטוריה נחשבת לבירת הספורט של אוסטרליה?
Question (translation): Which city in Victoria is considered the sporting capital of Australia?
Answer: קנברה
Answer (translation): Canberra
Expected Answer: מלבורן
Expected Answer (translation): Melbourne
####################
Question #2 with context:
Length of query with contexts: 3179
Question: איזה נהר ממוקם בקרבת אוניברסיטת הארווארד?
Question (translation): Which river is located in the vicinity of Harvard University?
Answer: I don't know
Answer (translation): I don't know
Expected Answer: נהר צ'ארלס
Expected Answer (translation): Charles River
####################
Question #3 with context:
Length of query with contexts: 3138
Question: כמה אנשים ברחבי העולם נספו בעקבות המוות השחור?
Question (translation): How many people in the world died from the black plague?
Answer: I don't know.
Answer (translation): I don't know.
Expected Answer: בין 75 ל200 מיליון איש
Expected Answer (translation): Between 75 and 200 million

OK, getting a bit better. The first answer is still wrong. In the second and third answers, we got “I don’t know” which I perceive as a major improvement over getting wrong answers. But still, we want to give current answers, how can we improve on this..?

Let’s add some additional subjects to the vector DB, maybe it didn’t have enough relevant information from which to pull a relevant context.

additional_subjects = [
    ('מלבורן', 'Melbourne'),
    ('סידני', 'Sydney'),
    ('אדלייד', 'Adelaide'),
    ('הרווארד', 'Harvard'),
    ("מסצ'וסטס", 'Massachusetts'),
    ("צ'ארלס", 'Charles'),
    ('בוסטון', 'Boston'),
    ('מגפה', 'plague'),
    ('האבעבועות השחורות', 'smallpox'),
    ('אבעבועות שחורות', 'smallpox'),
    ('מונגוליה', 'mongolia')
]

additional_subject_lines = {}
for subject, subject_translation in additional_subjects:
  additional_subject_lines[subject] = get_lines_per_subject(hebrew_wiki, subject, subject_translation)

for subject, lines in additional_subject_lines.items():
  insert_into_pinecone(lines, subject)

There are 948 sentences that have the words Melbourne in them
In total it amounts to 119669 tokens
With the current pricing of openAI it will cost 0.005$ to embed everything
There are 2812 sentences that have the words Sydney in them
In total it amounts to 347608 tokens
With the current pricing of openAI it will cost 0.014$ to embed everything
There are 213 sentences that have the words Adelaide in them
In total it amounts to 24480 tokens
With the current pricing of openAI it will cost 0.001$ to embed everything
There are 1809 sentences that have the words Harvard in them
In total it amounts to 228918 tokens
With the current pricing of openAI it will cost 0.009$ to embed everything
There are 1476 sentences that have the words Massachusetts in them
In total it amounts to 179227 tokens
With the current pricing of openAI it will cost 0.007$ to embed everything
There are 5025 sentences that have the words Charles in them
In total it amounts to 647501 tokens
With the current pricing of openAI it will cost 0.026$ to embed everything
There are 2912 sentences that have the words Boston in them
In total it amounts to 361337 tokens
With the current pricing of openAI it will cost 0.014$ to embed everything
There are 1007 sentences that have the words plague in them
In total it amounts to 116961 tokens
With the current pricing of openAI it will cost 0.005$ to embed everything
There are 39 sentences that have the words smallpox in them
In total it amounts to 5417 tokens
With the current pricing of openAI it will cost 0.000$ to embed everything
There are 135 sentences that have the words smallpox in them
In total it amounts to 15815 tokens
With the current pricing of openAI it will cost 0.001$ to embed everything
There are 742 sentences that have the words mongolia in them
In total it amounts to 90607 tokens
With the current pricing of openAI it will cost 0.004$ to embed everything

Great, we added richer context (you might even say we are cheating a bit — notice that we added all sentences that contain the words “Melbourne” and “Charles” amongst others, which are the answers to our questions. But it’s just a demonstration!)

for i, qa_dct in enumerate(questions_answers):
  print(f'####################')
  print(f'Question #{i + 1} with context:')
  query_with_contexts = retrieve(qa_dct['question'], add_dont_know=Tue)
  print(f'Length of query with contexts: {len(query_with_contexts)}')
  print_question_answer(qa_dct['question'],
                        query_with_contexts,
                        qa_dct['answer'],
                        qa_dct['question_translation'],
                        qa_dct['answer_translation'],
                        )

####################
Question #1 with context:
Length of query with contexts: 3158
Question: איזו עיר בויקטוריה נחשבת לבירת הספורט של אוסטרליה?
Question (translation): Which city in Victoria is considered the sporting capital of Australia?
Answer: מלבורן
Answer (translation): Melbourne
Expected Answer: מלבורן
Expected Answer (translation): Melbourne
####################
Question #2 with context:
Length of query with contexts: 3106
Question: איזה נהר ממוקם בקרבת אוניברסיטת הארווארד?
Question (translation): Which river is located in the vicinity of Harvard University?
Answer: נהר צ'ארלס
Answer (translation): Charles River
Expected Answer: נהר צ'ארלס
Expected Answer (translation): Charles River
####################
Question #3 with context:
Length of query with contexts: 3203
Question: כמה אנשים ברחבי העולם נספו בעקבות המוות השחור?
Question (translation): How many people in the world died from the black plague?
Answer: לפי הערכות שונות, כ-35 מיליון בני אדם בסין לבדה, ובין 20 ל-25 מיליון בני אדם באירופה.
Answer (translation): According to various estimates, about 35 million people in China alone, and between 20 and 25 million people in Europe.
Expected Answer: בין 75 ל200 מיליון איש
Expected Answer (translation): Between 75 and 200 million

Nice! We have our answers! Although the third answer is still not 100% correct, but way better than “I don’t know” and way better than the initial hallucination.

Part 5. Conclusions

In this article, we have seen how to automatically provide relevant context to an LLM using a vector database in order to improve the model outputs. In this specific case, it might not have been necessary because the performance of ChatGPT (which is available as an API) on Hebrew has improved tremendously and it is able to answer these questions very well out-of-the-box.

Yet still, the point of this article was to introduce this concept, which can be generalized for other purposes such as:

• Using an LLM to analyze proprietary data which is not available online
• Adding a form of explainability to the model, by appending the context which was used to the output of the user (to prove the credibility of the answer).

I hope you enjoyed this article and were motivated to try it for yourself. See you next time!

Elad

References

• Hebrew Wikipedia dump — https://u.cs.biu.ac.il/~yogo/hebwiki/
• SQuAD dataset — https://rajpurkar.github.io/SQuAD-explorer/
• Video tutorial by James Briggs — https://youtu.be/dRUIGgNBvVk
• OpenAI API — https://openai.com/blog/openai-api
• Pinecone DB — https://www.pinecone.io/

Improving Hebrew Q&A Models via Prompting was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I will present the topic of sentence embeddings. We will learn what sentence embeddings are and how they can be helpful. We will review some basic (and even naive) methods for creating sentence embeddings and also some recent advances in the field. We will end with an experiment I conducted on measuring document visualization quality when using different sentence embedding techniques. So without further ado, let’s jump right in!

The article will be organized as follows:

Part 1. What are Sentence Embeddings?
Part 2. An Example Application
Part 3. Some Basic Methods
Part 4. SentenceBERT [1] — The New Kid on the Block
Part 5. SimCSE [2] — Enter Unsupervised Sentence Embeddings
Part 6. Experiment — Measuring Document Visualization Quality — Kaggle News Dataset [3]

• The code I used for the experiment is available here — https://github.com/erap129/DocumentVisualizations

Part 1 — What are Sentence Embeddings?

Embeddings in machine learning are vector representations of entities in the world, so sentence embeddings are simply vector representations of sentences. Usually we say that they are distributed vector representations, and this means that we don’t have a 1:1 mapping from traits in the data to elements in the vector (for example — a single vector element representing the “happiness” of a sentence), but rather these traits are distributed amongst the different elements of the vector in ways that we cannot pre-define.

For example, in figure 1 we can see a toy example of some sentence embeddings being generated. We can see that the embeddings manage to capture the semantic similarity between the first two (questions about food) and last two (schedule-related facts) sentences while maintaining a difference between these two groups.

Part 2— An Example Application

So, what are these sentence embeddings good for, anyway? Let’s see a hypothetical example application that I thought of (this is just an example, I’m sure you will be able to conjure more of these ideas once you see it).

In figure 2 we can see an example application where a user writes a piece of code (insertion sort in Python) and wants to find similar code snippets (maybe to find better implementations, or to find the same function in other languages). The user takes this code and uses sentence embedding methods to transform the code snippet into a distributed vector representation. Then, the user queries a database of other code snippet embeddings and using K nearest neighbors with cosine similarity finds the most similar snippets to the input snippet.

Part 3— Some Basic Methods

The most basic methods in this field are counting-based. The simplest example of counting-based methods is shown in the top of figure 3, where the count of each word in a sentence is an entry in the vector representation of the sentence. This is a non-distributed vector representation, as a single trait of a sentence (a specific word count) can be mapped directly to a single entry in the representation vector. Other methods, such as TF-IDF have evolved this by normalizing these values by the frequency of each word in the entire dataset, but they are still counting-based and don’t take into account the ordering of the words in the sentence (which can be a major source of data loss).

A second method is shown in the bottom of figure 3, where certain word embedding methods are used out-of-the-box, and the representation for the sentence is taken as an average of these word embeddings. The word embeddings can be generated by any method (GloVe [5], word2vec, BERT, etc) but we will see later that there is a drawback in simply taking the average of these vectors without tuning them first to better fit the task of sentence embeddings.

Part 4— SentenceBERT — The New Kid on the Block

A major advancement in the field of sentence embeddings occurred in 2019 when the Sentence-BERT paper was released by Nils Reimers and Iryna Gurevych. They introduced a new method to fine-tune the well-known BERT model to produce embeddings that will be well comparable using cosine similarity.

The original BERT [4] paper dealt with the problem of semantic textual similarity, where the goal is to evaluate how similar a pair of sentences are (example shown in figure 4), and even reached new state-of-the-art results on a specific benchmark.

The problem with BERT’s approach was its incredible inefficiency. BERT solved this task by encoding the tokens of two sentences at once (separated by a special [SEP]token) and outputting the level of similarity at the end, as shown in figure 5, which is taken from the original BERT paper. To find the most similar pair in a dataset of 10K sentences would take BERT ~65 hours of inference time. Thus, the goal of SBERT was to create embeddings that can be compared simply using cosine similarity, while still yielding state-of-the-art results on benchmark STS tasks.

I will focus here on one of the solutions proposed in the SBERT paper, which is to use siamese networks on labeled sentence pairs from a natural language inference (NLI) task dataset. The sentence pairs in this dataset were labeled with one of the following options: entailment, contradiction or neutral.

The fine-tuning task introduced by SBERT and applied to the original BERT pre-trained model was to take each labeled sentence-pair and use siamese networks to generate a vector containing both BERT representations (the concatenation of v, u, |u-v|, where u and v are the outputs of BERT). A feedforward network with a softmax function is tacked on top of this structure and it’s output determines whether the sentence pair is an entailment, a contradiction or neutral. The process described above is depicted in figure 6.

This simple fine-tuning task created an improved BERT model which generates sentence representations that can be compared using simple cosine similarity! In table 1 (taken from the original SBERT paper), the authors show that they outperform naive methods (average GloVe embeddings, average BERT embeddings) by a large margin, and also outperform other sentence embedding methods which we have not covered here.

But what if our data doesn’t have labeled sentence pairs which we can use to train SBERT? That’s where SimCSE comes in…

Part 5— SimCSE — Enter Unsupervised Sentence Embeddings

SimCSE, written by Tianyu Gao, Xingcheng Yao and Danqi Chen introduces a novel way of fine-tuning language models for the purpose of creating sentence embeddings. In this paper the authors use contrastive learning, where the positive example is the sentence itself passed through a different dropout mask (more on this later).

Note — The paper also presents a supervised way to create sentence embeddings that uses contrastive learning. I chose to focus on the unsupervised part in this article. Refer to the original paper for full details.

First let’s have a refresher on contrastive learning: the main idea is to pull together close neighbors and push apart non-neighbors. We will be following Mario, Luigi and Yoshi in this example. Mario and Luigi are twins, so we want the embeddings that represent them to be close in vector space. Yoshi is cute, but not very similar to Mario and Luigi so we want his representation to be further away.

But how do we achieve this numerically? Using a loss function of course! In contrastive learning, we generally use a loss function called InfoNCE. Figure 8 is an infographic that will help us understand the InfoNCE loss function, let’s go over it step by step.

First, in the numerator, we see that we have an exponent of the similarity of h_i and (h_i)^+ (sorry for the bad notation). We see that h_i represents Mario and (h_i)^+ represents Luigi, thus we want the numerator to be large, of course. On the other hand in the denominator we have the sum of similarities of Mario h_i with all of the other elements in the batch (h_j)^+ , this includes Luigi but it also includes other stuff that we don’t want Mario to be similar to, including a plane, a dish and Yoshi of course. We see that the numerator is a part of the denominator and if we make the analogy to a cake — the numerator is a slice of the denominator. This slice should be as large as possible so as to maximize the similarity between Mario and Luigi while minimizing the similarity between Mario and everything else in the batch.

Notice that there is a negative logarithm function wrapping this whole thing. We see that as this fraction gets closer to 1, the negative logarithm gets smaller and thus by using gradient descent with this objective function we are pushing similar items to have a higher similarity, and dissimilar items to have a lower similarity.

Now we know how contrastive learning works, but which “Mario, Luigi and Yoshi” did SimSCE use when no labeled data of this sort was available? It turns out that for a given sentence they passed it through BERT twice, each time using a different dropout mask (reminder — dropout is a function which zeros out a fraction of the input randomly), thus receiving two slightly different embeddings for a given sentence. For negative examples they simply used all of the other sentences in the mini-batch (given good shuffling — the other sentences should be quite different).

Figure 9 shows an example of this (contains a figure from the original SimCSE paper). On the left we see that the sentence “Two dogs are running” is passed twice through the network, thus creating a positive pair, while all of the other sentences in the batch (A main surfing on the sea, A kid is on a skateboard) serve as negative pair samples. On the right is just a reminder of how dropout works by masking out inputs randomly.

OK, but why should this even work? What is the theoretical basis behind this? Here the authors of SimCSE explain to us about two terms called Alignment and Uniformity.

In figure 10 we can see the equations for alignment and uniformity, along with our trusty friends Mario, Yoshi and Luigi. Alignment is the expected distance between embeddings of paired instances. The term on the left is the alignment “loss” which we will aim to minimize. We see that it contains the distances between the embeddings of all of the positive pairs (Marios and Luigis) which we will want to minimize. On the right we see the uniformity loss. We have e to the negative power of the distances between all of the negative pairs (Marios and Yoshis), which we will want to maximize.

The authors argue that pre-trained models generally have good alignment — that is, pairs of positive samples will be close in the embedding space, but poor uniformity — that is, pairs of negative samples won’t be far enough from each other. Figure 11 is taken from the original SimCSE paper and it shows that they managed to keep a good alignment while improving the uniformity (minimizing the uniformity loss). The red stars represent the SimCSE model and the other symbols are other baselines that were tested.

When we look at the results of SimCSE on STS benchmark tasks in table 2, we see similar results to SBERT that we reviewed earlier, only this time — no labeled data regarding sentence similarity was used! They only fine-tuned on 1 million sentences from Wikipedia. That’s pretty impressive in my opinion.

Part 6— Experiment — Measuring Document Visualization Quality — Kaggle News Dataset

That’s it for the theoretical part of this article. I am now going to show an experiment I conducted using the above models regarding the visualization of documents in 2D space.

The code I used for this experiment is available here — https://github.com/erap129/DocumentVisualizations

The News Category Dataset from Kaggle — https://www.kaggle.com/datasets/rmisra/news-category-dataset (released under the Attribution 4.0 International (CC BY 4.0) license) contains news articles belonging to 42 different categories. Each article in the dataset contains the following data fields: category, headline, authors, link, short_description, date. I took only the headline, short_description fields and created a new field which I called long_description. My goal was to create embeddings of the long_description of each news article using several methods, and check which method results in the best 2D visualization of the data. So a few questions need to be asked here.

• How did I create a 2D visualization given the sentence embedding of a specific article?

I used the UMAP algorithm. There is a plethora of documentation regarding this great dimensionality reduction algorithm so I won’t go into details about it here. I will just say that I used the default hyper-parameters in all experiments with no tuning whatsoever.

• How did I quantitatively compare the qualities of the different visualizations? What does it mean to have a “good” visualization?

The comparison of visualization is done, of course, visually “by eye”. But what if we still want to get a metric which we can hang on to? Maybe we are pitting dozens of algorithms against each other and don’t have time to eyeball them all? I came across this paper — Selecting good views of high-dimensional data using class consistency which suggests a measure called distance consistency which is defined as so:

For each point x’ in the visualized space v(X), we check whether the closest class’s centroid is different than the class of x, and then divide by N the number of samples to get a number between 0 and 1. As the DSC of the visualization is lower it means that the clusters representing the different groups in the visualization are more prominent, which is good. This measure applies only when the data is labeled, which is our situation with the news dataset.

• Which methods yielded which results? Are my findings in line with the SBERT and SimCSE papers?

Let’s go into the experiment results. For simplicity, I used only 5 categories from the Kaggle news dataset: food&drink, worldpost, sports, divorce and style. I compared the following methods for creating visualizations:

• TF-IDF — The counting based method for creating document-representing vectors. Used as a baseline.
• bert_average — Average embedding values of all the tokens in the document, taken from the final layer of a BERT model (not fine-tuned for sentences)
• bert_cls — The embedding of the [CLS] token taken from the final layer of BERT.
• sbert_average — Average embedding values from SBERT (the supervised fine-tuning of BERT for sentences)
• sbert_cls — The embedding of the [CLS] token taken from the final layer of SBERT.
• simcse_average — Average embedding values from SimCSE (the unsupervised fine-tuning of BERT for sentences)
• simcse_cls — The embedding of the [CLS] token taken from the final layer of SimCSE.

Here are the results for the DSC measure:

We see very similar results to the SBERT and SimCSE papers. The non-finetuned BERT has hard competition from the simple TF-IDF baseline. After supervised finetuning (SBERT) the results improve dramatically, and also via unsupervised finetuning (SimCSE). In BERT and SBERT using the average token embeddings as a basis for dimensionality reduction was advantageous over taken the embedding of the [CLS] token, while for SimCSE the opposite is true.

Let’s view the actual visualization for each method to get a better feel for what has been done here. A quick inspections confirms the DSC metric — methods that got a lower DSC produce more discernable clusters in the visualization:

• TF-IDF

• bert_average

• bert_cls

• sbert_average

• sbert_cls

• simcse_average

• simcse_cls

Conclusion

In this article we:

• learned what sentence embeddings are
• learned about two state-of-the-art methods for creating sentence embeddings by finetuning BERT (both supervised and unsupervised)
• saw that the results from the above papers repeat also in a visualization quality experiment

For future work I think it would be interesting to try out SimCSE on other types of textual data and not just natural language. For example, to write a model that finds similar code snippets as I suggested in part 2 of this article. The fact that SimCSE does not need supervised data to fine-tune for sentences will allow training on various kinds of data.

I hope you enjoyed reading and learned something new. Thanks for reading!

References

[1] — SentenceBERT paper — https://arxiv.org/abs/1908.10084
[2] — SimCSE paper — https://arxiv.org/abs/2104.08821
[3] — Kaggle news categories dataset — https://www.kaggle.com/datasets/rmisra/news-category-dataset
[4] — BERT paper — https://arxiv.org/abs/1810.04805
[5] — GloVe embeddings — https://nlp.stanford.edu/projects/glove/

A Sentence Is Greater than the Sum of Its Words was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I will share the process of creating my personal website and portfolio using Jekyll. Jekyll is a Ruby Gem used to easily create static websites whose content is predetermined at build time and not dependent on any live user input or database connection.

As I am not a web designer and didn’t want to spend time designing in CSS and HTML, I searched for a nice and impressive template and stumbled upon Beautiful Jekyll (https://beautifuljekyll.com/) by Dean Atalli, which I will be customizing in this post. This is my site — https://erap129.github.io, which is what we will learn to create, and this is the site’s repository — https://github.com/erap129/erap129.github.io.

This article will be organized as follows:

1. Requirements — what I need/want from my site
2. Setups and installations
3. Building the website
4. Deploying the website

Part 1 — Requirements

Before I started working, I defined my main goals for this website. It should:

• Give general information about me to potential and existing clients/collaborators/employers, such as my interests, projects, and hobbies.
• Contain a page containing previews of all my articles on Medium and a link to them on the Medium platform. This page should not be built manually, as I am planning to keep writing on Medium, and I don’t want to copy and paste my posts each time to my personal website. Furthermore, it should be updated automatically so I won’t have to update my site whenever I post on Medium.
• Contain additional static pages — such as my scientific publications, public code contributions, and maybe more.

Part 2 — Setups and Installations

It is important to say — Beautiful-Jekyll is an awesome template. If the site structure defined by this template is right for you and you don’t need integration with Medium — just skip my article and follow the official instructions.

I really liked the design of this template, but to create an integration with Medium, I needed to do quite a bit of customization. To this end, I needed the ability to deploy the site locally, which is the purpose of the following installations:

This document — https://docs.github.com/en/pages/setting-up-a-github-pages-site-with-jekyll/creating-a-github-pages-site-with-jekyll provides an excellent explanation of how to get your environment set up for work with Jekyll and Github Pages, but I will go over the main steps here as well. I will assume that you are using Ubuntu Linux (if this is not the case — follow the instructions in the link above as per your OS).

• First of all, install Ruby — sudo apt-get install ruby-full
• Now install Bundler — sudo gem install bundler
• Fork Beautiful-Jekyll — The steps are written here -https://github.com/daattali/beautiful-jekyll#readme. They are:
  1. Fork the project as <yourusername>.github.io
  2. Clone the project to your PC
• To run the website locally, navigate to the local repository folder in the terminal and run bundle install. This will install the required Ruby software used to run and serve Jekyll. Now each time you wish to serve the website locally run bundle exec jekyll serve. Congrats! The website will be available on http://localhost:4000

Part 3 — Building the website

These are the things I will do to customize the Jekyll template to my needs:

1. Create the “Blog Posts” page, which will be integrated into Medium
2. Create additional markdown-based pages: Home and Publications

Let’s start!

Create the “Blog Posts” page

The first question is — how do we get information from Medium? The answer is — RSS! Medium has an RSS feed per author, which allows me to get all of my articles’ information using a simple URL (for me, it’s — https://medium.com/feed/@erap129).

I really liked how posts are shown in Beautiful Jekyll, but in my case, I don’t want each preview linking to an internal post but rather to the original post on Medium.

To generate content dynamically in Jekyll, we will need to create a generator plugin. Plugins in Jekyll are bits of Ruby code that run when you build your website and perform various functions. Our generator will collect all of my Medium posts’ information and store it in a collection we will use in our site’s HTML code.

Let’s create a folder named _plugins in the main directory of our repo, and create a file called jekyll-display-medium-posts-json.rb with the following content:

This plugin is an extension/modification of the plugin presented in the post by James Hamman — https://medium.com/@jameshamann/displaying-medium-posts-on-your-jekyll-website-7eef230309e4.

Let’s go over the code:

• In line 10, we create a new collection called medium_posts_json. We can use this name throughout the site’s code to access our posts.
• In line 12, we call the API of rss2json. The JSON format gives us better control over the content we receive from Medium, and thus we use this API to convert Medium’s XML format to JSON.
• In lines 18–33, we collect the information that we will be using in our “Blog Posts” page — the articles’ title, image, link, and date (we also collect the categories but don’t use them)
• Notice that we are using the nokogiri Ruby gem to traverse HTML content in line 31. Why? Because the description of each post in the RSS feed is raw HTML, we here want to create a short excerpt of the posts. To skip the headers and images — we extract only the paragraphs (which are in p tags in HTML).

It is important to add the following snippet to _config.yaml. Otherwise, the collection created by the generator won’t be available to use in the site:

collections:

- medium_posts_json

Now, we need to create a layout that will use this information and transform it into HTML code rendered by the browser. We will create a new file called blog.html in the _layouts directory with the following content:

This layout is basically a stripped-down version of the original layout in Beautiful-Jekyll — https://github.com/daattali/beautiful-jekyll/blob/master/_layouts/home.html. The only changes I made were to remove irrelevant content and make the layout use the collection I created earlier — site.medium_posts_json.

• Notice in line 5 that we iterate over the collection we made using our plugin and add the required HTML code for each element. Jekyll uses a syntax called Liquid to generate this content. You can read more about Liquid here — https://shopify.github.io/liquid/.

Now, all we need to do is to create a new file called blogposts.md in the main directory, and have it use the new layout we created — blog.

---

layout: blog

title: Blog Posts

subtitle: This is what keeps my mind busy

---

Also, edit the navbar-links section in _config.yaml to include the new blog posts page.

Great! If I now serve my website locally using bundle exec jekyll serve, I should be able to access my site in http://localhost:4000 and see the changes that I made.

2. Create additional markdown-based pages

Now, because we added a Blog Posts page, we no longer need the homepage to list local blog content. Additionally, we would like the home page to be easily editable using Markdown rather than HTML.

Thus, we will remove index.html and replace it with index.md. I added personal information about myself on this page, but you can do whatever you desire. I also added a publications.md page in which I list my scientific publications. The navbar-links section in my _config.yaml looks like this:

navbar-links:

Home: "index"

Blog Posts: "blogposts"

Publications: "publications"

Part 4 — Deploying the Website

Once we reached a stage where we run bundle exec jekyll serve and like what we see, we are ready to deploy.

We want our site to deploy in two cases:

1. When we push new/updated content into the repository’s master branch
2. Every time a new post on Medium is added, we wish to integrate into the site.

1 will be easy, and for 2, we will make this a daily process. This means that each day at a set time, the build process for the website will be built, and if I added new content to my Medium profile, it will get pulled to the website.

We can’t rely on the default GitHub page deployment actions for this because we are using custom plugins, which GitHub pages do not officially support. Plus, we want the build process to be timely and not just based on a push to the master branch. To this end, we will use this custom GitHub Action from the marketplace, called Jekyll Actions — https://github.com/marketplace/actions/jekyll-actions.

The link above has a pretty useful tutorial, but I will still take you through the steps needed to activate this action. From the site’s root directory, I created the following file: .github/workflows/github-pages.yml with the following content:

Let’s go over what we see here (the interesting bits)

• In lines 4–8, we define when the site deployment will occur — on a push event to the master branch and every day at midnight. I’m doing this to edit my website manually if I want, but if I publish new content on Medium, I won’t necessarily have to edit it.
• Notice that in lines 23 and 26, I refer to a secret called JEKYLL_PAT. To set this up, I created a GitHub personal access token — https://github.com/settings/tokens with full repo access and added it as a secret to my project settings (Repo page -> settings -> secrets -> actions).
• The rest of the code was taken from the original GitHub action and is explained here thoroughly — https://github.com/marketplace/actions/jekyll-actions.

Another important step is to set the deployment branch to gh-pages (the branch that the static content is being pushed to) so GitHub will know to host the files from this branch. I want to give a special thanks to Christian from StackOverflow, who helped me solve an issue regarding this — https://stackoverflow.com/questions/74244454/helaili-jekyll-action-does-not-update-the-browser-with-gh-pages-branch-content. This is done through the repository settings like this:

And that’s it!

Of course, you can keep customizing your website endlessly, and I didn’t go over trivial stuff I’m sure you will be able to figure out, such as changing the website title, adding your own photo, and more. The _config.yaml file is very self-explanatory and allows for easy manipulation and customization.

Thanks, folks! Cheers.

Bonus tip:

• This GitHub issue helped me set a favicon for the site — https://github.com/daattali/beautiful-jekyll/issues/312

Relevant Links:

• Jekyll homepage — https://jekyllrb.com/
• Beautiful Jekyll repository — https://beautifuljekyll.com/
• jekyll-action GitHub action — https://github.com/marketplace/actions/jekyll-actions
• Post by James Hamman on integration of Jekyll with Medium — https://medium.com/@jameshamann/displaying-medium-posts-on-your-jekyll-website-7eef230309e4.
• Explanation on liquid syntax — https://shopify.github.io/liquid/

How I Built a Website Integrating Medium Articles — Using Jekyll and GitHub Pages was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we are going to build together the classic game of Snake using Go. This is my first project in Go and what better opportunity to share my learning experience with you and hopefully help your journey to learning Go as well?

The full code for this project (separated into parts) is in this GitHub repo

The outline of this post is as follows:

• Part 1— Why Bother?
• Part 2— Setting up our environment
• Part 3— Implementing Snake in Go
  - Part 3.1— A moving blob
  - Part 3.2— A moving snake
  - Part 3.3— Adding food and points

Part 1— Why Bother?

Why learn a new language? And why not just follow the official tutorial and call it a day?

• Learning a new language makes us better programmers. Every programming language has its set of strengths and features and opening up to languages we haven’t seen before can open us to new ideas and programming paradigms and also add another tool to our toolchain.
• What better way to learn than by games? I personally think it’s way more fun than following an instruction manual.

So why Go?

Go is a language created by Google, with version 1.0 being released to the public back in 2012. Since then Go has developed and gained major traction amongst developers. Go is notable for being fast and easy to learn with a strong focus on efficient concurrency (read about goroutines [2] for more info). I decided that such a popular and trending language deserves a bit of my attention, even if I’m not planning to use it for anything practical in the near future, and thus this tutorial was born!

Part 2— Setting up our environment

The installation part is relevant only for Linux. Note that it will install an older version of Go (1.13 as of writing this article) but it is sufficient for our project.

Installation: Install Go using the following command: sudo apt install golang-go . After following through the installation, check that your system has installed Go by running go version . Your output should look something like this: go version go1.13.8 linux/amd64 (depending on the version you installed). After this is done we can create our first project!

Creating a project: We will be using Visual Studio Code (vscode) as our IDE (you can still follow through using other IDEs). Create a new folder for our project called Snake and open up the IDE in this folder. If using vscode you should install the Go extension via the Extensions tab.

Open up the terminal (can be done directly in VS Code), type go mod init Snake , and hit Enter.

This should create a new file go.mod in your current directory (the Snake directory). If we were publishing this project as a module we would have replaced Snake with a valid URL, such as our GitHub repository, but since this is just a toy project we don’t care about that.

Let’s just write a simple “Hello World” project to see that everything is up and running properly. Create a new file called main.go in the Snake directory and type in the following code:

package main

import "fmt"

func main() {

    fmt.Println("Hello World!")

}

In this code, we imported and used the standard fmt package which deals with command line input and output. In the terminal run go run main.go. You should see Hello World! printed in your terminal and the program should exit. If this is done we are good to go to the next part — building a Snake game!

When you want to run a project consisting of several files (as in this game) — from the directory that contains your code run go run *.go . What will happen is that your main function will run with awareness of the other files in the project (if you just run go run main.go it will ignore all of the other files and break due to missing dependencies).

Part 3— Implementing Snake in Go

This implementation is inspired by and based on the Pong Go tutorial [3] created by Josh Aletto.

Part 3.1— A moving blob

In this first part, we will create a keyboard-controlled blob on the screen. We are starting with this because it is a relatively simple logic task and it will allow us to focus on the basics of the language and the terminal-based GUI framework we will be using — tcell [4].

Part I will consist of three files —

• main.go will contain the UI loop which will listen for user input and update the relevant data structures accordingly. It will also initiate the subroutine in game.go.
• game.go deals with the main game loop — updating the state of the different elements in the game and drawing them on the terminal screen.
• snake.go handles the inner state of the snake object (currently just its speed and position)

Let’s view the files:

main.go —

• In lines 11–18 we get the screen object from tcell and do some error handling when screen fetching fails.
• In lines 23–33 we define our two objects, the game and the SnakeBody and initialize them with some initial parameters.
• In line 34 we run a goroutine which starts the main loop of the game in a different thread. We will view the game loop later.
• In lines 35–53 we deal with user input. Either the user presses ctrl+c and then we exit the program, or the user presses one of the arrow keys and then we send the relevant signal to the game.SnakeBody object, to change the direction of movement.

game.go—

• Lines 21–27 define the main game loop. At the start of each step we clear the screen, then we update the snakeBody location and draw it on the screen. After that, we perform time.Sleep to hold the previous frame on the screen (otherwise the movement would be too fast for us to visually perceive). Finally, we draw the new frame onto the screen.

snake.go —

• This file defines our snake object (currently just a blob) and deals with the changing of direction and the updating of the current location using the current speed. Notice that if the snake disappears into one edge of the screen we want it to reappear at the other edge, hence the complexity in lines 16–23.

The gameplay in this part should look like this:

Part 3.3— A moving snake

To implement a moving snake we will first need to define what being a “snake” means. In this context it means an entity consisting of several parts, where each part has its own X and Y coordinates. By noticing that the movement of the snake can be modelled using a queue we can implement this in a simple and elegant way, as shown in the following figure.

Most of our changes will occur in the file snake.go where the queue logic will be implemented (Go doesn’t have built-in queues so we will implement them using slices). Some minor changes will also happen in main.go and game.gomainly to support the changes in the former.

main.go —

• In lines 23–36 added an initialization to the snakeParts object which constitutes the initial building blocks of the snake.

game.go —

• Added the drawParts function which will draw the snake’s parts onto the screen in each iteration of the game loop.

snake.go—

• Added the Update function which implements the snake’s movement in a queue-like fashion. In each activation of this function a new head is enqueued (depending on the direction of the snake) and the tail’s end is dequeued.

The gameplay in this part should look like this:

Part 3.3— Adding food and points

In this part, we will add important logic to the game including:

• The ability to collect food and earn points
• Make the snake longer if it collected food
• Detecting collisions of the snake’s head with its body
• A “Game Over” screen, with the option to replay or to quit

Let’s discuss the changes required in each of the 3 files:

snake.go —

• Lines 21–23: check if the longerSnake parameter was passed as true. If so, Don’t perform a dequeue on the snake’s body and thus make it one tile longer.
• Added the ResetPos function for convenience (this function will be called from main.go)

game.go—

• Added drawText function to draw text on the screen. Taken from another snake implementation using cell [5].
• Added checkCollision function which receives a List of parts and another part. For each part in the list of parts, it checks whether it collides with the other part. We use this function to check 1) whether the snakehead has collided with food and 2) whether the snakehead has collided with the snake body.
• Added logic in the main loop (I will detail the less trivial additions):
  1) lines 72–76: check for collision with food and update score if collision detected. Also, redraw food in a random location.
  2) lines 86–88: Game Over screen. Notice that after displaying the score this subroutine will terminate (In main.go the user will have an option to start another goroutine with a fresh game)

main.go —

• Lines 43–47: These lines deal with the Game Over situation. If a game over is detected (using the GameOver bool in game), the user has an option to press y or n and the game will either restart or terminate accordingly. We restart the game simply by initiating another goroutine.

The gameplay in this part should look like this:

And we’re done with the implementation! Your snake game is up and running and you can invite friends over for a tournament :)

Thoughts Ahead

I hope you had fun implementing Snake in Go, I know I did. I think goroutines are a super-cool feature, I don’t know other languages where it is that easy to perform parallelization. Although this game is fairly simple I still liked the queue implementation of movement as a small mental exercise. This experience will also shorten the learning curve if I ever have to use this language for anything practical, and it’s always good to have another tool under my belt.

If I had more time I would extend this game and add cool power-ups to my snakes such as particle-shooting abilities, speed changing, and other goodies. I would also add unit tests to make sure that nothing breaks after these feature additions.

Thanks for reading, see you next time!

References

[1] — Full project code: https://github.com/erap129/SnakeInGo

[2] —goroutines: https://golangbot.com/goroutines/

[3] — Pong go tutorial: https://earthly.dev/blog/pongo/

[4] — tcell Go package: https://github.com/gdamore/tcell

[5] — Another snake implementation using tcell: https://github.com/liweiyi88/gosnakego

Build a Snake Game Using Go was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hello readers,

For those of you who haven’t read the previous part, here is the link:

MovieLens-1M Deep Dive — Part I

In that article, I present the MovieLens-1M [1] dataset (a movie recommendations dataset that contains 1 million ratings for movies made by different users) along with some exploratory data analysis and try out some classical recommender systems algorithms. Although, that article is not a pre-requisite and you will be able to understand the content here without reading it.

In the second installment of the MovieLens deep dive series, we will use TensorFlow recommenders (TFRS). TFRS will allow us to abstract away much of the gory details involved when dealing with recommender systems, such as evaluation metrics and the model training procedure. Our model will be based on the two-tower architecture, which we will review before delving into the code.

In the coding sections, I will somewhat extend an existing official TFRS tutorial — Building Deep Retrieval Models [3]. In this tutorial, the goal is to build a retrieval recommender system using rich context features of the items (movies) and the users. We will test the impact of adding different features and try to gain improvements by introducing smarter and more relevant features to the task at hand.

The outline of the article is as follows:

1. A brief overview of modern recommender system methods
2. Get to know TFRS and the retrieval task — the basic building blocks which we will be using
3. Beat the benchmark — attempt to outperform the original TFRS tutorial model by using more contextual features
4. Beat the benchmark — results
5. Conclusion and thoughts ahead

Let’s dive in!

Part I — A Brief Overview of Modern Recommender System Methods

The recommender systems pipeline is usually split into two stages — retrieval and ranking. In the retrieval stage, we generate representations for items and users in our system (usually in the form of embeddings) and select a subset of items that the user might be interested in. This stage may deal with millions of users/items and thus must be computationally efficient. In the ranking stage, we take the output of the retrieval stage and rank them to select the final few items that will be suggested to the user. This model may be more computationally intensive as it must deal with far fewer data. A depiction of this process is shown in figure 1.

In this walkthrough, we will focus on the retrieval stage, and try to make it more efficient and accurate by taking into account different contextual features in the embedding generation process.

Now for some terminology:

• Items — These are the items that we will recommend to users. In our case — movies.
• Queries — This is the information our system will use to make item recommendations. In our case, this will include the ID of the user we are recommending and other features such as the age and gender of the user. We will also include it here

Part II — Get to know TFRS and the retrieval task

Enter TFRS, a recommender systems library built on top of Tensorflow, intended on streamlining the process of building, training, and deploying recommender systems.

The models we build in TFRS will inherit from the tfrs. model.Model class. This class is itself a wrapper around tf.keras.Modeland saves us the hassle of writing the train_step and test_step methods. What we as users have to do is write the compute_loss function, which receives a batch of input queries and items and computes the loss for the model.

In our tutorial, we will build a two-tower architecture model. This model contains one tower for the items and one tower for the queries. As you can see in figure 2, the two-tower architecture is pretty simple. Embeddings are created to represent the queries and the users, and eventually, we perform a cosine similarity operation on the two embedding vectors to predict their affinity (is the query a good match for this item?). By sorting the results of a certain query using the cosine similarity with each item embedding the system can retrieve the top items for this user (usually in the order of 1000's), which will be then most likely fed into a ranking model for fine-grained control over the final items that will be presented to the user (usually up to 10 items).

What we don’t understand from the explanation above though is, how does the model learn? Well, of course by gradient descent. There are positive examples (in our case, movies that were rated by a certain user) and negative examples (movies that were not rated by a certain user). Our algorithm will treat this as a multiclass classification task and try to maximize the similarity between the rated items and the query and minimize the similarity between non-rated items and the query. This operation is done per batch (each batch consists of only positive examples, and the negative examples are generated by pairing the different queries and items in the batch, as shown in figure 3).

Part III — Beat the benchmark

In this part, I will present my code to you and we will walk through the process of trying to improve the performance of our deep retrieval model, by enriching the item and query representations using additional context features.

Let’s review the features we will use for both items and queries. For each feature, I will specify if it was used in the original TFRS tutorial or not.

Query features (user + context info):

• user_id [int] — the most basic feature. Used to index into the user embedding matrix. (Used in original tutorial)
• timestamp [int] — The time of the query performed by the user, represented in UNIX timestamp format. This can help us capture seasonal trends in the data (for example — on weekends people might prefer longer movies). (Used in original tutorial)
• user_age [int] — An additional numeric feature that might hint into patterns in user preferences. (Not used in original tutorial)

Item features (movie info):

• movie_title [string] — A unique identifier for items. Used to index into the item embedding matrix. This feature will also be used to create a sentence embedding of the movie title text (as opposed to just using it as a plain identifier). (Used in original tutorial)
• movie_genres [list<int>] — Each movie is associated with one or more genres, where each genre is defined by a unique integer. We will use these integers to index into a genre embedding matrix. For each movie, we will average the embeddings for all of its genres. (Not used in original tutorial)
• movie_length [int] — The length of the movie in minutes. This info was collected using the Cinemagoer[2] python package. Some of the movie length data was missing and for these movies, I gave a default length of 90 minutes.

The code:

I present here a non-conventional way of doing exploratory data science work, and that is via scripting and not via Jupyter notebooks. I think the advantages of scripting VS notebooks are:

1) Cleaner code. Encourages you to use good programming practices, such as extracting common procedures into functions instead of copy-pasting between cells.

2) The ability to track your work on it in a sane way, which is of utmost importance, especially when collaborating with other people.

The downside is that it might introduce the need for caching mechanisms, as I have done in this case (so you won’t have to run a heavy data loading/model training process for each experiment you wish to perform). In this example, I save the trained models for each configuration so I won’t have to re-run training every time I run the script. Additionally, I save and load the dataset using tf.data.Dataset.save and tf.data.Dataset.load because the dataset creation process can take a while. Finally, I utilize the cachier python package, which generates persistent caching of function results on disk, when loading the movie length information from Cinemagoer as this is a very lengthy process and we want to avoid running this function more than once for each movie.

The required non-standard packages to run this script are as follows: tensorflow, tensorflow_datasets, tensorflow_recommenders, plotly, numpy . Note that the script assumes you have two sub-directories in your main directory named datasets and saved_models.

The training process will be managed by the MovieLensTrainer class. Given training settings such as layer sizes, embedding size, and which features to use, it will either train models or load the relevant models from the disk. After obtaining the trained models for all of the feature combinations we defined via the command-line arguments, we will run the function plot_training_runs which will return a plotly line chart of the top-100 categorical accuracy of each run and will allow us to compare the different settings we used.

An important change I made to the model from the original TFRS tutorial is the ability to use additional features in the MovieModel (the item tower). In the tutorial, this model takes just the movie_title as input. In this project it can take any additional feature, eventually creating a concatenation of the embeddings of all features. This is important as it enables us to leverage additional movie information such as the genres, movie length, average rating on IMDB, and any other feature you may think of.

Command line arguments + Run example

We have several command-line arguments that control what our script does. I will go over the less obvious ones:

layer_sizes (list[int]) — This defines the number and size of the layers following the embedding layer for each of the two towers (query and item models). The more layers we add and the more neurons we give them, the more complex our model will become (and thus more prone to overfitting as well)

additional_feature_sets (list[list[string]]) — This argument is a list of lists, where each inner list is a set of features we wish to train a TFRS model on. For example, using this argument we can train two models, where one trains using the movie_genres, timestamp features, and the other trains using movie_title_text, timestamp features.

--retrain (bool) — This is a binary toggle. When off, we will load a previously trained model with the selected settings if we already trained before (loaded from the file system). If on, we will always re-train and overwrite the models currently on the disk.

--generate_recommendations_for_user (int) — This is an ID of a user for which we will generate recommendations for each trained model that we produced. This is meant to bring our model “down to earth” and see real examples of the recommendations it generates for a user whose preferences are known.

Example of a set of command line arguments for this process (this run will train 6 different models, each for 300 epochs. It will generate recommendations for a user with an ID of 1 for each trained model):

--num_epochs
300
--additional_feature_sets
None
--additional_feature_sets
timestamp
--additional_feature_sets
movie_title_text
--additional_feature_sets
timestamp
movie_title_text
--additional_feature_sets
timestamp
movie_title_text
movie_genres
--additional_feature_sets
timestamp
movie_title_text
movie_genres
movie_length
--embedding_size
32
--layer_sizes
64
32
--generate_recommendations_for_user
1

Let’s view the output of this run (removed warnings for clarity):

These are the ratings for user 1:

Recommendations for model ('timestamp',), user 1:
 [b'Alien (1979)' b'Sense and Sensibility (1995)' b'Sea Wolves, The (1980)'
 b'Ninth Gate, The (2000)' b'Beautiful (2000)' b'Sweet Nothing (1995)'
 b'Titan A.E. (2000)' b'Saint, The (1997)'
 b'Killer: A Journal of Murder (1995)' b'Man of the Year (1995)']
Recommendations for model ('movie_title_text',), user 1:
 [b'Angel and the Badman (1947)' b'Alien (1979)' b'Shadows (Cienie) (1988)'
 b'Two Moon Juction (1988)' b'Sea Wolves, The (1980)' b'Booty Call (1997)'
 b'Apple Dumpling Gang Rides Again, The (1979)'
 b'Psycho Beach Party (2000)' b'Local Hero (1983)'
 b"Shaft's Big Score! (1972)"]
Recommendations for model ('timestamp', 'movie_title_text'), user 1:
 [b'Fried Green Tomatoes (1991)' b'Alien (1979)'
 b'Angel and the Badman (1947)' b'Sarafina! (1992)'
 b'Color Purple, The (1985)' b'Sea Wolves, The (1980)'
 b'Madame Sousatzka (1988)' b'Little Shop of Horrors (1986)'
 b"One Flew Over the Cuckoo's Nest (1975)" b'L.A. Story (1991)']
Recommendations for model ('timestamp', 'movie_title_text', 'movie_genres'), user 1:
 [b'Being There (1979)' b'Alien (1979)' b'Crimes and Misdemeanors (1989)'
 b'Two Moon Juction (1988)' b'Psycho (1998)'
 b'Apple Dumpling Gang Rides Again, The (1979)' b'Me Myself I (2000)'
 b'Loser (2000)' b'Killer: A Journal of Murder (1995)'
 b"Barney's Great Adventure (1998)"]
Recommendations for model ('timestamp', 'movie_title_text', 'movie_genres', 'movie_length'), user 1:
 [b'Sid and Nancy (1986)' b'Angel and the Badman (1947)' b'Alien (1979)'
 b'Two Moon Juction (1988)' b'Crimes of the Heart (1986)'
 b'Sea Wolves, The (1980)' b'One False Move (1991)'
 b'Rich and Strange (1932)' b'Storefront Hitchcock (1997)'
 b'Celebration, The (Festen) (1998)']
Recommendations for model (), user 1:
 [b'Alien (1979)' b'Booty Call (1997)'
 b'Day the Earth Stood Still, The (1951)' b'Angel and the Badman (1947)'
 b'Sea Wolves, The (1980)' b'Celebration, The (Festen) (1998)'
 b'Midnight Cowboy (1969)' b'Desperado (1995)' b'Tingler, The (1959)'
 b'Blue Chips (1994)']

This is just a personal thought, but as user 1 I think I would be pretty disappointed. The only children’s movie that appears in all of the recommendations is “Barney’s Great Adventure (1998)”, even though the user has a clear affinity for these kinds of movies. Plus, we can even see some inappropriate movies (user 1 is categorized as age K-12). Some more modeling improvement and placement of heuristics are needed before this system is production-ready.

Now, let’s view some accuracy results and see if we managed to beat the tutorial benchmark by including more features in the training process.

Part IV — Beat the benchmark — Results

We will train models with the following feature combinations:

1. movie_title, user_id
2. movie_title, user_id, timestamp
3. movie_title, user_id, movie_title_text
4. movie_title, user_id, timestamp, movie_title_text (this setting was used in the official tutorial)
5. movie_title, user_id, timestamp, movie_title_text, movie_genres
6. movie_title, user_id, timestamp, movie_title_text, movie_genres, movie_length

And use the following layer sizes:

[64, 32]

For a total of 6 trained models. We will train each model for 300 epochs and use an embedding size of 32 for all embedding layers. I hypothesize that the additional features will enrich the item/query embeddings and help generate more accurate representations. The metric we will use to compare the models is top-100 accuracy (higher = better). This means, for each item/query pair in the validation set — was the affinity of the pair in the top-100 pairs generated for this query?

Shown in figure 4 (included a static image as well as an interactive graph because of visibility issues on mobile) is the validation top-100 accuracy (in 5 epoch intervals) for each model training.

We can see consistently that as we add more features, the validation accuracy increases. This is encouraging as it means our work had a positive effect! Note that the overall accuracy seems pretty low, this is due to this being a very difficult problem (need to guess the correct movie in the top-100 results out of around 6000 movies). The best approach in terms of validation accuracy was the one that used all of the proposed features, and indeed it beat the benchmark from the tutorial by about 0.015 after 300 epochs.

Part V — Conclusions and thoughts ahead

In this article we have seen:

1. How to build a recommendation system using tensorflow recommenders.
2. How to add context features to both the items and the queries to achieve better performance.
3. How to perform research via plain scripting rather than using Jupyter notebooks.

We can see that additional context, both for the queries and for the items, can boost our recommendation performance (even on relatively dense datasets such as MovieLens-1M). When building your own recommender systems try to think if you can add some more context to your embeddings rather than just using user and item identifiers :)

See you next time,

Elad

References:

[1] Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. https://doi.org/10.1145/2827872

[2] Tensorflow deep retrieval systems tutorial — https://www.tensorflow.org/recommenders/examples/deep_recommenders

[3] Cinemagoer python package — https://cinemagoer.github.io/

MovieLens-1M Deep Dive — Part II, Tensorflow Recommenders was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

A hands-on recommendation systems tour using the popular benchmark dataset

Recommendations. We all consume them. Be it through our favorite movie streaming apps, online shopping, or even passively as a target of advertising campaigns. How are these recommendations created? How does a recommendation system utilize enormous datasets of internet transactions to generate high quality and personalized recommendations? I find these questions fascinating, hence I decided to embark on a learning journey and this post is intended to share my findings with you, dear readers :)

I find that there are sufficient theoretical blog posts on the basics of recommendation systems and decided that this will be a hands-on practical tour of one of the most popular recommendation-focused datasets available — MovieLens-1M [1] (used with permission). In this dataset, we are given ~1 million historical ratings of 2894 movies by 6040 unique users.

The structure of this post is as follows:

• Part I — Exploratory data analysis (EDA)
• Part II — Data pre-processing (retrieving movie content information)
• Part III — Setting up the recommendation problem and obtaining a baseline score
• Part IV— Recommending movies with collaborative filtering
• Part V — Recommending movies with content-based filtering
• Part VI — Conclusions and thoughts ahead

The notebook containing all of this project’s code and more is available here: https://colab.research.google.com/drive/132N2lGQTT1NnzAU6cg1IfwpaeFKK9QQe?usp=sharing

This is the first of a two-part deep dive into MovieLens-1M. The next part, which will be released in the near future, will focus on more advanced algorithms and continue the never-ending EDA process. Happy reading!

Part I — EDA

First, download the MovieLens-1M dataset from here. Then, let’s make some necessary imports and get that out of the way for this project:

The MovieLens-1M dataset consists of 3 files — users.dat, ratings.dat and movies.dat. Let’s explore each of these files and understand what we are dealing with. We will start with the movie's data.

Let’s plot the distribution of movies per genre. For this, we will have to turn the genres column into a list and “explode” it. Note that each movie may be of several genres, and in this case, it will be counted multiple times. We will also plot the distribution of movies per movie release year.

Let’s move on to the users’ data and inspect it. The “occupation” column is encoded with a number representing each occupation, but for EDA we are interested in the actual data. We will get it by extracting the relevant rows from the README file of the dataset and swapping values in the dataset accordingly.

Let’s view the ratings now.

Let’s rank movie genres by their average rating. We will also plot how many times each genre was rated.

Now let’s combine all three tables (movies, users, and ratings) and see if there are any differences between male and female ratings per movie genre. Notice that here too we will normalize by the total amount of males/females to get a better answer to our question.

Again, we see some typical male/female differences in this dataset (men gave higher ratings to actions movies than women and vice versa for romance movies). This is a good sanity check for the data as the plots make sense.

Part II — Data pre-processing (retrieving movie content information)

In this part we will retrieve an important data source that will help us recommend movies later, and this is the movie plot summaries. We will obtain them from Wikipedia, using the Wikipedia python package.

There are 615 NaN movie plots

We see that we were not able to obtain movie plots for 615 movies. This might be because their Wikipedia page doesn’t have a “Plot” category or for some other reason. Since web crawling isn’t the main point here, we’ll just drop these movies and all ratings associated with them.

Part III — Setting up the recommendation problem and obtaining a baseline score

We will choose a user that will be acase study for which we will make actual predictions and see if the make sense to us. Most conveniently, we will choose the user with ID #1. Shown below in the user and the movies that she rated. We can see that she is a female K-12 student (age of 1 is an error in the data but I don’t think it’s very important anyhow). The top-20 movies she likes are mainly classical children’s hits with some anomalies here and there (Schindler’s List, One Flew Over the Cuckoo’s Nest).

We will also shine our spotlight on the movie Toy Story (1995), and for each prediction method we will check which are the movies that are closest to this movie in the embedding space.

Finally, let’s start recommending stuff! We will create two dataset splits:

1. Standard train/test split. This will be used for the rating prediction regression task

2. Leave-one-out cross-validation split. This will be used for hit-rate prediction (which is generally regarded as a more relevant metric in the field of recommender systems). The split is performed by taking one movie per user out of the dataset and using this as a train set. All the other user’s movies will be on the trainset. Then, the hit rate is the percentage of times the movie we took out was in the top-K ratings for that user.

For each method we will calculate:

• RMSE on test set
• hit-rate on leave-one-out cross validation set
• Top predictions for user #1 (case study)
• Top-10 most similar movies to Toy Story (1995)

Now, let’s define some auxiliary functions to help us evaluate our algorithms (mainly — the hit rate). We will inspect two measures for each algorithm — the RMSE and hit rate. Some of the code here is taken from the fabulous course on recommender systems by Sundog Education. I will also implement a simple tweak to the original KNN and SVD algorithms because calculating the hit rate takes a long time and I want to at least be able to measure the progress in real-time.

Let’s obtain results for a completely random recommender system, so we will be able to evaluate if our algorithms are better or worse than random (after all, we want to make things better not worse…)

• Note — hit rate takes a lot of time to compute

RMSE: 1.4960
HitRate: 0.02185430463576159

Part IV — Recommending movies with collaborative filtering

We will use the classic vanilla algorithms, out-of-the-box from the Surprise package (with some minor tweaks to show progress in KNN). I won’t go into the algorithmic details here because I think there are many great tutorials out there for learning these things (just google recommender systems SVD/KNN and you're good to go). I will check these algorithms:

1. SVD
2. User-based KNN
3. Item-based KNN

We will start with SVD:

RMSE: 0.8809
HitRate: 0.03923841059602649

Nice! That’s a major improvement over the random score. But when we look at the predictions for user #1 they still feel a bit “off”. The only children’s movie is Babe (1995) and I personally would expect more of these to appear in a good recommendation for this user.

Now onto KNN. We will start with user-based KNN. This means that predictions for movie m and user u will be based upon the tastes of users that are similar to u.

RMSE: 0.9595
HitRate: 0.002152317880794702

The RMSE is better than random but the hit rate is worse. We can also see that the top predictions for user_1 seem even further away than those that were given by the SVD algorithm. In user-based KNN there is no notion of similarity between items, so we won’t calculate the most similar movies to Toy Story (1995) for this algorithm.

Let us now check the performance of item-based KNN, which recommends items to users by the similarity of the items they already rated to other items they have not yet rated. Code-wise only a slight change to the class instantiation is required.

RMSE: 0.9851
HitRate: 0.002152317880794702

The RMSE is a bit worse than the user-based KNN and the hit rate reaches almost zero. The most similar movies to Toy Story (1995) seem reasonable but the predictions for user_1 are once again unsatisfying. So far the KNN algorithms have failed to outperform the SVD, which really is regarded as a much better algorithm for collaborative filtering.

Up until now, we haven’t taken into account the features of the movies themselves. Maybe if I know something about the details of the movies it will help me make better recommendations? Let’s check it out. This is called content-based filtering.

Part V — Recommending movies with content-based filtering

For the content-based filtering we will use KNN-based algorithms in three approaches (two of them item-based and one user-based):

1. Movie plots (item-based): Create a vector representation of all of the movies based on the plot descriptions. We will do this by first stemming all of the words in the plot description and then applying TF-IDF to vectorize each document. The similarity matrices we will generate will be based on:

a. Using the complete TF-IDF matrix

b. Using the TF-IDF matrix after feature selection

c. Using the TF-IDF matrix after feature selection and after removing peoples’ names

2. Movie genres (item-based): We will use the movie genres as the only source for recommendations and see how that goes.

3. User age+gender (user-based): We will use user data as features for our KNN predictor.

We will create a class that inherits from Surprise’s KNNBasic. Its functionality will be the same as item-based KNN for collaborative filtering, with the only difference being that we will supply a pre-calculated similarity matrix to the fit function, instead of having it calculated from the rating data.

Part V.I — Content-based filtering using movie plots

Now, let’s create the TF-IDF similarity matrix

Now, get the results for approach #1 (using the complete TF-IDF matrix). Note that we need a different cosine similarity matrix for the regular trainset and the leave-one-out trainset which is used to calculate hit-rate because they contain different inner ids for each movie.

RMSE: 1.0268
HitRate: 0.003973509933774834

Worse than collaborative filtering, without a doubt. We also see that the similarity matrix doesn’t really generate meaningful relationships between movies (the only children’s movie which is similar to Toy Story is Toy Story 2). Let’s see if reducing the number of features helps.

Number of selected features:  784

List of selected features:   ['abbi', 'abel', 'ace', 'adam', 'adel', 'adrian', 'adrien', 'affair', 'agent', 'agn', 'al', 'aladdin', 'alan', 'albert', 'alex', 'alfr', 'ali', 'alic', 'alicia', 'alien', 'allen', 'alli', 'alvin', 'alyssa', 'amanda', 'amelia', 'american', 'ami', 'amo', 'andi', 'andrea', 'andrew', 'angel', 'angela', 'angelo', 'angus', 'ann', 'anna', 'anni', 'antoin', 'anton', 'antonio', 'ape', 'archer', 'archi', 'ariel', 'arjun', 'armstrong', 'arni', 'arroway', 'art', 'arthur', 'arturo', 'ash', 'audrey', 'aurora', 'austin', 'axel', 'babe', 'babi', 'balto', 'bambi', 'band', 'bank', 'barbara', 'barn', 'barney', 'bastian', 'bate', 'bateman', 'batman', 'beach', 'beal', 'bean', 'beatric', 'beckett', 'becki', 'beldar', 'bella', 'ben', 'bendrix'...

RMSE: 1.0314
HitRate: 0.003642384105960265

Didn’t really help…We see that a large portion of the informative features are names. We don’t want that because names don’t really say anything about the movie content. Let’s remove the names.

RMSE: 1.0284
HitRate: 0.0041390728476821195

OK, this is somewhat more reasonable than the previous two approaches. First of all the hit-rate is higher (though still way less than the collaborative filtering methods), and second the similar movies to Toy Story(1995) are actually similar. Third, the top predictions for user_1 even seem passable. Let us now try the movie-poster approach and see how that goes.

Part V.II— Content-based filtering using movie genres

In this part I will tune down the algorithmic complexity and create a KNN recommender based only on the movie genres. This means that if a person rated children’s movies very high, we should expect the algorithm to recommend children’s movies. We will create all possible combinations of genres using pythons itertools package and then apply our old friend TF-IDF to generate a feature matrix from this data.

RMSE: 1.0138
HitRate: 0.0031456953642384107

Unsurprisingly the most similar movies to Toy Story (1995) are those that have identical or almost identical genres. Other than that the results are quite confusing, why did user_1 get recommendations only for war movies? Let’s remind ourselves of the table in part III of this article (which shows all of the ratings of user_1), we see that there are two movies with genre “war” in user_1’s rating list and both of them got a score of 5. The other genres that user_1 has rated appear more frequently and thus if we average user_1’s scores by genre — the “war” genre will receive a perfect score while the other genres won’t.

Because the item content-based KNN recommender predicts new items for user_1 by their similarity to other items that have been scored by user_1 it will give a perfect score for each war movie it will see in the context of user_1! The KNN recommender ignores scores belonging to 0 similarity items, it will only take into account the war movie scores when generating the recommendations for war movies and thus always predict perfect scores.

Part V.III — Content-based filtering using user User age+gender

Let us start by creating a feature vector from the users’ age and gender. We will assign 0 to ‘Male’ and 1 to ‘Female’ and we will take the age column as-is. Finally we will normalize the columns so as not to be affected by the difference in column magnitudes. Using these values we will create a cosine-similarity matrix as we did for the movies, but this time we will calculate a similarity score between each user. This similarity score will be an input to our user-based KNN recommender.

RMSE: 0.9784
HitRate: 0.00347682119205298

We see that the RMSE is a better than for the item-based methods, similarly to the results we got in part IV (collaborative filtering), where the user-based recommendations outperformed item-based ones. but the hit rate is a bit worse than some of the item-based methods we tried. It seems that each method has its strengths and weaknesses and a good approach will probably be to somehow combine them all, and this will be the focus of the next part.

Part VI — Conclusions and thoughts ahead

In this post, a lot was covered.

First, we inspected the MovieLens-1M dataset and got some pretty interesting insights from the graphs we saw, such as which movie genres tend to score higher than others in average. We then used some vanilla recommender systems algorithms from the Surprise python package, and got some pretty good results with the SVD algorithm.

After that, we tried our luck with content-based filtering, but unfortunately, this turned out to be a futile attempt. Despite that, in the research process we unveiled an inherent weakness in KNN recommender systems (or content-based filtering in general), which is that the recommendations for a specific user are based solely on the items that he/she has interacted with in the past, and if there was bias in this user’s interactions it will show up in the KNN recommendations. This is why our spotlight user (user_1) got top scores for war movies when we used genres as the basis of our recommendations, even though we as humans wouldn’t necessarily agree. A more probable recommendation for this user would be children’s/musicals and the like.

Additionally, we saw that when dealing with natural text, such as movie plots, it is worthwhile to understand which features are most affecting the recommendations. When we removed person names which were ranked very high in the TF-IDF algorithm, we suddenly got better results and this was clearly visible when the movies that were most similar to Toy Story (1995) suddenly made sense.

In the next part, I will combine content-based filtering and collaborative filtering and thus have the best of both worlds. I already have my eyes on the TensorFlow-recommenders package and I cant’ wait to give it a go.

Until next time!

Elad

References:

[1] Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. https://doi.org/10.1145/2827872

MovieLens-1M Deep Dive — Part I was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is a super-short weekend project. We are going to create a simple GUI application in python which highlights specific colors in an image and reports which percentage of the image was highlighted. Let’s show an example of the GUI:

Let’s start our project with some necessary imports:

Let’s create a MainWindow class which will be the one and only window in our application. We subclass the PyQt class QMainWindow and customize it to suit our needs. The main layout of this application will be a QVBoxLayout which stacks widgets vertically. Inside this layout, we will nest some QHBoxLayouts, which stack widgets horizontally. We also define additional variables which will be of use later in the code.

Let us continue the code for this class. We define the “select image” and “process buttons” and link their clicked event to some functions we will define later. We also set the RGB selectors, where each of them is a QSpinBox. We must define the maximum and minimum values so a user won’t enter illegal values (only numbers between 0 and 255 are valid RGB values).

We continue with the MainWindow class. We add the previously created widgets to the top_bar_layout. Then we start creating the widgets in the middle bar — image_bar_layout. This layout contains the source and processed images, including their labels. We also limit the image sizes by using the setMaximumSize methods, so our screen won’t explode with high-res images.

The following code ends the __init__() function of the MainWindow class. We define the bottom_bar_layout which will contain the “Save as file” button to save the processed image. It will also contain a label that will report which percentage of the image has been traced. We will end with the setCentralWidget method that will tell the MainWindow which widget to use. We set a dummy widget called widget whose whole purpose is to contain main_layout, which contains the “meat” of this window.

The following code contains the remaining methods in the MainWindow class. choose_source_image is used to select an image from the local filesystem. Notice that we resize the image. This is support the processing of larger files by the program (Qt limits image sizes to 128 megabytes).

process_image does the actual tracing by color. It later resizes the result image and sets the previously defined self.result_image to show the result.

save_as_file simly utilizes the imwrite function of opencv to save the result in the local file system, given a filename obtained from a filesystem dialog.

The following are helper functions which 1) resize the image using opencv, 2) create a Qt pixmap from an opencv image, and 3) apply the tracing process, which is the whole purpose of our program.

Finally, here is the whole code from start to finish:

As a bonus, to create a python executable from this, all you need to do is pip install pyinstaller and run pyinstaller image_tracer.py from the directory containing the code!

Thanks all, see you in the next post :)

Create a simple GUI image processor with PyQt6 and OpenCV was originally published in Towards Dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hello Medium. I have decided to pick up on the super-relevant subject of predictive maintenance and start with the industry gold standard — the NASA Turbofan Engine Degradation Simulation Data Set (https://ti.arc.nasa.gov/tech/dash/groups/pcoe/prognostic-data-repository/#turbofan).

My journey was highly influenced by Koen Peters’s awesome series regarding the analysis of this dataset and his solutions to the problem.

The dataset is a set of four different datasets: FD001, FD002, FD003, FD004 which are different variations of the same problem. This post will focus on the first dataset: FD001 .

Each of these datasets consists of numerous multivariate time series of simulated sensor measurements from a fleet of engines. The dataset was used as part of the Prognostics and Health Management (PHM) data competition at PHM’08. The contestants in this challenge were not given any background into the sensor data and were asked to perform pure data-driven predictions, thus we will act accordingly.

Each time series of sensor measurements starts with a different degree of engine wear and ends at system failure. In other words, we have data from N different scenarios of engine system failures, with N being the number of time individual time series. The test set in each dataset consists of time series with sensor measurements until a certain point in time before system failure which is referred to as remaining useful lifetime (RUL). The goal is to predict the RUL for each time series in the test set as accurately as possible.

The layout of this post is as follows:

• Part I — Exploratory data analysis (EDA): In this part, we will perform some basic plotting of the data and understand things such as which sensors contain relevant information and which do not, and we will try to understand the overall behavior of the data concerning the predicted variable.
• Part II — Baseline model: Here we perform some basic preprocessing and implement a very naive solution to predict the RUL for each time series in the test set. The results for this part will be our baseline score to beat when we try other, more advanced, methods.
• Part III — Feature engineering: Here we will try two methods of feature engineering. One automatic, using the python package tsfresh , and the other manual, using common sense :)
• Part IV — LSTM: In this part, we will use an LSTM neural network to process the raw data and predict the RUL for each time series in the test set.

Part I — EDA

We will start with the necessary imports. These are for the whole project.

Now we will start by loading the data and viewing a description of each column:

And view the first five rows as well:

We immediately suspect sensors 3, 4, 8, 9, 10, 11, 13, 19, 21, 22 to be non-informative because their standard deviation is very low in relation to their mean value. Sensors 25, 26 contain only NaNs, so we can just drop them.

Usually, having no domain knowledge hinders our ability to extract useful information from the data. Despite this, we will use this to our advantage. Because we don’t know anything about the individual sensors, we can start with normalizing all the data to be on the same scale, this will make our analyses much easier.
We will fit a MinMaxScalerto each sensor column, and save these scalers for later use (we will need to perform the same transformation on the test data). After that, we will plot the first unit’s measurements of each signal to verify our assumptions regarding which signals don’t contain relevant information.

The interactive plot below, of the first sensor recording in the dataset, allows us to visually divide the signals into four groups:

• irrelevant 3, 4, 8, 9, 13, 19, 21, 22
• upward trend: 5, 6, 7, 11, 14, 16, 18, 20
• downward trend: 10, 12, 15, 17, 23, 24
• no trend: 1, 2

Part II— Baseline Model

We will now drop the irrelevant columns, and continue to add labels to the data — the RUL. Our first labeling method will be naive and straightforward. We know that each unit in the training set is run until RUL = 0. Thus, we will assume that the degradation is linear and label each time step in the training set accordingly.

Great, now every row has a label. We will now perform regression where each row is a data sample, with x = row sensors and y = RUL. As in Koen Peters' post (https://towardsdatascience.com/predictive-maintenance-of-turbofan-engines-ec54a083127), we clipped the RUL values at a maximum value of 125 to match the engine degradation trend better.

Alright, we have our first model. We will use a vanilla XGBoost model throughout the post because of it is generally considered a very good performing model out-of-the-box. Now, let’s load the test set (and not forget to apply the scalers to the test set as well). Below we can see our baseline performance on the train and test sets.

Output:

RMSE on train set: 17.679585838619715
RMSE on test set: 17.35303931484297

Alright, so 17.353 is our test RMSE to beat in the following sections.

Part III — Feature engineering

Part IIIa — Windowing the data + a naive regression approach

Instead of taking each feature instance to be one timestep of sensor measurements, let’s try using a window of 20 timesteps. The rational behind this is that maybe data from the near past will help determine the RUL of the current timestep.
This operation of course will leave us with 3D data (n_samples, timesteps, n_sensors), which we will have to featurize or flatten in order to keep using traditional ML methods.

Our first attempt at this windowed approach will be to flatten the dimensions of each window and simply feed all of it to XGBoost. This will of course hide the time-related information from the model.

Output:

RMSE on train set: 15.908527707436281
RMSE on test set: 16.429387222606096

Surprisingly, this approach improves our test RMSE. The XGBoost model was able to make use of the previous 19 samples for each sensor to improve its RUL prediction ability, despite the unstructured nature of the input data (one long array containing measurements from all sensors).

Part IIIb — Automatic feature generation

Let us now try to generate features from each mini-time series of 20 time steps. We will use the python package tsfresh which generates features automatically from a set of fixed operations.

Output:

RMSE on train set: 14.486881328112556
RMSE on test set: 23.113512123440437

For computational efficiency we used only the features that are not computationally heavy to calculate. For each sensor in each window the following were calculated as features: root_mean_square, mean, median, sum, maximum, minimum, variance and standard_deviation. This resulted in a dataset containing 118 features.

We can see that the results are extremely overfitted for this method and it proves to be quite unsuccessful compared to the naive windowing approach.

Part IIIc— Manual feature generation

In this part, we will apply several operations to the measurements of each sensor, in each window. We will avoid creating features that combine sensors because we lack the domain knowledge (for example, the meaning of the sensors) required to generate these kinds of combinations. This approach is quite similar to the automatic feature generation, except the np.difffunction, which computes the difference between each two executive measurements, that was used here and isn’t used in the automatic version.
The operations for each sensor will be as follows:

• average value
• average diff value
• standard deviation
• maximum value
• minimum value

Output:

RMSE on train set: 14.58791927615462
RMSE on test set: 18.131375854948576

This approach is disappointing as well, with worse test RMSE than the naive windowing approach. Maybe we will have better luck with the LSTM model :)

Part III — LSTM

In this section we will use a simple LSTM network to process the data windows ans see if we gain any improvement from our baseline model (which the feature approach failed to deliver). We will use Pytorch lightning (the equivalent of Keras for Tensorflow) to streamline our model creation process.

Now let’s create the LSTM model:

Now, create the trainer and fit the model with the training data:

Finally, get the train and test RMSE for the LSTM approach:

Output:

RMSE on train set: 14.761564144403062
RMSE on test set: 16.065719776970667

We can see that the LSTM approach yielded the best test RMSE for this dataset, and this concludes this article. For future work it is worth investigating the feature generation approach and the reasons for it’s failure. (maybe some domain knowledge will be of use). Moreover, it may be beneficial to experiment with other deep learning architectures and not just LSTM (for example CNNs, transformers).

Full notebook link: https://colab.research.google.com/drive/12O670WevZRLm4AvQ_JtyU5cHN5PMnB1C?usp=sharing

Hope you enjoyed reading!