Let's start by setting up a dummy model, dataset, and hugging face trainer (forget about optimality, or if it makes sense at all, we just want to make sure it works end to end without failing).

Ok, so we are using IMDB dataset to classify reviews as good or bad. This a classification problem and hugging face correctly setup the loss function and everything else for us… But! Let's go the hard way, now we are going to use our own loss function, how?

Subclass Trainer and implement the method compute_loss:

What if my custom trainer is not receiving the labels?

This happened to us. You might have “remove_unused_columns=True”, meaning the trainer is dropping the inputs that are not expected on the forward function, so either set “remove_unused_columns=False” or add the labels arguments to your module:

How that works is by using “inspect.signature”, which gives you the ability to read the parameters of the given function: https://github.com/huggingface/transformers/blob/f1660d7e23d4432513fe060bde4f9b7b29f05204/src/transformers/trainer.py#L755

How to use a custom loss with hugging face was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Resources

By far the best way to learn about it is watching this three videos from databricks:

And here you have the documentation: https://www.mlflow.org/docs/latest/quickstart.html

Index

1. What is mlflow?
2. Server mode
   2.1. Experiments tracking
   2.2. Model tracking
3. Client mode
   3.1. Tracking experiments
   3.2. Running models
4. What else can you do with mlflow?

1. What is mlflow?

Basically it’s a library and through the library you can do a ton of things. It can be used as a client library to run experiments, execute projects, build models, etc. And is possible to run it as a server to receive experiment metrics, artifacts and even start a http server to serve a machine learning model… All from the same package.

2. Server mode

2.1 Experiments tracking

By running mlflow server you will start a http server that can receive tracking information to be stored on a database (sqlite, mysql, progresql…) and artifacts (google cloud, s3…). Example:

mlflow server \
--default-artifact-root gs://<bucket>/<folder> \
--backend-store-uri postgresql://user:pass@host:port/db

The UI offers very a nice and intuitive way to interact with the uploaded information.

You can compare different runs to find out what performed better using different models and/or different set of parameters.

Even includes a section for hyperparameter tuning.

2.2 Model tracking

For each experiment you can (and should) log artifacts including the model used. MLflow offers functions for automatically logging models to make it easier, but you can log anything you want (images, text, audio, binaries, parameters, metrics…). The cool thing about this models is that it adds a conda.yaml file to specify the dependencies. No more headaches with conflicting package versions or figuring out the setup of the author! (you might have headaches to setup the file correctly tho).

Another cool feature is the ability to upload a sample input and the schema for the expected input and output. If the schema is provided mlflow can validate it on inference time to reject invalid calls.

From the above interface you can create a model. A model is just a promoted experiment. Models have a unique name, are versioned and have stages so you know which one is in production stage easily.

3. Client mode

3.1 Tracking experiments

Tracking experiments is as simple as doing this:

with mlflow.start_run() as run:
   rfr = RandomForestRegressor(**params).fit([[0, 1]], [1])
   mlflow.log_params(params)
   mlflow.sklearn.log_model(rfr, artifact_path="sklearn-model")

This will record all parameters used to create the RandomForest (including the default values) and upload the model fitted with some extra parameters like the train set scores, oob score, etc.

After doing this multiple times you can go to the dashboard, select the model that performs better and promote it to a mlflow model.

3.2 Running models

Once you have a model you can load it with the command line executable:

mlflow models serve -m models:/model-name/version

The model might or not might contain a conda.yaml file. If it contains a conda.yaml all the dependencies and the correct python version will be installed automatically, that means the inference environment will be the same the research environment used to train the model.

4. What else can you do with mlflow?

This is just an introduction to the many features mlflow offers. There are many more things you can do with it:

• Parameterized tasks
• Execute jobs on kubernetes
• Create docker files
• Authenticated connection to the tracking server via http basic auth or ssl certificates
• Dataset versioning

MLFlow for MLOps was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Paper: From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

Another github repo implementing sparsemax:
* https://github.com/aced125/sparsemax

What can we learn from implementing this paper?

1. A new layer capable of producing sparse outputs while being differentiable everywhere.
2. How to implement the backward step.
3. How check the gradient is correct.

1) Sparsemax, a new layer

From the abstract:

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. […] We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.

So it can be an useful tool used on the intermediate layers (like for TabNet as an attention mechanism) or as the last layer for multi classification problems.

The algorithm is:

It’s pretty straight forward:

1. Sort the input on descending order.
2. Find the maximum index (from the sorted set) that meets some condition.
3. Calculate the threshold (tau(z)).
4. Subtract the threshold to each element of the original set and apply relu operation (convert negative values to 0).

A naive way of implementing it might look like this in numpy:

But this won’t work well when using a batch. With batch execution (2d case) where we have “columns” and “rows” looks as this in pytorch:

We can confirm it’s working fine by using the same test data with other existing implementations like I do on the demo notebook:…

2) The backward step

Typically custom pytorch functions are implemented this way:

ctx: is a “bag” where you can store information required for the backward step. So if you need something from the forward like the output, the input, etc you can store it on the “ctx” variable to use it later.

grad_output: the gradient of the previous step.

grad_input: the gradient of this function with respect to the output. Yeah it’s bit confusing to call something grad_input when is a variable you are returning.

Now the tricky thing is… how to implement it for sparsemax? On the paper you’ll find this formula that looks promising:

Ok, that’s the jacobian matrix of the function. Oh wait that’s sounds scary, what is a jacobian matrix? It’s a matrix composed of all partial derivatives of the output with respect to each element of the input.

Now that we have how to implement the backward step… not so fast! Actually if you implement that won’t work (trust me I already tried :D). We are missing to include the gradient of the previous step.

Lucky us the authors mention how to do it on the paper:

It can be implemented like:

3) Checking the gradient.

Pytorch offers the function “pytorch.autograd.gradcheck” to check the gradients are correct: https://pytorch.org/docs/stable/autograd.html#torch.autograd.gradcheck

Using it is pretty straight forward.

Conclusion

Implementing the gradient is much easier than I expected.

Google colab notebook with the code:
https://colab.research.google.com/drive/1YMIQgegWN2b_m1k5nwrOHOBd5B9ASV9E?usp=sharing

Sparsemax: from paper to code was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Related papers:

• https://arxiv.org/abs/1705.08741: Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• https://arxiv.org/abs/1908.07442: TabNet: Attentive Interpretable Tabular Learning

Description

This paper aims to solve the issue of the “generalization gap”. It seems neural networks tends to do worse for unseen data when being trained on large batch sizes.

One of the ways proposed to fix this is changing batchnorm layers from calculating statistics (remember BatchNorm layers changes the input data to make it “normally distributed”, 0 mean 1 standard deviation) from using the whole batch, to use small parts of the batch, what we can call nano batches (I just make it up, this term doesn’t appear on the paper).

Why this work?

It isn’t mentioned on the paper why this helps. My intuition is that as we will be changing small parts of the batch independently (and differently for each epoch) it’s like adding noise or maybe doing random data augmentation. We make it harder for the network to learn so it must become smarter!

I wonder if it’s the same as adding different noise to different part of the batch

Indeed it seems to be the case!! I did some tests on this colab notebook https://colab.research.google.com/drive/1cRNltVKTpkO47Wk85AaGPdmAkQmNHpyP#scrollTo=jtVydfmCY-2X and found similar results between GhostBatchNorm and what I’ll call NoisyBatchNorm, a method to add different noise to different slices of the batch.

The algorithm

Now let go for the meat and potatoes. The algorithm from the paper:

Might look bit cryptic, but the idea is simple.

• Calculate mean of each nano batch.
• Calculate std of each nano batch.
• Update running mean using an exponential moving average.
• Update running std using an exponential moving average.
• Return whole batch normalized.

Knowing this the following image should be easier to understand:

There is not much to say about mean and standard deviation, it’s what it is. Note the blue lines, I put them to make clear the 𝜖 is not inside the sum.

What does 𝜖 here? It helps with numerical stability. What if for whatever reason the operation under the square root is 0? Everything will fail. Instead adding a small value (like 1e-5) ensure we never have 0. This is one of those little details you must know when implementing papers, but it isn’t mentioned.

The exponential moving average is a simple trick to calculate a moving average using only the last value and the new value:

momentum = 0.1
old * (1- momentum) + new * momentum

Reviewing the algorithm

First of all this paper is pretty cool, I don’t consider myself smarter than the authors! But we all can make mistakes, that’s why there are reviews, etc. You can expect to see errors or things that aren’t well explained on papers (indeed papers have different versions with fixes). I think doing the following changes fix some problems with the original algorithm, missing/incorrect indexes.

Implementation

One naive way to implement this would be by doing everything with loops and that will be very very inefficient. Instead I’m going to show you the vectorized version directly:

And now the super simple & smart way:

What a difference! This last option is pretty cool. It takes every nano batch and calls an embedded batchnorm, instead of re-implenting batchnorm from scratch. This is from pytorch-tabnet repo.

Benchmark time

Which option is faster? Running both versions gives me this:

We can see for ghost batch sizes (< 512) the vectorized version is faster because we aren’t using loops, and as we get closer to the real batch size the second option (calling a batchnorm layer multiple times) is better. This is due to the fact that the pytorch implementation of batchnorm is highly optimized in C.

Conclusions

• Implementing papers can be pretty hard, even for simple algorithms like this one. Isn’t always easy to explain with math what you can explain quite easily with few words.
• It’s worth it to know how to write vectorized code, the improvement can be substantial.
• If you want to use large batch sizes consider using GhostBatchNorm.

Ghost BatchNorm explained was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

From paper to code

I have started looking into an architecture called TabNet that aims for tabular problems interpretability. As part of it I’ll do couple of posts about some of its components, in this case about GLU activation (gated linear units). Next one will be about Ghost BatchNorm.

Gated Linear Unit

Related papers:
* TabNet: https://arxiv.org/abs/1908.07442
* Language modeling with Gated Convolutional Networks: https://arxiv.org/abs/1612.08083

The idea is simple. I want to allow the network to decide how much information should flow through a given path, like a logical gate, hence the name. How?

• If we multiply X by 0, nothing passes.
• If we multiply X by 1, everything passes.
• If we multiply X by 0.5, half of it passes.
• …

Does this reminds you something? It does right? LSTMs! Because it’s inspired by the idea of the gates of LSTMs, but applied to convolutions and linear layers, but it’s the same idea.

How can we obtain a value between 0 and 1?

C’mon that’s an easy one!… Yes! The sigmoid function!

Implementation

The formula from the paper looks as this:

Sigma means the sigmoid function. So we have two set of weights W and V, and two biases, b and c. One naive way to implement this is:

• X*W + b is just a linear transformation, we can use a linear layer for it.
• Same for X*V + c.
• Then apply the sigmoid to one of them and we are done.

Making it faster

I said naive because there is a better (faster) way to implement it, using a fancy name like FastGLU it becomes:

What’s the difference? On the first snippet we do two matrix multiplications (when calling linear1 and linear2), and for this last one only one matrix multiplication, followed by splitting the output appropriately, but it’s the same thing.

How well does this compared with other activations?

From the paper:

[…]
In contrast, the gradient of the gated linear unit ∇[X ⊗ σ(X)] = ∇X ⊗ σ(X) + X ⊗ σ 0 (X)∇X (3) has a path ∇X ⊗ σ(X) without downscaling for the activated gating units in σ(X). This can be thought of as a multiplicative skip connection which helps gradients flow through the layers.
[…]

It reminds me ResNets.

Experiment time

I run some experiments to know what will happen if I compare it other activations: Sigmoid (because we are using a sigmoid here) and ReLU (because is the de facto standard).

If we ignore green and purple by now (2 linear layers + relu) we can see that FastGLU and SlowGLU versions perform quite similar, and better than the sigmoid version.

The purple line is so bumpy because values are too small (y axis is in log scale), it doesn’t really change much. Green line suffers from dead relus (must read https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b).

Notebook can be found here: https://github.com/hermesdt/machine_learning/blob/master/GLU.ipynb

Conclusions

Not much conclusions to made from this to be honest, but here are some:

• ReLU actually for this specific case did amazingly well, but it’s very unstable. This is just a toy to play with, but still this applies to real world project, there is no silver bullet.
• GLU is far more stable than ReLU and learns faster than sigmoid.
• I already tried with other fancy activations like Mish, not better than GLU here.
• Watch out for dead ReLU.

GLU: Gated Linear Unit was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Mean absolute value

Useful when having outliers but you don’t want them to have a big influence.

(y-ŷ).abs().mean()

Couple downsides with this loss:

• The derivative is always the same (+/- 1).
• The function isn’t differentiable at 0.

Huber loss is a good alternative for this reasons.

Mean squared error

Useful when you want to penalize big differences (high errors).

(y-ŷ).pow(2).mean()

Root mean squared error

The problem with MSE is that the obtained value is not in the original unit but on the root of the original unit, that means if we are using seconds, the value will be in seconds², which isn’t really useful.

To recover the original unit we take the square root obtaining something like an average where we strongly penalize big values. RMSE can be useful to use on reports in combination with other metrics like the MAE.

(y-ŷ).pow(2).mean().sqrt()

Huber loss

Similar to MAE, but differentiable everywhere with a smooth change in trend near 0. Stronger against outliers than MSE. It’s like a combination of L1 and L2 (MAE and MSE), depending on the magnitude of the difference between the target value and the predicted value. 𝛿 is another hyperparameter that indicates where to start applying a calculation similar to L2 (when approaching 0), L1 like otherwise.

delta = 1
mask = (y — ŷ).abs() <= delta

up = (0.5 * (y — ŷ)**2) * mask
down = (delta * (y -ŷ).abs() — 0.5 * delta**2) * ~mask
(up + down).mean()

List of deep learning losses for regression was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Classification & Probability

Classification & Probability? Why mix both? Well, because classification indeed can be thought as fitting a probability distribution, bear with me. I’ll comment a bit about how to fit distributions and then relate it to classification.

Probability

The loss used for fitting a probability distribution is the negative log likelihood, I explain it with more detail here. What it does is given a distribution defined by some parameters (obtained from the output of a NN in this case), what is the probability of observing, (the target / label) Y? And then update the parameters through backpropagation in the direction where we maximize the probability of observing Y with a different set of parameters.

The observation Y won’t change obviously, as we want something that can fit as best as possible the generation of those observed Y (for future inference), we will change the parameters towards increasing the likelihood of obtaining those observations if sampling from the probability distribution.

One last time, we want a probability distribution parameterized from past data to predict some future data Y given some input X.

Each probability distribution have a different maximum likelihood estimation function, so I’m not going to enter into more detail here but we can reduce the general case to the following, where “p” is the maximum likelihood estimation function (I’m omitting the parameters of the distribution) (some examples gaussian, categorical, poisson…):

And we are lucky because deep learning frameworks already provides them!

Classification

Classification can be seen as using probability distributions, for the binary case we have the bernoulli distribution, for multi-class we have the categorical distribution, and you can think on the binary case as a special case of the multi-class classification.

If we have a probability distribution we know we can fit it with the negative log likelihood as mentioned above.

Also you probably already know we also have cross entropy and binary cross entropy. More weirds terms! Why?! :’(… Don’t worry, although cross entropy and NLL isn’t the same, for our case it’s the same (very well explained here), or better say the math formulas at the end simplify to the exact same thing, therefore cross entropy = binary cross entropy = NLL. They just exposes different apis, but they are the same at the end. Don’t you believe me? Take a look on the following code:

Why so many options for doing the same thing? Depending on the use case you want to use one or the other. Binary? Multi class? Logits? Normalized? …

Why negative on the “negative log likelihood”? Because the frameworks usually only implements miniziation, so we maximize the likelihood by minimizing the negative value.

Deep learning losses for classification & probabilities was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Until M4 competition everyone thought that traditional statistic models were superior for forecasting, as happened with many other fields I would say. Today it seems deep learning is the best solution almost always… at least having enough data. Traditional forecasting techniques like ARIMA, etc seems to be doing better for small datasets still.

What about huge datasets where we have many different time series? Like amazon for example, or any retail business. Would be amazing to be able to throw all of that into a model and make predictions for each product/user/… individually.

And would even better to be able to use exchangeable time series. Like we have the history for a given author, probably another book of the same author of the same topic will have similar evolution, or a book from another author but same topic.

And even better if we can solve the cold start problem for fresh new products for which we have similar data.

Too good to be true… well actually that’s what DeepFactors does!

Paper Deep Factors for Forecasting: https://arxiv.org/abs/1905.12417
Older paper about the same architecture: https://arxiv.org/abs/1812.00098

I have to recognize I struggled to understand it (well that happens to me with all papers...), so to make it easier for you here you have a comprehensive list of terms, but first some excerpt from the paper:

Our new method is data-driven and scalable via a latent, global, deep component. It also handles uncertainty through a local classical model. We provide both theoretical and empirical evidence for the soundness of our approach through a necessary and sufficient decomposition of exchangeable time series into a global and a local part. Our experiments demonstrate the advantages of our model both in term of data efficiency, accuracy and computational complexity.
[…]
In this paper, we propose a novel global-local method, Deep Factor Models with Random Effects. It is based on a global DNN backbone and local probabilistic graphical models for computational efficiency. The global-local structure extracts complex non-linear patterns globally while capturing individual random effects for each time series locally.

Fixed? Random? Global? Local? What’s that?

• Fixed effect model: This just means that the model is deterministic, the output is always the same for the same input.
• Random effect model: The output is random, based on some probability distribution, ie. non deterministic.
• Global model: In this case refers to a model that learns some information shared across many time series.
• Local model: One model per time serie. Traditional statistical methods such as ARIMA are of this type.
• Global-local: Some sort of combination of shared high level information between time series and specific information for each specific time serie. First the global information is calculated and used to predict the local, specific, time serie.

Architecture

In the following graph we have the theoretical architecture of the model. Note about “g” and “r”.

g: global effect, common for all (no i subindex).
r: random effect, specific for each one.

Ok, not much to say about it, you have all you need to understand it on the paper. Let take a look on the implementation on gluon-ts.

DeepFactors gluon-ts architecture

The following schemas is based on gluon-ts repo code. I always find useful to see how to implement the esotherical maths from the papers.

Global model: The global model only receives the time features, that is, the time series past data plus some other useful features like hour of the day, day of the week, etc.

Deep Factors: The global deep factors comes from applying a matrix multiplication to the output of the LSTM, that’s almost the same as saying that we project that output into a different dimensional space, on the paper they use 10 as the rank of this new space.

Fixed effects: Now given those global factors we want to use them differently per category, therefore an attention block with the categories as input is used. The use of the category through an embedding layer should help to focus on different parts of the factors for each exchangeable time serie.

And with all of this we have the deterministic part of the model, the average without any kind of uncertainty associated. The output is 1 number per batch item.

Random effects: This is calculated from a combination of input features concatenated with the embedding of the category. The output is 1 number per batch item.

And now we have 2 numbers we can use as parameters of a gaussian distribution (code based on the gaussian process alternative).

Improvements

I’m far from being smarter than the people that coded this beautiful thing, but there are somethings I would like to try to see if it helps:

• QRNNs: https://arxiv.org/abs/1611.01576 Improved training speed.
• Activations: There are no activation layers in any part if I understood the code correctly.
• Different embedding objects for the fixed effects module and the random effects module. It’s shared on gluon-ts.
• Forcing the standard deviation, the output of the random effects module, to be higher than 0 always (using mx.nd.clip for example). On gluon-ts they use this neat trick:

log(exp(x) + 1)

That function have an asymptote on 0, should never be 0. But it can be so close to 0 that for a computer it’s indeed 0, causing NaNs.

DeepFactors for forecasting, architecture review was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Both aims to represent the “center” of some data. A proper way to define them might be that they define the location of a distribution, defining location as the middle point of it, or the position at which most of the data is located.

What’s the mean?

First thing to say is that there are multiple kinds of means, the most common is called the arithmetic mean, but there are others called geometric mean, weighted mean, harmonic mean or even the root mean square… all trying to summarize some sense os “middle” of the data considering the whole data, in different ways.

And that’s the key difference with the median. All previous kinds of means mentioned above are calculated considering the whole data you have available:

• Sum everything and divide by it’s cardinality…
• Multiply everything…
• square everything, sum it and then apply root square…

Doing some calculation over each data point and reducing it into 1 number. For the arithmetic mean, the “standard” we typically use, the formula is as:

What’s the median?

The median give us another way to get the center of the data, and it’s literally the center. The median is calculated by ordering all numbers and finding the value at the middle. Some examples:

• The median of [0,1,2,3,4] is 2.
• The median of [100, 1000, 10000, 100000, 1000000] is 10000.
• The median of [-100, -5, -4.3, 1.2, 1.8] is -4.3.

That was simple because we have an odd number of numbers (take the one in the middle), but what if we have an even number of numbers? Then we take the two that falls on the center and calculate the arithmetic mean:

• The median of [0, 1, 2, 3] is 1.5.
• The median of [100, 1000, 10000, 100000] is 5500.
• The median of [-100, -5, -4.3, 1.2] is -4.65.

Normal distribution

A normal distribution is parameterized by the mean and the standard deviation, sometimes called location and scale. For this distribution the mean and the median falls in the same place, why? Because it’s symmetrical:

Other distributions

For other distributions we can see the mean is shifted towards the tail.

And here you can start to have an idea of why this is important. If we have extreme values the mean will move, while the median not. Classic example:

There are 4 guys on a pub drinking a pint of beer. To keep it simple say all of them have a monthly income of 1,000€. The mean is 1,000€ and the median is 1,000€.

Then Bill Gates enters the pub, say he have a monthly income of 1,000,000€ (who knows!).

Now the mean is 200,800€, while the median is still 1,000€.

In cases like this where you have big outliers you should be using the median most likely to have a sense of where is the “middle”.

When to use each

Having outliers isn’t bad per se and something to avoid at all cost. For example on my job we have done a small change recently that might increase delivery times (it’s a logistic company) in some circumstances by a bit, should I be monitoring the median or the mean? Actually you want to monitor both:

• If the mean goes up means we have extreme values more often or extreme values have gone more extreme if possible.
• If the median goes up means the whole data pattern has changed, everything seems to be affected.

Monitoring the mean is useful to have a sense of how is the data distributed.

Monitoring the median is useful to have a sense of where is the exact midpoint of your data, if it changes something is changing your whole dataset rather than just affecting a subset of it. At the same time, if you don’t want to consider outliers typically the median is the better option.

The difference between the mean and the median. was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Very likely if you are reading this you already have some idea about this topic: it’s a way to use backpropagation when using random sampling.

That’s the thing, you can’t backpropagate when you pick up random numbers. Randomness on a computer usually means reading the voltage of some component, using the internal clock… some interaction with the world. There is no way to calculate the derivative of that.

This happens if we use variational auto-encoders for example, and maybe with bayesian networks? Things like that. This happens when:

• We obtain a probability distribution. Another non typical example: noise layers for DQN.
• We sample from the distribution.
• We use the sampled value as the input for another layer.

Then at some point we want to backpropagate, how do we propagate through a random number? ¯\_(ツ)_/¯

De-Standarization of the distribution

Say we have a normal distribution as N(10, 5), that means mean 10, standard deviation 5 (I prefer to read as location 10 and scale 5). It’s the same as doing:

N(0, 1) *5 + 10

Don’t you believe me? Here you have an example with numpy:

We just de-standarized the data. Instead of making it 0 mean 1 std, we did the opposite, we moved and scaled somewhere else. Do you know who is very good at discovering where is that somewhere else? Neural networks, among others.

So trick is as follows (for a normal distribution):

• Obtain new mean.
• Obtain new std.
• Apply those to a batch of random numbers from a N(0, 1) distribution.

The network will decide how shift and scale the distribution on it’s own. Then you can propagate the gradient of the number acting as the mean and the number acting as the std. Of course this have several implications that I’m not smart enough to even try to explain it, so I’m not going to do it.

Code example

An example might look like this:

As you can see from a linear layer we obtain the scale and location and apply those a batch of random samples from a N(0, 1) distribution.

Working example

You should try without BatchNorm as I think it might be able to predict sin function on it’s own because of its internal affine transformation matrix.

NOTE: I had to scale the input to make it 0–1, I couldn’t make it work without it.

Does this work for other distributions?

In theory yes according to this stack overflow answer although doesn’t look straightfoward at all.

Variational autoencoders paper: https://arxiv.org/abs/1312.6114

The reparameterization trick with code example was originally published in Deep Learning made easy on Medium, where people are continuing the conversation by highlighting and responding to this story.