Workflows and processes are a critical need for every machine learning and AI project. When done well, they can enable you to operate more efficiently and ensure that your hard work can successfully make it into production on time. But if you’re a data scientist, although workflows and processes are helpful, they’re not data science.

That raises the question — what steps can you take to make workflows and processes as easy as possible for you to spend less time on logistics, and more time on data science?

That’s the question we set out to answer as we worked on the next iteration of Comet’s Workspaces and Projects. Some of the features were straightforward — for example, making it easy to create and share interactive reports with your colleagues and executives. But others — like ML Templates — were designed to help you standardize how you approach and manage your experiments, ultimately enabling you to go from planning, to experimentation, to production as efficiently as possible while still supporting reproducibility and visibility as part the process.

With that in mind, let’s take a look at the three main features of this update:

Interactive Reports

Check out the full report: Visualizations to Interpret Convolutional Neural Networks

As you know, it’s not enough to simply do your experiment and push on with your day. At a certain point, you have to not only share the results with internal stakeholders, but may have to also report results to executives. With Comet Interactive Reports, you can create fully interactive reports that you can share with your team to provide updates on the status or results of your projects. You can also embed custom panels into each report as well, if there are custom items you need to report on.

Check out the full report: Tensorflow Model Analysis

You can share, download, or when appropriate, even publish your reports publicly to the Comet Report Library.

Machine Learning Templates

While every machine learning project is unique, that doesn’t mean the workflow or process has to be. No matter what you’re working on, there will likely be consistency in how you scope the project and what you have track. Rather than reinvent the wheel for every project, standardizing your approach to common ML requirements can help you spend less time on planning and more time on data science.

That’s where Comet ML Templates come in. The initial templates are below ( Free Comet account required):

Project Scoping Template — easily define and track the objectives, evaluation criteria, and deliverables for your machine learning project.

Baseline Classification Models — assess the performance of simple baseline models on a classification task.

Baseline Regression Models — assess the performance of simple baseline models on a regression task.

Hyperparameter Search — a pre-built template that provides the necessary panels and outline to analyze and present the results from a Hyperparameter Search.

CodeCarbon Panel & Emissions Tracking Template

CodeCarbon is a lightweight software package that seamlessly integrates into your Python codebase that estimates the amount of carbon dioxide (CO2) produced by the cloud or your personal computing resources when you execute the code. It shows developers how they can lower emissions by either optimizing code or by hosting cloud infrastructures in regions that use renewable energy sources. CodeCarbon was developed via a joint initiative between Mila, BCG GAMMA, Haverford College, and Comet ML.

As part of the initiative, we’ve developed a new interactive panel and Emissions Tracking Template ( Comet Account required) that you can use to proactively incorporate and consider the carbon emissions of your project.

If you missed the announcement earlier this week, you can check out our recent blog post.

Upcoming Live Demo

If you’d like to see all of the new features in action, be sure to sign up for our upcoming live demo on Thursday, December 10. Register here.

Originally published at https://www.comet.ml on December 7, 2020.

Introducing Comet Interactive Reports, ML Templates & More was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Comet recently hosted the online panel, “How do top AI researchers from Google, Stanford and Hugging Face approach new ML problems?” This is the second post in a series where we recap the questions, answers, and approaches that top AI teams in the world are taking to critical machine learning challenges. You can access the first post here.

Although every machine learning project is different, there are common pitfalls and challenges that machine learning teams face when building and training models, and then taking them into production. Many of these challenges can be addressed when taken into consideration upfront, such as understanding the end goal, as well as the limitations that will be faced in your production environment.

Gideon Mendels, Comet
You all have a lot of experience. You’ve seen a lot of models in production, models that didn’t make it to production. Where do you see most machine learning projects fail? I say projects and not models because we’re looking at how we bring value to the business or to the team. As a follow up, what would you tell a junior data scientist to be careful about? What is your number one tip for someone coming into the industry?

Piero Molino, Stanford & Ludwig
In terms of failures, I would say a couple the situations usually arise when you don’t know or understand what you’re optimizing for beforehand. Then when you try to deploy a model, the model is not really doing what you expect. You have to understand, “What’s the final goal?”

One example I can give — for recommender systems, if you have your model that has a higher mean reciprocal rank, or whatever metric that you care about, but then you put it in the hands of the users and find out what you’re really trying to optimize for is something like the click through rate or maybe an even more downstream metric, such as how many items did they end up buying or they end up watching.

There’s not always a one-to-one kind of relationship between the performance that you see offline and the performance that you see online. Things can look promising at the beginning, but don’t end up deployed in production.

The other aspect I want to stress — when you have a model and put it into production, in many cases, there will be a distribution shift between the training data and the real data. The more time that passes, the more this shifts. If you don’t do a good job at monitoring, improving the models, adapting them, to make sure they’re as aligned as possible, you can see degradation of performance over time.

Ambarish Jash, Google AI
Piero brings up really good points. In big systems, your model is not the only one in the system. Having big offline gains doesn’t always translate to online gains. One of the major reasons is you may not be passing any orthogonal signal to the system that you’re training. So it always makes sense to make sense what the final goal is.

Typically that final goal is not just one final object, like driving CTR or a person buys. There are auxiliary goals as well. You can’t create a model keeping these goals in mind at the same time, so you need to do A/B testing, look at the data when it comes back. You have to be willing to fail the first few times.

Looking at the distributional shift in your data, having continuous retraining pipelines is easier said than done. You have to understand how many steps you want to fine tune, how to set the learning rate, how you accommodate new and sparse objects. There’s a tone of systems work that must go on in the background to put something into production.

Victor Sanh, Hugging Face
One rookie mistake I see — if you don’t take into account production constraints from the very beginning, you can end up with an overcomplicated model that will never make it into production.

There’s stories where you have this great model with really high accuracy, but it took 24 hours to run. So you’d never take that into production. You have to understand those constraints at the beginning, or you’ll never make it to the end.

Another point is being stuck in “wishful thinking.” It’s when you look at the results and see what you want, not what they say. It can be super challenging at the beginning not to see the results for what they are. This is especially hard when you’re on deadline.

Want to watch the full panel? It’s available on-demand here.

Originally published at https://www.comet.ml on October 20, 2020.

Industry Q&A: Where do most machine learning projects fail? was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Comet recently hosted the online panel, “How do top AI researchers from Google, Stanford and Hugging Face approach new ML problems?” This post is our first in a series where we recap the questions, answers, and approaches that top AI teams in the world are taking to critical machine learning challenges.

One of the hardest parts of machine learning is simply getting started. Do you have the necessary data? Do you have the systems in place to manage your model? If you need to take it into production, do you have a good understanding of what the production environment looks like? All of these are considerations that need to be made in order to ensure your work is successful — and if the problem you’re trying to solve is worth pursuing.

Gideon Mendels, Comet
There are a number of challenges to even approaching a machine learning challenge. There are so many moving parts. For those of us in the industry, it’s very different from what you might see in something like a Kaggle competition, where you have a clean dataset and the metrics are figured out for you.

So how do you start the research process? What do you do when you have a new problem?

Ambarish Jash, Google AI
Two parts, one is the research problem and coming up with the problem definition. The other is the challenge of putting it in production. Production puts a significant amount of constraint on the research you can do. One approach is to:

• Define the problem
• Define the system you’re going to build

If you need to go into production, I would strongly recommend you about building the system first and maybe start with simpler models. In the long run, it makes things like debugging and maintenance simpler.

Keep things simple, build out pipelines. Once these are built, you can start to rapidly iterate on the model. You need a lot of data, and most of the time, the loss isn’t exactly what you care about. Having a strong evaluation framework is important too, because as you start to add complexity to your model, you will need to figure out if it’s making sense with your final task.

Piero Moilino, Stanford & Ludwig
This depends on the project, based on if it’s a theory applied project or a research project. For theory applied, I’m often the only one with access to data and there usual isn’t much historical progress on the data. While on a research problem, there may be papers about it and I can start from there.

If the project is applied, the first thing I do is try to understand the data. That’s the number one thing. Is there signal for the problem I want to solve? In many cases, a machine learning project won’t work simply because there isn’t enough signal for it in the data.

After looking at the data, I personally use tools that I build for myself. This is just because it’s easier for me to compare different models, have a standard pipeline, and then I can reuse it. Usually I train a simple model, see what’s there, look at the predictions, do some visualizations, understand the predictions and learning curves.

Once I feel I have a global understanding of the problem, the data, and an initial simple solution, then I double down on complex models or more sophisticated solutions. But something simple first, then scale up, is really good advice.

I believe that machine learning projects are much closer to research projects than software projects. For software, usually you define your constraints and implement, and you know it will work ahead of time. That’s not the case for machine learning.

In machine learning, you don’t know if the problem will work to begin with. By starting simple, and understanding if you have signal, you can get an idea if you have what you need to solve the problem. Otherwise you can spend a lot of time and end up with a model that doesn’t work. Fail fast and try to figure it out early if you can solve the problem.

Victor Sanh, Hugging Face
My approach is to “really lose.” We have a spreadsheet with a lot of ideas that come along, and we take the ones that excite us the most.

But I agree with Piero and Ambarish — you take the problems, then you want to start fast and iterate fast at the beginning. You want to understand the data and learning processes, so you can decide “is it worth it to pursue this problem for a few weeks?”

The first two weeks are decisive, because that’s when you get a sense of your data and understand if there’s actually improvement that you can do there.

Want to watch the full panel? It’s available on-demand here.

Originally published at https://www.comet.ml on October 13, 2020.

Industry Q&A: How do you start the machine learning research process? was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Of course, the internals of a pasta machine don’t change over time, and it doesn’t produce better “output” every time you cook (given appropriate “loss” ratings from your dinner guests). That would be cool! So, the metaphor is not complete, but I do like the idea of Pasta Machine Learning.

The pasta machine shows that the idea of a model can be varied between chefs, er, I mean data scientists.

Thanks to scikit learn, models in Python often have a similar interface to one another (for better or worse). For example, models often have methods like:

model.fit()
model.train()
model.test()

Regardless of what kind of model you are using in your Machine Learning code, Comet ML can handle it. Comet has a sophisticated system for logging, registering, versioning, and deploying machine learning models.

The steps of the Comet model pipeline are:

1. Log an Experiment Model, via Comet’s Python SDK Experiment
2. Register an Experiment Model
3. Track Model Versions of the Registered Models
4. Deploy a Registered Model

We’ll go through each of these steps.

1. Log an Experiment Model

To Comet, a model can be composed of any files or folders in the file system. That is, a model can be any collection of files.

The first step in the model pipeline is to log all of the associated files with a model name via an Experiment.

If all of the files connected to a model are in a single folder (and all of the files there are related to the model), you can simply log the model with a name and the path to the folder. Here, we use the name “MNIST CNN” and all of the model files are in the “../models/run-026/” folder:

from comet_ml import Experiment
experiment = Experiment()
experiment.log_model("MNIST CNN", "../models/run-026/")

After you log a model via an Experiment, you will see the model and all of its individual files in the Experiment’s Assets Tab.

From the Experiment’s Assets tab, you can download the model, see if it has been registered before, or register it as a new model, or a new version of an existing model.

Clicking the + Register link will take you to the Registry view:

2. Register a Model

After you log a model through an Experiment through the SDK, you can then register it. Registered Models belong to a workspace, and are shared with your team there.

You can register an Experiment’s model in two ways:

1. Through the Experiment’s Asset Tab in the Comet.ml User Interface
2. Programmatically through the Comet Python SDK

To register an experiment’s model via the Comet User Interface, simply go the Experiment’s Asset tab, locate the model in the asset list, and click on the Register link.

If you have previously registered a model, you can save this model to an Existing version:

However, if you would like to save this as a new model, simply click the “Register New Model” link. That will bring up the Register Model window:

Here, you can give the registered model a name, a version, and set its visibility to Public or Private.

You can also register an experiment’s model via the Python API. That would look similar to:

from comet_ml import API
api = API()
experiment = api.get("workspace/project/experiment")
experiment.register_model("MNIST CNN")

3. Track Model Versions

Registered models belong to the workspace. In this manner, you can share your models with the workspace team. At the workspace view, you can select between viewing the Projects or the Registered Models:

Selecting a registered model’s card in the Model Registry will take you to the detailed view of the registered model:

Here you can see the latest version of the registered model, and all previous versions as well. Registered models contain the following properties, which can all be edited here:

• Registered name
• Description
• Visibility setting (public vs. private)
• Notes

4. Deploy Models

To download a registered model as a zip file, click on the “Get Model” button. The zip file will contain all of the files that were logged with the Experiment model via the Python SDK.

You can download the latest versioned registered model, or any of the versions. Simply click on their download icon.

That’s it! I hope you enjoyed our quick snack overview of Comet’s Model Registry!

For more information, please see: comet.ml/docs/user-interface/models/

Bon appétit!

Originally published at https://www.comet.ml on September 9, 2020.

Using Comet’s Model Registry was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the last three years since Comet was founded, our users and customers trained millions of models on anything from self-driving cars to speech recognition, and from Covid-19 protein prediction to satellite image object detection. Based on your requests we added more and more visualization capabilities including chart builders, confusion matrices, audio players and many more. But as the complexity of the use cases, requirements and customization grew, we realized that we needed a more scalable way to grow Comet’s functionality and provide users the ability to build anything they want!

Today, we are excited to announce The key principles in designing Comet Panels were the following: they want without the limitations of a GUI or a specific dependency Comet Panels, which has been under development and testing for the past year. Starting today, you can tap into the ecosystems of JS/HTML/CSS and create new widgets with custom styles and behaviors, or add new visualizations and charting types. Unlimited possibilities!

1. Dynamic — Panels should be dynamic and update on new experiments and results when they arrive. There’s nothing worse than a stale and misleading visualization.
2. Flexible — Users should be able to build and customize
3. Reusable — teammates and community members should be able to share and reuse each other’s panels.

The Comet Panel Gallery

The heart of Panels lies in the Comet Panel Gallery. Check it out to see what others have built and shared with the community.

Comet experiments already store your hyper-parameters, metrics, model definition, trained models, dataset samples and much more. Panels build on top of that data to visualize and process anything you’d like.

You can create a new panel by going to the Panels Gallery and clicking on “Create New” on the top right corner. You should see a similar code editor:

The code editor is divided into three main panes. On the left we have a full code editor with code completion, syntax highlighting etc. On the right we have a live preview of our panel. On the bottom right we see our Console pane which shows errors and exceptions. You’ll also notice the tabs on the top that allow you to switch from code to description, HTML, CSS, resources and query.

To learn how to customize your machine learning visualizations, read the rest of the article on Comet.ml.

Originally published at https://www.comet.ml on August 5, 2020.

Introducing Panels: Custom Visualizations for Machine Learning was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deep Learning has changed the game in speech recognition with the introduction of end-to-end models. These models take in audio, and directly output transcriptions. Two of the most popular end-to-end models today are Deep Speech by Baidu, and Listen Attend Spell (LAS) by Google. Both Deep Speech and LAS, are recurrent neural network (RNN) based architectures with different approaches to modeling speech recognition. Deep Speech uses the Connectionist Temporal Classification (CTC) loss function to predict the speech transcript. LAS uses a sequence to sequence network architecture for its predictions.

These models simplified speech recognition pipelines by taking advantage of the capacity of deep learning system to learn from large datasets. With enough data, you should, in theory, be able to build a super robust speech recognition model that can account for all the nuance in speech without having to spend a ton of time and effort hand engineering acoustic features or dealing with complex pipelines in more old-school GMM-HMM model architectures, for example.

Deep learning is a fast-moving field, and Deep Speech and LAS style architectures are already quickly becoming outdated. You can read about where the industry is moving in the Latest Advancement Section below.

How to Build Your Own End-to-End Speech Recognition Model in PyTorch

Let’s walk through how one would build their own end-to-end speech recognition model in PyTorch. The model we’ll build is inspired by Deep Speech 2 (Baidu’s second revision of their now-famous model) with some personal improvements to the architecture. The output of the model will be a probability matrix of characters, and we’ll use that probability matrix to decode the most likely characters spoken from the audio. You can find the full code and also run the it with GPU support on Google Colaboratory.

Preparing the data pipeline

Data is one of the most important aspects of speech recognition. We’ll take raw audio waves and transform them into Mel Spectrograms.

You can read more on the details about how that transformation looks from this excellent post here. For this post, you can just think of a Mel Spectrogram as essentially a picture of sound.

For handling the audio data, we are going to use an extremely useful utility called torchaudio which is a library built by the PyTorch team specifically for audio data. We’ll be training on a subset of LibriSpeech, which is a corpus of read English speech data derived from audiobooks, comprising 100 hours of transcribed audio data. You can easily download this dataset using torchaudio:

import torchaudio train_dataset = torchaudio.datasets.LIBRISPEECH("./", url="train-clean-100", download=True) 
test_dataset = torchaudio.datasets.LIBRISPEECH("./", url="test-clean", download=True)

Each sample of the dataset contains the waveform, sample rate of audio, the utterance/label, and more metadata on the sample. You can view what each sample looks like from the source code here.

Data Augmentation — SpecAugment

Data augmentation is a technique used to artificially increase the diversity of your dataset in order to increase your dataset size. This strategy is especially helpful when data is scarce or if your model is overfitting. For speech recognition, you can do the standard augmentation techniques, like changing the pitch, speed, injecting noise, and adding reverb to your audio data.

We found Spectrogram Augmentation (SpecAugment), to be a much simpler and more effective approach. SpecAugment, was first introduced in the paper SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition, in which the authors found that simply cutting out random blocks of consecutive time and frequency dimensions improved the models generalization abilities significantly!

In PyTorch, you can use the torchaudio function FrequencyMasking to mask out the frequency dimension, and TimeMasking for the time dimension.

torchaudio.transforms.FrequencyMasking()
torchaudio.transforms.TimeMasking()

Now that we have the data, we’ll need to transform the audio into Mel Spectrograms, and map the character labels for each audio sample into integer labels:

class TextTransform:
    """Maps characters to integers and vice versa"""
    def __init__(self):
        char_map_str = """
        ' 0
        <SPACE> 1
        a 2
        b 3
        c 4
        d 5
        e 6
        f 7
        g 8
        h 9
        i 10
        j 11
        k 12
        l 13
        m 14
        n 15
        o 16
        p 17
        q 18
        r 19
        s 20
        t 21
        u 22
        v 23
        w 24
        x 25
        y 26
        z 27
        """
        self.char_map = {}
        self.index_map = {}
        for line in char_map_str.strip().split('\n'):
            ch, index = line.split()
            self.char_map[ch] = int(index)
            self.index_map[int(index)] = ch
        self.index_map[1] = ' '

    def text_to_int(self, text):
        """ Use a character map and convert text to an integer sequence """
        int_sequence = []
        for c in text:
            if c == ' ':
                ch = self.char_map['']
            else:
                ch = self.char_map[c]
            int_sequence.append(ch)
        return int_sequence

    def int_to_text(self, labels):
        """ Use a character map and convert integer labels to an text sequence """
        string = []
        for i in labels:
            string.append(self.index_map[i])
        return ''.join(string).replace('', ' ')


train_audio_transforms = nn.Sequential(
    torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128),
    torchaudio.transforms.FrequencyMasking(freq_mask_param=15),
    torchaudio.transforms.TimeMasking(time_mask_param=35)
)

valid_audio_transforms = torchaudio.transforms.MelSpectrogram()

text_transform = TextTransform()


def data_processing(data, data_type="train"):
    spectrograms = []
    labels = []
    input_lengths = []
    label_lengths = []
    for (waveform, _, utterance, _, _, _) in data:
        if data_type == 'train':
            spec = train_audio_transforms(waveform).squeeze(0).transpose(0, 1)
        else:
            spec = valid_audio_transforms(waveform).squeeze(0).transpose(0, 1)
        spectrograms.append(spec)
        label = torch.Tensor(text_transform.text_to_int(utterance.lower()))
        labels.append(label)
        input_lengths.append(spec.shape[0]//2)
        label_lengths.append(len(label))

    spectrograms = nn.utils.rnn.pad_sequence(spectrograms, batch_first=True).unsqueeze(1).transpose(2, 3)
    labels = nn.utils.rnn.pad_sequence(labels, batch_first=True)

    return spectrograms, labels, input_lengths, label_lengths

Define the Model — Deep Speech 2 (but better)

Our model will be similar to the Deep Speech 2 architecture. The model will have two main neural network modules — N layers of Residual Convolutional Neural Networks (ResCNN) to learn the relevant audio features, and a set of Bidirectional Recurrent Neural Networks (BiRNN) to leverage the learned ResCNN audio features. The model is topped off with a fully connected layer used to classify characters per time step.

Convolutional Neural Networks (CNN) are great at extracting abstract features, and we’ll apply the same feature extraction power to audio spectrograms. Instead of just vanilla CNN layers, we choose to use Residual CNN layers. Residual connections (AKA skip connections) were first introduced in the paper Deep Residual Learning for Image Recognition, where the author found that you can build really deep networks with good accuracy gains if you add these connections to your CNN’s. Adding these Residual connections also helps the model learn faster and generalize better. The paper Visualizing the Loss Landscape of Neural Nets shows that networks with residual connections have a “flatter” loss surface, making it easier for models to navigate the loss landscape and find a lower and more generalizable minima.

Recurrent Neural Networks (RNN) are naturally great at sequence modeling problems. RNN’s processes the audio features step by step, making a prediction for each frame while using context from previous frames. We use BiRNN’s because we want the context of not only the frame before each step, but the frames after it as well. This can help the model make better predictions, as each frame in the audio will have more information before making a prediction. We use Gated Recurrent Unit (GRU’s) variant of RNN’s as it needs less computational resources than LSTM’s, and works just as well in some cases.

The model outputs a probability matrix for characters which we’ll use to feed into our decoder to extract what the model believes are the highest probability characters that were spoken.

class CNNLayerNorm(nn.Module):
    """Layer normalization built for cnns input"""
    def __init__(self, n_feats):
        super(CNNLayerNorm, self).__init__()
        self.layer_norm = nn.LayerNorm(n_feats)

    def forward(self, x):
        # x (batch, channel, feature, time)
        x = x.transpose(2, 3).contiguous() # (batch, channel, time, feature)
        x = self.layer_norm(x)
        return x.transpose(2, 3).contiguous() # (batch, channel, feature, time) 


class ResidualCNN(nn.Module):
    """Residual CNN inspired by https://arxiv.org/pdf/1603.05027.pdf
        except with layer norm instead of batch norm
    """
    def __init__(self, in_channels, out_channels, kernel, stride, dropout, n_feats):
        super(ResidualCNN, self).__init__()

        self.cnn1 = nn.Conv2d(in_channels, out_channels, kernel, stride, padding=kernel//2)
        self.cnn2 = nn.Conv2d(out_channels, out_channels, kernel, stride, padding=kernel//2)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)
        self.layer_norm1 = CNNLayerNorm(n_feats)
        self.layer_norm2 = CNNLayerNorm(n_feats)

    def forward(self, x):
        residual = x  # (batch, channel, feature, time)
        x = self.layer_norm1(x)
        x = F.gelu(x)
        x = self.dropout1(x)
        x = self.cnn1(x)
        x = self.layer_norm2(x)
        x = F.gelu(x)
        x = self.dropout2(x)
        x = self.cnn2(x)
        x += residual
        return x # (batch, channel, feature, time)


class BidirectionalGRU(nn.Module):

    def __init__(self, rnn_dim, hidden_size, dropout, batch_first):
        super(BidirectionalGRU, self).__init__()

        self.BiGRU = nn.GRU(
            input_size=rnn_dim, hidden_size=hidden_size,
            num_layers=1, batch_first=batch_first, bidirectional=True)
        self.layer_norm = nn.LayerNorm(rnn_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        x = self.layer_norm(x)
        x = F.gelu(x)
        x, _ = self.BiGRU(x)
        x = self.dropout(x)
        return x


class SpeechRecognitionModel(nn.Module):
    """Speech Recognition Model Inspired by DeepSpeech 2"""

    def __init__(self, n_cnn_layers, n_rnn_layers, rnn_dim, n_class, n_feats, stride=2, dropout=0.1):
        super(SpeechRecognitionModel, self).__init__()
        n_feats = n_feats//2
        self.cnn = nn.Conv2d(1, 32, 3, stride=stride, padding=3//2)  # cnn for extracting heirachal features

        # n residual cnn layers with filter size of 32
        self.rescnn_layers = nn.Sequential(*[
            ResidualCNN(32, 32, kernel=3, stride=1, dropout=dropout, n_feats=n_feats) 
            for _ in range(n_cnn_layers)
        ])
        self.fully_connected = nn.Linear(n_feats*32, rnn_dim)
        self.birnn_layers = nn.Sequential(*[
            BidirectionalGRU(rnn_dim=rnn_dim if i==0 else rnn_dim*2,
                             hidden_size=rnn_dim, dropout=dropout, batch_first=i==0)
            for i in range(n_rnn_layers)
        ])
        self.classifier = nn.Sequential(
            nn.Linear(rnn_dim*2, rnn_dim),  # birnn returns rnn_dim*2
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(rnn_dim, n_class)
        )

    def forward(self, x):
        x = self.cnn(x)
        x = self.rescnn_layers(x)
        sizes = x.size()
        x = x.view(sizes[0], sizes[1] * sizes[2], sizes[3])  # (batch, feature, time)
        x = x.transpose(1, 2) # (batch, time, feature)
        x = self.fully_connected(x)
        x = self.birnn_layers(x)
        x = self.classifier(x)
        return x

Picking the Right Optimizer and Scheduler — AdamW with Super Convergence

The optimizer and learning rate schedule plays a very important role in getting our model to converge to the best point. Picking the right optimizer and scheduler can also save you compute time, and help your model generalize better to real-world use cases. For our model, we’ll be using AdamW with the One Cycle Learning Rate Scheduler. Adam is a widely used optimizer that helps your model converge more quickly, therefore, saving compute time, but has been notorious for not generalizing as well as Stochastic Gradient Descent AKA SGD.

AdamW was first introduced in Decoupled Weight Decay Regularization, and is considered a “fix” to Adam. The paper pointed out that the original Adam algorithm has a wrong implementation of weight decay, which AdamW attempts to fix. This fix helps with Adam ‘s generalization problem.

The One Cycle Learning Rate Scheduler was first introduced in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates. This paper shows that you can train neural networks an order of magnitude faster, while keeping their generalizable abilities, using a simple trick. You start with a low learning rate, which warms up to a large maximum learning rate, then decays linearly to the same point of where you originally started.

Because the maximum learning rate is magnitudes higher than the lowest, you also gain some regularization benefits which helps your model generalize better if you have a smaller set of data.

With PyTorch, these two methods are already part of the package.

optimizer = optim.AdamW(model.parameters(), hparams['learning_rate'])
scheduler = optim.lr_scheduler.OneCycleLR(optimizer,
	max_lr=hparams['learning_rate'],
	steps_per_epoch=int(len(train_loader)),
	epochs=hparams['epochs'],
	anneal_strategy='linear')

The CTC Loss Function — Aligning Audio to Transcript

Our model will be trained to predict the probability distribution of all characters in the alphabet for each frame (ie, timestep) in the spectrogram we feed into the model.

Traditional speech recognition models would require you to align the transcript text to the audio before training, and the model would be trained to predict specific labels at specific frames.

The innovation of the CTC loss function is that it allows us to skip this step. Our model will learn to align the transcript itself during training. The key to this is the “blank” label introduced by CTC, which gives the model the ability to say that a certain audio frame did not produce a character. You can see a more detailed explanation of CTC and how it works from this excellent post.

The CTC loss function is also built into PyTorch.

criterion = nn.CTCLoss(blank=28).to(device)

Evaluating Your Speech Model

When Evaluating your speech recognition model, the industry standard is using the Word Error Rate (WER) as the metric. The Word Error Rate does exactly what it says — it takes the transcription your model outputs, and the true transcription, and measures the error between them. You can see how that’s implemented here. Another useful metric is called the Character Error Rate (CER). The CER measures the error of the characters between the model’s output and the true labels. These metrics are helpful to measure how well your model performs.

For this tutorial, we’ll use a “greedy” decoding method to process our model’s output into characters that can be combined to create the transcript. A “greedy” decoder takes in the model output, which is a softmax probability matrix of characters, and for each time step (spectrogram frame), it chooses the label with the highest probability. If the label is a blank label, we remove it from the final transcript.

def GreedyDecoder(output, labels, label_lengths, blank_label=28, collapse_repeated=True):
    arg_maxes = torch.argmax(output, dim=2)
    decodes = []
    targets = []
    for i, args in enumerate(arg_maxes):
        decode = []
        targets.append(text_transform.int_to_text(labels[i][:label_lengths[i]].tolist()))
        for j, index in enumerate(args):
            if index != blank_label:
                if collapse_repeated and j != 0 and index == args[j -1]:
                    continue
                decode.append(index.item())
        decodes.append(text_transform.int_to_text(decode))
    return decodes, targets

Training and Monitoring Your Experiments Using Comet.ml

Comet.ml provides a platform that allows deep learning researchers to track, compare, explain, and optimize their experiments and models. Comet.ml has improved our productivity at AssemblyAI and we highly recommend using this platform for teams doing any sort of data science experiments. Comet.ml is super easy to set up. And works with just a few lines of code.

# initialize experiment object
experiment = Experiment(api_key=comet_api_key, project_name=project_name)
experiment.set_name(exp_name)

# track metrics
experiment.log_metric('loss', loss.item())

Comet.ml provides you with a very productive dashboard where you can view and track your model’s progress.

You can use Comet to track metrics, code, hyper parameters, your model’s graphs, among many other things! A really handy feature that Comet provides is the ability to compare your experiment among many other experiments.

Comet has a rich feature set that we won’t cover all here, but we highly recommended using it for a productivity and sanity boost.

Here is the rest of our training script.

class IterMeter(object):
    """keeps track of total iterations"""
    def __init__(self):
        self.val = 0

    def step(self):
        self.val += 1

    def get(self):
        return self.val


def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):
    model.train()
    data_len = len(train_loader.dataset)
    with experiment.train():
        for batch_idx, _data in enumerate(train_loader):
            spectrograms, labels, input_lengths, label_lengths = _data 
            spectrograms, labels = spectrograms.to(device), labels.to(device)

            optimizer.zero_grad()

            output = model(spectrograms)  # (batch, time, n_class)
            output = F.log_softmax(output, dim=2)
            output = output.transpose(0, 1) # (time, batch, n_class)

            loss = criterion(output, labels, input_lengths, label_lengths)
            loss.backward()

            experiment.log_metric('loss', loss.item(), step=iter_meter.get())
            experiment.log_metric('learning_rate', scheduler.get_lr(), step=iter_meter.get())

            optimizer.step()
            scheduler.step()
            iter_meter.step()
            if batch_idx % 100 == 0 or batch_idx == data_len:
                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                    epoch, batch_idx * len(spectrograms), data_len,
                    100. * batch_idx / len(train_loader), loss.item()))


def test(model, device, test_loader, criterion, epoch, iter_meter, experiment):
    print('\nevaluating…')
    model.eval()
    test_loss = 0
    test_cer, test_wer = [], []
    with experiment.test():
        with torch.no_grad():
            for I, _data in enumerate(test_loader):
                spectrograms, labels, input_lengths, label_lengths = _data 
                spectrograms, labels = spectrograms.to(device), labels.to(device)

                output = model(spectrograms)  # (batch, time, n_class)
                output = F.log_softmax(output, dim=2)
                output = output.transpose(0, 1) # (time, batch, n_class)

                loss = criterion(output, labels, input_lengths, label_lengths)
                test_loss += loss.item() / len(test_loader)

                decoded_preds, decoded_targets = GreedyDecoder(output.transpose(0, 1), labels, label_lengths)
                for j in range(len(decoded_preds)):
                    test_cer.append(cer(decoded_targets[j], decoded_preds[j]))
                    test_wer.append(wer(decoded_targets[j], decoded_preds[j]))


    avg_cer = sum(test_cer)/len(test_cer)
    avg_wer = sum(test_wer)/len(test_wer)
    experiment.log_metric('test_loss', test_loss, step=iter_meter.get())
    experiment.log_metric('cer', avg_cer, step=iter_meter.get())
    experiment.log_metric('wer', avg_wer, step=iter_meter.get())

    print('Test set: Average loss: {:.4f}, Average CER: {:4f} Average WER: {:.4f}\n'.format(test_loss, avg_cer, avg_wer))


def main(learning_rate=5e-4, batch_size=20, epochs=10,
        train_url="train-clean-100", test_url="test-clean",
        experiment=Experiment(api_key='dummy_key', disabled=True)):

    hparams = {
        "n_cnn_layers": 3,
        "n_rnn_layers": 5,
        "rnn_dim": 512,
        "n_class": 29,
        "n_feats": 128,
        "stride": 2,
        "dropout": 0.1,
        "learning_rate": learning_rate,
        "batch_size": batch_size,
        "epochs": epochs
    }

    experiment.log_parameters(hparams)

    use_cuda = torch.cuda.is_available()
    torch.manual_seed(7)
    device = torch.device("cuda" if use_cuda else "cpu")

    if not os.path.isdir("./data"):
        os.makedirs("./data")

    train_dataset = torchaudio.datasets.LIBRISPEECH("./data", url=train_url, download=True)
    test_dataset = torchaudio.datasets.LIBRISPEECH("./data", url=test_url, download=True)

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    train_loader = data.DataLoader(dataset=train_dataset,
                                batch_size=hparams['batch_size'],
                                shuffle=True,
                                collate_fn=lambda x: data_processing(x, 'train'),
                                **kwargs)
    test_loader = data.DataLoader(dataset=test_dataset,
                                batch_size=hparams['batch_size'],
                                shuffle=False,
                                collate_fn=lambda x: data_processing(x, 'valid'),
                                **kwargs)

    model = SpeechRecognitionModel(
        hparams['n_cnn_layers'], hparams['n_rnn_layers'], hparams['rnn_dim'],
        hparams['n_class'], hparams['n_feats'], hparams['stride'], hparams['dropout']
        ).to(device)

    print(model)
    print('Num Model Parameters', sum([param.nelement() for param in model.parameters()]))

    optimizer = optim.AdamW(model.parameters(), hparams['learning_rate'])
    criterion = nn.CTCLoss(blank=28).to(device)
    scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=hparams['learning_rate'], 
                                            steps_per_epoch=int(len(train_loader)),
                                            epochs=hparams['epochs'],
                                            anneal_strategy='linear')

    iter_meter = IterMeter()
    for epoch in range(1, epochs + 1):
        train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
        test(model, device, test_loader, criterion, epoch, iter_meter, experiment)

The train function trains the model on a full epoch of data. The test function evaluates the model on test data after every epoch. It gets the test_loss as well as the cer and wer of the model. You can start running the training script right now with GPU support in the Google Colaboratory.

How to Improve Accuracy

Speech Recognition Requires a ton of data and a ton of compute resources. The example laid out is trained on a subset of LibriSpeech (100 hours of audio) and a single GPU. To get state of the art results you’ll need to do distributed training on thousands of hours of data, on tens of GPU’s spread out across many machines.

Another way to get a big accuracy improvement is to decode the CTC probability matrix using a Language Model and the CTC beam search algorithm. CTC type models are very dependent on this decoding process to get good results. Luckily there is a handy open source library that allows you to do that.

This tutorial was made to be more accessible so it’s a relatively small model (23 million Parameters) compared to something like BERT (340 million Parameters). It seems to be the larger you can get your network, the better it performs, although there are diminishing returns. A larger model equating to better performance is not always the case though, as proven by OpenAI’s research Deep Double Descent.

This model has 3 residual CNN layers and 5 Bidirectional GRU layers which should allow you to train a reasonable batch size on a single GPU with at least 11GB of memory. You can tweak some of the hyper parameters in the main function to reduce or increase the model size for your use case and compute availability.

Latest Advancements In Speech Recognition with Deep Learning

Deep learning is a fast-moving field. It seems like you can’t go a week without some new technique getting state of the art results. Here are a few of things worth exploring int the world of speech recognition.

Transformers

Transformers have taken the Natural Language Processing world by storm! First Introduced in the paper Attention Is All You Need, transformers have been taking and modified to beat pretty much all existing NLP task dethroning RNN’s type architectures. The Transformer’s ability to see the full context of sequence data is transferable to speech as well.

Unsupervised Pre-training

If you follow deep learning closely you’ve probably heard of BERT, GPT, and GPT2. These Transformer models have first pertained on a language modeling task with unlabeled text data, and fine-tuned on a wide array of NLP task and get state of the art results! During pre-training, the model learns something fundamental on the statistics of language and uses that power to excel at other tasks. We believe this technique has great promises on speech data as well.

Word Piece Models

Our model defined above output characters. Some benefits to that are the model doesn’t have to worry about out of vocabulary words when running inference on speech. So for the word c h a t each character has is its own label. The downside to using characters are inefficiency and the model being prone to more errors because you’re predicting one character at a time. Using the whole word as labels have been explored, to some degree of success. Using this method, the entire word chat would be the label. But using whole words, you would have to keep an index of all possible vocabularies to make a prediction, which is memory inefficient with the possibility of running into out of vocabulary words during prediction. The sweet spot would be using word piece or sub-word units as labels. Instead of characters for the individual label, you can chop up the words into sub-word units, and use those as labels, i.e. ch at. This solves the out of vocabulary issue, and is much more efficient, as it needs fewer steps to decode then using characters, and without the need to have an index of all possible words. Word pieces have been used successfully with many NLP models, like BERT and would work natural with speech recognition problems as well.

Originally published at https://www.comet.ml on May 18, 2020.

Customer Case Study: Building an end-to-end Speech Recognition model in PyTorch with AssemblyAI was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author: Dhruv Nair, Data Scientist, Comet.ml

Introduction

Predictive Early Stopping is a state-of-the-art approach for speeding up model training and hyperparameter optimization. Our benchmarking studies have shown that Predictive Early Stopping can speed up model training by up to 30% independent of the underlying infrastructure.

We build on insights gathered from projects such as Learning Curve Extrapolation, Hyperband, and Median Stopping, in order to create a predictive model that can estimate the convergence value of a loss curve.

Comet is able to leverage model data, such as hyperparameters and loss curves, from over two million models in the public section of its platform to create a model whose predictions generalize across hyperparameters and model architectures.

In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs. In addition to predicting the convergence value, our Predictive Early Stopping product provides an estimate of the probability that the current model will outperform the best model result seen in the current training sweep.

In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs.

These predictions allow us to terminate the training of underperforming models, so that the search process is spent evaluating only the most promising candidates.

Benchmarking:

We tested our Predictive Early Stopping method in three different settings:

1. A hyperparameter search that optimizes the parameters of a function that acts as a surrogate for a neural network
2. A hyperparameter search to optimize a 6 layer CNN on CIFAR10 using the SMAC optimizer, with and without predictive early stopping.
3. A hyperparameter search to optimize the same 6 layer CNN using Random Search with Hyperband vs Random Search with Predictive Early Stopping.

Results for the Surrogate Function:

In our first test, we setup an exponentially decaying function as our surrogate for a Neural Network. We ran this surrogate model for 20 steps and determined the optimal values for the parameters of this function using Comet’s Bayesian Optimizer and Predictive Early Stopping.During the hyperparameter search we observed that suboptimal models were not allowed to train the full 20 steps. Figure 1, illustrates our stopping mechanism.

Results for the CNN model with SMAC:

Our benchmarking test for the CNN model was setup in the following way:

We used the SMAC optimizer to estimate the following hyperparameters in a 6-Layer CNN model. The model hyperparameters, and architecture were based on AlexNet.

{
    "learning_rate":{
        "type":"loguniform",
        "value":[0.0000001, 0.01]
    },
    "learning_rate_decay":{
        "type": "uniform",
        "value":[0.000001, 0.001]
    },
    "weight_decay": {
        "type": "loguniform",
        "value": [0.0000005, 0.005]
    }
}

We ran 8 trials of the optimizer, with and without Predictive Early Stopping. Each optimizer trial is given a 6 hour budget to evaluate as many configurations as possible. Each hyperparameter configuration was allowed to train for a maximum of 100 epochs, and the validation set was evaluated at the end of every epoch. The validation loss was used by our Predictive Early Stopping model to determine whether or not to terminate a hyperparameter configuration.

At the end of all the trials, we determined the mean value of the test loss across all trials as a function of the total number of epochs in the hyperparameter sweep that are required to achieve that loss value.

We can see in Figure 2, that using Predictive Early Stopping allows SMAC to get to a comparable loss value, almost 300 epochs faster. This is a 25% decrease in the amount of time spent doing hyperparameter optimization.

We also divided the hyperparameter configurations into quantiles, based on the final validation loss. We then calculated the average amount of epochs that the Optimizer spent in each quantile across all trials.

In Figure 3, we see that both SMAC and Predictive-Early-Stopping-SMAC spend roughly the same amount of time evaluating the top 25% and top 50% of configurations. However, Predictive Early Stopping spends 30 fewer epochs training models in the bottom 50% of results, and 20 fewer epochs in bottom 25% .

In Figure 4, we see sample loss curves from a hyperparameter sweep with Predictive Early Stopping. Suboptimal configurations are stopped well before the total number of allowed training steps.

Results for the CNN model with Hyperband:

We setup the test for Hyperband in a similar manner to SMAC. We specifically used the Asynchronous Successive Halving Pruner implemented in Optuna. We can think of this as Hyperband with a single bracket. We selected 120 hyperparameter configurations at random. Each configuration was allotted a minimum resource of 10 epochs, and allowed to train for a maximum of 100 epochs. This leads to a maximum training budget of 12000 epochs.

At every evaluation point, the number of configurations are reduced by a factor of N based on the worst performing validation losses . In our experiments we evaluated Hyperband with N values of 2, 4 and 8.

For the Predictive Early Stopping, we tested each configuration with different values of an interval parameter. Configurations were evaluated every 10, 15, and 20 epochs. At every evaluation we estimated the probability that the current configuration will be better than the best configuration seen so far. If this probability is less than a threshold, in our case 90%, we terminate the current configuration. Threshold, and Interval are both configurable hyperparameters for Predictive Early Stopping.

We then determine the best validation loss achieved in the hyperparameter sweep, and the number of epochs remaining in the budget after the sweep.

In Figure 5a, we see that all approaches find the same best value for the validation loss, however, Predictive Early Stopping is able to evaluate the configurations using just 15% of the total budget, compared to Hyperband which uses 25%. This is a 10% improvement in speed.

Conclusion

Predictive Early Stopping has very obvious time, energy, and cost saving benefits. Wasted compute cycles aren’t good for the environment, or a researcher’s budget.

The Allen Institute for AI recently published a report on the rising computational costs of training machine learning models, and how these increasingly large energy requirements adversely affect the environment. The paper states that current advancements in state of the art AI research have largely focussed on metrics such as accuracy, or error, at the cost of being environmentally unfriendly. They call this paradigm, Red AI. To counter the prevalence of Red AI research, they propose a shift towards AI research that places an emphasis on computational efficiency: Green AI.

The paper recommends tracking the efficiency of an AI algorithm based on the total number of floating point operations (FPO’s) required to generate a result. The total number of FPO’s directly correlates with the number of hyperparameter configurations that are evaluated during tuning, as well as the number of training iterations spent on each configuration.

We hope our efforts with Predictive Early Stopping contribute towards increasing the computational efficiency of the hyperparameter search process. We see the this tool as a way to lower some of the monetary barriers associated with AI research, and as a step towards adopting Green AI practices.

In our next post we will describe applying predictive early stopping in individual runs that do not belong to a larger parameter search.

*Predictive Early Stopping is available as an add-on with the purchase of Comet Teams or Comet Enterprise. Patent protection is being sought on Predictive Early Stopping via one or more pending patent applications.Learn more and sign-up here.

Originally published at https://www.comet.ml on April 20, 2020.

Predictive Early Stopping — A Meta Learning Approach was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

NEW YORK, NY -- , a leading provider of full-stack solutions for data science teams, and , the industry-leading meta machine learning experimentation platform, announced a collaboration with a reference architecture for data science teams looking to harness the power of the Dell EMC infrastructure in tandem with Comet’s meta machine learning platform. Read a whitepaper , authored by Comet and Dell, on using Comet with Dell EMC AI-enabled infrastructure.

With Dell EMC PowerEdge reference architectures, organizations can deploy artificial intelligence workload-optimized rack systems approximately 6–12 months faster than it would have taken to design the correct configurations and deploy the solution. Organizations can now rely on architectures that are tested and validated by our Dell engineers and know that services are available when and where you need them.

“Orchestrating and managing the stack for enterprise data science teams is a huge pain point for many of our customers,” said Gideon Mendels, Co-founder/CEO, Comet. “Dell EMC’s Kubeflow and Kubernetes solutions are best-in-class and an excellent choice for any data science team looking to build a robust and scalable ML platform.”

Comet is a meta machine learning experimentation platform allowing users to automatically track their metrics, hyperparameters, dependencies, GPU utilization, datasets, models, debugging samples, and more. By leveraging Comet, data science teams produce faster research cycles, and more transparent and collaborative data science. Comet also provides a built-in hyperparameter optimization service, interactive confusion matrices, full code tracking and reproducibility features. Comet on-premise installations can support teams of any size, from a single machine to distributed microservices.

“This is one of those products that makes you question how you functioned without it. Comet gives data science teams all the automation and productivity features they need, but that they never get around to developing themselves,” says Phil Hummel, Senior Principal Engineer and Distinguished Member of Technical Staff, Dell EMC.

The Reference architecture utilizes Dell EMC AI-Enabled Kubernetes solution backed by Canonical’s Charmed Kubernetes and Kubeflow that complies with all requirements of the AI workload. The solution includes 100% upstream Kubernetes latest code wrapped into easily consumable packages and supported by Canonical.

Dell EMC and Comet’s reference architecture and data science team user story illustrates how our joint solution can provide teams the tools and infrastructure they need to manage their machine learning workflows-data storage, experimentation and model building, and deployment-while providing flexible and robust deployment options as teams scale. To learn more about Dell EMC’s AI-Enabled Kubernetes Cluster options, read more or reach out to Dell EMC directly to schedule a call . To learn more about Comet’s meta-machine learning platform, read more about Comet or contact Comet’s customer solutions team for more information or to schedule a demo.

Read the full whitepaper here that covers (a) using Comet with Dell EMC AI-enabled infrastructure and (b) technical installation instructions.

Originally published at https://www.comet.ml on March 30, 2020.

Press Release: Dell EMC and Comet announce machine learning platform collaboration was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 1973, at the height of the OPEC oil crisis and skyrocketing fuel prices, NASA scientist and USC professor Jack Nilles began thinking about ways work could be done without the need for commuting. Nilles’ thought experiments evolved into case studies, numerous books, including The Telecommunications-Transportation Tradeoff the original book on telecommuting, as well as dozens of papers, articles and keynote speeches. To this day, Nilles remains one of the principal evangelists for remote work as a viable alternative to a traditional office.

Today, it would not be entirely hyperbolic to consider Nilles’ early work clairvoyant. The last decade has seen a dramatic rise in companies adopting a remote workforce. A 2019 report compiled by Flexjobs and Global Workplace Analytics found that the number of people telecommuting in the United States increased by 159% from 2005 to 2017. With the rapidly evolving global crisis surrounding COVID-19, forcing more and more companies to adopt strict work-from-home policies, remote work is further cementing itself as a workplace paradigm to be taken seriously.

The data science industry is no exception. Large computer, IT and software development industries are among the fastest-growing with respect to the share of employees working on a full or partially-remote basis. At , we see many data science teams going remote in some capacity (we, too, have several team members working on a fully remote basis). The perks of remote work are well-documented: a joint report published by Buffer and Angellist-a survey of more than 3,500 remote tech workers-found that a flexible schedule, the ability to work from anywhere, and not having a commute were among the most attractive benefits of remote work.

Despite these benefits, putting the systems in place to make remote teams effective and productive is not trivial. Simple things like collaboration and communication become challenging. Employees are more prone to feel lonely, distracted, or to feel like the workday never ends. In data science teams, where industry best practices are still very much taking shape , these issues can be even more pronounced.

This post aims to highlight some work from home best practices, both general and data science-specific, in order to help data scientists and teams remain productive, connected and happy while working remotely.

Best Practices for Members of Data Science Teams

Create a Productive Workspace

Have a dedicated workspace (read: not your bed). Keep your phone away from your workspace and only check it during breaks. Be sure to keep your workspace clean-working in a messy environment can be incredibly distracting.

Make sure to chat with your colleagues — even when you don’t have to

Schedule a couple of video calls a day to stay sane, even if they’re not strictly necessary. Have at least one of these be a non-work related ‘remote coffee’ with a colleague/friend. Always put the camera on and try to enlarge the video call window so it really feels like you’re having a real conversation.

Working habits

Block time in your calendar for undistracted work time, a workout, lunch, etc. Encourage your colleagues to do the same. Over-communicate and assume good intent. A quick update message can go a long way in ensuring your manager is reassured with the work’s progress. Relatedly, agree a daily cadence to stay in touch with your teammates — e.g. a daily standup meeting each morning.

Have a clear ‘turn-off time’ (agreed with your team) — one of the biggest risks of remote is that you never switch off and stop working so better to be proactive on this one

Tips for Managing Data Science Teams Remotely

When thinking about how to manage a remote team it’s important to start by highlighting the key benefits and downsides of remote work.

At a high-level, the benefit to you as a manager is the potential of increased productivity for your team due to no commute, fewer distractions (hopefully), and more time to focus.

The downsides generally are less effective communication and fewer opportunities for collaboration. Some may argue you can be just as collaborative and communicative remotely, but as any economist would tell you: there is no free lunch. While remote teams do come with downsides, there are some very good strategies by which you can minimize them.

Take Advantage of the Pros of Remote Work

First, focus on and enable the strengths of remote work. You have just been given a team with the capacity to focus even more than before. Use that! In order to do so, you will need to do the following:

1. Trust your team. If you take one thing away from how to manage a remote team well, let it be this: trust your team to work effectively and manage their time well. If you spend a lot of time doing more micromanaging you will give up almost all the benefits of remote work.
2. Set clear expectations. If you can set clear work priorities with measurable milestones, you can let your team go to work and focus. At this point, working from home can be a huge advantage because of hopefully fewer distractions and more time to get work done.

If you can set clear expectations and trust your team to get the work done, working remotely can be an advantage. For example, you could check-in at the beginning of the day with the team, set expectations, and then let everyone go to work.

Build Systems and Processes to Counterbalance the Downsides

Now — let’s discuss the downsides. Communication and collaboration. Focused work is great when you know what needs to be done and can accomplish it mostly on your own. But what about those serendipitous moments of in-person collaboration? How do you replicate that? I would argue that you can’t completely. Communication and collaboration are probably always going to be better in person (thus, no free lunch), but that doesn’t mean you can’t minimize the downsides of remote work. To do so, I would recommend the following:

1. Communicate over video. A lot of how we communicate is non-verbal and not turning on your video during a call hurts your ability to communicate.
2. Don’t always go straight to business in your meetings. With a remote team, you don’t get time to have “water cooler” discussions. Spend some time in meetings, when appropriate, just talking to stay connected.
3. Have scheduled, remote-friendly team building activities. The “water cooler” discussions should happen every day, but I would also suggest a monthly activity with concentrated time to relax and get to know each other. One of our team’s favorites is .
4. When the next steps are not clear and you need creative collaboration, do remote brainstorming using the whiteboard feature on video conferencing platforms. Don’t just get in a virtual meeting and start discussing ideas. Send out the goal in writing beforehand, have people brainstorm individually and bring ideas to the meeting, and allow everyone to share his or her ideas, then discuss and prioritize. This is sometimes called Silent Brainstorming and is very remote-friendly and tends to be more effective.
5. Develop a culture of radical candor and psychological safety. One of your weaknesses is now communication because you are remote. Make sure your culture is especially strong in being open and honest with feedback. Both positive and negative! As a leader, be an example in asking for, giving, and receiving feedback. And make sure everyone feels safe in expressing ideas and giving opinions.

These steps should help build camaraderie and trust within your remote team, both of which are necessary for strong communication and collaboration. You don’t have the luxury of falling back on serendipitous moments of in-person collaboration or the post-meeting discussions that happen when people are in the same room.

Invest in Tools to Encourage Collaboration

There are amazing tools that you can leverage to make you even more effective with your communication and collaboration.

Zoom

A good video conferencing tool is indispensable for all teams, including and especially teams who work remote. We use zoom every day for meetings with remote team members. Whatever video conferencing tool you choose, be sure it’s easily accessible so your team can have as little friction as possible getting on the phone with colleagues.

Slack

Real-time messaging allows the flow of information across a team to be ad-hoc, conversational and informal. These are crucial values to ensure you maintain the ability for your team to develop ideas easily , organically, and collaboratively. Slack is a popular and excellent tool that allows teams to do just this.

Comet

Data Science teams are prone to becoming packs of lone-wolf modelers even when working in the same place. With the switch to remote, it’s even easier to fall into this trap. Fortunately tools exist that make it far easier for data scientists, machine learning engineers, or anyone working on models to log, visualize, share and reproduce their work across distributed teams in real time. We at Comet provide a meta-machine learning platform, runnable in the cloud or on-premise/VPC, that allows data science teams to do just this: reproduce full experiments (and not just code), manage users across large distributed data science teams, and provide managers insight into team contributions and performance.

Comet allows data science teams to easily manage distributed teams by creating collaborative projects with many users:

If your team is part of a collaborative project, every data scientist can publish their experiments in one central location, where visualizing, comparing, analyzing and debugging team-wide work becomes easy to do-even when working remotely:

Conclusion

With COVID-19 forcing many teams into remote work, members and managers of newly-remote teams will be well-served taking stock of how to make the most of their new telecommute. Teams that take advantage of remote work’s benefits will remain productive, communicative and happy. What’s more, safeguards and processes can be put in place to effectively counterbalance many of the common challenges remote teams face. Do yourself a favor and schedule a 15-minute video call with one of your coworkers later today. It will be time well-spent.

Originally published at https://www.comet.ml on March 20, 2020.

How to Make Remote Work Effective for Data Science Teams was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.

“AI is the new electricity.” At least, that’s what Andrew Ng suggested at this year’s Amazon re:MARS conference. In his keynote address, Ng discussed the rapid growth of artificial intelligence (AI) — its steady march into industry after industry; the unrelenting presence of AI breakthroughs, technologies, or fears in the headlines each day; the tremendous amount of investment, both from established enterprises seeking to modernize (see: Sony, a couple of weeks ago) as well as from venture investors parachuting into the market riding a wave of AI-focused founders.

“AI is the next big transformation,” Ng insists, and we’re watching the transformation unfold.

While AI may be the new electricity (and as a Data Scientist at Comet, I don’t need much convincing), significant challenges remain for the field to realize this potential. In this blog post, I’m going to talk about why data scientists and teams can’t rely on the tools and processes that software engineering teams have been using for the last 20 years for machine learning (ML).

The reliance on the tools and processes of software engineering makes sense — data science and software engineering are both disciplines whose principal tool is code. Yet what is being done in data science teams is radically different from what is being done in software engineering teams. An inspection of the core differences between the two disciplines is a helpful exercise in clarifying how we should think about structuring our tools and processes for doing AI.

At Comet, we believe the adoption of tools and processes designed specifically for AI will help practitioners unlock and enable the type of revolutionary transformation Ng is speaking about.

Different Disciplines, Different Processes

Software engineering is a discipline whose aim is, considered broadly, the design and implementation of programs that a computer can execute to perform a defined function. Assuming the input to a software program is within the expected (or constrained) range of inputs, its behavior is knowable. In a talk at ICML in 2015, Leon Bottou formulated this well: in software engineering an algorithm or program can be proven correct, in the sense that given particular assumptions about the input, certain properties will be true when the algorithm or program terminates.

The provable correctness of software programs has shaped the tools and processes we have built for doing software engineering. Consider one corollary characteristic of software programming that follows from provable correctness: if a program is provably correct for some input values, then the program contains sub-programs that are also provably correct for those input values. This is why engineering processes like Agile are, broadly speaking, successful and productive for software teams. Breaking apart these projects into sub-tasks works. Most waterfall and scrum implementations also include sub-tasking as well.

We see a lot of data science teams using workflow processes that are identical or broadly similar to these software methodologies. Unfortunately, they don’t work very well. The reason? The provable correctness of software engineering does not extend to AI and machine learning. In (supervised) machine learning, the only guarantee we have about a model we’ve built is that if the training set is an iid (independent and identically distributed) sample from some distribution, then performance on another iid sample from the same distribution will be close to the performance on the training set. Because uncertainty is an intrinsic property of machine learning, sub-tasking can lead to unforeseeable downstream effects.

Why is uncertainty intrinsic to machine learning?

Part of the answer lies in the fact that the problems that are both (a) interesting to us and (b) amenable to machine learning solutions (self-driving cars, object recognition, labeling images, and generative language models, to name a few) do not have a clear reproducible mathematical or programmatic specification. In place of specifications, machine learning systems feed in lots of data in order to detect patterns and generate predictions. Put another way, the purpose of machine learning is to create a statistical proxy that can serve as a specification for one of these tasks. We hope our collected data is a representative subsample of the real-world distribution, but in practice we cannot know exactly how well this condition is met. Finally, the algorithms and model architectures we use are complex, sufficiently complex that we cannot always break them apart into sub-models to understand precisely what is happening.

From this description, obstacles to the knowability of machine learning systems should be somewhat obvious. Inherent to the types of problems amenable to machine learning is a lack of a clear mathematical specification. The statistical proxy we use in the absence of a specification is accumulating lots of environmental data we hope is iid and representative. And the models we use to extract patterns from this collected data are sufficiently complex that we cannot reliably break them apart and understand precisely how they work. My colleague at Comet, Dhruv Nair, has written a three-part series on uncertainty in machine learning (here’s a link to Part I) if you’d like to dig deeper into this topic.

Consider, then, the implications for something like the Agile methodology used on a machine learning project. We cannot possibly hope to break machine learning tasks into sub-tasks, tackled as part of some larger sprint and then pieced together like legos into a whole product, platform, or feature, because we cannot reliably predict how the sub-models, or the model itself, will function.

Ng discussed this topic at re:MARS as well. He revealed how his team adopted a workflow system designed specifically for ML: 1 day sprints, structured as follows:

1. Build models and write code each day
2. Set up training and run experiments overnight
3. Analyze results in the morning and…
4. Repeat

Ng’s 1 day sprints methodology reflects something crucial to understanding and designing teams that practice machine learning: it is an inherently experimental science. Because the systems being built lack a clear specification, because data collection is an imperfect science, and because machine learning models are incredibly complex, experimentation is necessary. Rather than structuring team processes around a multi-week sprint, it is usually more fruitful to test out many different architectures, feature engineering choices, and optimization methods rapidly until a rough image of what is working and what isn’t starts to emerge. 1 day sprints allow teams to move quickly, test many hypotheses in a short amount of time, and begin building intuition and knowledge around a modeling task.

Tools for ML: Experiment Management

Let’s say you adopt Andrew Ng’s 1 day sprints methodology or something similar ( and you should). You’re setting new hyperparameters, tweaking your feature selections, and running experiments each night. What tool are you using to keep track of these decisions for each model training? How are you comparing experiments to see how different configurations are working? How are you sharing experiments with co-workers? Can your manager or co-worker reliably reproduce an experiment you ran yesterday?

In addition to processes, the tools you use to do machine learning matter as well. At Comet, our mission is to help companies extract business value from machine learning by providing a tool that does this for you. Most of the data science teams we speak to are stuck using a combination of git, emails, and (believe it or not) spreadsheets to record all of the artifacts around each experiment.

Consider a modeling task where you’re keeping track of 20 hyperparameters, 10 metrics, dozens of architectures and feature engineering techniques, all while iterating quickly and running dozens of models a day. It can become incredibly tedious to manually track all of these artifacts. Building a good ML model can oftentimes resemble tuning a radio with 50 knobs. If you don’t keep track of all of the configurations you’ve tried, the combinatorial complexity of finding the signal in your modeling space can become cumbersome.

We’ve built Comet based on these needs (and what we wanted when we were working on data science and machine learning ourselves, at Google, IBM, and as part of research groups at Columbia University and Yale University). Every time you train a model, there should be something to capture all of the artifacts of your experiment and save them in some central ledger where you can look up, compare, and filter through all of your (or your team’s) work. Comet was built to provide this function to practitioners of machine learning.

Measuring workflow efficiency is a notoriously difficult thing to do, but on average our users report 20–30% time savings by using Comet (note: Comet is free for individuals and researchers — you can sign-up here). This doesn’t take into account unique insights and knowledge that arise from having insights to things like a visual understanding of your hyperparameter space, real-time metric tracking, team-wide collaboration and experiment comparison. Access to this knowledge enables time savings as well as, and perhaps more importantly, the ability to build better models.

Looking Ahead

It is tempting to ignore questions about ML tools and processes altogether. In a field responsible for self-driving cars, voice assistants, facial recognition, and many more groundbreaking technologies, one may be forgiven for leaping into the fray of building these tools themselves and not considering how best to build them.

If you are convinced that the software engineering stack works well enough for doing AI, you will not be proven definitively right or wrong. After all, this is a field defined by uncertainty. But perhaps it is best to consider this in the way a data scientist may consider a modeling task: what is the probability distribution of possible futures? What is more or less likely? That a field as powerful and promising as AI will continue to rely on the tools and processes built for a different discipline, or that new ones will emerge to empower practitioners to the fullest?

If you are curious about these ML tools or have any questions, feel free to reach out to me at niko@comet.ml.

Additional Reading
Blogs on the differences between Machine Learning and Software Engineering:

1. Futurice Blog on ML vs Software Engineering
2. KD Nuggets Blog on ML vs Software Engineering
3. Concur Labs Blog on ML vs Software Engineering
4. Microsoft Case Study on Building ML Team Processes
5. Leon Bottou Slides from 2015 ICML Talk

Originally published at https://www.comet.ml on December 4, 2019.

Why software engineering processes and tools don’t work for machine learning was originally published in Comet on Medium, where people are continuing the conversation by highlighting and responding to this story.