<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashley Binford on Medium]]></title>
        <description><![CDATA[Stories by Ashley Binford on Medium]]></description>
        <link>https://medium.com/@ashleybinford?source=rss-c4f9a177f432------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*XSrT20ADfJur0I7XcAn2sw.png</url>
            <title>Stories by Ashley Binford on Medium</title>
            <link>https://medium.com/@ashleybinford?source=rss-c4f9a177f432------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:32:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashleybinford/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Customer Case Study: Building an end-to-end Speech Recognition model in PyTorch with AssemblyAI]]></title>
            <link>https://medium.com/data-science/customer-case-study-building-an-end-to-end-speech-recognition-model-in-pytorch-with-assemblyai-473030e47c7c?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/473030e47c7c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[assemblyai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Tue, 19 May 2020 22:32:39 GMT</pubDate>
            <atom:updated>2020-05-19T22:32:39.465Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was written by Michael Nguyen, Machine Learning Research Engineer at </em><a href="https://www.assemblyai.com/"><em>AssemblyAI</em></a><em>, and Niko Laskaris at </em><a href="http://www.comet.ml/"><em>Comet.ml</em></a><em>. AssemblyAI uses Comet to log, visualize, and understand their model development pipeline.</em></p><p>Deep Learning has changed the game in speech recognition with the introduction of end-to-end models. These models take in audio, and directly output transcriptions. Two of the most popular end-to-end models today are Deep Speech by Baidu, and Listen Attend Spell (LAS) by Google. Both Deep Speech and LAS, are recurrent neural network (RNN) based architectures with different approaches to modeling speech recognition. Deep Speech uses the Connectionist Temporal Classification (CTC) loss function to predict the speech transcript. LAS uses a sequence to sequence network architecture for its predictions.</p><p>These models simplified speech recognition pipelines by taking advantage of the capacity of deep learning system to learn from large datasets. With enough data, you should, in theory, be able to build a super robust speech recognition model that can account for all the nuance in speech without having to spend a ton of time and effort hand engineering acoustic features or dealing with complex pipelines in more old-school GMM-HMM model architectures, for example.</p><p>Deep learning is a fast-moving field, and Deep Speech and LAS style architectures are already quickly becoming outdated. You can read about where the industry is moving in the Latest Advancement Section below.</p><h3>How to Build Your Own End-to-End Speech Recognition Model in PyTorch</h3><p>Let’s walk through how one would build their own end-to-end speech recognition model in PyTorch. The model we’ll build is inspired by Deep Speech 2 (Baidu’s second revision of their now-famous model) with some personal improvements to the architecture. The output of the model will be a probability matrix of characters, and we’ll use that probability matrix to decode the most likely characters spoken from the audio. You can find the full code and also run the it with GPU support on <a href="https://colab.research.google.com/drive/1IPpwx4rX32rqHKpLz7dc8sOKspUa-YKO">Google Colaboratory</a>.</p><h3>Preparing the data pipeline</h3><p>Data is one of the most important aspects of speech recognition. We’ll take raw audio waves and transform them into Mel Spectrograms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zD-ICYnOm6oGOcAj.png" /></figure><p>You can read more on the details about how that transformation looks from this excellent post <a href="https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html">here</a>. For this post, you can just think of a Mel Spectrogram as essentially a picture of sound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GQBJQnfeq7PZMHvC.jpeg" /></figure><p>For handling the audio data, we are going to use an extremely useful utility called <strong>torchaudio </strong>which is a library built by the PyTorch team specifically for audio data. We’ll be training on a subset of <a href="http://www.openslr.org/12/">LibriSpeech</a>, which is a corpus of read English speech data derived from audiobooks, comprising 100 hours of transcribed audio data. You can easily download this dataset using <strong>torchaudio</strong>:</p><pre>import torchaudio train_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./&quot;, url=&quot;train-clean-100&quot;, download=True) <br>test_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./&quot;, url=&quot;test-clean&quot;, download=True)</pre><p>Each sample of the dataset contains the waveform, sample rate of audio, the utterance/label, and more metadata on the sample. You can view what each sample looks like from the source code <a href="https://github.com/pytorch/audio/blob/master/torchaudio/datasets/librispeech.py#L40">here</a>.</p><h3>Data Augmentation — SpecAugment</h3><p>Data augmentation is a technique used to artificially increase the diversity of your dataset in order to increase your dataset size. This strategy is especially helpful when data is scarce or if your model is overfitting. For speech recognition, you can do the standard augmentation techniques, like changing the pitch, speed, injecting noise, and adding reverb to your audio data.</p><p>We found Spectrogram Augmentation (SpecAugment), to be a much simpler and more effective approach. SpecAugment, was first introduced in the paper <a href="https://arxiv.org/abs/1904.08779">SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition</a>, in which the authors found that simply cutting out random blocks of consecutive time and frequency dimensions improved the models generalization abilities significantly!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*xvkW5hm10ZktfE9q.png" /></figure><p>In PyTorch, you can use the<strong> torchaudio</strong> function <strong>FrequencyMasking</strong> to mask out the frequency dimension, and <strong>TimeMasking</strong> for the time dimension.</p><pre>torchaudio.transforms.FrequencyMasking()<br>torchaudio.transforms.TimeMasking()</pre><p>Now that we have the data, we’ll need to transform the audio into Mel Spectrograms, and map the character labels for each audio sample into integer labels:</p><pre>class TextTransform:<br>    &quot;&quot;&quot;Maps characters to integers and vice versa&quot;&quot;&quot;<br>    def __init__(self):<br>        char_map_str = &quot;&quot;&quot;<br>        &#39; 0<br>        &lt;SPACE&gt; 1<br>        a 2<br>        b 3<br>        c 4<br>        d 5<br>        e 6<br>        f 7<br>        g 8<br>        h 9<br>        i 10<br>        j 11<br>        k 12<br>        l 13<br>        m 14<br>        n 15<br>        o 16<br>        p 17<br>        q 18<br>        r 19<br>        s 20<br>        t 21<br>        u 22<br>        v 23<br>        w 24<br>        x 25<br>        y 26<br>        z 27<br>        &quot;&quot;&quot;<br>        self.char_map = {}<br>        self.index_map = {}<br>        for line in char_map_str.strip().split(&#39;\n&#39;):<br>            ch, index = line.split()<br>            self.char_map[ch] = int(index)<br>            self.index_map[int(index)] = ch<br>        self.index_map[1] = &#39; &#39;</pre><pre>def text_to_int(self, text):<br>        &quot;&quot;&quot; Use a character map and convert text to an integer sequence &quot;&quot;&quot;<br>        int_sequence = []<br>        for c in text:<br>            if c == &#39; &#39;:<br>                ch = self.char_map[&#39;&#39;]<br>            else:<br>                ch = self.char_map[c]<br>            int_sequence.append(ch)<br>        return int_sequence</pre><pre>def int_to_text(self, labels):<br>        &quot;&quot;&quot; Use a character map and convert integer labels to an text sequence &quot;&quot;&quot;<br>        string = []<br>        for i in labels:<br>            string.append(self.index_map[i])<br>        return &#39;&#39;.join(string).replace(&#39;&#39;, &#39; &#39;)</pre><pre>train_audio_transforms = nn.Sequential(<br>    torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128),<br>    torchaudio.transforms.FrequencyMasking(freq_mask_param=15),<br>    torchaudio.transforms.TimeMasking(time_mask_param=35)<br>)</pre><pre>valid_audio_transforms = torchaudio.transforms.MelSpectrogram()</pre><pre>text_transform = TextTransform()</pre><pre>def data_processing(data, data_type=&quot;train&quot;):<br>    spectrograms = []<br>    labels = []<br>    input_lengths = []<br>    label_lengths = []<br>    for (waveform, _, utterance, _, _, _) in data:<br>        if data_type == &#39;train&#39;:<br>            spec = train_audio_transforms(waveform).squeeze(0).transpose(0, 1)<br>        else:<br>            spec = valid_audio_transforms(waveform).squeeze(0).transpose(0, 1)<br>        spectrograms.append(spec)<br>        label = torch.Tensor(text_transform.text_to_int(utterance.lower()))<br>        labels.append(label)<br>        input_lengths.append(spec.shape[0]//2)<br>        label_lengths.append(len(label))</pre><pre>spectrograms = nn.utils.rnn.pad_sequence(spectrograms, batch_first=True).unsqueeze(1).transpose(2, 3)<br>    labels = nn.utils.rnn.pad_sequence(labels, batch_first=True)</pre><pre>return spectrograms, labels, input_lengths, label_lengths</pre><h3>Define the Model — Deep Speech 2 (but better)</h3><p>Our model will be similar to the Deep Speech 2 architecture. The model will have two main neural network modules — N layers of Residual Convolutional Neural Networks (ResCNN) to learn the relevant audio features, and a set of Bidirectional Recurrent Neural Networks (BiRNN) to leverage the learned ResCNN audio features. The model is topped off with a fully connected layer used to classify characters per time step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1TokC95mn1dVllEx.png" /></figure><p>Convolutional Neural Networks (CNN) are great at extracting abstract features, and we’ll apply the same feature extraction power to audio spectrograms. Instead of just vanilla CNN layers, we choose to use Residual CNN layers. Residual connections (AKA skip connections) were first introduced in the paper <a href="https://arxiv.org/abs/1512.03385">Deep Residual Learning for Image Recognition</a>, where the author found that you can build really deep networks with good accuracy gains if you add these connections to your CNN’s. Adding these Residual connections also helps the model learn faster and generalize better. The paper <a href="https://arxiv.org/abs/1712.09913">Visualizing the Loss Landscape of Neural Nets</a> shows that networks with residual connections have a “flatter” loss surface, making it easier for models to navigate the loss landscape and find a lower and more generalizable minima.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kR_1BxeE8lohr6El.png" /></figure><p>Recurrent Neural Networks (RNN) are naturally great at sequence modeling problems. RNN’s processes the audio features step by step, making a prediction for each frame while using context from previous frames. We use BiRNN’s because we want the context of not only the frame before each step, but the frames after it as well. This can help the model make better predictions, as each frame in the audio will have more information before making a prediction. We use Gated Recurrent Unit (GRU’s) variant of RNN’s as it needs less computational resources than LSTM’s, and works just as well in some cases.</p><p>The model outputs a probability matrix for characters which we’ll use to feed into our decoder to extract what the model believes are the highest probability characters that were spoken.</p><pre>class CNNLayerNorm(nn.Module):<br>    &quot;&quot;&quot;Layer normalization built for cnns input&quot;&quot;&quot;<br>    def __init__(self, n_feats):<br>        super(CNNLayerNorm, self).__init__()<br>        self.layer_norm = nn.LayerNorm(n_feats)</pre><pre>def forward(self, x):<br>        # x (batch, channel, feature, time)<br>        x = x.transpose(2, 3).contiguous() # (batch, channel, time, feature)<br>        x = self.layer_norm(x)<br>        return x.transpose(2, 3).contiguous() # (batch, channel, feature, time)</pre><pre>class ResidualCNN(nn.Module):<br>    &quot;&quot;&quot;Residual CNN inspired by <a href="https://arxiv.org/pdf/1603.05027.pdf">https://arxiv.org/pdf/1603.05027.pdf</a><br>        except with layer norm instead of batch norm<br>    &quot;&quot;&quot;<br>    def __init__(self, in_channels, out_channels, kernel, stride, dropout, n_feats):<br>        super(ResidualCNN, self).__init__()</pre><pre>self.cnn1 = nn.Conv2d(in_channels, out_channels, kernel, stride, padding=kernel//2)<br>        self.cnn2 = nn.Conv2d(out_channels, out_channels, kernel, stride, padding=kernel//2)<br>        self.dropout1 = nn.Dropout(dropout)<br>        self.dropout2 = nn.Dropout(dropout)<br>        self.layer_norm1 = CNNLayerNorm(n_feats)<br>        self.layer_norm2 = CNNLayerNorm(n_feats)</pre><pre>def forward(self, x):<br>        residual = x  # (batch, channel, feature, time)<br>        x = self.layer_norm1(x)<br>        x = F.gelu(x)<br>        x = self.dropout1(x)<br>        x = self.cnn1(x)<br>        x = self.layer_norm2(x)<br>        x = F.gelu(x)<br>        x = self.dropout2(x)<br>        x = self.cnn2(x)<br>        x += residual<br>        return x # (batch, channel, feature, time)</pre><pre>class BidirectionalGRU(nn.Module):</pre><pre>def __init__(self, rnn_dim, hidden_size, dropout, batch_first):<br>        super(BidirectionalGRU, self).__init__()</pre><pre>self.BiGRU = nn.GRU(<br>            input_size=rnn_dim, hidden_size=hidden_size,<br>            num_layers=1, batch_first=batch_first, bidirectional=True)<br>        self.layer_norm = nn.LayerNorm(rnn_dim)<br>        self.dropout = nn.Dropout(dropout)</pre><pre>def forward(self, x):<br>        x = self.layer_norm(x)<br>        x = F.gelu(x)<br>        x, _ = self.BiGRU(x)<br>        x = self.dropout(x)<br>        return x</pre><pre>class SpeechRecognitionModel(nn.Module):<br>    &quot;&quot;&quot;Speech Recognition Model Inspired by DeepSpeech 2&quot;&quot;&quot;</pre><pre>def __init__(self, n_cnn_layers, n_rnn_layers, rnn_dim, n_class, n_feats, stride=2, dropout=0.1):<br>        super(SpeechRecognitionModel, self).__init__()<br>        n_feats = n_feats//2<br>        self.cnn = nn.Conv2d(1, 32, 3, stride=stride, padding=3//2)  # cnn for extracting heirachal features</pre><pre># n residual cnn layers with filter size of 32<br>        self.rescnn_layers = nn.Sequential(*[<br>            ResidualCNN(32, 32, kernel=3, stride=1, dropout=dropout, n_feats=n_feats) <br>            for _ in range(n_cnn_layers)<br>        ])<br>        self.fully_connected = nn.Linear(n_feats*32, rnn_dim)<br>        self.birnn_layers = nn.Sequential(*[<br>            BidirectionalGRU(rnn_dim=rnn_dim if i==0 else rnn_dim*2,<br>                             hidden_size=rnn_dim, dropout=dropout, batch_first=i==0)<br>            for i in range(n_rnn_layers)<br>        ])<br>        self.classifier = nn.Sequential(<br>            nn.Linear(rnn_dim*2, rnn_dim),  # birnn returns rnn_dim*2<br>            nn.GELU(),<br>            nn.Dropout(dropout),<br>            nn.Linear(rnn_dim, n_class)<br>        )</pre><pre>def forward(self, x):<br>        x = self.cnn(x)<br>        x = self.rescnn_layers(x)<br>        sizes = x.size()<br>        x = x.view(sizes[0], sizes[1] * sizes[2], sizes[3])  # (batch, feature, time)<br>        x = x.transpose(1, 2) # (batch, time, feature)<br>        x = self.fully_connected(x)<br>        x = self.birnn_layers(x)<br>        x = self.classifier(x)<br>        return x</pre><h3>Picking the Right Optimizer and Scheduler — AdamW with Super Convergence</h3><p>The optimizer and learning rate schedule plays a very important role in getting our model to converge to the best point. Picking the right optimizer and scheduler can also save you compute time, and help your model generalize better to real-world use cases. For our model, we’ll be using <strong>AdamW</strong> with the <strong>One Cycle Learning Rate Scheduler</strong>. <strong>Adam</strong> is a widely used optimizer that helps your model converge more quickly, therefore, saving compute time, but has been notorious for not generalizing as well as <strong>Stochastic Gradient Descent</strong> AKA <strong>SGD</strong>.</p><p><strong>AdamW</strong> was first introduced in <a href="https://arxiv.org/abs/1711.05101">Decoupled Weight Decay Regularization</a>, and is considered a “fix” to <strong>Adam</strong>. The paper pointed out that the original <strong>Adam</strong> algorithm has a wrong implementation of weight decay, which <strong>AdamW</strong> attempts to fix. This fix helps with <strong>Adam</strong> ‘s generalization problem.</p><p>The <strong>One Cycle Learning Rate Scheduler</strong> was first introduced in the paper <a href="https://arxiv.org/abs/1708.07120">Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates</a>. This paper shows that you can train neural networks an order of magnitude faster, while keeping their generalizable abilities, using a simple trick. You start with a low learning rate, which warms up to a large maximum learning rate, then decays linearly to the same point of where you originally started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*R61grvRimy9VHEIg.jpeg" /></figure><p>Because the maximum learning rate is magnitudes higher than the lowest, you also gain some regularization benefits which helps your model generalize better if you have a smaller set of data.</p><p>With PyTorch, these two methods are already part of the package.</p><pre>optimizer = optim.AdamW(model.parameters(), hparams[&#39;learning_rate&#39;])<br>scheduler = optim.lr_scheduler.OneCycleLR(optimizer,<br>	max_lr=hparams[&#39;learning_rate&#39;],<br>	steps_per_epoch=int(len(train_loader)),<br>	epochs=hparams[&#39;epochs&#39;],<br>	anneal_strategy=&#39;linear&#39;)</pre><h3>The CTC Loss Function — Aligning Audio to Transcript</h3><p>Our model will be trained to predict the probability distribution of all characters in the alphabet for each frame (ie, timestep) in the spectrogram we feed into the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/0*ln_7M-tYyGuLTcrB.png" /><figcaption>Image taken from distill.pub</figcaption></figure><p>Traditional speech recognition models would require you to align the transcript text to the audio before training, and the model would be trained to predict specific labels at specific frames.</p><p>The innovation of the CTC loss function is that it allows us to skip this step. Our model will learn to align the transcript itself during training. The key to this is the “blank” label introduced by CTC, which gives the model the ability to say that a certain audio frame did not produce a character. You can see a more detailed explanation of CTC and how it works from <a href="https://distill.pub/2017/ctc/">this excellent post</a>.</p><p>The CTC loss function is also built into PyTorch.</p><pre>criterion = nn.CTCLoss(blank=28).to(device)</pre><h3>Evaluating Your Speech Model</h3><p>When Evaluating your speech recognition model, the industry standard is using the Word Error Rate (WER) as the metric. The Word Error Rate does exactly what it says — it takes the transcription your model outputs, and the true transcription, and measures the error between them. You can see how that’s implemented <a href="https://colab.research.google.com/drive/1IPpwx4rX32rqHKpLz7dc8sOKspUa-YKO">here</a>. Another useful metric is called the Character Error Rate (CER). The CER measures the error of the characters between the model’s output and the true labels. These metrics are helpful to measure how well your model performs.</p><p>For this tutorial, we’ll use a “greedy” decoding method to process our model’s output into characters that can be combined to create the transcript. A “greedy” decoder takes in the model output, which is a softmax probability matrix of characters, and for each time step (spectrogram frame), it chooses the label with the highest probability. If the label is a blank label, we remove it from the final transcript.</p><pre>def GreedyDecoder(output, labels, label_lengths, blank_label=28, collapse_repeated=True):<br>    arg_maxes = torch.argmax(output, dim=2)<br>    decodes = []<br>    targets = []<br>    for i, args in enumerate(arg_maxes):<br>        decode = []<br>        targets.append(text_transform.int_to_text(labels[i][:label_lengths[i]].tolist()))<br>        for j, index in enumerate(args):<br>            if index != blank_label:<br>                if collapse_repeated and j != 0 and index == args[j -1]:<br>                    continue<br>                decode.append(index.item())<br>        decodes.append(text_transform.int_to_text(decode))<br>    return decodes, targets</pre><h3>Training and Monitoring Your Experiments Using Comet.ml</h3><p><a href="https://www.comet.ml/site/">Comet.ml</a> provides a platform that allows deep learning researchers to track, compare, explain, and optimize their experiments and models. Comet.ml has improved our productivity at AssemblyAI and we highly recommend using this platform for teams doing any sort of data science experiments. Comet.ml is super easy to set up. And works with just a few lines of code.</p><pre># initialize experiment object<br>experiment = Experiment(api_key=comet_api_key, project_name=project_name)<br>experiment.set_name(exp_name)</pre><pre># track metrics<br>experiment.log_metric(&#39;loss&#39;, loss.item())</pre><p><a href="https://www.comet.ml/site/">Comet.ml</a> provides you with a very productive dashboard where you can view and track your model’s progress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iwawLH_3rwpSmM58.png" /></figure><p>You can use Comet to track metrics, code, hyper parameters, your model’s graphs, among many other things! A really handy feature that Comet provides is the ability to compare your experiment among many other experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xqIYrQAW66p9xk-k.png" /></figure><p>Comet has a rich feature set that we won’t cover all here, but we highly recommended using it for a productivity and sanity boost.</p><p>Here is the rest of our training script.</p><pre>class IterMeter(object):<br>    &quot;&quot;&quot;keeps track of total iterations&quot;&quot;&quot;<br>    def __init__(self):<br>        self.val = 0</pre><pre>def step(self):<br>        self.val += 1</pre><pre>def get(self):<br>        return self.val</pre><pre>def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):<br>    model.train()<br>    data_len = len(train_loader.dataset)<br>    with experiment.train():<br>        for batch_idx, _data in enumerate(train_loader):<br>            spectrograms, labels, input_lengths, label_lengths = _data <br>            spectrograms, labels = spectrograms.to(device), labels.to(device)</pre><pre>optimizer.zero_grad()</pre><pre>output = model(spectrograms)  # (batch, time, n_class)<br>            output = F.log_softmax(output, dim=2)<br>            output = output.transpose(0, 1) # (time, batch, n_class)</pre><pre>loss = criterion(output, labels, input_lengths, label_lengths)<br>            loss.backward()</pre><pre>experiment.log_metric(&#39;loss&#39;, loss.item(), step=iter_meter.get())<br>            experiment.log_metric(&#39;learning_rate&#39;, scheduler.get_lr(), step=iter_meter.get())</pre><pre>optimizer.step()<br>            scheduler.step()<br>            iter_meter.step()<br>            if batch_idx % 100 == 0 or batch_idx == data_len:<br>                print(&#39;Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}&#39;.format(<br>                    epoch, batch_idx * len(spectrograms), data_len,<br>                    100. * batch_idx / len(train_loader), loss.item()))</pre><pre>def test(model, device, test_loader, criterion, epoch, iter_meter, experiment):<br>    print(&#39;\nevaluating…&#39;)<br>    model.eval()<br>    test_loss = 0<br>    test_cer, test_wer = [], []<br>    with experiment.test():<br>        with torch.no_grad():<br>            for I, _data in enumerate(test_loader):<br>                spectrograms, labels, input_lengths, label_lengths = _data <br>                spectrograms, labels = spectrograms.to(device), labels.to(device)</pre><pre>output = model(spectrograms)  # (batch, time, n_class)<br>                output = F.log_softmax(output, dim=2)<br>                output = output.transpose(0, 1) # (time, batch, n_class)</pre><pre>loss = criterion(output, labels, input_lengths, label_lengths)<br>                test_loss += loss.item() / len(test_loader)</pre><pre>decoded_preds, decoded_targets = GreedyDecoder(output.transpose(0, 1), labels, label_lengths)<br>                for j in range(len(decoded_preds)):<br>                    test_cer.append(cer(decoded_targets[j], decoded_preds[j]))<br>                    test_wer.append(wer(decoded_targets[j], decoded_preds[j]))</pre><pre>avg_cer = sum(test_cer)/len(test_cer)<br>    avg_wer = sum(test_wer)/len(test_wer)<br>    experiment.log_metric(&#39;test_loss&#39;, test_loss, step=iter_meter.get())<br>    experiment.log_metric(&#39;cer&#39;, avg_cer, step=iter_meter.get())<br>    experiment.log_metric(&#39;wer&#39;, avg_wer, step=iter_meter.get())</pre><pre>print(&#39;Test set: Average loss: {:.4f}, Average CER: {:4f} Average WER: {:.4f}\n&#39;.format(test_loss, avg_cer, avg_wer))</pre><pre>def main(learning_rate=5e-4, batch_size=20, epochs=10,<br>        train_url=&quot;train-clean-100&quot;, test_url=&quot;test-clean&quot;,<br>        experiment=Experiment(api_key=&#39;dummy_key&#39;, disabled=True)):</pre><pre>hparams = {<br>        &quot;n_cnn_layers&quot;: 3,<br>        &quot;n_rnn_layers&quot;: 5,<br>        &quot;rnn_dim&quot;: 512,<br>        &quot;n_class&quot;: 29,<br>        &quot;n_feats&quot;: 128,<br>        &quot;stride&quot;: 2,<br>        &quot;dropout&quot;: 0.1,<br>        &quot;learning_rate&quot;: learning_rate,<br>        &quot;batch_size&quot;: batch_size,<br>        &quot;epochs&quot;: epochs<br>    }</pre><pre>experiment.log_parameters(hparams)</pre><pre>use_cuda = torch.cuda.is_available()<br>    torch.manual_seed(7)<br>    device = torch.device(&quot;cuda&quot; if use_cuda else &quot;cpu&quot;)</pre><pre>if not os.path.isdir(&quot;./data&quot;):<br>        os.makedirs(&quot;./data&quot;)</pre><pre>train_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./data&quot;, url=train_url, download=True)<br>    test_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./data&quot;, url=test_url, download=True)</pre><pre>kwargs = {&#39;num_workers&#39;: 1, &#39;pin_memory&#39;: True} if use_cuda else {}<br>    train_loader = data.DataLoader(dataset=train_dataset,<br>                                batch_size=hparams[&#39;batch_size&#39;],<br>                                shuffle=True,<br>                                collate_fn=lambda x: data_processing(x, &#39;train&#39;),<br>                                **kwargs)<br>    test_loader = data.DataLoader(dataset=test_dataset,<br>                                batch_size=hparams[&#39;batch_size&#39;],<br>                                shuffle=False,<br>                                collate_fn=lambda x: data_processing(x, &#39;valid&#39;),<br>                                **kwargs)</pre><pre>model = SpeechRecognitionModel(<br>        hparams[&#39;n_cnn_layers&#39;], hparams[&#39;n_rnn_layers&#39;], hparams[&#39;rnn_dim&#39;],<br>        hparams[&#39;n_class&#39;], hparams[&#39;n_feats&#39;], hparams[&#39;stride&#39;], hparams[&#39;dropout&#39;]<br>        ).to(device)</pre><pre>print(model)<br>    print(&#39;Num Model Parameters&#39;, sum([param.nelement() for param in model.parameters()]))</pre><pre>optimizer = optim.AdamW(model.parameters(), hparams[&#39;learning_rate&#39;])<br>    criterion = nn.CTCLoss(blank=28).to(device)<br>    scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=hparams[&#39;learning_rate&#39;], <br>                                            steps_per_epoch=int(len(train_loader)),<br>                                            epochs=hparams[&#39;epochs&#39;],<br>                                            anneal_strategy=&#39;linear&#39;)</pre><pre>iter_meter = IterMeter()<br>    for epoch in range(1, epochs + 1):<br>        train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)<br>        test(model, device, test_loader, criterion, epoch, iter_meter, experiment)</pre><p>The<strong> train</strong> function trains the model on a full epoch of data. The <strong>test</strong> function evaluates the model on test data after every epoch. It gets the<strong> test_loss</strong> as well as the <strong>cer</strong> and <strong>wer</strong> of the model. You can start running the training script right now with GPU support in the <a href="https://colab.research.google.com/drive/1IPpwx4rX32rqHKpLz7dc8sOKspUa-YKO">Google Colaboratory</a>.</p><h3>How to Improve Accuracy</h3><p>Speech Recognition Requires a ton of data and a ton of compute resources. The example laid out is trained on a subset of LibriSpeech (100 hours of audio) and a single GPU. To get state of the art results you’ll need to do distributed training on thousands of hours of data, on tens of GPU’s spread out across many machines.</p><p>Another way to get a big accuracy improvement is to decode the CTC probability matrix using a Language Model and the CTC beam search algorithm. CTC type models are very dependent on this decoding process to get good results. Luckily there is a handy <a href="https://github.com/parlance/ctcdecode">open source library</a> that allows you to do that.</p><p>This tutorial was made to be more accessible so it’s a relatively small model (23 million Parameters) compared to something like BERT (340 million Parameters). It seems to be the larger you can get your network, the better it performs, although there are diminishing returns. A larger model equating to better performance is not always the case though, as proven by OpenAI’s research <a href="https://openai.com/assets/deep-double-descent/">Deep Double Descent</a>.</p><p>This model has 3 residual CNN layers and 5 Bidirectional GRU layers which should allow you to train a reasonable batch size on a single GPU with at least 11GB of memory. You can tweak some of the hyper parameters in the main function to reduce or increase the model size for your use case and compute availability.</p><h3>Latest Advancements In Speech Recognition with Deep Learning</h3><p>Deep learning is a fast-moving field. It seems like you can’t go a week without some new technique getting state of the art results. Here are a few of things worth exploring int the world of speech recognition.</p><h3>Transformers</h3><p>Transformers have taken the Natural Language Processing world by storm! First Introduced in the paper <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a>, transformers have been taking and modified to beat pretty much all existing NLP task dethroning RNN’s type architectures. The Transformer’s ability to see the full context of sequence data is transferable to speech as well.</p><h3>Unsupervised Pre-training</h3><p>If you follow deep learning closely you’ve probably heard of BERT, GPT, and GPT2. These Transformer models have first pertained on a language modeling task with unlabeled text data, and fine-tuned on a wide array of NLP task and get state of the art results! During pre-training, the model learns something fundamental on the statistics of language and uses that power to excel at other tasks. We believe this technique has great promises on speech data as well.</p><h3>Word Piece Models</h3><p>Our model defined above output characters. Some benefits to that are the model doesn’t have to worry about out of vocabulary words when running inference on speech. So for the word <strong>c h a t</strong> each character has is its own label. The downside to using characters are inefficiency and the model being prone to more errors because you’re predicting one character at a time. Using the whole word as labels have been explored, to some degree of success. Using this method, the entire word <strong>chat</strong> would be the label. But using whole words, you would have to keep an index of all possible vocabularies to make a prediction, which is memory inefficient with the possibility of running into out of vocabulary words during prediction. The sweet spot would be using word piece or sub-word units as labels. Instead of characters for the individual label, you can chop up the words into sub-word units, and use those as labels, i.e. <strong>ch at</strong>. This solves the out of vocabulary issue, and is much more efficient, as it needs fewer steps to decode then using characters, and without the need to have an index of all possible words. Word pieces have been used successfully with many NLP models, like BERT and would work natural with speech recognition problems as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=473030e47c7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/customer-case-study-building-an-end-to-end-speech-recognition-model-in-pytorch-with-assemblyai-473030e47c7c">Customer Case Study: Building an end-to-end Speech Recognition model in PyTorch with AssemblyAI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer Case Study: Building an end-to-end Speech Recognition model in PyTorch with AssemblyAI]]></title>
            <link>https://medium.com/comet-ml/customer-case-study-building-an-end-to-end-speech-recognition-model-in-pytorch-with-assemblyai-cea744401fe8?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/cea744401fe8</guid>
            <category><![CDATA[assemblyai]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 18 May 2020 21:07:07 GMT</pubDate>
            <atom:updated>2020-05-19T13:02:17.798Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was written by Michael Nguyen, Machine Learning Research Engineer at </em><a href="https://www.assemblyai.com/"><em>AssemblyAI</em></a><em>. AssemblyAI uses Comet to log, visualize, and understand their model development pipeline.</em></p><p>Deep Learning has changed the game in speech recognition with the introduction of end-to-end models. These models take in audio, and directly output transcriptions. Two of the most popular end-to-end models today are Deep Speech by Baidu, and Listen Attend Spell (LAS) by Google. Both Deep Speech and LAS, are recurrent neural network (RNN) based architectures with different approaches to modeling speech recognition. Deep Speech uses the Connectionist Temporal Classification (CTC) loss function to predict the speech transcript. LAS uses a sequence to sequence network architecture for its predictions.</p><p>These models simplified speech recognition pipelines by taking advantage of the capacity of deep learning system to learn from large datasets. With enough data, you should, in theory, be able to build a super robust speech recognition model that can account for all the nuance in speech without having to spend a ton of time and effort hand engineering acoustic features or dealing with complex pipelines in more old-school GMM-HMM model architectures, for example.</p><p>Deep learning is a fast-moving field, and Deep Speech and LAS style architectures are already quickly becoming outdated. You can read about where the industry is moving in the Latest Advancement Section below.</p><h3>How to Build Your Own End-to-End Speech Recognition Model in PyTorch</h3><p>Let’s walk through how one would build their own end-to-end speech recognition model in PyTorch. The model we’ll build is inspired by Deep Speech 2 (Baidu’s second revision of their now-famous model) with some personal improvements to the architecture. The output of the model will be a probability matrix of characters, and we’ll use that probability matrix to decode the most likely characters spoken from the audio. You can find the full code and also run the it with GPU support on <a href="https://colab.research.google.com/drive/1IPpwx4rX32rqHKpLz7dc8sOKspUa-YKO">Google Colaboratory</a>.</p><h4>Preparing the data pipeline</h4><p>Data is one of the most important aspects of speech recognition. We’ll take raw audio waves and transform them into Mel Spectrograms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzORF_IT8BLP3zpYoif45A.png" /></figure><p>You can read more on the details about how that transformation looks from this excellent post <a href="https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html">here</a>. For this post, you can just think of a Mel Spectrogram as essentially a picture of sound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bRgvqH75Dz78YE3uyKZePQ.jpeg" /></figure><p>For handling the audio data, we are going to use an extremely useful utility called <strong>torchaudio </strong>which is a library built by the PyTorch team specifically for audio data. We’ll be training on a subset of <a href="http://www.openslr.org/12/">LibriSpeech</a>, which is a corpus of read English speech data derived from audiobooks, comprising 100 hours of transcribed audio data. You can easily download this dataset using <strong>torchaudio</strong>:</p><pre>import torchaudio train_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./&quot;, url=&quot;train-clean-100&quot;, download=True) <br>test_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./&quot;, url=&quot;test-clean&quot;, download=True)</pre><p>Each sample of the dataset contains the waveform, sample rate of audio, the utterance/label, and more metadata on the sample. You can view what each sample looks like from the source code <a href="https://github.com/pytorch/audio/blob/master/torchaudio/datasets/librispeech.py#L40">here</a>.</p><h4>Data Augmentation — SpecAugment</h4><p>Data augmentation is a technique used to artificially increase the diversity of your dataset in order to increase your dataset size. This strategy is especially helpful when data is scarce or if your model is overfitting. For speech recognition, you can do the standard augmentation techniques, like changing the pitch, speed, injecting noise, and adding reverb to your audio data.</p><p>We found Spectrogram Augmentation (SpecAugment), to be a much simpler and more effective approach. SpecAugment, was first introduced in the paper <a href="https://arxiv.org/abs/1904.08779">SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition</a>, in which the authors found that simply cutting out random blocks of consecutive time and frequency dimensions improved the models generalization abilities significantly!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*Hjkem8_fvGnPWcB8l3gUtA.png" /></figure><p>In PyTorch, you can use the<strong> torchaudio</strong> function <strong>FrequencyMasking</strong> to mask out the frequency dimension, and <strong>TimeMasking</strong> for the time dimension.</p><pre>torchaudio.transforms.FrequencyMasking()<br>torchaudio.transforms.TimeMasking()</pre><p>Now that we have the data, we’ll need to transform the audio into Mel Spectrograms, and map the character labels for each audio sample into integer labels:</p><pre>class TextTransform:<br>    &quot;&quot;&quot;Maps characters to integers and vice versa&quot;&quot;&quot;<br>    def __init__(self):<br>        char_map_str = &quot;&quot;&quot;<br>        &#39; 0<br>        &lt;SPACE&gt; 1<br>        a 2<br>        b 3<br>        c 4<br>        d 5<br>        e 6<br>        f 7<br>        g 8<br>        h 9<br>        i 10<br>        j 11<br>        k 12<br>        l 13<br>        m 14<br>        n 15<br>        o 16<br>        p 17<br>        q 18<br>        r 19<br>        s 20<br>        t 21<br>        u 22<br>        v 23<br>        w 24<br>        x 25<br>        y 26<br>        z 27<br>        &quot;&quot;&quot;<br>        self.char_map = {}<br>        self.index_map = {}<br>        for line in char_map_str.strip().split(&#39;\n&#39;):<br>            ch, index = line.split()<br>            self.char_map[ch] = int(index)<br>            self.index_map[int(index)] = ch<br>        self.index_map[1] = &#39; &#39;<br><br>    def text_to_int(self, text):<br>        &quot;&quot;&quot; Use a character map and convert text to an integer sequence &quot;&quot;&quot;<br>        int_sequence = []<br>        for c in text:<br>            if c == &#39; &#39;:<br>                ch = self.char_map[&#39;&#39;]<br>            else:<br>                ch = self.char_map[c]<br>            int_sequence.append(ch)<br>        return int_sequence<br><br>    def int_to_text(self, labels):<br>        &quot;&quot;&quot; Use a character map and convert integer labels to an text sequence &quot;&quot;&quot;<br>        string = []<br>        for i in labels:<br>            string.append(self.index_map[i])<br>        return &#39;&#39;.join(string).replace(&#39;&#39;, &#39; &#39;)<br><br><br>train_audio_transforms = nn.Sequential(<br>    torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128),<br>    torchaudio.transforms.FrequencyMasking(freq_mask_param=15),<br>    torchaudio.transforms.TimeMasking(time_mask_param=35)<br>)<br><br>valid_audio_transforms = torchaudio.transforms.MelSpectrogram()<br><br>text_transform = TextTransform()<br><br><br>def data_processing(data, data_type=&quot;train&quot;):<br>    spectrograms = []<br>    labels = []<br>    input_lengths = []<br>    label_lengths = []<br>    for (waveform, _, utterance, _, _, _) in data:<br>        if data_type == &#39;train&#39;:<br>            spec = train_audio_transforms(waveform).squeeze(0).transpose(0, 1)<br>        else:<br>            spec = valid_audio_transforms(waveform).squeeze(0).transpose(0, 1)<br>        spectrograms.append(spec)<br>        label = torch.Tensor(text_transform.text_to_int(utterance.lower()))<br>        labels.append(label)<br>        input_lengths.append(spec.shape[0]//2)<br>        label_lengths.append(len(label))<br><br>    spectrograms = nn.utils.rnn.pad_sequence(spectrograms, batch_first=True).unsqueeze(1).transpose(2, 3)<br>    labels = nn.utils.rnn.pad_sequence(labels, batch_first=True)<br><br>    return spectrograms, labels, input_lengths, label_lengths</pre><h3>Define the Model — Deep Speech 2 (but better)</h3><p>Our model will be similar to the Deep Speech 2 architecture. The model will have two main neural network modules — N layers of Residual Convolutional Neural Networks (ResCNN) to learn the relevant audio features, and a set of Bidirectional Recurrent Neural Networks (BiRNN) to leverage the learned ResCNN audio features. The model is topped off with a fully connected layer used to classify characters per time step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgSpH6OfrVWQXaF6K0580g.png" /></figure><p>Convolutional Neural Networks (CNN) are great at extracting abstract features, and we’ll apply the same feature extraction power to audio spectrograms. Instead of just vanilla CNN layers, we choose to use Residual CNN layers. Residual connections (AKA skip connections) were first introduced in the paper <a href="https://arxiv.org/abs/1512.03385">Deep Residual Learning for Image Recognition</a>, where the author found that you can build really deep networks with good accuracy gains if you add these connections to your CNN’s. Adding these Residual connections also helps the model learn faster and generalize better. The paper <a href="https://arxiv.org/abs/1712.09913">Visualizing the Loss Landscape of Neural Nets</a> shows that networks with residual connections have a “flatter” loss surface, making it easier for models to navigate the loss landscape and find a lower and more generalizable minima.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qTVxpLrpnOVzTiDnusJc8g.png" /></figure><p>Recurrent Neural Networks (RNN) are naturally great at sequence modeling problems. RNN’s processes the audio features step by step, making a prediction for each frame while using context from previous frames. We use BiRNN’s because we want the context of not only the frame before each step, but the frames after it as well. This can help the model make better predictions, as each frame in the audio will have more information before making a prediction. We use Gated Recurrent Unit (GRU’s) variant of RNN’s as it needs less computational resources than LSTM’s, and works just as well in some cases.</p><p>The model outputs a probability matrix for characters which we’ll use to feed into our decoder to extract what the model believes are the highest probability characters that were spoken.</p><pre>class CNNLayerNorm(nn.Module):<br>    &quot;&quot;&quot;Layer normalization built for cnns input&quot;&quot;&quot;<br>    def __init__(self, n_feats):<br>        super(CNNLayerNorm, self).__init__()<br>        self.layer_norm = nn.LayerNorm(n_feats)<br><br>    def forward(self, x):<br>        # x (batch, channel, feature, time)<br>        x = x.transpose(2, 3).contiguous() # (batch, channel, time, feature)<br>        x = self.layer_norm(x)<br>        return x.transpose(2, 3).contiguous() # (batch, channel, feature, time) <br><br><br>class ResidualCNN(nn.Module):<br>    &quot;&quot;&quot;Residual CNN inspired by https://arxiv.org/pdf/1603.05027.pdf<br>        except with layer norm instead of batch norm<br>    &quot;&quot;&quot;<br>    def __init__(self, in_channels, out_channels, kernel, stride, dropout, n_feats):<br>        super(ResidualCNN, self).__init__()<br><br>        self.cnn1 = nn.Conv2d(in_channels, out_channels, kernel, stride, padding=kernel//2)<br>        self.cnn2 = nn.Conv2d(out_channels, out_channels, kernel, stride, padding=kernel//2)<br>        self.dropout1 = nn.Dropout(dropout)<br>        self.dropout2 = nn.Dropout(dropout)<br>        self.layer_norm1 = CNNLayerNorm(n_feats)<br>        self.layer_norm2 = CNNLayerNorm(n_feats)<br><br>    def forward(self, x):<br>        residual = x  # (batch, channel, feature, time)<br>        x = self.layer_norm1(x)<br>        x = F.gelu(x)<br>        x = self.dropout1(x)<br>        x = self.cnn1(x)<br>        x = self.layer_norm2(x)<br>        x = F.gelu(x)<br>        x = self.dropout2(x)<br>        x = self.cnn2(x)<br>        x += residual<br>        return x # (batch, channel, feature, time)<br><br><br>class BidirectionalGRU(nn.Module):<br><br>    def __init__(self, rnn_dim, hidden_size, dropout, batch_first):<br>        super(BidirectionalGRU, self).__init__()<br><br>        self.BiGRU = nn.GRU(<br>            input_size=rnn_dim, hidden_size=hidden_size,<br>            num_layers=1, batch_first=batch_first, bidirectional=True)<br>        self.layer_norm = nn.LayerNorm(rnn_dim)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, x):<br>        x = self.layer_norm(x)<br>        x = F.gelu(x)<br>        x, _ = self.BiGRU(x)<br>        x = self.dropout(x)<br>        return x<br><br><br>class SpeechRecognitionModel(nn.Module):<br>    &quot;&quot;&quot;Speech Recognition Model Inspired by DeepSpeech 2&quot;&quot;&quot;<br><br>    def __init__(self, n_cnn_layers, n_rnn_layers, rnn_dim, n_class, n_feats, stride=2, dropout=0.1):<br>        super(SpeechRecognitionModel, self).__init__()<br>        n_feats = n_feats//2<br>        self.cnn = nn.Conv2d(1, 32, 3, stride=stride, padding=3//2)  # cnn for extracting heirachal features<br><br>        # n residual cnn layers with filter size of 32<br>        self.rescnn_layers = nn.Sequential(*[<br>            ResidualCNN(32, 32, kernel=3, stride=1, dropout=dropout, n_feats=n_feats) <br>            for _ in range(n_cnn_layers)<br>        ])<br>        self.fully_connected = nn.Linear(n_feats*32, rnn_dim)<br>        self.birnn_layers = nn.Sequential(*[<br>            BidirectionalGRU(rnn_dim=rnn_dim if i==0 else rnn_dim*2,<br>                             hidden_size=rnn_dim, dropout=dropout, batch_first=i==0)<br>            for i in range(n_rnn_layers)<br>        ])<br>        self.classifier = nn.Sequential(<br>            nn.Linear(rnn_dim*2, rnn_dim),  # birnn returns rnn_dim*2<br>            nn.GELU(),<br>            nn.Dropout(dropout),<br>            nn.Linear(rnn_dim, n_class)<br>        )<br><br>    def forward(self, x):<br>        x = self.cnn(x)<br>        x = self.rescnn_layers(x)<br>        sizes = x.size()<br>        x = x.view(sizes[0], sizes[1] * sizes[2], sizes[3])  # (batch, feature, time)<br>        x = x.transpose(1, 2) # (batch, time, feature)<br>        x = self.fully_connected(x)<br>        x = self.birnn_layers(x)<br>        x = self.classifier(x)<br>        return x</pre><h3>Picking the Right Optimizer and Scheduler — AdamW with Super Convergence</h3><p>The optimizer and learning rate schedule plays a very important role in getting our model to converge to the best point. Picking the right optimizer and scheduler can also save you compute time, and help your model generalize better to real-world use cases. For our model, we’ll be using <strong>AdamW</strong> with the <strong>One Cycle Learning Rate Scheduler</strong>. <strong>Adam</strong> is a widely used optimizer that helps your model converge more quickly, therefore, saving compute time, but has been notorious for not generalizing as well as <strong>Stochastic Gradient Descent</strong> AKA <strong>SGD</strong>.</p><p><strong>AdamW</strong> was first introduced in <a href="https://arxiv.org/abs/1711.05101">Decoupled Weight Decay Regularization</a>, and is considered a “fix” to <strong>Adam</strong>. The paper pointed out that the original <strong>Adam</strong> algorithm has a wrong implementation of weight decay, which <strong>AdamW</strong> attempts to fix. This fix helps with <strong>Adam</strong> ‘s generalization problem.</p><p>The <strong>One Cycle Learning Rate Scheduler</strong> was first introduced in the paper <a href="https://arxiv.org/abs/1708.07120">Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates</a>. This paper shows that you can train neural networks an order of magnitude faster, while keeping their generalizable abilities, using a simple trick. You start with a low learning rate, which warms up to a large maximum learning rate, then decays linearly to the same point of where you originally started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*anuZx5bSyspmGWGlp9ietw.jpeg" /></figure><p>Because the maximum learning rate is magnitudes higher than the lowest, you also gain some regularization benefits which helps your model generalize better if you have a smaller set of data.</p><p>With PyTorch, these two methods are already part of the package.</p><pre>optimizer = optim.AdamW(model.parameters(), hparams[&#39;learning_rate&#39;])<br>scheduler = optim.lr_scheduler.OneCycleLR(optimizer,<br>	max_lr=hparams[&#39;learning_rate&#39;],<br>	steps_per_epoch=int(len(train_loader)),<br>	epochs=hparams[&#39;epochs&#39;],<br>	anneal_strategy=&#39;linear&#39;)</pre><h3>The CTC Loss Function — Aligning Audio to Transcript</h3><p>Our model will be trained to predict the probability distribution of all characters in the alphabet for each frame (ie, timestep) in the spectrogram we feed into the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*GM8mOnGV8zz5OVt4QL9PjQ.png" /><figcaption>Image taken from distill.pub</figcaption></figure><p>Traditional speech recognition models would require you to align the transcript text to the audio before training, and the model would be trained to predict specific labels at specific frames.</p><p>The innovation of the CTC loss function is that it allows us to skip this step. Our model will learn to align the transcript itself during training. The key to this is the “blank” label introduced by CTC, which gives the model the ability to say that a certain audio frame did not produce a character. You can see a more detailed explanation of CTC and how it works from <a href="https://distill.pub/2017/ctc/">this excellent post</a>.</p><p>The CTC loss function is also built into PyTorch.</p><pre>criterion = nn.CTCLoss(blank=28).to(device)</pre><h3>Evaluating Your Speech Model</h3><p>When Evaluating your speech recognition model, the industry standard is using the Word Error Rate (WER) as the metric. The Word Error Rate does exactly what it says — it takes the transcription your model outputs, and the true transcription, and measures the error between them. You can see how that’s implemented <a href="https://colab.research.google.com/drive/1IPpwx4rX32rqHKpLz7dc8sOKspUa-YKO">here</a>. Another useful metric is called the Character Error Rate (CER). The CER measures the error of the characters between the model’s output and the true labels. These metrics are helpful to measure how well your model performs.</p><p>For this tutorial, we’ll use a “greedy” decoding method to process our model’s output into characters that can be combined to create the transcript. A “greedy” decoder takes in the model output, which is a softmax probability matrix of characters, and for each time step (spectrogram frame), it chooses the label with the highest probability. If the label is a blank label, we remove it from the final transcript.</p><pre>def GreedyDecoder(output, labels, label_lengths, blank_label=28, collapse_repeated=True):<br>    arg_maxes = torch.argmax(output, dim=2)<br>    decodes = []<br>    targets = []<br>    for i, args in enumerate(arg_maxes):<br>        decode = []<br>        targets.append(text_transform.int_to_text(labels[i][:label_lengths[i]].tolist()))<br>        for j, index in enumerate(args):<br>            if index != blank_label:<br>                if collapse_repeated and j != 0 and index == args[j -1]:<br>                    continue<br>                decode.append(index.item())<br>        decodes.append(text_transform.int_to_text(decode))<br>    return decodes, targets</pre><h3>Training and Monitoring Your Experiments Using Comet.ml</h3><p><a href="https://www.comet.ml/site/">Comet.ml</a> provides a platform that allows deep learning researchers to track, compare, explain, and optimize their experiments and models. Comet.ml has improved our productivity at AssemblyAI and we highly recommend using this platform for teams doing any sort of data science experiments. Comet.ml is super easy to set up. And works with just a few lines of code.</p><pre># initialize experiment object<br>experiment = Experiment(api_key=comet_api_key, project_name=project_name)<br>experiment.set_name(exp_name)<br><br># track metrics<br>experiment.log_metric(&#39;loss&#39;, loss.item())</pre><p><a href="https://www.comet.ml/site/">Comet.ml</a> provides you with a very productive dashboard where you can view and track your model’s progress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47U-COfhiP6wvEkr5Rzepg.png" /></figure><p>You can use Comet to track metrics, code, hyper parameters, your model’s graphs, among many other things! A really handy feature that Comet provides is the ability to compare your experiment among many other experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j926ql-Xkscq6BRmGH19eQ.png" /></figure><p>Comet has a rich feature set that we won’t cover all here, but we highly recommended using it for a productivity and sanity boost.</p><p>Here is the rest of our training script.</p><pre>class IterMeter(object):<br>    &quot;&quot;&quot;keeps track of total iterations&quot;&quot;&quot;<br>    def __init__(self):<br>        self.val = 0<br><br>    def step(self):<br>        self.val += 1<br><br>    def get(self):<br>        return self.val<br><br><br>def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):<br>    model.train()<br>    data_len = len(train_loader.dataset)<br>    with experiment.train():<br>        for batch_idx, _data in enumerate(train_loader):<br>            spectrograms, labels, input_lengths, label_lengths = _data <br>            spectrograms, labels = spectrograms.to(device), labels.to(device)<br><br>            optimizer.zero_grad()<br><br>            output = model(spectrograms)  # (batch, time, n_class)<br>            output = F.log_softmax(output, dim=2)<br>            output = output.transpose(0, 1) # (time, batch, n_class)<br><br>            loss = criterion(output, labels, input_lengths, label_lengths)<br>            loss.backward()<br><br>            experiment.log_metric(&#39;loss&#39;, loss.item(), step=iter_meter.get())<br>            experiment.log_metric(&#39;learning_rate&#39;, scheduler.get_lr(), step=iter_meter.get())<br><br>            optimizer.step()<br>            scheduler.step()<br>            iter_meter.step()<br>            if batch_idx % 100 == 0 or batch_idx == data_len:<br>                print(&#39;Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}&#39;.format(<br>                    epoch, batch_idx * len(spectrograms), data_len,<br>                    100. * batch_idx / len(train_loader), loss.item()))<br><br><br>def test(model, device, test_loader, criterion, epoch, iter_meter, experiment):<br>    print(&#39;\nevaluating…&#39;)<br>    model.eval()<br>    test_loss = 0<br>    test_cer, test_wer = [], []<br>    with experiment.test():<br>        with torch.no_grad():<br>            for I, _data in enumerate(test_loader):<br>                spectrograms, labels, input_lengths, label_lengths = _data <br>                spectrograms, labels = spectrograms.to(device), labels.to(device)<br><br>                output = model(spectrograms)  # (batch, time, n_class)<br>                output = F.log_softmax(output, dim=2)<br>                output = output.transpose(0, 1) # (time, batch, n_class)<br><br>                loss = criterion(output, labels, input_lengths, label_lengths)<br>                test_loss += loss.item() / len(test_loader)<br><br>                decoded_preds, decoded_targets = GreedyDecoder(output.transpose(0, 1), labels, label_lengths)<br>                for j in range(len(decoded_preds)):<br>                    test_cer.append(cer(decoded_targets[j], decoded_preds[j]))<br>                    test_wer.append(wer(decoded_targets[j], decoded_preds[j]))<br><br><br>    avg_cer = sum(test_cer)/len(test_cer)<br>    avg_wer = sum(test_wer)/len(test_wer)<br>    experiment.log_metric(&#39;test_loss&#39;, test_loss, step=iter_meter.get())<br>    experiment.log_metric(&#39;cer&#39;, avg_cer, step=iter_meter.get())<br>    experiment.log_metric(&#39;wer&#39;, avg_wer, step=iter_meter.get())<br><br>    print(&#39;Test set: Average loss: {:.4f}, Average CER: {:4f} Average WER: {:.4f}\n&#39;.format(test_loss, avg_cer, avg_wer))<br><br><br>def main(learning_rate=5e-4, batch_size=20, epochs=10,<br>        train_url=&quot;train-clean-100&quot;, test_url=&quot;test-clean&quot;,<br>        experiment=Experiment(api_key=&#39;dummy_key&#39;, disabled=True)):<br><br>    hparams = {<br>        &quot;n_cnn_layers&quot;: 3,<br>        &quot;n_rnn_layers&quot;: 5,<br>        &quot;rnn_dim&quot;: 512,<br>        &quot;n_class&quot;: 29,<br>        &quot;n_feats&quot;: 128,<br>        &quot;stride&quot;: 2,<br>        &quot;dropout&quot;: 0.1,<br>        &quot;learning_rate&quot;: learning_rate,<br>        &quot;batch_size&quot;: batch_size,<br>        &quot;epochs&quot;: epochs<br>    }<br><br>    experiment.log_parameters(hparams)<br><br>    use_cuda = torch.cuda.is_available()<br>    torch.manual_seed(7)<br>    device = torch.device(&quot;cuda&quot; if use_cuda else &quot;cpu&quot;)<br><br>    if not os.path.isdir(&quot;./data&quot;):<br>        os.makedirs(&quot;./data&quot;)<br><br>    train_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./data&quot;, url=train_url, download=True)<br>    test_dataset = torchaudio.datasets.LIBRISPEECH(&quot;./data&quot;, url=test_url, download=True)<br><br>    kwargs = {&#39;num_workers&#39;: 1, &#39;pin_memory&#39;: True} if use_cuda else {}<br>    train_loader = data.DataLoader(dataset=train_dataset,<br>                                batch_size=hparams[&#39;batch_size&#39;],<br>                                shuffle=True,<br>                                collate_fn=lambda x: data_processing(x, &#39;train&#39;),<br>                                **kwargs)<br>    test_loader = data.DataLoader(dataset=test_dataset,<br>                                batch_size=hparams[&#39;batch_size&#39;],<br>                                shuffle=False,<br>                                collate_fn=lambda x: data_processing(x, &#39;valid&#39;),<br>                                **kwargs)<br><br>    model = SpeechRecognitionModel(<br>        hparams[&#39;n_cnn_layers&#39;], hparams[&#39;n_rnn_layers&#39;], hparams[&#39;rnn_dim&#39;],<br>        hparams[&#39;n_class&#39;], hparams[&#39;n_feats&#39;], hparams[&#39;stride&#39;], hparams[&#39;dropout&#39;]<br>        ).to(device)<br><br>    print(model)<br>    print(&#39;Num Model Parameters&#39;, sum([param.nelement() for param in model.parameters()]))<br><br>    optimizer = optim.AdamW(model.parameters(), hparams[&#39;learning_rate&#39;])<br>    criterion = nn.CTCLoss(blank=28).to(device)<br>    scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=hparams[&#39;learning_rate&#39;], <br>                                            steps_per_epoch=int(len(train_loader)),<br>                                            epochs=hparams[&#39;epochs&#39;],<br>                                            anneal_strategy=&#39;linear&#39;)<br><br>    iter_meter = IterMeter()<br>    for epoch in range(1, epochs + 1):<br>        train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)<br>        test(model, device, test_loader, criterion, epoch, iter_meter, experiment)</pre><p>The<strong> train</strong> function trains the model on a full epoch of data. The <strong>test</strong> function evaluates the model on test data after every epoch. It gets the<strong> test_loss</strong> as well as the <strong>cer</strong> and <strong>wer</strong> of the model. You can start running the training script right now with GPU support in the <a href="https://colab.research.google.com/drive/1IPpwx4rX32rqHKpLz7dc8sOKspUa-YKO">Google Colaboratory</a>.</p><h3>How to Improve Accuracy</h3><p>Speech Recognition Requires a ton of data and a ton of compute resources. The example laid out is trained on a subset of LibriSpeech (100 hours of audio) and a single GPU. To get state of the art results you’ll need to do distributed training on thousands of hours of data, on tens of GPU’s spread out across many machines.</p><p>Another way to get a big accuracy improvement is to decode the CTC probability matrix using a Language Model and the CTC beam search algorithm. CTC type models are very dependent on this decoding process to get good results. Luckily there is a handy <a href="https://github.com/parlance/ctcdecode">open source library</a> that allows you to do that.</p><p>This tutorial was made to be more accessible so it’s a relatively small model (23 million Parameters) compared to something like BERT (340 million Parameters). It seems to be the larger you can get your network, the better it performs, although there are diminishing returns. A larger model equating to better performance is not always the case though, as proven by OpenAI’s research <a href="https://openai.com/assets/deep-double-descent/">Deep Double Descent</a>.</p><p>This model has 3 residual CNN layers and 5 Bidirectional GRU layers which should allow you to train a reasonable batch size on a single GPU with at least 11GB of memory. You can tweak some of the hyper parameters in the main function to reduce or increase the model size for your use case and compute availability.</p><h3>Latest Advancements In Speech Recognition with Deep Learning</h3><p>Deep learning is a fast-moving field. It seems like you can’t go a week without some new technique getting state of the art results. Here are a few of things worth exploring int the world of speech recognition.</p><h4>Transformers</h4><p>Transformers have taken the Natural Language Processing world by storm! First Introduced in the paper <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a>, transformers have been taking and modified to beat pretty much all existing NLP task dethroning RNN’s type architectures. The Transformer’s ability to see the full context of sequence data is transferable to speech as well.</p><h4>Unsupervised Pre-training</h4><p>If you follow deep learning closely you’ve probably heard of BERT, GPT, and GPT2. These Transformer models have first pertained on a language modeling task with unlabeled text data, and fine-tuned on a wide array of NLP task and get state of the art results! During pre-training, the model learns something fundamental on the statistics of language and uses that power to excel at other tasks. We believe this technique has great promises on speech data as well.</p><h4>Word Piece Models</h4><p>Our model defined above output characters. Some benefits to that are the model doesn’t have to worry about out of vocabulary words when running inference on speech. So for the word <strong>c h a t</strong> each character has is its own label. The downside to using characters are inefficiency and the model being prone to more errors because you’re predicting one character at a time. Using the whole word as labels have been explored, to some degree of success. Using this method, the entire word <strong>chat</strong> would be the label. But using whole words, you would have to keep an index of all possible vocabularies to make a prediction, which is memory inefficient with the possibility of running into out of vocabulary words during prediction. The sweet spot would be using word piece or sub-word units as labels. Instead of characters for the individual label, you can chop up the words into sub-word units, and use those as labels, i.e. <strong>ch at</strong>. This solves the out of vocabulary issue, and is much more efficient, as it needs fewer steps to decode then using characters, and without the need to have an index of all possible words. Word pieces have been used successfully with many NLP models, like BERT and would work natural with speech recognition problems as well.</p><p><em>Originally published at </em><a href="https://www.comet.ml/site/customer-case-study-building-an-end-to-end-speech-recognition-model-in-pytorch-with-assemblyai/"><em>https://www.comet.ml</em></a><em> on May 18, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cea744401fe8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comet-ml/customer-case-study-building-an-end-to-end-speech-recognition-model-in-pytorch-with-assemblyai-cea744401fe8">Customer Case Study: Building an end-to-end Speech Recognition model in PyTorch with AssemblyAI</a> was originally published in <a href="https://medium.com/comet-ml">Comet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predictive Early Stopping — A Meta Learning Approach]]></title>
            <link>https://medium.com/data-science/predictive-early-stopping-a-meta-learning-approach-90561f0e9454?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/90561f0e9454</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Wed, 22 Apr 2020 15:30:50 GMT</pubDate>
            <atom:updated>2020-04-22T15:30:50.916Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predictive Early Stopping — A Meta Learning Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQFvzNy0v7QFe7SuYAIh2g.png" /></figure><p><em>Author: Dhruv Nair, Data Scientist, Comet.ml</em></p><h4>Introduction</h4><p>Predictive Early Stopping is a state-of-the-art approach for speeding up model training and hyperparameter optimization. Our benchmarking studies have shown that Predictive Early Stopping can speed up model training <em>by up to 30% independent</em> of the underlying infrastructure.</p><p>We build on insights gathered from projects such as Learning Curve Extrapolation, Hyperband, and Median Stopping, in order to create a predictive model that can estimate the convergence value of a loss curve.</p><p>Comet is able to leverage model data, such as hyperparameters and loss curves, from <em>over </em>two million models in the <em>public</em> section of its platform to create a model whose predictions generalize across hyperparameters and model architectures.</p><p>In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs. In addition to predicting the convergence value, our Predictive Early Stopping product provides an estimate of the probability that the current model will outperform the best model result seen in the current training sweep.</p><p><em>In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs.</em></p><p>These predictions allow us to terminate the training of underperforming models, so that the search process is spent evaluating only the most promising candidates.</p><h4>Benchmarking:</h4><p>We tested our Predictive Early Stopping method in three different settings:</p><ol><li>A hyperparameter search that optimizes the parameters of a function that acts as a surrogate for a neural network</li><li>A hyperparameter search to optimize a 6 layer CNN on CIFAR10 using the SMAC optimizer, with and without predictive early stopping.</li><li>A hyperparameter search to optimize the same 6 layer CNN using Random Search with Hyperband vs Random Search with Predictive Early Stopping.</li></ol><h4>Results for the Surrogate Function:</h4><p>In our first test, we set up an exponentially decaying function as our surrogate for a Neural Network. We ran this surrogate model for 20 steps and determined the optimal values for the parameters of this function using Comet’s Bayesian Optimizer and Predictive Early Stopping. During the hyperparameter search, we observed that suboptimal models were not allowed to train the full 20 steps. <strong>Figure 1</strong>, illustrates our stopping mechanism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*p8ipe27ujbKoVwMWJ0EGrQ.jpeg" /></figure><h4>Results for the CNN model with SMAC:</h4><p>Our benchmarking test for the CNN model was set up in the following way:</p><p>We used the SMAC optimizer to estimate the following hyperparameters in a 6-Layer CNN model. The model hyperparameters and architecture were based on AlexNet.</p><pre>{<br>    &quot;learning_rate&quot;:{<br>        &quot;type&quot;:&quot;loguniform&quot;,<br>        &quot;value&quot;:[0.0000001, 0.01]<br>    },<br>    &quot;learning_rate_decay&quot;:{<br>        &quot;type&quot;: &quot;uniform&quot;,<br>        &quot;value&quot;:[0.000001, 0.001]<br>    },<br>    &quot;weight_decay&quot;: {<br>        &quot;type&quot;: &quot;loguniform&quot;,<br>        &quot;value&quot;: [0.0000005, 0.005]<br>    }<br>}</pre><p>We ran 8 trials of the optimizer, with and without Predictive Early Stopping. Each optimizer trial is given a 6 hour budget to evaluate as many configurations as possible. Each hyperparameter configuration was allowed to train for a maximum of 100 epochs, and the validation set was evaluated at the end of every epoch. The validation loss was used by our Predictive Early Stopping model to determine whether or not to terminate a hyperparameter configuration.</p><p>At the end of all the trials, we determined the mean value of the test loss across all trials as a function of the total number of epochs in the hyperparameter sweep that are required to achieve that loss value.</p><p>We can see in <strong>Figure 2</strong>, that using Predictive Early Stopping allows SMAC to get to a comparable loss value, almost 300 epochs faster. This is a 25% decrease in the amount of time spent doing hyperparameter optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByDaVQ4lCZXIoFF8oMYFyA.png" /></figure><p>We also divided the hyperparameter configurations into quantiles, based on the final validation loss. We then calculated the average amount of epochs that the Optimizer spent in each quantile across all trials.</p><p>In <strong>Figure 3</strong>, we see that both SMAC and Predictive-Early-Stopping-SMAC spend roughly the same amount of time evaluating the top 25% and top 50% of configurations. However, Predictive Early Stopping spends 30 fewer epochs training models in the bottom 50% of results, and 20 fewer epochs in bottom 25%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjGNOcpGgGwp6jutjS3N_A.png" /></figure><p>In <strong>Figure 4</strong>, we see sample loss curves from a hyperparameter sweep with Predictive Early Stopping. Suboptimal configurations are stopped well before the total number of allowed training steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hEo1RtcwZ9yrFmJ5-jZ-Lg.jpeg" /></figure><h3>Results for the CNN model with Hyperband:</h3><p>We set up the test for Hyperband in a similar manner to SMAC. We specifically used the <a href="https://optuna.readthedocs.io/en/latest/_modules/optuna/pruners/successive_halving.html#SuccessiveHalvingPruner">Asynchronous Successive Halving Pruner</a> implemented in <a href="https://optuna.org/">Optuna</a>. We can think of this as Hyperband with a single bracket. We selected 120 hyperparameter configurations at random. Each configuration was allotted a minimum resource of 10 epochs and allowed to train for a maximum of 100 epochs. This leads to a maximum training budget of 12000 epochs.</p><p>At every evaluation point, the number of configurations is reduced by a factor of N based on the worst performing validation losses. In our experiments, we evaluated Hyperband with N values of 2, 4 and 8.</p><p>For the Predictive Early Stopping, we tested each configuration with different values of an interval parameter. Configurations were evaluated every 10, 15, and 20 epochs. At every evaluation we estimated the probability that the current configuration will be better than the best configuration seen so far. If this probability is less than a threshold, in our case 90%, we terminate the current configuration. Threshold and Interval are both configurable hyperparameters for Predictive Early Stopping.</p><p>We then determine the best validation loss achieved in the hyperparameter sweep and the number of epochs remaining in the budget after the sweep.</p><p>In <strong>Figure 5a</strong>, we see that all approaches find the same best value for the validation loss, however, Predictive Early Stopping is able to evaluate the configurations using just 15% of the total budget, compared to Hyperband which uses 25%. This is a 10% improvement in speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIseFmsu7aH49C2M_yyG4A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_zHrp06YmcnpS0IeKZR9uA.jpeg" /></figure><h4>Conclusion</h4><p>Predictive Early Stopping has very obvious time, energy, and cost saving benefits. Wasted compute cycles aren’t good for the environment, or a researcher’s budget.</p><p>The <a href="https://allenai.org/">Allen Institute for AI</a> recently published a <a href="https://arxiv.org/pdf/1907.10597.pdf">report</a> on the rising computational costs of training machine learning models, and how these increasingly large energy requirements adversely affect the environment. The paper states that current advancements in state-of-the-art AI research have largely focussed on metrics such as accuracy, or error, at the cost of being environmentally unfriendly. They call this paradigm, Red AI. To counter the prevalence of Red AI research, they propose a shift towards AI research that places an emphasis on computational efficiency: Green AI.</p><p>The paper recommends tracking the efficiency of an AI algorithm based on the total number of floating point operations (FPO’s) required to generate a result. The total number of FPO’s directly correlates with the number of hyperparameter configurations that are evaluated during tuning, as well as the number of training iterations spent on each configuration.</p><p>We hope our efforts with Predictive Early Stopping contribute towards increasing the computational efficiency of the hyperparameter search process. We see this tool as a way to lower some of the monetary barriers associated with AI research, and as a step towards adopting Green AI practices.</p><p>In our next post, we will describe applying predictive early stopping in individual runs that do not belong to a larger parameter search.</p><p><em>*Predictive Early Stopping is available as an add-on with the purchase of Comet Teams or Comet Enterprise. Patent protection is being sought on Predictive Early Stopping via one or more pending patent applications.</em></p><p>Learn more and <a href="https://www.comet.ml/site/what-is-meta-ml/">sign-up here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90561f0e9454" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predictive-early-stopping-a-meta-learning-approach-90561f0e9454">Predictive Early Stopping — A Meta Learning Approach</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predictive Early Stopping — A Meta Learning Approach]]></title>
            <link>https://medium.com/comet-ml/predictive-early-stopping-a-meta-learning-approach-99bf4bc6921d?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/99bf4bc6921d</guid>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 20 Apr 2020 04:55:06 GMT</pubDate>
            <atom:updated>2020-04-22T13:37:25.337Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predictive Early Stopping — A Meta Learning Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQFvzNy0v7QFe7SuYAIh2g.png" /></figure><p>Author: Dhruv Nair, Data Scientist, Comet.ml</p><h3>Introduction</h3><p>Predictive Early Stopping is a state-of-the-art approach for speeding up model training and hyperparameter optimization. Our benchmarking studies have shown that Predictive Early Stopping can speed up model training <em>by up to 30% independent</em> of the underlying infrastructure.</p><p>We build on insights gathered from projects such as Learning Curve Extrapolation, Hyperband, and Median Stopping, in order to create a predictive model that can estimate the convergence value of a loss curve.</p><p>Comet is able to leverage model data, such as hyperparameters and loss curves, from <em>over </em>two million models in the <em>public</em> section of its platform to create a model whose predictions generalize across hyperparameters and model architectures.</p><p>In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs. In addition to predicting the convergence value, our Predictive Early Stopping product provides an estimate of the probability that the current model will outperform the best model result seen in the current training sweep.</p><p><em>In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs.</em></p><p>These predictions allow us to terminate the training of underperforming models, so that the search process is spent evaluating only the most promising candidates.</p><h3>Benchmarking:</h3><p>We tested our Predictive Early Stopping method in three different settings:</p><ol><li>A hyperparameter search that optimizes the parameters of a function that acts as a surrogate for a neural network</li><li>A hyperparameter search to optimize a 6 layer CNN on CIFAR10 using the SMAC optimizer, with and without predictive early stopping.</li><li>A hyperparameter search to optimize the same 6 layer CNN using Random Search with Hyperband vs Random Search with Predictive Early Stopping.</li></ol><h3>Results for the Surrogate Function:</h3><p>In our first test, we setup an exponentially decaying function as our surrogate for a Neural Network. We ran this surrogate model for 20 steps and determined the optimal values for the parameters of this function using Comet’s Bayesian Optimizer and Predictive Early Stopping.During the hyperparameter search we observed that suboptimal models were not allowed to train the full 20 steps. Figure 1, illustrates our stopping mechanism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*p8ipe27ujbKoVwMWJ0EGrQ.jpeg" /><figcaption>Figure 1. Surrogate Model Loss Curves</figcaption></figure><h3>Results for the CNN model with SMAC:</h3><p>Our benchmarking test for the CNN model was setup in the following way:</p><p>We used the SMAC optimizer to estimate the following hyperparameters in a 6-Layer CNN model. The model hyperparameters, and architecture were based on AlexNet.</p><pre>{<br>    &quot;learning_rate&quot;:{<br>        &quot;type&quot;:&quot;loguniform&quot;,<br>        &quot;value&quot;:[0.0000001, 0.01]<br>    },<br>    &quot;learning_rate_decay&quot;:{<br>        &quot;type&quot;: &quot;uniform&quot;,<br>        &quot;value&quot;:[0.000001, 0.001]<br>    },<br>    &quot;weight_decay&quot;: {<br>        &quot;type&quot;: &quot;loguniform&quot;,<br>        &quot;value&quot;: [0.0000005, 0.005]<br>    }<br>}</pre><p>We ran 8 trials of the optimizer, with and without Predictive Early Stopping. Each optimizer trial is given a 6 hour budget to evaluate as many configurations as possible. Each hyperparameter configuration was allowed to train for a maximum of 100 epochs, and the validation set was evaluated at the end of every epoch. The validation loss was used by our Predictive Early Stopping model to determine whether or not to terminate a hyperparameter configuration.</p><p>At the end of all the trials, we determined the mean value of the test loss across all trials as a function of the total number of epochs in the hyperparameter sweep that are required to achieve that loss value.</p><p>We can see in <strong>Figure 2</strong>, that using Predictive Early Stopping allows SMAC to get to a comparable loss value, almost 300 epochs faster. This is a 25% decrease in the amount of time spent doing hyperparameter optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByDaVQ4lCZXIoFF8oMYFyA.png" /><figcaption>Figure 2. Mean Test Loss across trials vs Total Number of Epochs</figcaption></figure><p>We also divided the hyperparameter configurations into quantiles, based on the final validation loss. We then calculated the average amount of epochs that the Optimizer spent in each quantile across all trials.</p><p>In <strong>Figure 3</strong>, we see that both SMAC and Predictive-Early-Stopping-SMAC spend roughly the same amount of time evaluating the top 25% and top 50% of configurations. However, Predictive Early Stopping spends 30 fewer epochs training models in the bottom 50% of results, and 20 fewer epochs in bottom 25% .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjGNOcpGgGwp6jutjS3N_A.png" /></figure><p>In <strong>Figure 4</strong>, we see sample loss curves from a hyperparameter sweep with Predictive Early Stopping. Suboptimal configurations are stopped well before the total number of allowed training steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hEo1RtcwZ9yrFmJ5-jZ-Lg.jpeg" /><figcaption>Figure 4. Sample validation loss curves from a hyperparameter sweep.</figcaption></figure><h3>Results for the CNN model with Hyperband:</h3><p>We setup the test for Hyperband in a similar manner to SMAC. We specifically used the <a href="https://optuna.readthedocs.io/en/latest/_modules/optuna/pruners/successive_halving.html#SuccessiveHalvingPruner">Asynchronous Successive Halving Pruner</a> implemented in <a href="https://optuna.org/">Optuna</a>. We can think of this as Hyperband with a single bracket. We selected 120 hyperparameter configurations at random. Each configuration was allotted a minimum resource of <strong>10 epochs, </strong>and allowed to train for a maximum of <strong>100 epochs. </strong>This leads to a maximum training budget of <strong>12000 epochs.</strong></p><p>At every evaluation point, the number of configurations are reduced by a factor of <strong>N </strong>based on the worst performing validation losses <strong>. </strong>In our experiments we evaluated Hyperband with <strong>N values of 2, 4 and 8.</strong></p><p>For the Predictive Early Stopping, we tested each configuration with different values of an <strong>interval</strong> parameter. Configurations were evaluated every 10, 15, and 20 epochs. At every evaluation we estimated the probability that the current configuration will be better than the best configuration seen so far. If this probability is less than a <strong>threshold, in our case 90%, </strong>we terminate the current configuration. <strong>Threshold</strong>, and <strong>Interval</strong> are both configurable hyperparameters for Predictive Early Stopping.</p><p>We then determine the best validation loss achieved in the hyperparameter sweep, and the number of epochs remaining in the budget after the sweep.</p><p>In <strong>Figure 5a, we see that all approaches find the same best value for the validation loss, </strong>however, Predictive Early Stopping is able to evaluate the configurations using just <strong>15% </strong>of the total budget, compared to Hyperband which uses <strong>25%. This is a 10%</strong> <strong>improvement in speed.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIseFmsu7aH49C2M_yyG4A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_zHrp06YmcnpS0IeKZR9uA.jpeg" /><figcaption>Figure 5. Predictive Early Stopping vs Hyperband a) Best Value of Validation Loss b) Number of Epochs left in the Training Budget</figcaption></figure><h3>Conclusion</h3><p>Predictive Early Stopping has very obvious time, energy, and cost saving benefits. Wasted compute cycles aren’t good for the environment, or a researcher’s budget.</p><p>The <a href="https://allenai.org/">Allen Institute for AI</a> recently published a <a href="https://arxiv.org/pdf/1907.10597.pdf">report</a> on the rising computational costs of training machine learning models, and how these increasingly large energy requirements adversely affect the environment. The paper states that current advancements in state of the art AI research have largely focussed on metrics such as accuracy, or error, at the cost of being environmentally unfriendly. They call this paradigm, Red AI. To counter the prevalence of Red AI research, they propose a shift towards AI research that places an emphasis on computational efficiency: Green AI.</p><p>The paper recommends tracking the efficiency of an AI algorithm based on the total number of floating point operations (FPO’s) required to generate a result. The total number of FPO’s directly correlates with the number of hyperparameter configurations that are evaluated during tuning, as well as the number of training iterations spent on each configuration.</p><p>We hope our efforts with Predictive Early Stopping contribute towards increasing the computational efficiency of the hyperparameter search process. We see the this tool as a way to lower some of the monetary barriers associated with AI research, and as a step towards adopting Green AI practices.</p><p>In our next post we will describe applying predictive early stopping in individual runs that do not belong to a larger parameter search.</p><p><em>*Predictive Early Stopping is available as an add-on with the purchase of Comet Teams or Comet Enterprise. Patent protection is being sought on Predictive Early Stopping via one or more pending patent applications.</em><strong>Learn more and </strong><a href="https://www.comet.ml/site/what-is-meta-ml/"><strong>sign-up here</strong></a><strong>.</strong></p><p><em>Originally published at </em><a href="https://www.comet.ml/site/predictive-early-stopping/"><em>https://www.comet.ml</em></a><em> on April 20, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99bf4bc6921d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comet-ml/predictive-early-stopping-a-meta-learning-approach-99bf4bc6921d">Predictive Early Stopping — A Meta Learning Approach</a> was originally published in <a href="https://medium.com/comet-ml">Comet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dell EMC and Comet release Kubernetes reference architecture]]></title>
            <link>https://medium.com/data-science/dell-emc-and-comet-release-kubernetes-reference-architecture-1706c1575530?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/1706c1575530</guid>
            <category><![CDATA[reference-architecture]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dell]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 15:08:56 GMT</pubDate>
            <atom:updated>2020-03-30T17:58:09.249Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dell EMC and Comet announce machine learning platform collaboration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7rWvNyDZH14-5zti3uGVg.png" /></figure><p><a href="https://www.delltechnologies.com/en-us/solutions/data-analytics/machine-learning/ready-solutions-for-ai.htm">Dell EMC</a>, a leading provider of full-stack solutions for data science teams, and <a href="http://comet.ml/">Comet</a>, the industry-leading meta machine learning experimentation platform, have released a reference architecture for data science teams looking to harness the power of the Dell EMC infrastructure in tandem with Comet’s meta machine learning platform. <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">Read a whitepaper</a>, authored by Comet and Dell, on using Comet with Dell EMC AI-enabled infrastructure.</p><p>With Dell EMC PowerEdge reference architectures, organizations can deploy artificial intelligence workload-optimized rack systems approximately 6–12 months faster than it would have taken to design the correct configurations and deploy the solution. Organizations can now rely on architectures that are tested and validated by our Dell engineers and know that services are available when and where you need them.</p><p>“Orchestrating and managing the stack for enterprise data science teams is a huge pain point for many of our customers,” said Gideon Mendels, Co-founder/CEO, Comet. “Dell EMC’s Kubeflow and Kubernetes solutions are best-in-class and an excellent choice for any data science team looking to build a robust and scalable ML platform.”</p><p>Comet is a meta machine learning experimentation platform allowing users to automatically track their metrics, hyperparameters, dependencies, GPU utilization, datasets, models, debugging samples, and more. By leveraging Comet, data science teams produce faster research cycles, and more transparent and collaborative data science. Comet also provides a built-in hyperparameter optimization service, interactive confusion matrices, full code tracking and reproducibility features. Comet on-premise installations can support teams of any size, from a single machine to distributed microservices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zc1Yv_eyICz9m9Wrr4H0rg.png" /></figure><p>“This is one of those products that makes you question how you functioned without it. Comet gives data science teams all the automation and productivity features they need, but that they never get around to developing themselves,” says Phil Hummel, Senior Principal Engineer and Distinguished Member of Technical Staff, Dell EMC.</p><p>The Reference architecture utilizes Dell EMC AI-Enabled Kubernetes solution backed by Canonical’s <a href="https://ubuntu.com/kubernetes/docs/overview">Charmed Kubernetes</a> and Kubeflow that complies with all requirements of the AI workload. The solution includes 100% upstream Kubernetes latest code wrapped into easily consumable packages and supported by Canonical.</p><p>Dell EMC and Comet’s reference architecture and data science team user story illustrates how our joint solution can provide teams the tools and infrastructure they need to manage their machine learning workflows — data storage, experimentation and model building, and deployment — while providing flexible and robust deployment options as teams scale. To learn more about Dell EMC’s AI-Enabled Kubernetes Cluster options, read more <a href="https://www.delltechnologies.com/en-us/servers/reference-architectures.htm#accordion0&amp;accordion1&amp;accordion2">here</a> or reach out to Dell EMC directly to <a href="https://www.delltechnologies.com/en-us/servers/reference-architectures.htm#accordion0&amp;accordion1&amp;accordion2">schedule a call</a>. To learn more about Comet’s meta-machine learning platform, read more about Comet <a href="https://www.comet.ml/site/enterprise/">here</a> or contact Comet’s <a href="https://www.comet.ml/site/contact-sales/">customer solutions team</a> for more information or to schedule a demo.</p><p>Read the <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">full whitepaper here</a> that covers (a) using Comet with Dell EMC AI-enabled infrastructure and (b) technical installation instructions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1706c1575530" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dell-emc-and-comet-release-kubernetes-reference-architecture-1706c1575530">Dell EMC and Comet release Kubernetes reference architecture</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Press Release: Dell EMC and Comet announce machine learning platform collaboration]]></title>
            <link>https://medium.com/comet-ml/press-release-dell-emc-and-comet-announce-machine-learning-platform-collaboration-8ba3207fd36d?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ba3207fd36d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dell]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 13:37:05 GMT</pubDate>
            <atom:updated>2020-03-30T17:30:01.526Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7rWvNyDZH14-5zti3uGVg.png" /></figure><p><strong>NEW YORK, NY -- </strong>, a leading provider of full-stack solutions for data science teams, and , the industry-leading meta machine learning experimentation platform, announced a collaboration with a reference architecture for data science teams looking to harness the power of the Dell EMC infrastructure in tandem with Comet’s meta machine learning platform. <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">Read a whitepaper </a>, authored by Comet and Dell, on using Comet with Dell EMC AI-enabled infrastructure.</p><p>With Dell EMC PowerEdge reference architectures, organizations can deploy artificial intelligence workload-optimized rack systems approximately 6–12 months faster than it would have taken to design the correct configurations and deploy the solution. Organizations can now rely on architectures that are tested and validated by our Dell engineers and know that services are available when and where you need them.</p><p>“Orchestrating and managing the stack for enterprise data science teams is a huge pain point for many of our customers,” said Gideon Mendels, Co-founder/CEO, Comet. “Dell EMC’s Kubeflow and Kubernetes solutions are best-in-class and an excellent choice for any data science team looking to build a robust and scalable ML platform.”</p><p>Comet is a meta machine learning experimentation platform allowing users to automatically track their metrics, hyperparameters, dependencies, GPU utilization, datasets, models, debugging samples, and more. By leveraging Comet, data science teams produce faster research cycles, and more transparent and collaborative data science. Comet also provides a built-in hyperparameter optimization service, interactive confusion matrices, full code tracking and reproducibility features. Comet on-premise installations can support teams of any size, from a single machine to distributed microservices.</p><p>“This is one of those products that makes you question how you functioned without it. Comet gives data science teams all the automation and productivity features they need, but that they never get around to developing themselves,” says Phil Hummel, Senior Principal Engineer and Distinguished Member of Technical Staff, Dell EMC.</p><p>The Reference architecture utilizes Dell EMC AI-Enabled Kubernetes solution backed by Canonical’s <a href="https://ubuntu.com/kubernetes/docs/overview">Charmed Kubernetes </a>and Kubeflow that complies with all requirements of the AI workload. The solution includes 100% upstream Kubernetes latest code wrapped into easily consumable packages and supported by Canonical.</p><p>Dell EMC and Comet’s reference architecture and data science team user story illustrates how our joint solution can provide teams the tools and infrastructure they need to manage their machine learning workflows-data storage, experimentation and model building, and deployment-while providing flexible and robust deployment options as teams scale. To learn more about Dell EMC’s AI-Enabled Kubernetes Cluster options, read more or reach out to Dell EMC directly to <a href="https://www.delltechnologies.com/en-us/servers/reference-architectures.htm#accordion0&amp;accordion1&amp;accordion2">schedule a call </a>. To learn more about Comet’s meta-machine learning platform, read more about Comet or contact Comet’s <a href="https://www.comet.ml/site/contact-sales/">customer solutions team </a>for more information or to schedule a demo.</p><p>Read the <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">full whitepaper here </a>that covers (a) using Comet with Dell EMC AI-enabled infrastructure and (b) technical installation instructions.</p><p><em>Originally published at </em><a href="https://www.comet.ml/site/dell-emc-and-comet-release-kubernetes-reference-architecture/"><em>https://www.comet.ml</em></a><em> on March 30, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ba3207fd36d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comet-ml/press-release-dell-emc-and-comet-announce-machine-learning-platform-collaboration-8ba3207fd36d">Press Release: Dell EMC and Comet announce machine learning platform collaboration</a> was originally published in <a href="https://medium.com/comet-ml">Comet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Press Release: Dell EMC and Comet release Kubernetes reference architecture]]></title>
            <link>https://medium.com/@ashleybinford/press-release-dell-emc-and-comet-release-kubernetes-reference-architecture-a01be7e0bef3?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/a01be7e0bef3</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[comet]]></category>
            <category><![CDATA[dell]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 13:37:05 GMT</pubDate>
            <atom:updated>2020-03-30T17:21:53.487Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dell EMC and Comet announce machine learning platform collaboration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7rWvNyDZH14-5zti3uGVg.png" /></figure><p><strong>NEW YORK, NY- </strong>, a leading provider of full-stack solutions for data science teams, and , the industry-leading meta machine learning experimentation platform, announced a collaboration with a reference architecture for data science teams looking to harness the power of the Dell EMC infrastructure in tandem with Comet’s meta machine learning platform. <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">Read a whitepaper </a>, authored by Comet and Dell, on using Comet with Dell EMC AI-enabled infrastructure.</p><p>With Dell EMC PowerEdge reference architectures, organizations can deploy artificial intelligence workload-optimized rack systems approximately 6–12 months faster than it would have taken to design the correct configurations and deploy the solution. Organizations can now rely on architectures that are tested and validated by our Dell engineers and know that services are available when and where you need them.</p><p>“Orchestrating and managing the stack for enterprise data science teams is a huge pain point for many of our customers,” said Gideon Mendels, Co-founder/CEO, Comet. “Dell EMC’s Kubeflow and Kubernetes solutions are best-in-class and an excellent choice for any data science team looking to build a robust and scalable ML platform.”</p><p>Comet is a meta machine learning experimentation platform allowing users to automatically track their metrics, hyperparameters, dependencies, GPU utilization, datasets, models, debugging samples, and more. By leveraging Comet, data science teams produce faster research cycles, and more transparent and collaborative data science. Comet also provides a built-in hyperparameter optimization service, interactive confusion matrices, full code tracking and reproducibility features. Comet on-premise installations can support teams of any size, from a single machine to distributed microservices.</p><p>“This is one of those products that makes you question how you functioned without it. Comet gives data science teams all the automation and productivity features they need, but that they never get around to developing themselves,” says Phil Hummel, Senior Principal Engineer and Distinguished Member of Technical Staff, Dell EMC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zc1Yv_eyICz9m9Wrr4H0rg.png" /></figure><p>The Reference architecture utilizes Dell EMC AI-Enabled Kubernetes solution backed by Canonical’s <a href="https://ubuntu.com/kubernetes/docs/overview">Charmed Kubernetes </a>and Kubeflow that complies with all requirements of the AI workload. The solution includes 100% upstream Kubernetes latest code wrapped into easily consumable packages and supported by Canonical.</p><p>Dell EMC and Comet’s reference architecture and data science team user story illustrates how our joint solution can provide teams the tools and infrastructure they need to manage their machine learning workflows-data storage, experimentation and model building, and deployment-while providing flexible and robust deployment options as teams scale. To learn more about Dell EMC’s AI-Enabled Kubernetes Cluster options, read more or reach out to Dell EMC directly to <a href="https://www.delltechnologies.com/en-us/servers/reference-architectures.htm#accordion0&amp;accordion1&amp;accordion2">schedule a call </a>. To learn more about Comet’s meta-machine learning platform, read more about Comet or contact Comet’s <a href="https://www.comet.ml/site/contact-sales/">customer solutions team </a>for more information or to schedule a demo.</p><p>Read the <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">full whitepaper here </a>that covers (a) using Comet with Dell EMC AI-enabled infrastructure and (b) technical installation instructions.</p><p><em>Originally published at </em><a href="https://www.comet.ml/site/dell-emc-and-comet-release-kubernetes-reference-architecture/"><em>https://www.comet.ml</em></a><em> on March 30, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a01be7e0bef3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Press Release: Dell EMC and Comet announce machine learning platform collaboration]]></title>
            <link>https://medium.com/@ashleybinford/press-release-dell-emc-and-comet-announce-machine-learning-platform-collaboration-4529d0c354ca?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/4529d0c354ca</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dell]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 13:37:05 GMT</pubDate>
            <atom:updated>2020-03-30T17:26:51.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7rWvNyDZH14-5zti3uGVg.png" /></figure><p><strong>NEW YORK, NY- </strong>, a leading provider of full-stack solutions for data science teams, and , the industry-leading meta machine learning experimentation platform, announced a collaboration with a reference architecture for data science teams looking to harness the power of the Dell EMC infrastructure in tandem with Comet’s meta machine learning platform. <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">Read a whitepaper </a>, authored by Comet and Dell, on using Comet with Dell EMC AI-enabled infrastructure.</p><p>With Dell EMC PowerEdge reference architectures, organizations can deploy artificial intelligence workload-optimized rack systems approximately 6–12 months faster than it would have taken to design the correct configurations and deploy the solution. Organizations can now rely on architectures that are tested and validated by our Dell engineers and know that services are available when and where you need them.</p><p>“Orchestrating and managing the stack for enterprise data science teams is a huge pain point for many of our customers,” said Gideon Mendels, Co-founder/CEO, Comet. “Dell EMC’s Kubeflow and Kubernetes solutions are best-in-class and an excellent choice for any data science team looking to build a robust and scalable ML platform.”</p><p>Comet is a meta machine learning experimentation platform allowing users to automatically track their metrics, hyperparameters, dependencies, GPU utilization, datasets, models, debugging samples, and more. By leveraging Comet, data science teams produce faster research cycles, and more transparent and collaborative data science. Comet also provides a built-in hyperparameter optimization service, interactive confusion matrices, full code tracking and reproducibility features. Comet on-premise installations can support teams of any size, from a single machine to distributed microservices.</p><p>“This is one of those products that makes you question how you functioned without it. Comet gives data science teams all the automation and productivity features they need, but that they never get around to developing themselves,” says Phil Hummel, Senior Principal Engineer and Distinguished Member of Technical Staff, Dell EMC.</p><p>The Reference architecture utilizes Dell EMC AI-Enabled Kubernetes solution backed by Canonical’s <a href="https://ubuntu.com/kubernetes/docs/overview">Charmed Kubernetes </a>and Kubeflow that complies with all requirements of the AI workload. The solution includes 100% upstream Kubernetes latest code wrapped into easily consumable packages and supported by Canonical.</p><p>Dell EMC and Comet’s reference architecture and data science team user story illustrates how our joint solution can provide teams the tools and infrastructure they need to manage their machine learning workflows-data storage, experimentation and model building, and deployment-while providing flexible and robust deployment options as teams scale. To learn more about Dell EMC’s AI-Enabled Kubernetes Cluster options, read more or reach out to Dell EMC directly to <a href="https://www.delltechnologies.com/en-us/servers/reference-architectures.htm#accordion0&amp;accordion1&amp;accordion2">schedule a call </a>. To learn more about Comet’s meta-machine learning platform, read more about Comet or contact Comet’s <a href="https://www.comet.ml/site/contact-sales/">customer solutions team </a>for more information or to schedule a demo.</p><p>Read the <a href="https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf">full whitepaper here </a>that covers (a) using Comet with Dell EMC AI-enabled infrastructure and (b) technical installation instructions.</p><p><em>Originally published at </em><a href="https://www.comet.ml/site/dell-emc-and-comet-release-kubernetes-reference-architecture/"><em>https://www.comet.ml</em></a><em> on March 30, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4529d0c354ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Make Remote Work Effective for Data Science Teams]]></title>
            <link>https://medium.com/comet-ml/how-to-make-remote-work-effective-for-data-science-teams-4d12bf2c6c08?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d12bf2c6c08</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[remote-work]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Fri, 20 Mar 2020 14:56:41 GMT</pubDate>
            <atom:updated>2020-03-30T13:47:41.358Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was written in collaboration with </em><a href="https://medium.com/@tylerfolkman/e784dee2f208"><em>Tyler Folkman</em></a><em>, Head of AI at Branded Entertainment Network. To read more of Tyler’s writing, check out his </em><a href="https://medium.com/@tylerfolkman"><em>Medium Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*LdOx3v3zn8PCIB-x5PurSg.png" /></figure><p>In 1973, at the height of the OPEC oil crisis and skyrocketing fuel prices, NASA scientist and USC professor Jack Nilles began thinking about ways work could be done without the need for commuting. Nilles’ thought experiments evolved into case studies, numerous books, including <a href="https://www.amazon.com/Telecommunications-Transportation-Tradeoff-Options-Tomorrow/dp/1419667297"><em>The Telecommunications-Transportation Tradeoff </em></a>the original book on telecommuting, as well as dozens of papers, articles and keynote speeches. To this day, Nilles remains one of the principal evangelists for remote work as a viable alternative to a traditional office.</p><p>Today, it would not be entirely hyperbolic to consider Nilles’ early work clairvoyant. The last decade has seen a dramatic rise in companies adopting a remote workforce. A <a href="https://www.flexjobs.com/blog/post/flexjobs-gwa-report-remote-growth/">2019 report </a>compiled by Flexjobs and Global Workplace Analytics found that the number of people telecommuting in the United States increased by 159% from 2005 to 2017. With the rapidly evolving global crisis surrounding COVID-19, forcing more and more companies to adopt strict work-from-home policies, remote work is further cementing itself as a workplace paradigm to be taken seriously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*DI_CXtM5aHzPPnjJ6OMW7A.png" /></figure><p>The data science industry is no exception. Large computer, IT and software development industries are among the <a href="https://www.flexjobs.com/blog/post/100-top-companies-with-remote-jobs-2020/">fastest-growing </a>with respect to the share of employees working on a full or partially-remote basis. At , we see many data science teams going remote in some capacity (we, too, have several team members working on a fully remote basis). The perks of remote work are well-documented: a <a href="https://lp.buffer.com/state-of-remote-work-2020">joint report </a>published by Buffer and Angellist-a survey of more than 3,500 remote tech workers-found that a flexible schedule, the ability to work from anywhere, and not having a commute were among the most attractive benefits of remote work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*gUtEUSAIyAjsoENs7tCFjQ.png" /></figure><p>Despite these benefits, putting the systems in place to make remote teams effective and productive is not trivial. Simple things like collaboration and communication become challenging. Employees are more prone to feel lonely, distracted, or to feel like the workday never ends. In data science teams, where industry best practices are <a href="https://www.comet.ml/site/why-software-engineering-processes-and-tools-dont-work-for-machine-learning/">still very much taking shape </a>, these issues can be even more pronounced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*WgB5V0SER8ORmupI_Ldbug.png" /></figure><p>This post aims to highlight some work from home best practices, both general and data science-specific, in order to help data scientists and teams remain productive, connected and happy while working remotely.</p><h3>Best Practices for Members of Data Science Teams</h3><h3>Create a Productive Workspace</h3><p>Have a dedicated workspace (read: not your bed). Keep your phone away from your workspace and only check it during breaks. Be sure to keep your workspace clean-working in a messy environment can be incredibly distracting.</p><h3>Make sure to chat with your colleagues — even when you don’t have to</h3><p>Schedule a couple of video calls a day to stay sane, even if they’re not strictly necessary. Have at least one of these be a non-work related ‘remote coffee’ with a colleague/friend. Always put the camera on and try to enlarge the video call window so it really feels like you’re having a real conversation.</p><h3>Working habits</h3><p>Block time in your calendar for undistracted work time, a workout, lunch, etc. Encourage your colleagues to do the same. Over-communicate and assume good intent. A quick update message can go a long way in ensuring your manager is reassured with the work’s progress. Relatedly, agree a daily cadence to stay in touch with your teammates — e.g. a daily standup meeting each morning.</p><p>Have a clear ‘turn-off time’ (agreed with your team) — one of the biggest risks of remote is that you never switch off and stop working so better to be proactive on this one</p><h3>Tips for Managing Data Science Teams Remotely</h3><p>When thinking about how to manage a remote team it’s important to start by highlighting the key benefits and downsides of remote work.</p><p>At a high-level, the benefit to you as a manager is the potential of increased productivity for your team due to no commute, fewer distractions (hopefully), and more time to focus.</p><p>The downsides generally are less effective communication and fewer opportunities for collaboration. Some may argue you can be just as collaborative and communicative remotely, but as any economist would tell you: there is no free lunch. While remote teams do come with downsides, there are some very good strategies by which you can minimize them.</p><h3>Take Advantage of the Pros of Remote Work</h3><p>First, focus on and enable the strengths of remote work. You have just been given a team with the capacity to focus even more than before. Use that! In order to do so, you will need to do the following:</p><ol><li><strong>Trust your team. I</strong>f you take one thing away from how to manage a remote team well, let it be this: trust your team to work effectively and manage their time well. If you spend a lot of time doing more micromanaging you will give up almost all the benefits of remote work.</li><li><strong>Set clear expectations.</strong> If you can set clear work priorities with measurable milestones, you can let your team go to work and focus. At this point, working from home can be a huge advantage because of hopefully fewer distractions and more time to get work done.</li></ol><p>If you can set clear expectations and trust your team to get the work done, working remotely can be an advantage. For example, you could check-in at the beginning of the day with the team, set expectations, and then let everyone go to work.</p><h3>Build Systems and Processes to Counterbalance the Downsides</h3><p>Now — let’s discuss the downsides. Communication and collaboration. Focused work is great when you know what needs to be done and can accomplish it mostly on your own. But what about those serendipitous moments of in-person collaboration? How do you replicate that? I would argue that you can’t completely. Communication and collaboration are probably always going to be better in person (thus, no free lunch), but that doesn’t mean you can’t minimize the downsides of remote work. To do so, I would recommend the following:</p><ol><li><strong>Communicate over video. </strong>A lot of how we communicate is non-verbal and not turning on your video during a call hurts your ability to communicate.</li><li><strong>Don’t always go straight to business in your meetings.</strong> With a remote team, you don’t get time to have “water cooler” discussions. Spend some time in meetings, when appropriate, just talking to stay connected.</li><li><strong>Have scheduled, remote-friendly team building activities.</strong> The “water cooler” discussions should happen every day, but I would also suggest a monthly activity with concentrated time to relax and get to know each other. One of our team’s favorites is .</li><li><strong>When the next steps are not clear and you need creative collaboration, do remote brainstorming using the whiteboard feature on video conferencing platforms.</strong> Don’t just get in a virtual meeting and start discussing ideas. Send out the goal in writing beforehand, have people brainstorm individually and bring ideas to the meeting, and allow everyone to share his or her ideas, then discuss and prioritize. This is sometimes called Silent Brainstorming and is very remote-friendly and tends to be more effective.</li><li><strong>Develop a culture of radical candor and psychological safety.</strong> One of your weaknesses is now communication because you are remote. Make sure your culture is especially strong in being open and honest with feedback. Both positive and negative! As a leader, be an example in asking for, giving, and receiving feedback. And make sure everyone feels safe in expressing ideas and giving opinions.</li></ol><p>These steps should help build camaraderie and trust within your remote team, both of which are necessary for strong communication and collaboration. You don’t have the luxury of falling back on serendipitous moments of in-person collaboration or the post-meeting discussions that happen when people are in the same room.</p><h3>Invest in Tools to Encourage Collaboration</h3><p>There are amazing tools that you can leverage to make you even more effective with your communication and collaboration.</p><h4>Zoom</h4><p>A good video conferencing tool is indispensable for all teams, including and especially teams who work remote. We use zoom every day for meetings with remote team members. Whatever video conferencing tool you choose, be sure it’s easily accessible so your team can have as little friction as possible getting on the phone with colleagues.</p><h4>Slack</h4><p>Real-time messaging allows the flow of information across a team to be ad-hoc, conversational and informal. These are crucial values to ensure you maintain the ability for your team to develop ideas easily , organically, and collaboratively. Slack is a popular and excellent tool that allows teams to do just this.</p><h4>Comet</h4><p>Data Science teams are prone to becoming packs of lone-wolf modelers even when working in the same place. With the switch to remote, it’s even easier to fall into this trap. Fortunately tools exist that make it far easier for data scientists, machine learning engineers, or anyone working on models to log, visualize, share and reproduce their work across distributed teams in real time. We at Comet provide a meta-machine learning platform, runnable in the cloud or on-premise/VPC, that allows data science teams to do just this: reproduce full experiments (and not just code), manage users across large distributed data science teams, and provide managers insight into team contributions and performance.</p><p>Comet allows data science teams to easily manage distributed teams by creating collaborative projects with many users:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*89toymJvceDfB_mZ3llKTw.png" /></figure><p>If your team is part of a collaborative project, every data scientist can publish their experiments in one central location, where visualizing, comparing, analyzing and debugging team-wide work becomes easy to do-even when working remotely:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*1WuJAgkszXuTkwRjkls2Fw.png" /></figure><h3>Conclusion</h3><p>With COVID-19 forcing many teams into remote work, members and managers of newly-remote teams will be well-served taking stock of how to make the most of their new telecommute. Teams that take advantage of remote work’s benefits will remain productive, communicative and happy. What’s more, safeguards and processes can be put in place to effectively counterbalance many of the common challenges remote teams face. Do yourself a favor and schedule a 15-minute video call with one of your coworkers later today. It will be time well-spent.</p><p><em>Originally published at </em><a href="https://www.comet.ml/site/how-to-make-remote-work-effective-for-data-science-teams/"><em>https://www.comet.ml</em></a><em> on March 20, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d12bf2c6c08" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comet-ml/how-to-make-remote-work-effective-for-data-science-teams-4d12bf2c6c08">How to Make Remote Work Effective for Data Science Teams</a> was originally published in <a href="https://medium.com/comet-ml">Comet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing our $22M Series A: a step closer to eliminating every barrier to mental health]]></title>
            <link>https://medium.com/spring-health/announcing-our-22m-series-a-a-step-closer-to-eliminating-every-barrier-to-mental-health-fd6b2796d77b?source=rss-c4f9a177f432------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd6b2796d77b</guid>
            <category><![CDATA[venture-capital]]></category>
            <category><![CDATA[mental-health]]></category>
            <category><![CDATA[series-a]]></category>
            <dc:creator><![CDATA[Ashley Binford]]></dc:creator>
            <pubDate>Mon, 27 Jan 2020 15:50:30 GMT</pubDate>
            <atom:updated>2020-01-28T01:59:39.205Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>A note from April Koh, CEO and Cofounder, Spring Health</em></strong></p><p>When Dr. Adam Chekroud and I founded Spring Health in 2016, we shared the same dream: <strong>a world with no barriers to mental health</strong>. We asked, what if everyone — no matter where they are in their mental health journey — could get frictionless access to the right care, at the right time?</p><p>We were fed up with the status quo and brokenhearted by the struggles of our loved ones — those who struggled in silence for months because they didn’t know where to start; those who dialed a dozen providers just to find one available slot; those who mustered up the hope, drug after drug after drug, provider after provider, to try yet another option — again, and again, and again.</p><p>When we started, there was already a lot of hype around reinventing mental healthcare. Many solutions promised fast access to <em>something</em> — therapy, meditation, drugs, or even a chatbot. <strong>But we knew fast access was meaningless — and potentially even dangerous — if the care was not right for the individual.</strong> Everyone is different, and these products treated everyone the same.</p><p>Building a world with no barriers to mental health<em> </em>meant that we needed to go <strong>beyond faster access<em> — </em>to faster <em>recovery</em></strong>, by connecting people to not just any<em> </em>care, but precisely the right care for them, from the start.</p><p>Today, we do this through a data-driven approach called <strong>Precision Mental Healthcare</strong>. Data collected through comprehensive and proprietary intakes allows us to personalize each mental health journey and match each person to the right pathway for them. We believe that a clinically rigorous foundation of clinically validated technology, licensed care navigators, and top tier mental health professionals will lead to the most effective model possible. But, we are also cognizant that there is much more work to do in our field.</p><p><strong>Today, we’re thrilled to announce our $22M Series-A, led by Northzone. </strong>Midas List investor <a href="https://www.forbes.com/profile/par-jorgen-parson/">Pär-Jörgen Pärson</a> will be joining our board. As the first investor in Spotify, PJ brings his deep experience in building B2B and D2C unicorns to help us grow our customer-obsessed and consumer-grade UX. Northzone will also help us scale internationally. Spring Health today serves members on at least 5 continents, and as employers in the US become increasingly global, we will help our customers develop unified offerings across their entire global workforce.</p><p>In addition to Northzone, we welcome two new investors: the venture arm of current customer Equinox — Equinox Ventures — and Able Partners. We are also grateful for the continued vote of confidence and participation from our earlier investors: Rethink Impact, Work-Bench, RRE, Sempervirens, and Warren Foundation.</p><p>We see this moment as an important inflection point not only for Spring Health but for mental healthcare broadly. $22M empowers us to scale a radically different approach to mental healthcare to millions of lives. <strong>This investment allows us to take a big step forward in building a world where guessing has no place in mental healthcare, where every barrier to mental health has been eliminated.</strong></p><p>Some have accused us of being “too clinical.”<strong> But we strongly believe that the lives entrusted to us — many who find us in times of crisis — deserve no less </strong>than a center of excellence upholding the highest clinical standards. We will continue to scrutinize our approach through data, and we will continue to insist on evidence of clinical efficacy from ourselves and our peers. <strong>Our clinical rigor has led some of the largest employers in the US to place their trust in us over other solutions.</strong></p><h3>A few things I am proud of today:</h3><ul><li>Our impact will reach over 2 million lives this year</li><li>We work across all industries — legal, tech, real estate, finance, pharmaceutical/biotech, wholesale, retail, education — with over 50 amazing customers like Sysco, Gap, and Equinox</li><li>As of 2019, we have started collaborating closely with national payers and their members</li><li>Cofounder and Chief Scientist Adam Chekroud now sits on the Editorial Board of Lancet Psychiatry, one of the most prestigious psychiatry journals in the world</li><li>Our 50-state in-person and virtual provider network is accessible in an average of 1.2 days</li><li>We consistently engage over 30% of employees in a clinically meaningful way</li></ul><p>I have told some that even if Spring Health were to change just one life, this chapter of my life would be well-spent. It is so humbling to think that we now have before us the opportunity to change hundreds of thousands — ultimately millions — of lives. We promise to treat this as a privilege every day and to do our best not to let our users down.</p><p>Thank you for being part of this journey with us. Please feel free to reach out if you want to be more involved:</p><ul><li>To bring Spring Health to your workplace: <a href="/service/mailto:sales@springhealth.com">sales@springhealth.com</a></li><li>To join our team: <a href="https://jobs.lever.co/springhealth/">our job board</a></li><li>To learn more: <a href="/service/mailto:hello@springhealth.com">hello@springhealth.com</a></li></ul><p>Here’s to a mentally healthier world —</p><p>April Koh, Cofounder &amp; CEO, Spring Health</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd6b2796d77b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spring-health/announcing-our-22m-series-a-a-step-closer-to-eliminating-every-barrier-to-mental-health-fd6b2796d77b">Announcing our $22M Series A: a step closer to eliminating every barrier to mental health</a> was originally published in <a href="https://medium.com/spring-health">Spring Health</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>