<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by MosaicML on Medium]]></title>
        <description><![CDATA[Stories by MosaicML on Medium]]></description>
        <link>https://medium.com/@MosaicML?source=rss-eb63b33497b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*zW9ApecJ3MZChrdP.jpg</url>
            <title>Stories by MosaicML on Medium</title>
            <link>https://medium.com/@MosaicML?source=rss-eb63b33497b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:11:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@MosaicML/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[BioMedLM: a Domain-Specific Large Language Model for Biomedical Text]]></title>
            <link>https://medium.com/@MosaicML/pubmed-gpt-a-domain-specific-large-language-model-for-biomedical-text-567b18e2b11?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/567b18e2b11</guid>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Fri, 27 Jan 2023 18:58:31 GMT</pubDate>
            <atom:updated>2023-01-31T18:56:33.210Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBD3CrXiPm8LqukEAP-FUQ.png" /></figure><h3><strong>The Stanford Center for Research on Foundation Models (CRFM) and MosaicML announce the release of BioMedLM, a purpose-built AI model trained to interpret biomedical language. </strong>Editorial update: this blog post was revised on 1/30/2023 to reflect name change from PubMed GPT.</h3><p>Large language models (LLMs) offer amazing capabilities for general-purpose natural language generation, image generation, speech synthesis, and multi-modal combinations of these applications. But is there more we can do when we know they will be used in industry-specific situations?</p><p>Today we announce the results of a partnership between MosaicML and the <a href="https://crfm.stanford.edu/">Stanford Center for Research on Foundation Models (CRFM)</a> that demonstrates the capabilities of industry-specific large language models-specifically for the field of biomedicine. Using the <a href="https://www.mosaicml.com/cloud">MosaicML Cloud</a> platform, CRFM trained a 2.7B parameter GPT on biomedical data from <a href="https://pubmed.ncbi.nlm.nih.gov/">PubMed</a> that achieves state-of-the art results on medical question and answer text from the <a href="https://www.usmle.org/">US Medical Licensing Exam (USMLE)</a> — highlighting the promise of domain-specific language generation models in real-world applications. The result: BioMedLM (formerly known as PubMed GPT).</p><blockquote><em>“We are excited to release a new biomedical model trained on PubMed, a first step in building foundation models that could support biomedical research.” — Percy Liang, Director, Stanford CRFM</em></blockquote><p>Our work reinforces <a href="https://arxiv.org/abs/2007.15779">existing</a> <a href="https://arxiv.org/abs/2211.09085">research</a> that shows standard LLMs trained on domain-specific data can outperform general-purpose models and compete with expert-designed, domain-specific model architectures. In this blog post, we outline the overall approach, our results, and our takeaway: custom LLMs are a turn-key solution for any organization with domain-specific data, not just a few companies with massive datasets and enormous compute budgets. Before we begin, a reminder: this model was developed solely for research purposes and is not suitable for production.</p><p>For additional analysis of the scientific results, as well as more information on how to access the model, take a look at CRFM’s <a href="https://crfm.stanford.edu/2022/12/15/pubmedgpt.html">companion blog post</a>.</p><h3>BioMedLM Recipe</h3><p><strong>Model. </strong>BioMedLM is based on a <a href="https://huggingface.co/stanford-crfm/pubmedgpt">HuggingFace GPT model</a> (decoder-only transformer) with 2.7B parameters and a maximum context length of 1024 tokens. It uses a custom biomedical tokenizer trained on PubMed Abstracts with a vocabulary size of 28896.</p><p>While CRFM has already made great strides in developing complex models that capture the <a href="https://arxiv.org/abs/2210.09338">structure</a> of <a href="https://arxiv.org/abs/2203.15827">knowledge</a> in biomedical text, for this project we wanted to keep the model design as simple as possible to demonstrate the power of off-the-shelf LLM training recipes. That way we could reuse the same recipe to train a state-of-the-art GPT model for other domain-specific applications like legal text.</p><p><strong>Data. </strong>We trained BioMedLM on the PubMed Abstracts and PubMed Central portions of the <a href="https://arxiv.org/abs/2101.00027">Pile</a> dataset. This dataset contains around 50B tokens and spans a collection of 16 million abstracts and 5 million full-text articles from the biomedical literature, as curated by the National Institute of Health.</p><p><strong>Compute. </strong>Although the dataset contains 50B tokens, this does not directly determine the training budget. GPT models of a similar size to BioMedLM are often trained on significantly more data. For example, GPT3–2.7B and GPT-J were trained on 300B and 400B tokens of data, respectively.</p><p>Within this design space, we elected to train BioMedLM for a long compute duration (300B tokens) by performing multiple passes, or epochs, over the 50B tokens. Our results show that even when limited data is available, one can still train a custom, high-quality LLMs from scratch!</p><h3>Training With MosaicML</h3><p>To train BioMedLM easily, quickly, and efficiently, we used the MosaicML Cloud for infrastructure and trained the model using MosaicML’s Composer and Streaming Dataset libraries. All model and training code is built off of PyTorch. <a href="https://github.com/stanford-crfm/pubmedgpt">See the code here!</a></p><h4>MosaicML Cloud</h4><p><strong>‍</strong> Using our cloud software stack, we orchestrated training on top of a cluster with 128 NVIDIA A100–40GB GPUs and 1600 Gb/s networking bandwidth between nodes. The physical GPUs were hosted on a leading cloud provider. The total training time for Pubmed GPT was ~ 6.25 days. Using placeholder pricing of $2/A100/hr, the total cost for this training run on MosaicML Cloud was ~ $38,000.</p><h4>Composer</h4><p><strong>‍</strong>For the optimal LLM training experience, we used Composer with its FSDP integration ( <a href="https://pytorch.org/docs/stable/fsdp.html">FSDP</a> is a PyTorch backend for fully sharded data parallel training). The open-source Composer library makes it easy to train large, custom models across hundreds of GPUs without imposing any restrictions on the model code. For example, we replaced the HuggingFace <a href="https://huggingface.co/docs/transformers/model_doc/gpt2">GPT2Model</a> attention implementation with <a href="https://github.com/HazyResearch/flash-attention">FlashAttention (Dao et. al),</a> which improved training throughput by nearly 2x while producing a math-equivalent model. Composer had no trouble handling the custom model definition, and training time was cut in half! Having the flexibility to easily add and test modifications greatly improved the training efficiency of BioMedLM, and we expect to make similar improvements in future LLM work.</p><h4>Streaming Datasets</h4><p><strong>‍</strong>To manage a training dataset containing over 100GB of text in a cloud-native way, we used MosaicML’s new <a href="https://github.com/mosaicml/streaming">StreamingDataset</a> library. This library enables users to host arbitrary data (text, images, etc.) as shards in object storage and then stream that data to a training job anywhere in the world. StreamingDataset works out of the box with vanilla PyTorch DataLoaders, and is compatible with multiple CPU workers, multi-GPUs, and multi-node training.</p><p>StreamingDataset made it fast, flexible, and cheap for us to manage a custom training dataset. There was no need to pre-tokenize the data; we were able to store the samples as raw text in object storage. At runtime, we streamed in text samples and tokenized on-the-fly, with no impact on training throughput and no data loader bottlenecks. This flexibility and performance enabled us to test different tokenization schemes for BioMedLM without having to regenerate the dataset.</p><p>As one last proof point for StreamingDataset, our final training run for BioMedLM did <strong>not</strong> use compute from AWS, despite the fact that the dataset was stored on AWS S3. Instead, we streamed the data from S3 to MosaicML Cloud without impacting training throughput, and without downloading the whole dataset at the start. Instead, shards were streamed in as they were needed during the training run and cached after the first epoch. This limited the cost of data egress to &lt;$10 for the whole training run, compared to ~$38,000 for the compute!</p><h3>How Does BioMedLM Perform?</h3><p>Let’s cut to the chase: does it work? We evaluated BioMedLM on several question and answer (QA) benchmarks, and manually assessed its generations for a question summarization task. One key benchmark was <a href="https://arxiv.org/abs/2009.13081">MedQA-USMLE</a>, which consists of question and answer pairs taken from previous Medical Licensing Exams given to doctors in the United States. As we see in Figure 1, the text contains detailed, technical medical queries related to a variety of health concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*JRqCnqSthtlyoPH1.png" /><figcaption><strong>Figure 1</strong>: An example question fromMedQA-USMLE, a question and answer benchmark suite design based on questions from the U.S. Medical Licensing Exam.</figcaption></figure><p>In our evaluation we compared our results against several models:</p><ul><li><a href="https://arxiv.org/abs/2210.09338">DRAGON</a> is a state-of-the-art biomedical language model, released last month by members of the CRFM team in a separate effort. The model is pre-trained from text (Pubmed Abstracts) and an expert-curated biomedical <em>knowledge graph</em> (i.e, the <a href="https://pubmed.ncbi.nlm.nih.gov/14681409/">Unified Medical Language System</a>, also known as UMLS). <br>‍<a href="https://github.com/EleutherAI/gpt-neo">‍</a></li><li><a href="https://github.com/EleutherAI/gpt-neo">GPT-Neo 2.7B</a> is an LLM of similar size and architecture as BioMedLM, but trained on the Pile and therefore not domain-specific.</li><li><a href="https://arxiv.org/abs/2211.09085">Galactica</a> is a 120B parameter LLM, trained on a corpus of over 48 million papers, textbooks, scientific websites, encyclopedias, and other sources of scientific knowledge across multiple domains.<br>‍</li><li><a href="https://arxiv.org/abs/2203.15827">BioLinkBERT</a> is another biomedical model trained by Stanford CRFM that uses the link structure of documents during training.<br>‍</li><li><a href="https://arxiv.org/abs/2007.15779">PubMedBERT</a> is another domain-specific language model for biomedical NLP.</li></ul><p>Due to time and resource constraints we did not evaluate other biomedical systems that use different evaluation tasks or setups than ours, such as <a href="https://arxiv.org/abs/2010.06060">BioMegatron</a>, <a href="https://arxiv.org/abs/2203.03540">GatorTron</a>, and <a href="https://arxiv.org/abs/2210.10341">BioGPT</a>. <a href="https://crfm.stanford.edu/2022/12/15/pubmedgpt.html"><em>See the CRFM companion blog post</em></a> for more details on their relation to BioMedLM.</p><p><strong>‍</strong></p><p>Our results as shown in Figure 2 demonstrate our findings: LLMs are remarkably versatile and deliver significant improvements when trained on domain-specific data, while focused models can achieve high quality with relatively few resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LI59yurBeEtwQdMs-V0Usw.png" /><figcaption><strong><em>Figure 2: </em></strong><em>A comparison of </em>BioMedLM<em> to DRAGON, Galactica, GPT-Neo, BioLinkBERT,and PubMedBERT. </em><a href="https://crfm.stanford.edu/2022/12/15/pubmedgpt.html"><em>See the CRFM companion blog post</em></a><em> for a more detailed analysis of PubMedQA and BioASQ. *The Galactica model is evaluated zero-shot with no task-specific finetuning.</em></figcaption></figure><h3>Conclusion #1: LLMs are remarkably versatile.</h3><p>BioMedLM outperforms DRAGON on MedQA-USMLE, setting a new state-of-the-art, and matches DRAGON’s performance on two other QA tasks, PubMedQA and BioASQ. Crucially, BioMedLM achieves this without any explicit knowledge graph. These results show that when trained from scratch with domain-specific data, standard LLMs can deliver comparable performance to custom, expert-designed systems.</p><p>Like most things in life, this performance does not come for free. BioMedLM has 2.7B parameters compared to DRAGON’s 360m, demonstrating a trade-off between model size/cost versus domain expertise and custom architectures. However, the general purpose nature of LLMs makes them more applicable to different domains. Using the same simple LLM training recipe, we could train a model for legal or financial domain expertise.</p><h3>Conclusion #2: Pre-training on domain-specific data beats general-purpose data.</h3><p>BioMedLM significantly outperforms the general-purpose <a href="https://www.eleuther.ai/projects/gpt-neo/">GPT-Neo</a>, a similar model with 2.7B parameters trained on text and code from many domains, by a significant margin (17% on MedQA), even after fine-tuning GPT-Neo for the tasks at hand.</p><p>As mentioned above, GPT-Neo is trained on the Pile, a massive corpus that spans PubMed Abstracts and PubMed Central but mainly comprises many other sources, such as Wikipedia, technical text from the US Patent and Trademark Office, GitHub, HackerNews, and Reddit. Although this data set includes sources of natural language that may boost a model’s ability in the domain (e.g, Wikipedia), or expand the model’s technical fluency across many domains (e.g., USPTO), it also includes sources such as HackerNews and Reddit that potentially mix valuable technical concepts with less trustworthy or outright incorrect and biased language. In total, PubMed Abstracts and PubMed Central, our data of interest, comprises only 17.5% of the Pile.</p><h3>Conclusion #3: Focused models achieve higher quality with fewer resources.</h3><p>Although the overall size of our dataset is significantly smaller than the Pile, our custom model outperforms GPT-Neo thanks to our selection of domain-specific data. BioMedLM also outperforms Galactica (120B) on MedQA, and competes with its behavior on PubMedQA and BioASQ. A key difference is that Galactica’s results are zero-shot, without the task-specific fine-tuning we apply for other models. While this zero-shot behavior is impressive, Galactica’s model targets myriad scientific domains and is 44x the size of our model, incurring significant costs for training and inference. By focusing on a single scientific domain, we arrive at a much smaller model that can still compete with Galactica in biomedical text.</p><p>Large language models offer the promise of new capabilities for many companies and researchers, with the potential to deliver <strong>increased quality with less data and compute</strong> than often assumed.</p><p>There still remains an elephant in the room: model size and training cost. However, through our work on efficient model implementations, training efficiency methods in <a href="https://www.mosaicml.com/composer">Composer</a>, and scalable cloud abstraction and orchestration with <a href="https://www.mosaicml.com/cloud">MosaicML Cloud</a>, we offer an accessible — and affordable — solution.</p><h3>Looking Forward</h3><p>The partnership announced today is just the first step of a larger journey to bring state-of-the art results in biomedical text, and other domains, into the hands of more individuals. As an example, consider our preliminary text generation results with BioMedLM. Below is an example from <a href="https://aclanthology.org/P19-1215/">MeQSum</a>, a medical question summarization benchmark. The task models the behavior of a system that can process a patient query via email or voicemail and present it to a physician in a clear and actionable format.</p><blockquote><strong><em>Input</em></strong><em>: SUBJECT: ball and socket joint MESSAGE: sir, with due respect I would like to state that there is a pain in my ball and socket joint. I didn’t feel and pain in normal or lower position there is a huge site of pain in my ball and socket joint. please prescribe a medicine for the cure.</em></blockquote><blockquote><strong><em>Output (Current)</em></strong><em>: What are the treatments for joint pain?</em></blockquote><blockquote><strong><em>Output (BioMedLM 2.7B</em></strong><em>): What are the treatments for ball and socket joint pain?</em></blockquote><p>From the text we can see that the system must distill a message with ambiguity, misspellings, and misstatements into a version that is succinct yet still preserves the message’s intended meaning. BioMedLM’s results are the first step along an exciting journey; however, our work should be seen as a proof-of-concept and is not intended for clinical use. The eventual hope is for a future of trustworthy, interactive AI systems that facilitate reliable interactions while triaging engagement with human experts.</p><p>Stay tuned for future improvements and releases of BioMedLM! If you’re interested in training LLMs on your own data, we invite you to <a href="https://forms.mosaicml.com/demo">schedule a demo.</a></p><p>To keep in touch with MosaicML, join our <a href="https://join.slack.com/t/mosaicml-community/shared_invite/zt-1btms90mc-GipE2ufuPkKY0QBrmF3LSA">Community Slack</a> or follow us on <a href="https://twitter.com/MosaicML">Twitter</a>!</p><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/introducing-pubmed-gpt"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=567b18e2b11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training Stable Diffusion from Scratch Costs <$160k]]></title>
            <link>https://medium.com/@MosaicML/training-stable-diffusion-from-scratch-costs-160k-e14c6facdb53?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e14c6facdb53</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Wed, 25 Jan 2023 19:28:26 GMT</pubDate>
            <atom:updated>2023-01-25T19:28:26.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dK4422Rhstkgm9larcxVOw.png" /></figure><h4>We wanted to know how much time (and money) it would cost to train a Stable Diffusion model from scratch using our Streaming datasets, Composer, and MosaicML Cloud Platform. Our results: it would take us 79,000 A100-hours in 13 days, for a total training cost of less than $160,000. Our tooling not only reduces time and cost by 2.5x, but it is also extensible and simple to use.</h4><h4><em>Try out our Stable Diffusion code </em><a href="https://github.com/mosaicml/diffusion-benchmark"><em>here</em></a><em>!</em></h4><p>The <a href="https://www.nytimes.com/2023/01/07/technology/generative-ai-chatgpt-investments.html">AI world is buzzing</a> with the power of large generative neural networks such as ChatGPT, Stable Diffusion, and more. These models are capable of impressive performance on a wide range of tasks, but due to their size and complexity, only a handful of organizations have the ability to train them. As a consequence, access to these models can be restricted by the organization that owns them, and users have no control over the data the model has seen during training.</p><p>That’s how we can help: at MosaicML, we make it easier to train large models efficiently, enabling more organizations to train <strong>their own models</strong> on <strong>their own data</strong>. As shown in a <a href="https://www.mosaicml.com/blog/gpt-3-quality-for-500k">previous blog post</a>, our <a href="https://github.com/mosaicml/streaming">StreamingDataset library</a>, our training framework <a href="https://github.com/mosaicml/composer">Composer</a>, and our <a href="https://www.mosaicml.com/blog/introducing-mosaicml-cloud">MosaicML Cloud</a> platform significantly simplify the process of <a href="https://www.mosaicml.com/blog/gpt-3-quality-for-500k">training large language models</a> (LLMs). For this blog post, we used that same process to measure the time and cost to train a <a href="https://stablediffusionweb.com/">Stable Diffusion model</a> from scratch. We estimated an upper-bound of <strong>79,000 A100-hours to train Stable Diffusion v2 base in 13 days on our MosaicML Cloud platform, corresponding to a total training cost of less than $160,000.</strong> This is a 2.5x reduction in the time and cost reported in the <a href="https://huggingface.co/stabilityai/stable-diffusion-2-base">model card</a> from Stability AI. In addition to saving time and money, our Streaming, Composer, and MosaicML Cloud tools make it dead-simple to set up and scale Stable Diffusion training across hundreds of GPUs without any additional effort. The code we used for this experiment is open-source and ready to run; check it out <a href="https://github.com/mosaicml/diffusion-benchmark">here</a>! And if you’re interested in training diffusion models yourself on the MosaicML Cloud, <a href="https://forms.mosaicml.com/demo">contact us for a demo</a>.</p><h3>Time and Cost Estimates</h3><p><strong>Table 1</strong> and <strong>Figure 1</strong> below illustrate how the Stable Diffusion V2 base training time and cost estimates vary by the number of GPUs used. Our final estimate for 256 A100s is 12.83 days to train with a cost of $160,000, a 2.5x reduction in the time and cost reported in the Stable Diffusion <a href="https://huggingface.co/stabilityai/stable-diffusion-2-base">model card</a>. These estimates were calculated using measured throughput and assumed training on 2.9 billion samples. Throughput was measured by training on 512x512 resolution images and captions with a max tokenized length of 77. We scaled GPUs from 8 to 128 NVIDIA 40GB A100s, then extrapolated throughput to 256 A100s based on these measurements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S4stxglq4G3P5Kg0.png" /><figcaption><strong><em>Table 1</em></strong><em>: Time and cost estimates to train a Stable Diffusion model on 2.9 billion samples when increasing the number of NVIDIA 40GB A100 GPUs. Throughput measurements were done with a global batch size of 2048 and a per device microbatch size of 4. *256 A100 throughput was extrapolated using the other throughput measurements.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nm7jSLsILBRzQDGlttKO6Q.png" /><figcaption><strong><em>Figure 1: </em></strong><em>Plot of days to train versus number of NVIDIA 40GB A100 GPUs. “Measured” indicates empirically verified throughput numbers. “Predicted” demonstrates how we estimated throughput at 256 GPUs.</em></figcaption></figure><h3>Benchmark Setup</h3><p>How did we get these results? We took advantage of a MosaicML Streaming dataset, our Composer training framework, and the MosaicML Cloud to measure throughput when training a Stable Diffusion model. You can reproduce our results using the code in <a href="https://github.com/mosaicml/diffusion-benchmark">this repo</a>. Read on for more details:</p><h3>Streaming</h3><p>A major pain point when training Stable Diffusion is working with enormous datasets such as <a href="https://laion.ai/blog/laion-5b/">LAION-5B</a>. The <a href="https://github.com/mosaicml/streaming">MosaicML StreamingDataset library</a> makes it significantly easier to manage and use these massive datasets. It works by converting the target dataset into a Streaming format, then storing the converted dataset to the desired cloud storage (e.g. an AWS S3 bucket). To use the stored dataset, we simply define a “StreamingDataset” class that pulls and transforms samples from the stored dataset.</p><p>For our results, we streamed a subset of the <a href="https://laion.ai/blog/laion-400-open-dataset/">LAION-400M</a> dataset with 256x256 images and their associated captions. Images were resized to 512x512 for final throughput measurements. We estimated the Streaming data loader throughput to be at least 30x higher than model throughput in all configurations we tested, so we’re unlikely to see data loader-related bottlenecks anytime soon. Check out our <a href="https://github.com/mosaicml/diffusion-benchmark/tree/main/data.py">data script</a> to see how we defined our streaming dataset and watch out for our soon-to-be-released Streaming blog post for more details!</p><h3>Composer</h3><p>Our open-source <a href="https://composer.dev/">Composer</a> library contains many state-of-the-art methods for accelerating neural network training and improving generalization. For this project, we first defined a <a href="https://docs.mosaicml.com/en/v0.12.0/composer_model.html">ComposerModel</a> for Stable Diffusion using models from <a href="https://huggingface.co/docs/diffusers/index">HuggingFace’s Diffusers library</a> and configs from “stabilityai/stable-diffusion-2-base”. The ComposerModel and Streaming dataset were provided to Composer’s <a href="https://docs.mosaicml.com/en/v0.12.0/trainer/using_the_trainer.html">Trainer</a> with an AdamW optimizer, an EMA algorithm 2, a throughput measurement callback, and a Weights and Biases logger. Finally, we called “fit()” on the Trainer object to start training.</p><h3>MosaicML Cloud</h3><p><a href="https://cloud.mosaicml.com/">MosaicML Cloud</a> orchestrates and monitors compute infrastructure for large-scale training jobs. Our job scheduler makes launching and scaling jobs easy.<strong> Figure 2</strong> shows the MosaicML Cloud training configuration (left) and the CLI used to launch the training run (right). From the configuration, we can easily scale the number of GPUs with the “gpu_num” parameter. The same code we write for one node can automatically leverage tens of nodes and hundreds of GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EleBS6i3ANSaK2f5.png" /><figcaption><strong><em>Figure 2</em></strong><em>: Running multi-node jobs is as easy as changing the “gpu_num” field. <br>‍</em><strong><em>Left: </em></strong><em>Example MosaicML Cloud training configuration. </em><strong><em>Right: </em></strong><em>Starting a training run with Mosaic CLI (mcli).</em></figcaption></figure><h3>What’s Next?</h3><p>In this blog post, we estimated a 2.5x reduction in time and cost to train Stable Diffusion when using our Streaming datasets, Composer library, and MosaicML Cloud. This is a great preliminary result, but we’re not done yet. In a future blog post, we’ll verify how we can train to convergence at this speed. For updates on our latest work, join our <a href="https://join.slack.com/t/mosaicml-community/shared_invite/zt-1btms90mc-GipE2ufuPkKY0QBrmF3LSA">Community Slack</a> or follow us on <a href="https://twitter.com/MosaicML">Twitter</a>. If your organization wants to start training diffusion models today, you can <a href="https://forms.mosaicml.com/demo">schedule a demo online</a> or email us at <a href="/service/mailto:demo@mosaicml.com">demo@mosaicml.com</a>.</p><p>Notes:</p><ol><li><em>Per device microbatch size is related to gradient accumulation, but microbatch size is easier to use when increasing the number of GPUs. Per device microbatch size and gradient accumulation are related as follows: grad_accum = global_batch_size / (n_devices * per_device_microbatch_size). For more details, check out the “device_train_microbatch_size” variable in Composer’s </em><a href="https://docs.mosaicml.com/en/v0.12.0/api_reference/generated/composer.Trainer.html#composer.Trainer"><em>Trainer API reference</em></a><em>.</em></li><li><em>Composer has a built-in </em><a href="https://docs.mosaicml.com/en/v0.12.0/method_cards/ema.html"><em>EMA algorithm</em></a><em>, but we had to improve its memory efficiency for our benchmark hence the EMA implementation in our benchmark repo. We will update the EMA algorithm in Composer to be more memory efficient soon.</em></li></ol><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e14c6facdb53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Enterprises Should Treat AI Models Like Critical IP (Part 2)]]></title>
            <link>https://medium.com/@MosaicML/why-enterprises-should-treat-ai-models-like-critical-ip-part-2-d5dc02143ca5?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5dc02143ca5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[intellectual-property]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Wed, 25 Jan 2023 19:10:32 GMT</pubDate>
            <atom:updated>2023-01-25T19:10:32.301Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cGq0IhcQEFvvFKkz.png" /></figure><p>In 2022, the potential of Large Language Models (LLM) and Generative AI entered the mainstream, while organizations began to recognize the value of state-of-the-art AI models to activate their data. In part 2 of this blog, I explore why enterprises should treat AI models as some of their most important intellectual property in 2023 and beyond.</p><p>In <a href="https://www.mosaicml.com/blog/ai-models-critical-ip-1">part 1</a> of this blog, I gave a brief history of the development of AI models — and why they should be considered an essential part of a company’s IP that leverages its most valuable asset: data. At the same time, access to AI model building and training has been limited due to the cost and complexity of model development. So — where do we go from here?</p><p>First and foremost: A central party serving a single, generic model provides <strong>no unique differential value</strong> to its users<strong>.</strong> Every user of that model has access to the same capabilities. When two competitors are using the same model, there’s<strong> little competitive advantage gained from the model itself. </strong>By definition, any competitive advantage comes from what each company is doing <em>differently</em> or <em>better</em> than the other.</p><p>However, there <em>are</em> scenarios where “genericized” data baked into a model enables innovation. For example, when the cost of data aggregation is very high, building a central model based on generic data can be useful when differentiation is added in other ways. For instance, in the medical field, a model built off of a single data source like cell pathology can be used to help automate cancer detection, search for genetic abnormalities, or identify tissue abnormalities. Applying a model to <em>adjacent, non-competitive</em> applications within the field of medical diagnosis can create unique value for each one.</p><h3>All Data is Different</h3><p>It is very unlikely that an AI model trained on generic data will derive insight from domain-specific inputs. For instance, models like GPT-3 are trained on Reddit and OpenWebText. If a company wants to use them with domain-specific data such as legal or medical text, the learned relationships in the generic model won’t have much predictive power. This should come as no surprise. Doctors, lawyers, and scientists all use specific ways to communicate that are precise to their field, require years of study, and often have large deviations from vernacular language.</p><p>If you’re well-versed in the AI and ML space, here’s the part of the blog where you ask: But what about fine-tuning? For everyone else: fine-tuning is the process of starting with a model trained on generic data (a pre-trained model) and continuing to train it on a more specific data set. Fine-tuning can improve a generic model and certainly has its place. When the domain-specific data points are much smaller in number than the generic data points, for example, fine tuning can augment a generic model to give it some domain specificity. However, this has limits. A pre-trained model has established representations of the data on which it was initially trained. If the statistics of the fine-tuning data are vastly different, the model simply won’t be able to capture these differences well. There’s a growing body of literature on this topic which is beyond the scope of this article.</p><p>We’ve observed that enterprises often have <em>vast</em> datasets as big or bigger than the generic datasets used to build the model initially. If that’s the case, why start with a generic model? It’s <strong>much</strong> <strong>more powerful to build a domain-specific model</strong> that’s<strong> unique to the data on which it’s trained</strong>. The provenance of that data is known, and companies can be certain that the ML model doesn’t contain any data from apocryphal (or worse, biased) sources.</p><h3>You’ve Got the Data — Let’s Put It to Use</h3><p>MosaicML was founded with the belief that it is crucial for enterprises to have the power to build, train, maintain, and update their own ML models. However, we recognize that resources for training large-scale neural network models are still not widely available to enterprises.</p><p>Building large models <strong>requires large, difficult-to-manage computing resources</strong>: A GPT-3 scale model, for example, requires at least 256 GPUs to complete the training in a tractable amount of time (approximately one month). Currently, this requires 1) an infrastructure operations team to set up and maintain compute at this scale, 2) software to coordinate and orchestrate these resources, 3) machine learning software knowledge required to optimize the compute, and 4) general machine learning knowledge to apply neural network techniques to the enterprise’s data sets. This equates to a team of ~100 people and tens of millions of dollars of infrastructure. That’s a high barrier to entry, and the reason these capabilities are only available to the largest and most advanced tech companies. But enterprises that can overcome these challenges will have a competitive advantage by creating differentiated value within their industries.</p><p>Ideally,<strong> any reasonably sized organization should be able to apply ML insights</strong> with access to tools that remove barriers to entry and make training neural network models faster, cheaper, and easier. Infrastructure can be accessed with stable APIs that abstract away the details of scaling, orchestration, optimization, and management of the underlying hardware. Technologies that greatly reduce the cost of training models by more efficiently using hardware resources will stabilize costs so that more enterprises have access to state-of-the-art capabilities.</p><p>These tools don’t have to be hard to use or require specialized talent. If we think of ML models as an activated form of data, the ability to train models becomes a basic need for organizations that store and generate lots of data. Data provenance is becoming increasingly important as we’ve seen LLMs either attempt to impute data points that don’t exist (ahem, <a href="https://arxiv.org/abs/2202.03629">hallucinate</a>) or produce data seen during training. Building smaller, more domain-specific models can not only increase in-area performance, but also reduce inference costs and latency. The impact of machine learning in our daily lives will be fully realized once it is accessible to a wide variety of enterprise companies.</p><p>Interested in learning more about our approach to making state-of-the-art AI &amp; ML widely available? <a href="https://www.youtube.com/watch?v=oDnKEppIFPo">Check out a demo</a> of our purpose-built platform for accessing the best algorithms, systems and architectures — or <a href="https://www.mosaicml.com/cloud">sign up for early access</a> to the MosaicML cloud today.</p><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/ai-models-critical-ip-2"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5dc02143ca5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Enterprises Should Treat AI Models Like Critical IP (Part 1)]]></title>
            <link>https://medium.com/@MosaicML/why-enterprises-should-treat-ai-models-like-critical-ip-part-1-dd214c02f3d?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd214c02f3d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[intellectual-property]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Mon, 23 Jan 2023 19:54:29 GMT</pubDate>
            <atom:updated>2023-01-23T19:54:29.972Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LH14rcVi5pEXh9JQ.png" /></figure><p>Five years ago, The Economist proclaimed that data was the new oil. Since then, the power of amassed data to impact the world has become even more undeniable. That’s why companies should treat AI models as some of their most important intellectual property, rather than setting them aside as something with the potential for future impact. Today’s large, state-of-the-art AI models can be viewed as a powerful tool to activate an organization’s data — and maximize its value.</p><p>AI models are simply an organized version of a company’s data in the form of a flexible, intuitively queryable database. As such, trained models represent the<strong> unique value a company brings to the world</strong> via the data it has collected by selling products and services to its customers. In order to take full advantage of their most valuable asset, companies must be able to build these models internally.</p><h3>Understanding AI Models: A Quick History</h3><p>To gain a better understanding of what AI models represent, it’s helpful to look at their inspiration: <strong>the human brain</strong>. Our brains encode information in the connection strength between neurons, known as a synapse. Information is typically organized into a learned conceptual hierarchy; raw data is transformed into concepts and relationships between concepts. Long-term memories form when these concepts and connections are encoded into persistent synaptic strengths. We’re able to retrieve those encoded memories by language, sight, touch, or any kind of trigger where relevant information is needed.</p><p>In the early days of artificial neural networks, scientists attempted to replicate some of these behaviors by modeling synaptic strength as a numeric <em>weight</em> and connections as a multiplication operation between weights and neurons. Biological neural networks (aka brains) use chemical processes to “update their weights” and modify the influence of one neuron upon another during learning. Artificial neural networks update their weights during learning using an algorithm called <em>backpropagation</em>; this is an iterative method that reduces the error of the output. Backpropagation generally requires global error information to update each weight after each sample has been pushed through. While artificial neural networks differ from biological ones in terms of how these weights are learned, they both produce the same result: input data is encoded in the weights of the neural network.</p><p>Up until about five years ago, AI model innovation was focused on the <em>topology</em> of the neural network- literally, how the model is organized. The idea was that the capabilities of the trained model largely stemmed from the topology and its ability to provide a better understanding of the inherent structure of the data on which it is trained. Every year, new topologies were introduced that increased benchmark performance on <em>industry-standard</em> datasets such as MNIST, CIFAR, and ImageNet. AlexNet, VGG, Inception Networks, and ResNets are all examples of networks able to achieve new levels of learning from the <em>same</em> dataset.</p><p>More recently, we’ve seen the advent of new types of models like large language models (LLMs) and diffusion models that learn about our world by exposure to data. LLMs like BERT learn about sentence structure and knowledge that’s embedded in the text they are fed. Diffusion models ( <a href="https://arstechnica.com/information-technology/2022/09/openai-image-generator-dall-e-now-available-without-waitlist/">most famously, DALL-E</a>) learn about relationships between textual descriptions of images and are able to produce images based on novel descriptions. In contrast to our previous focus on model topology, we’re beginning to understand that<strong> the data itself and how it is presented during learning dictate the performance achieved</strong>. In fact, there is a standardization of neural network topology occurring, similar to an evolutionary phenomenon observed in biology.</p><h3>AI Models Are Organized Data</h3><p>When an AI model is being trained, it encodes relationships observed in the data on which it is trained. The resultant model is <strong>like a fingerprint of its training data.</strong> It’s almost as if each AI model has its own identity, as each dataset and training method creates a unique set of representations. In fact, generative AI models can even memorize portions of the training dataset that can be retrieved when prompted. Neural networks are capable of representing vast data sets. Suddenly, the entire space represented by the data set can be used for decision making by expressing it through a generative model.</p><p>These models can be thought of as a form of database that is unique and specific to the data on which it is trained.But they aren’t databases in any standard sense. <strong>Traditional databases rely on rigidly structured input data</strong>, like columns and rows, a predefined organizational schema, and highly structured queries. SQL and other languages were designed to precisely express these queries within these organizational schemas.</p><p><strong>Large models handle data very differently.</strong> They can find inherent relationships within unstructured, noisy input data, and discover an organizational schema to represent this data. The data can be <strong>queried in unstructured and imprecise ways</strong>. Large language models like GPT-3 allow these queries to come in the form of natural human language with all of its imprecision. For example, users can essentially engage with OpenAI’s GPT-3 model by asking it questions about history, politics, or science. Diffusion models like Stable Diffusion allow human language queries that produce novel combinations of pixels (generated images) that represent associations between those words and images in the training data set. These models are able to extract meaningful relationships embedded in the data set, coming closer to the capability of knowledge extraction.</p><p>However, despite the magical-seeming qualities of these popular models, the power to interact with data in this way is only just now being realized. The precision of knowledge recall is tough to guarantee at the moment, making these models unreliable for mission-critical applications. For instance, a neural network that powers a healthcare chatbot would need to be incredibly accurate in order to prevent potential harm to patients. This is an active area of research and development.</p><h3>Why Model Building Must Be Accessible</h3><p>As we learn how to harness the power of data and build and train domain-specific AI models, we have the potential to transform nearly every industry. AI model training is in contrast to AI inference; the latter is much less computationally demanding and only deals with data points at time of usage. Model training, however, <strong>requires a large amount of three things</strong> that are in short supply: <strong>compute, data, and talent. </strong>And as model sizes increase, so does the need for these three elements.</p><p>Although many of the techniques commonly used in model training are publicly available, there is limited access to the knowledge — and technology — required to make them work. Simply put, the <strong>capabilities are obscured by complexity</strong>. What’s more, an enormous amount of compute capacity is required to build an AI model from scratch using a given dataset, and only a handful of companies like OpenAI, Meta, and Google have access to the hundreds of millions of dollars of processing power needed. This situation has created a world of haves and have-nots; AI model building and training has become inaccessible to most organizations since only the most advanced, most well-resourced groups can do it.</p><p>The result is a world where a handful of teams are building new models on datasets they have curated, then selling access to these models via API. This might seem like progress — sophisticated teams are building solutions and making them broadly available — but, as previously mentioned, models are a representation of their data. Companies<strong> that specialize in a given field tend to have the best understanding of the data</strong> and how it should be leveraged, but they lack the algorithmic and systems expertise required to build AI models. This siloing of capabilities is a fundamental obstacle to the broad and open application of AI to new fields.</p><p>In part two of this blog, I’ll go into more detail about why this state of affairs can be problematic…and how <strong>MosaicML is working to break down these silos</strong> and make ML models trained on domain-specific data widely available.</p><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/ai-models-critical-ip-1"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd214c02f3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New in Composer 0.12:]]></title>
            <link>https://medium.com/@MosaicML/new-in-composer-0-12-210799cffaa2?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/210799cffaa2</guid>
            <category><![CDATA[machine-learning-tools]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Sat, 21 Jan 2023 02:22:21 GMT</pubDate>
            <atom:updated>2023-01-21T02:22:21.778Z</atom:updated>
            <content:encoded><![CDATA[<h3>New in Composer 0.12: Mid-Epoch Resumption with MosaicML Streaming, CometML ImageVisualizer, HuggingFace Model and Tokenizer Loading, and more!</h3><figure><img alt="Composer 0.12 release feature bullet list" src="https://cdn-images-1.medium.com/max/1024/1*QdOdsUYBReTD9na0b-MWiw.png" /></figure><p>We are excited to announce the release of Composer 0.12 ( <a href="https://github.com/mosaicml/composer/releases">release notes</a>)! This release includes several new features, plus improvements to existing capabilities — kudos to the Composer community for your engagement, feedback, and contributions to this release.</p><p>Install via pip:</p><pre>pip install mosaicml==0.12.0</pre><p>For those who want to join the Composer community: learn more about contributing <a href="https://github.com/mosaicml/composer/blob/dev/CONTRIBUTING.md">here</a>, and message us on <a href="https://join.slack.com/t/mosaicml-community/shared_invite/zt-w0tiddn9-WGTlRpfjcO9J5jyrMub1dg">Slack</a> if you have any questions or suggestions!</p><p>There are many features included in this release, so let’s dive in and see what’s new.</p><h3>What’s in this release</h3><ul><li>CometML ImageVisualizer to visualize images and segmentation masks while training your computer vision models</li><li>Oracle Cloud Infrastructure ObjectStore and Google Cloud Storage support for saving and loading checkpoints</li><li>Mid-Epoch Resumption with MosaicML Streaming support</li><li>Support for loading HuggingFace models and tokenizers for easier pre-training to fine-tuning workflow</li><li>OptimizerMonitor updates to track optimizer-specific metrics</li><li>New PyTorch 1.13 + CUDA 11.7 Docker images</li><li>New community contribution for speedup algorithm — GyroDropout</li></ul><h3>About Composer</h3><p>The Composer library helps developers train PyTorch neural networks faster, at lower cost, and to higher accuracy. Composer includes:</p><ul><li>20+ methods for speeding up training networks for computer vision and language modeling.</li><li>An easy-to-use trainer that integrates best practices for efficient training.</li><li>Functional forms of all speedup methods that are easy to integrate into your existing training loop.</li><li>Strong and reproducible training baselines to get you started as quickly as possible.</li></ul><h3>Visualize your images and segmentation masks while training with CometML ImageVisualizer callback</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gVlRP3TdPy6b0Jvz.png" /></figure><p>Visualize your images and segmentation masks as you train with CometML and Composer using our ImageVisualizer callback with <a href="https://www.comet.com/site/">CometML</a>. Check out our latest <a href="https://www.mosaicml.com/blog/comet">MosaicML + Comet blog</a> to learn more about monitoring and logging your training workloads on the MosaicML Cloud with Comet.</p><pre>from composer.trainer import Trainer<br><br>trainer = Trainer(...,<br>    callbacks=[ImageVisualizer()],<br>    loggers=[CometMLLogger()]<br>)</pre><p>Train the MosaicML DeepLabV3 that yields a 5x faster time-to-train compared to a strong baseline. See our <a href="https://www.mosaicml.com/blog/mosaic-image-segmentation">latest segmentation blog</a> for more details and recipes, and start training today with our example <a href="https://github.com/mosaicml/examples/tree/main/deeplab">starter code for segmentation</a>.</p><h3>Save and load your checkpoints using Oracle Cloud Infrastructure (OCI) or Google Cloud Storage (GCS)</h3><p>With this latest Composer release, you can save or load your checkpoints using OCI or GCS. We added direct support for <a href="https://www.oracle.com/cloud/storage/object-storage/">Oracle Cloud Infrastructure (OCI)</a> as an ObjectStore and support for Google Cloud Storage (GCS) via URI.</p><pre>from composer.trainer import Trainer<br><br># Checkpoint saving to Google Cloud Storage.<br>trainer = Trainer(<br>    model=model,<br>    save_folder=&quot;gs://my-bucket/{run_name}/checkpoints&quot;,<br>    run_name=&#39;my-run&#39;,<br>    save_interval=&quot;1ep&quot;,<br>    save_filename=&quot;ep{epoch}.pt&quot;,<br>    save_num_checkpoints_to_keep=0, # delete all checkpoints locally<br>    ...<br>)<br><br>trainer.fit()</pre><h3>Track your training metrics with MLflow logging</h3><p>We’ve added support for using <a href="https://www.mlflow.org/docs/latest/tracking.html">MLflow</a> to log experiment metrics and hyperparameters. Please see our <a href="https://docs.mosaicml.com/en/stable/trainer/logging.html">Logging</a> documentation for additional details.</p><pre>from composer.loggers import MLFlowLogger<br>from composer.trainer import Trainer<br><br>mlflow_logger = MLFlowLogger(experiment_name=mlflow_exp_name,<br>                             run_name=mlflow_run_name,<br>                             tracking_uri=mlflow_uri)<br>trainer = Trainer(..., loggers=[mlflow_logger])</pre><h3>Simplified console and progress bar logging</h3><p>To turn off the progress bar, set composer.Trainer(progress_bar=False). To turn on logging directly to the console, set composer.Trainer(log_to_console=True). To control the frequency of logging to console, set the Trainer argument console_log_interval (e.g. to 1ep or 1ba).</p><h3>Support for Mid-Epoch Resumption and more with the latest MosaicML Streaming Library v0.2 release</h3><p>We’ve added support in Composer for the latest v0.2 release of our <a href="https://github.com/mosaicml/streaming">Streaming</a> library. This includes awesome new features like instant mid-epoch resumption and deterministic shuffling, regardless of the number of nodes.</p><p><a href="https://github.com/mosaicml/streaming">MosaicML Streaming</a> is a PyTorch-compatible dataset that enables users to stream training data from cloud-based object stores. MosaicML Streaming can read files from local disk or from cloud-based object stores.</p><p><strong>Key Benefits of the MosaicML Streaming library include:</strong></p><ul><li>High performance, accurate streaming of training data from cloud storage</li><li>Efficiently train anywhere, independent of training data location</li><li>Cloud-native, no persistent storage required</li><li>Enhanced data security — data exists ephemerally on training cluster</li></ul><p>See the <a href="https://github.com/mosaicml/streaming/releases/tag/v0.2.0">MosaicML Streaming release notes</a> for more about the latest updates, and stay tuned for a blog coming up detailing more about the MosaicML Streaming Library!</p><h3>Load a 🤗 HuggingFace Model and Tokenizer out of a Composer checkpoint</h3><p>The model pre-training to fine-tuning workflow in Composer with HuggingFace models is now easier than ever! We’ve added a new utility to load a HuggingFace model and tokenizer out of a Composer checkpoint.</p><p>Getting started is easy: check out our <a href="https://docs.mosaicml.com/en/latest/examples/pretrain_finetune_huggingface.html">example notebook</a> for a full tutorial on pre-training and fine-tuning a HuggingFace transformer using the Composer library. Read the <a href="https://docs.mosaicml.com/en/latest/api_reference/generated/composer.models.HuggingFaceModel.html#composer.models.HuggingFaceModel.hf_from_composer_checkpoint">docs</a> for more details.</p><h3>Track Optimizer Specific Metrics with OptimizerMonitor (previously GradMonitor)</h3><p>We’ve renamed our GradMonitor callback to OptimizerMonitor, and added the ability to track optimizer-specific metrics. OptimizerMonitor computes and logs the L2 norm of gradients after the reduction of gradients across GPUs.</p><p>Check out the <a href="https://docs.mosaicml.com/en/latest/api_reference/generated/composer.callbacks.OptimizerMonitor.html">docs</a> for more details, and add to your code just like any other callback!</p><h3>New PyTorch and CUDA versions</h3><p>We’ve expanded our library of Docker images with support for PyTorch 1.13 + CUDA 11.7:</p><ul><li>mosaicml/pytorch:1.13.0_cu117-python3.10-ubuntu20.04</li><li>mosaicml/pytorch:1.13.0_cpu-python3.10-ubuntu20.04</li></ul><p>The mosaicml/pytorch:latest, mosaicml/pytorch:cpu_latest and mosaicml/composer:0.12.0 tags are now built from PyTorch 1.13 based images. Please see our <a href="https://hub.docker.com/orgs/mosaicml/repositories">DockerHub repository</a> for additional details.</p><h3>New community speedup algorithm: Gyro Dropout</h3><p>Special thank you to Junyeol Lee and Gihyun Park at BDSL in Hanyang University for the Composer implementation of <a href="https://proceedings.mlsys.org/paper/2022/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract.html">Gyro Dropout</a> and the accompanying documentation.</p><p>Gyro Dropout is a variant of dropout that improves the efficiency of training neural networks. Instead of randomly dropping out neurons in every training iteration, gyro dropout pre-selects and trains a fixed number of subnetworks. Gyro Dropout replaces implementations of torch.nn.Dropout. You can read more about Gyro Dropout in the Composer <a href="https://docs.mosaicml.com/en/latest/method_cards/gyro_dropout.html">method card</a>.</p><p>We’re always looking for community contributions for speedup methods for Composer! Whether you’re looking to contribute your own speed-up method from your published research or need to do a final class project on efficient machine learning and want ideas, reach out to <a href="http://sendto:community@mosaicml.com/">community@mosaicml.com</a> to get started.</p><h3>Learn more!</h3><p>Thanks for reading! If you’d like to learn more about Composer and to be part of the community, you are welcome to download <a href="https://github.com/mosaicml/composer">Composer</a> and try it out for your training tasks. As you try it out, come be a part of our community by engaging with us on <a href="https://twitter.com/MosaicML">Twitter</a>, joining our <a href="https://join.slack.com/t/mosaicml-community/shared_invite/zt-w0tiddn9-WGTlRpfjcO9J5jyrMub1dg">Slack channel</a>, or just <a href="https://github.com/mosaicml/composer">giving us a star on Github</a>.</p><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/composer-0-12-release"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=210799cffaa2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficiently Estimating Pareto Frontiers with Cyclic Learning Rate Schedules]]></title>
            <link>https://medium.com/@MosaicML/efficiently-estimating-pareto-frontiers-with-cyclic-learning-rate-schedules-3d50e6fe8e0b?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d50e6fe8e0b</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 23:40:34 GMT</pubDate>
            <atom:updated>2022-08-19T23:40:34.125Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQsFKsdWS_1_kdMK930JsA.png" /></figure><h4><em>Benchmarking the tradeoff between model accuracy and training time is computationally expensive. Cyclic learning rate schedules can construct a tradeoff curve in a single training run. These cyclic tradeoff curves can be used to evaluate the effects of algorithmic choices on network training efficiency.</em></h4><h3>Problem Statement: Producing Tradeoff Curves Efficiently</h3><p>At MosaicML, we design changes to the neural network training algorithm (which we call <em>speedup methods</em>) that lead to better tradeoffs between the cost of training a model and the quality of the trained model. We evaluate the efficacy of these interventions by looking at the tradeoff between training time and model quality. Concretely, we look at <em>tradeoff curves</em> like the one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*br0Bv0qCHaHwQAe4.png" /><figcaption><strong>Tradeoff curve for a baseline ResNet-50 model on ImageNet</strong>. Each point on the accuracy-time tradeoff curve is a separate network, and networks are identical except for training time. Note that overfitting seems to occur after ~128 epochs.</figcaption></figure><p>‍With this as background, our problem statement is as follows: <strong>Given a fixed recipe for training (e.g. architecture, objective function, regularization methods), how can we efficiently produce a tradeoff curve?</strong></p><p>One approach might be to sparsely sample points across different possible training durations and then interpolate. For example, we could train a network for three different durations (e.g. 30, 90, 270 epochs on an exponential scale) and fit a parameterized function to interpolate the rest of the curve. In this scenario, training would take 390 epochs total. We use this approach extensively in our <a href="https://app.mosaicml.com/explorer/imagenet">MosaicML Explorer</a> benchmarking across cloud instances, hardware choices, and speedup methods. However, the added expense of doing so was especially painful given that we performed thousands of training runs to collect this data. For the remainder of the blog, we refer to any tradeoff curve generated this way as the <strong>standard tradeoff curve</strong>.</p><p>Another approach might be to do one long training run (e.g. for 270 epochs) and sample points on the tradeoff curve <em>during</em> training. However, modern training best-practices involve gradually decaying the learning rate over the course of training. Many of the points sampled in this procedure will therefore be at intermediate states where training is not complete and the learning rate schedule is atypically high, leading to lower-than-normal model quality.</p><p>Neither of these approaches is satisfactory. The first approach is too computationally expensive, and the second produces low-fidelity results that are not indicative of standard performance.</p><h3>The Solution: Cyclic Learning Rates</h3><p>In this blog post, we show that multiplicative cyclic learning rate schedules can be used to construct an accuracy-time tradeoff curve in linear time, i.e., <strong>in a single training run</strong>, saving time and money. Although using this method necessarily changes the outcome of training, we still find that it provides similar-fidelity information about the relative efficacy of different speedup methods. This is particularly exciting because it means that ML researchers can characterize the accuracy-time tradeoff more efficiently, and therefore make meaningful comparisons between choices of architectures, hyperparameters and network training methods.</p><p><strong>We recommend that accuracy-time tradeoff curves should be standard practice for ML research (in addition to accuracy tables!). With cyclic learning rate schedules, generating tradeoff curves is not computationally expensive.</strong></p><h3>Background: Characterizing Training Performance with Pareto Frontiers</h3><p>At MosaicML, we care about speeding up neural network training. We have found it particularly useful to frame neural network training as a tradeoff between <em>accuracy</em> and <em>time</em> as depicted in the <em>tradeoff curve</em> below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rzafASMbETXk8xN0.png" /><figcaption><strong>A tradeoff curve showing how combinations of training methods lead to Pareto improvements [ResNet-50, ImageNet]:</strong> Training with Blurpool (BP), Channels Last (CL), Label Smoothing (LS) and MixUp (MX) leads to speedup and accuracy boost relative to the baseline.</figcaption></figure><p>We produce tradeoff curves by varying the length of training for each model configuration; longer training runs take more time (further to the right on the x-axis) but tend to reach higher quality (higher on the y-axis). A Pareto frontier is the set of all of the best possible tradeoffs between training time and accuracy, where any further attempt to improve one of these metrics worsens the other. For a fixed model and task configuration, this method of generating tradeoff curves is an estimate of the theoretical Pareto frontier. Constructing a tradeoff curve allows us to then make claims such as:</p><ul><li>For a desired accuracy A, the network need only be trained for a particular amount of time T</li><li>Given a time budget T, accuracy A can be achieved</li></ul><p>We consider one configuration to be better than another if the tradeoff curve it produces is further upward and to the left (e.g., the orange curve in the figure above). Accuracy-time tradeoff curves also allow us to make quantitative statements about different hyperparameter choices and training algorithms, for example:</p><ul><li>Training the same network with <a href="https://docs.mosaicml.com/en/latest/method_cards/blurpool.html">Blurpool</a> (Zhang 2019), <a href="https://docs.mosaicml.com/en/latest/method_cards/channels_last.html">Channels Last</a>, <a href="https://docs.mosaicml.com/en/latest/method_cards/label_smoothing.html">Label Smoothing</a> (Szegedy et al. 2015, Müller et al. 2015), and <a href="https://docs.mosaicml.com/en/latest/method_cards/mixup.html">MixUp</a> (Zhang et al. 2017) shifts the tradeoff curve left along the time axis and up along the quality axis, leading to a Pareto improvement in both accuracy and speedup relative to the baseline</li></ul><h3>Cyclic Learning Rate Schedules</h3><p>The learning rate at which a neural network is trained is often called “the most important hyperparameter” (Howard and Gugger 2020). In many modern implementations, the learning rate is a scalar value that is first increased with a linear warmup, and then decreased with a cosine decay. Historically (i.e. 5 years ago!), there was a flurry of work exploring alternative learning rate schedules from exponential decays to trapezoids. Cyclic learning rates fall on the more exotic end of the spectrum. [ <a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers#endnotes">1</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*dBT5q_NTLOE8WsZs.png" /><figcaption><strong>Example Monotonic (Cosine Decay) and Cyclic Learning Rate Schedules:</strong> Linear warm up with cosine decay (<em>left</em>), linear warm up followed by cyclic learning rate schedule with constant period of 25 epochs (<em>middle</em>), and linear warm up followed by multiplicative cyclic learning rate schedule with increasing periods of 8, 16 ,32, 64 and 128 epochs (<em>right</em>).</figcaption></figure><p>Cyclic learning rate schedules for CIFAR-10 and ImageNet were first proposed in Smith (2015). They essentially increase and decrease the learning rate periodically during training. While Smith (2015) proposed various sawtooth-like schedules, Loshchilov and Hutter (2016) showed that a schedule designed with cycles of a cosine decay followed by a restart led to competitive accuracy on CIFAR-100. The basic (noncyclic) cosine decay schedule also introduced by this paper is widely used today. Some variations of cosine decay with restarts include schedules with a constant period and schedules with periods that increase multiplicatively, which we call <strong>multiplicative cyclic learning rate (CLR) schedules. </strong>[ <a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers#endnotes">2</a>]</p><p>Cyclic learning rate schedules were popularized by <a href="https://www.fast.ai/">Fast.ai</a> (Howard and Gugger 2020), but are not widely used in the research community. This is likely due to the fact that, with current best practices on benchmarks like ImageNet, cyclic learning rate schedules do not necessarily lead to improved (or even commensurate) accuracy when compared to cosine decay schedules (data not shown). And while some of the early motivation for cyclic learning rates was to be able to interrupt training at any time and still return good performance (a.k.a. an “anytime” algorithm), anytime performance is not necessarily of interest for heavily studied benchmarks with known, standard hyperparameters. However, we suspected that cyclic learning rate schedules could be used to efficiently construct accuracy-time tradeoff curves.</p><p><strong>In the rest of this blog post, we show how you can use multiplicative cyclic learning rate (CLR) schedules to efficiently construct useful accuracy-time tradeoff curves for benchmarking ImageNet. We then explore how these curves change for combinations of regularization and speedup methods, such as </strong><a href="https://docs.mosaicml.com/en/latest/method_cards/blurpool.html"><strong>Blurpool</strong></a><strong>, </strong><a href="https://docs.mosaicml.com/en/latest/method_cards/channels_last.html"><strong>Channels Last</strong></a><strong>, </strong><a href="https://docs.mosaicml.com/en/latest/method_cards/label_smoothing.html"><strong>Label Smoothing</strong></a><strong>, and </strong><a href="https://docs.mosaicml.com/en/latest/method_cards/mixup.html"><strong>Mixup</strong></a><strong>. </strong>All experiments use ResNet-50 on ImageNet, and report top-1 validation accuracy.</p><h3>Cyclic Learning Rate Schedules with Constant Period Generate Suboptimal Tradeoff Curves</h3><p>We first used cyclic learning rate (CLR) schedules with constant periods, and examined the effect of varying the periods (10, 20, 30, 40, and 50 epochs). This led to an interesting trend, whereby cyclic learning rate schedules with shorter periods (e.g. 10 epochs, <em>gray line</em>) led to suboptimal tradeoff curves, while CLRs with longer periods (e.g. 50 epochs, <em>magenta line</em>) led to curves that were on the standard tradeoff curve.</p><p><strong>The bottom line: </strong>Cyclic learning rate schedules aren’t always good, and high-frequency cycling can lead to poor validation accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D-pzDZ639D0TS00a.png" /><figcaption><strong>Accuracy-time tradeoff curves can be constructed with cyclic learning rate schedules [ResNet-50, ImageNet].</strong> Cyclic learning rate (CLR) schedule with a <strong>constant period</strong> of 25 epochs (<em>top, left</em>) leads to a top-1 validation accuracy on ImageNet with ResNet-50 that is also cyclic (<em>bottom, left</em>). Accuracy-time tradeoff curves can be constructed with cyclic learning rate schedules of different constant periods, for example 10 epochs (<em>gray</em>), 20 epochs (<em>cyan</em>), 25 epochs (<em>crimson</em>), 40 epochs (<em>gold</em>) and 50 epochs (<em>purple</em>).</figcaption></figure><h3>Multiplicative Cyclic Learning Rate Schedules Efficiently Estimate Tradeoff Curves</h3><p>We then used multiplicative cyclic learning rate schedules with doubling periods, and found that they approximated the baseline <strong>standard tradeoff curve</strong> quite well. For the rest of this blog post, we refer to these curves as <strong>cyclic learning rate (CLR) tradeoff curves</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*khrRd0fZBelLmRUV.png" /></figure><p><strong>Accuracy-time tradeoff curves can be constructed with multiplicative cyclic learning rate schedules [ResNet-50, ImageNet].</strong> A cyclic learning rate (CLR) schedule with <strong>multiplicative periods</strong> of 8, 16, 32, 64 and 128 epochs (<em>top, left</em>) also leads to a validation accuracy that is cyclic (<em>bottom, left</em>). An accuracy vs. training time tradeoff curve can be constructed with a multiplicative cyclic learning rate schedule (CLR, <em>dashed line</em>), and is close to the baseline standard tradeoff curve (<em>solid line</em>). Note the spikes and drops in accuracy during cycling.</p><p>Specifically, we designed the learning rate schedule to linearly warm up for 8 epochs to a maximum value of 2.048, and then cycle with a cosine decay for increasing periods of 8, 16, 32, 64, 128 and 256 epochs, for a total of 512 epochs of training. We then stored maximum accuracy values at the end of each cycle, and plotted an accuracy-time tradeoff curve. This was straightforward to implement in PyTorch with the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html">CosineAnnealingWarmRestarts</a> learning rate scheduler. [ <a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers#endnotes">3</a>]</p><p>This CLR tradeoff curve matched our ResNet-50 ImageNet baseline standard tradeoff curve (with a difference of -0.084% on average across three seeds). This result held across choices of periods (e.g. starting with 5 epochs, then doubling to 10, 20 etc. epochs; data not shown).</p><p>‍ <strong>The bottom line: </strong>Multiplicative cyclic learning rate schedules with doubling periods can be used to construct accuracy-time tradeoff curves.</p><h3>Cyclic Learning Rate Tradeoff Curves Approximate Pareto Frontiers Across Training Methods</h3><p>There are a handful of demonstrated methods that modify the training procedure to achieve better tradeoffs between the final model quality and the time or cost to train the model. Some of these methods include <a href="https://docs.mosaicml.com/en/latest/method_cards/blurpool.html">Blurpool</a> (Zhang 2019), <a href="https://docs.mosaicml.com/en/latest/method_cards/channels_last.html">Channels Last</a>, <a href="https://docs.mosaicml.com/en/latest/method_cards/label_smoothing.html">Label Smoothing</a> (Szegedy et al. 2015, Müller et al. 2015), and <a href="https://docs.mosaicml.com/en/latest/method_cards/mixup.html">MixUp</a> (Zhang et al. 2017). In our benchmarking work for <a href="https://app.mosaicml.com/explorer/imagenet">MosaicML Explorer</a>, we wanted to find methods in the literature that led to clear Pareto improvements.</p><p>We therefore asked whether multiplicative cyclic learning rate schedules could be used to construct competitive accuracy-time tradeoff curves for separate methods such as Blurpool, Channels Last, Label Smoothing, and MixUp. We found that using multiplicative cyclic learning rate schedules allowed us to generate tradeoff curves of comparable quality to standard tradeoff curves, but at substantial time savings (~2x).</p><p><strong>Blurpool, Channels Last</strong></p><p>We found that cyclic learning rate tradeoff curves were able to match baseline standard tradeoff curves for Blurpool and Channels Last almost exactly (Blurpool exceeded by 0.07%, and Channels by 0.09%, on average). Note that the simple Channels Last modification on RTX 3080 GPUs led to a leftward shift of the tradeoff curve along the time axis ( <em>solid cyan line</em>); thus training a network for 512 epochs took approximately 30 hours instead of 40 hours. Also note that Blurpool led to strong improvements for <em>shorter</em> training times (e.g. 16 epochs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzLD_IlkvlPGAcH31kWZjg.png" /><figcaption><strong>Cyclic Learning Rate (CLR) Tradeoff Curves for Blurpool and Channels Last [ResNet-50, ImageNet].</strong> The Blurpool (BP) tradeoff curve constructed with a cyclic learning rate schedule (<em>dashed gray line, left</em>) matches the Blurpool standard tradeoff curve. The Channels Last (CL) tradeoff curve (<em>dashed cyan line, right</em>) matches the Channels Last standard tradeoff curve.</figcaption></figure><p><strong>Label Smoothing, Mixup</strong></p><p>Cyclic learning rate tradeoff curves were slightly below baseline standard tradeoff curves for Label Smoothing and MixUp, on average differing roughly by -0.44% for Label Smoothing and -0.39% for MixUp. Importantly, CLR tradeoff curves for these two cases still trended with the standard tradeoff curves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XlpW0cnOg14kW72qmGbN3g.png" /><figcaption><strong>Cyclic Learning Rate (CLR) Tradeoff Curves for Label Smoothing and MixUp [ResNet-50, ImageNet].</strong> The Label Smoothing CLR tradeoff curve trended with the Label Smoothing standard tradeoff curve, with improvements over baseline for longer training times (<em>left</em>). The same was true for the MixUp CLR tradeoff curve (<em>right</em>).</figcaption></figure><p><strong>Blurpool + Channels Last, Label Smoothing + MixUp</strong></p><p>Constructing a separate tradeoff curve for every training method and/or combination of methods (for example Label Smoothing + Mixup) is also combinatorially expensive. We therefore asked whether cyclic learning rate schedules could accurately construct tradeoff curves for combinations of methods. We trained ResNet-50 with both Blurpool and Channels Last (BP + CL), and found that the cyclic learning rate schedule tradeoff curve matched the Pareto frontier for shorter training durations (on average, exceeded by 0.08%). This was not surprising, as the CLR tradeoff curves for Blurpool and Channels Last individually led to improvements with respect to the baseline standard tradeoff curve.</p><p>We then trained ResNet-50 with the combined methods of Label Smoothing and MixUp (LS + MX), and found that CLR tradeoff curves trended with standard tradeoff curves, but were slightly lower (by -0.46% on average). This made sense considering the results for Label Smoothing and MixUp individually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlOXq227TwW_A0DP-GiHPw.png" /><figcaption><strong>Cyclic Learning Rate Tradeoff Curves for Blurpool + Channels Last (BP+CL), and Label Smoothing + MixUp (LS+MX) [ResNet-50, ImageNet].</strong> BP+CL leads to a Pareto improvement early in training (<em>left</em>), while LS+MX leads to a Pareto improvement late in training (<em>right</em>).</figcaption></figure><p><strong>Blurpool + Channels Last + Label Smoothing + MixUp</strong></p><p>Finally, we trained ResNet-50 with the combined methods of Blurpool, Channels Last, Label Smoothing and MixUp (BL + CL + LS + MX), and found that CLR tradeoff curves trended with the standard tradeoff curves ( <em>crimson dashed line</em>, lower by -0.24% on average).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/0*m8KLjKwwOrxYdp2v.png" /><figcaption><strong>Cyclic Learning Rate Tradeoff Curves for Blurpool + Channels Last + Label Smoothing + MixUp (BP+CL+LS+MX) [ResNet-50, ImageNet].</strong> Combining training methods BP+CL+LS+MX leads to a Pareto improvement both early in training and late in training.</figcaption></figure><p>We then plotted the total wall clock time for standard tradeoff curves and cyclic learning rate (CLR) tradeoff curves with six sampled points at 16, 32, 64, 128, 256 and 512 epochs. As expected, standard tradeoff curve total wall clock time was almost twice as long as CLR tradeoff curve total wall clock time across all training methods and combinations of methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/0*Ir8EilA6gQjOul8o.png" /><figcaption><strong>Multiplicative cyclic learning rate (CLR) schedules can be used to construct accuracy-time tradeoff curves in less time than naive sampling. </strong>Total wall clock time for standard tradeoff curves constructed with six sampled points at 16, 32, 64, 128, 256 and 512 epochs is almost twice as large as CLR tradeoff curves sampled at the same six points.</figcaption></figure><p>The ratio of the time to construct a standard tradeoff curve versus the time to construct the cyclic tradeoff curve (where both tradeoff curves sample the same exact points) can be expressed in the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fUvN_EzYr6r-8KHd.png" /></figure><p>‍Here <em>a</em> is the starting number of epochs, <em>r</em> is the multiplicative factor, and <em>n</em>-1 is the total number of sampled points. When <em>r</em>=2, this ratio is approximately 2, meaning that using a cyclic learning rate schedule will lead to ~2x speedup.</p><p>We also found that CLR tradeoff curves matched relative trends between standard tradeoff curves across training methods. For example, MixUp (MX) has worse accuracy relative to baseline for shorter training times of 16 epochs (&lt;70%), and performs better than Label Smoothing (LS) for longer training times of 512 epochs (above 78.5%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKbjVa_AAd_5Qp8rciugxQ.png" /><figcaption><strong>Cyclic Learning Rate (CLR) tradeoff curves capture trends across training methods [Resnet-50, ImageNet].</strong>Relative ordering of tradeoff curves between training methods is preserved between standard tradeoff curves (<em>left</em>) and cyclic learning rate (CLR) tradeoff curves (<em>right</em>).</figcaption></figure><p>When comparing training methods such as Label Smoothing and MixUp, can we really trust that improvements in the Pareto frontier over the baseline will be matched by a tradeoff curve constructed with a cyclic learning rate? One way to assess this is simply to compare the relative improvements over baseline for each method. When we subtract the Blurpool standard tradeoff curve from the Baseline standard tradeoff curve, we can see that early on in training Blurpool provides a 2% accuracy boost, but provides no accuracy boost at 256 epochs ( <em>gray solid line</em>). Subtracting the Blurpool CLR tradeoff curve from the Baseline CLR tradeoff curve shows the same trend ( <em>gray dashed line</em>). We can also take the cosine similarity between these two lines. Across the training methods and combinations of training methods, we can see that CLR tradeoff curves match the trends nicely (cosine similarity BP=0.972, CL=-0.273, LS=0.992, MX=0.976, BP+CL=0.952, BP+CL=0.989, BP+CL+LS+MX=0.997).</p><p>This is all to say that if a method such as MixUp shifts the standard tradeoff curve, it also correspondingly shifts the CLR tradeoff curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gglWzufdlC_SAldL.png" /><figcaption><strong>Improvements over baseline are captured by cyclic learning rate tradeoff curves [ResNet-50, ImageNet].</strong> The difference between standard tradeoff curves (e.g. Blurpool minus Baseline) is matched by the difference between CLR tradeoff curves.</figcaption></figure><p><strong>The bottom line: </strong>Tradeoff curves constructed using multiplicative cyclic learning rate schedules can provide information on the relative performance of speedup methods.</p><h3>Why This Works</h3><p>It is somewhat surprising that multiplicative cyclic learning rates are able to approximate Pareto frontiers when constant period cyclic learning rates do not. Why does this work, particularly for increasing periods? One possibility might have to do with different phases of learning (Golatkar et al., 2019, Frankle et al., 2020). For example, rapid changes in learning rate might help find better minima early in training (10–20 epochs), while very slow changes in learning rate might help find minima in already “good” basins late in training (&gt;90 epochs). As training continues, it might become more difficult to find better minima, implying that it might be beneficial to take longer for each new cycle. [ <a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers#endnotes">4</a>]</p><p>We were particularly intrigued by the precipitous changes in accuracy during the learning rate cycles; we found this to be a clear illustration of explore versus exploit dynamics happening across different minima. This has been noted by multiple studies, and is one of the motivating factors for ensembling methods (e.g. Huang et al., 2017) and Stochastic Weight Averaging (SWA, Izmailov et al., 2018).</p><h3>Future Directions</h3><p>In many instances, the CLR tradeoff curve underestimated the standard tradeoff curve by a margin of ~0.5% validation accuracy. There are many possible variations to our approach that might improve this gap. Some ideas include:</p><ul><li>Allowing the entire cyclic learning rate schedule to decay over the course of learning. This would mean decreasing the maximum learning rate after each reset. This is one of many schedule variations explored by the literature (Schmidt et al. 2021)</li><li>Applying Stochastic Weight Averaging (SWA, Izmailov et al., 2018) at the end of each cycle. This would entail, at the end of each trough, cycling the learning rate (even more!) at low amplitude and at high frequency, and averaging the weights. Such an approach might boost the validation accuracy and lead to Pareto optimality in some instances. [<a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers#endnotes">5</a>]</li><li>Repeating linear warm up instead of a hard reset, although it is unclear if this would improve accuracy. We found that sawtooth/triangular cyclic learning rate schedules <em>did not</em> get close to the Pareto frontier for ResNet-50 on ImageNet (data not shown).</li><li>Experimenting with other multiplier values. The behavior of the PyTorch <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html">CosineAnnealingWarmRestarts</a> scheduler makes it difficult to control the spacing of points sampled along the Pareto curve, in part due to the constraint of epoch-wise (and not step-wise) scheduling. We hope to circumvent this problem in future versions of <a href="https://github.com/mosaicml/composer">Composer</a>.</li><li>Applying a cyclic learning rate to other hyperparameters such as weight decay, batch size, and gradient clipping (in line with Smith 2022).</li></ul><p>Our cyclic learning rate approach to constructing tradeoff curves worked nicely for training methods such as Blurpool and Label Smoothing. However, in the current implementation of <a href="https://github.com/mosaicml/composer">MosaicML Composer</a> (or PyTorch), it won’t work for methods that have their own independent schedules such as <a href="https://docs.mosaicml.com/en/latest/method_cards/layer_freezing.html">Layer Freezing</a> (Brock et al., 2017), <a href="https://docs.mosaicml.com/en/latest/method_cards/swa.html">Stochastic Weight Averaging (SWA)</a> (Izmailov et al., 2018), and <a href="https://docs.mosaicml.com/en/latest/method_cards/progressive_resizing.html">Progressive Resizing</a> (Fast.ai). It <em>can</em> work in principle, however, if the schedules for methods such as Progressive Resizing cycle with the learning rate schedule. We hope to implement this in future versions of the library.</p><p>The use of Pareto frontiers in ML research is surprisingly limited, and is mostly restricted to the subfield of multi-objective optimization (MOO) for multitask learning (some recent examples include Lin et al., 2020 and Navon et al., 2020). [ <a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers#endnotes">6</a>] At MosaicML, we have found that thinking about Pareto optimality is useful for framing more fundamental questions in neural network training.</p><p>Our results primarily hold for convolutional networks (ResNet-50, ResNet-101) applied to computer vision tasks (CIFAR-10 and ImageNet). Can the same approach be applied to NLP? A lot of the recent scaling law papers find a linear boundary based on single, full training runs, where the scaling is based on model size (Henighan et al., 2020). However, it is likely that each model size has an associated quality-time Pareto frontier. Is it possible to shift the scaling laws to have better exponents? Some of our preliminary experimentation is inconclusive. We hope to explore this in future work.</p><h3>Takeaways</h3><p>We have shown here that multiplicative cyclic learning rate schedules can be used to construct an accuracy-time tradeoff curve in a single training run, saving time and money. This is a practical means of assessing changes in tradeoff curves caused by different architectures, hyperparameters, and speedup methods.</p><p>We hope that this is a small step towards framing ML efficiency in terms of tradeoff curves and Pareto optimality.</p><p><em>This work was done as part of a research internship with MosaicML. Thanks to Cory Stephenson, Davis Blalock, Kobie Crawford and Jonathan Frankle for fruitful discussions.</em></p><h3>‍References</h3><ol><li>Blalock, D., Ortiz, J. J. G., Frankle, J., &amp; Guttag, J. (2020). <a href="https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html">What is the state of neural network pruning?</a>. arXiv preprint arXiv:2003.03033.</li><li>Blalock, D., Carbin, M., Florescu, L., Frankle, J., Leavitt, M., Lee, T., Nadeem, M., Portes, J., Rao, N., Seguin, L., Stephenson, C., Tang, H., Venigalla A., (2021) On Evaluating and Improving the Efficiency of Deep Networks Whitepaper</li><li>Brock, A., Lim, T., Ritchie, J. M., &amp; Weston, N. (2017). <a href="https://arxiv.org/abs/1706.04983">Freezeout: Accelerate training by progressively freezing layers</a>. arXiv preprint arXiv:1706.04983.</li><li>Fedyunin, V. (2021) <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html">Channels Last Memory Format in PyTorch</a> (PyTorch documentation)</li><li>Frankle, J., Schwab, D. J., &amp; Morcos, A. S. (2020). <a href="https://arxiv.org/abs/2002.10365">The early phase of neural network training</a>. arXiv preprint arXiv:2002.10365.</li><li>Golatkar, A. S., Achille, A., &amp; Soatto, S. (2019). <a href="https://arxiv.org/abs/1905.13277">Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence</a>. Advances in Neural Information Processing Systems, 32.</li><li>Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., … &amp; McCandlish, S. (2020). <a href="https://arxiv.org/abs/2010.14701">Scaling laws for autoregressive generative modeling</a>. arXiv preprint arXiv:2010.14701.</li><li>Howard, J., &amp; Gugger, S. (2020). <a href="https://www.mdpi.com/2078-2489/11/2/108/htm?ref=https://githubhelp.com">Fastai: a layered API for deep learning.</a> Information, 11(2), 108.</li><li>Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., &amp; Weinberger, K. Q. (2017). <a href="https://arxiv.org/abs/1704.00109">Snapshot ensembles: Train 1, get m for free</a>. arXiv preprint arXiv:1704.00109.</li><li>Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., &amp; Wilson, A. G. (2018). <a href="https://arxiv.org/abs/1803.05407">Averaging weights leads to wider optima and better generalization</a>. arXiv preprint arXiv:1803.05407.</li><li>Lin, X., Yang, Z., Zhang, Q., &amp; Kwong, S. (2020). <a href="https://arxiv.org/abs/2010.06313">Controllable pareto multi-task learning</a>. arXiv preprint arXiv:2010.06313.</li><li>Loshchilov, I., &amp; Hutter, F. (2016). <a href="https://arxiv.org/abs/1608.03983">Sgdr: Stochastic gradient descent with warm restarts</a>. arXiv preprint arXiv:1608.03983.</li><li>Loshchilov, I., &amp; Hutter, F. (2017). <a href="https://arxiv.org/abs/1711.05101">Decoupled weight decay regularization. arXiv preprint</a> arXiv:1711.05101.</li><li>Müller, R., Kornblith, S., &amp; Hinton, G. E. (2019). <a href="https://proceedings.neurips.cc/paper/2019/hash/f1748d6b0fd9d439f71450117eba2725-Abstract.html">When does label smoothing help?</a>. Advances in neural information processing systems, 32.</li><li>Navon, A., Shamsian, A., Chechik, G., &amp; Fetaya, E. (2020). <a href="https://arxiv.org/abs/2010.04104">Learning the pareto front with hypernetworks</a>. arXiv preprint arXiv:2010.04104.</li><li>Schmidt, R. M., Schneider, F., &amp; Hennig, P. (2021, July). <a href="http://proceedings.mlr.press/v139/schmidt21a.html">Descending through a crowded valley-benchmarking deep learning optimizers.</a> In International Conference on Machine Learning (pp. 9367–9376). PMLR.</li><li>Smith, L. N. (2015). <a href="https://arxiv.org/pdf/1506.01186.pdf%5D">Cyclical Learning Rates for Training Neural Networks</a>. arXiv preprint arXiv:1506.01186. (See also updated version Smith 2017 in IEEE WACV)</li><li>Smith, L. N. (2022). <a href="https://arxiv.org/abs/2202.08835">General Cyclical Training of Neural Networks</a>. arXiv preprint arXiv:2202.08835.</li><li>Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., &amp; Wojna, Z. (2016). <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html">Rethinking the inception architecture for computer vision.</a> In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818–2826).</li><li>Wu, Y., Liu, L., Bae, J., Chow, K. H., Iyengar, A., Pu, C., … &amp; Zhang, Q. (2019, December). <a href="https://ieeexplore.ieee.org/abstract/document/9006104">Demystifying learning rate policies for high accuracy training of deep neural networks</a>. In 2019 IEEE International conference on big data (Big Data) (pp. 1971–1980). IEEE.</li><li>You, K., Long, M., Wang, J., &amp; Jordan, M. I. (2019). <a href="https://arxiv.org/pdf/1908.01878.pdf">How does learning rate decay help modern neural networks?</a>. arXiv preprint arXiv:1908.01878.</li><li>Zhang, H., Cisse, M., Dauphin, Y. N., &amp; Lopez-Paz, D. (2017). <a href="https://arxiv.org/abs/1710.09412">mixup: Beyond empirical risk minimization.</a> arXiv preprint arXiv:1710.09412. (Published in ICLR 2018)</li><li>Zhang, R. (2019). <a href="http://proceedings.mlr.press/v97/zhang19a.html">Making convolutional networks shift-invariant again</a>. In International conference on machine learning (pp. 7324–7334). PMLR</li></ol><h3>Experimental Details</h3><p>We trained ResNet-50 on ImageNet with noncyclic and cyclic learning rate schedules for various training durations. All ImageNet experiments used 8 RTX 3080s with mixed precision training, 8 dataloader workers per GPU, a batch size of 2048, and a gradient accumulation of 4. We used SGDW (Loshchilov et al. 2017) with a maximum learning rate of 2.048, momentum of 0.875, and weight decay 5.0e-4. All runs began with a linear warm-up over 8 epochs to the maximum learning rate, and all learning rate schedules were stepwise and not epoch-wise. Training runs were done across 1–3 seeds for each data point. Experiments using cosine decay learning rate schedules were run using the CosineAnnealingLR scheduler in PyTorch, while cyclic learning rate (CLR) schedules were implemented using the PyTorch CosineAnnealingWarmRestarts scheduler. In select experiments, training methods such as <a href="https://docs.mosaicml.com/en/latest/method_cards/blurpool.html">Blurpool</a> (Zhang 2019), <a href="https://docs.mosaicml.com/en/latest/method_cards/channels_last.html">Channels Last</a>, <a href="https://docs.mosaicml.com/en/latest/method_cards/label_smoothing.html">Label Smoothing</a> (Szegedy et al. 2015, Müller et al. 2015) and <a href="https://docs.mosaicml.com/en/latest/method_cards/mixup.html">MixUp</a> (Zhang et al. 2017) were also implemented using the MosaicML <a href="https://github.com/mosaicml/composer">Composer</a> library for PyTorch.</p><h3>Endnotes</h3><ol><li>For a review of different shapes, see Table 3 of Appendix A in “Descending through a crowded valley — benchmarking deep learning optimizers”, Schmidt et al. 2021, and Wu et al. 2019.</li><li>In the original paper, the schedule is called SGDR, i.e. Stochastic Gradient Descent with Restarts. In PyTorch, the scheduler is called <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html">CosineAnnealingWarmRestarts</a></li><li>What we are essentially doing is chaining training runs together, by training for 16 epochs, then training for 32 epochs using the weights from the previous model for initialization, and so on to build up the curve.</li><li>See You et al., 2019 for a more nuanced understanding of the effects of learning rate decay</li><li>Interestingly, the original paper by Loshchilov and Hutter 2016 found a boost in accuracy when they combined a cyclic learning rate schedule with ensembling similar to Huang et al. 2017</li><li>Our approach here might be considered an A Posteriori MOO method.</li></ol><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/efficiently-estimating-pareto-frontiers"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d50e6fe8e0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Best Practices for Efficient Model Training]]></title>
            <link>https://medium.com/@MosaicML/5-best-practices-for-efficient-model-training-21faa1ff2606?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/21faa1ff2606</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-network-training]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 23:09:35 GMT</pubDate>
            <atom:updated>2022-08-19T23:09:35.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1bWLEFXnwjoKi6mgpV3eDg.jpeg" /></figure><p>In the course of our research and product development we’ve codified a number of best practices for efficient CNN training, and we’d like to share some of them with you here.</p><h3>tl;dr</h3><p>Train your CNN a lot faster by using <a href="https://github.com/mosaicml/composer">Composer</a> (our library for efficient training) and the <a href="https://hub.docker.com/r/mosaicml/composer">MosaicML docker image</a>, and enabling channels last via the command line argument “--algorithms channels_last”.</p><h3>Introduction</h3><p>MosaicML’s goal is to make neural network training more efficient through better algorithms and software. We do empirical research to understand the training process and develop methods that reduce the time or cost of training. We have already achieved speedups and cost reductions on standard benchmarks of 3.4x for vision and 1.6x for NLP. Our goal is to achieve 4x speedups and cost reductions per year, and repeat this feat every year.</p><p>In the course of our research and product development, we’ve codified a number of best practices for efficient training, and we’d like to share some of them with you here. Most of the best practices we discuss here are specific to training CNNs, but stay tuned for blog posts on more general and NLP-specific best practices. Some of the best practices in this post might be familiar to seasoned ML practitioners, but you’d be surprised at how many well-known papers and popular software libraries don’t make use of them. Whenever possible, we provide details or links on how to implement these best practices. But we’ve also saved you the work of implementing them by bringing them all together in one place: <a href="https://github.com/mosaicml/composer">Composer</a>, our library for efficient training.</p><p>Without further ado, let’s get started!</p><h3>1. No Mixed Opinions on Mixed Precision Training</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*D6ZLUfBNBPFy0OSM.png" /></figure><p>Most deep learning frameworks use 32-bit floating point (FP32, or “full precision”) for all values-model weights, activations, gradients, and optimizer state-and arithmetic. It’s possible to speed up training by reducing the precision of all values and operations to 16-bit floating point (FP16, or “half precision”), but this can seriously compromise model quality. However, reducing the precision of only <em>some</em> values and operations to FP16 can speed up training without any reduction in model quality, leading to substantial efficiency gains. This mix of FP32 and FP16 is referred to as <a href="https://arxiv.org/abs/1710.03740">Mixed Precision training</a>.</p><p>An easy way to perform Mixed Precision training is with <a href="https://developer.nvidia.com/blog/mixed-precision-training-deep-neural-networks/">NVIDIA’s AMP</a> (Automatic Mixed Precision) library-which is <a href="https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/">integrated with PyTorch</a>. AMP works by storing a primary copy of the model weights in FP32, then making an FP16 copy of the model and performing the full forward- and back-propagation in FP16. The weight gradients are then converted to FP32 before the primary model weights are updated. This final detail-doing the weight update in FP32-is critical for good results.</p><p>Model quality is <a href="https://arxiv.org/abs/1710.03740">unaffected</a> by mixed precision training, and it yields substantial benefits: training can be accelerated by 2x-4.5x, and memory requirements can be reduced by nearly 2x relative to FP32 training. The increased training speed is due to the fact that the speed of multiplying two numbers is approximately proportional to the square of the number bits. The memory reduction is because FP16 numbers are half the size of FP32 numbers. These benefits make mixed precision training especially valuable for training larger models and training models faster.</p><p><strong>AMP is toggled by a </strong><a href="https://github.com/mosaicml/composer/blob/main/composer/yamls/models/resnet50.yaml#L58"><strong>single hyperparameter</strong></a><strong> (precision: amp in a config file or --precision amp on the command line) and enabled by default in </strong><a href="https://github.com/mosaicml/composer"><strong>Composer</strong></a><strong>.</strong></p><h3>2. Channels Last for Training CNNs Fast</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*frWmHP2tKqK29sEh.jpeg" /></figure><p><a href="https://arxiv.org/abs/2010.11929">Vision transformers</a> might be the hot new thing in computer vision research, but convolutional neural networks (CNNs) are still very much <a href="https://arxiv.org/abs/2103.07579">alive</a> and <a href="https://arxiv.org/abs/2110.00476">kicking</a>, so there’s good reason to care about training them more efficiently.</p><p>Convolutions typically operate on four-dimensional tensors, where the dimensions correspond to a batch of N samples (e.g. images), C channels (e.g., red, green, and blue color channels in the input), and H x W feature maps (e.g., the individual pixels of these channels). For example, a batch of 128 RGB images of size 224 x 256 pixels would be represented as a 128 (images) x 3 (RGB) x 224 (pixel width) x 256 (pixel height) tensor. Most deep learning libraries store tensors in memory in NCHW format, that is, with width as the dimension of the tensor that is stored contiguously in memory. But modern NVIDIA GPUs — which are the most common hardware for training deep learning models — perform convolutions in NHWC format (i.e. with channels last). This means that NCHW tensors need to be transposed to NHWC before each convolution. Storing as NHWC removes the need for this transposition and saves time and compute.</p><p>NVIDIA <a href="https://docs.nvidia.com/deeplearning/performance/dl-performance-convolutional/index.html#tensor-layout">reports</a> 1.15x to 1.6x performance improvement (TFLOPS) when using channels-last, and <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html#performance-gains">this PyTorch tutorial</a>reports a 1.22x throughput (images/second) improvement. Our in-house benchmarks show that channels last can improve throughput by 1.2x-1.3x, depending on the hardware. This makes channels last a free lunch for anyone training a model with convolutional layers on a Volta architecture (V100) or newer NVIDIA GPU.</p><p><a href="https://docs.mosaicml.com/en/stable/method_cards/channels_last.html"><strong>Channels last</strong></a><strong> is one of the many speedup methods included in </strong><a href="https://github.com/mosaicml/composer"><strong>Composer</strong></a><strong>, where it can be enabled via a single hyperparameter (if applied at the command line: “--algorithms channels_last”).</strong></p><h3>3. Mind Your Step: Stepwise Learning Rate Schedules</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*TmHOJxZemAG5Ufm3.jpeg" /></figure><p>Learning rate (LR) scheduling — adjusting the learning rate over the course of training — is essential for achieving state-of-the-art results with deep networks. There are lots of different LR schedule patterns to follow (see this <a href="https://paperswithcode.com/methods/category/learning-rate-schedules">handy list</a> from Papers With Code), and each of these schedules can be enacted at different time resolutions. Just as wall-clock time can be measured in different units like minutes, hours, and days, training can be subdivided into individual <em>steps</em> (a single weight update) and <em>epochs</em> (enough steps to pass through the entire dataset). It is common to schedule the learning rate at an epoch resolution, which is to say that adjustments to the learning rate happen only once per epoch.</p><p>However, we at MosaicML have found that using a stepwise LR schedule resolution (i.e., changing the learning rate according to the schedule after each step) yields superior performance compared to epochwise resolution. Specifically, we found that stepwise LR schedules can yield 0.2–0.4% accuracy improvements for ResNet-50 trained on ImageNet compared to otherwise-identical models trained with epochwise schedules (see our training recipe here). Because training speed and accuracy are <a href="https://www.mosaicml.com/blog/methodology">often interchangeable</a>, this accuracy improvement translates into a training speed improvement: A ResNet50 trained with stepwise LR schedule trains to a target accuracy about 1.06x faster than an equivalent model trained with epochwise resolution.</p><p>The learning rate scheduler implementations in most deep learning frameworks (e.g. PyTorch, TensorFlow) are implemented assuming epochwise resolution by default, putting the burden on the user to convert between steps and epochs. <a href="https://docs.mosaicml.com/en/stable/optim.html#composer-optim-scheduler">Composer alleviates this headache</a> by making it possible to describe learning rate schedules at one resolution but execute them at another. For example, a user can describe a learning rate schedule in epochs (e.g. warmup for 5 epochs, cosine decay for 95 epochs), which is often the more intuitive and convenient resolution, but under the hood Composer executes the learning rate schedule in steps for superior performance.</p><p><strong>Stepwise resolution for LR schedules is toggled by a </strong><a href="https://github.com/mosaicml/composer/blob/d17e69f274bff129110c8fba29682c7c091a261a/composer/yamls/models/resnet50.yaml#L30"><strong>single hyperparameter</strong></a><strong> (--schedulers.{scheduler_name}.interval step) and </strong><a href="https://docs.mosaicml.com/en/stable/optim.html#composer-optim-scheduler"><strong>enabled by default</strong></a><strong> in </strong><a href="https://github.com/mosaicml/composer"><strong>Composer</strong></a><strong>.</strong></p><h3>4. Go Easy on the Image Augmentations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*OjcuvE8P2V4qcNwj.png" /></figure><p>Data augmentation is a technique in which new data samples are created by modifying existing data samples, for example by rotating or changing the contrast of an image. It is a <a href="https://link.springer.com/article/10.1186/s40537-019-0197-0?code=a6ae644c-3bfc-43d9-b292-82d77d5890d5">commonly-used technique</a> for improving the performance of deep learning models for computer vision. Data augmentation provides two benefits: it increases the effective number of data samples, and it can allow the model to learn to be invariant to the augmentations. Using image classification as an example, a good image classifier should recognize that an image of a dog should still be classified as a dog even after the image has been translated or flipped left-right.</p><p>Data augmentation for images (i.e. Image augmentation) is an essential part of most training recipes for reaching state-of-the-art performance in many computer vision tasks. Popular image augmentation schemes include AutoAugment, RandAugment,and AugMix, all of which perform many image augmentations in sequence. Image augmentations are by default performed by the CPU in standard deep learning frameworks. But this can have unintended consequences on training speed.</p><p>We at MosaicML conducted a series of experiments to examine the efficiency of image augmentation techniques. We found that image augmentation techniques do indeed increase accuracy on a per-step or per-epoch basis-it takes fewer steps to get to the same performance compared to not using augmentation techniques-but these techniques are often so CPU-intensive that they substantially increase the time per step. The end result was that the wall clock time to train to a given accuracy was actually increased by 1.5x-8x compared to training without image augmentations, depending on the hardware configuration and augmentation scheme. This means you’re bottlenecked on something other than your expensive GPU, in effect putting it to waste. Stay tuned for a deep-dive blog post about image augmentation soon!</p><p>The bottom line when it comes to the impact of image augmentations on training time is:</p><ul><li>Use augmentation schemes that are less CPU-intensive, such as TrivialAugment (which is like RandAugment but involves fewer augmentations per image).</li><li>Use hardware configurations with more CPU:GPU compute power (e.g. many CPUs per GPU, or relatively older/underpowered GPUs).</li><li>Train larger models. This lowers GPU throughput and gives the CPU more time to keep up.</li><li>Perform image augmentations on the GPU using a library such as <a href="https://kornia.github.io/">Kornia</a> or <a href="https://github.com/NVIDIA/DALI">DALI</a>.</li></ul><h3>5. Soften the blow of image processing with Pillow-SIMD</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*7sc6-g62s4dafCy2.jpeg" /></figure><p>Deep learning frameworks use third-party libraries for CPU image processing. These libraries handle everything from low-level operations like JPEG decoding to high-level operations like data augmentation. Pytorch and Keras do this using the <a href="https://pillow.readthedocs.io/en/stable/">Pillow</a> library by default, but we were able to speed up training of ResNet50 on ImageNet by 1.2x-2x (depending on the hardware platform and training recipe) by using <a href="https://github.com/uploadcare/pillow-simd">Pillow-SIMD</a>, a drop-in replacement for Pillow. Pillow-SIMD accelerates image processing by parallelizing certain operations. This in turn can speed up training because CPU image processing often imposes a bottleneck on training, as discussed in the previous section. In scenarios in which CPU image processing may not impose a bottleneck, such as training a large ViT or ResNet-152, we might see no throughput change from switching to Pillow-SIMD.</p><p>While it’s quite easy to install Pillow-SIMD, our <a href="https://hub.docker.com/r/mosaicml/composer">Composer docker image</a> comes loaded with it already, giving you one less thing to worry about.</p><h3>Conclusion</h3><p>We hope these best practices speed up your model training. Stay tuned for more dispatches from MosaicML on other topics in efficient ML! And as always, we’d love to <a href="https://twitter.com/MosaicML">answer your questions</a>, <a href="https://github.com/mosaicml/composer">accept your pull requests</a>, and <a href="https://mosaicml.com/jobs">have you join the team</a>!</p><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/5-best-practices-for-efficient-model-training"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21faa1ff2606" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Methodology]]></title>
            <link>https://medium.com/@MosaicML/methodology-64097877a20e?source=rss-eb63b33497b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/64097877a20e</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[research-methods]]></category>
            <dc:creator><![CDATA[MosaicML]]></dc:creator>
            <pubDate>Thu, 04 Aug 2022 21:20:00 GMT</pubDate>
            <atom:updated>2022-08-04T21:48:46.258Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2A3U2yZw4zGuIgRjWjujA.jpeg" /></figure><p>Authors: The MosaicML Research Team</p><p><strong>At MosaicML, we’re working on making neural network training more efficient algorithmically. In this post, we describe this research problem and how we’re solving it</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ascJPXd6tHRw6ZLVTRwXDQ.png" /></figure><p>At MosaicML, our mission is to reduce the cost of training neural networks. We do so by modifying the training process at the algorithmic level: changing the details of the training recipe. We believe that the existing way of training neural networks is inefficient but that, by studying how neural networks behave in practice, we can leverage those findings to develop better training algorithms and significantly reduce the cost of training.</p><p>We develop that modify the training procedure to achieve better tradeoffs between the final model quality and the time or cost to train the model. The end result is that, to train a model to a particular level of quality, we can get there in less time or for fewer dollars. These methods change the training procedure in a variety of ways: for example, altering the data or the order in which they are presented to the model; tweaking the structure of the model; and changing the way forward propagation and back propagation take place.</p><p>Over the past few months, we have pursued this approach on common settings for computer vision ( <a href="https://arxiv.org/abs/1512.03385">ResNets on ImageNet</a>, <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/nnUNet">UNet on BraTS</a>) and natural language processing ( <a href="https://openai.com/blog/better-language-models/">Transformer/GPT language models</a>). We carefully evaluated 20 different methods and how they compose, and we implemented them in our <a href="https://github.com/mosaicml/composer">Composer library</a>. The compositions in this library have allowed us to achieve training speedups and cost reductions of 2.9X on ResNet-50 on ImageNet, 3.5X on ResNet-101 on ImageNet, and 1.7X on the GPT-125 language models (as compared to the optimized baselines on 8xA100s on AWS), all while achieving the same model quality as the baselines. To make sure that these results reflect fair comparisons, all of these data come from training on a fixed hardware configuration on publicly available clouds, and none of these methods increase the cost of inference.</p><p>These results demonstrate the enormous opportunity to accelerate neural network training across a wide variety of settings. In the coming months, we will continue to grow our open-source library of methods and work with customers to reduce the time and cost of their existing workloads and take on more ambitious machine learning problems. We are excited to share our ideas, tools, and infrastructure with the community, and we are eager for feedback and collaboration as we pursue our mission to make deep learning efficient for everyone. Don’t hesitate to reach out if you’re interested in getting involved as a <a href="https://github.com/mosaicml/composer">contributor</a>, collaborator, customer, or — especially — as a <a href="https://boards.greenhouse.io/mosaicml/jobs/4144913004">researcher</a>, <a href="https://boards.greenhouse.io/mosaicml/jobs/4145010004">research engineer</a>, or <a href="https://boards.greenhouse.io/mosaicml/jobs/4133756004">intern</a> <a href="https://boards.greenhouse.io/mosaicml">on our team</a>.</p><h3>The Science of Algorithmic Improvements to Training</h3><p>Our goal in modifying the training algorithm is to improve the efficiency of training. We start with a baseline training algorithm and compare it to training with one or more methods. To evaluate the result, we typically ask one of the following questions:</p><ul><li>What is the fastest or cheapest way that we can train a particular model while maintaining the same quality as the baseline? For example, if the baseline ResNet-50 reaches 76.6% top-1 accuracy on ImageNet, can we reach this accuracy in less time or for less money than the baseline?</li><li>What is the highest quality we can reach while maintaining the same time or cost budget? For example, if the baseline ResNet-50 takes four hours to reach 76.6% top-1 accuracy, can we reach a higher quality model while investing those same four hours?</li></ul><h3>Measuring Success</h3><p>We measure three quantities when developing improvements to training efficiency:</p><ul><li><em>Quality: </em>Metrics that characterize the quality of the model. For example, accuracy in image classification and perplexity in language modeling.</li><li><em>Training Time</em>: The amount of time necessary to train the model on a given system configuration.</li><li><em>Training Cost</em>: The amount of money necessary to train the model on a given system configuration.</li></ul><p>There are many other quantities that we might wish to measure. For example, it is important to understand the effects that methods have on the time and cost required for <em>inference</em> (using the model after it is trained). For simplicity in our top-line numbers, we only consider changes to the training procedure that have a negligible impact on inference time and cost. For example, we do not modify the model architecture in significant ways, nor do we report numbers here that reflect comparisons across different model sizes. These constraints make it possible to isolate and characterize how changes to the training procedure affect the three metrics above.</p><p>In addition, there are many other inputs that could affect the time and cost of training, for instance, varying the hardware platform. In our experience, using an <a href="https://www.nvidia.com/en-us/data-center/tesla-t4/">NVIDIA T4</a> rather than an <a href="https://www.nvidia.com/en-us/data-center/a100/">NVIDIA A100</a> significantly increases training times, but it can also reduce training costs because the cost per hour for a T4 is often significantly lower than the cost per hour for an A100 on public clouds. For simplicity in our top-line numbers, we only make comparisons between models trained on identical hardware. In doing so, all improvements in training time lead to equivalent improvements in training cost.</p><p>For ease of interpreting the top-line numbers we report here, we make the simplifying assumption that inference costs and hardware configurations are fixed. In the <a href="https://app.mosaicml.com/">Explorer</a> — which includes thousands of data points we have collected on different combinations of models, hardware platforms, and methods — you can go beyond these constraints. For example, you can make comparisons across different model sizes (e.g., ResNet-50 vs. ResNet-101) and hardware platforms (e.g., A100s vs. V100s vs. T4s and Amazon Web Services vs. Google Cloud Platform) in order to find the most cost-effective training configuration.</p><h3>Evaluating Methods by Examining Tradeoffs</h3><p>Any method can have two effects on training a model:</p><ul><li>It can affect model quality, either positively or negatively.</li><li>It can affect training time, either positively or negatively. Training time can then be translated into training cost. Since we focus on fixed hardware here, all improvements to training speed lead to commensurate improvements in training cost.</li></ul><p>If a method improves quality at the same or better training time (or improves training time at the same or better quality), then this method is strictly better than the baseline and using it is clearly worthwhile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6X0d7XsJYuPgtBv2b2ybw.png" /></figure><p>But what if, instead, a method improves training time but reduces quality, or improves quality but worsens training time? How does one decide whether to use such a method?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tJoUkoXxm1HSdZo9ViqfdQ.png" /></figure><p>To answer this question, we evaluate the tradeoff between model quality and the available training budget (as measured in training time or cost). Consider a particular model, say ResNet-50 on ImageNet. In its standard configuration, we train for 90 epochs and reach 76.6% top-1 accuracy. But who’s to say that this is the correct configuration? If we train for 45 epochs, we will cut training time in half, but quality will decrease. If we train for 180 epochs, we will achieve higher quality, but training time will increase. No single one of these configurations is “right” per se, nor does any single data point accurately capture this time-quality tradeoff.</p><p>Instead, we describe tradeoffs in model training efficiency using a collection of points on a <em>tradeoff curve</em> that represent the quality attained for a given training budget. Ultimately, the researcher or engineer training the model must choose which point on a tradeoff curve best suits his or her goals, and only by showing the full collection of points may domain experts make informed decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9zAKO7dlm2jn84ksDQ6fg.png" /></figure><p>Let’s walk through an example <em>tradeoff curve</em>. On the x-axis, we plot the amount of time or money necessary to train a model. On the y-axis, we plot the quality of this model. For a given training configuration, we consider a wide range of training budgets. For example, on Resnet-50 on ImageNet, we train for 90, 72, 60, 54, 30, and 22.5 epochs (representing speedups and cost reductions of 1x, 1.25x, 1.5x, 2x, 3x, and 4x). When we scale the training time in this way, we correspondingly dilate the learning rate schedule. We then evaluate the resulting quality of these models and plot the points. The curve connecting these points characterizes the quality-speed tradeoff, shown in blue above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEKN2iMw__kt8XeD-7nAFA.png" /></figure><p>Now, let’s see the effect of applying a method to modify the training algorithm. When we modify the training configuration, we re-train the model for each of these budgets. This allows us to create a second tradeoff curve (red). To determine whether a method is beneficial, we compare the two tradeoff curves. If the curve moves upwards and to the left, it means that using this method attains a better overall tradeoff between quality and training time than the baseline. Note that, in the example above, the red points are all slower than the corresponding blue points because the method applied on its own costs extra time. However, the improvements in quality are such that the method still leads to a better overall tradeoff: to reach any particular quality, you can train for fewer steps, leading to an overall speedup despite the fact that each step is slightly more expensive.</p><p>This brings us to a key point in thinking about efficiency: <em>time and quality are typically interchangeable.</em> Most quality improvements can be converted into time improvements by training for fewer steps, and most time improvements can be converted into quality improvements by training for more steps. In this way, regularization, which is typically thought of as a way to improve model quality, can also be used as a speedup method.</p><p>One final note: the example above is particularly clean: one curve is above and to the left of the other across the full range of training budgets we consider. This is not always the case; it is possible that different configurations may be optimal depending on the time or cost budget available for training. We have found that certain methods excel when budgets are very small, while others (especially regularization) thrive in larger-budget regimes.</p><p>In summary, we think about efficiency of training in terms of tradeoffs between quality and time or cost, and we consider a method beneficial when it makes it possible to reach a better tradeoff curve than the baseline.</p><h3>The Costs of Neural Network Training</h3><p>If one is trying to improve the efficiency of neural network training, it is helpful to understand the steps of the training process and the system resources they utilize. As mentioned in the previous section, methods typically have one or more of the following effects on the cost of training:</p><ol><li>They reduce the number of optimization steps required for training to reach a particular quality.</li><li>They reduce the cost of each step in order to make it possible to train for the same number of steps in less time.</li></ol><p>Each step of training comprises several operations:</p><ol><li>Data is loaded off of the local disk or over the network and into main memory.</li><li>Data is pre-processed (e.g., JPEG decompression, data augmentation, tokenization). This can take place on CPU, an AI accelerator (e.g., a GPU), or some combination of the two.</li><li>Data is loaded onto the AI accelerator.</li><li>Data is forward propagated and backward propagated through the network, which involves several steps of loading data from accelerator memory, computing using the data, and writing data back to accelerator memory. This process may be bound by the amount of compute available on the accelerator or the amount of memory bandwidth on the accelerator, and different steps can be bound by one or the other.</li></ol><p>These operations are often performed in a pipelined manner, with data loaded for the next step while forward and backward propagation take place for the current step. One of these components will always impose a bottleneck on the overall speed of training. In an ideal world, that bottleneck will be the amount of compute or memory bandwidth on the accelerator, since accelerators are typically the most expensive resources. However, if a system is not tuned properly, it is possible that other steps — for example loading data from disk — may become a bottleneck, hampering the ability to take full advantage of the accelerator and wasting money.</p><p>Although our methods typically focus on reducing the cost of forward and backward propagation (Operation 4), they often have side-effects on Operations 1–3 that create new bottlenecks or shift the location of the bottleneck. For example, while the main effect of <a href="https://www.mosaicml.com/methods/randaugment">RandAugment</a> is regularization that improves model quality and reduces the number of steps necessary to train the model, it requires additional data augmentation that has the side-effect of significantly increasing CPU usage. In fact, in some cases, we found that CPU usage became such a bottleneck that it slowed down training. Similarly, some of our combinations of methods were so successful at speeding up Operation 4 that JPEG decoding on the CPU became a bottleneck, restricting our ability to speed up the model further without developing a more efficient dataloader pipeline.</p><p>In summary, it is important to understand how methods improve the efficiency of training and which resource is currently imposing a bottleneck in order to target further interventions. This is a basic tenet of computer system optimization, and it is important not to lose sight of it in the midst of the many other considerations present in deep learning.</p><h3>Composition: Combining Methods</h3><p>Thus far, we have only considered the effects of applying a single method to improve training. But what happens when we <em>compose</em> multiple methods? In an ideal world, the effects of different methods would compose linearly or even synergistically, where the benefits of composition are at least as good as the combined benefits of the constituent parts. But composition is complex in practice, and methods interact nonlinearly.</p><p>Composition is the cornerstone of MosaicML’s research program: it demonstrates how our diverse suite of methods and best practices may be combined together into a usable toolset. It is also the basis of our company name: we address the research problem of assembling the “mosaic” of individual methods proposed in the literature in order to see what the complete picture looks like. We have developed several strategies to reason about composition.</p><p>Central to our thinking about composition is <a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl’s Law</a>, an idea for reasoning about efficiency in computer architectures. In our context, Amdahl’s law states that optimizing a portion of a system responsible for N% of the costs can lead to, at most, an N% speedup. For example, assuming there are no other bottlenecks to the training process, back propagation consumes about 2/3 of the cost of training. Thus, any reductions to the cost of back propagation can reduce the cost of training by at most 2/3.</p><p>One consequence of Amdahl’s law is that there are diminishing returns in optimizing the same part of a system more than once. For example, suppose a method reduces the cost of back propagation by 50%. This will reduce the overall cost of training by 1/3, assuming there are no other bottlenecks in the training process. Because back propagation now comprises less of the training cost than it did before, introducing a second method that also reduces the cost of back propagation by 50% will only reduce the overall cost of training by 1/6 (assuming it composes perfectly with the first method). And if each of these methods also reduces model quality, it is possible that composing these methods could actually lead to a <em>worse</em> tradeoff curve than using either method individually. Early in our research, we encountered this exact problem. We combined two methods that each resulted in improved tradeoffs individually, but when combined led to worse tradeoffs than the baseline model because they were optimizing the same part of the training process.</p><p>As a consequence of this experience, we have learned that composition tends to be most successful when the methods affect all parts of training in a balanced way. For example, it is more beneficial to compose a method that reduces the cost of forward propagation with another that reduces the cost of back propagation. Or, alternatively, to compose a method that increases the cost of back propagation but improves quality with another method that reduces the computational cost of each step back to the original level.</p><p>Our best results occur when we combine many methods. For example, on ResNet-101, they occur when we combine BlurPool, Channels Last, Label Smoothing, Mixup, Progressive Resizing, and Sharpness-Aware Minimization while training for fewer steps using Scale Schedule. You can use the <a href="https://app.mosaicml.com/">Explorer</a> to view these combinations</p><h3>Setting Baselines</h3><p>As the previous section illustrates, an essential part of improving the efficiency of neural network training is choosing appropriate baselines to compare against. One common failure mode in machine learning research is to optimize the configuration of a proposed method without optimizing the corresponding baseline, potentially overestimating the value of the method. At MosaicML, we take great care to ensure that our baselines are as strong as possible, presenting the most difficult possible scenario in which to improve training time, training cost, or model quality.</p><p>Our baselines reflect standard best practices for efficient training:</p><ul><li>We use mixed precision training.</li><li>We have tuned the data loading pipeline to maximize throughput.</li><li>We have performed hyperparameter search across batch size and learning rate.</li><li>We use a step-wise (rather than epoch-wise) learning rate schedule, which we have found to improve quality across our range of settings.</li><li>Rather than select a specific training budget, we produce tradeoff curves reflecting a range of possible training budgets to ensure that our baselines reflect the minimum possible budget to reach a particular level of quality.</li><li>We use <a href="https://arxiv.org/abs/1711.05101">decoupled weight decay</a> to ensure that the level of weight decay is appropriate for a range of batch sizes and learning rates.</li><li>On GPT models, we have been guided by <a href="https://arxiv.org/abs/2001.08361">scaling laws</a> in choosing our model sizes and hyperparameters, but we have explored around these hyperparameters to ensure that our choices are as strong as possible.</li></ul><p>These baselines are challenging. Our ability to still reduce time, reduce cost, and improve quality in comparison to these baselines reflects the immense opportunity to improve the efficiency of training algorithmically.</p><p>These baselines reflect a significant amount of tuning. We recognize that many research and industrial models may not have been subjected to this level of tuning. In order to provide context for those scenarios, several of our benchmarks in the Explorer also include <em>unoptimized</em> baselines that use mixed precision training and tuned hyperparameters but no other enhancements.</p><h3>Data Collection</h3><p>To evaluate our methods, we collected data on thousands of combinations of models, datasets, methods, and hardware platforms. You can view these results in the <a href="https://app.mosaicml.com/">Explorer</a>. In this section, we provide the system details for how we collected this data so that you can replicate our results.</p><p>We collected all of our data on the publicly available Amazon (AWS) and Google (GCP) clouds. We determined the quality of each combination of model, dataset, and methods by training on a single hardware configuration (8xA100 on AWS — p4d.24xlarge). We determined the time and cost required for conducting this training run on all other hardware configuration by profiling the training throughput and extrapolating the time that would have been required for the full training run. For methods that behave uniformly throughout training, we profiled the initial epochs of training and extrapolated. For methods that change their behavior and throughput over the course of training according to a schedule, we profiled relevant points in training and extrapolated.</p><p>Performance and quality were obtained with our MosaicML Composer open-source library using the <a href="https://hub.docker.com/r/mosaicml/composer">Composer Docker Image</a>. The Docker Image is built with the following software dependencies: Ubuntu 18.04, Pytorch v1.9.0, Python 3.8.0. CUDA 11.1.1</p><p><strong>AWS Configuration</strong></p><p>Image: EKS Optimized AMI Version 1.19 with GPU Support</p><p>/aws/service/eks/optimized-ami/1.19/amazon-linux-2-gpu/recommended/image_id</p><p>CUDA Driver:460.73.01</p><p>For more information, see <a href="https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html">https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html</a></p><p><strong>GCP Configuration</strong></p><p>Image: COS with ContainerD, Version 1.18.16-gke.502</p><p>CUDA Driver: 450.119.04</p><p>For more information, see‍</p><ul><li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd">https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd</a></li><li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers">https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers</a></li></ul><p>‍</p><p><em>Originally published at </em><a href="https://www.mosaicml.com/blog/methodology"><em>https://www.mosaicml.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64097877a20e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>