Hydra, from Facebook AI, is a framework for elegantly configuring complex applications. Since its initial release, Hydra has become a popular framework adopted by researchers and practitioners. We are happy to announce that users can now scale and launch jobs to the cloud through the new Hydra Ray Launcher!

Ray is a library that fits Hydra’s needs perfectly. Ray is a simple yet powerful Python library for parallel and distributed programming with a great ecosystem of ML libraries (for distributed training, reinforcement learning, and model serving), as well as community libraries and integrations (e.g., Dask on Ray, Horovod on Ray).

Hydra Ray Launcher enables you to easily configure and launch your application on Ray in 3 different ways. You can launch your application by:

1. Starting or connecting to a Ray cluster on AWS EC2 for short lived clusters
2. Connecting to an existing Ray cluster if you have a long running Ray cluster.
3. Starting a new Ray cluster locally.

Walkthrough

Below, we walk you through installation and running the example applications provided by the launcher. Please check Hydra Ray Launcher’s documentation for more details.

Installation

pip install hydra-ray-launcher --pre

Launch to an AWS cluster

Launching a Ray application on AWS can be done by setting hydra/launcher=aws. This will allow your application to run on a new or existing AWS EC2 Ray Cluster. Launching on AWS is built on top of Ray’s cluster launcher CLI (learn more here). The cluster launcher CLI expects an autoscaler yaml for cluster configurations.

You can configure your cluster just like how you configure any Hydra application: yaml, strict config and command line override.

@hydra.main(config_name="config")
def my_app(cfg: DictConfig) -> None:
    log.info(f"Executing task {cfg.task}")
    time.sleep(1)


if __name__ == "__main__":
    my_app()

From the command line:

Launch on an existing Ray cluster

You can also launch the application on an existing Ray cluster by configuring the launcher to connect to the cluster. For instance, you might want to run the same Ray script on your local machine by accessing the Ray cluster you started previously.

Launch by spinning up a new Ray cluster locally (For testing)

For a quick test locally, you can spin up a Ray cluster at initialization time. Note how the only difference is to specify the hydra launcher as ray, as opposed to ray_aws shown in the previous example.

Next steps

Please check out Hydra Ray Launcher’s documentation for more details on the launcher. If you’d like to learn more about Hydra, check out the Hydra Website and join the community. If you’d like to learn more about Ray, check out the Ray Documentation, Ray Tutorials on Anyscale Academy, Ray Summit, and join the Ray forums.

We’d love to hear your feedback and experience with the new Hydra Ray launcher!

Originally published at https://www.anyscale.com.

Configuring and Scaling ML with Hydra + Ray was originally published in Distributed Computing with Ray on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ray provides a simple, universal API for scaling Python applications across compute clusters with minimal effort. Many users love using Ray since it reduces the complexity of building concurrent and distributed programs.

However, in the first two years of developing Ray, we neglected to provide native Windows support. Though most distributed execution environments ran Linux, developers needed to prototype on their local laptop or desktop, which often was a Windows machine.

In the past year, one of the Ray contributors, Mehrdad Niknami, led a huge effort to close this gap. There were countless technical challenges, including running Redis on Windows and building our own equivalent Windows shims for POSIX APIs.

After almost 9 months of work, we were finally able to build Windows Python wheels, which we released with version 0.8.6 in June. And, moving forward, we will be supporting and maintaining the Windows build to make sure more developers can leverage Ray.

As we approach the release of Ray 1.0, we will share a detailed account of our development efforts toward Windows support. In that post, we will take a deep dive into the challenges we faced and the solutions we came up with, which we hope will be valuable for other projects embarking on a similar journey.

While the port of Ray to Windows is not yet finished, it has been quite successful so far, and we are aiming to improve it over the next couple releases. Please try it out today and let us know if you run into any issues!

Ray for Windows was originally published in Distributed Computing with Ray on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pytorch Lightning is one of the hottest AI libraries of 2020, and it makes AI research scalable and fast to iterate on. But if you use Pytorch Lightning, you’ll need to do hyperparameter tuning.

Proper hyperparameter tuning can make the difference between a good training run and a failing one. Everyone knows that you can dramatically boost the accuracy of your model with good tuning methods!

In this blog post, we’ll demonstrate how to use Ray Tune, an industry standard for hyperparameter tuning, with PyTorch Lightning. Ray Tune is part of Ray, a library for scaling Python.

It is available as a PyPI package and can be installed like this:

pip install "ray[tune]"

To use Ray Tune with PyTorch Lightning, we only need to add a few lines of code!!

Getting started with Ray Tune + PTL!

To run the code in this blog post, be sure to first run:

pip install "ray[tune]" 
pip install "pytorch-lightning>=1.0" 
pip install "pytorch-lightning-bolts>=0.2.5"

The below example is tested on ray==1.0.1 , pytorch-lightning==1.0.2, and pytorch-lightning-bolts==0.2.5. See the full example here.

Let’s first start with some imports:

After imports, there are three easy steps.

1. Create your LightningModule
2. Create a function that calls Trainer.fit with the Tune callback
3. Use tune.run to execute your hyperparameter search.

Step 1: create your LightningModule

First step, create your LightningModule. Your LightningModule should take a configuration dict as a parameter on initialization. This dict should then set the model parameters you want to tune. Your module could look like this:

Step 2: Create a function that calls Trainer.fit with the Tune callback

We will use a callback to communicate with Ray Tune. The callback is very simple:

from ray.tune.integration.pytorch_lightning import TuneReportCallback

...
metrics = {"loss": "ptl/val_loss", "acc": "ptl/val_accuracy"}
callbacks = [TuneReportCallback(metrics, on="validation_end")]
trainer = pl.Trainer(... callbacks=callbacks)

This callback ensures that after each validation epoch, we report the loss metrics back to Ray Tune. The val_loss and val_accuracy keys correspond to the return value of the validation_epoch_end method.

Further, Ray Tune will start a number of different training runs. To create multiple training runs (for the hyperparameter search), we thus need to wrap the trainer call in a function:

The train_mnist() function expects a config dict, which it then passes to the LightningModule. This config dict will contain the hyperparameter values of one evaluation.

Step 3: Use tune.run to execute your hyperparameter search.

Finally, we need to call ray.tune to optimize our parameters. Here, our first step is to tell Ray Tune which values are valid choices for the parameters. This is called the search space, and we can define it like so:

# Defining a search space!

config = {
 "layer_1_size": tune.choice([32, 64, 128]),
 "layer_2_size": tune.choice([64, 128, 256]),
 "lr": tune.loguniform(1e-4, 1e-1),
 "batch_size": tune.choice([32, 64, 128])
}

Let’s take a quick look at the search space. For the first and second layer sizes, we let Ray Tune choose between three different fixed values. The learning rate is sampled between 0.0001 and 0.1. For the batch size, also a choice of three fixed values is given. Of course, there are many other (even custom) methods available for defining the search space.

Ray Tune will now proceed to sample ten different parameter combinations randomly, train them, and compare their performance afterwards.

We wrap the train_mnist function in tune.with_parameters to pass constants like the maximum number of epochs to train each model and the number of GPUs available for each trial. Ray Tune supports fractional GPUs, so something like gpus=0.25 is totally valid as long as the model still fits on the GPU memory.

# Execute the hyperparameter search
analysis = tune.run(
 tune.with_parameters(train_mnist_tune, epochs=10, gpus=0),
 config=config,
 num_samples=10)

The final invocation of tune.run can look like this:

And finally, the tuning result could look like this:

In this simple example, a number of configurations reached a good accuracy. The best result we observed was a validation accuracy of 0.978105 with a batch size of 32, layer sizes of 128 and 64, and a small learning rate around 0.001. We can also see that the learning rate seems to be the main factor influencing performance — if it is too large, the runs fail to reach a good accuracy.

You can retrieve the best score by using the return value of tune.run :

analysis = tune.run(...)

best_trial = analysis.best_trial  # Get best trial
best_config = analysis.best_config  # Get best trial's hyperparameters
best_logdir = analysis.best_logdir  # Get best trial's logdir
best_checkpoint = analysis.best_checkpoint  # Get best trial's best checkpoint
best_result = analysis.best_result  # Get best trial's last results

You can also easily leverage some of Ray Tune’s more powerful optimization features. For example, Ray Tune’s search algorithm allows you to easily optimize the landscape of hyperparameter combinations. Ray Tune’s schedulers can also stop bad performing trials early to save resources.

See the full example here.

Inspecting results in Tensorboard

Ray Tune automatically exports metrics into TensorBoard, and also easily supports W&B.

That’s it!

To enable easy hyperparameter tuning with Ray Tune, we only needed to add a callback, wrap the train function, and then start Tune.

Of course, this is a very simple example that doesn’t leverage many of Ray Tune’s search features, like early stopping of bad performing trials or population based training. If you would like to see a full example for these, please have a look at our full PyTorch Lightning tutorial.

If you’ve been successful in using PyTorch Lightning with Ray Tune, or if you need help with anything, please reach out by joining our Slack or dropping by our Github — we would love to hear from you!

If you liked this blog post, be sure to check out:

• Our Weights and Biases report on Hyperparameter Optimization for Transformers
• The simplest way to serve your NLP model from scratch

How to tune Pytorch Lightning hyperparameters was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Distributed training is annoying to set up and expensive to run. Here’s a library to make distributed Pytorch model training simple and cheap.

Distributing your deep learning model training has become a question of when you do it, not if. State-of-the-art ML models like BERT have 100s of millions of parameters, and training these large networks will take you days if not weeks on one machine.

Fundamentally, you have two choices when it comes to training deep learning models:

Option 1: Tolerate the 20 hour training times, or focus on models that are small enough to train on a single node (or single GPU) to keep things simple and be able to use standard tools like Jupyter Notebooks.

Option 2: Go through a mountain of pain and try to distribute your training.

So, what does it take to distribute training today?

To take your training beyond a single node, you’re going to have to deal with:

1. Messy distributed systems deployments (including setting up networking, containerization, credentials).
2. A huge AWS bill for expensive nodes (current solutions don’t allow you to use cheap, preemptible instances).
3. Losing access to your favorite tools such as Jupyter notebooks.

You can use one of the integrated tools for doing distributed training like Torch Distributed Data Parallel or tf.Distributed. While these are “integrated”, they are certainly not a walk in the park to use.

Torch’s AWS tutorial demonstrates the many setup steps you’re going to have to follow to simply get the cluster running, and Tensorflow 2.0 has a bunch of issues.

Maybe you might look at something like Horovod, but Horovod is going to require you to fight against antiquated frameworks like MPI and wait a long time for compilation when you launch.

Along with this complex setup, you’re going to need to give up the typical tools that you use, like Jupyter notebooks. On top of that you’re going to have to use expensive on-demand instances because none of these frameworks are fault-tolerant.

In our own work, we identified these issues as roadblocks to simplifying distributed deep learning training. We set out to create our own solution that solves these key problems.

So, what’s the better way?

RaySGD — the simple distributed training solution

To solve the problems above, we built RaySGD.

It’s a lightweight Python library built on top of distributed PyTorch that not only makes deployment easy, but it provides you with the flexibility you need to do cutting edge research and development.

Now, we’re sure you’re asking yourself,

How is this better than what already exists?

RaySGD focuses on several key benefits:

• Seamless Parallelization: Go from 1 GPU to 100 GPUs with a single argument.
• Accelerate Training: Built-in support for mixed precision training with NVIDIA Apex.
• Simple, Native Interfaces: We’ve kept the interface simple to make it easy to migrate existing training code and keep the mental overhead low — just learn a couple of new lines of code.
• Fault Tolerance: Supports automatic recovery when machines on the cloud are preempted. Now, you can use spot instances to reduce costs by up to 90%.
• Seamless Hyperparameter Tuning: RaySGD integrates with RayTune, a cutting edge, distributed, hyperparameter tuning framework.

On top of what we achieved above, RaySGD matches the performance of specialized, SOTA deep learning frameworks, like Horovod.

RaySGD is also able to outperform the default torch.nn.DataParallel by up to 20% on 8 GPUs.

RaySGD is built on top of Ray, a framework for fast and simple distributed computing. Training on spot or preemptible instances for huge cost savings is as simple as changing a single line of code: trainer.train(max_retries=100)

How do I get started?

RaySGD provides a minimal API that gives a user the typical customizability they’re already familiar with from TensorFlow or PyTorch. Here’s the minimum you need to run a multi-GPU training job.

1. pip install -U ray torch
2. Run the below script.

This simple script will download CIFAR10 and use a ResNet18 model to do image classification. You can run on multiple GPUs with a single parameter change ( num_workers=N ).

OK then, how do I scale out PyTorch training across a cluster?

Don’t worry, this is just 4 extra steps. I’ll demonstrate how to run RaySGD on AWS, but it’s just as easy to run this on SLURM, Azure, GCP, or your local cluster.

1. Download the following YAML file and the previous python script (save it as pytorch.py).
2. Change ray.init() in the script to ray.init(address="auto")
3. Change num_workers=16 in the TorchTrainer constructor.
4. Run ray submit ray-cluster.yaml pytorch.py --start --stop . This automatically starts the preemptible cluster (16 total V100 GPUs), and shuts down the cluster as soon as training is finished. For 30 minutes, this will cost… $7.44.

To run on GCP or Azure, simply change a couple lines in the above YAML — more instructions here.

As you can see, getting up and running with RaySGD is simple — you’ve learned almost everything you need in this blog post. So now get started and let us know what you think!

• Documentation
• Code
• Examples for customizing training
• Examples for distributed hyperparameter tuning
• Follow us on Twitter!
• Join our Discourse!
• Join our Slack!
• Join us at Ray Summit!

Faster and Cheaper Pytorch with RaySGD was originally published in Distributed Computing with Ray on Medium, where people are continuing the conversation by highlighting and responding to this story.

As AI research becomes more compute intensive, many AI researchers have become squeezed for time and resources. Many researchers now rely on cloud providers like Amazon Web Services or Google Compute Platform to access the huge amounts of computational resources necessary for training large models.

In this blog post, we share our experiences in developing two critical software libraries that many BAIR researchers use to execute large-scale AI experiments: Ray Tune and the Ray Cluster Launcher, both of which now back many popular open-source AI libraries.

Understanding research infrastructure

To put things into perspective, let’s first take a look at a standard machine learning workflow in industry (Figure 1).

Figure 1 represents the typical ML model development workflow in industry.

The typical research workflow is actually a tight loop between steps 2 and 3, generalizing to something like Figure 2.

Figure 2 represents the typical ML model development workflow for research. The research workflow is typically a subsection of the industry workflow.

The research workflow is very much an iterative process and is often bottlenecked by the experiment execution step (Figure 2, B). Typically, an “experiment” consists of multiple training jobs, or “trials”, where each trial is a job that trains a single model. Each trial might train a model using a different set of configuration parameters (hyperparameters) or a different seed.

At Berkeley, we saw that AI researchers moving to the cloud spent a lot of time writing their own experiment execution tools that wrangled cloud-provider APIs for starting instances, setting up dependencies, and launching experiments.

Unfortunately, despite the vast amounts of time poured into developing these tools, these ad-hoc solutions are typically limited in functionality:

• Simplistic Architecture: Each trial is typically launched on a separate node without any centralized control logic. This makes it difficult for researchers to implement optimization techniques such as Population-based Training or Bayesian Optimization that require coordination between different runs.
• Lack of Fault Handling: The results of training jobs are lost forever if an instance fails. Researchers often manually manage the failover by tracking live experiments on a spreadsheet, but this is both time consuming and error-prone.
• No Spot Instance Discount: The lack of fault tolerance capabilities also means forgoing the spot instance discount (up to 90%) that cloud providers offer.

To sum up, instrumenting and managing distributed experiments on cloud resources is both burdensome and difficult to get right. As such, easy-to-use frameworks that bridge the gap between execution and research can greatly accelerate the research process. Members from a couple of labs across BAIR collaborated to build two complementary tools for AI experimentation in the cloud:

Ray Tune: a fault-tolerant framework for training and hyperparameter tuning. Specifically, Ray Tune (or “Tune” for short):

1. Coordinates among parallel jobs to enable parallel hyperparameter optimization.
2. Automatically checkpoints and resumes training jobs in case of machine failures.
3. Offers many state-of-the-art hyperparameter search algorithms such as Population-based Training and HyperBand.

Ray Cluster Launcher: a utility for managing resource provisioning and cluster configurations across AWS, GCP, and Kubernetes.

Ray Tune

Tune was built to address the shortcomings of these ad-hoc experiment execution tools. This was done by leveraging the Ray Actor API and adding failure handling.

Actor-based Training

Many techniques for hyperparameter optimization require a framework that monitors the metrics of all concurrent training jobs and controls the training execution. To address this, Tune uses a master-worker architecture to centralize decision-making and communicates with its distributed workers using the Ray Actor API.

What is the Ray Actor API? Ray provides an API to create an “actor” from a Python class. This enables classes and objects to be used in parallel and distributed settings.

Tune uses a Trainable class interface to define an actor class specifically for training models. This interface exposes methods such as _train, _stop, _save, and _restore, which allows Tune to monitor intermediate training metrics and kill low-performing trials.

class NewTrainable(tune.Trainable):
    def _setup(self, config):
        ...

    def _train(self):
        """Run 1 step of training (e.g., one epoch).

        Returns:
            A dict of training metrics.
        """
        ...

    def _save(self, checkpoint_dir):
        ...

    def _restore(self, checkpoint_path):
        ...

More importantly, by leveraging the Actor API, we can implement parallel hyperparameter optimization methods in Tune such as HyperBand and Parallel Bayesian Optimization which was not possible with prior experiment execution tools that researchers used.

Fault Tolerance

Cloud providers often offer “preemptible instances” (i.e., spot instances) at a significant discount. The steep discounts allow researchers to lower their cloud computing costs significantly. However, the downside is that the cloud provider can terminate or stop your machine at any time, causing you to lose training progress.

To enable spot instance usage, we built Tune to automatically checkpoint and resume training jobs across different machines in the cluster so that experiments would be resilient to preemption and cluster resizing.

# Tune will resume training jobs from the last checkpoint
# even if machines are removed.

analysis = tune.run(
    NewTrainable,
    checkpoint_freq=5,  # Checkpoint every 5 epochs
    config={"lr": tune.grid_search([0.001, 0.01, 0.1])},
)

How does it work?

If a node is lost while a training job is still executing on that node and a checkpoint of the training job (trial) exists, Tune will wait until available resources are available to begin executing the trial again.

If the trial is placed on a different node, Tune will automatically push the previous checkpoint file to that node and restore the state, allowing the trial to resume from the latest checkpoint even after failure.

Ray Cluster Launcher

Above, we described the pains of wrangling cloud-provider APIs to automate the cluster setup process. However, even with a tool for spinning up clusters, researchers still have to go through a tedious workflow to run experiments:

To simplify this, we built the Ray Cluster Launcher, a tool for provisioning and autoscaling resources and starting a Ray cluster on AWS EC2, GCP, and Kubernetes. We then abstracted all of the above steps for running an experiment into just a short configuration file and a single command:

# CLUSTER.yaml
cluster_name: tune-default
provider: {type: aws, region: us-west-2}
auth: {ssh_user: ubuntu}
min_workers: 0
max_workers: 2
# Deep Learning AMI (Ubuntu) Version 21.0
head_node: {
    InstanceType: c4.2xlarge,
    ImageId: ami-0b294f219d14e6a82}
worker_nodes: {
    InstanceType: c4.2xlarge,
    ImageId: ami-0b294f219d14e6a82}
setup_commands: # Set up each node.
    - pip install ray numpy pandas
file_mounts: {
  '/home/ubuntu/files':'my_files/',
}

The below command starts a cluster, uploads and runs a script for distributed hyperparameter tuning, then shuts down the cluster.

$ ray submit CLUSTER.yaml --start --stop tune_experiment.py \
     --args="--address=auto"

Researchers are now using both Ray Tune and the Ray Cluster Launcher to launch hundreds of parallel jobs at once across dozens of GPU machines at once. The Ray Tune documentation page for distributed experiments shows you how you can do this too.

Putting things together

Over the last year, we’ve been working with different groups across BAIR to better allow researchers to leverage the cloud. We had to make Ray Tune and the Ray Cluster Launcher general enough to support a large body of research codebases while making the barrier to entry so low that anyone could try it out in a couple of minutes.

# An example Ray Tune script for PyTorch.
import torch.optim as optim
from ray import tune
from ray.tune.examples.mnist_pytorch import (
    get_data_loaders, ConvNet, train, test)

class TrainMNIST(tune.Trainable):
    def _setup(self, config):
        self.train_loader, self.test_loader = get_data_loaders()
        self.model = ConvNet()
        self.optimizer = optim.SGD(
            self.model.parameters(), lr=config.get("lr", 0.01))

    def _train(self):
        train(self.model, self.optimizer, self.train_loader)
        acc = test(self.model, self.test_loader)
        return {"mean_accuracy": acc}

    def _save(self, checkpoint_dir):
        checkpoint_path = os.path.join(checkpoint_dir, "model.pth")
        torch.save(self.model.state_dict(), checkpoint_path)
        return checkpoint_path

    def _restore(self, checkpoint_path):
        self.model.load_state_dict(torch.load(checkpoint_path))

analysis = tune.run(
    TrainMNIST,
    stop={"training_iteration": 50},
    config={"lr": tune.grid_search([0.001, 0.01, 0.1])})

print("Best hyperparameters: ", analysis.get_best_config(
    metric="mean_accuracy"))
# Get a dataframe for analyzing trial results.
df = analysis.dataframe()

Tune has grown to become a popular open-source project for hyperparameter tuning. It is used in many other popular research projects, ranging from Population-based Data Augmentation, to Hyperparameter Tuning for AllenNLP, to AutoML in AnalyticsZoo.

Numerous open-source research projects from BAIR now rely on the combination of Ray Tune and the Ray Cluster Launcher to orchestrate and execute distributed experiments, including rail-berkeley/softlearning, HumanCompatibleAI/adversarial-policies, and flow-project/flow.

So go ahead and try Ray Tune and Cluster Launcher yourself!

Links

• Previous BAIR blog posts about the Ray project:
• Ray: A Distributed System for AI
• Scaling Multi-Agent Reinforcement Learning
• Functional RL with Keras and Tensorflow Eager
• Ray website
• Ray GitHub page
• Tune
• Ray Cluster Launcher
• Ray-project landing page
• Installation

Scaling Experiments at Berkeley AI Research was originally published in riselab on Medium, where people are continuing the conversation by highlighting and responding to this story.

Behind most of the major flashy results in machine learning is a graduate student (me) or engineer spending hours training a model and tuning algorithm parameters. This is the tedious boring work that makes the headlines possible.

Here in the RISELab, we’re finding it more and more necessary to leverage cutting-edge hyperparameter tuning tools to keep up with the state of the art. Advancements in deep learning performance are becoming more and more dependent on newer and better hyperparameter tuning algorithms such as Population Based Training (PBT), HyperBand, and ASHA.

These algorithms provide two critical benefits:

• They maximize model performance: e.g., DeepMind uses PBT to achieve superhuman performance on StarCraft; Waymo uses PBT to enable self-driving cars.
• They minimize training costs: HyperBand and ASHA converge to high-quality configurations in half the time taken by previous approaches; population-based data augmentation algorithms cut costs by orders of magnitude.

Yet, we see that a vast majority of researchers and teams do not leverage such algorithms.

Why? Well, most existing hyperparameter search frameworks do not have these newer optimization algorithms. And once you reach a certain scale, most existing solutions for parallel hyperparameter search can be a hassle to use — you’ll need to configure each machine for each run and often manage a separate database.

Practically speaking, implementing and maintaining these algorithms requires a significant amount of time and engineering.

But it doesn’t need to be this way. We believe there’s no reason why hyperparameter tuning at scale needs to be this hard. All AI researchers and engineers should be able to seamlessly run a parallel asynchronous grid search across 8 GPUs and even scale out to leverage Population Based Training or any Bayesian optimization algorithm on the cloud.

In this blog post, we’ll introduce Tune, a powerful hyperparameter tuning library built on Ray designed to remove the friction from scaling and setting up experiment execution and hyperparameter tuning.

Tune is a powerful Python library that accelerates hyperparameter tuning. Here are some core features:

• Tune provides distributed asynchronous optimization out of the box by leveraging Ray.
• You can scale a hyperparameter search from a single machine to a large distributed cluster without changing your code.
• Tune offers state of the art algorithms including (but not limited to) ASHA, BOHB, and Population-Based Training.
• Automatically visualize results with TensorBoard or MLFlow.
• Tune integrates with many optimization libraries such as Ax/Botorch, HyperOpt, and Bayesian Optimization and enables you to scale them transparently.
• Tune supports any machine learning framework, including PyTorch, TensorFlow, XGBoost, LightGBM, and Keras.

Beyond Tune’s core features, there are two primary reasons why researchers and developers prefer Tune over other existing hyperparameter tuning frameworks: scale and flexibility.

Tune simplifies scaling.

Leverage all of the cores and GPUs on your machine to perform parallel asynchronous hyperparameter tuning by adding fewer than 10 lines of Python.

With another configuration file and 4 lines of code, launch a massive distributed hyperparameter tuning cluster on the cloud and automatically shut down the machines (we’ll show you how to do this below).

With Tune’s built-in fault tolerance, trial migration, and cluster autoscaling, you can safely leverage spot (preemptible) instances and reduce cloud costs by up to 90%.

Tune is flexible.

Tune integrates seamlessly with experiment management tools such as MLFlow and TensorBoard.

Tune provides a flexible interface for optimization algorithms, allowing you to easily implement and scale new optimization algorithms.

You can use Tune to leverage and scale many cutting edge optimization algorithms and libraries such as HyperOpt (below) and Ax without modifying any model training code.

Using Tune is Easy!

Let’s now dive into a concrete example that shows how you to leverage a popular early stopping algorithm (ASHA). We will start by running an example hyperparameter tuning script with Tune across all of the cores on your workstation. We’ll then scale out the same hyperparameter tuning experiment on the cloud with about 10 lines of code using Ray.

You can download a full version of the hyperparameter tuning code in this blog here (distributed experiment configuration here).

We’ll be using PyTorch in this example, but we also have examples for Tensorflow and Keras available.

Tune is packaged as part of Ray. To run this example, you will need to install the following: pip install ray torch torchvision

We first run some imports :

Let’s write a neural network with PyTorch:

To start using Tune, add a simple logging statement to the PyTorch training below function.

Notice that there’s a couple helper functions in the above training script; you can see their definitions here.

Running Tune

Let’s run 1 trial, randomly sampling from a uniform distribution for learning rate and momentum.

Now, you’ve run your first Tune run! You can easily enable GPU usage by specifying GPU resources — see the documentation for more details. We can then plot the performance of this trial (requires matplotlib).

Parallel execution and early stopping

Let’s integrate ASHA, a scalable algorithm for early stopping (blog post and paper). ASHA terminates trials that are less promising and allocates more time and resources to more promising trials.

Parallelize your search across all available cores on your machine with num_samples (extra trials will be queued).

You can use the same DataFrame plotting as the previous example. After running, if Tensorboard is installed, you can also use Tensorboard for visualizing results: tensorboard --logdir ~/ray_results

Going distributed

Setting up a distributed hyperparameter search is often too much work. Tune and Ray make this seamless.

Launching the cloud with a simple configuration file

First, we’ll create a YAML file which configures a Ray cluster. As part of Ray, Tune interoperates very cleanly with the Ray cluster launcher. The same commands shown below will work on GCP, AWS, and local private clusters. We’ll use 3 worker nodes in addition to a head node, so we should have a total of 32 vCPUs on the cluster — allowing us to evaluate 32 hyperparameter configurations in parallel.

Putting things together

To distribute your hyperparameter search across the Ray cluster, you’ll need to append this to the top of your script:

Given the large increase in compute, we should be able to increase our search space and number of samples in our search space:

You can download a full version of the script in this blog here (as tune_script.py).

Launching your experiment

To launch your experiment, you can run (assuming your code so far is in a file tune_script.py):

$ ray submit tune-default.yaml tune_script.py --start \
     --args=”localhost:6379”

This will launch your cluster on AWS, upload tune_script.py onto the head node, and run python tune_script localhost:6379, which is a port opened by Ray to enable distributed execution.

All of the output of your script will show up on your console. Note that the cluster will setup the head node first before any of the worker nodes, so at first you may see only 4 CPUs available. After some time, you can see 24 trials being executed in parallel, and the other trials will be queued up to be executed as soon as a trial is free.

To shut down your cluster, you can run:

$ ray down tune-default.yaml

And you’re done 🎉!

Learn more:

Tune has numerous other features that enable researchers and practitioners to accelerate their development. Other Tune features not covered in this blogpost include:

• A simple API for running distributed fault-tolerant experiments
• Distributed Hyperparameter search over Distributed Data Parallel training for PyTorch
• Population-based Training

For users that have access to the cloud, Tune and Ray provide a number of utilities that enable a seamless transition between development on your laptop and execution on the cloud. The documentation includes:

• running the experiment in a background session
• submitting trials to an existing experiment
• visualizing all results of a distributed experiment in TensorBoard.

Tune is designed to scale experiment execution and hyperparameter search with ease. If you have any comments or suggestions or are interested in contributing to Tune, you can reach out to me or the ray-dev mailing list.

Code: https://github.com/ray-project/ray/tree/master/python/ray/tune
Docs: http://ray.readthedocs.io/en/latest/tune.html

Thanks to Allan Peng, Eric Liang, Joey Gonzalez, Ion Stoica, Eugene Vinitsky, Lisa Dunlap, Philipp Moritz, Andrew Tan, Alvin Wan, Daniel Rothchild, Brijen Thananjeyan, Alok Singh, Robert Nishihara (and maybe others?) for reading through various versions of this blog post!

Originally published at https://towardsdatascience.com on August 20, 2019.

Cutting edge hyperparameter tuning with Ray Tune was originally published in riselab on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve ever tried to tune hyperparameters for a machine learning model, you know that it can be a very painful process. Simple approaches quickly become time-consuming.

And now more than ever, you absolutely need cutting-edge hyperparameter tuning tools to keep up with the state-of-the-art.

Model advancements are becoming more and more dependent on newer and better hyperparameter tuning algorithms such as Population Based Training (PBT), HyperBand, and ASHA.

These algorithms provide two critical benefits:

• They maximize model performance: e.g., DeepMind uses PBT to achieve superhuman performance on StarCraft; Waymo uses PBT to enable self-driving cars.
• They minimize training costs: HyperBand and ASHA converge to high-quality configurations in half the time taken by previous approaches; population-based data augmentation algorithms cuts costs by orders of magnitude.

The fact of the matter is that the vast majority of researchers and teams do not leverage such algorithms. Most existing hyperparameter search frameworks do not have these newer optimization algorithms. And once you reach a certain scale, most existing solutions for parallel hyperparameter search can be a hassle to use — you’ll need to configure each machine for each run and often manage a separate database.

Practically speaking, implementing and maintaining these algorithms requires a significant amount of time and engineering.

But it doesn’t need to be this way. There’s no reason why you can’t easily incorporate hyperparameter tuning into your machine learning project, seamlessly run a parallel asynchronous grid search on 8 GPUs in your cluster, and leverage Population Based Training or any Bayesian optimization algorithm at scale on the cloud.

In this blog post, we’ll introduce RayTune, a powerful hyperparameter optimization library designed to remove the friction from scaling experiment execution and hyperparameter search.

RayTune is a powerful library that accelerates hyperparameter optimization. Here are some core features:

• RayTune provides distributed asynchronous optimization out of the box.
• RayTune offers state of the art algorithms including (but not limited to) ASHA, BOHB, and Population-Based Training.
• You can scale a RayTune hyperparameter search from a single machine to a large distributed cluster without changing your code.
• RayTune integrates with many optimization libraries such as Ax/Botorch, HyperOpt, and Bayesian Optimization and enables you to scale them transparently.
• RayTune supports any machine learning framework, including PyTorch, TensorFlow, XGBoost, LightGBM, scikit-learn, and Keras.

Beyond RayTune’s core features, there are two primary reasons why researchers and developers prefer RayTune over other existing hyperparameter tuning frameworks: scale and flexibility.

Tune simplifies scaling.

Leverage all of the cores and GPUs on your machine to perform parallel asynchronous hyperparameter tuning by adding fewer than 10 lines of Python.

With another configuration file and 4 lines of code, launch a massive distributed hyperparameter search on the cloud and automatically shut down the machines (we’ll show you how to do this below).

With Tune’s built-in fault tolerance, trial migration, and cluster autoscaling, you can safely leverage spot (preemptible) instances and reduce cloud costs by up to 90%.

Tune is flexible.

Tune integrates seamlessly with experiment management tools such as MLFlow and TensorBoard.

Tune provides a flexible interface for optimization algorithms, allowing you to easily implement and scale new optimization algorithms.

You can use Tune to leverage and scale many state-of-the-art search algorithms and libraries such as HyperOpt (below) and Ax without modifying any model training code.

Using Tune is Easy!

Let’s now dive into a concrete example that shows how you to leverage a state-of-the-art early stopping algorithm (ASHA). We will start by running Tune across all of the cores on your workstation. We’ll then scale out the same experiment on the cloud with about 10 lines of code.

We’ll be using PyTorch in this example, but we also have examples for Tensorflow and Keras available.

Tune is installed as part of Ray. To run this example, you will need to install the following: pip install ray torch torchvision.

You can download a full version of the blog in this blog here.

We first run some imports :

Let’s write a neural network with PyTorch:

To start using Tune, add a simple logging statement to the PyTorch training below function.

def train_mnist(config):
    model = ConvNet()
    train_loader, test_loader = get_data_loaders()
    optimizer = optim.SGD(
        model.parameters(), 
        lr=config["lr"], 
        momentum=config["momentum"])
    for i in range(10):
        train(model, optimizer, train_loader, torch.device("cpu"))
        acc = test(model, test_loader, torch.device("cpu"))
        tune.track.log(mean_accuracy=acc)
        if i % 5 == 0:
            # This saves the model to the trial directory
            torch.save(model, "./model.pth")

Notice that there’s a couple helper functions in the above training script; you can see their definitions here.

Running Tune

Let’s run 1 trial, randomly sampling from a uniform distribution for learning rate and momentum.

Now, you’ve run your first Tune run! You can easily enable GPU usage by specifying GPU resources — see the documentation for more details. We can then plot the performance of this trial.

Parallel execution and early stopping

Let’s integrate ASHA, a scalable algorithm for early stopping (blog post and paper). ASHA terminates trials that are less promising and allocates more time and resources to more promising trials.

Parallelize your search across all available cores on your machine with num_samples (extra trials will be queued).

You can use the same DataFrame plotting as the previous example. After running, if Tensorboard is installed, you can also use Tensorboard for visualizing results: tensorboard --logdir ~/ray_results

Going distributed

Setting up a distributed hyperparameter search is often too much work. Tune and Ray make this seamless.

Launching the cloud with a simple configuration file

First, we’ll create a YAML file which configures a Ray cluster. As part of Ray, Tune interoperates very cleanly with the Ray cluster launcher. The same commands shown below will work on GCP, AWS, and local private clusters. We’ll use 3 worker nodes in addition to a head node, so we should have a total of 32 vCPUs on the cluster — allowing us to evaluate 32 hyperparameter configurations in parallel.

Putting things together

To distribute your hyperparameter search across the Ray cluster, you’ll need to append this to the top of your script:

Given the large increase in compute, we should be able to increase our search space and number of samples in our search space:

You can download a full version of the script in this blog here (as tune_script.py).

Launching your experiment

To launch your experiment, you can run (assuming your code so far is in a file tune_script.py):

$ ray submit tune-default.yaml tune_script.py --start \
     --args=”localhost:6379”

This will launch your cluster on AWS, upload tune_script.py onto the head node, and run python tune_script localhost:6379, which is a port opened by Ray to enable distributed execution.

All of the output of your script will show up on your console. Note that the cluster will setup the head node first before any of the worker nodes, so at first you may see only 4 CPUs available. After some time, you can see 24 trials being executed in parallel, and the other trials will be queued up to be executed as soon as a trial is free.

To shut down your cluster, you can run:

$ ray down tune-default.yaml

And you’re done 🎉!

Learn more:

Tune has numerous other features that enable researchers and practitioners to accelerate their development. Other Tune features not covered in this blogpost include:

• A simple API for running distributed fault-tolerant experiments
• Distributed Hyperparameter search over Distributed Data Parallel training for PyTorch
• Population-based Training

For users that have access to the cloud, Tune and Ray provide a number of utilities that enable a seamless transition between development on your laptop and execution on the cloud. The documentation includes:

• running the experiment in a background session
• submitting trials to an existing experiment
• visualizing all results of a distributed experiment in TensorBoard.

Tune is designed to scale experiment execution and hyperparameter search with ease. If you have any comments or suggestions or are interested in contributing to Tune, you can reach out to me or the ray-dev mailing list.

Code: https://github.com/ray-project/ray/tree/master/python/ray/tune
Docs: http://ray.readthedocs.io/en/latest/tune.html

For other readings on hyperparameter tuning, check out Neptune.ai’s blog post on Optuna vs HyperOpt!

Thanks to Allan Peng, Eric Liang, Joey Gonzalez, Ion Stoica, Eugene Vinitsky, Lisa Dunlap, Philipp Moritz, Andrew Tan, Alvin Wan, Daniel Rothchild, Brijen Thananjeyan, Alok Singh (and maybe others?) for reading through various versions of this blog post!

Ray Tune: a Python library for fast hyperparameter tuning at any scale was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.