Managing Experimentation in Deep Learning
Working with Deep Learning involves running innumerable experiments to test various hypotheses one might have, to solve towards a particular problem, or getting better results by trying out multiple hyperparameters. As the number of these experiments grow over time, it can be a bit difficult to manage them or even try to reflect back what was done a few weeks ago. In this post, I plan to share some tips or hacks that I use (as a graduate student working full time in deep learning) to keep things in order and prevent myself from getting lost in a bunch of lazily defined log files like exp0, exp1, newexp.
Keeping things in sync
I (and I assume a lot of others too) run experiments on a GPU-based linux server and then copy the results back to the host machine to analyse them. This type of data could consist of csv/txt files, tensorboard logs, or images.
I’ve found rsync as a perfect tool for that (I was used to scp previously), with the main advantage that it only syncs the diff between the source and the destination. This is helpful when the training is in process (and it’s supposed to take a long long time) and we want to monitor the progress. I usually work on generative models, so I need to check now-and-then if my GAN’s going wild or something by browsing through what it’s generating over the epochs.
The simplest way to do that would be:
rsync -av <username>@remote-host:<path_to_results_folder> .Here, -a would transfer everything recursively and -v stands for verbosity. And this is pretty fast. Also, if your server is via a gateway, you’ll need to login into that beforehand.
There are a lot of things that can be done with rsync, for example, deploying the models in production or setting up a cronjob to pull logs in fixed intervals. Refer this post to check out how the latter is achieved.
Tensorboard
Many are unaware that Tensorflow’s amazing visualisation tool Tensorboard can be used as a standalone utility to plot those nice and interactive curves of literally anything. I usually stick with just plotting scalars to keep things faster and lighter (transfer and load time). The API Tensorboard-Pytorch is really awesome for this. I’m not sure why they call it “-Pytorch” though, since it’s pretty generic. Install it via pip install tesorboardX .
from tensorboardX import SummaryWriter
writer = SummaryWriter(log_dir=<path_to_log>)
for i in range(num_epochs):
loss = forward_pass()
writer.add_scalar('loss', loss)And we’re done. Next, fire up tensorboard and view the plots: tensorboard --logdir <path_to_log> . To compare losses (or any other metric) from multiple experiments, the <path_to_log> can be tb_log/<exp_name> where tb_log is a directory containing subdirectories for each experiment. Consecutively, the value of the logdir flag while starting tensorboard would become tb_log .
If on a remote server, I’d recommend pulling the log file back to local and then starting up tensorboard from there. The rsync command from the previous section can be used to regularly update the files in tb_log while keeping tensorboard running in the background.
Get Anant Gupta’s stories in your inbox
Join Medium for free to get updates from this writer.
Alternatively, tensorboard can be started on the remote server and the dashboard can be navigated through public IP:port(6006 is default) . SSH tunnelling can be done if the remote server doesn’t have a public IP address. For example, if I start tensorboard on a server with some IP 10.11.12.13 on the default port 6006, I can tunnel it to my local machine via:
ssh -N -f -L localhost:6006:10.11.12.13:6006 username@10.11.12.13where localhost:6006:10.11.12.13:6006 is <destination_address>:<destination_port>:<source_address>:<source_port> .
Tracking image generation progress through gifs
To qualitatively evaluate the progress of your model easily, one can easily create gifs of generated samples over time.
from torchvision import make_grid
import imageio
grid = []
for i in range(num_epochs):
samples = forward_pass()
# the following functions are in context of pytorch
grid.append(make_grid(samples).permute(1, 2, 0).numpy())
imageio.mimsave('generations.gif', grid)We need to convert generated samples of shape (batch_size, channels, height, width) to (height, width, channel) and hence first join the images to make a grid, then transpose the dimensions and convert to numpy array.
This saves one from squinting over and comparing images generated and saved separately for each epoch.
Logging experiments
Not all of the experiments we run make it to that excel sheet table, and even then it’s difficult to capture all the information in a compact 2-d table, i.e., all the settings and changes associated with each experiment. To make sure all that I’ve run in the past is recorded somewhere without requiring any extra effort (because I wouldn’t care to log an experiment at 3am), run all the experiments from a bash file (and keep commenting out previously ran experiments).
This also gives you the freedom to enter any notes or comments pertaining to those experiments. You can loop over different values of certain hyperparameters to run multiple experiments at once. If your organisation uses a workload manager like Slurm (like mine), then it also makes more sense to follow this technique.
I understand there a bunch of amazing tools out there that are specifically made to manage this, like Sacred, but I just prefer to stick to the old-school ledger style of logging.
These are some of the things I practice to make my life a bit easier on an everyday basis. If you think these are sub-optimal or have some other tricks up your sleeve, let me know and I’d be glad to learn :)
This is my first post in a very long long time, and overall first post in Machine Learning, with which I’ve been working for some time. I plan to share what I’m learning and have learned so far. Feedbacks will be helpful to improve the quality of content in the future.
