Eole/benchmarking deployment

Dockerfile

@@ -0,0 +1,8 @@
+FROM nvidia/cuda:11.2.1-base-ubuntu20.04
+RUN apt-get update && \
+    apt-get install --no-install-recommends --no-install-suggests -y \
+    curl python3 python3-pip
+WORKDIR /lambda_diffusers
+COPY . .
+RUN pip3 install --no-cache-dir -r requirements.txt
+CMD ["python3", "-u", "scripts/benchmark.py", "--samples", "1,2,4,8,16"]

docs/benchmark.md

@@ -88,7 +88,9 @@ Interestingly, such a difference may not imply artifacts in half-precision's out
 
 You can use this [Lambda Diffusers](https://github.com/LambdaLabsML/lambda-diffusers) repository to reproduce the results presented in this article.
 
-## Setup
+### From your local machine
+
+#### Setup
 
 Before running the benchmark, make sure you have completed the repository [installation steps](../README.md#installation).
 
@@ -99,7 +101,7 @@ You will then need to set the huggingface access token:
 export ACCESS_TOKEN=<hf_...>
 ```
 
-## Usage
+#### Usage
 
 Launch the `benchmark.py` script to append benchmark results to the existing [benchmark.csv](../benchmark.csv) results file:
 ```
@@ -109,4 +111,72 @@ python ./scripts/benchmark.py
 Lauch the `benchmark_quality.py` script to compare the output of single-precision and half-precision models:
 ```
 python ./scripts/benchmark_quality.py
-```
+```
+
+
+### From a docker container
+
+The following instructions show how to run the benchmarking program from a docker container on Ubuntu.
+
+#### Prerequisites
+
+#### Get a huggingface access token
+
+Create a huggingface account.
+Get a [huggingface access token](https://huggingface.co/docs/hub/security-tokens).
+
+#### Install NVIDIA docker
+
+This section can be skipped if the environment already uses [Lambda Stack](https://lambdalabs.com/lambda-stack-deep-learning-software) or if the experiments are running on a [Lambda cloud](https://lambdalabs.com/service/gpu-cloud) instance as docker and `nvidia-container-toolkit` comes pre-installed in these cases.
+
+We first install docker:
+```
+# Install
+sudo apt-get update
+sudo apt-get remove docker docker-engine docker.io -y
+sudo apt install containerd -y
+sudo apt install docker.io -y
+sudo systemctl start docker
+sudo systemctl enable docker
+# Test install
+docker --version
+# Put the user in the docker group
+sudo usermod -a -G docker $USER
+newgrp docker
+```
+
+We install requirements to run docker containers leveraging NVIDIA GPUs:
+```
+# Install
+sudo apt install curl
+distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
+curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
+curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
+sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
+sudo systemctl restart docker
+# Test install
+sudo docker run --rm --gpus all nvidia/cuda:11.2.1-base-ubuntu20.04 nvidia-smi
+```
+
+
+3. Build the benchmark docker image
+
+```
+docker build -t benchmark -f ./benchmarking/Dockerfile .   
+```
+
+#### Running the benchmark
+
+Set the HuggingFace access token as environment variable:
+```
+export ACCESS_TOKEN=<your-hugging-face-access-token-here>
+```
+
+Run the benchmark program from the container and export the output `.csv` file to the host:
+```
+containerid=$(docker run --gpus all -e ACCESS_TOKEN=${ACCESS_TOKEN} -d --entrypoint "python3" benchmark:latest scripts/benchmark.py --samples=1,2,4,8,16) && \
+docker wait ${containerid} && \
+docker cp ${containerid}:/lambda_diffusers/benchmark_tmp.csv ./benchmark_tmp.csv
+```
+
+*Note that the arguments `scripts/benchmark.py --samples=1,2,4,8,16` can be changed to point to a different script or use different arguments.*