Python Environments
Getting Started With Docker
Docker, Virtual Machines, and Containerized Application Development With Python & pip
Introduction
If you’ve spent some time scrolling through medium stories in the data science community then you have undoubtedly come across a Docker article looking very similar to this one. Chances are that the article even had a similar picture of shipping containers like the one shown above. I was guilty of reading more than a few of these articles and still not fully grasping what a container is and how Docker works to build/deploy them. I understand that containers are a must for modern application development and that they are “lightweight and portable”, but I couldn’t fully grasp why.
The purpose of this article is to provide a clear and concise explanation of Docker containers in order to help you understand the why. I aim to explain from the bottom up how Docker differs from traditional virtual-machine solutions and why it is so revolutionary. Then I will walk through a basic example of how to create your first Docker container.
What is a Virtual Machine?
Before we dive into Docker and containers, let’s quickly touch on their distant older cousin, virtual machines. The concept of a virtual machine was introduced alongside the first personal computers in the early 1970’s. Computer resources were more expensive to provision back then, so researchers at IBM aimed to discover an economical method to distribute these physical resources among a large group of distinct users. Fast forward to today and virtualization has become the backbone of cloud computing. Companies like AWS and Google Cloud maintain massive data centers, and then provision virtual machines to users on-demand for extremely low cost.
A virtual machine operates nearly identically to a personal computer from the user’s point of view. Virtual machines will have their own “isolated” interface, storage, ram, and operating system. The key difference occurs in how the machine utilizes its physical hardware. On a standard non-virtual machine (think personal computer) physical hardware is directly tied to only that machine. With virtualization, we have multiple virtual machines that all pull from the same physical resources. The distribution of these resources is managed by a hypervisor.
Container vs. Virtual Machine
Now that we understand how virtual machines work, we can introduce what a container is and how the two differ.
A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. -Docker
The picture above highlights the key structural difference between containers and virtual machines. With the traditional virtual machine, a hypervisor manages multiple operating systems running in parallel that all share the same compute resources (The infrastructure). The hypervisor can allocate memory, processors and storage to each VM as needed, but each virtual machine needs its own dedicated guest operating system. A container on the other hand, leverages the operating system of the host to run its applications. The container engine (Docker) can perform virtualization at the OS-level by acting as a middle man between the application and the host operating system.
This major structural difference in OS utilization is how containers can remain so lightweight and portable. They don’t have to carry around the baggage of a beefy operating system. Because of this, containers can be only a couple megabytes in size while virtual machines may take up gigabytes of storage. This allows for containers to be quickly commissioned in seconds whereas virtual machines may take a couple minutes.
Infrastructure Of Docker
Docker utilizes a client-server architecture to build and run its containers. Below are the main components.
1. Docker Host
The docker host is the operating system on which docker is installed. This will be your local machine or the cloud instance where you are hosting your containers.
2. Docker Client
The Docker Client is the primary user interface for Docker. Commands are issued from the command line, then translated to a http request and sent to the Docker daemon.
3. Docker daemon
The Docker daemon is a persistent background process for managing containers under a given Docker Host. The daemon listens for http requests from the docker client and will communicate with the host os to manage docker containers, images, network, and storage.
3. Docker Registry
A public, open-source registry that you can push/pull container images from.
4. Docker Objects
Image: An image is the basic building block of a container. It contains a set of simple, read-only instructions that are used to construct a container. Oftentimes, images will stack on top of each other. For example, if we want to utilize a Python application in our container then we must first build a Linux image, then instruct our image to install Python as well. Images are stored in the form of a Dockerfile.
Container: A container is the executable form of an image and contains applications to be ran. Containers are logically isolated from each other and easily managed with the Docker Client.
How To Use Docker
Time to Dive In
In the below sections, I will first give a very brief explanation of how to install Docker and then list out some of the most common Docker Client commands. Next, we will develop a Docker image from scratch and run a basic “Hello, World!” Python application on it. Finally I will explain the easiest method to pip install packages into a container.
Installation
Docker desktop is available for installation on Windows, Mac, and Linux and contains everything you will need to begin developing and running Docker containers. Installation instructions can be found here. Note: If you are operating on a work computer where you do not have admin privileges then you may run into issues with Docker installation. Docker desktop has an extremely simple user interface so I will skip the detailed walkthrough.
Docker Commands
The primary method to manage and deploy Docker containers is via the command line on the host machine. Each command will start with “docker” and be followed by one of various base commands. Listed below are some of the most common commands.
docker search: Allows you to search all publicly available images from the Docker registry
docker pull: Once we have the name of our desired image, this command allows us pull it into our local host
docker images: Lists out each local image
docker build: Build an image from a local Dockerfile
docker run: Can build and deploy a container based on a saved image
docker ps: Lists all running containers (Can include stopped with — a)
docker stop: Stops a currently running container
docker start: Starts a stopped container
docker restart: Restarts a stopped container
docker exec: Allows you to access the command line of a running container
docker rm: Delete a stopped container and any volumes associated
docker rename: Rename an existing container
docker rmi: Deletes an image
Creating A Docker Container From Scratch
Step 1: Prepare your directory
For this example we will need to create a new directory with two files in it. One of those files is our Dockerfile, which is used to construct our image. The other fill will be main.py which holds our Python “application”.
.├── Dockerfile└── main.py
Step 2: Prepare your application script in main.py as shown below:
For this application we will simply just print “Hello, World!”.
#!/usr/bin/env python3
print("Hello, World!")Step 3: Modify the Dockerfile as shown below:
Get Eric Flynn’s stories in your inbox
Join Medium for free to get updates from this writer.
First, we import a base image containing Python using “FROM” and specify the latest version. Next, we will import our Python script main.py into our container with “COPY”. The “/” denotes the path to our file which is currently in the root directory. Finally, we can use CMD to specify a command to be executed when the container is run. In this case, we want Python to run our main.py script.
FROM python:latestCOPY main.py /CMD ["python", "./main.py"]
Step 3: Create image
To create an image from our Dockerfile we will use the docker build command from within the directory we created above. Make sure that you have the Docker service running on your computer. If not, then the daemon will not be able to receive your requests. There is no easy way to start the daemon from the command line on Mac OS or Windows so I recommend just opening Docker Desktop and letting it run in the background.
docker build -t my-first-python-image .Note: The “-t” specifies a tag for your image which is essentially its name. The “.” specifies that you want to create the image from your current directory.
Now, let’s verify that the image was created with docker images .
% docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmy-first-python-image latest 72fb37883adf 2 days ago 920MB
We can also check our Docker Desktop.
Step 4: Run container from image
Now that we have our image created, we can now create and run a container based on it. To do this we will use docker run .
% docker run --name python-hello my-first-python-imageHello, World!
Note: Here, we will use “ — name” to assign a name to our container upon launching it. The way I think about this command is:
Run a container named “python-hello” from “my-first-python-image”
We can verify this container was created with the command line by using docker ps -a . In this case -a is used to show all running/stopped containers. Since our script is not persistent it will stop right after execution.
% docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES28e578e72d3e my-first-python-image "python ./main.py" 4 minutes ago Exited (0) 4 minutes ago python-hello
We can also verify with Docker Desktop.
Step 5: Re-run your container
The beauty of Docker lies in the ability to easily re-run your containers once they have been created. To do this we will use docker start .
% docker start python-hello -aHello, World!
Note: The “-a” here is used so that we attach the output to our current command line. Without it, our container will run in detached mode and we wouldn’t see the output.
How To Add Packages To Your Container
The script above was extremely simple, and provides no real value beyond testing out how Docker works. Oftentimes, we will need to import third party packages like Pandas or Tensor Flow for any meaningful Python application. In the following section I will explain the easiest method to pip install a package into a Docker container.
Installing Packages Via Dockerfile
There are a handful of methods to pip install packages into your Docker container, but I feel that adding the pip installation into your Dockerfile is the easiest. For this example, we will modify the directory from above. Don’t worry, since you already have an image and container created you can still re-run them with ease!
Step 1: Modify your Dockerfile
In order to install a package into our Dockerfile we must utilize the RUN command. Keep in mind that the base image that we pulled from is a Python image, which will come with pip already installed. If not, we would first have to install pip before utilizing it. Below we will add instructions in our Dockerfile to pip install pandas:
FROM python:latestRUN pip install pandasCOPY main.py /CMD ["python", "./main.py"]
Step 2: Modify your Python script
The next step involves importing pandas into our Python script to verify it was installed correctly. To do this, modify your main.py as below:
#!/usr/bin/env python3
import pandas as pd
df = pd.DataFrame([1,2,3,4])
print(df)Step 3: Build your image, and run a new container
This step will be executed the exact same way as above so I will skip the in -depth explanation.
% docker build -t my-second-python-image ....% docker run --name python-pandas my-second-python-image00 11 22 33 4
We can see that the simple DataFrame was created and printed with no errors.
Conclusion
All in all, Docker is an extremely useful tool for any individual looking to advance their programming skills. With the rise of cloud computing, it is becoming ever more important to be able to produce agile scripts that can run with ease in a variety of computing environments. By virtualizing the OS, Docker was able to improve upon standard virtual machine technology and provide users with a truly portable and lightweight solution for application dev ops. I hope that this article was able to give you a fundamental understanding of Docker. Thank you for reading!
