Docker, Virtual Machines, and Containerized Application Development With Python & pip

Introduction

If you’ve spent some time scrolling through medium stories in the data science community then you have undoubtedly come across a Docker article looking very similar to this one. Chances are that the article even had a similar picture of shipping containers like the one shown above. I was guilty of reading more than a few of these articles and still not fully grasping what a container is and how Docker works to build/deploy them. I understand that containers are a must for modern application development and that they are “lightweight and portable”, but I couldn’t fully grasp why.

The purpose of this article is to provide a clear and concise explanation of Docker containers in order to help you understand the why. I aim to explain from the bottom up how Docker differs from traditional virtual-machine solutions and why it is so revolutionary. Then I will walk through a basic example of how to create your first Docker container.

What is a Virtual Machine?

Before we dive into Docker and containers, let’s quickly touch on their distant older cousin, virtual machines. The concept of a virtual machine was introduced alongside the first personal computers in the early 1970’s. Computer resources were more expensive to provision back then, so researchers at IBM aimed to discover an economical method to distribute these physical resources among a large group of distinct users. Fast forward to today and virtualization has become the backbone of cloud computing. Companies like AWS and Google Cloud maintain massive data centers, and then provision virtual machines to users on-demand for extremely low cost.

A virtual machine operates nearly identically to a personal computer from the user’s point of view. Virtual machines will have their own “isolated” interface, storage, ram, and operating system. The key difference occurs in how the machine utilizes its physical hardware. On a standard non-virtual machine (think personal computer) physical hardware is directly tied to only that machine. With virtualization, we have multiple virtual machines that all pull from the same physical resources. The distribution of these resources is managed by a hypervisor.

Container vs. Virtual Machine

Now that we understand how virtual machines work, we can introduce what a container is and how the two differ.

A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. -Docker

The picture above highlights the key structural difference between containers and virtual machines. With the traditional virtual machine, a hypervisor manages multiple operating systems running in parallel that all share the same compute resources (The infrastructure). The hypervisor can allocate memory, processors and storage to each VM as needed, but each virtual machine needs its own dedicated guest operating system. A container on the other hand, leverages the operating system of the host to run its applications. The container engine (Docker) can perform virtualization at the OS-level by acting as a middle man between the application and the host operating system.

This major structural difference in OS utilization is how containers can remain so lightweight and portable. They don’t have to carry around the baggage of a beefy operating system. Because of this, containers can be only a couple megabytes in size while virtual machines may take up gigabytes of storage. This allows for containers to be quickly commissioned in seconds whereas virtual machines may take a couple minutes.

Infrastructure Of Docker

Docker utilizes a client-server architecture to build and run its containers. Below are the main components.

1. Docker Host

The docker host is the operating system on which docker is installed. This will be your local machine or the cloud instance where you are hosting your containers.

2. Docker Client

The Docker Client is the primary user interface for Docker. Commands are issued from the command line, then translated to a http request and sent to the Docker daemon.

3. Docker daemon

The Docker daemon is a persistent background process for managing containers under a given Docker Host. The daemon listens for http requests from the docker client and will communicate with the host os to manage docker containers, images, network, and storage.

3. Docker Registry

A public, open-source registry that you can push/pull container images from.

4. Docker Objects

Image: An image is the basic building block of a container. It contains a set of simple, read-only instructions that are used to construct a container. Oftentimes, images will stack on top of each other. For example, if we want to utilize a Python application in our container then we must first build a Linux image, then instruct our image to install Python as well. Images are stored in the form of a Dockerfile.

Container: A container is the executable form of an image and contains applications to be ran. Containers are logically isolated from each other and easily managed with the Docker Client.

How To Use Docker

Time to Dive In

In the below sections, I will first give a very brief explanation of how to install Docker and then list out some of the most common Docker Client commands. Next, we will develop a Docker image from scratch and run a basic “Hello, World!” Python application on it. Finally I will explain the easiest method to pip install packages into a container.

Installation

Docker desktop is available for installation on Windows, Mac, and Linux and contains everything you will need to begin developing and running Docker containers. Installation instructions can be found here. Note: If you are operating on a work computer where you do not have admin privileges then you may run into issues with Docker installation. Docker desktop has an extremely simple user interface so I will skip the detailed walkthrough.

Docker Commands

The primary method to manage and deploy Docker containers is via the command line on the host machine. Each command will start with “docker” and be followed by one of various base commands. Listed below are some of the most common commands.

docker search: Allows you to search all publicly available images from the Docker registry

docker pull: Once we have the name of our desired image, this command allows us pull it into our local host

docker images: Lists out each local image

docker build: Build an image from a local Dockerfile

docker run: Can build and deploy a container based on a saved image

docker ps: Lists all running containers (Can include stopped with — a)

docker stop: Stops a currently running container

docker start: Starts a stopped container

docker restart: Restarts a stopped container

docker exec: Allows you to access the command line of a running container

docker rm: Delete a stopped container and any volumes associated

docker rename: Rename an existing container

docker rmi: Deletes an image

Creating A Docker Container From Scratch

Step 1: Prepare your directory

For this example we will need to create a new directory with two files in it. One of those files is our Dockerfile, which is used to construct our image. The other fill will be main.py which holds our Python “application”.

.

├── Dockerfile

└── main.py

Step 2: Prepare your application script in main.py as shown below:

For this application we will simply just print “Hello, World!”.

#!/usr/bin/env python3
print("Hello, World!")

Step 3: Modify the Dockerfile as shown below:

First, we import a base image containing Python using “FROM” and specify the latest version. Next, we will import our Python script main.py into our container with “COPY”. The “/” denotes the path to our file which is currently in the root directory. Finally, we can use CMD to specify a command to be executed when the container is run. In this case, we want Python to run our main.py script.

FROM python:latest

COPY main.py /

CMD ["python", "./main.py"]

Step 3: Create image

To create an image from our Dockerfile we will use the docker build command from within the directory we created above. Make sure that you have the Docker service running on your computer. If not, then the daemon will not be able to receive your requests. There is no easy way to start the daemon from the command line on Mac OS or Windows so I recommend just opening Docker Desktop and letting it run in the background.

docker build -t my-first-python-image .

Note: The “-t” specifies a tag for your image which is essentially its name. The “.” specifies that you want to create the image from your current directory.

Now, let’s verify that the image was created with docker images .

% docker images

REPOSITORY              TAG       IMAGE ID       CREATED      SIZE

my-first-python-image   latest    72fb37883adf   2 days ago   920MB

We can also check our Docker Desktop.

Step 4: Run container from image

Now that we have our image created, we can now create and run a container based on it. To do this we will use docker run .

% docker run --name python-hello my-first-python-image

Hello, World!

Note: Here, we will use “ — name” to assign a name to our container upon launching it. The way I think about this command is:

Run a container named “python-hello” from “my-first-python-image”

We can verify this container was created with the command line by using docker ps -a . In this case -a is used to show all running/stopped containers. Since our script is not persistent it will stop right after execution.

% docker ps -a

CONTAINER ID   IMAGE                   COMMAND              CREATED         STATUS                     PORTS     NAMES

28e578e72d3e   my-first-python-image   "python ./main.py"   4 minutes ago   Exited (0) 4 minutes ago             python-hello

We can also verify with Docker Desktop.

Step 5: Re-run your container

The beauty of Docker lies in the ability to easily re-run your containers once they have been created. To do this we will use docker start .

% docker start python-hello -a

Hello, World!

Note: The “-a” here is used so that we attach the output to our current command line. Without it, our container will run in detached mode and we wouldn’t see the output.

How To Add Packages To Your Container

The script above was extremely simple, and provides no real value beyond testing out how Docker works. Oftentimes, we will need to import third party packages like Pandas or Tensor Flow for any meaningful Python application. In the following section I will explain the easiest method to pip install a package into a Docker container.

Installing Packages Via Dockerfile

There are a handful of methods to pip install packages into your Docker container, but I feel that adding the pip installation into your Dockerfile is the easiest. For this example, we will modify the directory from above. Don’t worry, since you already have an image and container created you can still re-run them with ease!

Step 1: Modify your Dockerfile

In order to install a package into our Dockerfile we must utilize the RUN command. Keep in mind that the base image that we pulled from is a Python image, which will come with pip already installed. If not, we would first have to install pip before utilizing it. Below we will add instructions in our Dockerfile to pip install pandas:

FROM python:latest

RUN pip install pandas

COPY main.py /

CMD ["python", "./main.py"]

Step 2: Modify your Python script

The next step involves importing pandas into our Python script to verify it was installed correctly. To do this, modify your main.py as below:

#!/usr/bin/env python3
import pandas as pd
df = pd.DataFrame([1,2,3,4])
print(df)

Step 3: Build your image, and run a new container

This step will be executed the exact same way as above so I will skip the in -depth explanation.

% docker build -t my-second-python-image .

...

% docker run --name python-pandas my-second-python-image

0

0  1

1  2

2  3

3  4

We can see that the simple DataFrame was created and printed with no errors.

Conclusion

All in all, Docker is an extremely useful tool for any individual looking to advance their programming skills. With the rise of cloud computing, it is becoming ever more important to be able to produce agile scripts that can run with ease in a variety of computing environments. By virtualizing the OS, Docker was able to improve upon standard virtual machine technology and provide users with a truly portable and lightweight solution for application dev ops. I hope that this article was able to give you a fundamental understanding of Docker. Thank you for reading!

References

https://www.idkrtm.com/history-of-virtualization/

https://stackoverflow.com/questions/50333650/install-python-package-in-docker-file

https://www.freecodecamp.org/news/a-beginners-guide-to-docker-how-to-create-your-first-docker-application-cc03de9b639f/

https://www.geeksforgeeks.org/how-to-run-a-python-script-using-docker/

https://www.docker.com/

Getting Started With Docker was originally published in Python Environments on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Quick-Start Guide To Using venv For Python Environment Management

Introduction

The goal of this article is to provide you with a concise explanation of what venv is and then give you some real code examples on how to begin utilizing it for your Python environment management. Note: This article will assume you already have a working knowledge of Python/pip, and a basic understanding of what a Python environment is. If not, I encourage you to read Managing Python Environments With Anaconda: Part 1, where I give basic explanations of Python environments and pip.

What Is venv?

The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories.

-python.org

venv is the default virtual environment manager for Python 3 and is automatically included in the Python standard library. This means that when you install Python on your machine, venv will come with it.

Structure Of A Virtual Environment

Before diving into our code examples of venv utilization, it is helpful to explain the overall structure of an environment created with venv. Upon initiation of a new virtual environment, venv will create a directory on your machine. This directory is where the virtual environment lives. The typical structure of a venv virtual environment is as follows:

As you can see above, there are four main directories contained in our virtual environment. These are all created automatically by venv upon environment initiation.

1. bin : Contains binary executable files that are utilized in your environment. These are used to activate your environment, execute Python code and much more.
2. include : Contains C headers that compile Python packages
3. lib : Contains site-packages folder which is where the Python standard library and any pip-installed packages will be stored.
4. pyvenv.cfg : Configuration file that stores basic environment information.

venv Commands

In this section I will list out the most common commands used for different environment management applications.

Creating An Environment

The following command will create a Python virtual environment in the location that you specify. If no location is specified, then it will install in your current working directory. (Just be sure to still include a name for your environment) One thing to note is that whatever version of Python you issue the command with will be the version that the environment runs with. To specify your Python version, you must first be sure to have the desired version installed and then just simply replace “python3” with “python 3.10"

python3 -m venv path/to/new/virtual/environment

Activating An Environment

Depending on your os, there may be a different commands for activating the environment. Shown below are commands for several different shells.

Here is how to activate with a standard mac os shell. To deactivate simply type “deactivate”.

source venv_name/bin/activate

Installing Packages

Packages in venv are installed with pip, which will be automatically included when you initiate your environment. Be sure to activate your environment before installing any packages.

pip install "package"="version"

“Freezing” Your Environment

Freezing an environment is the first step you must take before being able to recreate it on a new machine. This creates a snapshot of your current pip-installed packages on your active environment and then saves it to a file typically named “requirements.txt”. This file will be automatically created in your current directory, but you can add in a path if you would like it to be stored somewhere else.

pip freeze > requirements.txt

Recreating Your Environment

Once you have a requirements.txt file, you can share this via email or however you typically share files. To recreate this environment on a new machine, first save the file into your projects working directory. You must first create and activate a new virtual environment. Then, you can issue the following command to copy over all of your packages from the requirements.txt file.

pip install -r requirements.txt

Full Code Example

Explanation

1. Create working directory for example (environments)
2. Create and activate test_env_1
3. Install pandas into test_env_1
4. Pip freeze environment and then deactivate test_env_1
5. Initiate new environment (test_env_2) and then activate it
6. Install requirements.txt file onto test_env_2
7. Verify that pandas as properly installed by importing

Conclusion

In conclusion, venv is a simple and lightweight solution for Python environment management. It comes preinstalled in the Python standard library and each virtual environment created with venv will come with everything you need to begin writing Python scripts. Environments also come with pip included so you can install and manage third party packages with ease.

I hope that this article provided you with a good introduction to venv. Please comment with any questions or disagreements. Thank you for reading! 👋

References:

[Image] https://www.pexels.com/search/relax/

[1]https://docs.python.org/3/library/venv.html

[2]https://zetcode.com/python/venv/#:~:text=The%20pyvenv.,the%20version%20of%20the%20interpreter.

Learn Python venv In Under 5 Minutes was originally published in Python Environments on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Collection of Advanced Methodologies

Introduction

This article is part two in a series dedicated to managing Python environments with Anaconda. This article will assume the reader already has a solid conceptual understanding of what a Python environment is and how to set one up using conda. If not, I highly recommend you to read part one which can be found here.

In this article I will first explain the concept of dependencies and then how to prevent and fix the notorious “dependency hell”. Next, I’ll cover the easiest way to export your current environment and then load it on a new machine. Lastly, the concept of environment variables will be introduced and you will learn an effective method of storing and retrieving sensitive information in Python with the help of conda.

What is a Dependency?

Any discussion regarding Python environments would be incomplete without mentioning dependencies. “Dependencies are all of the software components required by your project in order for it to work as intended and avoid runtime errors.”[2] These “software components” are packages that are used as a building block of another package. For example, in order for pandas to operate it must utilize numpy arrays. In this sense, pandas depends on numpy and cannot work without it. This is why when you install pandas into an environment, numpy will automatically be installed with it.

Managing Dependencies

One common issue you will run into when combining multiple third party packages in one environment is a dependency conflict. For example sake, consider three packages: A, B and C. Let’s say B depends on C_1.3 however A depends on C_1.4. Since both A and B both depend on different versions of C, we run into what is called a transitive dependency conflict.

To check which dependencies a given package has with pip type pip show “package"

(myenv) ericflynn@Erics-MacBook-Air ~ % pip show pandas

Name: pandas

Version: 1.4.2

Summary: Powerful data structures for data analysis, time series, and statistics

Home-page: https://pandas.pydata.org

Author: The Pandas Development Team

Author-email: pandas-dev@python.org

License: BSD-3-Clause

Location: /Users/ericflynn/opt/anaconda3/envs/myenv/lib/python3.10/site-packages

Requires: python-dateutil, numpy, pytz

Required-by:

As you can see above, the output provides you with relevant information pertaining to the package. In the “Requires” field we can view the package dependencies. To perform the same thing with conda, we can use conda search "package" --info The response for conda will include all possible distribution channels available on Anaconda repository so be prepared. Below, I show the result from only one channel. We can view details pertaining to the specific download channel, and package information regarding size and dependencies. One thing I like better about the conda method is that it also shows the version of each dependency. (numpy >=1.19.2,<2.0a0)

(myenv) ericflynn@Erics-MacBook-Air ~ % conda search pandas --info

pandas 1.4.2 py39he9d5cce_0

---------------------------

file name   : pandas-1.4.2-py39he9d5cce_0.conda

name        : pandas

version     : 1.4.2

build       : py39he9d5cce_0

build number: 0

size        : 9.4 MB

license     : BSD-3-Clause

subdir      : osx-64

url         : https://repo.anaconda.com/pkgs/main/osx-64/pandas-1.4.2-py39he9d5cce_0.conda

md5         : 9513b1735fc6feabfb647c545a5be53a

timestamp   : 2022-04-22 10:21:15 UTC

dependencies:

- bottleneck >=1.3.1

- libcxx >=12.0.0

- numexpr >=2.7.1

- numpy >=1.19.2,<2.0a0

- python >=3.9,<3.10.0a0

- python-dateutil >=2.8.1

- pytz >=2020.1

Avoiding Dependency Conflicts

Eventually, you are bound to run into a dependency conflict, and you may even find yourself in “dependency hell”. This is when you are stuck with a complex multi-layered dependency conflict. Conda does have a built in method for preventing dependency conflicts before they occur, but if you are using pip in conjunction with conda then you may still be prone to this issue. Here are some best-practice tips for managing dependencies and avoiding dependency hell.

1. Understand why the issue is occurring
2. Isolate each project in its own environment
3. Install all packages at the same time
4. Stick to only conda packages
5. Degrade packages to resolve if possible

Modifying Package Versions

Sometimes if you find yourself in a minor dependency conflict you may only need to modify one package to a slightly older version and this can fix your issue. Say for example, that a given package we just installed can only run with ≤ Python 3.9, but we’re currently using Python 3.10. To degrade a package on conda we simply just re-install it with the new version:

conda install "package"="version"

Below, we will first issue conda list to verify which version of Python this environment is currently using. Then we can issue our conda install command specifying our desired Python version. Conda will show you which new packages will be downloaded and which downgrades will occur. Take note that Python is not the only package that was modified from this command. Since those other packages depend on Python, they must also be downgraded in order avoid creating any new dependency conflicts.

(myenv) ericflynn@Erics-MacBook-Air envs_to_share % conda list

# packages in environment at /Users/ericflynn/opt/anaconda3/envs/myenv:

...

python                    3.10.4               hdfd78df_0

...

(myenv) ericflynn@Erics-MacBook-Air envs_to_share % conda install python=3.9

...

The following packages will be downloaded:

package                    |            build

---------------------------|-----------------

certifi-2022.5.18.1        |   py39hecd8cb5_0         148 KB

python-3.9.12              |       hdfd78df_0        10.2 MB

setuptools-61.2.0          |   py39hecd8cb5_0        1012 KB

------------------------------------------------------------

Total:        11.4 MB

The following packages will be DOWNGRADED:

certifi                       2022.5.18.1-py310hecd8cb5_0 --> 2022.5.18.1-py39hecd8cb5_0

pip                                21.2.4-py310hecd8cb5_0 --> 21.2.4-py39hecd8cb5_0

python                                  3.10.4-hdfd78df_0 --> 3.9.12-hdfd78df_0

setuptools                         61.2.0-py310hecd8cb5_0 --> 61.2.0-py39hecd8cb5_0

The same process can be done with pip using:

pip install "package"=="version"

The key difference here is “==” opposed to “=”. See below for how I downgraded pandas from 1.4.2 to 1.4.1. The process is very similar to conda, however there is no (y/n) prompt so be sure to type the correct version.

(myenv) ericflynn@Erics-MacBook-Air envs_to_share % pip install pandas==1.4.1

Collecting pandas==1.4.1

...

Attempting uninstall: pandas

Found existing installation: pandas 1.4.2

Uninstalling pandas-1.4.2:

Successfully uninstalled pandas-1.4.2

Successfully installed pandas-1.4.1

How to Share a Conda Environment

Python and its various package repositories are constantly being updated. Sometimes with a big enough change (Like a brand new version of Python), scripts that worked perfectly fine may not work in a new environment that contains different versions of packages. It can be difficult and time consuming to recreate an environment containing each package and version you utilize. Luckily, conda includes commands for this exact purpose! With conda you can create a snapshot of a given environment and then save it as a .yml file. These files can then be loaded up to a new machine, and used to run your scripts. This guarantees that each package will have the exact same version and will also maintain all dependency requirements.

Exporting An Environment to .yml

conda env export > "myenv".yml

This command will create a .yml file in your current directory which is a snapshot of the environment you have activated. (Note: You must be sure to first activate the desired environment) In the snippet below, first I created a new directory to store my environment in with mkdir. Next, I changed my directory to that new folder and issued the export command. Finally, we can verify that the file was created with ls . Don’t forget that wherever your current directory is set to when you issue conda env export, is where the .yml file will be stored.

(base) ericflynn@Erics-MacBook-Air desktop % conda activate myenv

(myenv) ericflynn@Erics-MacBook-Air desktop % mkdir envs_to_share

(myenv) ericflynn@Erics-MacBook-Air desktop % cd envs_to_share

(myenv) ericflynn@Erics-MacBook-Air envs_to_share % conda env export > myenv.yml

(myenv) ericflynn@Erics-MacBook-Air envs_to_share % ls

myenv.yml

Now that we have our environment recreated as a .yml file, we can share this file via email or however you traditionally share documents. Here is what the file itself looks like if you open it in a text editor:

name: myenv
channels:
  - defaults
dependencies:
  - bzip2=1.0.8=h1de35cc_0
  - ca-certificates=2022.4.26=hecd8cb5_0
  - certifi=2022.5.18.1=py310hecd8cb5_0
  - libcxx=12.0.0=h2f01273_0
  - libffi=3.3=hb1e8313_2
  - ncurses=6.3=hca72f7f_2
  - openssl=1.1.1o=hca72f7f_0
  - pip=21.2.4=py310hecd8cb5_0
  - python=3.10.4=hdfd78df_0
  - readline=8.1.2=hca72f7f_1
  - setuptools=61.2.0=py310hecd8cb5_0
  - sqlite=3.38.3=h707629a_0
  - tk=8.6.11=h3fd3227_1
  - tzdata=2022a=hda174b7_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.2.5=hca72f7f_1
  - zlib=1.2.12=h4dc903c_2
  - pip:
    - numpy==1.22.4
    - pandas==1.4.2
    - python-dateutil==2.8.2
    - pytz==2022.1
    - six==1.16.0
variables:
prefix: /Users/ericflynn/opt/anaconda3/envs/myenv

Now lets create a new environment from this .yml file. Since we are on the same machine, we must first delete the environment and then verify it is deleted.

(base) ericflynn@Erics-MacBook-Air ~ % conda remove --name myenv --all

...

(base) ericflynn@Erics-MacBook-Air ~ % conda activate myenv

Could not find conda environment: myenv

You can list all discoverable environments with `conda info --envs`.

Creating An Environment From .yml

Now that the environment is deleted, we can attempt to recreate it from our saved .yml file. First, you must navigate to the folder where you have saved the .yml file with cd . Then we can issue the following command:

conda anv create -f "myenv".yml

This will create a brand new environment based on the .yml file that we created previously. Just like before, make sure to be in the same directory that contains the .yml file you are creating from before issuing the create command.

(base) ericflynn@Erics-MacBook-Air ~ % cd desktop/envs_to_share

(base) ericflynn@Erics-MacBook-Air envs_to_share % conda env create -f myenv.yml

Collecting package metadata (repodata.json): done

...

Successfully installed numpy-1.22.4 pandas-1.4.2 python-dateutil-2.8.2 pytz-2022.1 six-1.16.0

done

We can validate that it was installed by activating and listing out our packages. Note: Packages that we installed with pip are still installed on our recreated environment! (We can see by the “pypi” build channel)

(base) ericflynn@Erics-MacBook-Air envs_to_share % conda activate myenv

(myenv) ericflynn@Erics-MacBook-Air envs_to_share % conda list

# packages in environment at /Users/ericflynn/opt/anaconda3/envs/myenv:

#

# Name                    Version                   Build  Channel

bzip2                     1.0.8                h1de35cc_0

ca-certificates           2022.4.26            hecd8cb5_0

certifi                   2022.5.18.1     py310hecd8cb5_0

libcxx                    12.0.0               h2f01273_0

libffi                    3.3                  hb1e8313_2

ncurses                   6.3                  hca72f7f_2

numpy                     1.22.4                   pypi_0    pypi

openssl                   1.1.1o               hca72f7f_0

pandas                    1.4.2                    pypi_0    pypi

pip                       21.2.4          py310hecd8cb5_0

python                    3.10.4               hdfd78df_0

python-dateutil           2.8.2                    pypi_0    pypi

pytz                      2022.1                   pypi_0    pypi

readline                  8.1.2                hca72f7f_1

setuptools                61.2.0          py310hecd8cb5_0

six                       1.16.0                   pypi_0    pypi

sqlite                    3.38.3               h707629a_0

tk                        8.6.11               h3fd3227_1

tzdata                    2022a                hda174b7_0

wheel                     0.37.1             pyhd3eb1b0_0

xz                        5.2.5                hca72f7f_1

zlib                      1.2.12               h4dc903c_2

Cloning Environments

Another way of recreating environments is using conda’s --clone feature. This can be useful when you would like to add a new package to your environment, but you don’t want to run the risk of dependency issues. A clone can act as a great testing ground so you don’t mess up your perfectly integrated environment. In the command, “CloneName” will be the new name we set for the environment and “myenv” is the environment which we are cloning. We will skip the detailed code example for this one because it is pretty self explanatory.

conda create --name "CloneName" --clone "myenv"

Environment Variables

Environment variables are key-value pairs that you can manually set from within a given conda environment. These variables can then be accessed from any Python script when that environment is activated. One application for environment variables is for storing passwords and secret API keys, since you can call the variables from within Python and don’t have to explicitly type them out in your code. To set a new environment variable, first activate your desired environment and then use the following:

conda env config vars set "key"="value"

Here we will set our variable key “secret_api_key” to be equal to “123abc”. Make sure to re-activate your environment once the variable is set with conda activate "myenv" . You can then verify that the variable was set with conda env config vars list . You may check the value of a single environment variable with echo $"var_name" .

(base) ericflynn@Erics-MacBook-Air ~ % conda activate myenv

(myenv) ericflynn@Erics-MacBook-Air ~ % conda env config vars set secret_api_key=123abc

To make your changes take effect please reactivate your environment

(myenv) ericflynn@Erics-MacBook-Air ~ % conda activate myenv

(myenv) ericflynn@Erics-MacBook-Air ~ % conda env config vars list

secret_api_key = 123abc

(myenv) ericflynn@Erics-MacBook-Air ~ % echo $secret_api_key

123abc

Now that we have our environment variable set, lets launch Python and call it.

(myenv) ericflynn@Erics-MacBook-Air ~ % python

Python 3.10.4 (main, Mar 31 2022, 03:38:35) [Clang 12.0.0 ] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> print(secret_api_key)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

NameError: name 'secret_api_key' is not defined

Hmmm…. so Python says that my “secret_api_key” isn’t defined but I clearly just defined it and even just verified that it is there within conda. The reason that Python cannot call my environment variable from just its key is because environment variables don’t behave like normal variables. They get stored in your system and not Python. So in order to access these variables from Python, we must utilize a package that can integrate Python and our operating system.

Environment Variables and OS

The os package comes pre-installed by default with any Python installation, and it provides numerous classes for interacting with a machine’s operating system. For this article, we will primarily be concerned with one class:

os.environ

Let’s continue where we left off before, after being denied access to our “secret_api_key” within Python. First, we want to import our os package and then we will print our os.environ object.

>>> import os

>>> print(os.environ)

This object will be a dictionary that contains all of the environment variables stored on this environment within your machine. In that dictionary, you should be able to find the “secret_api_key” that we set up before along with multiple other key:value pairs. Environment variables are utilized for much more than just storing of sensitive information. Conda uses them by default to facilitate internal operations that occur during environment management. In the code below, we’re able to fetch certain values from our environment variable dictionary by using .get("Key") .

>>> print(os.environ.get('CONDA_PREFIX'))

/Users/ericflynn/opt/anaconda3/envs/myenv

‘CONDA_PREFIX’ is the path to where our current environment resides. This is an integral piece of information that conda must continuously call on to perform different operations. Without these environment variables, you would have to manually direct the environment where to install new packages by specifying the path explicitly each time. Instead, conda can just check its prefix path and automatically install any new packages there.

Below I will show how we can access our previously created “secret_api_key” environment variable. If you have launched a new instance, be sure to have the correct environment activated and don’t forget to import os before trying the below code.

>>> Key = os.environ.get('secret_api_key')

>>> print(Key)

123abc

Congratulations, you can now utilize your manually defined environment variable from within Python! We can even re-export our environment as a .yml file and we can see the “variables:” section now contains our secret api key.

name: myenv
channels:
  - defaults
dependencies:
  - bzip2=1.0.8=h1de35cc_0
  - ca-certificates=2022.4.26=hecd8cb5_0
  - certifi=2022.5.18.1=py310hecd8cb5_0
  - libcxx=12.0.0=h2f01273_0
  - libffi=3.3=hb1e8313_2
  - ncurses=6.3=hca72f7f_2
  - openssl=1.1.1o=hca72f7f_0
  - pip=21.2.4=py310hecd8cb5_0
  - python=3.10.4=hdfd78df_0
  - readline=8.1.2=hca72f7f_1
  - setuptools=61.2.0=py310hecd8cb5_0
  - sqlite=3.38.3=h707629a_0
  - tk=8.6.11=h3fd3227_1
  - tzdata=2022a=hda174b7_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.2.5=hca72f7f_1
  - zlib=1.2.12=h4dc903c_2
  - pip:
    - numpy==1.22.4
    - pandas==1.4.2
    - python-dateutil==2.8.2
    - pytz==2022.1
    - six==1.16.0
variables:
  secret_api_key: 123abc
prefix: /Users/ericflynn/opt/anaconda3/envs/myenv

If you would like to remove an environment variable you can use conda env config vars unset "var_name" . Be sure to re-activate your environment after.

Conclusion

Conda is an extremely powerful tool and mastering it has helped me in taking my Python programming to the next level. I can now spend much less time stressing over dependency conflicts and “module not found” errors and spend more time actually writing valuable code.

If there is anything that you are confused on or disagree with please leave a comment so we can discuss. Thank you for reading and I hope you learned something!😄

References:

[1] https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

[2] https://www.activestate.com/resources/quick-reads/python-dependencies-everything-you-need-to-know/#:~:text=Dependencies%20are%20all%20of%20the,to%20web%20development%2C%20and%20more.

[3] https://www.nylas.com/blog/making-use-of-environment-variables-in-python/

Managing Python Environments With Anaconda: Part 2 was originally published in Python Environments on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Comprehensive Starters Guide To Anaconda

Introduction

As you work on more advanced Python projects, you will eventually need to install third party packages. These packages are collections of .py files that contain predefined functions and classes that can make your life a whole lot easier when writing Python code. One way to organize these packages is through the use of “environments”. Managing Python environments can be one of the most difficult concepts for a beginner to grasp. If you’re like me, then you probably just pip-installed every package into your base environment. This may work just fine for a while, but eventually you are bound to run into issues.

This article is part one of a series dedicated to explaining different methodologies for managing Python environments. The topic of this writing will be the Anaconda Distribution, the world’s most popular open-source Python distribution platform.

What is a Python Environment?

At the simplest level, a Python environment is a folder located on your machine that contains everything you will need to run your Python scripts. Typically, this includes a Python interpreter, and any third party packages that are to be used within the environment.

As you can see in the picture above, each environment may have a specific installation of Python, and also different versions of each package. This is especially useful when working on multiple projects on the same machine, that must each utilize different versions of the same packages. Environments can also facilitate sharing of scripts with co-workers. Instead of having to install each and every package to run your code, a colleague can just import the entire environment and run your script with ease. (Refer to part two for a specific example)

The Anaconda Distribution

Anaconda is marketed as “the easiest way to perform Python/R data science and machine learning on a single machine”.

The Anaconda distribution is composed of three main components.

1. Anaconda Repository: Collection of commonly used Python/R packages. Similar to the PyPI Index.

“…[The Anaconda repository] features over 8,000 open-source data science and machine learning packages, Anaconda-built and compiled for all major operating systems and architectures.”

2. Conda: Command line environment manager compatible with all major operating systems.

“Conda is an open-source package and environment management system that runs on Windows, macOS, and Linux. Conda quickly installs, runs, and updates packages and their dependencies. It also easily creates, saves, loads, and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.”

3: Anaconda Navigator: Desktop GUI to manage environments without command line.

Anaconda Installation

The download process for Anaconda is fairy simple, and Anaconda maintains excellent documentation so click here to access the detailed guide based on your operating system. Once installed, navigate to your Anaconda 3 folder and explore its contents. Understanding how the Anaconda directories are stored will help drastically in conceptualizing how Python environments work.

A Quick Note On pip

By default, each new Anaconda environment that includes Python will also come with pip pre-installed. Its documentation sums up the capabilities elegantly.

“pip is the package installer for Python. You can use it to install packages from the Python Package Index and other indexes.”

Pip truly is one of my favorite Python package installers. It works great in conjunction with Anaconda, and can allow you to install packages that may not be available on the Anaconda repository. One thing to note, however is that working with pip and Conda in conjunction can lead to potential issues. The rule of thumb is Conda before pip.

Issues may arise when using pip and conda together. When combining conda and pip, it is best to use an isolated conda environment. Only after conda has been used to install as many packages as possible should pip be used to install any remaining software.

Using Anaconda To Manage A Python Environment

For this tutorial, I am going to be using Conda for all of my environment management needs. If you can grasp how to use Conda from the command line, then utilizing the Anaconda Navigator GUI will be a breeze. I use Conda exclusively for environment management because it is fast and simple. (And I enjoy command line programming because it reminds me of the matrix 😎)

Setting Up Conda

Depending on what OS you are operating on, there may be different ways of setting up Conda for use. For this example, I will go over how to set up on Windows, but I will provide a resource for MacOS here. On Windows, Anaconda is nice enough to come pre-installed with Anaconda Prompt, which is an isolated command line dedicated to running Conda commands. If you still prefer to run Conda from the standard Windows command line, you must first add it to your environment path.

1. Type “env” into the Windows search and click the first result

2. Navigate to “advanced” from the tab, and click “Environment Variables…”

3. Click on your “Path” variable and then edit

4. Add new path to the condabin folder within your anaconda3 directory. (“wherever_you_installed\anaconda3\condabin”)

5. Congrats! You should now be able to issue Conda commands from the Windows command line. Test it out by typing “Conda env list” in a new cmd window. You should see your base environment.

Conda Commands

Note: All of these commands will work by default on Anaconda prompt. In order to use in command prompt/terminal, be sure to first enable Conda for your specific operating system.

There are very few Conda commands that you will use on a day to day basis. Here I will show you the 5 top commands and give a brief explanation.

conda activate "env"

This is the most basic of all Conda commands, and it is used to activate a given environment. For this example, since we haven’t created any environments yet, I will activate “base”. Base environment is the standard environment that comes with Anaconda upon installation. As you can see, once activated you will see (base) next to your curser. For those using Anaconda prompt, you will start in your base environment by default so you can skip this one.

conda env list

“conda env list” is used to show all the current Anaconda environments you have installed on your machine. This can be utilized without first activating an environment. As you can see, I have many different Conda environments. This is because I chose to isolate each project to an environment. If you haven’t yet created any Conda environments, then you will only see your base environment here. On the right hand side, you can see the path to each environment directory.

conda create -n "Name_Of_Env" "Package"="Version"

The “conda create” is used to create a brand new Conda environment for use. You will just have to input the desired name of the environment and any packages that you would like to have installed upon initial setup. For this example, we will create a new environment “myenv” with python 3.10 installed on it. By typing “y” + enter you will initiate the installation. Once the download is open, try running “conda env list” again to see it now added into your environment list.

Note: to add multiple packages to initial installation, just separate them with spaces and follow the same “package=version” format.

conda list

“conda list” is similar to “conda env list” however it is used to list out each individual package you have installed in a given environment. For this command, you must first activate your environment of choice.

conda remove --name "Name_Of_Env" --all

“conda remove” is pretty self explanatory and allows you to delete a Conda environment. Make sure to deactivate any open environments before running this command.

Installing Packages

Installing packages into an Anaconda environment is extremely easy. First, we will launch Python from our previously created environment and attempt to import pandas. In order to launch Python, you must first change your directory to the location of where your Python interpreter is saved. Remember, Python environments are just folders that are stored locally on your machine. We must tell our computer which version of Python we want to load. To do this, we will cd (change directory) to the Python installation within our “myenv” environment folder. Recall from before, if you type “conda env list” you can see the location where each of your Conda environments are stored on your local machine. Locate your “myenv” folder, and navigate to it using cd. (Shown below). Then type “python” and you will see the curser change to >>> . This means you are now operating in Python.

If we try importing pandas at this point, we will get an error message because we haven’t yet downloaded the pandas package into “myenv”

To install pandas, open another cmd window and activate “myenv”. Once activated, we can either pip/conda install pandas. In this example, I used pip. The only difference is the location of the repository from which you download the package. (Either PyPi or Conda repository)

As you can see, when we go back into Python and try to import pandas we now have no issue. We can even call on a pandas function to verify that it worked properly.

Conclusion

In conclusion, this article has explained the basics of what a Python environment is and why every Python programmer should be utilizing them. The Anaconda Distribution is an industry leader in Python environment management due to its simplicity, open-source nature, and extensive documentation. Conda is Anaconda’s command line environment manager. This article reviewed how to get started using Conda and then showed the top five Conda commands. These commands should be more than enough to get started and begin experimenting on your own with Anaconda environments. Thank you for reading and I hope that you now have a better understanding of Python environments and Anaconda! 👋

References

https://www.anaconda.com/

https://docs.conda.io/en/latest/

https://pypi.org/

https://pip.pypa.io/en/stable/

https://www.dataquest.io/blog/a-complete-guide-to-python-virtual-environments/

Managing Python Environments With Anaconda: Part 1 was originally published in Python Environments on Medium, where people are continuing the conversation by highlighting and responding to this story.