<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sabitha Manoj on Medium]]></title>
        <description><![CDATA[Stories by Sabitha Manoj on Medium]]></description>
        <link>https://medium.com/@sabitha.manoj0891?source=rss-b7b73768c0de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EM-ncgx7afycMKtCYHmr1A.png</url>
            <title>Stories by Sabitha Manoj on Medium</title>
            <link>https://medium.com/@sabitha.manoj0891?source=rss-b7b73768c0de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:01:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sabitha.manoj0891/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Docker Mastery Series — Part 6: Advanced Docker Concepts and Best Practices]]></title>
            <link>https://medium.com/@sabitha.manoj0891/docker-mastery-series-part-6-advanced-docker-concepts-and-best-practices-71f4233ed541?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/71f4233ed541</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 16:33:00 GMT</pubDate>
            <atom:updated>2026-11-18T16:33:00.896Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nMLS3MGE7n27i9Igm8H6g.jpeg" /><figcaption>Image created by author using CoPilot</figcaption></figure><p>Welcome back to the <strong>Docker Mastery Series</strong>.</p><p>In Part 1, we explored the fundamentals of Docker, including images, containers, Docker architecture, and essential Docker commands.</p><p>In Part 2, we moved one step further and learned how to build our own Docker images using Dockerfiles. We explored Dockerfile instructions, image layers, build caching, optimization techniques, security practices, and production-ready Dockerfile design.</p><p>In Part 3, we explored two essential building blocks for real-world containerized applications: <strong>Docker Volumes</strong> and <strong>Docker Networking</strong>. We learned how to persist data beyond the lifecycle of a container, use named volumes and bind mounts, and allow containers to communicate using Docker networks and service names.</p><p>In Part 4, we brought these concepts together with <strong>Docker Compose</strong>. We learned how to define and manage multiple services using a single YAML configuration file and built a complete Flask + PostgreSQL application.</p><p>In Part 5, we applied Docker to <strong>Machine Learning Deployment</strong>. We built a TensorFlow model, created a Flask prediction API, packaged the application into a Docker image, and deployed it inside a container. We also explored CPU and GPU inference and important considerations for production machine learning applications.</p><p>Throughout this series, we have learned how to build, run, connect, and deploy containerized applications.</p><p>But building a Docker container is only one part of the story.</p><p>When applications move closer to production, additional questions become important:</p><ul><li>How can we reduce Docker image size and build time?</li><li>How can we make containers more secure?</li><li>How do we manage secrets and sensitive configuration?</li><li>How can we monitor container health and manage resource consumption?</li><li>How do we distribute and maintain Docker images?</li><li>What practices should we follow when running containers in production?</li></ul><p>These questions lead us to an important topic:</p><blockquote><strong>How do we move from simply running Docker containers to building efficient, secure, and production-ready containerized applications?</strong></blockquote><p>The answer lies in understanding <strong>Advanced Docker Concepts and Best Practices</strong>.</p><p>In this final article, we&#39;ll explore advanced Docker techniques, including multi-stage builds, image optimization, non-root containers, health checks, resource limits, logging, image scanning, and container registries.</p><p>We&#39;ll also bring several of these concepts together through a practical example.</p><h3>Docker Mastery Series</h3><p>✅ <strong>Part 1 — Introduction to Docker</strong></p><p>✅ <strong>Part 2 — Mastering Dockerfiles</strong></p><p>✅ <strong>Part 3 — Mastering Docker Volumes &amp; Networking</strong></p><p>✅ <strong>Part 4 — Docker Compose</strong></p><p>✅ <strong>Part 5 — Docker for Machine Learning</strong></p><p><strong>✅ Part 6 — Advanced Docker (Final Part)</strong></p><h3>1. Moving Beyond Basic Docker</h3><p>In the earlier parts, we learned how to build images, run containers, persist data, configure networks, and manage multi-container applications.</p><p>These fundamentals are essential, but production environments introduce additional requirements.</p><p>A production-ready containerized application should ideally have:</p><ul><li>Small and efficient Docker images</li><li>Secure configurations and minimal privileges</li><li>Reliable startup and shutdown behavior</li><li>Health monitoring</li><li>Controlled resource consumption</li><li>Appropriate logging and error handling</li><li>Versioned and traceable images</li><li>A reliable image distribution process</li></ul><p>Docker provides several features that help achieve these goals.</p><p>Let&#39;s explore them one by one.</p><h3>2. Multi-Stage Docker Builds</h3><p>One common challenge when building Docker images is that the final image may contain unnecessary tools and dependencies.</p><p>For example, an application might require compilers, development libraries, and build tools during installation. However, these tools may not be required when the application is running.</p><p>Including everything in the final image increases its size and potentially expands its attack surface.</p><p><strong>Multi-stage builds</strong> help solve this problem.</p><p>They allow us to use multiple FROM instructions in a single Dockerfile. Each stage can perform a specific task, and we can selectively copy only the required files into the final image.</p><h3>Understanding the concept</h3><p>Consider a Python application that uses a virtual environment.</p><p>The first stage installs the dependencies. The second stage copies the application and installed environment into a clean runtime image.</p><pre>       Build Stage<br>             |<br>             v<br>    Install Dependencies<br>             |<br>             v<br>     Create Virtual Environment<br>             |<br>             v<br>       Required Files<br>             |<br>             v<br>        Runtime Stage<br>             |<br>             v<br>      Final Docker Image</pre><h3>Example: Multi-stage Dockerfile</h3><pre># Stage 1: Build dependencies<br>FROM python:3.11-slim AS builder<br><br># Set the working directory<br>WORKDIR /app<br><br># Create a virtual environment<br>RUN python -m venv /opt/venv<br><br># Use the virtual environment<br>ENV PATH=&quot;/opt/venv/bin:$PATH&quot;<br><br># Install application dependencies<br>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br><br># Stage 2: Create the runtime image<br>FROM python:3.11-slim AS runtime<br><br># Set the working directory<br>WORKDIR /app<br><br># Copy the virtual environment from the builder stage<br>COPY --from=builder /opt/venv /opt/venv<br><br># Activate the virtual environment<br>ENV PATH=&quot;/opt/venv/bin:$PATH&quot;<br><br># Copy application code<br>COPY app.py .<br><br># Start the application<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><h3>Understanding the stages</h3><p><strong>Stage 1 — Builder</strong></p><p>The builder stage creates a virtual environment and installs the required Python packages.</p><p><strong>Stage 2 — Runtime</strong></p><p>The runtime stage starts from a clean Python image and copies only the virtual environment and application code from the builder stage.</p><p>The compiler and other build-only tools are not automatically included in the final image.</p><p>This example assumes that the installed dependencies are compatible with the runtime image and do not require additional system libraries that were omitted.</p><h3>Why use multi-stage builds?</h3><ul><li>Reduce the final image size.</li><li>Keep build tools out of the runtime image.</li><li>Reduce unnecessary software in production containers.</li><li>Separate build-time and runtime responsibilities.</li></ul><p>Multi-stage builds are particularly useful for compiled applications, frontend applications, and projects with complex build dependencies.</p><h3>3. Optimizing Docker Images</h3><p>Docker image size affects storage requirements, image transfer time, and deployment efficiency.</p><p>A large image takes longer to push to a registry, pull onto another machine, and distribute across deployment environments.</p><p>There are several ways to optimize Docker images.</p><h3>3.1 Choose appropriate base images</h3><p>The base image contributes significantly to the final image size.</p><p>For example:</p><pre># Use a smaller Python base image<br>FROM python:3.11-slim</pre><p>A slim image generally contains fewer preinstalled packages than a full Python image.</p><p>However, smaller is not always better. Make sure the selected image includes the operating-system libraries required by your application.</p><h3>3.2 Use Docker layer caching</h3><p>Docker builds images in layers. When an instruction and its relevant inputs remain unchanged, Docker can reuse the previously built layer.</p><p>Consider:</p><pre># Use the lightweight Python 3.11 slim image as the base image<br>FROM python:3.11-slim<br><br># Set /app as the working directory inside the container<br># All subsequent commands will run from this directory<br>WORKDIR /app<br><br># Copy the requirements.txt file from the host machine<br># into the /app directory inside the container<br>COPY requirements.txt .<br><br># Install all Python dependencies listed in requirements.txt<br># --no-cache-dir prevents pip from storing unnecessary cache files<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Copy the application source code (app.py)<br># from the host machine into the container&#39;s /app directory<br>COPY app.py .</pre><p>If only app.py changes, Docker can reuse the dependency-installation layer.</p><p>This can significantly reduce build time during development.</p><h3>3.3 Use .dockerignore</h3><p>Not every file in the project directory needs to be included in the Docker build context.</p><p>Create a .dockerignore file:</p><pre><br># Ignore Python cache directories generated during execution<br>__pycache__/<br><br># Ignore compiled Python bytecode files<br>*.pyc<br><br># Ignore Python virtual environment directories<br>venv/<br>.venv/<br><br># Ignore Git version control metadata and history<br>.git/<br><br># Ignore application log files<br>*.log<br><br># Ignore temporary files created during development or execution<br>*.tmp<br><br># Ignore local environment configuration files<br># These may contain sensitive information such as API keys or passwords<br>.env<br><br># Ignore Pytest cache files generated during test execution<br>.pytest_cache/</pre><p>The .dockerignore file prevents matching files from being sent as part of the build context.</p><p>This helps reduce unnecessary data transfer and avoids accidentally including files that should not be part of the image.</p><p><strong>Important:</strong> Do not exclude files that the application actually needs at runtime.</p><h3>4. Running Containers as a Non-Root User</h3><p>By default, many Docker images run processes as the root user inside the container.</p><p>Running an application as root is generally unnecessary and can increase the impact of a security vulnerability.</p><p>A better practice is to create a dedicated user with only the permissions required by the application.</p><h3>Example</h3><pre><br># Use the lightweight Python 3.11 slim image as the base image<br>FROM python:3.11-slim<br><br># Set /app as the working directory inside the container<br>WORKDIR /app<br><br># Create a dedicated non-root user named appuser<br># --create-home creates a home directory for the new user<br>RUN useradd --create-home appuser<br><br># Copy the application file (app.py) into the container<br># --chown sets appuser as the owner of the copied file<br>COPY --chown=appuser:appuser app.py .<br><br># Switch from the root user to the non-root appuser<br># This improves container security by limiting user privileges<br>USER appuser<br><br># Define the default command to start the Python application<br># Executes app.py when the container starts<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>The useradd instruction creates a user named appuser.</p><p>The COPY --chown instruction assigns ownership of the copied file to that user.</p><p>Finally, USER appuser ensures that subsequent commands and the application process run as the non-root user.</p><p>In production, also consider:</p><ul><li>Restricting container capabilities.</li><li>Avoiding privileged containers.</li><li>Using read-only filesystems where practical.</li><li>Limiting access to mounted host directories.</li><li>Keeping base images and dependencies updated.</li></ul><p>The objective is to give the application only the permissions it needs.</p><h3>5. Managing Environment Variables and Secrets</h3><p>Applications often require configuration values such as:</p><ul><li>Database hostnames</li><li>Database usernames</li><li>API endpoints</li><li>Application modes</li><li>Database passwords</li><li>External service credentials</li></ul><p>Hardcoding these values directly into application code or Dockerfiles is not recommended.</p><p>Environment variables provide a way to separate configuration from application code.</p><h3>Example: Using environment variables</h3><pre># Use the lightweight Python 3.11 slim image as the base image<br>FROM python:3.11-slim<br><br># Set /app as the working directory inside the container<br>WORKDIR /app<br><br># Copy the application source file (app.py)<br># from the host machine into the container&#39;s /app directory<br>COPY app.py .<br><br># Set the default application environment variable to production<br># The application can read this value using os.environ.get(&quot;APP_ENV&quot;)<br>ENV APP_ENV=production<br><br># Define the default command to start the Python application<br># Executes app.py when the container starts<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>The application can read the environment variable:</p><pre><br># Import the os module to access operating system environment variables<br>import os<br><br># Read the APP_ENV environment variable<br># If APP_ENV is not set, use &quot;development&quot; as the default value<br>app_env = os.getenv(&quot;APP_ENV&quot;, &quot;development&quot;)<br><br># Display the current application environment (e.g., production or development)<br>print(f&quot;Running in {app_env} mode&quot;)</pre><p>We can override the value when starting the container:</p><pre># Run the container in detached mode with a custom environment variable<br>docker run -d \<br>  # Assign a name to the container for easy identification<br>  --name my-app \<br>  # Set the APP_ENV environment variable to production inside the container<br>  -e APP_ENV=production \<br>  # Specify the Docker image used to create the container<br>  my-app-image</pre><h3>What about sensitive information?</h3><p>Environment variables are useful for ordinary configuration, but they are not automatically a secure secret-management solution.</p><p>Avoid placing passwords, API keys, or other sensitive information directly into:</p><ul><li>Dockerfiles</li><li>Source code</li><li>Image layers</li><li>Public repositories</li><li>Version-controlled configuration files</li></ul><p>For sensitive values, use appropriate secret-management mechanisms.</p><p>Docker Compose supports secrets for services:</p><pre><br># Define the services (containers) that make up the application<br>services:<br><br>  # Define the web application service<br>  web:<br>    # Specify the Docker image used to create the web container<br>    image: my-app-image<br><br>    # Attach the required secrets to the web service<br>    secrets:<br>      # Make the db_password secret available inside the container<br>      - db_password<br><br># Define the secrets used by the application<br>secrets:<br>  # Declare a secret named db_password<br>  db_password:<br>    # Read the secret value from a local file<br>    # The file should contain the database password<br>    file: ./secrets/db_password.txt</pre><p>The application can access the mounted secret file at:</p><pre>/run/secrets/db_password</pre><p>This example uses a file-backed Compose secret. Its security properties depend on the deployment environment and file permissions. For production, use a suitable secrets-management platform and protect the source secret files.</p><h3>6. Docker Health Checks</h3><p>A container can be running without the application inside it being healthy.</p><p>For example, a web application process might still be running while its API endpoint is failing.</p><p>Docker provides the HEALTHCHECK instruction to periodically test whether an application is responding as expected.</p><h3>Example</h3><pre><br># Use the lightweight Python 3.11 slim image as the base image<br>FROM python:3.11-slim<br><br># Set /app as the working directory inside the container<br>WORKDIR /app<br><br># Copy the dependency list into the container<br>COPY requirements.txt .<br><br># Install all Python dependencies listed in requirements.txt<br># --no-cache-dir prevents pip from storing unnecessary cache files<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Copy the application source file (app.py) into the container<br>COPY app.py .<br><br># Configure a health check to monitor the application&#39;s availability<br># Run the health check every 30 seconds<br>HEALTHCHECK --interval=30s \<br>  # Allow a maximum of 5 seconds for each health check<br>  --timeout=5s \<br>  # Wait 10 seconds after container startup before the first check<br>  --start-period=10s \<br>  # Mark the container as unhealthy after 3 consecutive failures<br>  --retries=3 \<br>  # Send an HTTP request to the application&#39;s health endpoint<br>  # The check succeeds if the endpoint responds successfully<br>  CMD python -c &quot;import urllib.request; urllib.request.urlopen(&#39;http://127.0.0.1:5000/health&#39;, timeout=3)&quot;<br><br># Start the Flask application using Gunicorn as the production WSGI server<br># --bind 0.0.0.0:5000 makes the application listen on all network interfaces<br># app:app refers to the Flask application object named &#39;app&#39; in app.py<br>CMD [&quot;gunicorn&quot;, &quot;--bind&quot;, &quot;0.0.0.0:5000&quot;, &quot;app:app&quot;]</pre><p>Let&#39;s understand the options:</p><p>Let’s understand the options:</p><ul><li>--interval=30s: runs the health check every 30 seconds.</li><li>--timeout=5s: allows five seconds for the check to complete.</li><li>--start-period=10s: provides an initial startup period.</li><li>--retries=3: marks the container unhealthy after three consecutive failures.</li><li>CMD: defines the command used to check the application.</li></ul><p>The example assumes that the Flask application exposes a /health endpoint.</p><p>Check the health status with:</p><pre># Display container health information<br>docker ps</pre><p>Docker reports the container state as healthy, unhealthy, or starting when a health check is configured.</p><p>A health check does not automatically restart an unhealthy container. Restart policies and orchestration tools handle different aspects of recovery.</p><h3>7. Managing Container Resources</h3><p>Containers share the host machine’s resources. Without suitable limits, a resource-intensive application can consume more CPU or memory than intended.</p><p>Docker allows us to configure resource constraints.</p><h3>7.1 Limit memory</h3><pre><br># Start the container in detached mode with a memory limit<br>docker run -d \<br>  # Assign a name to the container for easy identification<br>  --name memory-limited-app \<br>  # Limit the container&#39;s maximum memory usage to 512 MB<br>  --memory=512m \<br>  # Specify the Docker image used to create the container<br>  my-app-image</pre><p>The --memory option sets the container&#39;s memory limit.</p><p>If the application exceeds the available memory, it may be terminated by the operating system or container runtime.</p><h3>7.2 Limit CPU usage</h3><pre><br># Start the container in detached mode with a CPU usage limit<br>docker run -d \<br>  # Assign a name to the container for easy identification<br>  --name cpu-limited-app \<br>  # Limit the container to a maximum of 1 CPU core worth of processing power<br>  --cpus=&quot;1.0&quot; \<br>  # Specify the Docker image used to create the container<br>  my-app-image</pre><p>The --cpus option limits the CPU resources available to the container.</p><h3>7.3 Monitor resource usage</h3><pre># Display live resource usage for running containers<br>docker stats</pre><p>The output includes information such as:</p><ul><li>CPU usage</li><li>Memory usage</li><li>Network input and output</li><li>Block input and output</li><li>Process count</li></ul><p>Resource limits are particularly important for machine learning applications, where model loading and inference can consume substantial memory and CPU resources.</p><p>Limits should be selected based on workload measurements rather than arbitrary values.</p><h3>8. Logging and Troubleshooting</h3><p>Logs are essential for understanding what happens inside a running container.</p><p>Docker captures the standard output and standard error streams of the container’s main process.</p><p>For example:</p><pre># Display the logs of a container<br>docker logs my-app</pre><p>Follow logs continuously:</p><pre># Follow the container logs in real time<br>docker logs -f my-app</pre><p>Display only the most recent lines:</p><pre># Display the last 100 lines of logs<br>docker logs --tail 100 my-app</pre><h3>Inspecting a container</h3><p>Docker also provides the inspect command to retrieve detailed information about a container.</p><pre># Display detailed container configuration<br>docker inspect my-app</pre><p>This can help investigate:</p><ul><li>Container configuration</li><li>Network settings</li><li>Mounted volumes</li><li>Environment variables</li><li>Restart policies</li><li>Health status</li></ul><p>For production environments, consider a centralized logging solution rather than relying exclusively on local container logs.</p><p>Avoid logging passwords, access tokens, personal data, or other sensitive information.</p><h3>9. Restart Policies</h3><p>What happens if a container stops unexpectedly?</p><p>Docker supports restart policies that determine whether a container should restart automatically.</p><p>Common options include:</p><p>PolicyBehaviornoDoes not automatically restart the containeron-failureRestarts when the container exits with a non-zero statusalwaysRestarts the container whenever it stopsunless-stoppedRestarts unless the container was explicitly stopped</p><h3>Example</h3><pre><br># Start the container in detached mode with an automatic restart policy<br>docker run -d \<br>  # Assign a name to the container for easy identification<br>  --name production-app \<br>  # Automatically restart the container if it stops,<br>  # unless it was explicitly stopped by the user<br>  --restart unless-stopped \<br>  # Specify the Docker image used to create the container<br>  my-app-image</pre><p>A restart policy does not fix application errors. It only controls restart behavior.</p><p>For more advanced deployments, orchestration platforms provide additional mechanisms for managing application availability and recovery.</p><h3>10. Docker Image Tagging and Registries</h3><p>Building an image locally is useful during development. However, deploying applications to other machines requires a way to distribute those images.</p><p>Docker registries provide a central location to store and distribute Docker images.</p><p>Examples include:</p><ul><li>Docker Hub</li><li>GitHub Container Registry</li><li>Amazon Elastic Container Registry (ECR)</li><li>Google Artifact Registry</li><li>Azure Container Registry</li></ul><h3>10.1 Tagging an image</h3><p>An image can have multiple tags to identify different versions.</p><pre># Tag the local image with version 1.0<br>docker tag my-app-image myusername/my-app:1.0</pre><p>Here:</p><ul><li>my-app-image is the existing local image.</li><li>myusername/my-app identifies the repository.</li><li>1.0 is the image version tag.</li></ul><p>Avoid relying exclusively on the latest tag in production. Explicit version tags make deployments easier to track and roll back.</p><h3>10.2 Logging in to Docker Hub</h3><pre># Authenticate with Docker Hub<br>docker login</pre><p>Use the authentication method supported by your registry. Avoid putting passwords directly into shell commands or scripts.</p><h3>10.3 Pushing an image</h3><pre># Upload the tagged image to Docker Hub<br>docker push myusername/my-app:1.0</pre><p>The image is now available in the registry, subject to its access permissions.</p><h3>10.4 Pulling an image</h3><p>On another machine, retrieve the image:</p><pre># Download the image from Docker Hub<br>docker pull myusername/my-app:1.0</pre><p>Run the application:</p><pre><br># Start a container using the downloaded Docker image<br>docker run -d \<br>  # Assign a name to the container for easy identification<br>  --name my-app \<br>  # Map port 5000 on the host machine to port 5000 inside the container<br>  # This allows external clients to access the application through the host<br>  -p 5000:5000 \<br>  # Specify the Docker image, username, and version tag<br>  myusername/my-app:1.0</pre><p>This is one of the key benefits of containerization: we can build an image once and deploy that same image to multiple compatible environments.</p><p>For stronger image identity and traceability, deployments can also reference images by digest rather than relying only on mutable tags.</p><h3>11. Docker Image Security</h3><p>Security should be considered throughout the container lifecycle, starting with the base image and continuing through deployment.</p><p>A few important practices include:</p><h3>Use trusted base images</h3><p>Choose official or otherwise trusted images and keep them updated.</p><h3>Scan images for vulnerabilities</h3><p>Image scanning tools can identify known vulnerabilities in operating-system packages and application dependencies.</p><p>Examples include:</p><ul><li>Docker Scout</li><li>Trivy</li><li>Grype</li></ul><p>For example, using Trivy:</p><pre># Scan the Docker image for known vulnerabilities<br>trivy image my-app-image</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*7m7CmYrV0TZrZGAb8f1JCg.png" /></figure><p>The scanner reports vulnerabilities associated with the image’s components.</p><p>Scanning results depend on the vulnerability database and tool version. A scan also does not guarantee that an image is completely secure.</p><h3>Avoid running privileged containers</h3><p>Do not use --privileged unless there is a specific and justified requirement.</p><h3>Keep images updated</h3><p>Regularly rebuild images using maintained base images and updated dependencies. Apply security updates after testing compatibility.</p><h3>Use a minimal attack surface</h3><p>Remove unnecessary packages, avoid including development tools in runtime images, and run applications with limited privileges.</p><h3>Protect the Docker daemon</h3><p>Access to the Docker daemon can provide extensive control over the host system. Restrict access to trusted users and secure remote Docker access.</p><h3>12. Docker BuildKit</h3><p>Docker BuildKit is a modern build backend that improves Docker image building through features such as efficient caching, parallel execution, and improved build controls.</p><p>It can help make builds faster and more flexible.</p><p>For example:</p><pre><br># Enable Docker BuildKit for this image build<br># BuildKit is Docker&#39;s modern build engine that provides improved<br># build performance, caching, and advanced build features<br>DOCKER_BUILDKIT=1 docker build -t my-app-image .</pre><p>BuildKit also supports build secrets, which can provide sensitive values to a build step without baking them into the final image.</p><h3>Example: Using a build secret</h3><p>BuildKit Dockerfile syntax:</p><pre><br># Specify the Dockerfile syntax version to enable modern BuildKit features<br># syntax=docker/dockerfile:1<br><br># Use the lightweight Alpine Linux 3.20 image as the base image<br>FROM alpine:3.20<br><br># Access the build secret only during this build step<br># --mount=type=secret mounts the secret temporarily without storing it<br># id=mysecret identifies the secret passed to the build<br>RUN --mount=type=secret,id=mysecret \<br>    # Read the secret from its temporary mount location<br>    # Write its contents to /tmp/secret-output<br>    cat /run/secrets/mysecret &gt; /tmp/secret-outputFROM alpine:3.20</pre><p>Build the image using:</p><pre><br># Build the Docker image and securely pass a secret file to the build process<br>docker build \<br>  # Provide the secret file to Docker BuildKit<br>  # id=mysecret identifies the secret inside the Dockerfile<br>  # src=secret.txt specifies the local file containing the secret<br>  --secret id=mysecret,src=secret.txt \<br>  # Assign the name &quot;secure-build&quot; to the generated Docker image<br>  -t secure-build \<br>  # Use the current directory as the Docker build context<br>  .</pre><p>This example demonstrates how a build secret can be mounted temporarily during a build instruction.</p><p><strong>Important:</strong> The example writes the secret into /tmp/secret-output, which would persist in the image layer. Do not do this in a real application. Use the secret only for the operation that requires it, and avoid copying it into the image or its layers.</p><p>Build secrets are intended for build-time credentials. Runtime secrets should be managed separately.</p><h3>13. Putting It All Together: A Production-Oriented Dockerfile</h3><p>Let’s combine several of the techniques discussed in this article.</p><p>We will create a simple Flask application image that includes:</p><ul><li>A slim Python base image</li><li>Dependency installation</li><li>A non-root application user</li><li>A health check</li><li>Gunicorn as the application server</li></ul><h3>Project structure</h3><pre>advanced-docker/<br>├── app.py<br>├── requirements.txt<br>├── Dockerfile<br>└── .dockerignore</pre><h3>app.py</h3><pre><br># Import Flask to create the web application<br># Import jsonify to return responses in JSON format<br>from flask import Flask, jsonify<br><br># Create a Flask application instance<br>app = Flask(__name__)<br><br><br># Define a health-check endpoint<br># This endpoint allows Docker or monitoring tools to check application health<br>@app.route(&quot;/health&quot;)<br>def health():<br>    # Return a JSON response indicating that the application is healthy<br>    return jsonify({&quot;status&quot;: &quot;healthy&quot;})<br><br><br># Define the root URL endpoint<br># This endpoint is accessible through http://localhost:5000/<br>@app.route(&quot;/&quot;)<br>def home():<br>    # Return a JSON response with an application status message<br>    return jsonify({<br>        &quot;message&quot;: &quot;Advanced Docker application is running&quot;<br>    })<br><br><br># Check whether this Python file is executed directly<br># rather than being imported as a module<br>if __name__ == &quot;__main__&quot;:<br><br>    # Start the Flask development server<br>    # host=&quot;0.0.0.0&quot; makes the application accessible from outside the container<br>    # port=5000 specifies the port on which the application listens<br>    app.run(host=&quot;0.0.0.0&quot;, port=5000)</pre><h3>requirements.txt</h3><pre>Flask<br>gunicorn</pre><p>For a real deployment, pin the tested versions of these dependencies.</p><h3>Dockerfile</h3><pre><br># Use a lightweight Python 3.11 slim image as the base image<br>FROM python:3.11-slim<br><br># Set /app as the working directory inside the container<br>WORKDIR /app<br><br># Copy the dependency list into the container<br>COPY requirements.txt .<br><br># Install all Python dependencies listed in requirements.txt<br># --no-cache-dir prevents pip from storing unnecessary cache files<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Create a dedicated non-root user named appuser<br># --create-home creates a home directory for the new user<br>RUN useradd --create-home appuser<br><br># Copy the application file into the container<br># --chown assigns ownership of the file to appuser<br>COPY --chown=appuser:appuser app.py .<br><br># Switch from the root user to the non-root appuser<br># This improves container security by limiting user privileges<br>USER appuser<br><br># Document that the application listens on port 5000<br># This does not automatically publish the port on the host machine<br>EXPOSE 5000<br><br># Configure a health check to monitor application availability<br># Run the health check every 30 seconds<br>HEALTHCHECK --interval=30s \<br>  # Allow a maximum of 5 seconds for each health check<br>  --timeout=5s \<br>  # Wait 10 seconds after container startup before the first check<br>  --start-period=10s \<br>  # Mark the container as unhealthy after 3 consecutive failures<br>  --retries=3 \<br>  # Send an HTTP request to the application&#39;s health endpoint<br>  # The check succeeds if the endpoint returns a successful HTTP response<br>  CMD python -c &quot;import urllib.request; urllib.request.urlopen(&#39;http://127.0.0.1:5000/health&#39;, timeout=3)&quot;<br><br># Start the application using Gunicorn, a production WSGI server<br># --bind 0.0.0.0:5000 makes the application listen on all network interfaces<br># app:app refers to the Flask application object named &#39;app&#39; in app.py<br>CMD [&quot;gunicorn&quot;, &quot;--bind&quot;, &quot;0.0.0.0:5000&quot;, &quot;app:app&quot;]</pre><h3>.dockerignore</h3><pre><br># Ignore Git repository files and version control metadata<br>.git/<br><br># Ignore Python cache directories generated during execution<br>__pycache__/<br><br># Ignore compiled Python bytecode files<br>*.pyc<br><br># Ignore Python virtual environment directories<br>venv/<br>.venv/<br><br># Ignore application log files<br>*.log<br><br># Ignore temporary files created during development or execution<br>*.tmp<br><br># Ignore local environment configuration files<br># These may contain sensitive information such as API keys or passwords<br>.env<br><br># Ignore the directory containing local secret files<br># Prevents secret files from being included in the Docker build context<br>secrets/</pre><h3>Build the image</h3><pre># Build the production-oriented Docker image<br>docker build -t advanced-docker-app:1.0 .</pre><h3>Run the container</h3><pre><br># Start the application container in detached mode with production-oriented settings<br>docker run -d \<br>  # Assign a name to the container for easy identification<br>  --name advanced-app \<br>  # Automatically restart the container unless it was explicitly stopped<br>  --restart unless-stopped \<br>  # Map port 5000 on the host machine to port 5000 inside the container<br>  -p 5000:5000 \<br>  # Limit the container&#39;s maximum memory usage to 512 MB<br>  --memory=512m \<br>  # Limit the container&#39;s CPU usage to the equivalent of one CPU core<br>  --cpus=&quot;1.0&quot; \<br>  # Specify the Docker image and version to run<br>  advanced-docker-app:1.0Test the application</pre><pre><br># Send an HTTP GET request to the application&#39;s home endpoint<br># localhost:5000 connects to the application running on port 5000<br>curl http://localhost:5000/</pre><pre># Send an HTTP GET request to the application&#39;s health-check endpoint. <br># Verify that the application is running and responding #successfully<br><br>curl http://localhost:5000/health</pre><h3>Inspect the container</h3><pre><br># Check running containers, their status, port mappings, and health status<br>docker ps<br><br># View the application logs to check startup messages and runtime errors<br>docker logs advanced-app<br><br># Monitor the container&#39;s real-time resource usage,<br># including CPU, memory, network, and disk I/O<br>docker stats advanced-app</pre><p>This example brings together several advanced Docker practices. It is a useful starting point, but a real production deployment would also require tested dependency versions, vulnerability scanning, secure configuration, appropriate monitoring, and an operational deployment strategy.</p><h3>14. Common Advanced Docker Mistakes</h3><p>Even experienced developers can encounter problems when building and deploying containers.</p><p>Here are some mistakes worth avoiding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*XsJmh8NAU7UvBdNMUixWIw.png" /></figure><p>Understanding these practices helps avoid common problems as applications grow and move into production environments.</p><h3>15. Where Does Docker Fit in a Production Workflow?</h3><p>Docker is an important part of modern application deployment, but it is not the entire deployment ecosystem.</p><p>A typical workflow may look like this:</p><pre>        Source Code<br>             |<br>             v<br>        Dockerfile<br>             |<br>             v<br>        Build Image<br>             |<br>             v<br>      Security Scanning<br>             |<br>             v<br>      Push to Registry<br>             |<br>             v<br>      Deploy Container<br>             |<br>             v<br>    Monitor Application<br>             |<br>             v<br>     Update and Maintain</pre><p>For larger applications, this process can be automated using Continuous Integration and Continuous Deployment (CI/CD) tools.</p><p>When an application is deployed across multiple machines or requires automated scaling, scheduling, and recovery, container orchestration platforms such as Kubernetes may become relevant.</p><p>Docker provides the foundation for packaging and running containers, while orchestration platforms help manage containers at a larger scale.</p><p>Not every application needs Kubernetes. For many small and medium-sized applications, Docker Compose or a managed container service may be sufficient.</p><h3>16. Key Takeaways</h3><p>In this final part, we explored advanced Docker concepts and practices that help improve containerized applications.</p><p>We covered:</p><ol><li><strong>Multi-stage builds:</strong> Separating build-time and runtime dependencies.</li><li><strong>Image optimization:</strong> Reducing unnecessary image content and improving build caching.</li><li><strong>Non-root containers:</strong> Running applications with limited privileges.</li><li><strong>Environment variables and secrets:</strong> Separating configuration from application code.</li><li><strong>Health checks:</strong> Checking whether applications are responding correctly.</li><li><strong>Resource limits:</strong> Controlling CPU and memory consumption.</li><li><strong>Logging and troubleshooting:</strong> Inspecting container behavior and diagnosing problems.</li><li><strong>Restart policies:</strong> Defining how containers should behave after stopping.</li><li><strong>Image registries:</strong> Storing and distributing Docker images.</li><li><strong>Image security:</strong> Scanning images and following security best practices.</li><li><strong>BuildKit:</strong> Using modern Docker build features to improve the build process.</li></ol><p>These techniques help make Docker applications more efficient, secure, and manageable.</p><h3>Conclusion: Completing the Docker Mastery Series</h3><p>Over the course of this six-part series, we started with the fundamentals of Docker and gradually moved towards more advanced deployment practices.</p><p>We learned how to:</p><ul><li>Understand Docker architecture, images, and containers.</li><li>Build custom Docker images using Dockerfiles.</li><li>Persist application data using volumes.</li><li>Connect containers using Docker networking.</li><li>Manage multi-container applications with Docker Compose.</li><li>Deploy machine learning models using TensorFlow, Flask, and Docker.</li><li>Optimize, secure, monitor, and maintain containerized applications.</li></ul><p>The most important takeaway is that Docker is more than a tool for running applications in containers. It provides a consistent way to package software, manage dependencies, and simplify application deployment.</p><p>Whether you are developing a web application, building a machine learning API, or managing a multi-service application, understanding Docker fundamentals and best practices is a valuable skill.</p><p><strong>This concludes the Docker Mastery Series.</strong></p><p>Thank you for following the series. I hope these articles have helped you understand Docker concepts and apply them to practical projects.</p><p>Keep building, experimenting, and learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71f4233ed541" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Mastery Series — Part 5: Docker for Machine Learning]]></title>
            <link>https://medium.com/@sabitha.manoj0891/docker-mastery-series-part-5-docker-for-machine-learning-10090bc1a506?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/10090bc1a506</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 15:57:39 GMT</pubDate>
            <atom:updated>2026-11-11T15:57:55.345Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzPIWhaJrYMNFF1nlQqAMg.jpeg" /><figcaption>Image created using Copilot</figcaption></figure><p>Welcome back to the <strong>Docker Mastery Series</strong>.</p><p>In Part 1, we explored the fundamentals of Docker, including images, containers, Docker architecture, and essential Docker commands.</p><p>In Part 2, we moved one step further and learned how to build our own Docker images using Dockerfiles. We explored Dockerfile instructions, image layers, build caching, optimization techniques, security practices, and production-ready Dockerfile design.</p><p>In Part 3, we explored two essential building blocks for real-world containerized applications: <strong>Docker Volumes</strong> and <strong>Docker Networking</strong>. We learned how to persist data beyond the lifecycle of a container, use named volumes and bind mounts, and allow containers to communicate using Docker networks and service names.</p><p>In Part 4, we brought these concepts together with <strong>Docker Compose</strong>. We learned how to define and manage multiple services using a single YAML configuration file and built a complete Flask + PostgreSQL application.</p><p>Now, it’s time to apply what we’ve learned to another important area: <strong>Machine Learning Deployment</strong>.</p><p>Imagine you have trained a machine learning model using TensorFlow. The model works perfectly on your development machine, but when you try to deploy it on another system, you encounter problems with Python versions, library dependencies, missing files, or incompatible environments.</p><p>This raises an important question:</p><blockquote><strong>How can we package and deploy a machine learning application so that it runs consistently across different environments?</strong></blockquote><p>The answer is <strong>Docker</strong>.</p><p>Docker allows us to package a trained machine learning model, its inference application, and all the required dependencies into a container. This makes it easier to deploy the application across compatible environments without manually configuring each system.</p><p>In this article, we’ll build a practical <strong>TensorFlow + Flask machine learning prediction API</strong>, create a Docker image, run it inside a container, and test the deployed API.</p><p>We’ll also explore CPU and GPU inference, dependency management, and important considerations for production-ready machine learning deployments.</p><h3>Docker Mastery Series</h3><p>✅ <strong>Part 1 — Introduction to Docker</strong></p><p>✅ <strong>Part 2 — Mastering Dockerfiles</strong></p><p>✅ <strong>Part 3 — Mastering Docker Volumes &amp; Networking</strong></p><p>✅ <strong>Part 4 — Docker Compose</strong></p><p><strong>✅ Part 5 — Docker for Machine Learning (Current)</strong></p><p>➡️ <strong>Part 6 — Advanced Docker</strong></p><h3>1. Understanding the Machine Learning Workflow</h3><p>A typical machine learning project involves several stages:</p><ol><li>Data collection and preprocessing</li><li>Model training</li><li>Model evaluation</li><li>Saving the trained model</li><li>Building an application for inference</li><li>Deploying the application</li><li>Monitoring the deployed service</li></ol><p>Training and inference serve different purposes. Training uses data to learn model parameters, while inference uses the trained model to generate predictions for new inputs.</p><p>For example, a trained model might classify images, predict customer churn, or estimate the risk of a particular outcome. Once training is complete, we can save the model and load it into an application that accepts input data and returns predictions.</p><p>Docker helps package the inference application so that it can run consistently across compatible environments.</p><h3>How Docker fits into the workflow</h3><pre>Training Data<br>     |<br>     v<br>Train Machine Learning Model<br>     |<br>     v<br>Save Trained Model<br>     |<br>     v<br>Build Prediction API<br>     |<br>     v<br>Create Docker Image<br>     |<br>     v<br>Run Docker Container<br>     |<br>     v<br>Send Request and Receive Prediction</pre><p>Docker does not train the model or improve its accuracy. Instead, it provides a consistent environment for running the application that serves the model.</p><h3>2. Why Use Docker for Machine Learning?</h3><p>Machine learning applications often depend on several libraries, including TensorFlow, NumPy, pandas, and scikit-learn. These libraries may require specific Python versions or compatible dependency versions.</p><p>Docker offers several advantages.</p><p><strong>Consistent environments:</strong> The application can use the same packaged dependencies in development, testing, and deployment.</p><p><strong>Dependency isolation:</strong> Libraries installed inside the container are separated from the host machine’s Python environment.</p><p><strong>Portability:</strong> The image can be deployed on compatible Docker environments without manually reinstalling every dependency.</p><p><strong>Reproducibility:</strong> A version-controlled Dockerfile documents how the application environment is built. Pinning dependency versions and using a controlled base image improves reproducibility.</p><p><strong>Simplified deployment:</strong> Developers can distribute the same container image to other machines or container platforms.</p><p>However, Docker alone does not guarantee complete reproducibility. Hardware differences, external services, random seeds, and unpinned dependencies can still affect results.</p><h3>3. Project Overview</h3><p>We will create a simple binary classification application using TensorFlow and Flask.</p><p>The model will accept two numerical features and predict one of two classes. We will use a synthetic dataset to keep the example small and focused on deployment rather than data preparation.</p><p>Our project structure will look like this:</p><pre>docker-ml-api/<br>├── train_model.py<br>├── app.py<br>├── requirements.txt<br>└── Dockerfile</pre><p>After training, the directory will also contain model.keras, the saved model artifact.</p><p>The files have the following responsibilities:</p><ul><li>train_model.py trains and saves the model.</li><li>app.py loads the saved model and exposes an HTTP API.</li><li>requirements.txt defines the Python dependencies.</li><li>Dockerfile defines the environment used to build the Docker image.</li></ul><p>We will train the model before building the Docker image. This keeps the training workflow separate from the inference application.</p><h3>4. Training and Saving the Model</h3><p>Create a file named train_model.py:</p><pre><br># Import NumPy for numerical operations and data generation<br>import numpy as np<br><br># Import TensorFlow for building and training the neural network<br>import tensorflow as tf<br><br><br># Set random seeds to make the results more reproducible<br># This helps generate consistent random numbers across different runs<br>np.random.seed(42)<br>tf.random.set_seed(42)<br><br><br># Generate 1,000 samples, each containing two numerical features<br># np.random.rand() generates random values between 0 and 1<br># The resulting shape is (1000, 2)<br># astype(np.float32) converts the data to 32-bit floating-point values<br>X = np.random.rand(1000, 2).astype(np.float32)<br><br><br># Assign binary class labels using a simple decision rule<br># If the sum of the two feature values is greater than 1.0, assign class 1<br># Otherwise, assign class 0<br># astype(np.float32) converts the labels into floating-point values<br>y = ((X[:, 0] + X[:, 1]) &gt; 1.0).astype(np.float32)<br><br><br># Define a simple feedforward neural network using Keras Sequential API<br>model = tf.keras.Sequential([<br><br>    # Define the input layer with two features per sample<br>    tf.keras.layers.Input(shape=(2,)),<br><br>    # Hidden layer with 8 neurons and ReLU activation<br>    # ReLU introduces non-linearity, allowing the network to learn patterns<br>    tf.keras.layers.Dense(8, activation=&quot;relu&quot;),<br><br>    # Output layer with one neuron and sigmoid activation<br>    # Sigmoid produces a probability between 0 and 1<br>    # This is suitable for binary classification<br>    tf.keras.layers.Dense(1, activation=&quot;sigmoid&quot;)<br>])<br><br><br># Configure the model before training<br>model.compile(<br><br>    # Adam optimizer adjusts model weights to minimize the loss<br>    optimizer=&quot;adam&quot;,<br><br>    # Binary cross-entropy measures the prediction error<br>    # It is commonly used for binary classification problems<br>    loss=&quot;binary_crossentropy&quot;,<br><br>    # Accuracy measures the percentage of correctly classified samples<br>    metrics=[&quot;accuracy&quot;]<br>)<br><br><br># Train the neural network using the generated dataset<br>model.fit(<br><br>    # Input features used for training<br>    X,<br><br>    # Corresponding binary class labels<br>    y,<br><br>    # Train the model for 20 complete passes through the dataset<br>    epochs=20,<br><br>    # Process 32 samples at a time before updating model weights<br>    batch_size=32,<br><br>    # Reserve 20% of the training data for validation<br>    # Validation data is used to monitor model performance during training<br>    validation_split=0.2,<br><br>    # Suppress detailed training output<br>    verbose=0<br>)<br><br><br># Save the trained model to disk in Keras native format<br># The .keras format stores the model architecture, weights, and configuration<br>model.save(&quot;model.keras&quot;)<br><br><br># Display a confirmation message after saving the model<br>print(&quot;Model saved successfully.&quot;)</pre><p>Run the script in a Python environment with TensorFlow installed:</p><pre>python train_model.py</pre><p>After training, the file model.keras will be created.</p><p>The model uses two input features, a hidden layer with eight neurons, and a sigmoid output layer. The sigmoid output represents the estimated probability of class 1.</p><p>The dataset is synthetic and uses a simple rule to generate labels. It is intended only to demonstrate the deployment workflow, not to represent a real-world prediction problem.</p><p><strong>Why save the model?</strong></p><p>Training can require substantial computation. Saving the model allows the application to load the learned parameters directly instead of retraining whenever the service starts.</p><h3>5. Building a Flask Prediction API</h3><p>Flask is a lightweight Python web framework that allows us to expose the model through HTTP endpoints.</p><p>Create app.py:</p><pre><br># Import Flask to create the web application<br># request handles incoming HTTP requests<br># jsonify converts Python dictionaries into JSON responses<br>from flask import Flask, request, jsonify<br><br># Import NumPy for numerical operations and input validation<br>import numpy as np<br><br># Import TensorFlow for loading the trained machine learning model<br>import tensorflow as tf<br><br><br># Create a Flask application instance<br>app = Flask(__name__)<br><br><br># Load the previously trained Keras model when the application starts<br># Loading the model once avoids reloading it for every prediction request<br>model = tf.keras.models.load_model(&quot;model.keras&quot;)<br><br><br># Define a health-check endpoint<br># This endpoint allows users or monitoring tools to check<br># whether the Flask application is running correctly<br>@app.route(&quot;/health&quot;, methods=[&quot;GET&quot;])<br>def health():<br><br>    # Return a JSON response indicating that the application is healthy<br>    # HTTP status code 200 indicates a successful request<br>    return jsonify({&quot;status&quot;: &quot;ok&quot;}), 200<br><br><br># Define the prediction endpoint<br># This endpoint accepts POST requests containing input features in JSON format<br>@app.route(&quot;/predict&quot;, methods=[&quot;POST&quot;])<br>def predict():<br><br>    # Read the incoming JSON request body<br>    # silent=True prevents Flask from raising an error if JSON parsing fails<br>    data = request.get_json(silent=True)<br><br><br>    # Validate the request structure<br>    # Check that the request contains a JSON dictionary<br>    # and that the required &quot;features&quot; field is present<br>    if not isinstance(data, dict) or &quot;features&quot; not in data:<br><br>        # Return an error message with HTTP 400 (Bad Request)<br>        return jsonify({<br>            &quot;error&quot;: &quot;Provide a &#39;features&#39; field&quot;<br>        }), 400<br><br><br>    try:<br><br>        # Convert the input features into a NumPy array<br>        # float32 is the data type expected by the trained TensorFlow model<br>        features = np.asarray(<br>            data[&quot;features&quot;],<br>            dtype=np.float32<br>        )<br><br><br>        # Validate the input shape<br>        # The trained model expects exactly two numerical features<br>        # The shape must therefore be (2,)<br>        if features.shape != (2,):<br>            raise ValueError(<br>                &quot;Exactly two numerical features are required&quot;<br>            )<br><br><br>        # Validate the numerical values<br>        # Reject NaN (Not a Number) and infinite values<br>        # Machine learning models should receive valid finite inputs<br>        if not np.isfinite(features).all():<br>            raise ValueError(<br>                &quot;Features must contain finite values&quot;<br>            )<br><br><br>        # Reshape the input array into a batch containing one sample<br>        # Original shape: (2,)<br>        # New shape: (1, 2)<br>        # TensorFlow models expect input data in batch format<br>        input_data = features.reshape(1, 2)<br><br><br>        # Pass the input features to the trained model<br>        # model.predict() returns the predicted probability<br>        # verbose=0 suppresses unnecessary prediction output<br>        # float() converts the NumPy value into a standard Python float<br>        probability = float(<br>            model.predict(input_data, verbose=0)[0][0]<br>        )<br><br><br>        # Convert the predicted probability into a binary class label<br>        # Probability &gt;= 0.5 is classified as class 1<br>        # Probability &lt; 0.5 is classified as class 0<br>        prediction = int(probability &gt;= 0.5)<br><br><br>        # Return the prediction and probability as a JSON response<br>        # HTTP status code 200 indicates a successful prediction<br>        return jsonify({<br>            &quot;prediction&quot;: prediction,<br>            &quot;probability&quot;: probability<br>        }), 200<br><br><br>    # Handle invalid input types and validation errors<br>    # Return a client-friendly error message instead of crashing the application<br>    except (TypeError, ValueError):<br><br>        # HTTP 400 indicates that the client provided invalid input<br>        return jsonify({<br>            &quot;error&quot;: &quot;Provide exactly two finite numerical features&quot;<br>        }), 400<br><br><br># Check whether this Python file is executed directly<br># rather than imported as a module<br>if __name__ == &quot;__main__&quot;:<br><br>    # Start Flask&#39;s built-in development server<br>    # host=&quot;0.0.0.0&quot; makes the application accessible from outside<br>    # the container when the port is appropriately exposed<br>    # port=5000 specifies the application listening port<br>    #<br>    # This development server is suitable for local testing only.<br>    # In the Docker production setup, Gunicorn is used instead.<br>    app.run(host=&quot;0.0.0.0&quot;, port=5000)</pre><p>Let’s examine the important components.</p><h3>Loading the model</h3><pre>model = tf.keras.models.load_model(&quot;model.keras&quot;)</pre><p>The saved model is loaded when the application starts. Loading it once avoids repeatedly reading the model file for every request.</p><h3>Creating the prediction endpoint</h3><pre># Define the /predict endpoint to handle HTTP POST requests.<br># This endpoint accepts input features in JSON format,<br># passes them to the trained machine learning model,<br># and returns the predicted class and probability as a JSON response.<br><br>@app.route(&quot;/predict&quot;, methods=[&quot;POST&quot;])</pre><p>This endpoint accepts HTTP POST requests containing JSON input.</p><h3>Generating predictions</h3><pre>model.predict(input_data, verbose=0)</pre><p>The model receives a batch containing one sample. The output probability is converted into a class label using a threshold of 0.5.</p><h3>Returning the result</h3><p>Flask returns the prediction and probability as JSON. The client can process the response without needing direct access to the TensorFlow model.</p><p>In a real application, the preprocessing, feature ordering, and decision threshold must match those used during training. Input validation should also reflect the expected data types and valid ranges.</p><h3>6. Defining the Dependencies</h3><p>Create requirements.txt:</p><pre>Flask<br>gunicorn<br>numpy<br>tensorflow</pre><p>These packages serve different purposes:</p><ul><li><strong>Flask:</strong> handles HTTP requests and responses.</li><li><strong>Gunicorn:</strong> runs the Flask application through a production-oriented WSGI server.</li><li><strong>NumPy:</strong> supports numerical operations.</li><li><strong>TensorFlow:</strong> loads the trained model and performs inference.</li></ul><p>For an actual deployment, use tested, compatible, and pinned package versions. TensorFlow compatibility depends on the Python version and platform, so the versions should be selected and tested together.</p><h3>7. Creating the Dockerfile</h3><p>Now we will define the environment required to run the prediction API.</p><p>Create a file named Dockerfile:</p><pre># Use a Python base image<br>FROM python:3.11-slim<br><br># Set the working directory inside the container<br>WORKDIR /app<br><br># Copy the dependency list<br>COPY requirements.txt .<br><br># Install the required Python packages<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Copy the application code<br>COPY app.py .<br><br># Copy the trained model<br>COPY model.keras .<br><br># Document the application port<br>EXPOSE 5000<br><br># Start the API using Gunicorn<br>CMD [&quot;gunicorn&quot;, &quot;--bind&quot;, &quot;0.0.0.0:5000&quot;, &quot;app:app&quot;]</pre><p>Let’s understand the instructions.</p><p><strong>FROM</strong> selects the base image. The chosen Python version must be supported by the TensorFlow version installed in the image.</p><p><strong>WORKDIR</strong> sets /app as the working directory inside the container.</p><p><strong>COPY requirements.txt .</strong> copies the dependency list into the image.</p><p><strong>RUN pip install ...</strong> installs the Python packages needed by the application.</p><p><strong>COPY app.py . and </strong><strong>COPY model.keras .</strong> copy the API code and trained model into the image.</p><p><strong>EXPOSE 5000</strong> documents the port used by the application. It does not publish the port on the host.</p><p><strong>CMD</strong> specifies the command that runs when the container starts. Gunicorn serves the Flask application using the app:app reference, where the first app identifies app.py and the second identifies the Flask object inside that file.</p><p>Notice that we install dependencies before copying the application code. Docker can reuse the dependency-installation layer when the requirements remain unchanged, making subsequent builds more efficient.</p><p>For production, consider using a pinned base-image digest, tested dependency versions, a non-root user, and a .dockerignore file. These practices improve build reliability and reduce unnecessary files in the build context.</p><h3>8. Building the Docker Image</h3><p>Before building the image, ensure that model.keras exists in the project directory.</p><p>Run:</p><pre># Build the image and assign it a name<br>docker build -t ml-prediction-api .</pre><p>The command uses the Dockerfile in the current directory to create an image named ml-prediction-api.</p><p>The final dot specifies the build context: the files Docker can access during the build.</p><p>List the available images:</p><pre># Display locally available Docker images<br>docker images</pre><p>You should see ml-prediction-api in the output if the build succeeds.</p><h3>9. Running the Container</h3><p>Start the API in detached mode:</p><pre># Run the machine learning API in the background<br>docker run -d \<br>  --name ml-api \<br>  -p 5000:5000 \<br>  ml-prediction-api</pre><p>The options mean:</p><ul><li>-d runs the container in the background.</li><li>--name ml-api assigns a name to the container.</li><li>-p 5000:5000 maps port 5000 on the host to port 5000 inside the container.</li><li>ml-prediction-api identifies the image used to create the container.</li></ul><p>The API is now accessible at http://localhost:5000 from the host machine.</p><p>Check the container status:</p><pre># List running containers<br>docker ps</pre><p>If the container exits or fails to start, inspect the logs:</p><pre># Display the container logs<br>docker logs ml-api</pre><p>TensorFlow may require significant memory and take time to initialize. If the container fails, check the logs, memory availability, dependency compatibility, and whether the model file was copied into the image.</p><h3>10. Testing the API</h3><p>First, check whether the API is responding:</p><pre># Check the API health endpoint<br>curl http://localhost:5000/health</pre><p>An expected response is:</p><pre>{<br>  &quot;status&quot;: &quot;ok&quot;<br>}</pre><p>Next, send a prediction request:</p><pre># Submit two numerical features for prediction<br>curl -X POST http://localhost:5000/predict \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;features&quot;: [0.7, 0.6]}&#39;</pre><p>The API returns a response similar to:</p><pre>{<br>  &quot;prediction&quot;: 1,<br>  &quot;probability&quot;: 0.85<br>}</pre><p>The numerical probability shown here is illustrative. Your actual output depends on the trained model.</p><p>The prediction field contains the predicted class, while probability contains the estimated probability of class 1.</p><p>We can also test input validation:</p><pre># Send invalid input containing only one feature<br>curl -X POST http://localhost:5000/predict \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;features&quot;: [0.7]}&#39;</pre><p>The API should return an HTTP 400 response because the model requires exactly two features.</p><h3>11. Managing the Container</h3><p>Docker provides useful commands for inspecting and managing the deployed application.</p><pre># View the running container<br>docker ps<br># Follow the application logs<br>docker logs -f ml-api<br># Stop the container<br>docker stop ml-api<br># Restart the stopped container<br>docker start ml-api<br># Remove the container after stopping it<br>docker rm -f ml-api</pre><p>The last command removes the container. It does not remove the Docker image.</p><p>To remove the image after removing the container, run:</p><pre># Remove the machine learning API image<br>docker rmi ml-prediction-api</pre><p>Remove an image only when you no longer need it and no remaining container depends on it.</p><h3>12. CPU and GPU Inference</h3><p>Machine learning inference can run on a CPU or a GPU, depending on the model, workload, hardware, and installed software.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*XqsGIVBFcufyhC6BiAgrPQ.png" /></figure><p>For this example, CPU inference is sufficient.</p><p>GPU inference requires more than simply adding a GPU option to docker run. The host needs compatible GPU hardware and drivers, and the container must use a compatible framework build and GPU runtime configuration.</p><p>For NVIDIA GPUs, Docker deployments commonly use the NVIDIA Container Toolkit to make the host GPU available to containers.</p><p>GPU acceleration should be measured rather than assumed. For small models and low request volumes, CPU inference may be simpler and more cost-effective.</p><h3>13. Improving the Deployment for Production</h3><p>The example above demonstrates the core deployment workflow, but a real machine learning service needs additional safeguards.</p><h3>Pin dependency versions</h3><p>Unpinned dependencies can change between builds. Use versions that have been tested together, and consider a lock file or other dependency-management approach.</p><h3>Separate model training from inference</h3><p>Training and serving have different resource requirements. In many projects, model training runs as a separate job, while the inference API loads a versioned model artifact.</p><h3>Avoid placing sensitive data in images</h3><p>Do not copy credentials, private datasets, API keys, or other secrets into a Docker image. Use appropriate secret-management mechanisms and restrict access to sensitive files.</p><h3>Improve health checks</h3><p>A health endpoint can indicate whether the application responds, but it does not necessarily prove that the model is ready to serve predictions. A production deployment should check the relevant readiness conditions and distinguish them from simple process liveness.</p><h3>Monitor performance</h3><p>Track request latency, error rates, memory consumption, throughput, and model-specific metrics where appropriate. Monitoring helps identify infrastructure problems and changes in prediction behavior.</p><h3>Use appropriate resource limits</h3><p>TensorFlow applications can consume substantial memory. Measure the application’s resource requirements and configure container or orchestration limits accordingly.</p><h3>Keep the image lean</h3><p>A .dockerignore file can prevent unnecessary files from entering the build context. Exclude items such as local virtual environments, Git metadata, temporary files, and datasets that are not needed at inference time.</p><h3>14. What About Docker Compose?</h3><p>In Part 4, we learned how Docker Compose manages multiple services.</p><p>For a simple prediction API, a single Docker container may be enough. However, a larger machine learning application might include:</p><ul><li>A Flask or FastAPI inference service</li><li>A database for application metadata</li><li>A message queue for asynchronous prediction jobs</li><li>A monitoring service</li></ul><p>Docker Compose can define and run these services together during development and testing.</p><p>For example, a prediction API could communicate with a database through a Compose network using the database service name rather than localhost.</p><p>The API container and database container would remain separate services, allowing their dependencies and configurations to be managed independently.</p><h3>15. Key Takeaways</h3><p>In this article, we built a simple TensorFlow prediction API and packaged it using Docker.</p><p>The main steps were:</p><ol><li>Train a model and save it as a model artifact.</li><li>Build a Flask API that loads the model and accepts prediction requests.</li><li>Define the required dependencies.</li><li>Create a Dockerfile to package the application.</li><li>Build and run the Docker image.</li><li>Test the API using HTTP requests.</li><li>Review important considerations for production deployment.</li></ol><p>The most important lesson is that <strong>model training and model deployment are separate stages of the machine learning lifecycle</strong>. Docker helps standardize the environment in which the deployed application runs, making it easier to move the application between compatible environments.</p><p>The example is intentionally small. Real-world deployments also require careful data preprocessing, model versioning, security, monitoring, performance testing, and reliable artifact management.</p><h3>Conclusion</h3><p>Docker is a valuable tool for machine learning engineers because it brings consistency and portability to the deployment process. By packaging an inference application and its dependencies into a container, teams can reduce environment-related problems and simplify the process of delivering predictions to other applications.</p><p>This workflow is also a useful starting point for more advanced deployment practices, including automated builds, model versioning, monitoring, GPU inference, and multi-service architectures.</p><p>In Part 6, we will explore advanced Docker concepts and practices that help make containerized applications more secure, efficient, and suitable for production environments.</p><p><strong>Next: Part 6 — Advanced Docker</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10090bc1a506" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Mastery Series — Docker Compose]]></title>
            <link>https://medium.com/@sabitha.manoj0891/docker-mastery-series-docker-compose-23cc76990e2e?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/23cc76990e2e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 07:41:35 GMT</pubDate>
            <atom:updated>2026-10-12T07:41:35.063Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NPhRTlb6D4N8fkfHvgo1Gw.jpeg" /></figure><p>Welcome back to the <strong>Docker Mastery Series</strong>.</p><p>In Part 1, we explored the fundamentals of Docker, including images, containers, Docker architecture, and essential Docker commands.</p><p>In Part 2, we moved one step further and learned how to build our own Docker images using Dockerfiles. We explored Dockerfile instructions, image layers, build caching, optimization techniques, security practices, and production-ready Dockerfile design.</p><p>In Part 3, we explored two essential building blocks for real-world containerized applications: <strong>Docker Volumes</strong> and <strong>Docker Networking</strong>. We learned how to persist data beyond the lifecycle of a container, use named volumes and bind mounts, and allow containers to communicate using Docker networks and service names.</p><p>Now it’s time to bring these concepts together.</p><p>Imagine an application consisting of a Flask API, a PostgreSQL database, a persistent database volume, and a network connecting the services.</p><p>Without Docker Compose, we would need to create and manage each container, network, volume, port mapping, and environment variable separately.</p><p>This raises an important question:</p><blockquote><strong>Is there an easier way to define and manage an entire multi-container application?</strong></blockquote><p>The answer is <strong>Docker Compose</strong>.</p><p>Docker Compose allows us to define the services that make up an application in a single configuration file. Instead of manually running multiple Docker commands, we can describe the application’s containers, networks, volumes, environment variables, and dependencies in one place.</p><p>In this article, we’ll start with the basics of Docker Compose and gradually build a complete <strong>Flask + PostgreSQL application</strong>.</p><h3>Docker Mastery Series</h3><p>✅ <strong>Part 1 — Introduction to Docker</strong></p><p>✅ <strong>Part 2 — Mastering Dockerfiles</strong></p><p>✅ <strong>Part 3 — Mastering Docker Volumes &amp; Networking</strong></p><p><strong>✅ Part 4 — Docker Compose (Current)</strong></p><p>➡️ <strong>Part 5 — Docker for Machine Learning</strong></p><p>➡️ <strong>Part 6 — Advanced Docker</strong></p><h3>What Is Docker Compose?</h3><p>Docker Compose is a tool for defining and running <strong>multi-container applications</strong>.</p><p>Instead of starting each container separately with multiple docker run commands, we can describe the entire application in a single Compose file.</p><p>For example, consider an application with:</p><pre>┌─────────────────┐<br>│    Flask API    │<br>│    Container    │<br>└────────┬────────┘<br>         │<br>         │ Docker Network<br>         │<br>         ▼<br>┌─────────────────┐<br>│   PostgreSQL    │<br>│    Container    │<br>└────────┬────────┘<br>         │<br>         │<br>         ▼<br>┌─────────────────┐<br>│ Persistent      │<br>│ Volume          │<br>└─────────────────┘</pre><p>Without Compose, we might need several commands to create the network, volume, containers, port mappings, and configuration.</p><p>With Compose, the same architecture can be described in a compose.yaml file.</p><p>We can then start the complete application with:</p><pre># Start the services defined in compose.yaml<br># and display their logs in the terminal<br>docker compose up</pre><p>And stop it with:</p><pre># Stop and remove the containers and network created by Docker Compose<br># Named volumes are preserved by default<br>docker compose down</pre><p>This is the fundamental idea behind Docker Compose.</p><h3>Why Do We Need Docker Compose?</h3><p>Let’s consider the application we discussed in Part 3.</p><p>We have two containers:</p><pre>flask-api      # Flask application service that runs the API<br>postgres-db    # PostgreSQL database service that stores application data</pre><p>The Flask application needs to communicate with PostgreSQL.</p><p>The PostgreSQL container needs persistent storage.</p><p>Both containers need to communicate through a Docker network.</p><p>We might manually create everything like this:</p><pre># Create a Docker network<br>docker network create app-network</pre><pre># Create a persistent volume for PostgreSQL<br>docker volume create postgres-data</pre><pre># Start the PostgreSQL container<br>docker run -d \<br>  --name postgres-db \<br>  --network app-network \<br>  -v postgres-data:/var/lib/postgresql/data \<br>  postgres</pre><pre># Start the Flask application container<br>docker run -d \<br>  --name flask-api \<br>  --network app-network \<br>  -p 5000:5000 \<br>  flask-app</pre><p>This works, but there are several things to remember:</p><ul><li>The network name</li><li>The volume name</li><li>Container names</li><li>Port mappings</li><li>Environment variables</li><li>Volume mappings</li><li>Which services depend on each other</li><li>How to start everything again later</li></ul><p>As the application grows, the number of commands and configuration details also grows.</p><p>Docker Compose solves this by allowing us to define the entire application in one configuration file.</p><p>Instead of remembering all these commands, we can simply run:</p><pre># Build the images if needed and start all services defined in docker-compose.yml<br>docker compose up</pre><h3>Docker Compose File</h3><p>The central part of a Compose-based application is the <strong>Compose file</strong>.</p><p>The recommended modern filename is:</p><pre># Define and configure the application services, networks, and persistent volumes<br># in the Docker Compose configuration file<br><br>compose.yaml</pre><p>A simple example looks like this:</p><pre>services:</pre><pre>  web:<br>    image: nginx<br>    ports:<br>      - &quot;8080:80&quot;</pre><p>Let’s break this down.</p><h3>services</h3><p>The services section defines the containers that make up the application.</p><p>In this example, there is one service:</p><pre>web</pre><p>The service uses the Nginx image:</p><pre>image: nginx</pre><p>and publishes port 80 inside the container as port 8080 on the host:</p><pre>ports:<br>  - &quot;8080:80&quot;</pre><p>The application can then be accessed through:</p><pre>http://localhost:8080</pre><h3>Service vs Container</h3><p>One important concept to understand is the difference between a <strong>service</strong> and a <strong>container</strong>.</p><p>Consider:</p><pre>services:</pre><pre>  web:<br>    image: nginx</pre><p>Here:</p><pre>Service name: web<br>Image:       nginx</pre><p>Docker Compose uses this service definition to create and manage the corresponding container.</p><p>In a Compose application, we generally work with the <strong>service name</strong> rather than manually assigning container names.</p><p>For example:</p><pre># Display the logs for the web service defined in the Docker Compose configuration<br>docker compose logs web</pre><p>instead of having to remember a generated container name.</p><h3>A Multi-Container Application</h3><p>Now let’s define two services:</p><pre>services:</pre><pre>  web:<br>    image: nginx</pre><pre>  database:<br>    image: postgres</pre><p>We now have:</p><pre>Docker Compose<br>      │<br>      ├── web<br>      │    └── Nginx<br>      │<br>      └── database<br>           └── PostgreSQL</pre><p>This is the foundation of a multi-container application.</p><p>Each service can have its own:</p><ul><li>Image</li><li>Dockerfile</li><li>Ports</li><li>Environment variables</li><li>Volumes</li><li>Networks</li><li>Dependencies</li><li>Health checks</li></ul><p>Compose brings all of these configurations together.</p><h3>Docker Compose Networking</h3><p>One of the most useful features of Compose is that services can communicate with each other automatically.</p><p>When we start a Compose application, Docker creates a default network for the application.</p><p>For example:</p><pre>Docker Compose<br>      │<br>      ▼<br>app_default network<br>      │<br>      ├───────────────┐<br>      │               │<br>      ▼               ▼<br>    web            database</pre><p>The services can communicate using their <strong>service names</strong>.</p><p>For example, if the database service is:</p><pre>services:</pre><pre>  database:<br>    image: postgres</pre><p>the Flask application can connect to:</p><pre>database</pre><p>as the database hostname.</p><p>This is the same networking principle we learned in Part 3.</p><p>We do not need to find the database container’s IP address.</p><h3>Why Service Names Matter</h3><p>Suppose the Flask application contains:</p><pre>DB_HOST = &quot;localhost&quot;</pre><p>This would be incorrect when PostgreSQL is running in a separate container.</p><p>Inside the Flask container:</p><pre>localhost<br>    ↓<br>Flask container itself</pre><p>Instead, the Flask application should use:</p><pre>DB_HOST = &quot;database&quot;</pre><p>because database is the Compose service name.</p><p>The communication then looks like:</p><pre>Flask container<br>      │<br>      │ database<br>      ▼<br>PostgreSQL container</pre><p>Docker’s internal DNS resolves the service name to the appropriate container.</p><p>This means we do not need to hard-code container IP addresses.</p><h3>Building Our Own Image</h3><p>Compose can also build an image from a Dockerfile.</p><p>Suppose our project looks like this:</p><pre>flask-app/<br>│<br>├── Dockerfile<br>├── app.py<br>├── requirements.txt<br>└── compose.yaml</pre><p>The Compose file can contain:</p><pre>services:</pre><pre>  web:<br>    build: .</pre><p>The build instruction tells Compose to build the Docker image using the Dockerfile in the current directory.</p><p>We can then start the application with:</p><pre>docker compose up --build</pre><p>The --build option tells Compose to build the image before starting the services.</p><h3>image vs build</h3><p>These two options are commonly used in Compose.</p><h3>Using an existing image</h3><pre>services:</pre><pre>  web:<br>    image: nginx</pre><p>Compose uses the specified image.</p><h3>Building an image</h3><pre>services:</pre><pre>  web:<br>    build: .</pre><p>Compose builds the image using the Dockerfile.</p><p>We can also specify both:</p><pre>services:</pre><pre>  web:<br>    build: .<br>    image: flask-app:1.0</pre><p>In this case, Compose builds the image and assigns it the specified image name and tag.</p><h3>Publishing Ports</h3><p>Suppose our Flask application listens on port 5000 inside the container.</p><p>We can publish that port using:</p><pre>services:</pre><pre>  web:<br>    build: .<br>    ports:<br>      - &quot;5000:5000&quot;</pre><p>The format is:</p><pre>HOST_PORT:CONTAINER_PORT</pre><p>Therefore:</p><pre>5000:5000</pre><p>means:</p><pre>Host<br>Port 5000<br>   │<br>   ▼<br>Container<br>Port 5000</pre><p>The Flask application can then be accessed from the host at:</p><pre>http://localhost:5000</pre><h3>Adding Environment Variables</h3><p>Applications often require configuration values such as:</p><pre>DATABASE_HOST<br>DATABASE_PORT<br>DATABASE_NAME<br>DATABASE_USER<br>DATABASE_PASSWORD</pre><p>Compose allows us to provide these values to a container:</p><pre>services:</pre><pre>  web:<br>    build: .<br>    environment:<br>      DATABASE_HOST: database<br>      DATABASE_PORT: &quot;5432&quot;<br>      DATABASE_NAME: appdb<br>      DATABASE_USER: appuser<br>      DATABASE_PASSWORD: examplepassword</pre><p>The Flask application can then read them using Python:</p><pre>import os</pre><pre>DB_HOST = os.getenv(&quot;DATABASE_HOST&quot;)<br>DB_PORT = os.getenv(&quot;DATABASE_PORT&quot;)<br>DB_NAME = os.getenv(&quot;DATABASE_NAME&quot;)<br>DB_USER = os.getenv(&quot;DATABASE_USER&quot;)<br>DB_PASSWORD = os.getenv(&quot;DATABASE_PASSWORD&quot;)</pre><p>This keeps environment-specific configuration outside the application code.</p><h3>Adding PostgreSQL</h3><p>Now let’s add PostgreSQL as a second service.</p><pre>services:</pre><pre>  web:<br>    build: .<br>    ports:<br>      - &quot;5000:5000&quot;</pre><pre>  database:<br>    image: postgres:18<br>    environment:<br>      POSTGRES_DB: appdb<br>      POSTGRES_USER: appuser<br>      POSTGRES_PASSWORD: examplepassword</pre><p>Our application now contains two services:</p><pre>┌─────────────────┐<br>│      web        │<br>│  Flask service  │<br>└────────┬────────┘<br>         │<br>         │ database<br>         │<br>         ▼<br>┌─────────────────┐<br>│    database     │<br>│ PostgreSQL      │<br>└─────────────────┘</pre><p>The Flask service can communicate with PostgreSQL using:</p><pre>database</pre><p>as the hostname.</p><h3>Adding Persistent Storage</h3><p>From Part 3, we know that databases need persistent storage.</p><p>We can define a named volume:</p><pre>services:</pre><pre>  database:<br>    image: postgres:18<br>    volumes:<br>      - postgres-data:/var/lib/postgresql/data</pre><pre>volumes:</pre><pre>  postgres-data:</pre><p>The volume:</p><pre>postgres-data</pre><p>is mounted inside the PostgreSQL container at:</p><pre>/var/lib/postgresql/data</pre><p>The architecture is now:</p><pre>┌─────────────────┐<br>│  Flask service  │<br>└────────┬────────┘<br>         │<br>         │ Docker Network<br>         │<br>         ▼<br>┌─────────────────┐<br>│ PostgreSQL      │<br>│ service         │<br>└────────┬────────┘<br>         │<br>         ▼<br>┌─────────────────┐<br>│ postgres-data   │<br>│ named volume    │<br>└─────────────────┘</pre><p>Now we have both concepts from Part 3 working together:</p><p><strong>Network → communication</strong></p><p><strong>Volume → persistence</strong></p><h3>Starting the Application</h3><p>Once the Compose file is ready, the entire application can be started with:</p><pre>docker compose up</pre><p>To run it in the background:</p><pre>docker compose up -d</pre><p>To build the images before starting:</p><pre>docker compose up --build</pre><p>This is one of the biggest advantages of Compose.</p><p>Instead of running multiple docker run commands, we can start the entire application with a single command.</p><h3>Checking Running Services</h3><p>Use:</p><pre>docker compose ps</pre><p>This shows the services managed by the Compose project.</p><p>For example:</p><pre>NAME                    SERVICE     STATUS<br>flask-app-web           web         Up<br>flask-app-database      database    Up</pre><p>The exact container names may differ depending on the Compose project configuration.</p><h3>Viewing Logs</h3><p>To view logs from all services:</p><pre>docker compose logs</pre><p>To continuously follow the logs:</p><pre>docker compose logs -f</pre><p>To view logs from only the Flask service:</p><pre>docker compose logs -f web</pre><p>To view PostgreSQL logs:</p><pre>docker compose logs -f database</pre><p>This is particularly useful when debugging communication or startup problems.</p><h3>Executing Commands Inside a Service</h3><p>We can also execute commands inside a running service.</p><p>For example:</p><pre>docker compose exec web sh</pre><p>This opens a shell inside the Flask container.</p><p>We can also execute a specific command:</p><pre>docker compose exec web python --version</pre><p>The service name is used instead of manually finding the container name.</p><h3>Stopping the Application</h3><p>To stop the services:</p><pre>docker compose stop</pre><p>The containers remain available and can be started again with:</p><pre>docker compose start</pre><p>This is useful when temporarily stopping an application without removing its containers.</p><h3>Removing the Application</h3><p>To stop and remove the containers and Compose-created networks:</p><pre>docker compose down</pre><p>This does <strong>not normally remove named volumes</strong>.</p><p>That distinction is important when working with databases.</p><p>For example:</p><pre>docker compose down<br>        │<br>        ├── Containers removed<br>        ├── Network removed<br>        └── Named volume remains</pre><p>If you also want to remove the volumes:</p><pre>docker compose down -v</pre><p>Be careful with this command.</p><p>For a PostgreSQL application, removing the volume can remove the persisted database data.</p><h3>depends_on</h3><p>Our Flask application depends on PostgreSQL.</p><p>Compose allows us to express this relationship:</p><pre>services:</pre><pre>  web:<br>    build: .<br>    depends_on:<br>      - database</pre><pre>  database:<br>    image: postgres:18</pre><p>This tells Compose that the database service is a dependency of the web service.</p><p>However, there is an important detail.</p><p>depends_on controls startup order, but simply declaring:</p><pre>depends_on:<br>  - database</pre><p>does not necessarily mean that PostgreSQL is already ready to accept connections.</p><p>A container can be running while the application inside it is still initializing.</p><h3>Health Checks</h3><p>A health check can be used to determine whether PostgreSQL is ready.</p><p>For example:</p><pre>database:<br>  image: postgres:18<br>  environment:<br>    POSTGRES_DB: appdb<br>    POSTGRES_USER: appuser<br>    POSTGRES_PASSWORD: examplepassword</pre><pre>  healthcheck:<br>    test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U appuser -d appdb&quot;]<br>    interval: 10s<br>    timeout: 5s<br>    retries: 5</pre><p>Then the Flask service can depend on the database being healthy:</p><pre>web:<br>  build: .<br>  depends_on:<br>    database:<br>      condition: service_healthy</pre><p>This provides a more reliable startup sequence.</p><h3>Complete Compose Example</h3><p>Now let’s bring everything together.</p><pre>services:</pre><pre>  web:<br>    build: ./app<br>    ports:<br>      - &quot;5000:5000&quot;</pre><pre>    environment:<br>      DATABASE_HOST: database<br>      DATABASE_PORT: &quot;5432&quot;<br>      DATABASE_NAME: appdb<br>      DATABASE_USER: appuser<br>      DATABASE_PASSWORD: examplepassword</pre><pre>    depends_on:<br>      database:<br>        condition: service_healthy</pre><pre>  database:<br>    image: postgres:18</pre><pre>    environment:<br>      POSTGRES_DB: appdb<br>      POSTGRES_USER: appuser<br>      POSTGRES_PASSWORD: examplepassword</pre><pre>    volumes:<br>      - postgres-data:/var/lib/postgresql/data</pre><pre>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U appuser -d appdb&quot;]<br>      interval: 10s<br>      timeout: 5s<br>      retries: 5</pre><pre>volumes:<br>  postgres-data:</pre><p>This single file defines:</p><ul><li>Flask service</li><li>PostgreSQL service</li><li>Port mapping</li><li>Environment variables</li><li>Service dependency</li><li>PostgreSQL health check</li><li>Persistent volume</li><li>Automatic Compose networking</li></ul><p>The application can be started with:</p><pre>docker compose up --build</pre><h3>The Complete Architecture</h3><p>The final architecture looks like this:</p><pre>Host Machine<br>                              │<br>                         Port 5000<br>                              │<br>                              ▼<br>                    ┌─────────────────┐<br>                    │   Flask API     │<br>                    │    Service      │<br>                    └────────┬────────┘<br>                             │<br>                             │ database<br>                             │<br>                    Docker Network<br>                             │<br>                             ▼<br>                    ┌─────────────────┐<br>                    │   PostgreSQL    │<br>                    │    Service      │<br>                    └────────┬────────┘<br>                             │<br>                             ▼<br>                    ┌─────────────────┐<br>                    │ postgres-data   │<br>                    │  Named Volume   │<br>                    └─────────────────┘</pre><p>Each component has one clear responsibility:</p><p>ComponentResponsibilitywebFlask applicationdatabasePostgreSQL databaseCompose networkContainer-to-container communicationpostgres-dataPersistent database storagePort 5000Access Flask from the host</p><h3>Using a .env File</h3><p>Putting passwords directly inside compose.yaml is convenient for a demonstration, but it is not a good practice for sensitive production credentials.</p><p>Compose supports environment-variable substitution.</p><p>Create a .env file:</p><pre>POSTGRES_DB=appdb<br>POSTGRES_USER=appuser<br>POSTGRES_PASSWORD=examplepassword</pre><p>Then reference those values in compose.yaml:</p><pre>services:</pre><pre>  database:<br>    image: postgres:18<br>    environment:<br>      POSTGRES_DB: ${POSTGRES_DB}<br>      POSTGRES_USER: ${POSTGRES_USER}<br>      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}</pre><p>This separates configuration from the Compose file.</p><p>If the .env file contains sensitive information, it should not be committed to a public Git repository.</p><p>Add it to .gitignore:</p><pre>.env</pre><p>For production environments, dedicated secrets-management mechanisms may be more appropriate.</p><h3>Development with Bind Mounts</h3><p>Compose can also be used with bind mounts during development.</p><p>For example:</p><pre>services:</pre><pre>  web:<br>    build: ./app<br>    volumes:<br>      - ./app:/app</pre><p>This maps:</p><pre>Host:<br>./app</pre><pre>      │<br>      ▼</pre><pre>Container:<br>/app</pre><p>If we modify app.py on the host, the container can see the changed file.</p><p>This can make development much more convenient because the image does not need to be rebuilt for every source-code change, depending on the application’s development setup.</p><h3>Named Volumes vs Bind Mounts</h3><p>The same distinction from Part 3 applies to Compose.</p><h3>Named volume</h3><pre>volumes:<br>  - postgres-data:/var/lib/postgresql/data</pre><p>Best suited for persistent application data such as databases.</p><h3>Bind mount</h3><pre>volumes:<br>  - ./app:/app</pre><p>Useful when the host’s files need to be directly shared with a container, particularly during development.</p><p>A common development architecture is therefore:</p><pre>Source code<br>    │<br>    └── Bind Mount ──&gt; Flask container</pre><pre>Database data<br>    │<br>    └── Named Volume ──&gt; PostgreSQL container</pre><h3>Common Docker Compose Mistakes</h3><h3>1. Using localhost for another container</h3><p>Incorrect:</p><pre>DB_HOST = &quot;localhost&quot;</pre><p>Correct:</p><pre>DB_HOST = &quot;database&quot;</pre><p>Remember:</p><blockquote><em>localhost inside a container refers to that container itself.</em></blockquote><h3>2. Forgetting a database volume</h3><p>Without persistent storage, database data may not survive container removal.</p><p>Use:</p><pre>volumes:<br>  - postgres-data:/var/lib/postgresql/data</pre><h3>3. Assuming depends_on means &quot;ready&quot;</h3><p>This:</p><pre>depends_on:<br>  - database</pre><p>controls startup order but does not necessarily mean that PostgreSQL is ready.</p><p>Use a health check when application readiness matters.</p><h3>4. Publishing unnecessary ports</h3><p>The Flask service may need:</p><pre>ports:<br>  - &quot;5000:5000&quot;</pre><p>But PostgreSQL does not necessarily need:</p><pre>ports:<br>  - &quot;5432:5432&quot;</pre><p>if only the Flask container needs to communicate with it.</p><p>The containers can communicate through the internal Compose network without exposing PostgreSQL to the host.</p><h3>5. Hard-coding passwords</h3><p>Avoid committing real credentials directly into:</p><pre>POSTGRES_PASSWORD: my-real-password</pre><p>Use appropriate configuration or secret-management mechanisms instead.</p><h3>6. Accidentally deleting volumes</h3><p>Be careful with:</p><pre>docker compose down -v</pre><p>When a database volume is removed, its persisted data can also be removed.</p><h3>7. Using latest unnecessarily</h3><p>For reproducible environments, explicitly selecting an image version is generally preferable.</p><p>For example:</p><pre>image: postgres:18</pre><p>rather than:</p><pre>image: postgres:latest</pre><p>The exact version should be selected based on the application’s compatibility requirements.</p><h3>Docker Compose Command Reference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*4qkFC3J1zeIKk1B9fi60_A.png" /></figure><h3>Docker CLI vs Docker Compose</h3><p>It is useful to understand what Compose actually changes.</p><p>Without Compose:</p><pre>docker network create<br>        ↓<br>docker volume create<br>        ↓<br>docker build<br>        ↓<br>docker run database<br>        ↓<br>docker run web</pre><p>With Compose:</p><pre>            compose.yaml<br>                  │<br>                  ▼<br>        docker compose up<br>                  │<br>        ┌─────────┼─────────┐<br>        ▼         ▼         ▼<br>     Services  Network   Volumes</pre><p>Docker Compose does not replace Docker.</p><p>Instead, it provides a convenient way to <strong>define and manage multiple Docker resources as one application</strong>.</p><h3>Quick Visual Summary</h3><pre>                    compose.yaml<br>                         │<br>                         ▼<br>                 Docker Compose<br>                         │<br>          ┌──────────────┼──────────────┐<br>          │              │              │<br>          ▼              ▼              ▼<br>       Services       Network         Volumes<br>          │              │              │<br>     ┌────┴────┐         │              │<br>     │         │         │              │<br>     ▼         ▼         │              ▼<br>   Flask   PostgreSQL    │       postgres-data<br>     │         │         │<br>     └────┬────┘         │<br>          │              │<br>          └──────────────┘</pre><p>The key idea is simple:</p><p><strong>Docker provides the containers.</strong></p><p><strong>Dockerfiles build the images.</strong></p><p><strong>Volumes provide persistent storage.</strong></p><p><strong>Networks allow containers to communicate.</strong></p><p><strong>Docker Compose brings these components together and manages them as one application.</strong></p><h3>Conclusion</h3><p>Docker Compose makes multi-container applications much easier to define, start, stop, and reproduce.</p><p>Instead of maintaining a long sequence of Docker commands, we can describe the application’s architecture in a single compose.yaml file.</p><p>We can define:</p><ul><li>Application services</li><li>Database services</li><li>Images</li><li>Dockerfile builds</li><li>Port mappings</li><li>Environment variables</li><li>Networks</li><li>Volumes</li><li>Dependencies</li><li>Health checks</li></ul><p>The Flask + PostgreSQL example demonstrated how these pieces work together:</p><pre>              Docker Compose<br>                     │<br>          ┌──────────┴──────────┐<br>          │                     │<br>          ▼                     ▼<br>     Flask Service         PostgreSQL Service<br>          │                     │<br>          └────── Network ──────┘<br>                                │<br>                                ▼<br>                         Persistent Volume</pre><p>With Docker Compose, a complete multi-container application can be described as configuration and started with a simple command:</p><pre>docker compose up</pre><p>This is an important step toward managing real-world applications with Docker.</p><p>But Docker is not limited to web applications and databases.</p><p>The same concepts can also be applied to <strong>machine learning applications</strong>, where reproducibility, dependency management, model packaging, APIs, and deployment environments are especially important.</p><p>In <strong>Part 5 — Docker for Machine Learning</strong>, we will take what we have learned throughout this series and apply Docker to a practical machine learning workflow.</p><p><strong>Next: Part 5 — Docker for Machine Learning</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23cc76990e2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Docker Mastery Series — Docker Volumes & Networking]]></title>
            <link>https://medium.com/@sabitha.manoj0891/docker-mastery-series-docker-volumes-networking-6575d043969f?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/6575d043969f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-networking]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 16:00:30 GMT</pubDate>
            <atom:updated>2026-09-24T07:12:23.007Z</atom:updated>
            <content:encoded><![CDATA[<h3>🐳 Docker Mastery Series — Part 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Cg-ECNK256ixb3pwQuhdw.jpeg" /><figcaption>Image created using Copilot</figcaption></figure><h4>Learn how Docker persists data, use volumes and bind mounts effectively, understand container networking, and connect multiple containers through practical Python and Flask examples.</h4><h3>Docker Mastery Series</h3><p>Welcome back to the <strong>Docker Mastery Series</strong>.</p><p>In Part 1, we explored the fundamentals of Docker, including images, containers, Docker architecture, and essential Docker commands.</p><p>In Part 2, we moved one step further and learned how to build our own Docker images using Dockerfiles. We explored Dockerfile instructions, image layers, build caching, optimization techniques, security practices, and production-ready Dockerfile design.</p><p>Now it’s time to address two problems that appear as soon as we start building real applications:</p><blockquote><strong><em>Where should application data be stored?</em></strong></blockquote><p>and</p><blockquote><strong><em>How do multiple containers communicate with each other?</em></strong></blockquote><p>A container is designed to be disposable. This is one of Docker’s strengths, but it also creates an important challenge: data stored inside a container can disappear when that container is removed.</p><p>At the same time, modern applications rarely consist of a single process. A typical application might contain a web application, a database, a cache, and several backend services. These components need a reliable way to communicate.</p><p>Docker provides solutions for both problems through <strong>Volumes</strong> and <strong>Networking</strong>.</p><p>In this article, we’ll start with the basics and gradually build toward a multi-container application.</p><h3>Docker Mastery Series</h3><p>✅ <strong>Part 1 — Introduction to Docker</strong></p><p>✅ <strong>Part 2 — Mastering Dockerfiles</strong></p><p><strong>✅ Part 3 — Mastering Docker Volumes &amp; Networking (Current)</strong></p><p>➡️ <strong>Part 4 — Docker Compose</strong></p><p>➡️ <strong>Part 5 — Docker for Machine Learning</strong></p><p>➡️ <strong>Part 6 — Advanced Docker</strong></p><h3>What You Will Learn</h3><p>By the end of this article, you will understand:</p><ul><li>Why container data is ephemeral(short-lived).</li><li>How Docker’s writable container layer works.</li><li>What Docker volumes are.</li><li>How named volumes work.</li><li>How bind mounts work.</li><li>The difference between volumes and bind mounts.</li><li>How to create, inspect, use, and remove volumes.</li><li>How containers communicate with each other.</li><li>How Docker networks work.</li><li>The difference between EXPOSE and port publishing.</li><li>Docker’s main network drivers.</li><li>How to create custom Docker networks.</li><li>How containers discover each other by name.</li><li>Why localhost behaves differently inside containers.</li><li>How to connect a Flask application to PostgreSQL.</li><li>How storage and networking work together in a realistic application.</li><li>Common mistakes and best practices.</li></ul><h3>1. The Problem With Container Storage</h3><p>Let’s start with a simple Python application.</p><p>Suppose our application writes a file:</p><pre>with open(&quot;/app/data.txt&quot;, &quot;w&quot;) as file:<br>    file.write(&quot;Important application data&quot;)</pre><p>Inside the container, the file exists:</p><pre>Container<br><br>/app<br>├── app.py<br>└── data.txt<br><br>#/app → directory/path inside the container<br># app.py → application file<br># data.txt → data file</pre><p>Now imagine the container is removed:</p><pre>docker rm my-container</pre><p>The container disappears.</p><p>The file disappears with it.</p><p>This can be surprising when you’re first learning Docker.</p><h3>Containers Are Designed to Be Disposable</h3><p>A useful way to think about containers is:</p><blockquote><strong><em>A container contains the running environment of an application, not necessarily its permanent data.</em></strong></blockquote><p>For example:</p><pre>Docker Container<br>              ┌─────────────────────┐<br>              │                     │<br>              │   Application       │<br>              │                     │<br>              │   Python            │<br>              │   Libraries         │<br>              │   Temporary files   │<br>              │                     │<br>              └─────────────────────┘<br>                       │<br>                       │ delete<br>                       ▼<br>                    Gone</pre><p>If important data is stored only inside the container, deleting the container can delete that data.</p><p>This is perfectly acceptable for temporary files.</p><p>It is obviously not acceptable for something like a database.</p><p>Imagine storing a production PostgreSQL database entirely inside a disposable container without persistent storage.</p><p>If the container is deleted, the database files could disappear along with it.</p><p>Docker therefore provides several storage mechanisms that allow data to live independently from the container.</p><h3>2. Understanding the Container Writable Layer</h3><p>When Docker creates a container from an image, the image itself remains read-only.</p><p>Docker adds a thin writable layer on top.</p><p>Conceptually:</p><pre>Container<br>        ┌───────────────────┐<br>        │ Writable Layer    │<br>        │                   │<br>        │ Application Data  │<br>        └───────────────────┘<br>                 ▲<br>        ┌───────────────────┐<br>        │ Image Layer       │<br>        │                   │<br>        │ Application       │<br>        │ Python            │<br>        │ Libraries         │<br>        └───────────────────┘</pre><p>The underlying image layers are immutable.</p><p>Changes made by the running container are stored in the container’s writable layer.</p><p>When the container is removed, that writable layer is removed as well.</p><p>Therefore:</p><pre>Container created<br>       ↓<br>Writable layer created<br>       ↓<br>Application writes data<br>       ↓<br>Container removed<br>       ↓<br>Writable layer removed<br>       ↓<br>Data disappears</pre><p>This is why persistent application data should normally be stored outside the container’s writable layer.</p><h3>3. Docker Storage Options</h3><p>Docker provides several ways to handle data.</p><p>The most important ones are:</p><ol><li><strong>Named volumes</strong></li><li><strong>Bind mounts</strong></li><li><strong>tmpfs mounts</strong></li></ol><p>There is also the container’s normal writable layer, which is useful for temporary data.</p><p>A simplified comparison looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*Eh8onVeF-_X1CYBJDDoQeQ.png" /></figure><p>Let’s examine them individually.</p><h3>4. Docker Volumes</h3><p>A <strong>Docker volume</strong> is a storage location managed by Docker that exists independently of the container using it.</p><p>Instead of storing data inside:</p><pre>Container<br>   ↓<br>Writable layer</pre><p>we can store it in:</p><pre>Container<br>   ↓<br>Docker Volume<br>   ↓<br>Persistent Storage</pre><p>The container can be deleted while the volume remains.</p><h3>5. Named Volumes</h3><p>A named volume has an explicit name.</p><p>For example:</p><pre>docker volume create mydata</pre><p>Docker creates a volume called:</p><pre>mydata</pre><p>You can verify it:</p><pre>docker volume ls</pre><p>You might see:</p><pre>DRIVER    VOLUME NAME<br><br>local     mydata</pre><p>The volume now exists independently of any particular container.</p><h3>Using a Named Volume</h3><p>Suppose a container expects its persistent data at:</p><pre>/app/data</pre><p>We can mount our volume there:</p><pre>docker run \<br>  -v mydata:/app/data \<br>  myapp</pre><p>The syntax is:</p><pre>-v &lt;volume-name&gt;:&lt;container-path&gt;</pre><p>So:</p><pre>mydata:/app/data</pre><p>means:</p><pre>Docker Volume<br>     │<br>     │ mounted at<br>     ▼<br>/app/data<br>     │<br>     ▼<br>Container</pre><h3>6. A Practical Volume Example</h3><p>Let’s create a simple Python application.</p><pre>import os<br><br># Define the path where the data file will be stored inside the container<br>DATA_FILE = &quot;/app/data/message.txt&quot;<br><br># Create the /app/data directory if it does not already exist<br># exist_ok=True prevents an error if the directory already exists<br>os.makedirs(&quot;/app/data&quot;, exist_ok=True)<br><br># Open the file in write mode (&quot;w&quot;)<br># If the file does not exist, it will be created<br>with open(DATA_FILE, &quot;w&quot;) as file:<br>    # Write the message to the file<br>    file.write(&quot;This data should survive container deletion.&quot;)<br><br># Display a confirmation message<br>print(&quot;Data written successfully.&quot;)</pre><p>Suppose this application is packaged into an image called:</p><pre>volume-demo</pre><p>First create a volume:</p><pre># Create a named Docker volume called &quot;app-data&quot;<br># The volume is managed by Docker and can persist data<br># independently of the container<br>docker volume create app-data</pre><p>Then run the container:</p><pre># Run a container and mount the &quot;app-data&quot; volume<br>docker run \<br>  --name volume-container \<br>  -v app-data:/app/data \<br>  volume-demo</pre><p>The application writes:</p><pre>/app/data/message.txt</pre><p>But that directory is backed by the Docker volume.</p><p>Now remove the container:</p><pre>docker rm volume-container</pre><p>The container is gone.</p><p>But the volume remains.</p><p>Check:</p><pre>docker volume ls</pre><p>You should still see:</p><pre>app-data</pre><p>Create another container using the same volume:</p><pre>docker run \<br>  --rm \<br>  -v app-data:/app/data \<br>  volume-demo</pre><p>The volume can be reused independently of the previous container.</p><h3>7. Managing Docker Volumes</h3><p>Docker provides several commands for managing volumes.</p><h4>List volumes</h4><pre>docker volume ls</pre><h4>Create a volume</h4><pre>docker volume create app-data</pre><h4>Inspect a volume</h4><pre>docker volume inspect app-data</pre><h4>Remove a volume</h4><pre>docker volume rm app-data</pre><h4>Remove unused volumes</h4><pre>docker volume prune</pre><p>Be careful with:</p><pre>docker volume prune</pre><p>It removes unused local volumes.</p><p>If a volume contains important data, make sure it is no longer needed before deleting it.</p><h3>8. Inspecting a Volume</h3><p>Run:</p><pre>docker volume inspect app-data</pre><p>Docker returns metadata similar to:</p><pre>[<br>    {<br>        &quot;Name&quot;: &quot;app-data&quot;,<br>        &quot;Driver&quot;: &quot;local&quot;,<br>        &quot;Mountpoint&quot;: &quot;/var/lib/docker/volumes/app-data/_data&quot;<br>    }<br>]</pre><p>The exact location can vary depending on the operating system and Docker configuration.</p><p>The important point is that Docker manages this storage location for you.</p><p>You normally don’t need to manipulate the underlying files directly.</p><h3>9. Volumes in Dockerfile</h3><p>In Part 2, we introduced the VOLUME Dockerfile instruction.</p><p>For example:</p><pre>VOLUME [&quot;/app/data&quot;]</pre><p>This indicates that /app/data is intended to be used as a mount point for persistent data.</p><p>However, it is important to understand the distinction between <strong>declaring a volume mount point</strong> in an image and <strong>choosing a specific volume</strong> when running a container.</p><p>For example:</p><pre>VOLUME [&quot;/app/data&quot;]</pre><p>declares the location.</p><p>At runtime, you can explicitly mount a named volume:</p><pre>docker run \<br>  -v app-data:/app/data \<br>  myapp</pre><p>For modern applications, explicitly managing volumes at deployment time often provides clearer control over storage.</p><h3>10. Bind Mounts</h3><p>Named volumes are managed by Docker.</p><p>A <strong>bind mount</strong>, on the other hand, maps a specific directory or file from the host machine into the container.</p><p>For example:</p><pre>docker run \<br>  -v $(pwd):/app \<br>  python-app</pre><p>This maps the current directory on your computer to /app inside the container.</p><p>Conceptually:</p><p><strong><em>Your computer</em></strong></p><pre>project/<br>├── app.py<br>├── requirements.txt<br>└── Dockerfile<br>       │<br>       │ bind mount<br>       ▼<br>Container</pre><pre>/app<br>├── app.py<br>├── requirements.txt<br>└── Dockerfile</pre><p>The files are not copied into the image.</p><p>The container accesses the files directly through the mount.</p><h3>11. Why Bind Mounts Are Useful</h3><p>Bind mounts are particularly useful during development.</p><p>Suppose your Flask application contains:</p><pre>print(&quot;Hello Docker&quot;)</pre><p>You start the container with your source directory mounted:</p><pre>docker run \<br>  -v $(pwd):/app \<br>  flask-app</pre><p>Now edit the file on your host:</p><pre>print(&quot;Hello Docker Volumes&quot;)</pre><p>The container sees the changed file because both the host and container are accessing the same mounted directory.</p><p>This is extremely useful for development workflows.</p><h3>12. Named Volumes vs Bind Mounts</h3><p>This distinction is important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*akeONVMiMTqI3ALjEKblyg.png" /></figure><p>A useful rule of thumb is:</p><blockquote><strong><em>Use named volumes for application data that Docker should manage. Use bind mounts when you intentionally need to share specific host files or directories with a container.</em></strong></blockquote><h3>13. Bind Mount Syntax</h3><p>The traditional -v syntax is:</p><pre>docker run \<br>  -v /host/path:/container/path \<br>  image-name</pre><p>Docker also supports the more explicit --mount syntax:</p><pre># Run a container and bind-mount the current host directory to /app<br>docker run \<br>  --mount type=bind,source=$(pwd),target=/app \<br>  image-name</pre><p>Here:</p><ul><li>type=bind → creates a bind mount</li><li>source=$(pwd) → uses the current directory on the host</li><li>target=/app → mounts that directory at /app inside the container</li><li>image-name → the Docker image used to create the container</li></ul><h3>14. Read-Only Mounts</h3><p>Sometimes a container should be able to read files but should not be allowed to modify them.</p><p>Docker supports read-only mounts.</p><p>For example:</p><pre>docker run \<br>  -v $(pwd):/app:ro \<br>  myapp</pre><p>The :ro means:</p><pre>read-only</pre><p>The container can read the files but cannot modify them through that mount.</p><p>With the --mount syntax:</p><pre>docker run \<br>  --mount type=bind,source=$(pwd),target=/app,readonly \<br>  myapp</pre><p>Read-only mounts can be useful when containers need access to configuration or reference data without requiring write permissions.</p><h3>15. tmpfs Mounts</h3><p>A tmpfs mount stores data in the host&#39;s memory rather than persistent disk storage.</p><p>For example:</p><pre>docker run \<br>  --tmpfs /app/temp \<br>  myapp</pre><p>Data written to:</p><pre>/app/temp</pre><p>exists only while the container is running.</p><p>When the container stops, the data is lost.</p><p>This can be useful for temporary data that does not need to persist.</p><p>For example:</p><pre>Temporary processing<br>Temporary cache<br>Short-lived files</pre><p>Because the data is held in memory, available memory limits must also be considered.</p><h3>16. Choosing the Right Storage Type</h3><p>A simple decision process can help.</p><h3>Need persistent application data?</h3><p>Use a <strong>named volume</strong>.</p><pre>Database<br>Uploaded files<br>Persistent application data</pre><h3>Need your local source code inside the container?</h3><p>Use a <strong>bind mount</strong>.</p><pre>Local development<br>Live code editing<br>Configuration files</pre><h3>Need temporary in-memory data?</h3><p>Consider <strong>tmpfs</strong>.</p><pre>Temporary files<br>Short-lived sensitive data<br>Memory-backed scratch space</pre><h3>Need temporary container-only data?</h3><p>The normal writable layer may be sufficient.</p><h3>17. Why Networking Matters</h3><p>We’ve solved one major problem:</p><blockquote><strong><em>How do containers keep data after they are removed?</em></strong></blockquote><p>Now we need to solve another:</p><blockquote><strong><em>How do containers communicate with each other?</em></strong></blockquote><p>Modern applications are rarely single-container applications.</p><p>Consider a typical web application:</p><pre>                 ┌──────────────┐<br>                 │   Browser    │<br>                 └──────┬───────┘<br>                        │<br>                        ▼<br>                 ┌──────────────┐<br>                 │ Flask API    │<br>                 └──────┬───────┘<br>                        │<br>              ┌─────────┴─────────┐<br>              ▼                   ▼<br>       ┌─────────────┐     ┌─────────────┐<br>       │ PostgreSQL  │     │    Redis    │<br>       └─────────────┘     └─────────────┘</pre><p>Each component could run in its own container.</p><p>Those containers need to communicate reliably.</p><p>Docker Networking provides the infrastructure for this communication.</p><h3>18. Docker Networking Basics</h3><p>Docker creates virtual networks that allow containers to communicate.</p><p>List the available networks:</p><pre>docker network ls</pre><p>A typical installation might show:</p><pre>NETWORK ID     NAME      DRIVER<br><br>abc123         bridge    bridge<br>def456         host      host<br>ghi789         none      null</pre><p>Docker provides several network drivers.</p><p>The most important ones for beginners are:</p><ul><li>bridge</li><li>host</li><li>none</li></ul><p>Other drivers, such as overlay and macvlan, are useful in more specialized environments</p><h3>19. The Default Bridge Network</h3><p>When Docker is installed, it creates a default network called:</p><pre>bridge</pre><p>If you run a container without specifying a network:</p><pre>docker run nginx</pre><p>Docker normally connects it to the default bridge network.</p><p>You can inspect it:</p><pre>docker network inspect bridge</pre><p>This displays information about the network and the containers connected to it.</p><h3>20. Why Create a Custom Network?</h3><p>Docker’s default bridge network works for simple scenarios, but user-defined bridge networks provide better container-to-container communication.</p><p>Create a custom network:</p><pre>docker network create app-network</pre><p>Verify it:</p><pre>docker network ls</pre><p>You should now see:</p><pre>app-network</pre><h3>21. Connecting Containers to a Custom Network</h3><p>Suppose we have two containers:</p><pre>flask-api<br>postgres-db</pre><p>Start PostgreSQL on the custom network:</p><pre>docker run -d \<br>  --name postgres-db \<br>  --network app-network \<br>  postgres</pre><p>Then start the Flask application on the same network:</p><pre>docker run -d \<br>  --name flask-api \<br>  --network app-network \<br>  flask-app</pre><p>Now both containers belong to:</p><pre>app-network</pre><p>They can communicate over that Docker network.</p><pre>Docker<br>│<br>├── Container: flask-api<br>│   └── Flask application<br>│<br>└── Container: postgres-db<br>    └── PostgreSQL database</pre><pre>┌─────────────────┐       app-network       ┌─────────────────┐<br>│   flask-api     │ ──────────────────────&gt; │   postgres-db   │<br>│                 │                         │                 │<br>│ Flask API       │                         │ PostgreSQL      │<br>└─────────────────┘                         └─────────────────┘</pre><h3>22. Container-to-Container Communication</h3><p>Here’s an important Docker networking concept.</p><p>Suppose the PostgreSQL container is named:</p><pre>postgres-db</pre><p>From the Flask container, you generally do <strong>not</strong> connect to:</p><pre>localhost</pre><p>Instead, you use the PostgreSQL container’s network name:</p><pre>postgres-db</pre><p>For example:</p><pre>DB_HOST = &quot;postgres-db&quot;</pre><p>This is because:</p><pre>localhost</pre><p>inside the Flask container refers to the Flask container itself.</p><p>It does not refer to the PostgreSQL container.</p><h3>23. Understanding localhost in Docker</h3><p>This is one of the most common Docker networking mistakes.</p><p>Imagine:</p><pre>Host Computer<br>      │<br>      ├──────────────┐<br>      ▼              ▼<br>Flask Container   PostgreSQL Container</pre><p>Inside the Flask container:</p><pre>localhost</pre><p>means:</p><pre>Flask Container</pre><p>Inside the PostgreSQL container:</p><pre>localhost</pre><p>means:</p><pre>PostgreSQL Container</pre><p>Each container has its own network namespace.</p><p>Therefore, if Flask needs to communicate with PostgreSQL, using:</p><pre>DB_HOST = &quot;localhost&quot;</pre><p>is usually incorrect.</p><p>Instead:</p><pre>DB_HOST = &quot;postgres-db&quot;</pre><p>allows Docker’s user-defined network to route the connection to the PostgreSQL container.</p><h3>24. Docker’s Built-In Name Resolution</h3><p>On a user-defined Docker network, Docker provides internal DNS-based service discovery.</p><p>For example:</p><pre>app-network</pre><pre>┌─────────────────────┐<br>│                     │<br>│ flask-api           │<br>│                     │<br>└──────────┬──────────┘<br>           │<br>           │ postgres-db<br>           ▼<br>┌─────────────────────┐<br>│                     │<br>│ postgres-db         │<br>│                     │<br>└─────────────────────┘</pre><p>The Flask application can connect using:</p><pre>postgres-db</pre><p>rather than needing to know the PostgreSQL container’s IP address.</p><p>This is extremely useful because container IP addresses can change when containers are recreated.</p><p>Names provide a stable way for services to find each other.</p><h3>25. Port Mapping vs Container Networking</h3><p>Another common source of confusion is the difference between:</p><pre>-p 5000:5000</pre><p>and:</p><pre>--network app-network</pre><p>They solve different problems.</p><h3>Port publishing</h3><pre>-p 5000:5000</pre><p>makes a container port accessible from the host.</p><pre>Host<br>Port 5000<br>   │<br>   ▼<br>Container<br>Port 5000</pre><h3>26. EXPOSE vs Port Publishing</h3><p>Remember the EXPOSE instruction from Part 2?</p><p>For example:</p><pre>EXPOSE 5000</pre><p>This documents that the application listens on port 5000.</p><p>It does <strong>not</strong> publish the port.</p><p>To make it accessible from the host, you still need:</p><pre>docker run -p 5000:5000 myapp</pre><p>Therefore:</p><pre>EXPOSE<br>   ↓<br>Documentation / image metadata</pre><pre>-p<br>   ↓<br>Actual host-to-container port publishing</pre><p>This distinction is important.</p><h3>27. A Practical Flask + PostgreSQL Architecture</h3><p>Let’s combine everything we’ve learned so far.</p><p>Our application will contain two containers:</p><pre>Host Machine<br>                      │<br>                      │ Port 5000<br>                      ▼<br>              ┌───────────────┐<br>              │   Flask API   │<br>              │   Container   │<br>              └───────┬───────┘<br>                      │<br>                app-network<br>                      │<br>                      ▼<br>              ┌───────────────┐<br>              │  PostgreSQL   │<br>              │   Container   │<br>              └───────┬───────┘<br>                      │<br>                      ▼<br>                Docker Volume</pre><p>The responsibilities are separated:</p><p><strong>Flask container</strong></p><ul><li>Runs the application.</li><li>Handles HTTP requests.</li><li>Connects to PostgreSQL.</li></ul><p><strong>PostgreSQL container</strong></p><ul><li>Runs the database.</li><li>Stores persistent data in a volume.</li></ul><p><strong>Docker network</strong></p><ul><li>Allows Flask and PostgreSQL to communicate.</li></ul><p><strong>Docker volume</strong></p><ul><li>Keeps PostgreSQL data independent of the database container lifecycle.</li></ul><p>This architecture is a major step toward real-world containerized applications.</p><h3>28. Why Separate the Database Into Its Own Container?</h3><p>A common beginner approach is to install everything into one container:</p><pre>One Container</pre><pre>├── Flask<br>├── Python<br>├── PostgreSQL<br>├── Redis<br>└── Other Services</pre><p>This creates a tightly coupled environment.</p><p>A more container-friendly approach is:</p><pre>Flask Container<br>       │<br>       │ network<br>       ▼<br>PostgreSQL Container</pre><p>Each container has a focused responsibility.</p><p>This follows an important containerization principle:</p><blockquote><strong><em>One container should generally have one primary responsibility.</em></strong></blockquote><p>This doesn’t mean a container can never run multiple processes, but separating independently managed services usually makes systems easier to operate, scale, and maintain.</p><h3>29. Conclusion</h3><p>Docker containers are designed to be temporary and replaceable, but application data often needs to survive beyond the lifecycle of an individual container. Docker <strong>volumes</strong> provide a reliable way to persist data independently of containers, while <strong>bind mounts</strong> allow applications to work directly with files and directories on the host system.</p><p>Docker <strong>networking</strong> solves another important problem: allowing containers to communicate with each other reliably. With user-defined networks and Docker’s built-in DNS-based service discovery, containers can communicate using container names instead of relying on hard-coded IP addresses.</p><p>Together, volumes and networks form two essential building blocks of multi-container applications:</p><p><strong>Volumes</strong> provide persistent storage.</p><p><strong>Networks</strong> provide communication between containers.</p><p>In this part, we explored these concepts individually and saw how they fit into a typical application architecture such as:</p><pre>                   Host Machine<br>                         │<br>                         │ HTTP<br>                         ▼<br>                  ┌─────────────┐<br>                  │  Flask API  │<br>                  │  Container  │<br>                  └──────┬──────┘<br>                         │<br>                    app-network<br>                         │<br>                         ▼<br>                  ┌─────────────┐<br>                  │ PostgreSQL  │<br>                  │  Container  │<br>                  └──────┬──────┘<br>                         │<br>                    app-data<br>                         │<br>                         ▼<br>                  ┌─────────────┐<br>                  │   Volume    │<br>                  └─────────────┘</pre><p>Understanding these building blocks makes it much easier to work with real-world containerized applications.</p><p>However, managing multiple containers manually can quickly become repetitive. Creating networks, starting containers, configuring environment variables, mapping ports, and attaching volumes requires multiple commands.</p><p>This is where <strong>Docker Compose</strong> becomes useful.</p><p>In <strong>Part 4 — Docker Compose</strong>, we will bring these concepts together and define a complete multi-container application using a single compose.yaml file. We will build a Flask + PostgreSQL application, configure its services, networks, volumes, environment variables, and dependencies, and then manage the entire application with simple Docker Compose commands.</p><p><strong>Next: Part 4 — Docker Compose: Build and Manage Multi-Container Applications</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6575d043969f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Mastering Dockerfiles: Build Efficient Docker Images Step by Step]]></title>
            <link>https://medium.com/@sabitha.manoj0891/mastering-dockerfiles-build-efficient-docker-images-step-by-step-b2b03fd54426?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2b03fd54426</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 08:40:40 GMT</pubDate>
            <atom:updated>2026-08-07T08:40:40.274Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how Dockerfiles work, understand image layers and build caching, explore every Dockerfile instruction, and create optimized, production-ready Docker images using practical Python examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*TLZAeoR-kK1B_rE18mZhsg.jpeg" /></figure><h3>Docker Mastery Series</h3><p>Welcome back to the <strong>Docker Mastery Series</strong>, where we learn Docker from the fundamentals to production-ready deployments through practical examples.</p><h3>📚 Docker Mastery Series</h3><p><strong>✅ Part 1 — Introduction to Docker</strong></p><p><strong>✅ Part 2 — Mastering Dockerfiles (Current)</strong></p><p>➡️ Part 3 — Docker Volumes &amp; Networking</p><p>➡️ Part 4 — Docker Compose</p><p>➡️ Part 5 — Docker for Machine Learning</p><p>➡️ Part 6 — Advanced Docker</p><p>In <strong>Part 1</strong>, we explored the foundations of Docker and learned how containers have transformed modern software development. We covered Docker architecture, images, containers, the container lifecycle, and the essential Docker commands needed to run and manage applications.</p><p>However, downloading images from Docker Hub is only part of the story.</p><p>Sooner or later, every developer needs to package their own application into a Docker image.</p><p>Imagine you’ve built:</p><ul><li>a Python automation script,</li><li>a Flask REST API,</li><li>a Django web application,</li><li>a FastAPI service,</li><li>or a machine learning model for inference.</li></ul><p>How do you share your application with someone else without asking them to install Python, libraries, dependencies, and configure the environment manually?</p><p>This is exactly where <strong>Dockerfiles</strong> become indispensable.</p><p>A Dockerfile provides a repeatable set of instructions that tells Docker how to build an image for your application. Instead of documenting setup steps in a README and hoping every developer follows them correctly, you define the process once in a Dockerfile. Docker then produces identical images every time the build runs.</p><p>This consistency is one of the biggest reasons Docker has become a standard tool in modern software engineering.</p><p>In this article, you’ll learn how Dockerfiles work from the ground up. We’ll build images step by step, understand Docker’s layer caching mechanism, examine every major Dockerfile instruction, and finish with best practices that help create efficient, maintainable, and production-ready images.</p><h3>What You Will Learn</h3><p>By the end of this article, you’ll be able to:</p><ul><li>Understand what a Dockerfile is and why it is important.</li><li>Build Docker images from your own source code.</li><li>Understand Docker image layers and caching.</li><li>Use every major Dockerfile instruction correctly.</li><li>Optimize Dockerfiles for faster builds and smaller image sizes.</li><li>Avoid common mistakes when writing Dockerfiles.</li><li>Follow industry best practices for production deployments.</li></ul><h3>Why Do We Need Dockerfiles?</h3><p>Let’s consider a simple Python application.</p><p>Your project looks like this:</p><pre>hello-python/<br><br>├── app.py<br>└── requirements.txt</pre><p>The application is straightforward.</p><pre>The application is straightforward.</pre><p>To run it on another computer, the recipient must:</p><ul><li>Install Python</li><li>Install pip</li><li>Install all required packages</li><li>Ensure the correct Python version</li><li>Configure environment variables</li><li>Execute the application</li></ul><p>Even for a small project, these manual steps introduce opportunities for errors.</p><p>Now imagine a machine learning application requiring:</p><ul><li>Python 3.11</li><li>TensorFlow</li><li>NumPy</li><li>OpenCV</li><li>Pillow</li><li>Flask</li></ul><p>Installing and configuring these dependencies manually becomes much more challenging.</p><p>Docker solves this problem by packaging everything into a single image.</p><pre> Application<br>      │<br>      ▼<br>  Dockerfile<br>      │<br>      ▼<br>Docker Image<br>      │<br>      ▼<br>Docker Container</pre><p>Anyone with Docker installed can run the application without manually recreating the environment.</p><h3>What is a Dockerfile?</h3><p>A Dockerfile is a plain text file containing a sequence of instructions that Docker follows to build an image.</p><p>Think of it as a recipe.</p><p>When baking a cake, a recipe might say:</p><pre>Mix ingredients<br><br>   ↓<br><br>  Bake<br><br>   ↓<br><br>  Cool<br><br>   ↓<br><br>  Serve</pre><p>A Dockerfile works in a similar way.</p><pre>Choose a Base Image<br><br>        ↓<br><br>Copy Application Files<br><br>        ↓<br><br>Install Dependencies<br><br>        ↓<br><br>Configure Environment<br><br>        ↓<br><br>Specify Startup Command<br><br>        ↓<br><br> Create Docker Image</pre><p>Docker executes these instructions one by one until the image is complete.</p><h3>Your First Dockerfile</h3><p>Let’s containerize our Python application.</p><p>Project structure:</p><pre>hello-python/<br><br>├── app.py<br>└── Dockerfile</pre><p>Create a file named:</p><pre>Dockerfile</pre><p>Notice that it has <strong>no file extension</strong>.</p><p>Add the following content:</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY . .<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Although this file contains only four instructions, Docker performs a significant amount of work behind the scenes.</p><p>Let’s examine each instruction.</p><h3>Understanding the Dockerfile</h3><h4>FROM</h4><pre>FROM python:3.11-slim</pre><p>Every Docker image begins with a base image.</p><p>In this case, we’re using the official Python 3.11 slim image.</p><pre> Docker Hub<br><br>      ↓<br><br>python:3.11-slim<br><br>      ↓<br><br> Our Application</pre><p>Instead of installing Python ourselves, Docker downloads a ready-to-use Python environment.</p><h4>WORKDIR</h4><pre>WORKDIR /app</pre><p>This instruction sets the working directory inside the container.</p><p>It’s similar to running:</p><pre>cd /app</pre><p>All subsequent instructions execute relative to this directory.</p><p><strong>COPY</strong></p><pre>COPY . .</pre><p>This copies the contents of the current project directory from your computer into the container.</p><pre>Host Computer<br><br>app.py<br><br>Dockerfile<br><br>↓<br><br>Docker Image<br><br>/app<br><br>├── app.py<br>└── Dockerfile</pre><p>Without this step, the image would not contain your application.</p><h4>CMD</h4><pre>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>This defines the default command executed when the container starts.</p><p>When you run the container:</p><pre>docker run hello-python</pre><p>Docker automatically executes:</p><pre>python app.py</pre><h3>Building Your First Image</h3><p>Once the Dockerfile is ready, build the image.</p><pre>docker build -t hello-python .</pre><p>Let’s understand this command.</p><pre>docker build</pre><p>Starts the image build process.</p><pre>-t hello-python</pre><p>Assigns a name (tag) to the image.</p><p>If everything is configured correctly, Docker creates a new image.</p><p>Verify it:</p><pre>docker images</pre><p>Example output:</p><pre>REPOSITORY       TAG        IMAGE ID<br><br>hello-python    latest     ab12cd34ef56</pre><h3>Running the Image</h3><p>Now start a container.</p><pre>docker run hello-python</pre><p>Output:</p><pre>Hello, Docker!</pre><p>You have successfully created and executed your first custom Docker image.</p><p>Unlike downloading an existing image from Docker Hub, you’ve now built an image specifically for your own application.</p><h3>What Happens During docker build?</h3><p>Many beginners think Docker simply copies files into an image.</p><p>In reality, Docker performs a sequence of operations.</p><pre> Read Dockerfile<br>        │<br>        ▼<br>Download Base Image<br>        │<br>        ▼<br>  Create Layer<br>        │<br>        ▼<br>Execute Instruction<br>        │<br>        ▼<br>Create Next Layer<br>        │<br>        ▼<br>Repeat Until Complete<br>        │<br>        ▼<br>Final Docker Image</pre><p>Each instruction generates a new layer, which becomes one of Docker’s most powerful optimization features.</p><h3>The Docker Build Context</h3><p>One concept that often confuses beginners is the <strong>build context</strong>.</p><p>Consider this command:</p><pre>docker build -t hello-python .</pre><p>The final . is not part of the image name.</p><p>It tells Docker:</p><blockquote><em>“Use the current directory as the build context.”</em></blockquote><p>The build context includes all files and folders that Docker can access during the build.</p><p>For example:</p><pre>hello-python/<br><br>├── app.py<br>├── requirements.txt<br>├── README.md<br>├── images/<br>├── notebooks/<br>└── Dockerfile</pre><p>When you run:</p><pre>docker build .</pre><p>Docker sends the entire build context to the Docker daemon.</p><p>This is why large projects can take longer to build if they include unnecessary files such as datasets, notebooks, or virtual environments.</p><p>In the next section, we’ll learn how to avoid this using a .dockerignore file.</p><h3>Understanding Docker Image Layers</h3><p>One of Docker’s most powerful features is its <strong>layered architecture</strong>.</p><p>Unlike traditional virtual machines, Docker does not create a completely new copy of the operating system for every application. Instead, Docker builds images as a stack of <strong>read-only layers</strong>, where each Dockerfile instruction creates a new layer.</p><p>This design makes Docker images smaller, faster to build, and more efficient to distribute.</p><h3>What Is a Docker Layer?</h3><p>Think of building a house.</p><p>First, you lay the foundation.</p><p>Then you build the walls.</p><p>Next comes the roof.</p><p>Finally, you paint the house and add furniture.</p><p>Each step builds upon the previous one.</p><p>Docker images are created in much the same way.</p><pre>Base Operating System<br>        │<br>        ▼<br>  Python Installed<br>        │<br>        ▼<br>Application Files<br>        │<br>        ▼<br>Dependencies Installed<br>        │<br>        ▼<br>  Startup Command</pre><p>Each block represents a <strong>layer</strong>.</p><p>Docker stores these layers separately and combines them into a single image.</p><h3>Every Dockerfile Instruction Creates a Layer</h3><p>Consider the following Dockerfile.</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY requirements.txt .<br><br>RUN pip install -r requirements.txt<br><br>COPY . .<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Docker processes it step by step.</p><pre>Layer 1<br>FROM python:3.11-slim<br><br>↓<br><br>Layer 2<br>WORKDIR /app<br><br>↓<br><br>Layer 3<br>COPY requirements.txt<br><br>↓<br><br>Layer 4<br>RUN pip install<br><br>↓<br><br>Layer 5<br>COPY application files<br><br>↓<br><br>Layer 6<br>CMD</pre><p>Each instruction becomes a separate layer in the image.</p><p>This layered approach is the foundation of Docker’s caching mechanism.</p><h3>Why Layers Matter</h3><p>Suppose you build your application today.</p><p>Everything builds successfully.</p><p>Tomorrow, you modify only one line inside app.py.</p><p>Should Docker reinstall Python?</p><p>Should it download every package again?</p><p>Fortunately, the answer is <strong>no</strong>.</p><p>Docker detects which layers have changed and reuses the layers that remain unchanged.</p><p>For example:</p><pre>Layer 1<br>Python<br>✔ Cached<br><br>   ↓<br><br>Layer 2<br>WORKDIR<br>✔ Cached<br><br>   ↓<br><br>Layer 3<br>requirements.txt<br>✔ Cached<br><br>   ↓<br><br>Layer 4<br>pip install<br>✔ Cached<br><br>   ↓<br><br>Layer 5<br>Application Files<br>✖ Rebuilt<br> <br>  ↓<br><br>Layer 6<br>CMD<br>Reused</pre><p>Only the layer affected by the modified file — and any layers after it — are rebuilt.</p><p>Everything before it is retrieved from Docker’s cache.</p><h3>Docker Build Cache</h3><p>The <strong>build cache</strong> is one of Docker’s biggest performance advantages.</p><p>The first build may take several minutes because Docker needs to:</p><ul><li>Download the base image.</li><li>Install dependencies.</li><li>Copy files.</li><li>Create image layers.</li></ul><p>Subsequent builds are often much faster because Docker reuses cached layers whenever possible.</p><p>For example, the first build might look like this:</p><pre>docker build -t hello-python .</pre><pre>Downloading Python image...<br>Installing packages...<br>Copying files...<br>Building image...<br><br>Completed in 90 seconds</pre><p>Now suppose you only change the text inside app.py.</p><p>Running the build again might produce:</p><pre>Using cache<br>Using cache<br>Using cache<br>Copying updated files...<br><br>Completed in 6 seconds</pre><p>Instead of rebuilding everything, Docker only rebuilds the affected layers.</p><h3>How Docker Decides to Use the Cache</h3><p>Docker compares each instruction in the Dockerfile with the corresponding cached layer.</p><p>If both the instruction and the files it depends on are unchanged, Docker reuses the cached layer.</p><p>For example:</p><pre>FROM python:3.11-slim</pre><p>If the base image hasn’t changed:</p><pre>✔ Cache Hit</pre><pre>COPY requirements.txt .</pre><p>If requirements.txt hasn&#39;t changed:</p><pre>✔ Cache Hit</pre><pre>RUN pip install -r requirements.txt</pre><p>Since the requirements file is identical:</p><pre>✔ Cache Hit</pre><pre>COPY . .</pre><p>If app.py changed:</p><pre>✖ Cache Miss</pre><p>Docker rebuilds this layer and every instruction that follows it.</p><h3>Why Instruction Order Matters</h3><p>The order of Dockerfile instructions has a significant impact on build performance.</p><p>Consider the following Dockerfile.</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY . .<br><br>RUN pip install -r requirements.txt<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>At first glance, it looks reasonable.</p><p>However, it has an important drawback.</p><p>Every time <strong>any</strong> file changes, Docker must:</p><ul><li>Copy the application.</li><li>Reinstall all Python packages.</li></ul><p>Even changing a comment inside app.py causes the dependency installation layer to be rebuilt.</p><p>For projects with large dependencies like TensorFlow or PyTorch, this can add several minutes to every build.</p><h3>A Better Dockerfile</h3><p>A more efficient approach is:</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY . .<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Let’s see why this is better.</p><pre>requirements.txt<br>     ↓<br><br>pip install<br>     ↓<br><br>Application Files</pre><p>Now imagine you edit only app.py.</p><p>Docker recognizes that:</p><ul><li>requirements.txt is unchanged.</li><li>The dependency installation layer is still valid.</li></ul><p>Only the application files need to be copied again.</p><p>This reduces build times dramatically.</p><h3>Real-World Example</h3><p>Suppose your application depends on TensorFlow.</p><p>Installing TensorFlow may take several minutes.</p><p>With a poorly ordered Dockerfile:</p><pre>Modify app.py<br><br>      ↓<br><br>Reinstall TensorFlow<br><br>      ↓<br><br>Build Complete</pre><p>Time:</p><pre>4–6 minutes</pre><p>With an optimized Dockerfile:</p><pre>Modify app.py<br><br>     ↓<br><br>Copy Updated File<br><br>     ↓<br><br>Build Complete</pre><p>Time:</p><pre>5–10 seconds</pre><p>This is why experienced Docker users pay close attention to the order of instructions.</p><h3>Viewing Image Layers</h3><p>Docker provides commands to inspect how images are constructed.</p><p>To view image history:</p><pre>docker history hello-python</pre><p>Example output:</p><pre>IMAGE          CREATED         <br><br>CMD            0B<br><br>COPY           12KB<br><br>RUN            180MB<br><br>COPY           2KB<br><br>WORKDIR        0B<br><br>FROM           45MB</pre><p>This history shows the sequence of layers that make up the image.</p><p>It also highlights which instructions contribute most to the final image size.</p><h3>Inspecting Image Details</h3><p>You can also inspect an image to view metadata.</p><pre>docker inspect hello-python</pre><p>The output includes information such as:</p><ul><li>Image ID</li><li>Creation date</li><li>Environment variables</li><li>Working directory</li><li>Entrypoint</li><li>Labels</li><li>Exposed ports</li></ul><p>This command is especially useful when debugging or verifying image configurations.</p><h3>Best Practices for Layer Optimization</h3><p>To make builds faster and images more efficient:</p><ul><li>Place instructions that rarely change (such as the base image and dependency installation) near the top of the Dockerfile.</li><li>Copy dependency files (for example, requirements.txt) before copying the rest of the application.</li><li>Group related commands together where appropriate to reduce unnecessary layers.</li><li>Avoid repeatedly installing the same dependencies.</li><li>Use a .dockerignore file to exclude unnecessary files from the build context.</li></ul><p>Following these practices leads to faster builds, smaller images, and a smoother development workflow.</p><h3>Dockerfile Instructions Explained</h3><p>A Dockerfile is made up of a series of instructions. Each instruction performs a specific task while Docker builds the image.</p><p>Although Docker supports many instructions, a relatively small set is used in most applications.</p><p>In this section, we’ll explore the most important instructions used in real-world projects.</p><pre>Dockerfile<br><br>│<br><br>├── FROM<br><br>├── WORKDIR<br><br>├── COPY<br><br>├── ADD<br><br>├── RUN<br><br>├── CMD<br><br>├── ENTRYPOINT<br><br>├── EXPOSE<br><br>├── ENV<br><br>├── ARG<br><br>├── LABEL<br><br>├── USER<br><br>├── VOLUME<br><br>└── HEALTHCHECK</pre><p>Let’s examine them one by one.</p><h3>1. FROM</h3><p>The <strong>FROM</strong> instruction specifies the base image used to build your Docker image.</p><p>Every Dockerfile must begin with a FROM instruction (except for advanced multi-stage builds, where additional FROM statements define new stages).</p><p>Think of it as the foundation of a house. Every other instruction builds on top of it.</p><h4>Syntax</h4><pre>FROM &lt;image&gt;:&lt;tag&gt;</pre><p>Example:</p><pre>FROM python:3.11-slim</pre><p>Docker downloads the official Python image (if it isn’t already available locally) and uses it as the starting point.</p><h4>Another Example</h4><pre>FROM ubuntu:24.04</pre><p>This creates an image based on Ubuntu Linux.</p><h3>Why Use Base Images?</h3><p>Imagine installing everything yourself:</p><ul><li>Linux</li><li>Python</li><li>pip</li><li>SSL certificates</li><li>System libraries</li><li>Timezone data</li></ul><p>That would be tedious and error-prone.</p><p>Instead, Docker lets us build upon existing images.</p><pre>Docker Hub<br><br>    ↓<br><br>Python Image<br><br>    ↓<br><br>Our Application<br><br>    ↓<br><br>Final Image</pre><p>This saves time and ensures consistency.</p><h3>Choosing the Right Base Image</h3><p>Some commonly used base images include:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*HIK5oRVF2pXqx09rtHVa2g.jpeg" /></figure><h3>Best Practices</h3><p>✔ Choose official images whenever possible.</p><p>✔ Prefer lightweight variants such as -slim or Alpine-based images <strong>when they meet your application&#39;s compatibility needs</strong>.</p><p>✔ Pin a specific version (for example, python:3.11-slim) instead of relying on latest to improve build reproducibility.</p><h3>2. WORKDIR</h3><p>The WORKDIR instruction sets the working directory inside the image.</p><p>All subsequent instructions execute relative to this directory.</p><h4>Syntax</h4><pre>WORKDIR /app</pre><p>Suppose your Dockerfile contains:</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY . .<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Docker behaves as though it had executed:</p><pre>cd /app</pre><p>Every following instruction uses /app as the current directory.</p><h3>Why Is WORKDIR Useful?</h3><p>Without WORKDIR, you would need to specify the full path repeatedly.</p><p>For example:</p><pre>COPY . /app<br><br>RUN pip install -r /app/requirements.txt<br><br>CMD [&quot;python&quot;,&quot;/app/app.py&quot;]</pre><p>This is harder to read and maintain.</p><p>Using WORKDIR makes the Dockerfile cleaner.</p><pre>WORKDIR /app<br><br>COPY . .<br><br>RUN pip install -r requirements.txt<br><br>CMD [&quot;python&quot;,&quot;app.py&quot;]</pre><h3>Automatic Directory Creation</h3><p>One useful feature is that Docker creates the directory automatically if it doesn’t exist.</p><pre>WORKDIR /project</pre><p>Even if /project is missing, Docker creates it during the build.</p><h3>Multiple WORKDIR Instructions</h3><p>Docker also supports nested working directories.</p><pre>WORKDIR /app<br><br>WORKDIR src<br><br>WORKDIR api</pre><p>The final working directory becomes:</p><pre>/app/src/api</pre><h3>Best Practices</h3><p>✔ Set WORKDIR near the beginning of the Dockerfile.</p><p>✔ Use absolute paths rather than relying on relative paths.</p><p>✔ Choose meaningful directory names such as /app or /usr/src/app.</p><h3>3. COPY</h3><p>The COPY instruction copies files from the build context on your computer into the Docker image.</p><p>It is one of the most frequently used Dockerfile instructions.</p><h4>Syntax</h4><pre>COPY &lt;source&gt; &lt;destination&gt;</pre><p>Example:</p><pre>COPY app.py .</pre><p>This copies app.py into the current working directory inside the image.</p><h4>Copy an Entire Project</h4><p>Most applications copy all project files.</p><pre>COPY . .</pre><p>Meaning:</p><pre>Host Computer<br><br>     ↓<br><br>Current Folder<br><br>     ↓<br><br>Docker Image<br><br>    /app</pre><p>Everything in the build context is copied into /app.</p><h4>Copy Specific Files</h4><p>Instead of copying everything, you can copy only selected files.</p><pre>COPY requirements.txt .</pre><pre>COPY config.json .</pre><pre>COPY app.py .</pre><p>This approach is particularly useful for optimizing Docker’s build cache.</p><h4>Copy Directories</h4><p>Suppose your project contains:</p><pre>project/<br><br>├── app.py<br><br>├── models/<br><br>│   ├── model.keras<br><br>│   └── labels.json</pre><p>You can copy only the models directory.</p><pre>COPY models models</pre><p>Result:</p><pre>/app<br><br>├── models<br><br>│   ├── model.keras<br><br>│   └── labels.json</pre><h4>Why COPY Is So Important</h4><p>Remember the build cache from the previous section?</p><p>A common optimization is:</p><pre>COPY requirements.txt .<br><br>RUN pip install -r requirements.txt<br><br>COPY . .</pre><p>Docker installs dependencies only when requirements.txt changes.</p><p>If you modify only app.py, Docker reuses the cached dependency layer, making rebuilds much faster.</p><h4>Best Practices</h4><p>✔ Copy only the files you need.</p><p>✔ Copy dependency files before application code.</p><p>✔ Use .dockerignore to exclude unnecessary files from the build context.</p><p>✔ Avoid copying sensitive files such as private keys, credentials, or large datasets into the image.</p><h3>4. ADD</h3><p>At first glance, ADD looks very similar to COPY.</p><pre>ADD . .</pre><p>Many beginners assume they are interchangeable.</p><p>However, ADD has additional capabilities beyond simply copying files.</p><p>It can:</p><ul><li>Copy local files and directories.</li><li>Automatically extract supported local archive files (such as .tar archives).</li><li>Retrieve files from remote URLs (although this is generally discouraged for reproducible builds).</li></ul><p>For example:</p><pre>ADD archive.tar.gz /app/</pre><p>Docker automatically extracts the archive into /app.</p><p>With COPY, the archive would be copied without being extracted.</p><h4>Should You Use ADD?</h4><p>In most cases, <strong>no</strong>.</p><p>Unless you specifically need one of its extra features, COPY is preferred because it is more explicit and easier to understand.</p><p>As a general rule:</p><ul><li>Use <strong>COPY</strong> for everyday file copying.</li><li>Use <strong>ADD</strong> only when you need automatic archive extraction or another feature unique to ADD.</li></ul><p>We’ll compare COPY and ADD in more detail later in this article.</p><h3>5. RUN</h3><p>The RUN instruction executes a command <strong>while the Docker image is being built</strong>.</p><p>Think of RUN as preparing the environment for your application. It is used to install software, update packages, create directories, or perform any setup required before the image is complete.</p><p>Every RUN instruction creates a new image layer.</p><h4>Syntax</h4><pre>RUN &lt;command&gt;</pre><p>Example:</p><pre>RUN apt-get update</pre><p>Docker executes this command <strong>during the build</strong>, not when the container starts.</p><h4>Installing Python Packages</h4><p>One of the most common uses of RUN is installing application dependencies.</p><pre>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt</pre><p>When Docker reaches this instruction, it installs every package listed in requirements.txt and saves them inside the image.</p><p>Once the image is built, these packages are already available whenever a container starts.</p><h4>Installing System Packages</h4><p>Sometimes applications require operating system packages.</p><p>For example:</p><pre>RUN apt-get update &amp;&amp; \<br>    apt-get install -y curl </pre><p>This installs the curl utility into the image.</p><p>Other commonly installed packages include:</p><ul><li>git</li><li>wget</li><li>vim</li><li>ffmpeg</li><li>build-essential</li><li>libgl1 (for OpenCV)</li></ul><h4>Multiple RUN Instructions</h4><p>A Dockerfile like this works correctly:</p><pre>RUN apt-get update<br><br>RUN apt-get install -y curl<br><br>RUN apt-get install -y git</pre><p>However, it creates <strong>three separate layers</strong>.</p><p>A better approach is:</p><pre>RUN apt-get update &amp;&amp; \<br>    apt-get install -y \<br>    curl \<br>    git</pre><p>This reduces the number of layers and keeps the image more efficient.</p><h4>RUN Happens Only Once</h4><p>A common misconception is that RUN executes every time the container starts.</p><p>It does <strong>not</strong>.</p><p>Consider this Dockerfile:</p><pre>RUN echo &quot;Installing packages...&quot;</pre><p>During docker build, Docker prints:</p><pre>Installing packages...</pre><p>After the image is built, running the container will <strong>not</strong> execute this command again.</p><h4>Best Practices</h4><p>✔ Combine related commands into a single RUN instruction where appropriate.</p><p>✔ Remove temporary files created during installation to reduce image size.</p><p>✔ Use --no-cache-dir with pip to avoid storing unnecessary package caches.</p><h3>6. CMD</h3><p>The CMD instruction specifies the <strong>default command</strong> that Docker runs when a container starts.</p><p>Unlike RUN, which executes during the image build, CMD executes <strong>when the container is launched</strong>.</p><h4>Syntax</h4><pre>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>This is the recommended <strong>exec form</strong>, where each argument is provided as a separate element in a JSON array.</p><h4>Example</h4><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY . .<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Build the image:</p><pre>docker build -t hello-python .</pre><p>Run it:</p><pre>docker run hello-python</pre><p>Docker automatically executes:</p><pre>python app.py</pre><h4>Only One CMD</h4><p>A Dockerfile may contain multiple CMD instructions, but only the <strong>last one</strong> takes effect.</p><p>For example:</p><pre>CMD [&quot;python&quot;, &quot;app.py&quot;]<br><br>CMD [&quot;python&quot;, &quot;server.py&quot;]</pre><p>The container starts with:</p><pre>python server.py</pre><p>The first CMD is ignored.</p><h3>Overriding CMD</h3><p>One advantage of CMD is that it can be overridden when running a container.</p><p>Dockerfile:</p><pre>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Run normally:</p><pre>docker run myapp</pre><p>Result:</p><pre>python app.py</pre><p>Override the command:</p><pre>docker run myapp python test.py</pre><p>Now Docker executes:</p><pre>python test.py</pre><p>This flexibility makes CMD ideal for providing sensible defaults while still allowing users to run alternative commands.</p><h4>Typical Uses</h4><p>CMD is commonly used to start:</p><ul><li>Python scripts</li><li>Flask applications</li><li>FastAPI applications</li><li>Node.js servers</li><li>Nginx</li><li>Database servers</li></ul><h3>7. ENTRYPOINT</h3><p>ENTRYPOINT also specifies what runs when a container starts, but it serves a different purpose.</p><p>While CMD provides a <strong>default command</strong>, ENTRYPOINT defines the <strong>main executable</strong> of the container.</p><p>Think of ENTRYPOINT as saying:</p><blockquote><em>“This container is designed to run this application.”</em></blockquote><h4>Syntax</h4><pre>ENTRYPOINT [&quot;python&quot;]</pre><p>Unlike CMD, the entrypoint is <strong>not replaced</strong> by default when additional arguments are supplied.</p><h4>Example</h4><pre>ENTRYPOINT [&quot;python&quot;]<br><br>CMD [&quot;app.py&quot;]</pre><p>When the container starts:</p><pre>docker run myapp</pre><p>Docker combines the two instructions and runs:</p><pre>python app.py</pre><p>Now suppose you execute:</p><pre>docker run myapp test.py</pre><p>Docker runs:</p><pre>python test.py</pre><p>Here, ENTRYPOINT remains fixed, while CMD provides the default argument.</p><h4>A Practical Example</h4><p>Imagine you’re creating a Docker image that behaves like a command-line tool.</p><p>Dockerfile:</p><pre>ENTRYPOINT [&quot;python&quot;, &quot;calculator.py&quot;]</pre><p>Run:</p><pre>docker run calculator add 5 7</pre><p>Docker executes:</p><pre>python calculator.py add 5 7</pre><p>The user only supplies the arguments, while the application itself is fixed.</p><p>This is a common pattern for CLI utilities distributed as Docker images.</p><h3>CMD vs ENTRYPOINT</h3><p>This is one of the most frequently asked Docker interview questions.</p><p>Although both instructions execute commands when a container starts, they are designed for different purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*2MCaVS-j0X4v6FcF1c2wKg.jpeg" /></figure><h3>Which One Should You Use?</h3><p>For most Python web applications, such as Flask or FastAPI APIs, CMD is usually sufficient.</p><p>Example:</p><pre>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Use ENTRYPOINT when your container is intended to behave like a dedicated executable or command-line utility.</p><p>In many production images, you’ll see both instructions used together:</p><pre>ENTRYPOINT [&quot;python&quot;]<br><br>CMD [&quot;app.py&quot;]</pre><p>This provides a fixed executable while still allowing the default script or arguments to be customized.</p><h4>Best Practices</h4><p>✔ Use the <strong>exec form</strong> ([&quot;command&quot;, &quot;arg1&quot;]) instead of the shell form (command arg1) whenever possible. It handles signals more cleanly and is generally recommended for containerized applications.</p><p>✔ Use RUN for build-time setup.</p><p>✔ Use CMD for the default runtime command.</p><p>✔ Use ENTRYPOINT when the container should always execute a specific program.</p><p>✔ Avoid multiple CMD instructions in the same Dockerfile, as only the last one is used.</p><h3>8. EXPOSE</h3><p>Most applications need to communicate with the outside world.</p><p>A web application, for example, listens for incoming requests on a specific network port.</p><p>The EXPOSE instruction documents which port the application inside the container intends to listen on.</p><blockquote><strong><em>Important:</em></strong><em> </em><em>EXPOSE does </em><strong><em>not</em></strong><em> publish the port to your host machine. It serves as metadata and helps document the intended network interface of the container.</em></blockquote><h4>Syntax</h4><pre>EXPOSE &lt;port&gt;</pre><p>Example:</p><pre>EXPOSE 5000</pre><p>For a Flask application, port <strong>5000</strong> is commonly used.</p><p>For a FastAPI application:</p><pre>EXPOSE 8000</pre><p>For an Nginx web server:</p><pre>EXPOSE 80</pre><h4>How It Works</h4><p>Suppose your Flask application starts like this:</p><pre>app.run(host=&quot;0.0.0.0&quot;, port=5000)</pre><p>Your Dockerfile includes:</p><pre>EXPOSE 5000</pre><p>This tells anyone reading the Dockerfile (and many Docker tools) that the application is designed to listen on port <strong>5000</strong>.</p><h4>Publishing the Port</h4><p>Even after using EXPOSE, the application is <strong>not</strong> automatically accessible from your computer.</p><p>You still need to publish the port when running the container.</p><pre>docker run -p 5000:5000 flask-api</pre><p>Meaning:</p><pre>Host Computer<br>Port 5000<br>      │<br>      ▼<br>Container<br>Port 5000</pre><p>Now the application can be accessed from:</p><pre>http://localhost:5000</pre><h3>Best Practices</h3><p>✔ Use EXPOSE to document the application&#39;s listening port.</p><p>✔ Use docker run -p (or Docker Compose port mappings) to make the application accessible from outside the container.</p><p>✔ Keep the exposed port consistent with the application configuration.</p><h3>9. ENV</h3><p>Applications often need configuration values that differ between environments.</p><p>For example:</p><ul><li>Database host</li><li>API keys</li><li>Debug mode</li><li>Application version</li><li>Model location</li></ul><p>Hardcoding these values directly into your code makes applications difficult to configure and maintain.</p><p>The ENV instruction allows you to define environment variables inside the Docker image.</p><h4>Syntax</h4><pre>ENV KEY=value</pre><p>Example:</p><pre>ENV APP_ENV=production</pre><p>Inside Python:</p><pre>import os<br><br>environment = os.getenv(&quot;APP_ENV&quot;)<br><br>print(environment</pre><p>Output:</p><pre>production</pre><h4>Multiple Environment Variables</h4><pre>ENV APP_ENV=production<br><br>ENV PORT=5000<br><br>ENV MODEL_PATH=/models/model.keras</pre><p>Or, more compactly:</p><pre>ENV APP_ENV=production \<br>    PORT=5000 \<br>    MODEL_PATH=/models/model.keras</pre><h4>Best Practices</h4><p>✔ Store configuration in environment variables rather than hardcoding it.</p><p>✔ Avoid storing secrets (such as passwords or API keys) directly in the Dockerfile.</p><p>✔ Use sensible default values where appropriate.</p><h3>10. ARG</h3><p>At first glance, ARG looks similar to ENV.</p><p>However, they serve different purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*rMi50_9FIW43dFK_8XH3pA.jpeg" /></figure><h4>Syntax</h4><pre>ARG VERSION=3.11</pre><p>Example:</p><pre>ARG PYTHON_VERSION=3.11<br><br>FROM python:${PYTHON_VERSION}-slim</pre><p>Now you can build different versions of the image.</p><pre>docker build \<br>--build-arg PYTHON_VERSION=3.12 \<br>-t myapp </pre><p>Docker substitutes the value during the build.</p><h4>Best Practices</h4><p>✔ Use ARG for values needed only during the image build.</p><p>✔ Use ENV for configuration required while the container is running.</p><p>✔ Avoid using ARG or ENV for sensitive information such as passwords or API keys.</p><h3>11. LABEL</h3><p>Docker images can include metadata that helps identify and manage them.</p><p>This is done using the LABEL instruction.</p><h4>Syntax</h4><pre>LABEL key=&quot;value&quot;</pre><p>Example:</p><pre>LABEL maintainer=&quot;Sabitha Manoj&quot;</pre><p>You can add multiple labels.</p><pre>LABEL maintainer=&quot;Sabitha Manoj&quot;<br><br>LABEL version=&quot;1.0&quot;<br><br>LABEL description=&quot;Flask REST API&quot;</pre><p>Or combine them into one instruction:</p><pre>LABEL maintainer=&quot;Sabitha Manoj&quot; \<br>      version=&quot;1.0&quot; \<br>      description=&quot;Flask REST API&quot;</pre><h4>Best Practices</h4><p>✔ Include version information.</p><p>✔ Add a meaningful description.</p><p>✔ Use labels consistently across projects.</p><h3>Our Dockerfile So Far</h3><p>Our Dockerfile has gradually evolved throughout this article.</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY . .<br><br>EXPOSE 5000<br><br>ENV PORT=5000<br><br>LABEL maintainer=&quot;Your Name&quot;<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><h3>12. USER</h3><p>By default, Docker containers run as the <strong>root</strong> user.</p><p>While this is convenient during development, running applications as root in production is generally discouraged because it grants the application unrestricted privileges inside the container.</p><p>As a security best practice, containers should run as a <strong>non-root user</strong> whenever possible.</p><h4>Syntax</h4><pre>USER &lt;username&gt;</pre><p>or</p><pre>USER &lt;uid&gt;</pre><h3>Why Should You Avoid Running as Root?</h3><p>Imagine your application has a security vulnerability.</p><p>If the application runs as the <strong>root</strong> user, an attacker who exploits that vulnerability could gain elevated privileges inside the container. Although containers provide isolation, following the <strong>principle of least privilege</strong> reduces the potential impact of security issues.</p><p>Running as a non-root user limits what the application can access and modify.</p><h4>Creating a Non-Root User</h4><p>You can create a dedicated user during the image build.</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>RUN useradd -m appuser<br><br>USER appuser</pre><p>Here:</p><ul><li>useradd -m appuser creates a new user named appuser.</li><li>USER appuser instructs Docker to run all subsequent instructions—and the application itself—as appuser.</li></ul><h4>Copying Files with the Correct Ownership</h4><p>If files are copied into the image before switching users, they may still be owned by root.</p><p>Docker allows you to assign ownership while copying files.</p><pre>COPY --chown=appuser:appuser . .</pre><p>This ensures the application files belong to appuser, avoiding permission-related issues.</p><h4>Checking the Current User</h4><p>To verify which user is running inside the container:</p><pre>docker exec -it &lt;container_name&gt; whoami</pre><p>Example output:</p><pre>appuser</pre><h4>Best Practices</h4><p>✔ Avoid running containers as root in production.</p><p>✔ Create a dedicated application user with only the required permissions.</p><p>✔ Use COPY --chown where appropriate to simplify file ownership management.</p><h3>13. VOLUME</h3><p>Containers are <strong>ephemeral</strong>, which means data stored inside a container can be lost when the container is removed.</p><p>Consider a simple example.</p><p>Your application writes uploaded files to:</p><pre>/app/uploads</pre><p>If the container is deleted:</p><pre>docker rm my-container</pre><p>everything stored in /app/uploads is removed with it.</p><p>For applications that need to preserve data — such as databases, uploaded files, or logs — we use <strong>Docker Volumes</strong>.</p><p>The VOLUME instruction marks a directory as persistent storage.</p><h4>Syntax</h4><pre>VOLUME [&quot;/path&quot;]</pre><p>Example:</p><pre>VOLUME [&quot;/app/uploads&quot;]</pre><p>Docker now treats /app/uploads as a location intended for persistent data.</p><h4>How It Works</h4><pre>Container<br><br>  /app/uploads<br>        │<br>        ▼<br>   Docker Volume<br>        │<br>        ▼<br>   Host Storage</pre><p>Even if the container is deleted, the data stored in the volume remains available.</p><h4>Running a Container with a Volume</h4><pre>docker run \<br>-v mydata:/app/uploads \<br>flask-api</pre><p>Here:</p><ul><li>mydata is the Docker-managed volume.</li><li>/app/uploads is the directory inside the container.</li></ul><p>Uploaded files will now persist even if the container is recreated.</p><h3>When Should You Use Volumes?</h3><p>Volumes are commonly used for:</p><ul><li>Database files</li><li>Machine learning models</li><li>User uploads</li><li>Application logs</li><li>Configuration files</li><li>Cache directories</li></ul><p>We’ll explore Docker volumes in much greater detail in <strong>Part 3</strong> of this series.</p><h4>Best Practices</h4><p>✔ Store important data in volumes rather than inside the container’s writable layer.</p><p>✔ Use named volumes for persistent application data.</p><p>✔ Keep application code separate from persistent data.</p><h3>14. HEALTHCHECK</h3><p>Just because a container is running doesn’t necessarily mean the application inside it is working correctly.</p><p>For example:</p><ul><li>A Flask application may have crashed.</li><li>A web server may be stuck waiting indefinitely.</li><li>A database connection may have failed.</li></ul><p>From Docker’s perspective, the container is still running.</p><p>The HEALTHCHECK instruction allows Docker to periodically verify that the application is functioning as expected.</p><h4>Syntax</h4><pre>HEALTHCHECK [OPTIONS] CMD &lt;command&gt;</pre><p>Example:</p><pre>HEALTHCHECK CMD curl -f http://localhost:5000/ || exit 1</pre><p>Docker periodically sends a request to the application.</p><p>If the command succeeds, the container is marked as <strong>healthy</strong>.</p><p>If it repeatedly fails, Docker reports the container as <strong>unhealthy</strong>.</p><h4>A More Practical Example</h4><pre>HEALTHCHECK --interval=30s \<br>            --timeout=5s \<br>            --retries=3 \<br>CMD curl -f http://localhost:5000/health || exit 1</pre><p>Explanation:</p><ul><li>--interval=30s → Run the check every 30 seconds.</li><li>--timeout=5s → Wait up to 5 seconds for a response.</li><li>--retries=3 → Mark the container as unhealthy after three consecutive failures.</li></ul><p>Many production applications expose a dedicated /health endpoint that returns a simple success response when the application is ready to serve requests.</p><h4>Viewing Container Health</h4><p>To inspect the health status:</p><pre>docker ps</pre><p>Example output:</p><pre>STATUS<br><br>Up 5 minutes (healthy)</pre><p>Or inspect detailed health information:</p><pre>docker inspect &lt;container_name&gt;</pre><h3>Why Is HEALTHCHECK Important?</h3><p>Container orchestration platforms and monitoring systems often rely on health checks to detect application failures.</p><p>For example, if a container becomes unhealthy, an orchestrator may automatically restart it or stop routing traffic to it until it recovers.</p><p>Even in local development, health checks provide an additional layer of confidence that your application is actually responding — not just running.</p><h4>Best Practices</h4><p>✔ Create a lightweight health endpoint (such as /health) in web applications.</p><p>✔ Keep health check commands simple and fast.</p><p>✔ Avoid expensive database queries or long-running operations inside health checks.</p><p>✔ Choose reasonable intervals and timeouts to balance responsiveness with resource usage.</p><h3>Our Dockerfile So Far</h3><p>After adding the new instructions, our Dockerfile has evolved into:</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY . .<br><br>EXPOSE 5000<br><br>ENV PORT=5000<br><br>LABEL maintainer=&quot;Your Name&quot;<br><br>RUN useradd -m appuser<br><br>USER appuser<br><br>VOLUME [&quot;/app/uploads&quot;]<br><br>HEALTHCHECK --interval=30s --timeout=5s \<br>CMD curl -f http://localhost:5000/health || exit 1<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Notice how the Dockerfile has gradually transformed from a minimal example into a much more robust, production-oriented configuration.</p><h3>COPY vs ADD</h3><p>One of the most common questions beginners ask is:</p><blockquote><strong><em>Should I use </em></strong><strong><em>COPY or </em></strong><strong><em>ADD?</em></strong></blockquote><p>Since both instructions can copy files into a Docker image, they often appear interchangeable. However, they are designed for different purposes.</p><p>As a general rule:</p><blockquote><strong><em>Use </em></strong><strong><em>COPY unless you specifically need a feature provided by </em></strong><strong><em>ADD.</em></strong></blockquote><p>Let’s compare them side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*Pt8nROLnHId4D4KEd0vrag.jpeg" /></figure><p>Example: COPY</p><pre>COPY requirements.txt .</pre><p>Docker simply copies the file into the image.</p><p>Example: ADD</p><pre>ADD application.tar.gz /app/</pre><p>If application.tar.gz is a supported local archive, Docker automatically extracts it into /app.</p><p>With COPY, the archive would be copied as a single compressed file.</p><h4>Which Should You Use?</h4><p>For most Python, Flask, FastAPI, and machine learning projects, <strong>COPY is the right choice</strong>.</p><p>Using COPY makes your Dockerfile easier to understand because its behavior is explicit—it copies files and nothing more.</p><p>Reserve ADD for the relatively uncommon situations where you intentionally need one of its additional features.</p><h4>Best Practice</h4><p>✔ Prefer <strong>COPY</strong> for everyday file copying.</p><p>✔ Use <strong>ADD</strong> only when its additional functionality is genuinely required.</p><h3>CMD vs ENTRYPOINT</h3><p>Another area of confusion is the difference between CMD and ENTRYPOINT.</p><p>Although both affect what happens when a container starts, they have distinct roles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*eKhDka7oNTbMthxPDr7aVw.jpeg" /></figure><h3>Visual Comparison</h3><h4>Using CMD</h4><pre>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Running:</p><pre>docker run myapp</pre><p>Executes:</p><pre>python app.py</pre><p>Override it:</p><pre>docker run myapp python test.py</pre><p>Docker now executes:</p><pre>python test.py</pre><p>The original command is completely replaced.</p><h4>Using ENTRYPOINT</h4><pre>ENTRYPOINT [&quot;python&quot;]<br><br>CMD [&quot;app.py&quot;]</pre><p>Running:</p><pre>docker run myapp</pre><p>Executes:</p><pre>python app.py</pre><p>Now run:</p><pre>docker run myapp test.py</pre><p>Docker executes:</p><pre>python test.py</pre><p>Here, ENTRYPOINT (python) stays the same, while CMD (app.py) acts as the default argument.</p><h3>When Should You Use Each?</h3><p>Use <strong>CMD</strong> when:</p><ul><li>Running Flask applications</li><li>Running FastAPI applications</li><li>Running Python scripts</li><li>Starting web servers</li></ul><p>Use <strong>ENTRYPOINT</strong> when:</p><ul><li>Building command-line utilities</li><li>Creating specialized Docker images with a fixed executable</li><li>You always want the same executable to run</li></ul><p>For many applications, using both together provides the best balance between flexibility and predictable behavior.</p><h3>Building a Production-Ready Dockerfile</h3><p>Throughout this article, we’ve introduced Dockerfile instructions one at a time.</p><p>Let’s now combine them into a clean, well-structured Dockerfile for a simple Flask application.</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY . .<br><br>ENV PORT=5000<br><br>EXPOSE 5000<br><br>RUN useradd -m appuser<br><br>USER appuser<br><br>HEALTHCHECK --interval=30s --timeout=5s --retries=3 \<br>CMD python -c &quot;import urllib.request; urllib.request.urlopen(&#39;http://localhost:5000/health&#39;)&quot;<br><br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Notice the order of the instructions.</p><pre>Choose Base Image<br>        │<br>        ▼<br>Set Working Directory<br>        │<br>        ▼<br>Copy Dependency File<br>        │<br>        ▼<br>Install Dependencies<br>        │<br>        ▼<br>Copy Application Code<br>        │<br>        ▼<br>Configure Environment<br>        │<br>        ▼<br>Expose Port<br>        │<br>        ▼<br>Create Non-root User<br>        │<br>        ▼<br>Health Check<br>        │<br>        ▼<br>Start Application</pre><p>This order isn’t accidental.</p><p>It allows Docker to maximize layer caching while keeping the image organized and easy to maintain.</p><h3>Why This Dockerfile Is Efficient</h3><p>Suppose you change only app.py.</p><p>Will Docker reinstall every dependency?</p><p>No.</p><p>Since requirements.txt hasn&#39;t changed, Docker reuses the cached dependency layer.</p><p>Only the application code is copied again.</p><pre>FROM<br>✔ Cached<br><br>↓<br><br>WORKDIR<br>✔ Cached<br><br>↓<br><br>COPY requirements.txt<br>✔ Cached<br><br>↓<br><br>RUN pip install<br>✔ Cached<br><br>↓<br><br>COPY Application<br>Rebuilt<br><br>↓<br><br>CMD<br>Reused</pre><p>For projects with large libraries such as TensorFlow, PyTorch, or OpenCV, this optimization can reduce rebuild times from several minutes to just a few seconds.</p><h3>Using a .dockerignore File</h3><p>Earlier, we learned that Docker sends the entire <strong>build context</strong> to the Docker daemon.</p><p>If your project contains unnecessary files, they are transferred during every build unless explicitly excluded.</p><p>A .dockerignore file works much like .gitignore. It tells Docker which files and directories should <strong>not</strong> be included in the build context.</p><p>Create a file named:</p><pre>.dockerignore</pre><p>A typical Python project might include:</p><pre>__pycache__/<br>*.pyc<br>*.pyo<br>.git/<br>.gitignore<br>venv/<br>.env<br>.ipynb_checkpoints/<br>*.log<br>tests/<br>README.md</pre><p>By excluding these files, you can:</p><ul><li>Reduce build time.</li><li>Create smaller Docker images.</li><li>Prevent accidental inclusion of development artifacts or sensitive files.</li></ul><h3>Common Dockerfile Mistakes</h3><p>Even experienced developers occasionally write inefficient Dockerfiles. Fortunately, many common issues are easy to avoid once you understand Docker’s best practices.</p><p>Let’s look at some of the most frequent mistakes and how to fix them.</p><h4>1. Using the latest Tag</h4><p>Many beginners write:</p><pre>FROM python:latest</pre><p>Although this works, it can lead to inconsistent builds because the latest tag may point to different image versions over time.</p><p>Instead, pin a specific version:</p><pre>FROM python:3.11-slim</pre><p>This ensures that every build uses the same base image, making your application more predictable and reproducible.</p><h4>2. Copying Everything Too Early</h4><p>Consider the following Dockerfile:</p><pre>COPY . .<br><br>RUN pip install -r requirements.txt</pre><p>Every time any file changes, Docker invalidates the cache and reinstalls all dependencies.</p><p>A better approach is:</p><pre>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY . .</pre><p>This allows Docker to reuse the dependency layer whenever requirements.txt remains unchanged.</p><h4>3. Ignoring .dockerignore</h4><p>Without a .dockerignore file, Docker includes every file in the build context.</p><p>That might unintentionally include:</p><ul><li>Virtual environments</li><li>Git history</li><li>Jupyter notebooks</li><li>Large datasets</li><li>Log files</li><li>Temporary files</li></ul><p>A properly configured .dockerignore keeps your builds faster and your images smaller.</p><h4>4. Running Containers as Root</h4><p>Containers run as the root user by default.</p><p>Although this may be acceptable for quick experiments, production applications should use a dedicated non-root user whenever possible.</p><pre>RUN useradd -m appuser<br><br>USER appuser</pre><p>This follows the principle of least privilege and improves the security of your container.</p><h4>5. Installing Unnecessary Packages</h4><p>Avoid installing software that your application doesn’t need.</p><p>Every additional package:</p><ul><li>Increases the image size.</li><li>Expands the attack surface.</li><li>Slows down builds.</li></ul><p>Start with a minimal base image and install only the required dependencies.</p><h4>6. Hardcoding Sensitive Information</h4><p>Never include passwords, API keys, or database credentials directly in your Dockerfile.</p><p>❌ Avoid:</p><pre>ENV API_KEY=my-secret-key</pre><p>Instead, provide sensitive values at runtime using environment variables or a dedicated secrets management solution.</p><h4>7. Creating Too Many Layers</h4><p>This Dockerfile creates three separate layers:</p><pre>RUN apt-get update<br><br>RUN apt-get install -y curl<br><br>RUN apt-get install -y git</pre><p>A cleaner approach is:</p><pre>RUN apt-get update &amp;&amp; \<br>    apt-get install -y \<br>    curl \<br>    git</pre><p>Combining related commands helps keep images more efficient.</p><h3>Dockerfile Best Practices</h3><p>Before building your next Docker image, use the following checklist.</p><h3>✅ Dockerfile Checklist</h3><p>✔ Start with an official and trusted base image.</p><p>✔ Pin specific image versions instead of relying on latest.</p><p>✔ Choose lightweight base images when they meet your application’s compatibility needs.</p><p>✔ Copy dependency files before application code to maximize Docker’s build cache.</p><p>✔ Install only the packages your application requires.</p><p>✔ Use --no-cache-dir when installing Python packages with pip.</p><p>✔ Keep the build context small by using a .dockerignore file.</p><p>✔ Store configuration in environment variables instead of hardcoding values.</p><p>✔ Avoid including secrets in Docker images.</p><p>✔ Run applications as a non-root user whenever possible.</p><p>✔ Use health checks for long-running services.</p><p>✔ Keep Dockerfiles simple, readable, and well-documented.</p><p>Following these practices will help you create Docker images that are smaller, faster to build, easier to maintain, and more suitable for production deployments.</p><h3>Key Takeaways</h3><p>In this article, you learned how Dockerfiles provide a repeatable way to package applications into portable Docker images.</p><p>You explored the complete image build process, learned how Docker layers and build caching improve performance, and examined the most commonly used Dockerfile instructions, including FROM, WORKDIR, COPY, ADD, RUN, CMD, ENTRYPOINT, EXPOSE, ENV, ARG, LABEL, USER, VOLUME, and HEALTHCHECK.</p><p>You also learned how to optimize Dockerfiles, reduce build times through effective layer ordering, minimize the build context using a .dockerignore file, and apply best practices that make your images more secure, efficient, and maintainable.</p><p>With these concepts, you now have the knowledge to build your own Docker images with confidence and create Dockerfiles that are suitable for both development and production environments.</p><h3><strong>A Quick Reference Guide</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rIycChGSN75H58_1h2n3rA.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Conclusion</h3><p>Congratulations on completing <strong>Part 2</strong> of the <strong>Docker Mastery Series</strong>!</p><p>Dockerfiles are the foundation of every Docker image. While they may appear to be simple text files, they define the complete process of packaging an application — from selecting a base image and installing dependencies to configuring the runtime environment and specifying how the application starts.</p><p>Writing an effective Dockerfile is more than learning individual instructions. It’s about understanding how Docker builds images, how layer caching works, and how thoughtful instruction ordering can significantly improve build performance. By following the best practices covered in this article, you’ll create Docker images that are reproducible, efficient, secure, and easier to maintain.</p><p>As your applications grow in complexity, you’ll often need multiple containers working together. For example, a web application might require a database, a cache, and a backend service, all communicating over a network while persisting data beyond the lifecycle of individual containers.</p><p>That’s exactly what we’ll explore next.</p><h3>🚀 Continue the Docker Mastery Series</h3><p><strong>✅ Part 1 — Introduction to Docker</strong></p><p><strong>✅ Part 2 — Mastering Dockerfiles (Completed)</strong></p><p>➡️ <strong>Part 3 — Docker Volumes &amp; Networking</strong> <em>(Coming Next)</em></p><p>➡️ Part 4 — Docker Compose</p><p>➡️ Part 5 — Docker for Machine Learning</p><p>➡️ Part 6 — Advanced Docker</p><p>In <strong>Part 3</strong>, you’ll learn how to persist application data using Docker volumes, understand bind mounts, explore Docker networking concepts, publish ports, create custom networks, and enable communication between multiple containers. These concepts form the foundation for building real-world, multi-container applications and prepare you for Docker Compose in Part 4.</p><h3>References</h3><ul><li>Docker Documentation — <a href="https://docs.docker.com/">https://docs.docker.com/</a></li><li>Dockerfile Reference — <a href="https://docs.docker.com/reference/dockerfile/">https://docs.docker.com/reference/dockerfile/</a></li><li>Docker CLI Reference — <a href="https://docs.docker.com/reference/cli/docker/">https://docs.docker.com/reference/cli/docker/</a></li><li>Docker Build Documentation — <a href="https://docs.docker.com/build/">https://docs.docker.com/build/</a></li><li>Docker Best Practices — <a href="https://docs.docker.com/build/building/best-practices/">https://docs.docker.com/build/building/best-practices/</a></li><li>Python Documentation — <a href="https://docs.python.org/3/">https://docs.python.org/3/</a></li><li>Flask Documentation — <a href="https://flask.palletsprojects.com/">https://flask.palletsprojects.com/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2b03fd54426" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Mastery Series: From Beginner to Production]]></title>
            <link>https://medium.com/@sabitha.manoj0891/docker-mastery-series-from-beginner-to-production-9bf4e453fc2c?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bf4e453fc2c</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 11:13:24 GMT</pubDate>
            <atom:updated>2026-07-27T11:13:24.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*TLZAeoR-kK1B_rE18mZhsg.jpeg" /></figure><p>Welcome to the <strong>Docker Mastery Series</strong> — a comprehensive, hands-on guide designed to take you from the fundamentals of Docker to deploying production-ready applications.</p><p>Whether you’re a Python developer, data scientist, machine learning engineer, DevOps enthusiast, or software developer looking to modernize your development workflow, this series will help you build a solid understanding of Docker through clear explanations, practical examples, and real-world projects.</p><p>Throughout the series, you’ll learn how to package applications, manage containers, persist data, connect multiple services, deploy machine learning models, and apply industry best practices used in production environments.</p><p>Each article builds upon the previous one, so by the end of the series you’ll have a strong foundation in Docker and be ready to explore container orchestration platforms such as Kubernetes.</p><h3>📚 Docker Mastery Series</h3><p><strong>✅ Part 1 — Introduction to Docker </strong><br>Learn what Docker is, why containers have transformed modern software development, understand Docker architecture, install Docker, work with images and containers, and master the essential Docker commands every developer should know.</p><p><strong>➡️ Part 2 — Mastering Dockerfiles</strong><br>Learn how to build efficient Docker images using Dockerfiles, understand image layers and caching, explore every Dockerfile instruction, and apply best practices for creating clean, optimized images.</p><p><strong>➡️ Part 3 — Docker Volumes &amp; Networking</strong><br>Understand persistent storage with Docker volumes and bind mounts, learn how Docker networking works, connect multiple containers, publish ports, and build applications that communicate seamlessly.</p><p><strong>➡️ Part 4 — Docker Compose</strong><br>Discover how to manage multi-container applications using Docker Compose, define complete application stacks with a single configuration file, and build a real-world Flask application with PostgreSQL and Redis.</p><p><strong>➡️ Part 5 — Docker for Machine Learning</strong><br>Deploy machine learning models using Docker and Flask, containerize TensorFlow applications, optimize inference containers, manage model files, and learn practical deployment strategies for AI applications.</p><p><strong>➡️ Part 6 — Advanced Docker</strong><br>Explore production-ready Docker practices including multi-stage builds, image optimization, health checks, security best practices, resource limits, container registries, CI/CD integration, and an introduction to Kubernetes.</p><p>By the end of this series, you’ll understand not only <strong>how Docker works</strong>, but also <strong>how to use it effectively in real-world software engineering and machine learning projects</strong>.</p><p>Let’s begin with the fundamentals and discover why Docker has become one of the most important technologies in modern application development.</p><h3><strong>✅ Part 1 — Introduction to Docker</strong></h3><h3>Introduction</h3><p>Imagine spending several weeks developing a Python application on your laptop. It runs perfectly on your machine, passes every test, and you’re confident it’s ready for deployment. You send the project to a teammate, only to receive the dreaded message:</p><blockquote><strong><em>“It doesn’t work on my computer.”</em></strong></blockquote><p>This is one of the most common challenges in software development.</p><p>Perhaps your teammate has a different version of Python installed. Maybe a required library is missing, or an operating system dependency behaves differently. Even something as simple as an environment variable or file path can cause an application to fail.</p><p>Before containerization, developers spent countless hours solving these environment-related issues instead of building new features.</p><p>Docker revolutionized software development by solving this problem through <strong>containers</strong> — lightweight, portable environments that package an application together with everything it needs to run.</p><p>With Docker, your application behaves the same way on:</p><ul><li>Your laptop</li><li>Your teammate’s computer</li><li>A testing server</li><li>A cloud virtual machine</li><li>A Kubernetes cluster</li></ul><p>Instead of saying:</p><blockquote><em>“It works on my machine.”</em></blockquote><p>Developers can confidently say:</p><blockquote><strong><em>“It works inside the container.”</em></strong></blockquote><p>Today Docker has become one of the most important tools in modern software engineering. It is used by startups, enterprises, research labs, and cloud providers to package and deploy applications consistently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfES5Gezf8mQYXqjYy2cgQ.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Why Do We Need Docker?</h3><p>Before learning Docker, it’s important to understand the problem it was designed to solve.</p><p>Imagine a team of three developers working on the same Python project.</p><p>Alice develops on Windows.</p><p>Bob uses Ubuntu.</p><p>Charlie works on macOS.</p><p>Although all three developers write the same Python code, their environments are completely different.</p><p>For example:</p><pre>Python Version<br><br>Alice      Python 3.10<br>Bob        Python 3.11<br>Charlie    Python 3.12</pre><p>Installed libraries may also differ.</p><pre>Alice<br>TensorFlow 2.15<br><br>Bob<br>TensorFlow 2.19<br><br>Charlie<br>TensorFlow 2.18</pre><p>Operating systems introduce another layer of differences.</p><pre>Windows<br><br>Linux<br>  <br>macOS</pre><p>A program that works perfectly on one operating system may fail on another due to missing system libraries, incompatible commands, or filesystem differences.</p><p>As projects grow larger, these inconsistencies become increasingly difficult to manage.</p><p>For machine learning applications, the problem becomes even more severe because projects often depend on:</p><ul><li>CUDA versions</li><li>NVIDIA drivers</li><li>TensorFlow or PyTorch versions</li><li>OpenCV libraries</li><li>FFmpeg</li><li>Database drivers</li><li>System packages</li></ul><p>Installing the correct versions manually on every computer can take hours — or even days.</p><p>Now imagine deploying the same application to production.</p><p>Your production server may have:</p><ul><li>a different Linux distribution,</li><li>a newer Python version,</li><li>different security libraries,</li><li>missing dependencies.</li></ul><p>The application that worked during development suddenly crashes after deployment.</p><p>This is known as the <strong>environment inconsistency problem</strong>.</p><p>Docker solves this issue by packaging everything required to run an application into a single portable unit called a <strong>container</strong>.</p><p>Instead of sharing only your source code, Docker allows you to share:</p><ul><li>Your application</li><li>Python</li><li>Libraries</li><li>Operating system dependencies</li><li>Environment variables</li><li>Configuration files</li></ul><p>Everything travels together.</p><p>As a result, the application behaves identically regardless of where it is executed.</p><h3>Problems Before Containers</h3><p>Before Docker became popular, developers typically relied on one of three approaches.</p><h3>1. Installing Everything Directly on the Host</h3><p>The simplest approach was to install every dependency directly on the operating system.</p><p>For example:</p><pre><br>Python 3.11<br>TensorFlow<br>NumPy<br>OpenCV<br>Flask<br>PostgreSQL<br>Redis<br>CUDA</pre><p>While straightforward initially, this approach quickly became problematic. Different projects often required different versions of the same software, leading to dependency conflicts.</p><p>Suppose:</p><p>Project A requires:</p><pre>TensorFlow 2.10</pre><p>Project B requires:</p><pre>TensorFlow 2.19</pre><p>Installing one version could break the other.</p><p>The same issue occurred with Python versions, databases, CUDA toolkits, and many other dependencies.</p><h3>2. Using Virtual Environments</h3><p>Python introduced virtual environments to isolate packages.</p><pre>Project A<br>│<br>├── venv<br>│   ├── TensorFlow 2.10<br>│   └── NumPy<br><br>Project B<br>│<br>├── venv<br>│   ├── TensorFlow 2.19<br>│   └── NumPy</pre><p>This solved Python package conflicts but not system-level dependencies.</p><p>For example:</p><ul><li>CUDA</li><li>OpenCV system libraries</li><li>Java</li><li>FFmpeg</li><li>Database servers</li></ul><p>These still had to be installed manually.</p><h3>3. Virtual Machines</h3><p>The next solution was virtualization.</p><p>Each application ran inside its own virtual machine.</p><pre>Host Computer<br>│<br>├── VM 1<br>│     Ubuntu<br>│     Python<br>│     TensorFlow<br>│<br>├── VM 2<br>│     Ubuntu<br>│     Flask<br>│<br>└── VM 3<br>      Windows<br>      SQL Server</pre><p>Virtual machines solved compatibility issues because each VM included its own operating system.</p><p>However, they came with significant drawbacks:</p><ul><li>Large disk usage (often several gigabytes)</li><li>High memory consumption</li><li>Slow startup times</li><li>Resource-intensive operation</li><li>Limited portability</li></ul><p>Running several virtual machines simultaneously could easily exhaust the resources of a typical laptop.</p><p>Docker introduced a more efficient alternative: <strong>containers</strong>.</p><p>Unlike virtual machines, containers share the host operating system’s kernel while keeping applications isolated from one another. Because they don’t need a full guest operating system, containers are much smaller, start in seconds, and use far fewer resources.</p><p>This lightweight approach is one of the main reasons Docker has become the standard for modern application development and deployment.</p><h3>What is Docker?</h3><p>At first glance, Docker may seem like another virtualization technology, but it is fundamentally different.</p><p>Docker is an <strong>open-source containerization platform</strong> that packages an application together with everything it needs to run, including:</p><ul><li>Source code</li><li>Runtime (such as Python)</li><li>Libraries and dependencies</li><li>System tools</li><li>Configuration files</li><li>Environment variables</li></ul><p>This package is called a <strong>container</strong>.</p><p>Think of a container as a <strong>portable software package</strong>.</p><p>No matter where you move it — your laptop, another developer’s computer, an on-premises server, or a cloud platform — it behaves exactly the same.</p><p>Instead of installing software manually on every machine, you simply run the container.</p><h3>A Real-World Analogy</h3><p>Before shipping containers were invented, transporting goods around the world was inefficient.</p><p>Each item had to be loaded individually:</p><ul><li>Furniture</li><li>Electronics</li><li>Food</li><li>Clothing</li></ul><p>Every port had different loading procedures, making transportation slow and expensive.</p><p>Then came the <strong>standard shipping container</strong>.</p><p>Everything was packed into one standardized box.</p><p>Whether transported by:</p><ul><li>Truck</li><li>Ship</li><li>Train</li></ul><p>the container itself never changed.</p><p>The transportation system simply moved the container.</p><p>Docker works in exactly the same way.</p><p>Instead of shipping software piece by piece, Docker packages everything into a standardized container.</p><pre>Application<br>      │<br>Python<br>      │<br>Libraries<br>      │<br>Configuration<br>      │<br>Dependencies<br>      │<br>──────────────────<br> Docker Container<br>──────────────────</pre><p>The host computer simply runs the container.</p><p>This eliminates installation problems and configuration errors.</p><h3>What Makes a Container Special?</h3><p>A Docker container is <strong>isolated</strong> from the host system.</p><p>It has its own:</p><ul><li>filesystem</li><li>processes</li><li>networking</li><li>installed packages</li><li>environment variables</li></ul><p>Although multiple containers share the host operating system’s kernel, each behaves as though it has its own dedicated environment.</p><p>For example, your computer might simultaneously run:</p><pre>Host Computer<br>│<br>├── Python API Container<br>│<br>├── Database Container<br>│<br>├── Redis Container<br>│<br>├── Nginx Container<br>│<br>└── Machine Learning Container</pre><p>Each container operates independently.</p><p>Updating one container does not affect the others.</p><h3>Containers Are Lightweight</h3><p>Unlike virtual machines, containers do <strong>not</strong> include an entire operating system.</p><p>Instead, they share the host’s operating system kernel.</p><p>This dramatically reduces their size.</p><p>Typical comparison:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*B2Ewu3BfUT2QmN73EmER4A.png" /></figure><p>Because containers don’t need to boot an operating system, they start almost instantly.</p><p>This is one reason cloud providers can launch hundreds of containers within seconds.</p><h3>Virtual Machines vs Docker Containers</h3><p>One of the most common beginner questions is:</p><blockquote><strong><em>If virtual machines already isolate applications, why do we need Docker?</em></strong></blockquote><p>The answer lies in architecture.</p><h4>Virtual Machine Architecture</h4><p>A virtual machine includes an entire operating system.</p><pre>+--------------------------------+<br>|         Application            |<br>+--------------------------------+<br>|      Python / Libraries        |<br>+--------------------------------+<br>|      Guest Operating System    |<br>+--------------------------------+<br>|         Hypervisor             |<br>+--------------------------------+<br>|      Host Operating System     |<br>+--------------------------------+<br>|         Hardware               |<br>+--------------------------------+</pre><p>Every VM requires:</p><ul><li>Linux or Windows installation</li><li>Kernel</li><li>Device drivers</li><li>System services</li><li>Memory allocation</li></ul><p>Even a simple Python application carries the overhead of a complete operating system.</p><p>Running five virtual machines means running five operating systems.</p><h3>Docker Architecture</h3><p>Containers eliminate this unnecessary overhead.</p><pre>+--------------------------------+<br>|    Container 1 (Python App)    |<br>+--------------------------------+<br>|    Container 2 (Flask API)     |<br>+--------------------------------+<br>|    Container 3 (Redis Cache)   |<br>+--------------------------------+<br>|        Docker Engine           |<br>+--------------------------------+<br>|     Host Operating System      |<br>+--------------------------------+<br>|          Hardware              |<br>+--------------------------------+</pre><p>Instead of multiple operating systems, all containers share the host kernel.</p><p>Only the application and its dependencies are isolated.</p><p>This makes containers:</p><ul><li>much smaller</li><li>much faster</li><li>easier to deploy</li><li>easier to scale</li></ul><h4>Side-by-Side Comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*dQp68KJLUmT8Y_rMBXDkmw.png" /></figure><h3>When Should You Use Each?</h3><h3>Use Virtual Machines when:</h3><ul><li>Running different operating systems on one machine.</li><li>Strong hardware-level isolation is required.</li><li>You need to test kernel modifications.</li><li>Legacy applications depend on a specific operating system.</li></ul><p>Examples include:</p><ul><li>Running Windows applications on a Linux host.</li><li>Testing a new Linux distribution.</li><li>Hosting multiple operating systems on a single server.</li></ul><h3>Use Docker Containers when:</h3><ul><li>Developing software.</li><li>Deploying web applications.</li><li>Building microservices.</li><li>Creating machine learning APIs.</li><li>Running CI/CD pipelines.</li><li>Scaling cloud applications.</li></ul><p>Today, most cloud-native applications are deployed as Docker containers.</p><h3>Docker Architecture</h3><p>Understanding Docker becomes much easier once you know what happens behind the scenes.</p><p>When you type:</p><pre>docker run hello-world</pre><p>a surprising amount of work happens automatically.</p><p>Docker follows a client-server architecture.</p><pre>  Docker Hub<br>                      ▲<br>                      │ Pull Images<br>                      │<br>        +----------------------------+<br>        |       Docker Client        |<br>        | (docker commands / CLI)    |<br>        +-------------+--------------+<br>                      │<br>          API Request │<br>                      ▼<br>        +----------------------------+<br>        |      Docker Engine         |<br>        |  (Docker Daemon)           |<br>        +-------------+--------------+<br>                      │<br>          Creates Container<br>                      ▼<br>          +------------------+<br>          |    Container      |<br>          +------------------+</pre><p>Let’s understand each component.</p><h3>1. Docker Client</h3><p>The Docker Client is the command-line interface (CLI) that users interact with.</p><p>Examples include:</p><pre>docker run<br>docker build<br>docker ps<br>docker stop<br>docker images</pre><p>These commands don’t directly create containers. Instead, they send requests to the Docker Engine.</p><p>Think of the Docker Client as a remote control.</p><p>It tells Docker what to do but doesn’t perform the work itself.</p><h3>2. Docker Engine (Docker Daemon)</h3><p>The Docker Engine is the core of Docker.</p><p>It runs in the background as a service and is responsible for:</p><ul><li>downloading images</li><li>creating containers</li><li>starting containers</li><li>stopping containers</li><li>managing networks</li><li>managing volumes</li><li>allocating resources</li></ul><p>Every Docker command ultimately communicates with the Docker Engine.</p><h3>3. Docker Images</h3><p>An image is a <strong>read-only blueprint</strong> used to create containers.</p><p>Think of an image as a recipe.</p><p>For example:</p><pre>Ubuntu Image<br>│<br>├── Ubuntu OS files<br>├── Python<br>├── Flask<br>├── NumPy<br>└── Application Code</pre><p>The image never changes while running.</p><p>Whenever Docker starts a container, it creates a writable layer on top of the image.</p><h3>4. Containers</h3><p>A container is a <strong>running instance of an image</strong>.</p><p>One image can produce many containers.</p><pre>Python Image<br>      │<br>      ├────────► Container 1<br>      │<br>      ├────────► Container 2<br>      │<br>      ├────────► Container 3<br>      │<br>      └────────► Container 4</pre><p>Each container has:</p><ul><li>its own processes</li><li>filesystem</li><li>IP address</li><li>hostname</li><li>memory allocation</li></ul><p>But all are created from the same image.</p><h3>5. Docker Registry</h3><p>Docker images are stored in a registry.</p><p>The most popular registry is <strong>Docker Hub</strong>, which hosts millions of public images such as:</p><ul><li>Ubuntu</li><li>Python</li><li>MySQL</li><li>PostgreSQL</li><li>Redis</li><li>Nginx</li><li>TensorFlow</li><li>PyTorch</li></ul><p>When you execute:</p><pre>docker run python:3.11</pre><p>Docker checks your local machine first.</p><p>If the image isn’t available locally, it automatically downloads it from Docker Hub.</p><pre>Docker Hub<br>      │<br>      ▼<br>Downloads Image<br>      │<br>      ▼<br>Stored Locally<br>      │<br>      ▼<br>Creates Container</pre><p>This automatic image retrieval is one of Docker’s biggest conveniences.</p><h3>Docker Workflow</h3><p>The complete lifecycle can be summarized as:</p><pre>Write Application<br>        │<br>        ▼<br>Create Dockerfile<br>        │<br>        ▼<br>Build Docker Image<br>        │<br>        ▼<br>Store Image<br>(Local / Docker Hub)<br>        │<br>        ▼<br>Run Container<br>        │<br>        ▼<br>Application Executes</pre><p>This simple workflow is the foundation of almost every Docker-based development and deployment process.</p><h3>Installing Docker</h3><p>Before creating containers, we first need to install Docker.</p><p>Docker is available for all major operating systems, including:</p><ul><li>Windows</li><li>macOS</li><li>Linux</li></ul><p>The installation process is straightforward, but the exact steps differ slightly depending on your operating system.</p><h3>Installing Docker on Windows</h3><p>Windows users should install <strong>Docker Desktop</strong>, which includes:</p><ul><li>Docker Engine</li><li>Docker CLI</li><li>Docker Compose</li><li>Docker Desktop GUI</li></ul><h3>Step 1: Download Docker Desktop</h3><p>Visit the official Docker website:</p><p><a href="https://www.docker.com/products/docker-desktop/">Docker Desktop: The #1 Containerization Tool for Developers | Docker</a></p><p>Download the version compatible with your system.</p><h3>Step 2: Install Docker Desktop</h3><p>Run the installer.</p><p>During installation, Docker will ask whether to enable <strong>WSL2 (Windows Subsystem for Linux)</strong>.</p><p>Choose:</p><pre>✓ Use WSL2</pre><p>WSL2 provides better performance and compatibility than the older Hyper-V backend.</p><h3>Step 3: Restart Your Computer</h3><p>After installation, restart your system.</p><p>Launch Docker Desktop.</p><p>You should see the Docker whale icon in the system tray.</p><h3>Installing Docker on macOS</h3><p>Download Docker Desktop for macOS.</p><p>Docker supports both:</p><ul><li>Apple Silicon (M-series chips)</li><li>Intel processors</li></ul><p>Install the application like any other macOS application.</p><h3>Installing Docker on Ubuntu</h3><p>Docker can be installed using the package manager.</p><p>First update your packages:</p><pre>sudo apt update</pre><p>Install Docker:</p><pre>sudo apt install docker.io</pre><p>Start Docker:</p><pre>sudo systemctl start docker</pre><p>Enable Docker on startup:</p><pre>sudo systemctl enable docker</pre><p>Verify installation:</p><pre>docker --version</pre><p>Example:</p><pre>Docker version 28.2.2</pre><h3>Verifying the Installation</h3><p>Regardless of your operating system, open a terminal (or PowerShell on Windows) and type:</p><pre>docker --version</pre><p>Example output:</p><pre>Docker version 28.2.2, build xxxxx</pre><p>Next verify Docker is running.</p><pre>docker info</pre><p>If Docker is running correctly, you’ll see information about:</p><ul><li>Containers</li><li>Images</li><li>Storage Driver</li><li>CPU</li><li>Memory</li><li>Docker Root Directory</li></ul><h3>Your First Docker Container</h3><p>Let’s run the famous Docker example.</p><pre>docker run hello-world</pre><p>You should see something similar to:</p><pre>Unable to find image &#39;hello-world:latest&#39; locally<br><br>latest: Pulling from library/hello-world<br><br>Hello from Docker!<br><br>This message shows that your installation appears to be working correctly.</pre><p>You have just executed your first Docker container.</p><p>But what actually happened?</p><h3>What Happens Behind the Scenes?</h3><p>Running a single Docker command triggers several automated steps.</p><pre>docker run hello-world</pre><p>Internally, Docker performs the following operations:</p><pre>Step 1<br>Search local images<br>        │<br>        ▼<br>Image not found<br>        │<br>        ▼<br>Connect to Docker Hub<br>        │<br>        ▼<br>Download Image<br>        │<br>        ▼<br>Store Image Locally<br>        │<br>        ▼<br>Create Container<br>        │<br>        ▼<br>  Run Program<br>        │<br>        ▼<br>Display Output</pre><p>This entire process typically completes in just a few seconds.</p><h3>Understanding docker run</h3><p>The docker run command performs multiple operations in one step.</p><p>When you execute:</p><pre>docker run python:3.11</pre><p>Docker automatically:</p><ol><li>Searches for the image locally.</li><li>Downloads it if necessary.</li><li>Creates a new container.</li><li>Starts the container.</li><li>Executes the default command defined by the image.</li></ol><p>This makes docker run one of the most frequently used Docker commands.</p><h3>Downloading Images Without Running Them</h3><p>Sometimes you only want to download an image.</p><p>Use:</p><pre>docker pull python:3.11</pre><p>Output:</p><pre>3.11: Pulling from library/python<br>...<br>Status: Downloaded newer image</pre><p>Now the image exists on your computer.</p><p>Running:</p><pre>docker images</pre><p>might display:</p><pre>REPOSITORY   TAG      IMAGE ID      SIZE<br><br>python       3.11     xxxxxxxx      1.02GB<br><br>hello-world  latest   xxxxxxxx      13kB</pre><h3>Listing Docker Images</h3><p>To display all downloaded images:</p><pre>docker images</pre><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Hbz24oUsiEiogGbpOPeYWQ.png" /></figure><p>Think of images as templates.</p><p>Nothing is actually running yet.</p><h3>Starting an Interactive Container</h3><p>Let’s start an Ubuntu container.</p><pre>docker run -it ubuntu</pre><p>The prompt changes to something like:</p><pre>root@6af14c3b2:/#</pre><p>You are now inside a Linux environment running within a Docker container.</p><p>Try the following commands:</p><pre>ls</pre><pre>pwd</pre><pre>whoami</pre><pre>cat /etc/os-release</pre><p>You’ll notice you’re interacting with the container’s filesystem — not your host machine.</p><p>To exit the container:</p><pre>exit</pre><h3>Running a Python Container</h3><p>Now let’s launch Python inside a container.</p><pre>docker run -it python:3.11</pre><p>The Python interpreter starts:</p><pre>&gt;&gt;&gt;</pre><p>Let’s execute some Python code.</p><pre>print(&quot;Hello Docker&quot;)</pre><p>Output:</p><pre>Hello Docker</pre><p>Or calculate a factorial:</p><pre>import math<br><br>print(math.factorial(6))</pre><p>Output:</p><pre>720</pre><p>Exit Python:</p><pre>exit()</pre><p>Docker automatically stops the container.</p><h3>Running Commands Without Entering the Container</h3><p>Instead of opening a shell, Docker can execute a single command.</p><p>Example:</p><pre>docker run python:3.11 python --version</pre><p>Output:</p><pre>Python 3.11.13</pre><p>Or run inline Python code:</p><pre>docker run python:3.11 python -c &quot;print(5 + 10)&quot;</pre><p>Output:</p><pre>15</pre><p>This is particularly useful for automation and CI/CD pipelines.</p><h3>Container Lifecycle</h3><p>Every Docker container goes through a lifecycle.</p><pre>Docker Image<br>      │<br>      ▼<br>Container Created<br>      │<br>      ▼<br>Container Running<br>      │<br>      ▼<br>Container Stopped<br>      │<br>      ▼<br>Container Removed</pre><p>Understanding this lifecycle helps explain why containers sometimes disappear after they finish executing.</p><h3>Listing Running Containers</h3><p>To see active containers:</p><pre>docker ps</pre><p>Example:</p><pre>CONTAINER ID   IMAGE       STATUS<br><br>abc123         ubuntu      Up 2 minutes</pre><h3>Listing All Containers</h3><p>Stopped containers are hidden by default.</p><p>Display all containers:</p><pre>docker ps -a</pre><p>Example:</p><pre>CONTAINER ID   IMAGE        STATUS<br><br>abc123         ubuntu       Exited<br><br>xyz456         python       Exited</pre><h3>Stopping a Running Container</h3><p>Suppose your container ID is:</p><pre>7d54bc19</pre><p>Stop it using:</p><pre>docker stop 7d54bc19</pre><p>Docker sends a graceful termination signal, allowing the application to shut down cleanly.</p><h3>Removing Containers</h3><p>Stopped containers still occupy disk space.</p><p>Remove a container:</p><pre>docker rm 7d54bc19</pre><p>To remove all stopped containers:</p><pre>docker container prune</pre><p>Docker will ask for confirmation before deleting them.</p><h3>Removing Images</h3><p>To delete an image:</p><pre>docker rmi ubuntu</pre><p>If an image is still being used by a container, Docker prevents its removal until the dependent containers are deleted.</p><h3><strong>Essential Docker Commands</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*5vQD6LnYLKyomTLk1gP1iA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rvapIqXI0tRi30V1anmKsw.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Conclusion</h3><p>Congratulations on completing <strong>Part 1</strong> of the <strong>Docker Mastery Series</strong>!</p><p>In this article, we explored the foundations of Docker and learned why containers have become an essential part of modern software development. We covered the core concepts of Docker architecture, images, containers, the container lifecycle, and the most commonly used Docker commands. By the end of this guide, you should be comfortable pulling images, running containers, inspecting them, managing their lifecycle, and understanding how Docker simplifies application deployment across different environments.</p><p>These fundamentals form the foundation for everything that follows. As you continue through the series, you’ll build on this knowledge to create optimized Docker images, manage persistent storage, connect multiple containers, orchestrate applications with Docker Compose, deploy machine learning models, and apply production-ready best practices.</p><p>Docker is more than just a tool for running applications — it has become a standard technology for building portable, reproducible, and scalable software systems. Investing time in mastering Docker will benefit you whether you’re developing web applications, data science projects, APIs, or AI solutions.</p><h3>References</h3><ul><li>Docker Inc. <strong>Docker Documentation</strong> — <a href="https://docs.docker.com/">https://docs.docker.com/</a></li><li>Docker Inc. <strong>Get Started Guide</strong> — <a href="https://docs.docker.com/get-started/">https://docs.docker.com/get-started/</a></li><li>Docker Inc. <strong>Dockerfile Reference</strong> — <a href="https://docs.docker.com/reference/dockerfile/">https://docs.docker.com/reference/dockerfile/</a></li><li>Docker Inc. <strong>Docker CLI Reference</strong> — <a href="https://docs.docker.com/reference/cli/docker/">https://docs.docker.com/reference/cli/docker/</a></li><li>Docker Inc. <strong>Docker Compose Documentation</strong> — <a href="https://docs.docker.com/compose/">https://docs.docker.com/compose/</a></li><li>Python Software Foundation. <strong>Python Documentation</strong> — <a href="https://docs.python.org/3/">https://docs.python.org/3/</a></li><li>Flask Documentation — <a href="https://flask.palletsprojects.com/">https://flask.palletsprojects.com/</a></li><li>TensorFlow Documentation — <a href="https://www.tensorflow.org/">https://www.tensorflow.org/</a></li><li>OCI (Open Container Initiative) Specifications — <a href="https://opencontainers.org/">https://opencontainers.org/</a></li></ul><h3>🚀 Continue the Docker Mastery Series</h3><p><strong>✅ Part 1 — Introduction to Docker (Completed)</strong></p><p>➡️ <strong>Part 2 — Mastering Dockerfiles</strong> <em>(Coming Next)</em></p><p>➡️ <strong>Part 3 — Docker Volumes &amp; Networking</strong></p><p>➡️ <strong>Part 4 — Docker Compose</strong></p><p>➡️ <strong>Part 5 — Docker for Machine Learning</strong></p><p>➡️ <strong>Part 6 — Advanced Docker</strong></p><p>In <strong>Part 2</strong>, we’ll dive deep into <strong>Dockerfiles</strong> — the blueprint for creating Docker images. You’ll learn how each instruction works, how Docker’s layer caching improves build performance, the differences between CMD and ENTRYPOINT, and the best practices for writing efficient, production-ready Dockerfiles.</p><p>Stay tuned for the next article in the series!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bf4e453fc2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Workflow with Flask: From Trained Model to Web Application]]></title>
            <link>https://medium.com/codetodeploy/machine-learning-workflow-with-flask-from-trained-model-to-web-application-06aa563ba006?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/06aa563ba006</guid>
            <category><![CDATA[python-flask]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 11:45:05 GMT</pubDate>
            <atom:updated>2026-07-06T11:45:05.181Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*MvAq09rcwopypVTOo2ol-Q.jpeg" /></figure><p>Machine learning models are valuable only when they can be used by real users. After training a model, the next step is deployment — making predictions available through an application or API.</p><blockquote><em>💥 </em><strong><em>Master Any Skills in 3 Months</em></strong><br> 📚 Up to 50% OFF Premium Courses<br> ⏳ Limited-Time Offer<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><p>One of the simplest and most popular ways to deploy machine learning models in Python is using Flask, a lightweight web framework. Flask allows you to create REST APIs and web applications that interact with trained machine learning models.</p><p>In this article, we’ll explore:</p><ul><li>What Flask is</li><li>Why Flask is used for ML deployment</li><li>End-to-end ML workflow with Flask</li><li>Building a Dog Breed Classifier API</li><li>Complete Python code</li><li>Best practices for production deployment</li></ul><h3>Why Flask for Machine Learning?</h3><p>Flask is a lightweight Python web framework that allows developers to expose machine learning models through HTTP endpoints.</p><p>Benefits:</p><p>✅ Easy to learn</p><p>✅ Minimal setup</p><p>✅ Flexible architecture</p><p>✅ Integrates well with Scikit-Learn, TensorFlow, and PyTorch</p><p>✅ Suitable for prototypes and production systems</p><h3>Typical Machine Learning Workflow with Flask</h3><p>A deployed ML application usually follows this workflow:</p><pre>Data Collection<br>      │<br>      ▼<br>Data Preprocessing<br>      │<br>      ▼<br>Model Training<br>      │<br>      ▼<br>Model Evaluation<br>      │<br>      ▼<br>Save Trained Model<br>      │<br>      ▼<br>  Flask API<br>      │<br>      ▼<br>User Request<br>      │<br>      ▼<br>  Prediction<br>      │<br>      ▼<br>Response to User</pre><p>The machine learning model is trained once and saved to disk. Flask loads the model and serves predictions whenever requests arrive.</p><h3>Example Problem</h3><p>Let’s build a simple Dog Breed Classifier.</p><p>The model predicts whether an image contains:</p><ul><li>Husky</li><li>Golden Retriever</li><li>German Shepherd</li></ul><h3>Step 1: Train the Model</h3><p>For demonstration purposes, we’ll use TensorFlow and MobileNetV2.</p><pre>import tensorflow as tf<br><br># Define the input image size expected by the model<br>IMG_SIZE = (224, 224)<br><br># Load images from the dataset directory<br># The directory should contain one subfolder per class<br># Example:<br># dog_dataset/<br># ├── husky/<br># ├── golden_retriever/<br># └── german_shepherd/<br>train_ds = tf.keras.preprocessing.image_dataset_from_directory(<br>    &quot;dog_dataset&quot;,<br>    image_size=IMG_SIZE,   # Resize all images to 224×224<br>    batch_size=32          # Number of images processed in one training batch<br>)<br><br># Create a MobileNetV2 model<br># weights=None means training starts from scratch<br># classes=3 specifies three output classes<br>model = tf.keras.applications.MobileNetV2(<br>    weights=None,<br>    classes=3<br>)<br><br># Configure the training process<br>model.compile(<br>    optimizer=&quot;adam&quot;,                       # Adam optimizer for weight updates<br>    loss=&quot;sparse_categorical_crossentropy&quot;, # Loss function for integer class labels<br>    metrics=[&quot;accuracy&quot;]                    # Track classification accuracy during training<br>)<br><br># Train the model on the dataset<br>model.fit(<br>    train_ds,<br>    epochs=10   # Number of complete passes through the training dataset<br>)<br><br># Save the trained model to disk<br># The .h5 file can later be loaded for inference or further training<br>model.save(&quot;dog_classifier.h5&quot;)</pre><p>The model is trained and saved for deployment.</p><h3><strong>Step 2: Create Flask Project Structure</strong></h3><pre>project/<br>│<br>├── app.py<br>├── dog_classifier.h5<br>├── templates/<br>│   └── index.html<br>│<br>├── static/<br>│   └── uploads/<br>│<br>└── requirements.txt</pre><h3><strong>Step 3: Install Dependencies</strong></h3><pre># Install required Python packages<br># Flask      : Creates the web application and REST API<br># TensorFlow : Loads and runs the trained deep learning model<br># Pillow     : Handles image loading and preprocessing<br># NumPy      : Performs numerical operations on image arrays<br><br>pip install flask tensorflow pillow numpy</pre><p>requirements.txt</p><pre># Flask web framework for creating the API<br>flask<br><br># TensorFlow for loading and running the deep learning model<br>tensorflow<br><br># Pillow for image processing and manipulation<br>pillow<br><br># NumPy for numerical computations and array operations<br>numpy</pre><h3><strong>Step 4: Create Flask Application</strong></h3><p>app.py</p><pre># Import Flask components for creating the web application,<br># handling file uploads, and rendering HTML templates<br>from flask import Flask, request, render_template<br><br># Import TensorFlow for loading and running the trained model<br>import tensorflow as tf<br><br># Import NumPy for numerical operations and array manipulation<br>import numpy as np<br><br># Import Pillow for image loading and preprocessing<br>from PIL import Image<br><br># Create a Flask application instance<br>app = Flask(__name__)<br><br># Load the trained MobileNetV2 dog breed classification model<br>model = tf.keras.models.load_model(<br>    &quot;dog_classifier.h5&quot;<br>)<br><br># Define the class labels corresponding to the model outputs<br>class_names = [<br>    &quot;German Shepherd&quot;,<br>    &quot;Golden Retriever&quot;,<br>    &quot;Husky&quot;<br>]<br><br># Function to preprocess uploaded images before prediction<br>def preprocess_image(image):<br><br>    # Resize image to the input size expected by the model<br>    image = image.resize((224, 224))<br><br>    # Convert PIL image to NumPy array<br>    image = np.array(image)<br><br>    # Normalize pixel values from [0, 255] to [0, 1]<br>    image = image / 255.0<br><br>    # Add batch dimension<br>    # Shape changes from (224, 224, 3) to (1, 224, 224, 3)<br>    image = np.expand_dims(image, axis=0)<br><br>    return image<br><br># Define the home page route<br># Supports both GET (load page) and POST (submit image) requests<br># Defines URL endpoint : http://localhost:5000<br>@app.route(&quot;/&quot;, methods=[&quot;GET&quot;, &quot;POST&quot;])<br>def predict():<br><br>    # Variable to store the prediction result<br>    prediction = None<br><br>    # Process image only when the form is submitted<br>    if request.method == &quot;POST&quot;:<br><br>        # Retrieve uploaded image file from the form<br>        file = request.files[&quot;image&quot;]<br><br>        # Open the uploaded image using Pillow<br>        image = Image.open(file)<br><br>        # Preprocess image for model inference<br>        image = preprocess_image(image)<br><br>        # Generate prediction probabilities<br>        preds = model.predict(image)<br><br>        # Find the class with the highest probability<br>        predicted_class = np.argmax(preds)<br><br>        # Convert class index to human-readable breed name<br>        prediction = class_names[predicted_class]<br><br>    # Render the webpage and display the prediction result<br>    return render_template(<br>        &quot;index.html&quot;,<br>        prediction=prediction<br>    )<br><br># Start the Flask development server<br>if __name__ == &quot;__main__&quot;:<br>    app.run(debug=True)</pre><h3>Step 5: Create HTML Interface</h3><p>templates/index.html</p><pre>&lt;!-- HTML5 document declaration --&gt;<br>&lt;!DOCTYPE html&gt;<br><br>&lt;html&gt;<br>&lt;head&gt;<br>    &lt;!-- Title displayed in the browser tab --&gt;<br>    &lt;title&gt;Dog Breed Classifier&lt;/title&gt;<br>&lt;/head&gt;<br><br>&lt;body&gt;<br><br>&lt;!-- Main page heading --&gt;<br>&lt;h1&gt;Dog Breed Classifier&lt;/h1&gt;<br><br>&lt;!--<br>    Image upload form<br>    method=&quot;POST&quot; sends the uploaded file to the Flask server<br>    enctype=&quot;multipart/form-data&quot; is required for file uploads<br>--&gt;<br>&lt;form method=&quot;POST&quot;<br>      enctype=&quot;multipart/form-data&quot;&gt;<br><br>    &lt;!-- File selection field for uploading an image --&gt;<br>    &lt;input type=&quot;file&quot;<br>           name=&quot;image&quot;&gt;<br><br>    &lt;!-- Button to submit the image for prediction --&gt;<br>    &lt;button type=&quot;submit&quot;&gt;<br>        Predict<br>    &lt;/button&gt;<br><br>&lt;/form&gt;<br><br>&lt;!--<br>    Jinja2 template block<br>    Display prediction only if a prediction result exists<br>--&gt;<br>{% if prediction %}<br><br>&lt;!-- Display the predicted dog breed --&gt;<br>&lt;h2&gt;<br>Prediction:<br>{{ prediction }}<br>&lt;/h2&gt;<br><br>{% endif %}<br><br>&lt;/body&gt;<br>&lt;/html&gt;</pre><h3>Running the Application</h3><p>Start Flask:</p><pre>python app.py</pre><p>Output:</p><pre>* Running on http://127.0.0.1:5000</pre><p>Open the browser:</p><pre>http://127.0.0.1:5000</pre><p>Upload an image and receive predictions.</p><h3>Alternative: Building a REST API</h3><p>Instead of HTML, Flask can expose a prediction endpoint.</p><pre># Import Flask for creating the web application<br># Import request for accessing incoming HTTP requests and data<br># Import jsonify for returning JSON responses from API endpoints<br>from flask import Flask, request, jsonify</pre><p>Create an API route:</p><pre># Define an API endpoint for image prediction<br># The endpoint accepts only POST requests<br>@app.route(&quot;/predict&quot;, methods=[&quot;POST&quot;])<br>def predict_api():<br><br>    # Retrieve the uploaded image file from the request<br>    file = request.files[&quot;image&quot;]<br><br>    # Open the uploaded image using Pillow<br>    image = Image.open(file)<br><br>    # Preprocess the image to match the model&#39;s input requirements<br>    image = preprocess_image(image)<br><br>    # Generate prediction probabilities using the trained model<br>    preds = model.predict(image)<br><br>    # Get the index of the class with the highest probability<br>    predicted_class = np.argmax(preds)<br><br>    # Return the predicted dog breed as a JSON response<br>    return jsonify({<br>        &quot;prediction&quot;:<br>        class_names[predicted_class]<br>    })</pre><h3>Testing the API</h3><p>Using cURL:</p><p><strong>cURL</strong> (Client URL) is a command-line tool used to <strong>send HTTP requests and transfer data between a client and a server</strong>. It is widely used for testing APIs, downloading files, uploading files, and interacting with web services.</p><p>In machine learning deployments, cURL is often used to test REST APIs without needing a web browser or a separate application.</p><h3>Why Use cURL?</h3><ul><li>Test Flask, FastAPI, or Django APIs</li><li>Upload files to a server</li><li>Send JSON data</li><li>Download files from the internet</li><li>Automate API calls in scripts</li></ul><pre># cURL command to test the Flask prediction API<br># curl           : Command-line tool for sending HTTP requests<br># -X POST        : Sends a POST request to the server<br># -F             : Sends form data (multipart/form-data)<br># image=@dog.jpg : Uploads the file dog.jpg using the form field name &quot;image&quot;<br># http://127.0.0.1:5000/predict : URL of the prediction endpoint<br><br>curl -X POST \<br>-F image=@dog.jpg \<br>http://127.0.0.1:5000/predict</pre><p>Response:</p><pre>{<br>  &quot;prediction&quot;: &quot;Golden Retriever&quot;<br>}</pre><h3>Production Workflow</h3><p>A real-world deployment often looks like:</p><pre>Client Application<br>        │<br>        ▼<br>  Load Balancer<br>        │<br>        ▼<br>   Flask API<br>        │<br>        ▼<br>  Preprocessing<br>        │<br>        ▼<br>   ML Model<br>        │<br>        ▼<br>   Prediction<br>        │<br>        ▼<br>Database / Logging</pre><h3>Improving the Flask ML Workflow</h3><h3>1. Save Models Efficiently</h3><p>Instead of:</p><pre>model.save(&quot;model.h5&quot;)</pre><p>Use:</p><pre>model.save(&quot;saved_model&quot;)</pre><p>Benefits:</p><ul><li>Faster loading</li><li>Better TensorFlow compatibility</li><li>Easier production deployment</li></ul><h3><strong>2. Add Input Validation</strong></h3><pre># Check whether the user selected a file<br># An empty filename indicates that no file was chosen<br># Return an error message if no file was uploaded<br><br>if file.filename == &quot;&quot;:<br>    return &quot;No file uploaded&quot;</pre><p>Prevents invalid requests.</p><h3><strong>3. Add Error Handling</strong></h3><pre># Use a try-except block to handle prediction errors gracefully<br>try:<br>    # Run inference using the trained model<br>    prediction = model.predict(image)<br><br>except Exception as e:<br>    # Catch any exception that occurs during prediction<br>    # and return the error message for debugging<br>    return str(e)</pre><p>Makes debugging easier.</p><h3><strong>4. Log Predictions</strong></h3><pre># Import Python&#39;s built-in logging module<br># Logging is commonly used to record application events,<br># debugging information, warnings, and errors<br>import logging<br><br># Write an informational log message containing<br># the model prediction result<br>logging.info(<br>    f&quot;Prediction: {prediction}&quot;<br>)</pre><p>Useful for monitoring model behaviour.</p><h3><strong>5. Containerize with Docker</strong></h3><pre># Use the official Python 3.11 image as the base image<br>FROM python:3.11<br><br># Set the working directory inside the container<br># All subsequent commands will be executed from /app<br>WORKDIR /app<br><br># Copy all project files from the local machine<br># into the container&#39;s /app directory<br>COPY . .<br><br># Install all required Python dependencies<br># listed in requirements.txt<br>RUN pip install -r requirements.txt<br><br># Specify the command to run when the container starts<br># Launches the Flask application<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p>Build:</p><pre># Build a Docker image from the Dockerfile in the current directory<br># docker build : Creates a Docker image<br># -t dog-classifier : Assigns the name (tag) &quot;dog-classifier&quot; to the image<br># . : Uses the current directory as the build context<br><br>docker build -t dog-classifier .</pre><p>Run:</p><pre># Run a container from the &quot;dog-classifier&quot; Docker image<br># docker run      : Starts a new container<br># -p 5000:5000   : Maps port 5000 on the host machine<br>#                  to port 5000 inside the container<br># dog-classifier : Name of the Docker image to run<br><br>docker run -p 5000:5000 dog-classifier</pre><p>This allows you to access the Flask application from your browser at:</p><pre>http://localhost:5000</pre><p>or test the API with:</p><pre>curl -X POST -F image=@dog.jpg http://localhost:5000/predict</pre><h3><strong>Common Flask Deployment Options</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*j4dZKh2KpPrBbyWvxR7Jbg.png" /></figure><h3><strong>Advantages of Flask for ML Deployment</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*-05xMgwDHfAfALh34r41jA.png" /></figure><h3><strong>Limitations</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*ZLV44tDdtHwD-fOl42QkWA.png" /></figure><h3><strong>End-to-End Workflow Summary</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*PFDFX2AeoOQoqh00roTfjw.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>A Quick Reference Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IeGd7KWUDSN1Lg66CGnIiw.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Conclusion</h3><p>Flask provides one of the easiest paths from a trained machine learning model to a usable application. By combining TensorFlow or Scikit-Learn models with Flask, developers can quickly build web applications and REST APIs that deliver predictions in real time.</p><p>Whether you’re deploying a simple image classifier, a recommendation system, a fraud detection model, or a large deep learning model, Flask offers a lightweight and flexible foundation for serving machine learning predictions. As projects grow, the same Flask-based workflow can be extended with Docker, cloud platforms, load balancers, monitoring systems, and orchestration tools to support production-scale machine learning applications.</p><h3>References</h3><ul><li>Flask Documentation — <a href="https://flask.palletsprojects.com/">https://flask.palletsprojects.com/</a></li><li>TensorFlow Documentation — <a href="https://www.tensorflow.org/">https://www.tensorflow.org/</a></li><li>Keras API Documentation — <a href="https://keras.io/">https://keras.io/</a></li><li>Docker Documentation — <a href="https://docs.docker.com/">https://docs.docker.com/</a></li><li>Gunicorn Documentation — <a href="https://docs.gunicorn.org/">https://docs.gunicorn.org/</a></li><li>REST API Tutorial — <a href="https://restfulapi.net/">https://restfulapi.net/</a></li><li>Python Documentation — <a href="https://docs.python.org/3/">https://docs.python.org/3/</a></li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/bhuwan-kc-583505319/?isSelfProfile=true"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06aa563ba006" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/machine-learning-workflow-with-flask-from-trained-model-to-web-application-06aa563ba006">Machine Learning Workflow with Flask: From Trained Model to Web Application</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to EfficientNet with Python Examples]]></title>
            <link>https://pub.aimind.so/a-practical-guide-to-efficientnet-with-python-examples-e6f11856e4eb?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6f11856e4eb</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[efficientnet]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 20:47:51 GMT</pubDate>
            <atom:updated>2026-06-29T20:47:51.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ruhsxF3O00PlEQBf4Z9zw.jpeg" /></figure><p>Deep learning models have become increasingly powerful over the years. Researchers often improved performance by making networks:</p><ul><li>Deeper (more layers)</li><li>Wider (more channels)</li><li>Higher resolution (larger input images)</li></ul><p>However, simply increasing one of these dimensions often leads to inefficient use of computational resources.</p><p>This challenge led researchers at Google to introduce <strong>EfficientNet</strong>, a family of convolutional neural networks that achieved state-of-the-art accuracy while using significantly fewer parameters and FLOPs.</p><p>In this article, we’ll explore:</p><ul><li>Why EfficientNet was created</li><li>The concept of Compound Scaling</li><li>EfficientNet architecture</li><li>EfficientNet variants (B0–B7)</li><li>Dog breed classification example</li><li>Python implementation using TensorFlow</li></ul><h3>The Problem with Traditional Scaling</h3><p>Suppose we have a CNN that classifies dogs.</p><h4>Option 1: Make it Deeper</h4><p>Increase layers:</p><pre>20 Layers → 50 Layers → 100 Layers</pre><p>Benefits:</p><ul><li>Learns more complex features</li></ul><p>Problems:</p><ul><li>More memory</li><li>Longer training time</li><li>Diminishing returns</li></ul><h4>Option 2: Make it Wider</h4><p>Increase channels:</p><pre>64 Filters → 128 Filters → 256 Filters</pre><p>Benefits:</p><ul><li>Captures richer features</li></ul><p>Problems:</p><ul><li>Parameter count explodes</li></ul><h4>Option 3: Increase Image Resolution</h4><p>Increase image size:</p><pre>224×224 → 380×380 → 600×600</pre><p>Benefits:</p><ul><li>More visual detail</li></ul><p>Problems:</p><ul><li>Higher computational cost</li></ul><p>Researchers discovered that scaling only one dimension is not optimal.</p><p>The solution?</p><p>Scale all dimensions together in a balanced manner.</p><p>This idea became the foundation of EfficientNet.</p><h3>What Makes EfficientNet Different?</h3><p>EfficientNet introduces a technique called:</p><h4>Compound Scaling</h4><p>Instead of arbitrarily increasing depth, width, or resolution, EfficientNet scales them simultaneously.</p><p>Mathematically:</p><pre>Depth = α^φ <br>Width = β^φ <br>Resolution = γ^φ</pre><p>where:</p><ul><li>α controls depth</li><li>β controls width</li><li>γ controls resolution</li><li>φ determines model size</li></ul><p>Subject to:</p><pre>α × β² × γ² ≈ 2</pre><p>This ensures computational resources are allocated efficiently.</p><h3>Intuition Using Dog Breed Classification</h3><p>Imagine we want to distinguish between:</p><ul><li>Husky</li><li>Golden Retriever</li><li>German Shepherd</li></ul><p>A small model may focus on:</p><ul><li>General shape</li><li>Fur color</li></ul><p>A larger EfficientNet model learns:</p><h4>More Depth</h4><p>Recognizes:</p><ul><li>Ear structure</li><li>Facial proportions</li></ul><h4>More Width</h4><p>Learns multiple feature types simultaneously:</p><ul><li>Fur texture</li><li>Eye shape</li><li>Tail characteristics</li></ul><h4>Higher Resolution</h4><p>Detects fine details:</p><ul><li>Fur patterns</li><li>Nose structure</li><li>Eye color variations</li></ul><p>Because all three dimensions grow together, the network learns richer representations without wasting computation.</p><h3>EfficientNet Architecture Overview</h3><p>The EfficientNet family starts with a baseline model called:</p><h4>EfficientNet-B0</h4><p>All larger models are scaled versions of B0.</p><p>Architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tl1hQbBQWSuGPgNMgvdh2Q.png" /></figure><h3>What is an MBConv Block?</h3><p>EfficientNet is built using:</p><h3>Mobile Inverted Bottleneck Convolution (MBConv)</h3><p>Originally introduced in MobileNetV2.</p><p>An MBConv block contains:</p><pre>   Input <br>     │<br>     ▼<br>1×1 Expansion Conv <br>     │ <br>     ▼ <br>Depthwise Convolution <br>     │ <br>     ▼ <br>Squeeze-and-Excitation <br>     │ <br>     ▼ <br>1×1 Projection Conv<br>     │ <br>     ▼ <br>   Output</pre><p>Benefits:</p><ul><li>Fewer parameters</li><li>Faster inference</li><li>Better accuracy</li></ul><h3>Squeeze-and-Excitation (SE) Module</h3><p>EfficientNet also uses:</p><h3>Channel Attention</h3><p>The network learns:</p><pre>Which channels are important?</pre><p>Example:</p><p>For a Husky image:</p><ul><li>Fur texture channel → important</li><li>Background channel → less important</li></ul><p>The SE module automatically increases useful channels and suppresses irrelevant ones.</p><p>This improves feature quality significantly.</p><h3>EfficientNet Variants</h3><p>EfficientNet comes in multiple sizes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*VmAyicgHuO5xY7yBBeQpOA.png" /></figure><p>As we move from B0 to B7:</p><ul><li>Depth increases</li><li>Width increases</li><li>Resolution increases</li></ul><p>using compound scaling.</p><h3>Understanding Feature Learning</h3><p>Let’s classify a German Shepherd.</p><h3>Early Layers</h3><p>Learn:</p><ul><li>Edges</li><li>Corners</li><li>Simple textures</li></ul><pre>//// <br>---- <br>|||||</pre><h3>Middle Layers</h3><p>Learn:</p><ul><li>Eyes</li><li>Ears</li><li>Fur patterns</li></ul><h3>Deep Layers</h3><p>Learn:</p><pre>German Shepherd Face</pre><p>rather than individual edges.</p><p>This hierarchical learning enables accurate classification.</p><h3>Why EfficientNet Became Popular</h3><p>Compared to older CNNs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*7pkuaez5pbnO9PvBrG5IbQ.png" /></figure><p>EfficientNet achieves an outstanding balance between:</p><ul><li>Accuracy</li><li>Speed</li><li>Memory usage</li></ul><h3>Transfer Learning with EfficientNet</h3><p>Most real-world projects use pretrained EfficientNet.</p><p>Advantages:</p><ul><li>Faster training</li><li>Less data required</li><li>Better performance</li></ul><p>Instead of learning from scratch, the model starts with knowledge from ImageNet’s millions of images.</p><h3><strong>Python Example Using TensorFlow</strong></h3><p><strong>Load EfficientNetB0</strong></p><pre># Import EfficientNetB0 model from Keras applications (pretrained CNN architecture)<br>from tensorflow.keras.applications import EfficientNetB0<br><br># Load EfficientNetB0 pretrained on ImageNet dataset<br># weights=&#39;imagenet&#39; → uses learned weights from ImageNet (transfer learning)<br># include_top=False → removes the final classification layer (we add our own head)<br># input_shape=(224, 224, 3) → expects 224x224 RGB images as input<br>base_model = EfficientNetB0(<br>    weights=&#39;imagenet&#39;,<br>    include_top=False,<br>    input_shape=(224, 224, 3)<br>)</pre><h4>Freeze Pretrained Layers</h4><pre># Freeze the pretrained EfficientNetB0 base model<br># This prevents its weights from being updated during training<br># Useful in transfer learning to retain learned ImageNet features<br># and train only the custom classification head on top<br>base_model.trainable = False</pre><h4>Build Classification Model</h4><pre># Import necessary Keras modules for building the neural network<br>from tensorflow.keras import layers<br>from tensorflow.keras import models<br><br># Build a Sequential model by stacking layers on top of the pretrained EfficientNetB0 base<br>model = models.Sequential([<br>    base_model,  # Pretrained EfficientNetB0 feature extractor (frozen if base_model.trainable = False)<br><br>    # Global Average Pooling reduces spatial dimensions (H x W) into a single vector per feature map<br>    layers.GlobalAveragePooling2D(),<br><br>    # Dropout layer to reduce overfitting by randomly deactivating 30% of neurons during training<br>    layers.Dropout(0.3),<br><br>    # Fully connected dense layer to learn higher-level patterns from extracted features<br>    layers.Dense(128, activation=&#39;relu&#39;),<br><br>    # Output layer for 3-class classification using softmax activation<br>    # Produces probability distribution across the 3 classes<br>    layers.Dense(3, activation=&#39;softmax&#39;)<br>])</pre><h4>Compile Model</h4><pre># Compile the model by defining how it will learn and be evaluated during training<br><br>model.compile(<br>    optimizer=&#39;adam&#39;,<br>    # Adam optimizer: adaptive learning rate optimizer that combines momentum + RMSprop<br>    # Works well for most deep learning classification problems<br><br>    loss=&#39;categorical_crossentropy&#39;,<br>    # Loss function used for multi-class classification when labels are one-hot encoded<br>    # Measures how far predicted probabilities are from true class distribution<br><br>    metrics=[&#39;accuracy&#39;]<br>    # Evaluation metric to track during training and validation<br>    # Accuracy = percentage of correctly predicted samples<br>)</pre><h4>Train</h4><pre># Train the model using the training dataset and validate on validation dataset<br><br>history = model.fit(<br>    train_dataset,<br>    # Training data pipeline (input images + labels)<br><br>    validation_data=val_dataset,<br>    # Validation data used to monitor model performance on unseen data during training<br><br>    epochs=10<br>    # Number of complete passes through the entire training dataset<br>    # Each epoch updates model weights based on backpropagation<br>)</pre><h3>Example Predictions</h3><p>Suppose the model sees these images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*TI0SOcwy9xN9ZV1WPKpzlQ.png" /></figure><p>The pretrained EfficientNet backbone already understands general visual concepts, making transfer learning highly effective.</p><h3>When Should You Use EfficientNet?</h3><p>EfficientNet is an excellent choice when:</p><p>✅ High accuracy is required</p><p>✅ GPU memory is limited</p><p>✅ Dataset size is moderate</p><p>✅ Transfer learning is desired</p><p>Examples:</p><ul><li>Medical imaging</li><li>Wildlife monitoring</li><li>Industrial inspection</li><li>Dog breed classification</li><li>Agricultural AI</li></ul><h3>EfficientNet vs Previous Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*1OFwYhDnhc9_TiQe9_ZKwg.png" /></figure><h3><strong>A Quick Reference Guide</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4SyQFV7hAabRnwNUnpEYjA.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Conclusion</h3><p>EfficientNet revolutionized CNN design by introducing Compound Scaling, a smarter way to increase model capacity. Instead of simply making networks deeper or wider, EfficientNet balances depth, width, and image resolution simultaneously.</p><p>The result is a family of models that delivers exceptional accuracy while remaining computationally efficient.</p><p>For many computer vision projects, EfficientNet remains one of the strongest choices for transfer learning, offering an ideal combination of performance, speed, and resource efficiency.</p><h3>References</h3><ol><li>EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks<br> <a href="https://arxiv.org/abs/1905.11946">https://arxiv.org/abs/1905.11946</a></li><li>TensorFlow EfficientNet Documentation<br> <a href="https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet">https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet</a></li><li>Google Research Blog — EfficientNet<br> <a href="https://research.google/blog/efficientnet-improving-accuracy-and-efficiency-through-automl-and-model-scaling/">https://research.google/blog/efficientnet-improving-accuracy-and-efficiency-through-automl-and-model-scaling/</a></li><li>Keras EfficientNet API Documentation<br> <a href="https://keras.io/api/applications/efficientnet">https://keras.io/api/applications/efficientnet</a></li></ol><h4>A Message from AI Mind</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*5Wm7sOfTpe5DEbhg.gif" /></figure><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://pub.aimind.so/">AI Mind Publication</a></li><li>🧠 Improve your <a href="https://www.aimind.so/prompt-generator?utm_source=pub&amp;utm_medium=message">AI prompts effortlessly and FREE</a></li><li><strong>🧰 Discover </strong><a href="https://www.aimind.so/?utm_source=pub&amp;utm_medium=message"><strong>Intuitive AI Tools</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6f11856e4eb" width="1" height="1" alt=""><hr><p><a href="https://pub.aimind.so/a-practical-guide-to-efficientnet-with-python-examples-e6f11856e4eb">A Practical Guide to EfficientNet with Python Examples</a> was originally published in <a href="https://pub.aimind.so/">AI Mind</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MobileNet Explained: Fast and Efficient Deep Learning for Mobile Devices]]></title>
            <link>https://medium.com/data-and-beyond/mobilenet-explained-fast-and-efficient-deep-learning-for-mobile-devices-cf8b4bdbe77c?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf8b4bdbe77c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[mobilenet]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 11:32:01 GMT</pubDate>
            <atom:updated>2026-06-24T11:32:01.535Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t7HCQyplR5XN5BwArUCAAw.png" /><figcaption>Designed using Copilot</figcaption></figure><p><em>Learn MobileNet architecture step-by-step using a Husky, Golden Retriever, and German Shepherd classification example.</em></p><p>Deep learning models such as VGG16, ResNet, and DenseNet achieve impressive accuracy, but they often require millions of parameters and significant computational power. Running these models on smartphones, embedded systems, drones, or IoT devices can be challenging.</p><p>This is where <strong>MobileNet</strong> comes in.</p><p>MobileNet was designed by Google to provide:</p><p>· Smaller model size</p><p>· Faster inference</p><p>· Lower memory usage</p><p>· Good accuracy</p><p>Instead of using standard convolutions everywhere, MobileNet introduces a clever technique called <strong>Depthwise Separable Convolution</strong>, dramatically reducing computation while maintaining performance.</p><p>In this article, we’ll explore MobileNet layer-by-layer using a dog breed classification example.</p><h3>The Problem</h3><p>Suppose we want to classify images into three dog breeds:</p><ul><li>Husky</li><li>Golden Retriever</li><li>German Shepherd</li></ul><p>Our dataset contains thousands of labeled images.</p><p>A traditional CNN can solve this problem, but deploying it on:</p><ul><li>Mobile phones</li><li>Raspberry Pi</li><li>Smart cameras</li><li>Edge AI devices</li></ul><p>may be too expensive computationally.</p><p>MobileNet solves this challenge.</p><h3>Traditional CNN Approach</h3><p>Consider an input image:</p><pre>224 × 224 × 3</pre><p>A standard convolution layer might use:</p><pre>32 filters<br>3 × 3 kernel</pre><p>Output:</p><pre>224 × 224 × 32</pre><p>The problem is that every filter operates across all input channels simultaneously.</p><p>This requires many multiplications.</p><p>For mobile devices, this becomes expensive.</p><h3>MobileNet’s Key Idea</h3><p>Instead of one large convolution operation:</p><pre>Standard Convolution<br>       ↓<br>Depthwise Convolution<br>       +<br>Pointwise Convolution</pre><p>This approach is called:</p><p><strong>Depthwise Separable Convolution</strong></p><p>It reduces computation by approximately:</p><pre>8–9 times</pre><p>compared to standard convolutions.</p><h3>Understanding Depthwise Convolution</h3><p>Assume the input image is:</p><pre>224 × 224 × 3</pre><p>representing:</p><pre>Red<br>Green<br>Blue</pre><p>channels.</p><p>Instead of applying filters across all channels together, MobileNet processes each channel independently.</p><h3>Husky Example</h3><p>Suppose the image contains a Husky.</p><h3>Channel 1 (Red)</h3><p>Detects:</p><ul><li>Fur brightness</li><li>Facial contrast</li></ul><h3>Channel 2 (Green)</h3><p>Detects:</p><ul><li>Eye region</li><li>Ear boundaries</li></ul><h3>Channel 3 (Blue)</h3><p>Detects:</p><ul><li>Background snow</li><li>Fur texture</li></ul><p>Each channel gets its own convolution filter.</p><pre>Red Channel   → Filter 1<br>Green Channel → Filter 2<br>Blue Channel  → Filter 3</pre><p>Output:</p><pre>224 × 224 × 3</pre><p>This operation is called:</p><h4>Depthwise Convolution</h4><h3>Why Depthwise Convolution Alone Isn’t Enough</h3><p>After depthwise convolution, channels remain separate.</p><p>The network knows:</p><ul><li>Fur texture</li><li>Ear shape</li><li>Eye regions</li></ul><p>But it hasn’t combined these clues yet.</p><p>To recognize a Husky, features must interact.</p><p>That’s where Pointwise Convolution comes in.</p><h3>Pointwise Convolution</h3><p>Pointwise convolution uses:</p><pre>1 × 1 convolution</pre><p>across all channels.</p><p>Think of it as feature mixing.</p><p>For example:</p><pre>Fur Texture<br>+<br>Pointed Ears<br>+<br>Blue Eyes<br>=<br>Husky</pre><p>The network combines information from all channels to form meaningful patterns.</p><p>Output become:</p><pre>224 × 224 × 32</pre><p>feature maps.</p><h3>Why It Saves Computation</h3><p>Suppose:</p><p>Input:</p><pre>224 × 224 × 3</pre><p>Output:</p><pre>224 × 224 × 32</pre><p>Kernel:</p><pre>3 × 3</pre><h3>Standard Convolution Cost</h3><p>Parameters:</p><pre>3 × 3 × 3 × 32<br>=<br>864</pre><h3>MobileNet Cost</h3><p>Depthwise:</p><pre>3 × 3 × 3<br>=<br>27</pre><p>Pointwise:</p><pre>1 × 1 × 3 × 32 = 96</pre><p>Total:</p><pre>27 + 96<br>=<br>123</pre><p>Instead of:</p><pre>864</pre><p>we only need:</p><pre>123</pre><p>parameters.</p><p>Huge savings!</p><h3>MobileNet Architecture</h3><p>A simplified MobileNet architecture looks like:</p><pre>Input Image<br>      ↓<br>  3×3 Conv<br>      ↓<br>Depthwise Conv<br>      ↓<br>Pointwise Conv<br>      ↓<br>Depthwise Conv<br>      ↓<br>Pointwise Conv<br>      ↓<br>Depthwise Conv<br>      ↓<br>Pointwise Conv<br>      ↓<br>Global Average Pooling<br>      ↓<br>Dense Layer<br>      ↓<br>Output Classes</pre><p>The network repeatedly stacks:</p><pre>Depthwise Conv<br>      +<br>Pointwise Conv</pre><p>blocks.</p><h3><strong>What Does MobileNet Learn?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MfNb_01lYrEFlQ9rvM-PuQ.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Early Layers</h3><p>The model learns simple features:</p><h4>Husky</h4><ul><li>Fur edges</li><li>Eye boundaries</li></ul><h4>Golden Retriever</h4><ul><li>Smooth fur contours</li></ul><h4>German Shepherd</h4><ul><li>Dark facial patterns</li></ul><h3>Middle Layers</h3><p>Features become more meaningful.</p><h4>Husky</h4><ul><li>Pointed ears</li><li>Thick fur</li></ul><h4>Golden Retriever</h4><ul><li>Long ears</li><li>Golden coat</li></ul><h4>German Shepherd</h4><ul><li>Black saddle pattern</li><li>Upright ears</li></ul><h3>Deep Layers</h3><p>The model learns breed-level concepts.</p><h4>Husky</h4><pre>Blue eyes<br>   +<br>Dense fur<br>   +<br>Snow-like texture</pre><h4>Golden Retriever</h4><pre>Golden coat<br>      +<br>Friendly facial structure</pre><h4>German Shepherd</h4><pre>Black-and-tan coloration<br>     +<br>Strong muzzle</pre><h3>Global Average Pooling</h3><p>Traditional CNNs often use large fully connected layers.</p><p>These layers add many parameters.</p><p>MobileNet replaces them with:</p><pre>Global Average Pooling</pre><p>Example:</p><p>Before:</p><pre>7 × 7 × 1024</pre><p>After pooling:</p><pre>1 × 1 × 1024</pre><p>Benefits:</p><ul><li>Fewer parameters</li><li>Less overfitting</li><li>Faster inference</li></ul><h3>Final Classification Layer</h3><p>The final layer predicts probabilities.</p><p>Example output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*bqpifJM3lErISt_-8QztCQ.png" /></figure><p>Prediction:</p><pre>Husky</pre><h3>MobileNet in TensorFlow</h3><h4>Import Libraries</h4><pre>from tensorflow.keras.applications import MobileNet<br>from tensorflow.keras import layers, models</pre><h4>Load Pretrained MobileNet</h4><pre># Load MobileNet pretrained on ImageNet<br>base_model = MobileNet(<br>    weights=&#39;imagenet&#39;,<br>    include_top=False,<br>    input_shape=(224,224,3)<br>)</pre><h4><strong>Freeze Feature Extractor</strong></h4><pre># Prevent pretrained weights from being updated<br>base_model.trainable = False</pre><h4><strong>Build Classification Model</strong></h4><pre># Create a custom classifier on top of MobileNet<br><br>model = models.Sequential([<br>    <br>    # Pretrained MobileNet feature extractor<br>    base_model,<br>    <br>    # Convert feature maps into a feature vector<br>    layers.GlobalAveragePooling2D(),<br>    <br>    # Learn higher-level breed-specific features<br>    layers.Dense(128, activation=&#39;relu&#39;),<br>    <br>    # Reduce overfitting<br>    layers.Dropout(0.3),<br>    <br>    # Output layer for 3 dog breeds<br>    layers.Dense(3, activation=&#39;softmax&#39;)<br>])</pre><h4><strong>Compile Model</strong></h4><pre># Configure training process<br><br>model.compile(<br>    optimizer=&#39;adam&#39;,<br>    loss=&#39;categorical_crossentropy&#39;,<br>    metrics=[&#39;accuracy&#39;]<br>)</pre><h4><strong>Train Model</strong></h4><pre># Train on dog breed images<br><br>history = model.fit(<br>    train_ds,<br>    validation_data=val_ds,<br>    epochs=10<br>)</pre><h3>Advantages of MobileNet</h3><h3>Extremely Fast</h3><p>Suitable for:</p><ul><li>Smartphones</li><li>Tablets</li><li>Embedded systems</li></ul><h3>Small Memory Footprint</h3><p>Requires much less storage than larger CNNs.</p><h3>Good Accuracy</h3><p>Provides an excellent trade-off between:</p><ul><li>Speed</li><li>Size</li><li>Accuracy</li></ul><h3>Easy Transfer Learning</h3><p>Can be fine-tuned for:</p><ul><li>Medical imaging</li><li>Face recognition</li><li>Defect detection</li><li>Wildlife classification</li></ul><h3>Limitations of MobileNet</h3><h4>Slightly Lower Accuracy</h4><p>Usually less accurate than:</p><ul><li>ResNet</li><li>DenseNet</li><li>EfficientNet</li></ul><h4>Less Suitable for Large Servers</h4><p>If computational resources are abundant, larger models may perform better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*nv07_CY-1Ir9fkanHeMQDQ.png" /></figure><h3>When Should You Use MobileNet?</h3><p>MobileNet is an excellent choice when:</p><p>✅ Running on smartphones</p><p>✅ Deploying to edge devices</p><p>✅ Real-time inference is required</p><p>✅ Memory is limited</p><p>✅ Building lightweight AI applications</p><p>Examples include:</p><ul><li>Dog breed recognition apps</li><li>Wildlife monitoring cameras</li><li>Smart doorbells</li><li>Medical screening tools</li><li>Industrial defect detection systems</li></ul><h3><strong>A Quick Reference Guide</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HvJ3cE9iysnMtixIQgnaTw.png" /><figcaption>Image created by author using AI guidance</figcaption></figure><h3>Final Thoughts</h3><p>MobileNet revolutionized lightweight deep learning by replacing expensive convolutions with <strong>Depthwise Separable Convolutions</strong>.</p><p>Using our dog breed example:</p><ul><li>Early layers detected fur edges and textures.</li><li>Middle layers identified ears, eyes, and coat patterns.</li><li>Deep layers learned breed-specific concepts.</li><li>The final classifier distinguished between Huskies, Golden Retrievers, and German Shepherds.</li></ul><p>The result is a model that is significantly smaller and faster than traditional CNNs while maintaining strong classification performance, making MobileNet one of the most practical architectures for real-world AI deployment.</p><h3><strong>References</strong></h3><p>MobileNet Paper : <a href="https://arxiv.org/abs/1704.04861">https://arxiv.org/abs/1704.04861</a>TensorFlow MobileNet</p><p>Keras Applications Guide: <a href="https://keras.io/api/applications/">https://keras.io/api/applications/</a></p><p>MobileNetV2 Paper : <a href="https://arxiv.org/abs/1801.04381">https://arxiv.org/abs/1801.04381</a></p><p>MobileNetV3 Paper : <a href="https://arxiv.org/abs/1905.02244">https://arxiv.org/abs/1905.02244</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf8b4bdbe77c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-and-beyond/mobilenet-explained-fast-and-efficient-deep-learning-for-mobile-devices-cf8b4bdbe77c">MobileNet Explained: Fast and Efficient Deep Learning for Mobile Devices</a> was originally published in <a href="https://medium.com/data-and-beyond">Data And Beyond</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Model Development on HPC Clusters with SLURM: From Basics to Production]]></title>
            <link>https://ai.plainenglish.io/machine-learning-model-development-on-hpc-clusters-with-slurm-from-basics-to-production-12d16765874f?source=rss-b7b73768c0de------2</link>
            <guid isPermaLink="false">https://medium.com/p/12d16765874f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[slurm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sabitha Manoj]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 05:36:24 GMT</pubDate>
            <atom:updated>2026-06-20T05:36:24.749Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*qTTCm5F6-2ue6m8PkqRoKA.jpeg" /><figcaption>Designed with Co-pilot</figcaption></figure><p>Training machine learning models on a laptop is great for learning, but real-world ML projects often involve:</p><ul><li>Millions of training samples</li><li>Large deep learning models</li><li>Multiple GPUs</li><li>Hyperparameter tuning</li><li>Distributed training</li></ul><p>As datasets and models grow, a single machine quickly becomes a bottleneck.</p><p>This is where <strong>High Performance Computing (HPC) Clusters</strong> and <strong>SLURM</strong> come into play.</p><p>In this article, we’ll explore:</p><p>· What an HPC cluster is</p><p>· Why ML engineers use clusters</p><p>· Understanding SLURM</p><p>· Running Python ML jobs on a cluster</p><p>· GPU training</p><p>· Distributed training</p><p>· Best practices for production environments</p><h3>Why Not Train Everything on a Laptop?</h3><p>Suppose you’re training an image classification model.</p><p>Dataset:</p><ul><li>500,000 images</li><li>224×224 RGB</li><li>ResNet50 architecture</li></ul><p>Training may take:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*0jIXTGYrGoFDmAenoJv9GA.png" /></figure><h3>What Is an HPC Cluster?</h3><p>An HPC (High Performance Computing) cluster is a collection of computers connected together.</p><p>Think of it as:</p><pre>+----------------------+<br>| Login Node           |<br>+----------+-----------+<br>           |<br>           |<br>+----------+-----------+<br>| Scheduler (SLURM)    |<br>+----------+-----------+<br>           |<br> ----------------------------------<br> |               |                |<br> |               |                |<br>Node 1         Node 2          Node 3<br>CPU/GPU       CPU/GPU         CPU/GPU</pre><h3><strong>Components:</strong></h3><h4>Login Node</h4><p>Used for:</p><ul><li>Writing code</li><li>Submitting jobs</li><li>Monitoring jobs</li></ul><p>Not used for training.</p><h4>Compute Nodes</h4><p>Actual machines where training happens.</p><p>Examples:</p><pre>Node 1<br>- 64 CPU cores<br>- 256 GB RAM<br>- 4 NVIDIA GPUs<br><br>Node 2<br>- 128 CPU cores<br>- 512 GB RAM<br>- 8 NVIDIA GPUs</pre><h4>Shared Storage</h4><p>All nodes can access the same data.</p><pre>/data<br>/models<br>/results</pre><p>This allows any node to train on the same dataset.</p><h3>What Is SLURM?</h3><p>SLURM stands for:</p><p><strong>Simple Linux Utility for Resource Management</strong></p><p>It is one of the most popular workload managers used in:</p><ul><li>Universities</li><li>Research labs</li><li>National supercomputers</li><li>Enterprise AI platforms</li></ul><p>SLURM manages:</p><ul><li>CPU allocation</li><li>GPU allocation</li><li>Memory allocation</li><li>Job scheduling</li><li>Queue management</li></ul><p>Instead of directly running:</p><pre>python train.py</pre><p>you submit a job to SLURM.</p><p>SLURM decides:</p><ul><li>Which node to use</li><li>When resources are available</li><li>How many GPUs to allocate</li></ul><h3><strong>Typical ML Workflow</strong></h3><pre>Write Code<br>    ↓<br>Submit Job<br>    ↓<br>SLURM Queue<br>    ↓<br>Compute Node<br>    ↓<br>Training<br>    ↓<br>Results Saved</pre><h3>A Simple Python Training Script</h3><p>Consider a small example.</p><pre># Import the Iris flower dataset<br>from sklearn.datasets import load_iris<br><br># Import the Random Forest classification algorithm<br>from sklearn.ensemble import RandomForestClassifier<br><br># Load the Iris dataset<br># X = feature matrix (sepal length, sepal width, petal length, petal width)<br># y = target labels (flower species)<br>X, y = load_iris(return_X_y=True)<br><br># Create a Random Forest classifier with default parameters<br>model = RandomForestClassifier()<br><br># Train the model using the feature data (X) and labels (y)<br># During training, the Random Forest builds multiple decision trees<br># and learns patterns that distinguish the flower species<br>model.fit(X, y)<br><br># Print a confirmation message when training is complete<br>print(&quot;Training Complete&quot;)</pre><p>Save as:</p><pre>train.py</pre><h3>Running Locally</h3><p>Normally:</p><pre>python train.py</pre><p>On a cluster we submit it through SLURM.</p><h3>Your First SLURM Job</h3><p>Create:</p><pre>train.slurm</pre><pre>#!/bin/bash<br><br># SLURM job name shown in the job queue<br>#SBATCH --job-name=rf_train<br><br># File where standard output (print statements) will be saved<br>#SBATCH --output=output.log<br><br># File where error messages will be saved<br>#SBATCH --error=error.log<br><br># Maximum job runtime (HH:MM:SS)<br>#SBATCH --time=00:10:00<br><br># Number of CPU cores allocated to this job<br>#SBATCH --cpus-per-task=4<br><br># Amount of memory allocated to this job<br>#SBATCH --mem=4G<br><br># Execute the Python training script<br>python train.py</pre><h3>Understanding SLURM Parameters</h3><h4>Job Name</h4><pre>#SBATCH --job-name=rf_train</pre><p>Appears in queue.</p><h4>Output File</h4><pre>#SBATCH --output=output.log</pre><p>Stores standard output.</p><h4>Error File</h4><pre>#SBATCH --error=error.log</pre><p>Stores errors.</p><h4>Runtime Limit</h4><pre>#SBATCH --time=00:10:00</pre><p>Maximum runtime:</p><pre>HH:MM:SS</pre><h4>CPU Allocation</h4><pre>#SBATCH --cpus-per-task=4</pre><p>Allocates:</p><pre>4 CPU cores</pre><h4>Memory</h4><pre>#SBATCH --mem=4G</pre><p>Allocates:</p><pre>4 GB RAM</pre><h3><strong>Submitting the Job</strong></h3><pre># Submit the SLURM job script to the scheduler<br># SLURM places the job in a queue and runs it when<br># the requested resources (CPUs, memory, GPUs, etc.)<br># become available.<br>sbatch train.slurm</pre><p>Output:</p><pre>Submitted batch job 12345</pre><p>Job ID:</p><pre>12345</pre><h3>Monitoring Jobs</h3><p>Check queue:</p><pre># Display all SLURM jobs belonging to a specific user<br># Useful for monitoring running, pending, and completed jobs<br># in the cluster queue.<br>squeue -u username</pre><p>Cancel Job:</p><pre># Cancel a running or pending Slurm job with job ID 12345<br>scancel 12345</pre><p>Job Details</p><pre># Display detailed information about a specific Slurm job (job ID: 12345)<br># This includes job state, resources requested, nodes allocated, runtime, etc.<br>scontrol show job 12345</pre><p>Resource Usage</p><pre># Show job efficiency report for a completed Slurm job (job ID: 12345)<br># Displays CPU usage, memory usage, and overall resource efficiency<br>seff 12345</pre><h3>Best Practices for ML Engineers</h3><h4>1. Never Train on Login Nodes</h4><p>Login nodes are for:</p><ul><li>Editing code</li><li>Submitting jobs</li></ul><p>Training belongs on compute nodes.</p><h4>2. Save Checkpoints Frequently</h4><pre># Save only the model parameters (state_dict) to a file<br># This does NOT save the full model architecture, only learned weights<br># Useful for checkpointing and later reloading with the same model definition<br><br>torch.save(<br>    model.state_dict(),   # dictionary containing model weights<br>    &quot;checkpoint.pt&quot;       # output file path for saved checkpoint<br>)</pre><p>Protects against interruptions.</p><h3>3. Log Metrics</h3><p>Use:</p><ul><li>TensorBoard</li><li>MLflow</li><li>Weights &amp; Biases</li></ul><p>Track:</p><ul><li>Loss</li><li>Accuracy</li><li>GPU utilization</li></ul><h3>4. Use Job Arrays</h3><p>For:</p><ul><li>Hyperparameter tuning</li><li>Cross-validation</li><li>Model comparisons</li></ul><h3>5. Request Only Needed Resources</h3><p>Bad:</p><pre>--gres=gpu:8</pre><p>when using only one GPU.</p><p>Good resource requests improve cluster efficiency.</p><h3>Where Clusters and SLURM Are Used</h3><p>Machine learning clusters are common in:</p><ul><li>AI research labs</li><li>Pharmaceutical companies</li><li>Autonomous driving</li><li>Medical imaging</li><li>Finance</li><li>Aerospace</li><li>Large-scale recommendation systems</li></ul><p>Whether training a simple Random Forest or a billion-parameter deep learning model, understanding clusters and SLURM is a valuable skill for every Machine Learning Engineer.</p><h3><strong>A Quick Reference Guide</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqY6x82rWWUBw8epj9E48Q.png" /><figcaption>Image created using AI guidance</figcaption></figure><h3>Final Thoughts</h3><p>Building machine learning models is only part of the job. Production-scale ML requires efficient use of compute resources, reproducible experiments, and scalable training workflows.</p><p>SLURM provides a powerful way to manage CPUs, GPUs, memory, and distributed workloads across HPC clusters. As datasets and models continue to grow, understanding cluster-based development becomes an essential skill for ML engineers moving from experimentation to large-scale production systems.</p><h3>References</h3><ul><li>SLURM Documentation: <a href="https://slurm.schedmd.com/">https://slurm.schedmd.com</a></li><li>PyTorch Distributed Training: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></li><li>TensorFlow Distributed Training: <a href="https://www.tensorflow.org/guide/distributed_training">https://www.tensorflow.org/guide/distributed_training</a></li><li>OpenMPI: <a href="https://www.open-mpi.org/">https://www.open-mpi.org</a></li><li>MLflow: <a href="https://mlflow.org/">https://mlflow.org</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12d16765874f" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/machine-learning-model-development-on-hpc-clusters-with-slurm-from-basics-to-production-12d16765874f">Machine Learning Model Development on HPC Clusters with SLURM: From Basics to Production</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>