<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vardhan Devireddy on Medium]]></title>
        <description><![CDATA[Stories by Vardhan Devireddy on Medium]]></description>
        <link>https://medium.com/@vardhanreddy1228?source=rss-f58e913da9fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*WQeXVwVtsJ0tNN2M</url>
            <title>Stories by Vardhan Devireddy on Medium</title>
            <link>https://medium.com/@vardhanreddy1228?source=rss-f58e913da9fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:15:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vardhanreddy1228/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[P1 — MLOps: DVC — Pipelines Management and Experiment Tracking]]></title>
            <link>https://medium.com/@vardhanreddy1228/p1-mlops-dvc-pipelines-management-and-experiment-tracking-1efe8f4d66fd?source=rss-f58e913da9fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/1efe8f4d66fd</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[data-version-control]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vardhan Devireddy]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 12:32:33 GMT</pubDate>
            <atom:updated>2025-07-27T12:32:33.779Z</atom:updated>
            <content:encoded><![CDATA[<h3>P2 — MLOps: DVC — Pipelines Management and Experiment Tracking</h3><p>In machine learning, experimentation is the name of the game to optimize the model. You change a model’s architecture, adjust hyperparameters, shuffle training data, or even just refactor code — and suddenly, your results shift. The question then becomes: <strong>Which change made the model better?</strong> Without tracking, you’re looking numb.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzg3q48ZQboyW3VGazDeuw.jpeg" /><figcaption>Fig -1 : Data Version Control</figcaption></figure><p>This is where <strong>DVC (Data Version Control)</strong> steps in. It brings Git-like capabilities to data science: tracking datasets, parameters, models, and metrics — all versioned and reproducible alongside your Git commits.</p><h3>Pipeline Management</h3><h4>Adding a pipeline stage</h4><p>Let’s start setting up a DVC pipeline using the example project provided by the DVC team <a href="https://code.dvc.org/get-started/code.zip">here</a>. This sample project includes a data file (data.xml), parameters file(params.yaml), a feature generation script (src/prepare.py), and a model training script (src/train.py).</p><p>To build a reproducible and trackable ML workflow, DVC expects to define each processing step as a <strong>stage</strong>. Each stage describes its dependencies (input data or scripts), parameters(hyperparameters), outputs (transformed data, models, metrics), and the command to execute.</p><p>Let’s add the first stage to our pipeline — data preparation:</p><pre>dvc stage add -n prepare \<br>              -p prepare.seed,prepare.split \<br>              -d src/prepare.py -d data/data.xml \<br>              -o data/prepared \<br>              python src/prepare.py data/data.xml</pre><p><strong>Command breakdown:</strong></p><ul><li>-n — Specifies the name of the stage (prepare). This name identifies the stage within the pipeline.</li><li>-p— Declares the parameters used in this stage (prepare.seed, prepare.split), sourced from params.yaml.</li><li>-d— Lists the dependencies — in this case, the script and input data file.</li><li>-o — Points to the output directory (data/prepared) where the results of this stage are stored.</li></ul><p><strong>Key Points to consider:</strong></p><ul><li>A pipeline definition is generated as dvc.yamlfile, and includes information about the stages we want to run, their dependencies, and outputs.</li><li>The output(data/prepared)produced in this stage will be automatically tracked by DVC, so there&#39;s no need to run dvc add manually.</li></ul><p>We can add multiple stages to the pipeline by using <a href="https://dvc.org/doc/command-reference/stage/add">dvc stage add</a> multiple times. Let’s add a 2nd stage to perform featurize extraction:</p><pre>dvc stage add -n featurize \<br>              -p featurize.max_features,featurize.ngrams \<br>              -d src/featurization.py -d data/prepared \<br>              -o data/features \<br>              python src/featurization.py data/prepared data/features</pre><p>And finally, let’s add a 3rd stage to train a model:</p><pre>dvc stage add -n train \<br>               -p train.seed,train.n_est,train.min_split \<br>               -d src/train.py -d data/features \<br>               -o model.pkl \<br>               python src/train.py data/features model.pkl</pre><h4>Running a pipeline</h4><p>Once the stages are added, we can run the entire pipeline using:</p><pre># Reproducing a entire pipline<br>dvc repro</pre><p><strong>Key Points to consider:</strong></p><ul><li>DVC uses the pipeline definition in dvc.yamlto reproduce the pipeline.</li><li>We can observe that a dvc.lock is generated to capture the pipeline results that contain md5/hashes of the dependencies and values of the parameters that were used. Simply, A state file, and we can correlate it to the <strong>Terraform</strong> state file.</li><li>Stages that haven’t changed will be skipped in the next run, as DVC uses the dvc.lock file to store metadata about executed commands, dependencies, and outputs.</li><li>Both dvc.yaml and dvc.lock should be committed to Git. This ensures that when you check out a specific Git commit, you can easily reproduce the exact results of your pipeline.</li></ul><h3>Experimenting Tracking</h3><p>Once your DVC pipeline is set up with stages, the next logical step is to <strong>experiment with different parameter values</strong> to tune the model’s performance. DVC provides a powerful feature for this, and it is <strong>dvc exp</strong> — for running the <strong>experiments</strong>.</p><h4><strong>Running Experiments</strong></h4><p>We can tweak the hyperparameters in params.yaml and trigger a new pipeline run using dvc exp run and it will execute the pipeline with the updated parameters and track the changes <strong>without affecting your Git workspace</strong>. Each experiment is versioned and stored internally by DVC, so you can always go back and compare results.</p><pre>dvc exp run</pre><h4>Running <strong>Experiments</strong> by <strong>setting parameters</strong> on the fly</h4><p>Rather than manually editing params.yaml, DVC lets us override parameter values directly using an argument --set-param . The below command updates the specified parameters and launches a new experiment without modifying the original files.</p><pre>dvc exp run --set-param train.n_est=200 --set-param train.min_split=4</pre><h4><strong>Running Batch Experiments</strong></h4><p>When we need to test multiple combinations of parameters for identifying the better performance, then use the --queue flag is advisable.</p><pre># Queues experiments to override the 8,16,24 to train.batch_size parameter.<br>dvc exp run --queue --set-param &quot;train.batch_size=8,16,24&quot;<br><br># Queues experiments to override with train.arch and train.img_size<br># parameters with all possible combination.<br>dvc exp run --queue \<br>-S &#39;train.arch=alexnet,resnet34,squeezenet1_1&#39; \<br>-S &#39;train.img_size=128,256&#39;<br><br>Note: -S and --set-param does the same usage for us.</pre><p>Once you’ve queued all desired configurations, run them all together:</p><pre>dvc exp run --run-all</pre><h4><strong>Listing experiments</strong></h4><p>After running experiments, we can view a <strong>table</strong> of experiments, showing metrics, parameter values, and outputs side-by-side. It helps you visually identify which parameter combinations lead to better results.</p><pre>dvc exp show</pre><h4><strong>Filter metrics/params columns</strong></h4><p>Helps to <strong>focus on what matters</strong> in large pipelines, <strong>simplifies your experiment review process, and reduces clutter</strong> when sharing results with your team</p><pre># To include<br>dvc exp show --include metrics.accuracy params.train.n_est<br><br># To exclude<br>dvc exp show --exclude params.prepare seed</pre><h4><strong>Comparing Experiments</strong></h4><p>While dvc exp show gives you a <strong>table of all experiments</strong>, sometimes you want to <strong>compare two specific experiments</strong> in detail — especially their parameter changes, metrics, or even output differences. That’s where dvc exp diff comes in.</p><pre># By default, this compares the latest experiment with the workspace. It shows:<br># Which parameters changed<br># How metrics improved or degraded<br># Any output files that were modified<br>dvc exp diff<br><br># Compares b/n exp-b123 and exp-c456<br>dvc exp diff exp-b123 exp-c456</pre><h4>Promoting the Best Experiment</h4><p>Once you’re happy with an experiment result — say the model trained with n_est=150 gives you the best performance — you can <strong>promote</strong> that experiment to your main workspace:</p><pre>dvc exp apply &lt;experiment-name&gt;</pre><p>This applies the experiment’s parameter values and outputs to your working directory. Now, we should version it properly using Git and DVC:</p><pre># Stage params.yaml and dvc.lock files to commit<br>git add params.yaml dvc.lock<br><br># Commit to track under GIT<br>git commit -m &quot;Promote experiment with n_est=150&quot;</pre><h4>Reproducing Past Experiments</h4><p>Since each experiment is tied to a Git commit and has all its parameters, inputs, and outputs tracked, you can reproduce a specific run at any time:</p><pre># Look for required commit and checkout to it.<br>git checkout &lt;commit-id&gt;<br><br># Inform DVC to point to right information as per commit. <br>dvc checkout<br><br># Now, Reproduce the pipeline and results remain same.<br>dvc repro</pre><h4>Sharing Experiments Without Git Commit</h4><p>DVC enables seamless collaboration by allowing you to share experiments even before committing them to Git. This is done using the dvc exp push and dvc exp pull commands.</p><ul><li><strong>dvc exp push origin &lt;experiment-name&gt;</strong>: Pushes your local experiment to the remote Git repository (e.g., origin) as a special Git reference, <strong>without requiring a Git commit</strong>.</li><li><strong>dvc exp pull origin &lt;experiment-name&gt;</strong>: Allows your teammates (or other machines) to fetch those experiments and reproduce them locally — again, without needing them to be part of the main Git history.</li><li>It saves the experiment as a special Git reference (refs/exps/...), <strong>not as a regular commit</strong>.</li></ul><p>This feature is especially useful during collaborative experimentation, enabling teams to test and compare different results without cluttering the Git log.</p><h3>Conclusion</h3><p>DVC is not only for versioning data, but also excels at managing ML pipelines and tracking experiments — all while remaining tightly integrated with Git. It empowers teams to quickly reproduce results, compare experiments with ease, and confidently promote the most promising models.</p><h3>References</h3><ul><li>DVC YouTube <a href="https://www.youtube.com/@dvcorg8370">Channel</a>.</li><li><a href="https://dvc.org/doc/start/data-pipelines">Pipeline Management</a>.</li><li><a href="https://dvc.org/doc/start/experiments">Experiment Tracking</a>.</li></ul><p>Thank You and Enjoy Learning!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1efe8f4d66fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[P1 — MLOps: DVC — Data Version Control]]></title>
            <link>https://medium.com/@vardhanreddy1228/p1-mlops-dvc-data-version-control-39e7adf45863?source=rss-f58e913da9fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/39e7adf45863</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dvc]]></category>
            <category><![CDATA[data-version-control]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Vardhan Devireddy]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 09:22:19 GMT</pubDate>
            <atom:updated>2025-07-23T09:22:19.249Z</atom:updated>
            <content:encoded><![CDATA[<h3>P1 — MLOps: DVC — Data Version Control</h3><p><strong>DVC (Data Version Control)</strong> is an open-source tool that brings version control to the world of machine learning. Just like Git tracks changes in code, DVC helps you track changes in your datasets, models, and experiments. It is designed to make ML models shareable and experiments reproducible and to track versions of models, data, and pipelines. More simply, “<em>Code has Git. Data has DVC.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQ3_QnmqzqfA_EdoiI9QqQ.png" /><figcaption>Fig 1: DVC — Data Version Control</figcaption></figure><h3>Why DVC is essential in MLOps</h3><p>Software development benefitted from Git for SCM- Source Code Management, but ML development faced the following challenges:</p><ul><li>Datasets and models are <strong>too large</strong> for Git. So, Git bloats and slows down with large files.</li><li><strong>Collaboration</strong> becomes painful when teammates have to clone huge repos.</li></ul><p>Here’s where DVC solves these problems. It:</p><ul><li><strong>Keeps your Git repo clean</strong> by storing only lightweight .dvc metafiles.</li><li><strong>Links code and data</strong> versions together.</li><li><strong>Supports remote storage</strong> (S3, GCS, Azure, SSH, etc.) to store data separately from code.</li><li>Makes data collaboration <strong>as easy as code collaboration</strong>.</li></ul><h3>What all can DVC do for us?</h3><ul><li><strong>Experiment Tracking — </strong>Manage and compare different ML experiments using dvc exp.</li><li><strong>Pipeline Management — </strong>DVC enables you to define and run data pipelines using dvc.yaml. This makes your ML workflow structured, repeatable, and transparent.</li><li><strong>Model Registry — S</strong>imilar to datasets, DVC can version and organise your trained models. You can tag your best models, push them to remote storage, and share them with your team.</li></ul><h3>DVC + GIT Workflow</h3><figure><img alt="This diagram outlines how to use DVC with Git for dataset storage." src="https://cdn-images-1.medium.com/max/1024/1*HJhroliBON9U73nTbCMiIg.png" /><figcaption>Fig 2: dvc + git workflow</figcaption></figure><h3>Practical Engagement with DVC</h3><h4>Initialization</h4><p>We should begin by setting up our project for both code and data versioning.</p><ul><li><strong>git init</strong> – Initializes a Git repository to version control your code and lightweight metadata.</li><li><strong>dvc init</strong> – Initializes DVC in the project, creating a .dvc directory to manage data versioning.</li><li><strong>dvc remote add -d myremote /path/to/storage</strong> – Configures where your data will be pushed or pulled from. You can add remotes like local folders, S3, GCS, etc</li></ul><blockquote><em>At this stage, your code is versioned via Git, and DVC is configured to track and store your data remotely.</em></blockquote><h4>Adding and Tracking Data</h4><p>Now we bring dataset under DVC’s version control.</p><ul><li><strong>dvc add</strong>– Tracks the dataset, stores it in the local DVC cache, and generates a .dvc metafile. <strong><em>E.g.</em></strong> dvc add path/to/dataset</li><li><strong>dvc status</strong> – Checks if there are changes between your workspace, cache, and remote.</li><li><strong>dvc commit</strong> – If the data is already tracked and you modify it, use this to manually update the DVC cache.</li></ul><blockquote><em>These commands allow you to manage and track large files/data just like you do with code — but efficiently.</em></blockquote><h4>Pushing Code and Data</h4><p>Now that code and data are tracked, we store them safely in remote locations.</p><ul><li><strong>git commit</strong> – Commits your code and the .dvc files to Git.</li><li><strong>git push</strong> – Pushes your Git project (code + DVC metadata) to a Git hosting service.</li><li><strong>dvc push</strong> – Pushes actual data files from the local DVC cache to the configured remote storage.</li></ul><blockquote><em>This ensures your team or other machines can access both code and exact data versions without manual transfers.</em></blockquote><h3>Restoring or Verifying Data</h3><p>From a new machine or teammate’s setup:</p><ul><li><strong>git clone &lt;repo&gt;</strong> – Clones the Git repo with all code and DVC metafiles.</li><li><strong>dvc pull</strong> – Downloads the actual data from the remote storage using the .dvc files.</li><li><strong>dvc checkout</strong> – Updates the working directory to reflect the correct data version as per Git/DVC state.</li></ul><blockquote><em>This makes collaboration and reproducibility seamless.</em></blockquote><h3>Switching Versions: Git Checkout + DVC Checkout</h3><p>To switch to a different version of code <em>and</em> the corresponding dataset/model, you need to use both Git and DVC:</p><ul><li><strong>git checkout &lt;branch/commit&gt;</strong> — This changes the Git-tracked files, including your .dvc metafiles, to a previous commit or branch.</li><li><strong>dvc checkout</strong> — This uses the .dvc files from the checked-out Git commit to update your workspace with the correct versions of data files from the DVC cache.</li></ul><blockquote><em>git checkout changes metadata, </em><em>dvc checkout restores actual data versions.</em></blockquote><h3>Conclusion</h3><p>So, DVC simplifies managing the data-heavy world of machine learning by seamlessly integrating with Git and enabling efficient data versioning. I hope this article helped you understand the basics of DVC — from initializing a project, adding and tracking data, switching between data versions, to storing and retrieving data from remote storage.</p><h3>References</h3><ul><li>Check here for official DVC <a href="https://dvc.org/">website</a>.</li><li>Look at this <a href="https://www.youtube.com/@dvcorg8370">channel</a> on YouTube by DVC.</li></ul><p>Thank You and Enjoy Learning!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39e7adf45863" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[P1: HELM — User Input Validation]]></title>
            <link>https://medium.com/@vardhanreddy1228/p1-helm-user-input-validation-714b3d45f7fa?source=rss-f58e913da9fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/714b3d45f7fa</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[validation]]></category>
            <category><![CDATA[required]]></category>
            <category><![CDATA[fail]]></category>
            <category><![CDATA[helm-chart]]></category>
            <dc:creator><![CDATA[Vardhan Devireddy]]></dc:creator>
            <pubDate>Sat, 08 Mar 2025 16:55:55 GMT</pubDate>
            <atom:updated>2025-03-08T16:55:55.439Z</atom:updated>
            <content:encoded><![CDATA[<h3>P1: HELM — User Input Validation</h3><p>Building a Helm chart simplifies deployment, but enforcing robust input validation and error handling is crucial to ensure the user does not provide the faulty values, and having a strict validation over user-provided data prevents misconfigurations and improves the user experience. In Helm, the required and fail functions play a key role in performing validation over user-supplied values, and they ensure errors are managed gracefully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*vi6Wx5P9QQf54PyFVsU4aA.png" /><figcaption>Helm — User Input Validation</figcaption></figure><h3>Required Function</h3><p>The required function ensures that a particular value is provided in the Helm values.yaml file. If the value is <strong>missing</strong> or <strong>empty</strong>, Helm will throw an error supplied by the Helm Chart developer and prevent the chart from being installed.</p><h4><strong>Syntax</strong></h4><pre>{{ required &quot;Error Message&quot; .Values.someValue }}</pre><h4>Example</h4><pre>{{ required &quot;A valid database host entry is required.&quot; .Values.database.host | quote }}</pre><ul><li>If database hostis not defined in values.yaml, then Helm Chart installation will fail and return an error.</li></ul><h4>Use Cases</h4><ul><li>If the Database Host is not missing.</li><li>If the Docker Image Registry is not provided.</li></ul><h3>Fail Function</h3><p>The fail function is used to terminate the Helm rendering process forcefully with a custom error message supplied by the Helm Chart developer. If the user-supplied values <strong>aren’t</strong> <strong>valid(Runtime/Dynamic condition checks) </strong>during the Helm Chart installation.</p><h4>Syntax</h4><pre>{{ fail &quot;Error message&quot; }}</pre><h4>Example</h4><pre>{{- if not (gt (int .Values.replicaCount) 0) }}<br>  {{ fail &quot;replicaCount must be greater than 0 in values.yaml&quot; }}<br>{{- end }}</pre><ul><li>If replicaCount is not greater than <strong>0</strong>, then Helm Chart installation will fail and return an error.</li></ul><h4>Use Cases</h4><ul><li>ReplicaCount is provided but not supplied the proper value.</li><li>If a valid proxy tlsmode(tls|mtls|insecure) is not provided.</li></ul><h3>Conclusion</h3><p>To summarize, the <strong>Required</strong> and <strong>Fail</strong> functions serve distinct purposes. The <strong>Required</strong> function is to address missing or empty values in the values.yaml file, while the <strong>Fail</strong> function is utilized to ensure that the provided values are valid for the Helm Chart installation.</p><h3>References</h3><ul><li>Helm Chart <a href="https://helm.sh/docs/">Documentation</a>.</li></ul><p>Thank You and Enjoy Learning!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=714b3d45f7fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Squid as a Forward Proxy via Docker]]></title>
            <link>https://medium.com/@vardhanreddy1228/squid-as-a-forward-proxy-via-docker-dc3b4e11578c?source=rss-f58e913da9fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc3b4e11578c</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[proxy-server]]></category>
            <category><![CDATA[squid]]></category>
            <category><![CDATA[forward-proxy]]></category>
            <category><![CDATA[proxy]]></category>
            <dc:creator><![CDATA[Vardhan Devireddy]]></dc:creator>
            <pubDate>Sat, 15 Feb 2025 12:27:22 GMT</pubDate>
            <atom:updated>2025-02-15T14:51:40.759Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog provides you with detailed instructions on how to set up a TLS and MTLS forward proxy using Squid using Docker as it is machine-independent and helps to run consistently across different environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJK32hvBANtwO6xvtjjteA.png" /><figcaption><strong>Request-Response Cycle</strong></figcaption></figure><h4>What is a Forward Proxy?</h4><p>A forward proxy is an intermediary layer(Man-In-The-Middle) between a user’s device/application and the Internet. It forwards the user’s requests to the destination server and returns the server’s response to the user without revealing the user’s identity.</p><h3>Let’s Get Started</h3><h4>#1 Create a Dockerfile</h4><p>This docker file helps us to create a container with Ubuntu as a base image and installs Squid binary.</p><pre>FROM ubuntu:22.04<br><br>ARG DEBIAN_FRONTEND=noninteractive<br><br>EXPOSE 3128<br>VOLUME /data<br><br>RUN apt-get update &amp;&amp; apt-get upgrade -y<br>RUN apt-get install -y squid-openssl</pre><h4>#2 Create a Squid Conf file</h4><p>we need to know about a few parameters that are mentioned below to run a proxy in mTLS or TLS mode:</p><blockquote><strong>tls-cert</strong> — Path to a file containing the Self-Signed certificate of the proxy server.<br><strong>tls-key</strong> — Path to a file containing the private key of the proxy server.<br><strong>clientca</strong> — Path to a file containing the CAs of the client to verify when a client makes the request.</blockquote><p>If the proxy is required to operate in <strong>mTLS</strong> mode, it is essential to provide the tls-cert, tls-key, andclientca. These are required to establish <strong>mutual trust</strong> between both the client and the proxy server.</p><pre>http_access allow all<br><br>https_port 3128 tls-cert=/path/to/server.crt tls-key=/path/to/server.key clientca=/path/to/client.crt<br><br>refresh_pattern . 0 0% 0<br><br>access_log stdio:/data/access.log<br>cache_log stdio:/data/cache.log<br>pid_filename /data/squid.pid</pre><p>If the proxy is required to operate in <strong>TLS</strong> mode, it is required to provide only the tls-cert and tls-key because the client only verifies the proxy server, but the proxy does not verify the client.</p><pre>http_access allow all<br><br>https_port 3128 tls-cert=/path/to/server.crt tls-key=/path/to/server.key<br><br>refresh_pattern . 0 0% 0<br><br>access_log stdio:/data/access.log<br>cache_log stdio:/data/cache.log<br>pid_filename /data/squid.pid</pre><h4>#3 Start Squid Server</h4><p>Execute the below script which does 3 major steps, Which are outlined below:</p><ol><li>Creates a few directories to store the <em>certs</em> and <em>logs</em>.</li><li>Generates self-signed certs based on the PROXY_TLS_MODE (tls|mtls). <strong>Note:</strong> <br> a. By default, Proxy operates in <em>TLS</em> mode. I would suggest using an mtls mode as it is a more secure way to establish the connection.<br> b. Proxy server can be accessed through a couple of DNS - <em>squid-proxy</em> and <em>host.docker.internal</em>.</li><li>A Docker image is built and runs as a Docker container.</li></ol><blockquote><strong>Note: <br> </strong>1.<strong> </strong>It is expected that Docker is already running on the host machine.<br> 2. We can check the logs of each request made by a client through the access.log present under data-proxy directory.</blockquote><pre>#!/usr/bin/env bash<br><br>set -xv<br>set -euo pipefail<br><br># Proxy Variables.<br><br>export PROXY_PORT=${PROXY_PORT:-3128}<br>export PROXY_IMAGE=${PROXY_IMAGE:-&quot;localhost:5000/squid-proxy:latest&quot;}<br>export PROXY_NAME=${PROXY_NAME:-&quot;squid-proxy&quot;}<br>export PROXY_TLS_MODE=${PROXY_TLS_MODE:-&quot;tls&quot;}<br><br># Knowing Current Path to create required directories.<br>BASE_DIR=$(realpath $(dirname &quot;$0&quot;))<br><br># Initializing required directories.<br><br>CERTS_DIR=&quot;$BASE_DIR/certs&quot;<br>PROXY_DATA_DIR=&quot;$BASE_DIR/data-proxy&quot;<br>LOG_DIR=&quot;$BASE_DIR/logs&quot;<br><br># Creating required directories.<br><br>mkdir -p &quot;$CERTS_DIR&quot;<br>mkdir -p &quot;$PROXY_DATA_DIR&quot;<br>mkdir -p &quot;$LOG_DIR&quot;<br><br>########<br># Create certs.<br><br>cd &quot;$CERTS_DIR&quot;<br><br># Create server cert.<br>openssl genrsa -out server.key 2048<br>openssl req -x509 -new \<br>    -subj &quot;/C=US/CN=squid-proxy&quot; \<br>    -addext &quot;subjectAltName = DNS:squid-proxy,DNS:host.docker.internal&quot; \<br>    -key server.key -out server.crt<br><br># Create client cert - It requires only when TLS mode is MTLS(mutual TLS).<br>if [[ $PROXY_TLS_MODE == &quot;mtls&quot; ]]; then<br>    openssl genrsa -out client.key 2048<br>    openssl req -x509 -new \<br>        -subj &#39;/C=US/CN=client&#39; \<br>        -key client.key -out client.crt<br>fi<br><br>cd -<br><br>########<br># Create container image.<br><br>docker build -t &quot;$PROXY_IMAGE&quot; &quot;$BASE_DIR&quot;<br><br>cd &quot;$PROXY_DATA_DIR&quot;<br># Copy Squid Conf based on TLS mode.<br>if [[ $PROXY_TLS_MODE == &quot;mtls&quot; ]]; then<br>    cp &quot;$BASE_DIR/squid.mtls.conf&quot; squid.conf<br>else<br>    cp &quot;$BASE_DIR/squid.tls.conf&quot; squid.conf<br>fi<br><br>install -m 644 &quot;$CERTS_DIR/client.crt&quot; client.crt<br>install -m 644 &quot;$CERTS_DIR/server.crt&quot; server.crt<br>install -m 600 &quot;$CERTS_DIR/server.key&quot; server.key<br><br>cd -<br><br>########<br>echo &quot;Starting proxy container&quot;<br>docker run --rm \<br>    -v &quot;$PROXY_DATA_DIR:/data:rw&quot; \<br>    -p &quot;$PROXY_PORT:3128&quot; \<br>    --name &quot;$PROXY_NAME&quot; \<br>    &quot;$PROXY_IMAGE&quot; \<br>    /usr/sbin/squid -N -f /data/squid.conf \<br>    &gt; &quot;$LOG_DIR/proxy-log.txt&quot; 2&gt;&amp;1 &amp;<br><br>echo &quot;Proxy at https://squid-proxy:$PROXY_PORT&quot;</pre><h3>More Information</h3><ol><li>We can restrict both client addresses (who can use the proxy) and destination addresses (where clients need to connect) using ACLs (Access Control Lists) in the squid.conf file</li><li>Refer <a href="https://hub.docker.com/r/ubuntu/squid">here</a> to launch the insecure mode of the squid proxy.</li><li>Do check out <a href="https://www.squid-cache.org/Doc/config/http_port/">here</a> for the Squid proxy parameters.</li></ol><h3>Conclusion</h3><p>To wrap up, We have reached the end of this post, and I hope you can set up either TLS or mTLS mode of forward proxy on your host machine. I always suggest you set up the mTLS mode of proxy as it offers more security and we should be more restrictive by adding the ACLs to constrain the clients and destination addresses.</p><h3>Future Considerations</h3><ol><li>Need to explore more about the usage of ACLs and use it with Squid proxy.</li><li>Running Squid within a Kubernetes (K8s) cluster.</li></ol><h3>References</h3><ul><li>Docker <a href="https://hub.docker.com/r/ubuntu/squid">Page</a> to launch Squid.</li><li>Squid <a href="https://www.squid-cache.org/">Page</a>.</li></ul><p>Thank You and Enjoy Learning!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc3b4e11578c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker: Multi-Arch Imaging]]></title>
            <link>https://medium.com/@vardhanreddy1228/docker-multi-arch-imaging-6ac81acf1d8b?source=rss-f58e913da9fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ac81acf1d8b</guid>
            <category><![CDATA[multi-architecture]]></category>
            <category><![CDATA[docker-build]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[multiarch]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Vardhan Devireddy]]></dc:creator>
            <pubDate>Sun, 09 Feb 2025 16:38:36 GMT</pubDate>
            <atom:updated>2025-02-09T16:43:49.048Z</atom:updated>
            <content:encoded><![CDATA[<p>This article focuses on utilizing the docker buildx imagetools create command for the below-mentioned purposes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEQZY_aDk5Be0N-dre2BmQ.png" /></figure><ol><li><strong>Combing</strong> different platform images(e.g., linux/amd64, linux/arm64, etc.) to have a single reference that points to the <em>multi-arch</em> image, helps to deploy the same image in a K8s cluster even though nodes are of different CPU architectures.</li><li><strong>Copying </strong>the<strong> </strong>open-source<strong> </strong><em>multi-architecture image</em> to a private registry for minimizing internet interactions from a private cloud environment to avoid pull rate issues.</li></ol><h3><strong>Combining Different Platform Images</strong></h3><p>The following steps give you a hands-on to combine different platform images into a single multi-arch image.</p><h4>Step 1: Create a Dockerfile</h4><pre>FROM ubuntu:latest<br><br>ARG TARGETARCH<br><br>COPY run.sh /run.sh<br><br>CMD [&quot;/bin/bash&quot;, &quot;/run.sh&quot;]</pre><h4>Step 2: Build a different platform image</h4><pre>docker buildx build --push --platform linux/amd64 \<br>--tag localhost:5000/testimage:amd64 .<br><br>docker buildx build --push --platform linux/arm64/v8 \<br>--tag localhost:5000/testimage:arm64 .</pre><h4>Step 3: Combine a multi-platform image</h4><ul><li>I suggest performing a <em>dry run</em> prior to directly creating a multi-arch image using:</li></ul><pre>docker buildx imagetools create --tag localhost:5000/finalimage:latest \<br>localhost:5000/testimage:arm64 \<br>localhost:5000/testimage:amd64 \<br>--dry-run</pre><ul><li>If all looks good as you expect then start combining different platform images. <strong>Note: </strong>This command is not restricted to combining only 2 source image manifests.</li></ul><pre># Commnad<br><br>docker buildx imagetools create --tag localhost:5000/finalimage:latest \<br>localhost:5000/testimage:arm64 \<br>localhost:5000/testimage:amd64<br><br>---<br><br># Reference for combining more than 2 source images.<br><br>docker buildx imagetools create --tag localhost:5000/finalimage:latest \<br>localhost:5000/testimage1:version \<br>localhost:5000/testimage2:version \<br>.<br>.<br>.<br>localhost:5000/testimagen:version \<br></pre><h4>Step 4: Verify the built image</h4><ul><li>We should expect to see the manifest list containing both linux/amd64 and linux/arm64 . <strong>Note: </strong>--format is the option to format the output template.</li></ul><pre>docker buildx imagetools inspect localhost:5000/finalimage:latest \<br>--format &quot;{{json .Manifest}}&quot;</pre><h3>Copying Of Multi-Arch Image</h3><p>The following steps give you a hands-on to copying a multi-arch image from the public/private registry and pushing it into the private/required registry:</p><pre># Command<br><br>docker buildx imagetools create \<br>--tag &lt;dest-registry&gt;/&lt;dest-image&gt;:&lt;dest-image-version&gt; \<br>&lt;source-registry&gt;/&lt;source-image&gt;:&lt;source-image-version&gt;<br><br>----<br># Example<br><br>docker buildx imagetools create \<br>--tag localhost:5000/finalimage:latest \<br>localhost:5000/testimage:latest</pre><h4>Conclusion</h4><p>To summarize, Hopes this article provided you with complete information and practical guidance on creating a multi-arch image from different platform images, as well as how to copy this multi-arch image into the destination registry.</p><p>Enjoy Learning!!!</p><h4>References</h4><p><strong>Docker Doc:</strong> <a href="https://docs.docker.com/reference/cli/docker/buildx/imagetools/">https://docs.docker.com/reference/cli/docker/buildx/imagetools/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ac81acf1d8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>