<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bhaskar Agarwal on Medium]]></title>
        <description><![CDATA[Stories by Bhaskar Agarwal on Medium]]></description>
        <link>https://medium.com/@a.bhaskar.is?source=rss-3a1273d509cf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*6YojQxc_Ttf3AV4LooN-cQ.jpeg</url>
            <title>Stories by Bhaskar Agarwal on Medium</title>
            <link>https://medium.com/@a.bhaskar.is?source=rss-3a1273d509cf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:47:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@a.bhaskar.is/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My journey from a data scientist to a product owner (Part 1/2)]]></title>
            <link>https://levelup.gitconnected.com/my-journey-from-a-data-scientist-to-a-product-owner-part-1-2-6bd91f065331?source=rss-3a1273d509cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bd91f065331</guid>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[business-strategy]]></category>
            <dc:creator><![CDATA[Bhaskar Agarwal]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 15:11:35 GMT</pubDate>
            <atom:updated>2025-12-18T12:49:20.154Z</atom:updated>
            <content:encoded><![CDATA[<h4>5 Key Lessons from transitioning to a tech-lead role</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HIrS3ZN4t_Lmlc9H" /><figcaption>Photo by <a href="https://unsplash.com/@lunarts?utm_source=medium&amp;utm_medium=referral">Volodymyr Hryshchenko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>“You’re over-explaining again.”</em></p><p>This became the refrain of many meetings as I stepped into a more hybrid product owner role. As a senior data scientist with a background rooted in academic research, my instinct was to explain <em>everything</em> right from the methodology to the caveats, from the assumptions to the modeling decisions. But stakeholders rarely wanted the whole story. They wanted impact. In retrospect, I get it. But it took some unlearning.</p><p>Here are five key lessons that helped me navigate the transition from senior data scientist to hybrid product owner: something that blends delivery with direction, and code with context.</p><h3>1. Research Trains You for Depth, Not Brevity</h3><p>As a researcher, the goal was clarity through complexity. Every assumption was justified, every limitation explored. That mindset carried over into my early data science roles where I found myself obsessing over precision, making sure the models could defend themselves in court, if it came to it.</p><p>However, stakeholders don’t want a courtroom; they want a dashboard, or better yet a number. <em>The number</em> that tells them if they’re winning or losing.</p><p>The shift to product ownership forced me to compress information. Communicate like a billboard, not a thesis. It was painful at first, like throwing away hours of work. But eventually, I realized: the work still matters. It’s just not always the headline.</p><h3>2. One KPI to Rule Them All</h3><p>This was the hardest part: reducing a rich, multi-layered analysis into a single metric. It felt like betrayal. Like condensing a novel into a tweet.</p><p>But over time, I understood the power of that single KPI. It wasn’t just about summarizing the work but about <em>championing</em> it. A good KPI becomes a proxy, a torchbearer. It shows up in boardrooms, dashboards, and decisions. It carries your work forward even when you’re not in the room.</p><p>At first, I resisted it. It felt too reductive, as if it invalidated all the nuance we’d worked so hard to capture. But eventually, I realized: it doesn’t replace the work. It <em>represents</em> it.</p><p>Now, I still go deep. But I lead with one line. One number. One outcome that matters to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WiBI0mxeWOab-WEjxxzkbQ.png" /><figcaption>source: generated with ChatGPT</figcaption></figure><h3>3. Build It First, Understand It Better</h3><p>Before I ever led a team or pitched a roadmap, I built stuff. Real, functioning MVPs ranging from Dash apps, Selenium workflows, Dockerized microservices running on custom VMs etc.. Not because I had to but because I wanted to understand the whole process.</p><p>I spun up cloud instances, configured environments, wrestled with containerization, and for the first time <em>really</em> understood what it meant when someone said “make sure port xyz is open.” I learned how APIs communicate, how services expose ports, how reverse proxies actually work.</p><p>This hands-on experience made product ownership easier later. I wasn’t guessing timelines, I had lived them. I didn’t just say “we’ll need monitoring here,” I knew what that looked like in logs. I could advocate for scrappy MVPs because I’d built them with their flaws, duct tape and all.</p><p>If I have any advice for aspiring product owners from a DS background: build it once. End-to-end. Not for perfection, but for perspective.</p><h3>4. Product Thinking Starts With the Outcome</h3><p>In data science, the finish line is often the model. In product, it’s the outcome. This simple difference reshaped how I approached my work.</p><p>When I think like a product owner, I ask different questions:</p><ol><li>Who uses this?</li><li>How often?</li><li>What changes because of it?</li><li>What’s the cost of getting it wrong?</li></ol><p>That shift doesn’t just help users but it also sharpens the science. When you anchor your models in user impact, you naturally prioritize interpretability, robustness, and resilience. You stop chasing marginal accuracy and start delivering actual value.</p><h3>5. Sometimes the Real Product is… a Spreadsheet</h3><p>One of the biggest mindset shifts was learning to separate <em>what I thought was valuable</em> from <em>what the user actually needed</em>. I learned this the hard way on my first internal product in a corporate setting.</p><p>We built an elegant dashboard. Real-time filters, clean visuals, click-throughs, you name it we had it! So what got the most love?</p><p>A filterable Excel export.</p><p>That was the “real” product for many internal users. Something they could manipulate, annotate, blend with other datasets, and use in high-stakes meetings. It enabled action. Not just analysis.</p><p>That moment changed how I think about “deliverables.” Because the best product isn’t the one you’re proudest of but it’s the one people use.</p><p>Stay tuned for <a href="https://levelup.gitconnected.com/my-journey-from-a-data-scientist-to-a-product-owner-part-2-2-5ad5ace3ed7b?postPublishedType=repub"><strong>Part 2</strong></a>, where I’ll talk about the hard pivots, blind spots, and new muscles I had to build especially around cross-functional leadership, stakeholder alignment, and letting go of the urge to optimize everything.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bd91f065331" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/my-journey-from-a-data-scientist-to-a-product-owner-part-1-2-6bd91f065331">My journey from a data scientist to a product owner (Part 1/2)</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My journey from a data scientist to a product owner (Part 2/2)]]></title>
            <link>https://levelup.gitconnected.com/my-journey-from-a-data-scientist-to-a-product-owner-part-2-2-5ad5ace3ed7b?source=rss-3a1273d509cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ad5ace3ed7b</guid>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[business-strategy]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[careers]]></category>
            <dc:creator><![CDATA[Bhaskar Agarwal]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 15:11:32 GMT</pubDate>
            <atom:updated>2025-12-18T12:48:40.313Z</atom:updated>
            <content:encoded><![CDATA[<h4>5 Key Lessons from managing a cross-functional tech team</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WKVBmHVerzxZS8LT" /><figcaption>Photo by <a href="https://unsplash.com/@jannerboy62?utm_source=medium&amp;utm_medium=referral">Nick Fewings</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The shift from building things solo to leading a team that builds together is both liberating and terrifying. Suddenly it’s not just about your velocity but alignment, delegation, motivation, and momentum.</p><p>When I took on my first product build as a hybrid product owner, I was lucky to work with a sharp, cross-functional team:</p><ul><li>1 front-end designer with a pre-existing design system</li><li>1 senior solution architect</li><li>1 data scientist</li><li>1 back-end engineer</li></ul><p>We had one common goal but diverse backgrounds and areas of expertise. Now, it was my job to bring it all together . The aim was not by forcing the team to follow me blindly, but by creating the structure and clarity needed for everyone to move in sync. That meant defining just enough to get us started, stepping back to let people lead in their own domains, and stepping in when things got stuck. It also meant owning the un-glamorous parts, i.e. writing docs, pre-empting bugs, clarifying priorities, and sometimes just asking the dumb (but necessary) questions.</p><p>What follows are five key lessons from that journey. Lessons in trust, simplicity, communication, and course correction.</p><h3>1. Document Everything — It Still Won’t Be Enough (and that’s fine)</h3><p>Before we started, I documented everything right from the concept, the logic, to the MVP code. I broke down expected flows, edge cases, and example data structures. But one of the first pieces of feedback I got was from our lead front-end dev, someone with many years of experience, was:</p><p><em>“It’s not your job to make the perfect documentation for the front end. Just tell us what you want in enough detail so we can work from it.”</em></p><p>That stuck with me.</p><p>Because the truth is: <strong>it’s never enough</strong>. It never is. That’s not even the goal. Documentation isn’t about perfection, it’s about reducing ambiguity enough to get things moving.</p><p>Yes, there were gaps. Yes, questions came up. But having that base helped everyone ramp up faster, and gave us a shared mental model to work from. Over time, it became the artifact we rallied around: not because it was exhaustive; but because it <em>existed</em>.</p><h3>2. Start Simple, but Think Long-Term</h3><p>We began lean. Minimal scope. No unnecessary complexity. Just a functional version we could build on.</p><p>But I was always clear about the longer-term arc, i.e. where we might go, what the product could grow into, how our sprints aligned to that vision.</p><p>That mix of <strong>keep it simple now, but purposeful overall</strong> gave us momentum <em>and</em> direction.</p><p>By the time the team had on-boarded fully, they weren’t just executing. They were making smart decisions in-context, because they knew what we were building <em>towards</em>.</p><h3>3. Empower the Team and Explain the Why</h3><p>Once everyone was rolling, I shifted from directing to enabling. I set up the initial Jira flow and seeded a few foundational issues. By the time we shipped V1, the team had multiplied it into 50+.</p><p>That didn’t happen because I micromanaged. It happened because I spoke to people’s strengths, let them lead their verticals, and gave them room to build.</p><p>But there’s one piece that made it all click: <strong>I always explained the why</strong>.</p><ol><li>Why we’re building this feature.</li><li>Why this timeline matters.</li><li>Why this is the right thing to prioritize right now.</li></ol><p>Once people understand the mission, they don’t just build what’s asked but they improve it. They see problems early. They propose better paths. You stop being the only one steering.</p><h3>4. Build Trust by Staying Technical (and Human)</h3><p>I never pretended to know everything. Especially not about complex systems or scalable infrastructure. I asked questions. I admitted what I didn’t know. And that vulnerability helped.</p><p>But what helped even more? Knowing <em>enough</em> about everything to get conversations unstuck.</p><p>At one point, the backend and frontend teams were misaligned on how to handle a piece of data. I didn’t dive into code, I just asked a few targeted questions about the config file structure and how the metadata was being passed around. That was enough to get the right people talking, in the right direction.</p><p>That kind of context-switching or in other words being able to move between design, dev, data, and delivery is where hybrid roles become fun.</p><p>And it goes beyond coordination. Ten days before delivery, we hit a major bug with the mathematical logic. The lead dev was fried. I tried stepping in, debugged for a day, and resolved it. The fix stuck.</p><p>That earned trust. Not because I knew everything, but because I <em>showed up</em> when it mattered. With clarity, context, and sleeves rolled up.</p><h3>5. Don’t Be Afraid to Course-Correct (Even If It’s Uncomfortable)</h3><p>Mid-sprint, an external senior stakeholder went deep into a visual redesign tangent.</p><p>Interesting? Sure.</p><p>Critical? Not even close.</p><p>Meanwhile, the rest of the team was drowning in blockers. So I had to step in.</p><p><em>“This isn’t the sprint for that. Let’s table it.”</em></p><p>It wasn’t the most fun call to make but it was the right one. Here’s where that early documentation came through again. A few times, when issues were forgotten or sidelined, I could pull up the original spec:</p><p><em>“This has been in scope from the start. We’re not improvising — we’re just remembering.”</em></p><p>That kind of paper trail turns conflict into course correction. It’s not about being right but about being aligned.</p><p><strong>In the end, managing a cross-functional team isn’t about knowing everything, it’s about creating the clarity, context, and confidence that lets everyone do their best work. That’s the real product.</strong></p><p>Leadership in a hybrid role isn’t about having all the answers. It’s about seeing the whole board. Knowing which pieces move when, and why. Knowing when to step in, and when to get out of the way.</p><p>Sometimes your biggest contribution is writing the first line of code.<br> Sometimes it’s just creating the space where great code gets written.</p><p>Hope you enjoyed reading about my journey! Stay tuned for more and feel free to reach out to me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ad5ace3ed7b" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/my-journey-from-a-data-scientist-to-a-product-owner-part-2-2-5ad5ace3ed7b">My journey from a data scientist to a product owner (Part 2/2)</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Reduce the Training Time of Your Neural Network from Hours to Minutes]]></title>
            <link>https://medium.com/data-science/how-to-reduce-the-training-time-of-your-neural-network-from-hours-to-minutes-fe7533a3eec5?source=rss-3a1273d509cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe7533a3eec5</guid>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Bhaskar Agarwal]]></dc:creator>
            <pubDate>Fri, 06 May 2022 05:07:50 GMT</pubDate>
            <atom:updated>2022-07-01T06:24:11.903Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2 of the articles on AI with HPC: parallelising a CNN with Horovod and GPUs to obtain a 75x-150x speed-up.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fSWbNI1zIkE25lUl" /><figcaption>Photo by <a href="https://unsplash.com/@ro_ka?utm_source=medium&amp;utm_medium=referral">Robert Katzki</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://towardsdatascience.com/machine-learning-on-supercomputers-data-processing-aa403e51516b">part 1</a> of the series we looked at how it is possible to get a <strong>~1500x speed-up in IO operations</strong> with a few lines of Python using the multiprocessing module. <strong>In this article, we will look at parallelising a deep learning code and reducing the training time from roughly 13 hours to 13 minutes!</strong></p><h3>The Context</h3><p>As a data scientist, you will eventually face the following problem (if you haven’t faced it already).</p><p><strong><em>“I have a neural network to train but the input data doesn’t fit in memory!” </em></strong>or<strong><em> “My neural network takes forever to train with this amount of data!”</em></strong></p><p>It would surely be a pity to exclude a substantial part of your data for training, or wait for hours (even days) for your neural network to finish training. There has to be a better way and there is, it is called <strong>data parallelism</strong>.</p><h3>The Problem</h3><p>For this post, we will use a modified version of the Pepsico-Lab potato chips control dataset available on <a href="https://www.kaggle.com/datasets/concaption/pepsico-lab-potato-quality-control">Kaggle</a>. The original images are in a 2000 x 2000 pixel format with 3 channels. I modify the images by scaling them to 500 x 500 and applying 5 transformations (a vertical flip, a horizontal flip and three 90 degree rotations) to create 6 times more images than in the original dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O57Y6_w4SYSVSa4Km5hkVA.png" /><figcaption>The two types of chip to distinguish in this post.</figcaption></figure><p>I then wrote a simple convolutional neural network (CNN) code for classification based on the input data I created from the original dataset. All the material can be found <a href="https://github.com/codiusmaximus/chips_cnn">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*sIyp5jTkVm0gsQhCq3tQiw.png" /><figcaption>The CNN used in this post visualised using <a href="https://github.com/paulgavrikov/visualkeras">visualkeras</a>. Note that the first layer is not to-scale for the sake of aesthetics, but it gives an idea of the tiny CNN I created for this problem.</figcaption></figure><h4>The resources and the repository</h4><p>For this article we will be using:</p><ul><li>TensorFlow2 and Keras</li><li>Horovod</li><li>Python 3.8+</li><li>A computer with at least 1 GPU</li><li>All the codes and data can be found in this <a href="https://github.com/codiusmaximus/chips_cnn">repository</a>.</li></ul><h3>Data Parallelism</h3><p>In data parallelism, one deploys the same model to be trained on different processing units (think GPUs) but with a unique subset of the input data. After every epoch, the model parameters are collected, aggregated and communicated back to the processing units to continue the training of their respective model. This way, each processing unit trains the same model with its own unique subset of the data (thus a fraction of it), but the model parameters are updated with an insight to the entire input dataset.</p><h4>How can I use this approach with neural networks?</h4><p>If you use TensorFlow, PyTorch or any of the popular public deep learning frameworks, you can use <a href="https://github.com/horovod/horovod">Horovod</a> for distributed training. Horovod was developed by Uber in order to bring the ideas of HPC to deep learning. Installation is fairly straight-forward and I won’t go into details as their <a href="https://github.com/horovod/horovod">GitHub page</a> is excellent in explaining the same. For the remainder of this post, we will talk about how one can data parallelise their TensorFlow code using Horovod.</p><p>Think of Horovod as a plug-in to your existing code. The steps involved in <em>Horovod-ising </em>your code follow the idea of data-parallelism, so before anything else you must fully understand how your code will be executed with Horovod. You muse then decide on a way to distribute your data. Once this is done, you can start implementing Horovod in your code.</p><p>To me, it is easier to approach this subject in the following steps.</p><ol><li>Understand how your code will be executed with Horovod, i.e. <em>horovodrun</em></li><li>Make sure you have a good data distribution strategy</li><li>Intialise Horovod in your code</li><li>Make sure your optimizer is distributed</li><li>Broadcast your variables from the coordinator and fix your callbacks</li></ol><p>We will now discuss each of these points in some detail.</p><h3>1. How Horovod operates</h3><p>Horovodrun wraps some essential message passing interface (MPI) commands and concepts into a higher level API. Thus, under the hood one can find a lot of similarities between the two if they are familiar with MPI. On a system with <em>n</em> GPUs one would execute a CNN code, where Horovod has been implemented, as</p><pre>horovodrun -np <strong>n</strong> python cnn_parallel.py</pre><p>Codes that have been modified with Horovod need to be executed with either <em>horovodrun </em>or <em>mpirun</em>. For the data parallelism framework, with <em>n </em>GPUs one would create n tasks. Each task is running your modified CNN code with python but on a unique subset of the data. In the image below I explain the same. We are working on a machine with 2 GPUs. The input dataset D is split into two subsets a and b such that there is no element in common between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*0XFnT76gBBqK7T1kCfSUQA.png" /><figcaption>Understanding how Horovod executes your code in parallel.</figcaption></figure><p>The important thing to remember here is that your entire python code will be run <em>n</em> times, where n is the number of GPUs. Thus every print command, every IO operation, every plotting section etc. will be executed n times. This can be a confusing concept at first but the good part is that every on-screen message (e.g. print(…)) comes with its own GPU identifier.</p><p>Horovod comes with some useful calls (similar to the ones defined by the MPI standard) that help define the architecture of the system on which we will be running the code. The ‘hvd.size()’ command displays how many GPUs are actually available for use, the ‘hvd.local_rank()’ tells you the local rank of the GPU, and the ‘hvd.rank()’ tells you the global rank of the GPU. The rank can be thought of as an id for that GPU. The image below explains the idea better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*lmhZiB-ILyUTo2-jG21sRg.png" /><figcaption>Understanding the size and ranks of GPUs in single node vs. multi-node setup.</figcaption></figure><h3>2. Data distribution strategy</h3><p>So why must you distribute the data? In order to make full use of the data parallelism, each GPU must see a unique subset of data. However if you pass the same dataset for training, you are training a model n times with the same samples on n GPUs, or in other words you are running your code in exactly the same way with the same data n times. This is not what we want to achieve with data parallelism.</p><p>There are many strategies one can adopt in order to distribute the data correctly. There are inbuilt data distribution methods in Pytorch, Tensorflow etc., but the actual strategy will depend on how your data is structured and the amount of it.</p><p>Let us create some data distribution strategies with a simple example. Assume that our data comprises of all integers from 1 -100 in increasing order without repetition. We want to distribute this data on 2 GPUs.</p><h4><strong>Option 1: TensorFlow Data Generator</strong></h4><p>We will employ the idea of distributing batches on GPUs randomly at the start of training. We will trick the fitting call in TensorFlow to assign 1/n batches to each of the GPUs using the ‘steps_per_epoch’ keyword. This method works well with a data generator, in our case, the image data generator. We keep the same batch size as we had when we were training without any GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*ooLXWb5tDKFeOtwtDTHeZg.png" /><figcaption>Data distribution with a TensorFlow data generator.</figcaption></figure><pre>img_gen = tf.keras.preprocessing.image.ImageDataGenerator(<br>                                           validation_split=split)</pre><pre>train_dataset = img_gen.flow_from_directory(<br>                           directory= train_data_path,<br>                           target_size=(img_height,img_width),              <br>                           color_mode=<strong>&#39;rgb&#39;</strong>,<br>                           batch_size=<strong>batch_size</strong>,<br>                           class_mode=<strong>&#39;sparse&#39;</strong>,<br>                           seed=123, <br>                           shuffle=True, <br>                           subset=<strong>&#39;training&#39;</strong>)</pre><pre>val_dataset=img_gen.flow_from_directory(directory=...  <br>                           subset=<strong>&#39;validation&#39;</strong>)</pre><pre><strong>total_train_batches</strong> = train_dataset.samples//batch_size</pre><pre><strong>total_val_batches</strong> = val_dataset.samples//batch_size</pre><p>Then when we perform the fitting, we instruct TensorFlow to only use 1/n of the total batches for training and validation. This is because the training will happen on each GPU, and by specifying the ‘steps_per_epoch’ as total_batches/n, we instruct TensorFlow to split the batches on the two GPUs such that each GPU gets a disjoint subset of batches from our data generator.</p><pre>history = model.fit(<br>    train_dataset,<br>    epochs=epochs,<br>    batch_size=batch_size,<br><strong>    steps_per_epoch=total_train_batches//hvd.size(), </strong><em><br>    </em>validation_data=val_dataset,<br><strong>    validation_steps=total_val_batches//hvd.size(), </strong><em><br>    ...</em><br>)</pre><h4><strong>Option 2: TensorFlow data.Dataset object</strong></h4><p>We will employ the concept of ‘sharding’ which is equivalent to taking every q-th element in a dataset. The pseudo-code below should explain it better.</p><pre>D = [1,2,...,99,100]</pre><pre># split into 2 shards<br>shard_1 = [1,3,5,...]<br>shard_2 = [2,4,6,...]</pre><pre># split into 3 shards<br>shard_1 = [1,4,7,...]<br>shard_2 = [2,5,8,...]<br>shard_3 = [3,6,9,...]</pre><p>Below I describe 2 different ways for the same using a batch size of 4. Note that these are not the only strategies one may adopt to distribute the data and you are free to devise your own.</p><ul><li>method A: shard the dataset 2 times → batch it</li><li>method B: shuffle the dataset → shard the dataset 2 times → batch it</li></ul><p>From a machine learning point of view, can you reason why only one of these might be the right way?</p><pre>D = [1,2,...,99,100]</pre><pre><strong># method A</strong></pre><pre>shard_1 = [1,3,5...]<br>shard_2 = [2,4,6...]</pre><pre>shard_1_batch = [[1,3,5,7],[9,11,13,15]...] # split 1<br>shard_2_batch = [[2,4,6,8],[10,12,14,16]...] # split 2</pre><pre><strong># method B</strong></pre><pre>D_shuffle = [63,74,22,26,36,14,34,94,60,42,56,17,65,1,12,51...]</pre><pre>shuffle_shard_1 = [63,22,36,34,60,56,65,12]<br>shuffle_shard_2 = [74,26,14,94,42,17,1,51]</pre><pre>shuffle_shard_1_batch = [[63,22,36,34],[60,56,65,12]...] # split 1<br>shuffle_shard_2_batch = [[74,26,14,94],[42,17,1,51]...] # split 2</pre><p>With method A, all odd numbers end up in split 1 and even numbers up in split 2. Clearly, method B is the best as the resulting splits have a fair representation (and distribution) of even and odd numbers.</p><p>Now think about the input for a CNN. The input folder would ideally contain thousands (if not millions) of images that you need to train on, generally grouped into different classes (sub folders). When you create a TensorFlow dataset from a folder of images, it infers the classes from the directory structure. The snippet below should clarify the point better.</p><pre><strong>input_folder<br>|-train<br>| |-class_1</strong><br>| | |-image_1.jpg<br>| | |-...<br>| | |-image_100.jpg<br><strong>| |-class_2</strong><br>| | |-image_1.jpg<br>| | |-...<br>| | |-image_50.jpg<br><strong>| |-class_3</strong><br>| | |-image_1.jpg<br>| | |-...<br>| | |-image_500.jpg</pre><p>Thus, once such a folder structure has been passed to TensorFlow, it automatically maps the filenames to their respective class, ready to be processed and passed to the CNN for training. <strong>Note that by default, </strong><em>image_dataset_from_directory</em><strong> shuffles the filenames, or in other words your dataset is already shuffled if you use this to ingest the images.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*lcxZEEcPimCv3Tam-8BxHQ.png" /><figcaption>Data distribution using sharding.</figcaption></figure><pre>import tensorflow.keras.preprocessing as tkp</pre><pre>local_bs = .. # will be scaled with the hvd.size() <br>              # we will discuss this later</pre><pre>X_train = tkp.image_dataset_from_directory(..., batch_size = 1) </pre><pre>train_dataset = X_train.<br>                        <strong>unbatch()</strong>.<br>                        <strong>shard(hvd.size(), hvd.rank()).<br>                        batch(local_bs).<br>                        cache()</strong></pre><p>The <em>hvd.shard() </em>call takes the number of GPUs (how many shards to create) and the <em>id</em> of the GPU to create a unique shard of the data. Using hvd.size() and hvd.rank() as inputs to this call, we can easily manage the data splits that will be processed by each GPU. Till this point, <em>train_dataset </em>has not been created and thus does not exist in memory. It will be created when one finally trains the model with the command below.</p><pre>model.fit(train_dataset,...)</pre><p>Note that here we used the <em>local_bs, </em>instead of the original batch size. I will explain the reason shortly.</p><h4><strong>Training the model after data distribution</strong></h4><p>Once we have decided and implemented the data distribution strategy correctly, we are ready to train the CNN. At this point, with 2 GPUs, the data training strategy will be put into execution as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8o1-3zj2td-ZD9D3LyXqA.png" /><figcaption>The data distribution strategy being executed when the CNN is trained.</figcaption></figure><h4>Batch size &amp; learning rate (Option 2 in data distribution strategy)</h4><p>If you have used a neural network (NN), you must be familiar with the concept of learning rate and batch size. In a data distributed framework one might need to scale these correctly in order to compensate for the change in our training strategy. Since each GPU is now training the model with a unique subset of data, the number of samples the NN now <em>effectively </em>processes during every epoch is not the same as the batch size. Note that you might not need to perform this scaling depending on how you decide to distribute your data and how your code is written. In my experience, the Option 2 above for data distribution generally works better if one scales the the batch size and/or the learning rate accordingly. Basically in case you find strange loss and accuracy curves after implementing data parallelism, then this might be something to keep in mind.</p><p>Let us go back to Option 2 for data distribution. We have 2 GPUs and the dataset comprises of 100 integers with a batch size of 4. Each GPU gets 50 unique integers grouped into batches of size 4. Each GPU carries out training by processing its own 13 batches (last batch will have only 2 elements) and after every epoch, the model parameters are passed to Horovod for aggregation. Horovod aggregates model parameters from GPU 0 with a batch size of 4 and GPU 1 with a batch size of 4, so <em>effectively</em> the batch size is 8! In other words, when Horovod updates the model parameters at the end of every epoch, the model has technically ‘seen’ a batch where each batch had 8 unique samples. The image below should explain the point better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFA0IVTE9X7DUCWtaNoABQ.png" /><figcaption>Data distributed training of the CNN with two GPUs. At the end of each epoch, Horovod (orange) aggregates the model parameters from each GPU (teal and fuchsia) and updates the CNN model, now ready for training in the next epoch.</figcaption></figure><p>In the case where we do not change the batch size, i.e. keep it fixed to the same value as in the non data distributed version of the code, we must scale the learning rate linearly* with the total number of GPUs. This is because with more samples in our <em>effective batch</em>, one can increase the size of the step that the algorithm is allowed to take in the process of finding the minimum of the loss function. In other words, we have less noise (more samples) so we can afford a larger jump in the parameter space in order to find the minimum.</p><p>If we do however wish to keep the learning rate fixed, i.e. the same as in the non data distributed version of the code, we must scale the batch size inversely with the number of GPUs. Let us take our case of integers. With 2 GPUs, if I batch each shard of my data with a batch size of 2, then the <em>effective batch size</em> that Horovod sees at the end of each epoch is 4. Thus, I can maintain the same learning rate as I had before data distribution.</p><p>To summarise:</p><pre>############################################ THEORY<br># original batch size: batch_size<br># local batch size: local_bs (batch size on each gpu)</pre><pre>#original learning rate: lr<br>#scaled learning rate: scaled_Llr</pre><pre>## option 1: <br># bs= local_bs : therefore, the effective batch size is larger<br># therefore scaled_lr = lr*n_gpus</pre><pre>## option 2:<br># scaled_lr = lr<br># therefore local_bs = batch_size/n_gpus <br># now the effective batch size will be the same as before<br>############################################</pre><pre>############################################ IMPLEMENTATION<br>#<strong>pick one</strong> below:<br><strong># Option 1:</strong><br># local_bs = batch_size<br># scaled_lr = base_lr * hvd.size()<br># <strong>OR<br># Option 2:<br></strong>local_bs = int(batch_size/hvd.size())<br>scaled_lr = base_lr</pre><p>*<em>sometimes instead of a strictly linear scaling, a square-root scaling, or a power 2/3 scaling of the learning rate works better</em></p><h3>3. Initialising Horovod in your code</h3><p>The first step is telling your code you will be using Horovod.</p><pre><strong>import horovod.tensorflow as hvd</strong></pre><pre><strong>hvd.init()</strong> # this initiliases horovod</pre><pre>print(&#39;Total GPUs available:&#39;, hvd.size())<br>print(&#39;Hi I am a GPU and my rank is:&#39;, hvd.rank())</pre><p>The next step is to pin each available GPU to a ‘process’ using the GPU’s local rank as an identifier.</p><pre><em># Pin GPU to be used to process local rank (one GPU per process)</em><br>gpus = tf.config.experimental.list_physical_devices(&#39;GPU&#39;)<br>for gpu in gpus:<br>    tf.config.experimental.set_memory_growth(gpu, True)<br>if gpus:<br>  tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], &#39;GPU&#39;)</pre><h3>4. Distribute the optimizer</h3><p>This is where the aggregation of the optimizer takes place. Horovod makes this very easy as we only need to add one line to our code to distribute the optimizer.</p><pre>opt = keras.optimizers.Adam(learning_rate=learning_rate)</pre><pre># Horovod: add Horovod Distributed Optimizer.<br><strong>opt = hvd.DistributedOptimizer(opt)</strong></pre><h3>5. Fix the call backs</h3><p>The next step is to make sure we do not corrupt our training by allowing all GPUs to update the model parameters. Thus we declare one of our GPUs as the ‘coordinator’. This GPU will be responsible for collecting the model parameters, aggregating them and broadcasting them back to the other GPUs after every epoch of training. Remember that it will also be training the CNN with its own subset of the data. Usually we declare GPU with rank 0 to be our coordinator.</p><pre>callbacks = [<br>    # Horovod: broadcast initial variable states from rank 0 to<br>    # all other processes. This is necessary to ensure consistent<br>    # initialization of all workers when training is started with<br>    # random weights or restored from a checkpoint.<br><strong>    hvd.callbacks.BroadcastGlobalVariablesCallback(0),</strong><br><br>    # Horovod: average metrics among workers at the end of every<br>    # epoch.<br><strong>    hvd.callbacks.MetricAverageCallback(),</strong><br>    ]</pre><pre># Horovod: save checkpoints only on worker 0 to prevent other<br># workers from corrupting them.<br><strong>if hvd.rank() == 0:<br>    callbacks.append(keras.callbacks.ModelCheckpoint(&#39;...h5&#39;))</strong></pre><h3>Running the code</h3><p>We are now ready to run the code. If you are using a computer with just 1 GPU, you may execute the command from a terminal as</p><pre>horovodrun -np 1 python python_code/horovod_dist_gen.py</pre><p>If instead we were to run it on a supercomputer, one would need to write a script that submits our ‘job’, where a job is a piece of code to be run with a given combination of resources. Below is an example of a script where I run the code on a machine with 4 GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LbjKLZl_7PwyJB-tHTmsRQ.png" /><figcaption>The slurm script to run the CNN code using GPUs with <em>Horovod</em>.</figcaption></figure><h3>What is the result?</h3><p><strong>Time with no GPU: 14 hours</strong></p><p><strong>Time with 1 GPU: 12 minutes</strong></p><p>For data distribution approach using shards,<strong> we obtain a 75x speed up using 1 GPU</strong> and <strong>150x scale up using 2</strong> or more GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*2oJtAW3Bcu03HbZRfrfLTA.png" /><figcaption>Speed up obtained after implementing Horovod in our CNN code and running it with 1, 2 and 4 GPUs.</figcaption></figure><p>Data distributed as shards performs better as compared to the generator approach. The speed-up from 1 GPU to 2 GPUs is almost linear, i.e. 75x to 150x, but after that there is no more speed-up. Why is that? It is because when we parallelise a code, the idea is to maximise the usage of resources available. With 1 GPU, even if the code trains 75 times faster as compared to the run with no GPUs, there is still scope for benefiting from more computational power. When we add the second GPU, the run-time halves thus leading to the ideal linear speed-up. At this point we are at a stage where the resources provided to the code are optimal for the amount of computation. Adding more resources will not benefit us. Thus we see no more speed up with 4 GPUs. We can monitor the resource usage (in our case the GPU usage) using the command <em>nvidia-smi. </em>I will not delve further into this topic but if you are interested feel free to contact me.</p><h3>Final Remarks</h3><ul><li>Feel free to devise your own data distribution strategy. It will often depend on how your data is organised and the sheer size of it.</li><li>When using GPUs, there is always an overhead for the initial set-up, i.e. depending on the system you are using there might be some time needed for setting up the GPU-CPU communication, environment variables etc. This is done by the system under the hood, so if at the start the code seems to stall for a few seconds it most likely is just setting up the GPUs.</li><li>For the sake of simplicity, I only time the training part. In principle one might need to benchmark the run-time for the entire code. It depends on what you are trying to optimise.</li><li>In order to benchmark the performance correctly, one would need to run the same code multiple times to get a better statistic of the run-time.</li><li>One must also check the convergence of the model, i.e. the parallelised training with GPUs should converge to the same loss and accuracy values as seen in the run without GPUs. Furthermore, regardless of the number of GPUs used the accuracy and loss curves should converge to the same values.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe7533a3eec5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-reduce-the-training-time-of-your-neural-network-from-hours-to-minutes-fe7533a3eec5">How to Reduce the Training Time of Your Neural Network from Hours to Minutes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I reduced data processing time from 5 days to 5 hours]]></title>
            <link>https://medium.com/data-science/machine-learning-on-supercomputers-data-processing-aa403e51516b?source=rss-3a1273d509cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa403e51516b</guid>
            <category><![CDATA[supercomputer]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Bhaskar Agarwal]]></dc:creator>
            <pubDate>Tue, 15 Mar 2022 13:08:14 GMT</pubDate>
            <atom:updated>2022-04-01T12:58:10.126Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1 of the articles on AI with high-performance computing: data processing in parallel</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tt95A9phyIh-6r0T6ZwTOg.jpeg" /><figcaption>Image courtesy: <a href="https://www.flickr.com/photos/cineca/25618574166">CINECA</a>.</figcaption></figure><p>In this series of articles, I will outline various ways one can leverage high-performance computing (HPC) in developing faster and more precise machine learning (ML) projects.</p><p>As a data scientist in the HPC division of <a href="https://www.cineca.it/">CINECA</a>, I was tasked with the problem of day-ahead hail prediction for the EU-funded <a href="https://www.cybele-project.eu/">CYBELE</a> project. The purpose of this project was to develop a hail prediction model to be integrated in the Cybele Demonstrator 3, which was a result of a collaboration between CINECA, <a href="https://gmv.com/en-es">GMV</a>, and <a href="http://www.cooperativesagroalimentariescv.com/">CACV</a>- a farmer federation based in Valencia that provided the user requirements and the historical observational data. The work presented in these posts was done with my colleagues at CINECA, namely Giorgio Pedrazzi, <a href="https://it.linkedin.com/in/roberta-turra-3897826">Roberta Turra</a>, <a href="https://it.linkedin.com/in/gian-franco-marras-27442217">Gian Franco Marras</a> and <a href="https://it.linkedin.com/in/michele-bottazzi-phd-a0a28990">Michele Bottazzi</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyIO-1qJPdZFPWGaEnjwnA.png" /><figcaption>Overview of the prediction region in red and historical hail events in black. Image courtesy: author.</figcaption></figure><p>Hail is a phenomenon that is likely to occur only when specific atmospheric conditions are met. This uncertainty makes the hail event rare and very hard to predict. Our approach was aimed at discriminating hail events from non-hail events, given the weather conditions, by learning from real events that occurred in the past. Thus, our hail prediction model was built based on automatic learning techniques, using the weather forecast data and the derived climate instability indices as input, and performing<br>validations against hail event data collected in the field.</p><h4>The Problem</h4><p>The input data was provided by <a href="https://www.ecmwf.int/en/forecasts/datasets/set-i">ECMWF</a> at a temporal resolution of 1 hour and spatial resolution of 5 Km covering around 200000 square kilometres in Spain. The data consisted of 73 fields describing the weather conditions for each grid point (described by its longitude and latitude) for each hour of the day, where some fields were computed at 65 different levels above the ground. Thus, the resulting number of variables was ~ 1000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cSRH3dkqReKgLzomwf6sAQ.png" /><figcaption>Overview of a GRIB file in our raw dataset. Image courtesy: author.</figcaption></figure><p>The input data needed to be downscaled to a 2.2 Km resolution in order to match the extension of the agricultural parcels and the crops grown in the target region of Spain. The finer grid of 205 x 333 ~ 68000 grid points was obtained by small scale modelling simulations, adapted to the area of interest, that required 640 cores (20 nodes) on an HPC machine, consuming 128 core hours for each forecast. Two forecasts (48 hours and 24 hours in advance) were run for each day in the historical data, provided by ECMWF, from 2015-2019 amounting to roughly 2100 days. This task was managed by my colleagues at CINECA, namely <a href="https://it.linkedin.com/in/gian-franco-marras-27442217">Gian Franco Marras</a> and <a href="https://it.linkedin.com/in/michele-bottazzi-phd-a0a28990">Michele Botazzi</a>. <strong>The resulting volume of input data amounted to roughly 6 TB, i.e. 3 GB of data for each day for 5 years</strong>.</p><h4>The Solution</h4><p>The first task was to make this data ready for ML. The original data was written as hourly <a href="https://en.wikipedia.org/wiki/GRIB">GRIB</a> files, a binary format widely used in meteorology. One can think of it as a tensor (or N-dimensional array) where any given grid-point comes with its own set of values, in our case each of the final 68000 grid points have values corresponding to the 1000 meteorological features.</p><h4>Data Format</h4><p>Given that in order to tabulate the data, I had to write 2100 x 24 ~ <strong>50000 files (one for every hour of every day for 5 years of data), with 68000 rows (data samples at that hour) and 1000 columns (the features at that hour)</strong>, I decided to use a more compact data format with respect to <em>csv</em>. I went with <strong><em>feather</em></strong> (or <em>parquet</em>), a format that is <strong>highly portable and light-weight. </strong>Feather files are easily handled by Python if one installs the correct Apache Arrow bindings or simply put, the ‘<a href="https://arrow.apache.org/docs/python/index.html">PyArrow</a>’ package. An excellent overview of the same is provided <a href="https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d">here</a>.</p><h4>Data Preparation</h4><p>The task now was to read a binary GRIB file and rewrite it in the form of a feather file. This had to be done for all the 50000 files. While this step was not necessary, it was convenient as it is possible to read feather files with pandas, but the same is not true for GRIB files. The pseudo-code for the IO operation can be defined as follows:</p><pre>def grib_to_feather(grib_file):<br>    grib_data = pygrib.read(grib_file)<br>    tabulated_file = tabulate(grib_data)<br>    pyarrow.write(tabulated_file)</pre><pre>for grib_file in GRIB_FILES:<br>    grib_to_feather(grib_file)</pre><p>The function <em>grib_to_feather</em> takes <strong>10 seconds per file</strong> (on average), thus to process all the <strong>50000 files we would need roughly 5 days!</strong></p><h4>Data Preparation in Parallel</h4><p>I decided to parallelise the IO operation under the ‘embarrassingly parallel’ paradigm. It is called so as little effort is required in parallelising the task at hand, especially because there is no communication between the resulting individual parallel tasks. In our case we can leverage this ideology since the raw data is written as separate files (a file for every hour), and we just need to read it in and write it out in a different form (and format). In other words, we can read-in the raw data and write-out the processed files asynchronously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LlUZved9e5TNk_xd4hP4nw.png" /><figcaption>Carrying out IO operations in parallel asynchronously using 8 cpus. Image courtesy: author.</figcaption></figure><p>At CINECA we host one of the largest super-computing facilities of Europe. Thus, I decided to use just one node (32 cores) on one of our machines called ‘<a href="https://www.hpc.cineca.it/hardware/marconi100">Marconi 100</a>’. I modified the code by using the <a href="https://docs.python.org/3/library/multiprocessing.html">multiprocessing</a> module in python. The pseudo-code is as follows:</p><pre>from multiprocessing import Pool</pre><pre>n_workers = 30<br>p = Pool(n_workers)<br>p.map(grib_to_feather,GRIB_FILES)</pre><p>As simple as that! Since the files are independent of each other and the processing is stand-alone for each file, this method works perfectly. The <em>map</em> call can ‘map’ a function with a given input, to a list of viable inputs. In other words, it creates 30 parallel tasks where a task is defined as running the function <em>grib_to_feather</em> on a file name extracted from the list of file names stored in the list <em>GRIB_FILES</em>. As one task is completed on a processor (i.e. one raw GRIB file is read in, processed and written out as a feather file), the next available file name is passed to the processor. The call <em>map</em> takes care of this file-processor association under-the-hood.</p><p>Thus, with a parallel version of my IO script I obtained a speed up in proportion to the number of cores used, i.e. <strong>4 hours instead of 5 days!</strong></p><p>In the next article, we will see how one can speed up neural networks on supercomputers with <em>data parallelism</em>.</p><p><em>Note: All images in the article are created by, and property of the author.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa403e51516b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-on-supercomputers-data-processing-aa403e51516b">How I reduced data processing time from 5 days to 5 hours</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to write code that solves logical constraints instead of testing them]]></title>
            <link>https://levelup.gitconnected.com/how-to-write-a-code-that-solves-logical-constraints-instead-of-testing-them-57d7575e8070?source=rss-3a1273d509cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/57d7575e8070</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[logical-thinking]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Bhaskar Agarwal]]></dc:creator>
            <pubDate>Sun, 26 Apr 2020 15:02:33 GMT</pubDate>
            <atom:updated>2020-04-27T03:10:36.161Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to write code that can solve logic without a loop</h3><h4>Using integer linear programming to solve complex logic</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xNzOXA7pBrtKZNX-JW4Oqw.jpeg" /><figcaption>Image by author.</figcaption></figure><p>Imagine the following scenario. You have a set of logical constraints. These constraints can only be satisfied by a certain combination of input variables. You need to write code that returns the valid input , i.e. one that satisfies all the logical constraints. One can proceed in two ways.</p><h4><strong>Option 1: Testing in a loop</strong></h4><p>The first idea is to come up with code that tests each possible input combination against the logic. Start with a list of all possible input combinations, pass each one of them to a loop containing all logical constraints, test each input combination against the constraints in the loop, repeat the process for all the possible combinations. In the end, return the combinations that satisfy all constraints. We can make such algorithms faster, more efficient, cleaner etc., but it doesn’t refute the fact that we still run a loop over all possibilities to return the right output.</p><h4><strong>Option 2: No-testing, no loop</strong></h4><p>There is no loop here whatsoever. This means a method where the code arrives at the answer without ‘testing’ anything. Instead of checking all possibilities one-by-one against the logical constraints, the constraints are ‘solved’ to arrive at the only possible outcome. The objective of the post it to show how one can write such a code.</p><h4>The no-testing approach</h4><p>The first idea that comes to mind is parallelisation. One would check the logical constraints in parallel and arrive at the right outcome. The problem is, even if we parallelise the code, we are still checking all possibilities against the constraints. The loop is still present in one form or another. This does not serve our purpose.</p><p>What if we could write logical constraints as linear equations? Doing so would reduce a bunch of if-else clauses into a set of linear equations. Then all that remains is solving those linear equations and finding the right answer! So is there a method where we can solve all linear constraints <em>simultaneously</em> to arrive at the right outcome? Yes there is. <strong>Solving a bunch of constraints simultaneously sounds a lot like linear programming</strong> (if the said constraints are linear in nature). If you followed my <a href="https://towardsdatascience.com/solving-your-first-linear-program-in-python-9e3020a9ad32">last post</a>, or have come across linear programming you can guess where this is going. As I mentioned in my last post</p><blockquote>A linear program finds an optimum solution for a problem where the variables are subject to numerous linear relationships.</blockquote><p><strong>So how can one write a logical constraint in the form of a linear relationship, i.e. a linear equation?</strong> The answer is by using binary variables and a bit of creativity. Let us consider the example of the AND logic gate. The output is TRUE if both the inputs are the same. In binary form, we can write 0 as FALSE (or OFF) and 1 as TRUE (or ON). Thus, I could write the equation for this gate with two inputs In1 and In2, where both can be either 0 or 1, as</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2F5NVaAJ9FrgeFTENAZbdD7F&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2F5NVaAJ9FrgeFTENAZbdD7F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="135" frameborder="0" scrolling="no"><a href="https://medium.com/media/ceb545942b753cd22652fb1624454453/href">https://medium.com/media/ceb545942b753cd22652fb1624454453/href</a></iframe><p>Now that we understand how a logic gate can be transformed into a linear equation (in binary variables), we are ready to write a code that can actually solve a bunch of such constraints in one shot!</p><h4>The challenge</h4><p>I recently came across a puzzle in the periodical <a href="https://www.lasettimanaenigmistica.com/index.html?m=site">La Settimana Enigmastica</a>, which I wanted to solve by writing a code that does not loop over all possibilities but instead solves for the right logical outcome. I paraphrase the problem below.</p><p><em>A person goes to a carousel operator at the fair. The carousel operator tells him that anyone who can guess what happens when he turns on all 5 switches gets free rides for life. The 5 switches do the following:</em></p><ul><li><em>Switch 1: Turns on red when yellow is on</em></li><li><em>Switch 2: Red and green are never on together</em></li><li><em>Switch 3: Blue and green can only be on and off together</em></li><li><em>Switch 4: Blue and purple cannot be off at the same time</em></li><li><em>Switch 5: If purple is on, blue and yellow have to be on</em></li></ul><h4>The approach: integer linear program</h4><p>We will solve this problem by writing a code that does not really &#39;test&#39; anything. It just solves the equations (logical constraints) and arrives at the answer.</p><p>First, let us define variables for coloured lights</p><ul><li><em>Y</em>: Yellow</li><li><em>R</em>: Red</li><li><em>G</em>: Green</li><li><em>B</em>: Blue</li><li><em>P</em>: Purple</li></ul><p>Each of these lights (or variables) can either be ON or OFF, and we further define</p><ul><li>value of 1: ON</li><li>value of 0: OFF</li></ul><p>By applying the same logic as we did for the AND gate, we will try and understand the logic behind each switch and write it as an equation. This will require us to first write down what the switch allows (or does not allow), and how those combinations can be expressed as a linear equation for each switch.</p><p><strong>Switch 1: Turns on Red when Yellow is on<br></strong>This allows for the following values for [Y,R]: [1,1],[0,1],[0,0].<br>Note that in the first case, Yellow is ON and Red is ON. In the second and third cases, if Yellow is OFF, Red could be ON or OFF as the condition does not say anything about the status of Red if Yellow is OFF. I can write it in equation form as</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FuVUNtVQUcmjDp27tDYwUJ3&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FuVUNtVQUcmjDp27tDYwUJ3&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="372" height="96" frameborder="0" scrolling="no"><a href="https://medium.com/media/e74094cc1580cbac2cc5402e42227d12/href">https://medium.com/media/e74094cc1580cbac2cc5402e42227d12/href</a></iframe><p><strong>Switch 2: Red and Green are never on together<br></strong>This allows for the following values of [R,G]: [0,0],[0,1], [1,0].<br>This is because both can be off together but not ON together. In equation form</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FkPXYMQ27iDJgtK4j1uVZzz&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FkPXYMQ27iDJgtK4j1uVZzz&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="136" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0acf7af15f23e836acd71dd042ce7dc/href">https://medium.com/media/e0acf7af15f23e836acd71dd042ce7dc/href</a></iframe><p><strong>Switch 3: Blue and Green can only be on and off together<br></strong>The possible [B,G] combinations are: [0,0],[1,1]. As an equation</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FcV5nZbHksifmH5HAJWsCH6&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FcV5nZbHksifmH5HAJWsCH6&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="136" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e502c86d81470d1b457c5b4fcecf1ba/href">https://medium.com/media/0e502c86d81470d1b457c5b4fcecf1ba/href</a></iframe><p><strong>Switch 4: Blue and Purple cannot be off at the same time<br></strong>The possible [B,P] combinations are: [0,1],[1,0],[1,1]. The condition does not say anything about them being ON together, hence the last option. As an equation</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FnshmRjY7pwS1axTtAgXpVF&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FnshmRjY7pwS1axTtAgXpVF&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="161" frameborder="0" scrolling="no"><a href="https://medium.com/media/b85a13be8736aedfa3ea709537ee6df9/href">https://medium.com/media/b85a13be8736aedfa3ea709537ee6df9/href</a></iframe><p><strong>Switch 5: If Purple is on, Blue and Yellow have to be on<br></strong>This condition is better seen as a combination of two sub-constraints.<br>If Purple is ON, Blue has to be ON and if Purple is ON, Yellow has to be ON.<br>It looks a lot like Switch 1 but we need to apply the logic twice, i.e.<br>maximum of Purple minus Blue is 0 and maximum of Purple minus Yellow is 0. Or in other words</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2Fr64me9FRMY5fd4mJ6wgBR1&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2Fr64me9FRMY5fd4mJ6wgBR1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="209" frameborder="0" scrolling="no"><a href="https://medium.com/media/e76c950db6ae0861236e50e9eba0c1a3/href">https://medium.com/media/e76c950db6ae0861236e50e9eba0c1a3/href</a></iframe><p>Thus, the binary nature of our variables allowed us to re-write the logical constraints in a simple mathematical form. We express the above 4 inequalities (Switch 1,2,4,5) and one equality (Switch 3) in <a href="http://cs.brown.edu/stc/summer/94GeoTrans/94GeoTrans_31.html">matrix form</a> by writing</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2F9N3Yvm4aRDeGTzATArrKUZ&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2F9N3Yvm4aRDeGTzATArrKUZ&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="336" height="368" frameborder="0" scrolling="no"><a href="https://medium.com/media/c22778f079ccb7eaf433cbe0acb0e1ba/href">https://medium.com/media/c22778f079ccb7eaf433cbe0acb0e1ba/href</a></iframe><p>This is now beginning to look a lot like a linear program except all variables only take the value of 0 or 1 (a light can be either ON or OFF). A linear program (LP) where all variables can only assume integer values is called an integer linear program or an ILP. Ours is a special case 0–1 ILP as the variables are only allowed two values in order to simulate a boolean logic (note that an ILP is different from a mixed-integer linear program, or MILP, as in a MILP only some variables take integer values).</p><p><strong>Cost function<br></strong>So far I have not touched upon the cost function. In a linear program, a cost function is designed to optimise the result, i.e. minimise or maximise a linear combination of the variables we are solving for. In this particular case we are trying to find the (only) possible answer. So is there even something to optimise? I would say that in this case a cost function might not even be necessary as <strong>we are only checking the feasibility of the problem</strong> (the only possible answer). Thus, I define the cost function simply as a matrix containing five elements all equal to 0.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2F4SWxdbjUtVgiEKfaSZQ15o&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2F4SWxdbjUtVgiEKfaSZQ15o&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="228" frameborder="0" scrolling="no"><a href="https://medium.com/media/d322703c168ea0b7e4f7482961218a68/href">https://medium.com/media/d322703c168ea0b7e4f7482961218a68/href</a></iframe><p>The problem is now reduced to a familiar LP construct</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FgnvFivGRuU2tf5pmjLwSyP&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FgnvFivGRuU2tf5pmjLwSyP&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="412" height="535" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e3cde5d72739391c2618285ec4d8610/href">https://medium.com/media/5e3cde5d72739391c2618285ec4d8610/href</a></iframe><p>where of course the first condition (of minimising the cost function) is purely written for the sake of completeness.</p><h4>Coding the problem in Python</h4><p>My Python setup is as follows</p><ul><li>Python 3.8.2</li><li>Numpy 1.4.1</li><li>Cvxopt 1.2.3</li></ul><p>My previous post discussed how one can implement a LP in Python using the optimize library in <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linprog.html">SciPy</a>, and <a href="https://www.gnu.org/software/glpk/">GLPK</a> solver in <a href="http://cvxopt.org/">CVXOPT</a>. Our current problem at hand is an ILP, i.e. I only permit integer values (0,1 to be more specific) for my variables. The SciPy implementation in Python does not allow one to pass such constraints. However, this is certainly possible with other libraries.</p><p>The CVXOPT library with the GLPK solver allows you to specify which variables are binary in nature, i.e. assume either a 0 or 1 value. In our case, all our 5 variables are binary in nature thus the binary indices would be 0,1,2,3,4.</p><pre># define the indices of the variables that are binary in nature<br># in our case all 5 variables are binary</pre><pre>Binary_ind = range(5)</pre><pre># …(rest of the code)…</pre><pre># pass these indices to the glpk ilp solver as a set<br># of indices under the parameter ‘B’, i.e. binary indices</pre><pre>status, solution = glpk.ilp(c, A_ineq, B_ineq, A_eq, B_eq, B=set(Binary_ind))</pre><p>Equipped with our matrix formulation and knowledge of how to tell the solver which variables are binary, we can now code up the problem as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c98f7ea09cb64801ac976fb1318c47bf/href">https://medium.com/media/c98f7ea09cb64801ac976fb1318c47bf/href</a></iframe><p>This returns</p><pre>solution found <br>Yellow is OFF , Red is OFF , Green is ON , Blue is ON , Purple is OFF</pre><p>We have found the right answer!</p><p>The code for this article (and the previous one on LP) is hosted on my <a href="https://github.com/codiusmaximus/lights">GitHub repo</a>.</p><h4>Outlook</h4><p>A simple logical riddle prompted the exploration of the different ways it could be answered using Python. Furthermore, it forced me to ask the question ‘could you make the code mimic a human thought process?’. <br>In my opinion, a human thought process in such cases is one where we eliminate the clearly implausible solutions and reason our way through the remaining options. The ILP approach mimics this process, by limiting the solution space and only navigating a sub-set of possibilities, ultimately arriving at the right solution. This is very different to an ensemble of if-else conditions against which we test all possibilities. Thus by using an ILP in such cases, the plausible outcome of a set of logical constraints can be found in one go without actually going through all the possibilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57d7575e8070" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/how-to-write-a-code-that-solves-logical-constraints-instead-of-testing-them-57d7575e8070">How to write code that solves logical constraints instead of testing them</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving your first linear program in Python]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/solving-your-first-linear-program-in-python-9e3020a9ad32?source=rss-3a1273d509cf------2"><img src="https://cdn-images-1.medium.com/max/1920/1*8ZWMIx46hxGg8kiQbKh2Vw.jpeg" width="1920"></a></p><p class="medium-feed-snippet">The &apos;why&apos;, &apos;what&apos; and &apos;how&apos; of linear programming in Python.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/solving-your-first-linear-program-in-python-9e3020a9ad32?source=rss-3a1273d509cf------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/solving-your-first-linear-program-in-python-9e3020a9ad32?source=rss-3a1273d509cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e3020a9ad32</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[learn-data-science]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[linear-programming]]></category>
            <dc:creator><![CDATA[Bhaskar Agarwal]]></dc:creator>
            <pubDate>Sun, 05 Apr 2020 14:00:56 GMT</pubDate>
            <atom:updated>2020-12-28T11:40:43.152Z</atom:updated>
        </item>
    </channel>
</rss>