<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piyush Daga on Medium]]></title>
        <description><![CDATA[Stories by Piyush Daga on Medium]]></description>
        <link>https://medium.com/@piyushdaga5?source=rss-eeb84aa2bcc7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7-WLkkcT_XI7fFB341V8gQ.jpeg</url>
            <title>Stories by Piyush Daga on Medium</title>
            <link>https://medium.com/@piyushdaga5?source=rss-eeb84aa2bcc7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:02:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@piyushdaga5/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Capture and Stream Kubernetes Network Traffic]]></title>
            <link>https://medium.com/better-programming/capture-and-stream-kubernetes-network-traffic-9ab299974946?source=rss-eeb84aa2bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ab299974946</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Piyush Daga]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 15:33:19 GMT</pubDate>
            <atom:updated>2022-03-17T15:33:19.831Z</atom:updated>
            <content:encoded><![CDATA[<h4>With New Relic’s most recent addition, Pixie</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hkHlEvtsLiNhR6JWd1Lq_w.png" /><figcaption>Source: <a href="https://undraw.co/">Undraw</a></figcaption></figure><h3>Objective</h3><p>Tracking network traffic on a Kubernetes (k8s) cluster, capturing request-response tuple and associated metadata. Streaming the captured data out of the cluster to be analysed further.</p><p>Since we’re planning on tracing each network request (OSI Layer 7), there are a few considerations that form a part of our requirements. They are:</p><h3>Considerations</h3><ul><li>Lightweight →Low memory and CPU footprint are critical.</li><li>Scalable → New nodes are added and removed from a cluster, and the solution should scale accordingly.</li><li>Language and Framework agnostic → Containers might be created in any number of languages and frameworks, this should not affect/influence our tracking.</li><li>Rapid → The scale of all network data in a server would be massive. Our solution must be able to track and report that data to us within considerable time limits.</li><li>Comfortable → Users should feel comfortable/safe/assured when deploying this on their production clusters.</li><li>Plug and Play → Low onboarding time and cost. No code changes to attach hooks.</li></ul><p>Based on the above details, we chose <a href="https://px.dev/">Pixie</a> for this task. Pixie is an open source observability platform built by <a href="https://newrelic.com/">New Relic</a>, one of the leaders in the APM space.</p><p>The fact that Pixie is a <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a> project and New Relic uses it for its Kubernetes observability lends credibility to the project. Alleviating the rapid and comfortable considerations made above.</p><h3>What is Pixie?</h3><p>In their own words:</p><blockquote><em>“Open source Kubernetes observability for developers</em></blockquote><blockquote><em>Auto-instrumented. Scriptable. Kubernetes native.”</em></blockquote><p>For us, it means easy deployments, custom data points tracking (if required), and native capture of requests and responses made on the cluster.</p><h4><strong>So, how does Pixie do that? And what makes it special?</strong></h4><p>Pixie uses eBPF technology to run event-driven scripts at the kernel level. Bypassing all code requirements, making the Linux kernel its first-class citizen.</p><p>This makes Pixie language and framework agnostic, and since eBPF works on any system running the Linux kernel, Pixie has a wide install base.</p><p><strong><em>Note:</em></strong><em> This also means Pixie will not run on Windows servers, though the volume of production k8s clusters running on Windows servers might be pretty low.</em></p><p>Full list of supported Pixie environments and cluster types: <a href="https://docs.px.dev/installing-pixie/requirements">Install | Requirements</a>.</p><p>[Optional] Further resources on eBPF technology:</p><ul><li><a href="https://newrelic.com/blog/best-practices/what-is-ebpf">What Is eBPF and Why Does It Matter for Observability?</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F16slh29iN1g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D16slh29iN1g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F16slh29iN1g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/105078cea510b769138238db570209b9/href">https://medium.com/media/105078cea510b769138238db570209b9/href</a></iframe><h4><strong>But still, why Pixie? What are its advantages?</strong></h4><ul><li>Pixie’s Edge Module is deployed at a node level. So, irrespective of the pods and services running, only a single edge module deployment is required per node. Addressing lightweight and scalable.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEDBhwJNQknv0kTtkpFdbQ.png" /><figcaption>Pixie deployment on a k8s cluster. Image from <a href="https://docs.pixielabs.ai/about-pixie/what-is-pixie/#architecture">Pixie’s Website</a></figcaption></figure><ul><li>Pixie has a very simple deployment and installation. Once authenticated, only the px deploy command needs to be run. Addresses plug and play.</li><li>As soon as it is deployed, Pixie is ready to report on Network data. No other setup is required.</li><li>Pixie provides an easy-to-use API (in Python and Go) for extracting/fetching/streaming that data via the internet. It also supports in-transit e2e encryption of the data.</li></ul><p>We go ahead with Pixie as it addresses most of our questions while providing additional features which make development easier.</p><p>In the next sections, we’ll discuss pixie specific things, such as:</p><ul><li>Installation</li><li>What all data does Pixie track and its impact on the cluster?</li><li>Finally, streaming data out from the cluster using Pixie’s API.</li></ul><h3>Installation</h3><p>Installing Pixie is pretty easy, there are only a few steps to follow. Pixie’s documentation: <a href="https://docs.px.dev/installing-pixie/install-guides/community-cloud-for-pixie/">Comprehensive Install Guide for Pixie Community Edition.</a></p><p>If you already have a Kubernetes cluster available you can skip this section and go straight to the installation steps for Pixie. Here, we’re going to install and run a minikube cluster for local testing.</p><p>Steps to set up a local Kubernetes environment:</p><p>Step 1: Install <a href="https://minikube.sigs.k8s.io/docs/start/">minikube</a>.</p><p>Step 2: Increase the configured CPU and memory limits. This is required as we’ll be deploying Pixie and a microservice for request simulation.</p><pre>minikube config set cpus 4<br>minikube config set memory 8192</pre><p>If you already have minikube installed and running, then you need to perform minikube delete before your changes are materialized.</p><p>To check if the changes took place correctly run the below commands.</p><pre>minikube config get memory<br># Output: 8192</pre><pre>minikube config get cpus<br># Output: 4</pre><p>Step 3: Start minikube with the hyperkit driver.</p><pre>minikube start --driver=hyperkit</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p71WLG9bUb7AVf-ZUNzYvw.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Steps to install Pixie on a cluster</strong>:</h4><p>Step 1: The main interface for Pixie to interact with the cluster is its CLI. Install it via:</p><pre># Copy and run command to install the Pixie CLI.<br>bash -c &quot;$(curl -fsSL <a href="https://withpixie.ai/install.sh">https://withpixie.ai/install.sh</a>)&quot;</pre><p>Step 2: Pixie (CLI as well) requires an account to work and authenticate with the cluster. To authenticate Pixie CLI, run:</p><pre>px auth login</pre><p>Step 3: Once login is successful. Deploy Pixie on the cluster.</p><pre>px deploy</pre><p>That’s it ✅ for Pixie’s installation. We’ve set up a cluster and installed Pixie on it. But, we can’t track traffic if there isn’t any so we deploy a demo application. <a href="https://microservices-demo.github.io/">Sock Shop Weaveworks</a>.</p><pre>px demo deploy px-sock-shop</pre><p>Once this command finishes, to check if everything is installed correctly, run the following command:</p><pre>kubectl get pods -A</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZMnNHa212wuxkoUYcXOezA.png" /><figcaption>Final state after Pixie + sock-shop installation. Image by author.</figcaption></figure><p>If you look closely, there is a load-test-* pod in Init state, which is simulating traffic.</p><p>Go to Pixie’s <a href="https://work.withpixie.ai/live">Live UI</a> for exploring all the metrics that it collects:</p><p>Some example views:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sseF7Y3YgYdJIJmMlgPFxQ.png" /><figcaption>Cluster Level Metrics. Image by author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gX_UR0IMFxgJr8gFa_EENA.png" /><figcaption>Namespaces. Image by author.</figcaption></figure><p>The most relevant for us: http_data .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iccOnWBf9MFsOO2ERxnFVg.png" /><figcaption>HTTP Data. Image by author.</figcaption></figure><h3>What Data Does Pixie Track, and What’s Its Impact on the Cluster?</h3><p>Pixie comes with a lot of scripts preinstalled to make all sorts of tracking easier — <a href="https://docs.px.dev/about-pixie/data-sources/">About Pixie | Data Sources</a></p><p>Below is a list of important things to note regarding its data collection and impact on the cluster:</p><ul><li>Pixie stores the data it collects in-memory on the nodes in your cluster; no data is sent to a centralized backend outside of the cluster.</li><li>Pixie has a <a href="https://docs.px.dev/installing-pixie/requirements/#memory">2GiB memory requirement</a>. After installing Pixie, it is normal to see a temporary increase in memory usage.</li><li>How much data does Pixie store?<br>Retention time depends on the level of traffic in your cluster, but will generally be on the order of hours. Each data table in Pixie (e.g. http_events) has its maximum size. These tables collect data until the maximum size is reached, and then begin dropping the oldest data.</li><li>Typical performance overhead for node CPU usage is between 2–5%, depending on the amount and type of traffic. We are in the process of optimizing this, however, so watch for that to go down even further.</li></ul><h3>Streaming Data From the Cluster Using Pixie’s API</h3><p>Now that we have installed Pixie and it has started collecting HTTP events data, our goal is to stream that data out.</p><p>Pixie provides an API that we can use. To get started we require two values:</p><ul><li>Cluster ID → <a href="https://docs.px.dev/reference/admin/cluster-id/#find-the-cluster-id">Reference | Admin | Find a Cluster ID</a></li><li>API Key → <a href="https://docs.px.dev/reference/admin/api-keys/#create-an-api-key">Reference | Admin | API Keys</a></li></ul><p>Once we have these values ready, we can set up a new Python virtual environment and install pxapi in it.</p><pre>pip install pxapi</pre><p>If the above fails on MacOS, try:</p><pre>GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install pxapi</pre><p>Script to stream http_data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/778553cfa78093bc635159253a3b4957/href">https://medium.com/media/778553cfa78093bc635159253a3b4957/href</a></iframe><p>Pixie provides a very powerful API and framework to debug and observe applications deployed on Kubernetes. This post just scratches the surface on what Pixie is about and what it can deliver for your business.</p><p>For further reading, please go through:</p><ul><li><a href="https://docs.px.dev/">Pixie docs</a>, <a href="https://docs.px.dev/using-pixie/api-quick-start/">Pixie API</a> and <a href="https://docs.pixielabs.ai/about-pixie/what-is-pixie/">Pixie Architecture</a></li><li><a href="https://newrelic.com/blog/best-practices/what-is-ebpf">What Is eBPF and Why Does It Matter for Observability?</a></li><li><a href="https://www.youtube.com/watch?v=16slh29iN1g">BPF performance analysis at Netflix</a></li></ul><p>Let me know if you’d like to cover the second half of this problem statement: <em>Building a streaming pipeline to handle the massive scale of network data</em>.</p><p>Until then, take care.</p><pre><strong>Want to connect?</strong></pre><pre>Twitter: <a href="https://twitter.com/piyush_daga5">https://twitter.com/piyush_daga5</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ab299974946" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/capture-and-stream-kubernetes-network-traffic-9ab299974946">Capture and Stream Kubernetes Network Traffic</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey from easy to robust python code]]></title>
            <link>https://towardsdev.com/a-journey-from-easy-to-robust-python-code-df526efbaef4?source=rss-eeb84aa2bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/df526efbaef4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-dataclass]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[intermediate-python]]></category>
            <category><![CDATA[class]]></category>
            <dc:creator><![CDATA[Piyush Daga]]></dc:creator>
            <pubDate>Fri, 25 Feb 2022 04:41:40 GMT</pubDate>
            <atom:updated>2022-02-28T11:53:28.475Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W5M4-Ag5jNNhVoYV" /><figcaption>Photo by <a href="https://unsplash.com/@clemensvanlay?utm_source=medium&amp;utm_medium=referral">Clemens van Lay</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article explores how we can take the next step in writing mature python code. Shifting from “<em>easily written but painful to maintain</em>” to a more robust approach, that we so often witness in large open-source projects.</p><p>Any mid to large scale system is built up of small composable blocks, which try to solve one problem and do it well. The block or component is responsible for the complete representation: how it looks, what interfaces it exposes and how it behaves with other systems.</p><p>When we start writing our implementation of a component, we tend to begin with the most “expressive” data structure we can find for the task.<br><strong>Note:</strong> Not the easiest, but the most expressive. As such if we have to build a model for a user, having a first name, last name and age. <strong>We’ll prefer the dict implementation over the tuple </strong>owing to its expressiveness as key-value pairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*KO5iLLOf5QwrvRMxO1xu2w.png" /><figcaption>Users modelled using a tuple and a dict</figcaption></figure><p>But, do we stop to consider how it would scale when we have to instantiate thousands of users? How difficult will it be to maintain them down the road? Will more engineering time be spent on maintenance instead of development?</p><blockquote>Engineers like to gloss over actualising the time required to maintain a new system. Hence the phrase:</blockquote><blockquote>Building stuff is easy, but maintaining them is hard.</blockquote><p>We already saw how we would start off instantiating users (as dicts) and we asked ourselves some challenging questions about our choice. But those questions were a bit generic and difficult to answer.<br><strong>Asking a question too vague will eventually drive us to give up on answering it altogether</strong>.</p><p>So, let’s narrow down the questions, make them more granular and try to answer them.</p><pre>✋ All the code referenced in this article can be found in the github repository here: <a href="https://github.com/piyush-daga/better_to_robust">https://github.com/piyush-daga/better_to_robust</a>. Feel free to raise PRs for any changes or improvements.</pre><pre><br>Journey:<br> - Questioning our current approach.<br> - Devising a better approach with more flexibility.<br> - Understanding the challenges with the newer approach.<br> - Using an idiomatic python concise solution.</pre><p><strong>Q1: How would we “print/display” our users?</strong></p><p>Well, the simple trick would be to call print(spidy_dict), but that would lead to an output such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUfe692-8zllU547TaquZg.png" /><figcaption>Printing the spidy_dict user</figcaption></figure><p>Okay for debugging but in no way representative of what this component is.</p><p>We can easily do better. If we write a function that takes in this user and prints out a more reasonable output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0_sy1zC_1YZ2OBsM0eXIQ.png" /><figcaption>Custom print function for User class</figcaption></figure><p>Now the pieces slowly start to crumble, since the user is a builtin dict type, there is no way to confirm that the object passed to the function is a user.</p><p>If any of the dict-keys referred to print the user does not exist, python will throw a KeyError. As we want to print the correct user, we cannot use .get calls to “avoid” errors. We’ll have to check for the existence of those keys and abort if any of them are missing before we call the print function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-BLpqOyKCJ2QWpGk_OLeQ.png" /><figcaption>Updated print_user with safety checks</figcaption></figure><p>I believe we’re starting to get a sense of how a simple operation is getting complicated. But this is not the worst part, in any moderately large project where the user is passed around multiple files/modules, each module might create its own way of representing the user as the print_user method is not <strong>bound to </strong>the user.</p><blockquote>Inviting Inconsistent implementations of the same functionality.</blockquote><p><strong>Q2: What if someone knowingly/unknowingly modifies the </strong><strong>spidy_dict user?</strong></p><p>It would have major ramifications across all other modules using it. Frankly, as a dict implementation, we cannot prevent other modules from NOT changing it.<br>We could use a frozendict to prevent it from changing, but that would involve <a href="https://pypi.org/project/frozendict/">downloading a package</a> and well the <a href="https://www.python.org/dev/peps/pep-0416/#id2">python community does not like it very much</a>.</p><p>These challenges are just the tip of the iceberg, employing builtin dicts to model an entity leads to tons of messy code changes, for the smallest business logic change. Making programmers hesitant to pick up a bug knowing even a small change would require re-writes or re-checks in multiple files.</p><p>Well, we’ve been through the gloomy half, and we’ve questioned, thought about and berated ourselves on our coding practices or lack thereof. Now it’s time to look at how we can do better. We will build our solution incrementally, understanding why each piece is important and what value it serves. Eventually, looking at a much better, easier and faster way to achieve it all.</p><p>Shall we.</p><p>When an entity is important and its use-cases varied, the solution is to build a class and instantiate the objects of this class. The class (User) will hold all the required methods to interact with users properly and in a defined fashion.</p><p>First, let’s instantiate our class. Pretty basic stuff here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CwtSWHKkNFkTgELVJD9emA.png" /><figcaption>Instantiating the user class</figcaption></figure><p>Now, people would like to print a User or display it, so let’s add a __repr__ function to it. <a href="https://stackoverflow.com/questions/1984162/purpose-of-repr-method">__repr__ function</a> is called to show a printable representation of an object, i.e. print() function on a class defaults to __repr__ .</p><p>I went with a simple print, we can change it to anything we like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87-CtjPNv6cqD3W3wl6UxQ.png" /><figcaption>Comment shows how the output will look like when print is called on an object of the class</figcaption></figure><p>Let’s revisit the problems faced with the dict implementation:</p><ul><li>How would we “print/display” our users?</li><li>What if someone knowingly/unknowingly modifies the spidy_dict user?</li></ul><p>In one fell swoop, we solved both. We get a standard printable representation and we can be sure that no one can add extra attributes to our objects. Towards the end, we’ll also look at how to make our attribute values, i.e. first_name = Tom immutable, so no one can modify the same.</p><p>Now that printing is out of the way, let’s suppose we want to compare users based on their age, to answer the age-old question: Who came first??</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*fscnGZRpI9CcGSTj1QL8lQ.png" /><figcaption>We want to compare spideys</figcaption></figure><p>To do the above, python provides dunder operators.</p><p>They are: __eq__, __gt__, __lt__, __le__, __ge__. They stand for equals, greater than, less than, less than or equals to and greater than or equals to respectively. There are more operators available, but these will do nicely for now.<br><strong>Don’t worry, we don’t need to implement all of them</strong>.</p><p>The __eq__ function needs to be written in a way to correctly identify similar users, while the other four are for sorting the users based on their ages.</p><p>To say that two users are the same they need to have the same first name, last name and age. Essentially, all the attributes that define the make-up of a person.</p><p>A simple implementation that checks if all attributes are the same works for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J-324oIq8SS6xd7fWXS-3w.png" /><figcaption>__eq__ method</figcaption></figure><p>But, we can be better here, we can check if the object being compared is of the User class or not. If not, we raise an appropriate error.<br>So let’s change that up, create some users and check if our implementation works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WeenSfbc3wn3NRnm2zgenA.png" /><figcaption>tom is equal to cloned tom 😱</figcaption></figure><p>Now that we can tell tom and andrew apart let’s start comparing their ages. To do that, we’ll implement the less-than (__lt__) criteria first, basically checking against the age attribute and returning a boolean output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*UPK8YYoYwJJW8BEWooxnww.png" /><figcaption>__lt__ method</figcaption></figure><p>Based on the above code, do you notice what python does?</p><p>Whenever we call tom &lt; andrew, or anytime the &lt; operator is invoked, the call is translated to tom.__lt__(andrew). Fascinating, we can even call dir(tom) to see the methods that it implements, and hence the functions that can be run on it.</p><p>By implementing __lt__, we can run, print(tom &lt; andrew), but we can’t run print(andrew &gt; tom), as we’re missing the greater than’s implementation. Try the same, and python gives this error:</p><pre>Operator &quot;&gt;&quot; not supported for types &quot;User&quot; and &quot;User&quot;</pre><p>Fortunately, we do not need to implement all the comparison methods, <a href="https://docs.python.org/3/library/functools.html#functools.total_ordering">the </a><a href="https://docs.python.org/3/library/functools.html#functools.total_ordering">functools library has a </a><a href="https://docs.python.org/3/library/functools.html#functools.total_ordering">total_ordering decorator</a>, which requires us to implement an __eq__ method and one of the 4 comparison methods (we went with __lt__ ) and it will provide the remaining for us.</p><p>Let’s see what our class looks like with everything implemented.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gMgPJTLiAbIBhnvcBQpquw.png" /><figcaption>Things are coming together nicely</figcaption></figure><p>Now, we can successfully run the below operations on our objects.</p><p>The cherry on the cake?<br>The implementations are tied to the object and every downstream implementation can use our standards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lGAAkoCHeG6T2oafv00aA.png" /><figcaption>Compare, compare and compare all the way !!</figcaption></figure><p>Well, onto our initial promise now, we don’t want people to access and change properties of our class (Note: They can’t add a new entry anyways).</p><p>We’d like to make our attributes immutable. We can do that by slapping the property decorator and renaming our attributes.</p><p>To do that we have to alter the __init__ method slightly (prefix double underscores to each attribute) and add their property implementations. property is the pythonic way of enabling getters and setters for our classes, you can read in-depth about it <a href="https://www.programiz.com/python-programming/property">here</a>.</p><p>So now underlying implementations can read these properties but cannot change them 🎉.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qwarpoObQqbG5Pry8Lj4zQ.png" /><figcaption>Immutable and Indestructible. I’m feeling strong 💪</figcaption></figure><p>We started from a small class with just an __init__ method and then built upon it to introduce printing (__repr__), similarity (__eq__) and comparisons (__lt__ and total_ordering). We even went ahead and made our attributes immutable.</p><p>But, you might have this sinking feeling that things are somewhat getting out of hand. A simple dictionary implementation has turned into a class with several methods. Though it offers flexibility, is it frankly worth the effort?</p><p>Let’s have a look at the complete class code we’ve written till now. This is still a pretty barebones implementation to be fair, there are a lot more methods we can implement to gain additional functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ofyy_GX25exSNDQADTYX3A.png" /><figcaption>Class based implementation of the user model</figcaption></figure><h3>Rescue Me Please 🙏</h3><p>We found a solution to our initial problem statements, but it came with its challenges. Writing so much code for a simple implementation is unlikely to go over well with anyone, especially within tight deadlines and harrowing deliverables.</p><p>Enter dataclasses. Take all that code and well, delete it. Here is the dataclass implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*kjhcVyYc-PNOYBLDdHVksQ.png" /><figcaption>That’s it. Notice: <strong>age</strong> is the first property because we want to compare <strong>users</strong> against it.</figcaption></figure><blockquote>Dataclasses are an in-built pythonic implementation of “classes storing state” or modelling entities. They take care of a lot of boiler-plate for us and provide a clean interface for the end-user.</blockquote><p>For our use case, dataclasses are the perfect drop-in replacement. Let’s validate by creating our users and checking if everything works as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i1sRjfVGVmOU9kqB9gmijQ.png" /><figcaption>Tada 🎉</figcaption></figure><p>Apart from being a cleaner alternative, dataclasses have a few tricks up their sleeve.</p><p>If we want to use the created user as a tuple or a dict (our original implementations), dataclasses has a method for that as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSshqjr5rV-oISiRMQFCag.png" /><figcaption>asdict and astuple when we need it</figcaption></figure><p>The benefit of using dataclasses is the peace of mind that a higher-order implementation provides.</p><p>We get all the bells and whistles of the class we just wrote, with the added advantage, that all implementations are correct and the chance of missing a rogue self or . is close to zero.</p><p>Dataclasses also has the benefit that adding a new attribute is super easy. Suppose we wanted to add hometown as an attribute.<br>For dataclasses, we would just add another entry, and all the functionality we’ve come to expect would still just work. But in our original Users class, we’d have to change __init__, __repr__, __eq__ methods and add a property for hometown 😞.</p><p>Let’s look at one more feature of dataclasses before we wrap this up.</p><p><strong>Default arguments.<br></strong>Dataclasses allow us to set default arguments for any of the attributes. Let’s set one for hometown, as that’s optional.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*OJ0jf7G6BzH4WPUKMSe5_A.png" /><figcaption>Default hometown argument</figcaption></figure><p>That’s it !!</p><p>This has been some journey, starting from a simple and easy dict implementation to a powerful, but verbose class. Finally, we got a gentle introduction to dataclasses, which is a “best of both worlds” solution.</p><p>There is a lot more to dataclasses, classes and certainly this topic as a whole.<br>I’ll write more around this, until then some references and further reading:</p><ul><li><a href="https://docs.python.org/3/library/dataclasses.html">https://docs.python.org/3/library/dataclasses.html</a></li><li><a href="https://docs.python.org/3/library/functools.html#module-functools">https://docs.python.org/3/library/functools.html#module-functools</a></li><li>Github repo for final code: <a href="https://github.com/piyush-daga/better_to_robust">https://github.com/piyush-daga/better_to_robust</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df526efbaef4" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/a-journey-from-easy-to-robust-python-code-df526efbaef4">A journey from easy to robust python code</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trigger “specific” ECS Tasks with S3 payloads]]></title>
            <link>https://medium.com/analytics-vidhya/trigger-specific-ecs-tasks-with-s3-payloads-79c9ce6aa57d?source=rss-eeb84aa2bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/79c9ce6aa57d</guid>
            <category><![CDATA[ec]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Piyush Daga]]></dc:creator>
            <pubDate>Sun, 23 Jan 2022 06:54:04 GMT</pubDate>
            <atom:updated>2022-02-25T16:31:53.178Z</atom:updated>
            <content:encoded><![CDATA[<p>And inject the payload to the triggered task</p><p>With the launch of AWS Fargate, ECS (Elastic Container Service) becomes that much more lucrative an offering for organisations that want managed container orchestration, but do not have the need or expertise to switch to Kubernetes.</p><p>The king of serverless function invocations, AWS Lambda has a major bottleneck that it can be run for 15 minutes maximum on a single invocation. There are <a href="https://lumigo.io/learn/aws-lambda-timeout-best-practices/">design patterns</a> we can choose to overcome this, but for long event-driven workflows, ECS on Fargate offers a nice alternative without the time limit constraints.</p><p>Let’s have a look at the problem statement to get more clarity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2ajdt-w3nOlGjuL45lrSQ.png" /><figcaption>S3 data-dumps triggering respective ECS tasks</figcaption></figure><p>There are multiple processes dumping data in a S3 bucket, and each process’s data requires a separate workflow to process/transform the same. In our case, each type of transformation is handled by a separate ECS Task.</p><p>As a design pattern consideration, if we want to process salesforce data, and our ECS task is named as salesforce pipeline. The process dumping data should dump it under the S3 prefix app_salesforce . This embeds the task name in the S3 key and helps intrinsically map data to its pipeline.</p><p>Now, let’s consider that S3 has been setup to emit events when data dumps on specific prefixes occur (how to do the same is discussed below). The bridge mapping (implementation below) those multiple events to their correct tasks is a lambda function which is triggered by the event and runs the correct ECS task.</p><ul><li>S3 Event Notifications→ Lambda → ECS Task Trigger</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCY5ragfEAMl7A5eD4dXZQ.png" /><figcaption>Lambda used as a mediator in directing correct task invocation</figcaption></figure><blockquote>Note: This article assumes that the ECS Task exists and is correctly set up.<br>We are not going into how to create a lambda function or an S3 bucket, as multiple references can be found on the same, this solution would tackle the integration between these components.</blockquote><h3>1. Setting up the Lambda</h3><p>We are starting with the lambda setup as it is easy to test whether it can trigger the correct ECS Task, given a mock payload.</p><p>The lambda is written in python and it extracts the folder name from the s3 key and calls the respective ECS Task for it. For details on the API call, the core logic code is attached for reference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f7d26d82f7e21d4581515ca3d5eb549/href">https://medium.com/media/1f7d26d82f7e21d4581515ca3d5eb549/href</a></iframe><blockquote>The <strong>containerOverrides</strong> section in the <strong>run_task function call</strong> is the crux of the solution.</blockquote><p>This lambda can then be tested using mock S3 payloads to validate if it can correctly trigger the ECS task, while injecting the payload in Environment Variables.</p><h3>2. Setting up S3 to trigger the Lambda</h3><p>To add the trigger for the lambda</p><ol><li>Go to the bucket → Properties Tab → Create event notification</li><li>Add the event name and keep the prefix as “<strong>app”</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*8eXPSlhFmxsCZy-FJ0NHdA.png" /><figcaption>Create Event on S3</figcaption></figure><p>3. Event Types: <strong>All object create events</strong></p><p>4. Add the Lambda function which we created above. Keep the name as the one that you set up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*o_fF3bjMl8Rp3Iun-wBfiw.png" /><figcaption>Add the lambda function</figcaption></figure><h3>3. Bringing it all together</h3><p>Now, when the data is dumped in S3 by any source we can check that the corresponding task has been executed and validate that the S3 bucket and key are injected as environment variables to be consumed.</p><p>In parting, I’d like to leave you with the thought that there is another way of achieving the above using Cloudtrail + EventBridge rules as well, would love to discuss the scalability issues around it in the comments.</p><p>Hopefully, you liked reading this article, let me know if you’d prefer deeper dives into other AWS nuances and architecture discussions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79c9ce6aa57d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/trigger-specific-ecs-tasks-with-s3-payloads-79c9ce6aa57d">Trigger “specific” ECS Tasks with S3 payloads</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 rationalities gained from reading 5 chapters of The Upside of Irrationality]]></title>
            <link>https://medium.com/@piyushdaga5/5-rationalities-gained-from-reading-5-chapters-of-the-upside-of-irrationality-17116b7a345b?source=rss-eeb84aa2bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/17116b7a345b</guid>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[review]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[book-review]]></category>
            <dc:creator><![CDATA[Piyush Daga]]></dc:creator>
            <pubDate>Mon, 22 Jun 2020 14:35:51 GMT</pubDate>
            <atom:updated>2020-06-22T14:35:51.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nfNhm7J4GQnEuJfZ" /><figcaption>Photo by <a href="https://unsplash.com/@dtopkin1?utm_source=medium&amp;utm_medium=referral">Dayne Topkin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the successor to his international bestseller, behavioral economist Dan Ariely pens down the unexpected ways we defy logic at home and work. Emphasizing specifically, on how the knowledge of these irrationalities can become a tool in our arsenal, helping us comprehend our actions to make informed &amp; conscious decisions.</p><p>The book is detailed out as a series of experiments ranging from topics such as the joy of work, why my ideas are better than yours to how to use our intrinsic adaptation capabilities to our advantage. The premise of this book is to challenge everything, ask questions, and NOT take things for granted.</p><p>Reflecting back these are the 5 life lessons I learned from the book, in no particular order:</p><h3>A vantage point for creating engaging experiences (Chapter: The IKEA Effect)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RN8vIMiHSUeWVbsu" /><figcaption>Photo by <a href="https://unsplash.com/@jonnysplsh?utm_source=medium&amp;utm_medium=referral">Jonny Caspari</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The IKEA effect in essence details that when our efforts are involved in the creation process of an object to completion, be it as small as assembling a piece of furniture, folding pair to form an origami or adding ingredients to off-the-shelf easy to cook recipes, we feel a sense of pride and ownership towards our creations. This ownership also leads us to overvalue our creations.</p><p><strong>Learning</strong>: Now, whenever I look at semi-DIY websites or offerings by phone case manufacturers, letting us customize not only the color, but also the material, the finish, and unique engravings. I can’t help but feel this is The IKEA effect done right.</p><h3>Manipulative Biases (Chapter: The Not-Invented Here Bias)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VgheChk0UnQ3qwqI" /><figcaption>Photo by <a href="https://unsplash.com/@dallasreedy?utm_source=medium&amp;utm_medium=referral">Dallas Reedy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The not-invented-here bias is a simple fallacy that makes us believe that anything that was NOT our idea does not deserve merit.</p><p>Edison could have had the patent for Alternating Current as Nikola Tesla was under his stewardship when he invented A.C… Still, as an irrational human, he fell for the not-invented-here bias. He considered that if the idea did not originate from his brain, it lacked merit.</p><p><strong>Learning</strong>: At first glance, the not-invented-here bias is NOT at all manipulative. Instead, it is a deformity of the human psyche, but history is a witness to baser instincts.</p><p>Andrew Carnegie had to make a business deal with his bitter rival, a merger of companies was in their individual benefit. But the competitor would not be easily swayed, so the end result? The new company thus formed had the name of his rival, followed by Carnegie’s name. By doing so, Carnegie allowed the man to believe that the company was his invention and legacy, utilizing the analogy of the not-invented-here bias to benefit.</p><p>P.S. I don’t remember the name of the rival 😅</p><p>P.P.S. Thinking out loud: What would the outcome be if Carnegie did not allow his opponent to feel the sense of ownership offered?</p><h3>A large splash for a vivid experience (Chapter: Adaptation)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jKwypPOt9ydDlwqT" /><figcaption>Photo by <a href="https://unsplash.com/@johnnyb803?utm_source=medium&amp;utm_medium=referral">Johnny Brown</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This chapter goes deep into details about physical and hedonic adaptation and how confidently competent the human mind is at facilitating adaptation.</p><blockquote>It turns out that we are very good at conceiving the future, but we can’t foresee how we will adapt to it.</blockquote><p><strong>Learning</strong>: Our mind is adept at adapting to small, inconsequential changes in our day-to-day life. If we wish to leave an impression / create a lasting memory, a big splash is the way to go about it.</p><h3>Take breaks doing what you love and grind through the rest (Chapter: On Adaptation: Why we get used to things)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XO4Y9F8MS9fsuC7d" /><figcaption>Photo by <a href="https://unsplash.com/@epicantus?utm_source=medium&amp;utm_medium=referral">Daria Nepriakhina</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the best inferences personally for me was accepting the conclusion that our brain quickly numbs just about every experience.</p><p><strong>Learning</strong>: It is better to do things we hate in one-go / one-sitting; that way, our brain gets to adapt to the work and makes the overall experience less jarring. On the other hand, it is beneficial for us to break down the things we enjoy so that the brain does not adapt and tune out the exciting bits.</p><h3><strong>Sleep off your emotions (Chapter: The Long-Term Effects of Short-Term Emotions)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OVsgWWLGCtEivtaE" /><figcaption>Photo by <a href="https://unsplash.com/@davidclode?utm_source=medium&amp;utm_medium=referral">David Clode</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>decisions → emotions → DECISIONS (short-term) → DECISIONS (long-term)</blockquote><p>The above flow can be read as, decisions made under the influence of an underlying emotion changes the outcome of that decision to a different one, i.e., DECISION (short-term). And then the short-term decision acts as our North Star for future similar actions, even when the underlying emotion has long worn off.</p><p><strong>Learning</strong>: If there is one learning we can take from this article, I hope this is IT.</p><p>It is better to not act and let the emotion wear off when making a decision because the effects of that short-term outburst may alter our long-term outlook. Or as people commonly say, we should <strong>Sleep On It</strong>.</p><h3>While the above is a manicured list, there are some other small but practical learnings I was able to churn out, namely:</h3><ul><li>Wash dishes in one go (Dislike it, do it in one go and let your mind adapt)</li><li>Stop binge-watching and take breaks in-between episodes (If you are binging something, you probably like it. Its best to savor the taste and not let our mind adapt and start tuning details out).</li><li>Self-herding as a way to automate mundane tasks, like where I keep my keys, wallet, phone, and so on.</li></ul><h3>Some things that still puzzle me</h3><blockquote>Do we cherish a Subway sub more as compared to a McDonald’s burger?</blockquote><p>Per the IKEA effect, the above has a sense of creation attached. So do we care more about the sub, or is the quantifiable amount of ownership insufficient for the bias to kick in?</p><blockquote>decisions → emotions → DECISIONS (short-term) → DECISIONS (long-term)</blockquote><blockquote>a new emotion (slowly creeping)⤴</blockquote><p>The book details that short-term decisions made under the influence of an emotion govern our long-term choices, even when the underlying feeling wears off.</p><p>I’m puzzled by what happens when we are under a new emotion (different from the original), say when we made the short-term decision we were primed with anger, now we are joyful. Will the new emotion overpower our self-herding tendency to fall back to the original decision’s response, or will it start a new cycle which overrides our initial stored sequence.</p><h3>Summary</h3><p>I would highly encourage everyone to sift through the book. Each page is a treasure trove of experiments or the stimulus for one. Please feel free to comment if you have read or are planning to read the same, and let’s have a discussion about it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17116b7a345b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PersonAttributes Classifier — training a multi-task neural network to detect attributes like age…]]></title>
            <link>https://medium.com/analytics-vidhya/personattributes-classifier-training-a-multi-task-neural-network-to-detect-attributes-like-age-a3b877d21d22?source=rss-eeb84aa2bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3b877d21d22</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Piyush Daga]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 10:15:56 GMT</pubDate>
            <atom:updated>2020-01-23T11:57:20.147Z</atom:updated>
            <content:encoded><![CDATA[<h3>PersonAttributes Classifier — training a multi-task neural network to detect attributes like age, gender, …, emotion</h3><p>The goal of this challenge is to detect a person’s gender, age, weight, carryingbag, footwear, emotion, bodypose as well as the imagequality all from a single image. Any neural network architecture is allowed, but the <strong>training has to be done from scratch, that means no pre-trained weights, and no transfer learning.</strong><br> To complete this successfully we will require a convolutional neural network with multiple outputs, each output catering to a prediction.<br> For this challenge I ended up using a <a href="https://towardsdatascience.com/understanding-and-visualizing-densenets-7f688092391a"><strong>Densenet121</strong></a> architecture, but any modern parallel network with skip connections should provide good results.</p><p>The training is done using <strong>Keras</strong>. I also tried <strong>one-cycle policy and cyclic LR in tensorflow</strong>, but found the keras solution simple and effective. <br> Before we take a look at the proposed network architecture let&#39;s take a look at the dataset we are dealing with.</p><p>You can download the dataset from <a href="https://drive.google.com/file/d/1Abe336Tzvi8BC6bVGTN3unlAtj1XdiTT/view">person-data-gdrive</a> for experimentation.<br> Also find the <a href="https://github.com/baronrogers5/person-classifier">person-classifier-github-link</a> which contains the notebook that can directly be run on <a href="https://colab.research.google.com/">google-colab</a>.<br> If you are running the notebook on colab, just make sure to download the data and put it in your google drive’s My Drive as hvc_data.zip. <br> All the required files visualized below are present in the hvc_data.zip file.</p><h3>Visualizing the data</h3><p>Each image is labelled according to the dataframe given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*juuiP2QI9uxB9zX1.png" /></figure><p>Some examples of the images we have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p9CVXmSLlcMUniET.png" /></figure><p>Let’s see all possible predictions for each category</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac7ab28a462f6248e0fd6a2dfdeb8c60/href">https://medium.com/media/ac7ab28a462f6248e0fd6a2dfdeb8c60/href</a></iframe><h3>Viewing the network’s head</h3><p>As promised let’s have a look at the head (where the output’s arrive) of the network, so that we can understand the code better.<br> Some connections are not-visible to post a clearer image. Find the full network image <a href="https://github.com/baronrogers5/person-classifier/blob/master/images/model.png">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*46LO89V-Afi2S8MS.png" /></figure><p>Now that we have an idea about the problem at hand, let’s start with the actual steps required to train an awesome classifier from scratch.</p><ul><li><strong>Data Preprocessing </strong>-</li><li>Converting to one-hot encoded.</li><li>Normalizing using the mean and std-dev.</li><li>Data Augmentation(<a href="https://github.com/yu4u/cutout-random-erasing">cutout</a>)</li><li>Building a keras Sequence to feed to the training loop.</li><li><strong>Designing the architecture</strong> -</li><li>Choosing a backbone</li><li>Constructing the tower.</li><li>Constructing heads [final predictions] for each class.</li><li><strong>Defining the training callbacks</strong></li><li><strong>Actual training.</strong></li></ul><h3>Data PreProcessing</h3><p>Neural networks expect the prediction labels to be one-hot encoded. So that the predicted probabilities [equal to the number of classes] can be directly matched against the correct labels.</p><p>Hence, we need to convert all our prediction labels to one-hot encoded <em>with a prefix</em>, so that it becomes easier to identify them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce13e713e46315466b6ad83fd4e02144/href">https://medium.com/media/ce13e713e46315466b6ad83fd4e02144/href</a></iframe><p>(13573, 9)</p><p>The one-hot encoded data</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/628c7fe075ce48d96eaab8b4bcd2ecd3/href">https://medium.com/media/628c7fe075ce48d96eaab8b4bcd2ecd3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*J7_8c6oChGSJiQ9l.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*r5WMwwngeOUfPWLw.png" /></figure><h3>Normalizing using the mean and std-dev</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6fe65c3b9a8c8d0e0adef0dce31f4f6/href">https://medium.com/media/c6fe65c3b9a8c8d0e0adef0dce31f4f6/href</a></iframe><p>The original size of the images were (200x200), but our dataset contains resized versions of them at (224x224).</p><p>Since, we will be using a <a href="https://towardsdatascience.com/review-densenet-image-classification-b6631a8ef803">densenet</a> as our architecture without the head, we don’t need the extra computation and hence we resize the images to (200x200).</p><p>This resizing is done in the PersonDataGenerator class.</p><p>Let’s create the keras Sequence which can be called to give a sequence of batches when called by fit_generator</p><h3>Store attribute column names in variables</h3><p>We prefixed columns with attribute names while creating one-hot encoded versions, so that we could collate columns with specific attribute names together, this will help identify which columns does a specific target belong to.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa1bc2c15e25a295036a4711fc6f5842/href">https://medium.com/media/aa1bc2c15e25a295036a4711fc6f5842/href</a></iframe><p>Let’s build the sequence class which will give batches of data as a generator, shuffle it and apply data augmentations if specified.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a36dfa0719615f0924d3580ded77a292/href">https://medium.com/media/a36dfa0719615f0924d3580ded77a292/href</a></iframe><p>Let’s quickly call the PersonDataGenerator class to create the train and test generators. To implement cutout we use <a href="https://github.com/yu4u/cutout-random-erasing/blob/master/random_eraser.py">get_random_generator</a> function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07dd8f4698b735d6b18ff3bade443279/href">https://medium.com/media/07dd8f4698b735d6b18ff3bade443279/href</a></iframe><h3>Designing the architecture</h3><h3>Choosing a backbone</h3><p>We are not using pre-trained weights and starting from scratch as that was one of the requirements of this exercise. We take a <strong>DenseNet121</strong> backbone, without the head. This was chosen over <strong>Resnet50</strong>, but that should also provide comparable results.</p><p>Since this is a multi-task classification. The final model we are going for would be something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34ca9ea18a2609317940a617e0fd1109/href">https://medium.com/media/34ca9ea18a2609317940a617e0fd1109/href</a></iframe><ol><li>Choose an architecture as a backbone (here we are choosing densenet121, as it worked better than resnet50 in intial tests), do not include the head, as we would build our own.</li><li>Build towers for each class, the architecture of the towers would largely remain the same.</li><li>Build the respective heads which are the outputs for each class.</li><li>Construct the overall model, specifying the inputs and the outputs.</li></ol><p>Code for the backbone.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f30095f40490e892eceef5df65950e04/href">https://medium.com/media/f30095f40490e892eceef5df65950e04/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/0*nDEA_q-XcxyUP3X_.png" /></figure><p>The final shape after relu is (6, 6, 1024).</p><p>This is passed through GlobalAveragePooling (further referenced as GAP) to average each feature map across the channel dimension. This brings the tensor shape to a flat (None, 1024).</p><p>I strongly believe this can be improved if instead of only using GAP, we also use GlobalMaxPooling and concatenate the outputs. That results in 2048 rank 1 tensor, which would have advantages of both mean and max results.</p><p>Here is a representation of what I mean:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9aacc23dedce648e5aee5aa3687845d/href">https://medium.com/media/c9aacc23dedce648e5aee5aa3687845d/href</a></iframe><h3>Constructing the tower</h3><p>The tower adds a batchnorm after GAP, to normalize the results after GAP and also a small amount of dropout, to improve resilience.<br> It is followed by a densely connected layer that reduces the nodes to 128, on which each head builds.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e50a3294a0a4aa9aba5eba8378e976f/href">https://medium.com/media/8e50a3294a0a4aa9aba5eba8378e976f/href</a></iframe><h3>Constructing the heads</h3><p>The head is the final layer that produces outputs.<br> Let’s take <strong>pose</strong>(one of the attributes we need to predict), it has 3 possible outputs, namely <strong>(front-frontish, back and side)</strong>, so each possible output needs a final node which represents the probability of it’s occurrence.<br> num_units contains the mapping of each name and the number of categories present. build_head builds the heads of each category assigning appropriate final nodes as per num_units.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09715221bc3cb8fe2ce0918cd1798d00/href">https://medium.com/media/09715221bc3cb8fe2ce0918cd1798d00/href</a></iframe><h3>Code to build the complete Model</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/441476b09bc5ae3ef748a97642900afe/href">https://medium.com/media/441476b09bc5ae3ef748a97642900afe/href</a></iframe><p>You can check out how the complete network looks <a href="https://github.com/baronrogers5/person-classifier/blob/master/images/model.png">here</a>.</p><h3>Defining the training callbacks</h3><ol><li>ReduceLROnPlateau -&gt; To reduce learning_rate when the model val_loss does not improve by min_delta for some patience epochs</li><li>ModelCheckpoint -&gt; To save the model weights, in specified directory.</li><li>EarlyStopping -&gt; To stop training if val_loss does not improve by min_delta for patience epochs, restore the best_weights during training after stopping.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcb172cffda187c6521075d36eb5b096/href">https://medium.com/media/dcb172cffda187c6521075d36eb5b096/href</a></iframe><p>Let’s compile the model before we start the actual training.</p><p>The model compilation step has been implemented as a function, so that it can be called from any cell, and on any model (backbone or partial models).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b28bd4c79fe0238fd9ced2346e107ecf/href">https://medium.com/media/b28bd4c79fe0238fd9ced2346e107ecf/href</a></iframe><h3>Let the training begin</h3><p>The training was carried on in the following sequence:</p><ol><li>(100x100) images with all layers trainable.</li><li>(100x100) images with backbone frozen, so that the final layers could be fine-tuned.</li><li>(100x100) images with all trainable layers but aggressive data augmentation with a lr = 1e-4.</li><li>(200x200) images with all trainable layers.</li><li>(200x200) images with backbone frozen.</li></ol><p>All the training loops had EarlyStopping enabled, so the number of epochs ranged from 8-15 in each case, after which the training stopped and the next step was run.</p><p>For the purpose of brevity, I will not include all training steps here, but you can always check the <a href="https://github.com/baronrogers5/person-classifier/blob/master/Person_using_DenseNet.ipynb">actual colab file</a> , which has all the steps with logs.</p><p>If you wish to continue training, and wish to use my pre-trained weights, you can find them in the <a href="https://github.com/baronrogers5/person-classifier/tree/master/models">github repo models folder</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b9a83072ecd92e911848a2631142587/href">https://medium.com/media/4b9a83072ecd92e911848a2631142587/href</a></iframe><h3>Results</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/984bc96c738f7e5ae8a95b796f26266e/href">https://medium.com/media/984bc96c738f7e5ae8a95b796f26266e/href</a></iframe><h3>Further experiments / Ways to improve the model</h3><p>There can be a lot of ways to improve the results that we achieved here. I will keep updating the actual <a href="https://github.com/baronrogers5/person-classifier">github repo</a> with some of these recommendations down the line. If you liked the article or would like further updates, please `watch` the github repo or `star` it. It would motivate me to work on other similar articles.</p><p>So without further ado, I’m listing some recommendations in no particular order.</p><p>1. Better normalization and weight initialization.</p><p>2. Loss weights for each individual class.</p><p>3. Better image augmentation especially <a href="https://github.com/aleju/imgaug">imgaug</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/1*twdVtdNWqsQYjQYYgKflCw.gif" /></figure><p>4. Try different architectures example Inceptionv4, EfficientNet, ResNext.</p><p>5. Loose the regularizations such as dropout.</p><p>6. Try One Cycle Policy and Cyclic Learning Rate, especially the implemenation by <a href="https://www.fast.ai/">fast.ai</a>.</p><p>7. Instead of only GAP use `concatenate(GAP, GMP)`, and build tower after that.</p><p>I hope to implement some of these and see the loss decrease and get better results, also if any of you implement and see some interesting results, would love to hear more about it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b877d21d22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/personattributes-classifier-training-a-multi-task-neural-network-to-detect-attributes-like-age-a3b877d21d22">PersonAttributes Classifier — training a multi-task neural network to detect attributes like age…</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>