<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Surya Chakraborty on Medium]]></title>
        <description><![CDATA[Stories by Surya Chakraborty on Medium]]></description>
        <link>https://medium.com/@suryashekhar?source=rss-3f2a779a2e4d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*0-oO1tWS5inTpVPJ-o4Vog.jpeg</url>
            <title>Stories by Surya Chakraborty on Medium</title>
            <link>https://medium.com/@suryashekhar?source=rss-3f2a779a2e4d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:23:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suryashekhar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Wrote a Pipeline to Publish the Best of Reddit to Instagram]]></title>
            <link>https://medium.com/better-programming/i-wrote-a-pipeline-to-fetch-posts-from-reddit-and-post-them-to-instagram-d073e55fd258?source=rss-3f2a779a2e4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d073e55fd258</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Surya Chakraborty]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 15:28:24 GMT</pubDate>
            <atom:updated>2020-07-07T14:41:29.791Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here’s what I learned</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*waTVY4uYjm8DRYpz" /><figcaption>Photo by <a href="https://unsplash.com/@impatrickt?utm_source=medium&amp;utm_medium=referral">Patrick Tomasso</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>If you’ve been on Instagram long enough, you’ll have seen profiles posting screenshots of interesting Reddit posts. That makes sense if you think about it: Reddit is heavily moderated. This ensures only quality posts show up on your feed.</p><p>Combine that with Instagram, a social medium with over 1 billion active accounts per month, and you have a recipe for success.</p><p>Now if you see a sample post from one of these Instagram pages, you will realize that it takes quite a bit of effort to create these posts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCxY7llSZ0RHX9Rw-uDkMw.png" /><figcaption>A whole lot of effort</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yThusiLbE6Yqtwu9zjqBzg.png" /></figure><p>First, the owner of the page has to scour Reddit to find posts worth posting.</p><ol><li>They will judge the Reddit posts by relevance and pick the ones worth posting.</li><li>Then, they have to take a screenshot of a post and paste it on a black background to give it Instagram-friendly dimensions.</li><li>They have to think of a caption that is related to the post and will drive engagement.</li><li>They have to come up with hashtags to boost the visibility of the post.</li><li>And, trivially, they have to remember all the posts they have made — it would not make sense to post the same content twice.</li></ol><p>I started wondering, “Is there a way we can make their life easier through… <em>automation</em>?”</p><h3>Lesson #1: Plan It Out Before Diving Headfirst</h3><p>My first mistake was sorting everything in my head and just “coding it up.”</p><p>I started exploring the APIs that Reddit and Instagram provide. Within a day, I realized that this was not going to be a one-weekend project. The APIs were not sufficient for what I had in mind. For instance, Reddit would provide the text of the post. I had to paste the text on the image myself.</p><p>I created a doc to track the important milestones:</p><ol><li>Scrape Reddit for top posts.</li><li>Create an image out of the text.</li><li>Upload it to Instagram.</li><li>Automate the entire thing.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-iXj-Q-p858vc205ne8qg.jpeg" /><figcaption>Photo by the author.</figcaption></figure><h3>Lesson #2: Don’t Over-Engineer It. The Simplest Solution May Be Staring You in the Face</h3><p>What this planning would have also saved me is a lot of heartache in creating the database of posts.</p><p>I needed to save the posts so that I knew which ones have already been uploaded. Think about it: If I request the “Top 10 posts this month” from Reddit on two consecutive days, it’s almost impossible that there won’t be any overlap. So I needed to save the posts to a database and avoid appending posts if they had already been fetched.</p><p>Initially, I started doing this through a local CSV. But as the project progressed, I started wondering, “Hey, if I spill coffee on my laptop, my DB is gone. Poof!”</p><p>To ensure data backup, I had to use a database living on the internet. I started exploring the options. I had seen folks at my workplace use <a href="https://firebase.google.com/">Firebase</a>. Would that be the best solution? Or do I explore the features of <a href="https://www.heroku.com/">Heroku</a> that I had seen folks at my college use?</p><p>I sat down to brainstorm with my far more experienced colleague Puneet to create something robust. After an hour’s worth of weighing all the different options, Puneet asked, “Remind me, Surya, why do we need a <em>database</em> database again?”</p><p>We realized that we only needed basic operations on a data table: Add a row every time there’s a new post to upload, update a row once it has been uploaded, and query the rows to see which ones have yet to be uploaded. We weren’t performing complex joins. If internet backup was our only concern, why not simply use<em> </em>Google Sheets?</p><h3>Lesson #3: It Pays to Refactor. I Only Had to Change Certain Function Calls From CSV to GSheets</h3><p>In the spirit of automation, we had to use the GSheets API rather than, you know, <em>any</em> sort of manual labour. It took some effort to set up the credentials, but soon it was up and running. However, the API only provided some blanket options. I wrote a wrapper on top of the API so that it could use GSheets as a DB.</p><p>When the time came to replace the local CSV-based approach with the online GSheets-based approach, I thanked my past self for obsessively modularizing and refactoring my code. I did not have to bother with all the locations where I may have used the CSV operations.</p><h3>Lesson #4 (That Your Scrum Manager Won’t Admit): Timelines Are Treacherous</h3><p>Surprisingly, the most time-consuming part of setting up this entire pipeline was creating an image out of the text.</p><p>Since there are no out-of-the-box solutions available, I had to modify an existing image manipulation library (<a href="https://pypi.org/project/Pillow/">Pillow</a>) to accommodate features like dividing an image and keeping the text in each division centrally aligned between the borders so that it looks good on an Instagram feed. That took a good part of two weekends.</p><h3>Lesson #5: Pick Your Battles. You Cannot Get Stuck in the Details</h3><p>Avishek, who was helping me with the aesthetic part of the images generated, is picky about his fonts. He got me to use a font that isn’t there on a standard font palette. Since I wanted this project to be easily reproducible, I had to add this font to the bundle. While figuring out a way to “check in” the font, I dug deep inside the image manipulation code only to find out that the font is being picked up directly from the system fonts (and not from say a font file location).</p><p>Now sure, I could spend some time to see if there was a way to pick up a font from a location. But at that point, I decided that it’s far simpler to add a note in the README to download the font from a website.</p><h3>Conclusion</h3><p>And here we are: The pipeline is ready and the code is checked in. The only remaining part is to figure out a way to trigger the daily upload automatically. Is Cron the right tool for the job? Colleague Bhavya suggested using AWS Lambda. Or is a much simpler solution staring us in the face?</p><p>If you want to look at the code or replicate it, here’s the well-documented <a href="https://github.com/suryashekharc/insta_reddit">GitHub link</a>. Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d073e55fd258" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/i-wrote-a-pipeline-to-fetch-posts-from-reddit-and-post-them-to-instagram-d073e55fd258">I Wrote a Pipeline to Publish the Best of Reddit to Instagram</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A new way to visualize complex Python objects]]></title>
            <link>https://medium.com/analytics-vidhya/a-new-way-to-visualize-complex-python-objects-bbea595bb1c9?source=rss-3f2a779a2e4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbea595bb1c9</guid>
            <category><![CDATA[print]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[schema]]></category>
            <dc:creator><![CDATA[Surya Chakraborty]]></dc:creator>
            <pubDate>Sat, 29 Feb 2020 05:28:02 GMT</pubDate>
            <atom:updated>2020-03-18T08:14:22.779Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQWwXSpODQGW4i3oe_DIXA.png" /></figure><p>At a fast-paced startup like the one where I work, every day throws a new challenge at you. And on this one (fateful?) day, I was looking for bus-routes… in Colombia. Here’s a screengrab from the <a href="https://transit.land/api/v1/routes.geojson?operated_by=o-d2g6-transmilenio~alimentadores&amp;per_page=false">website</a> which provides the geo-data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2hUBcxfJtcpGqXDO" /><figcaption>Props if you tried to read it.</figcaption></figure><p>Now, I wanted to convert this data into a GeoJSON. A <strong>GeoJSON</strong> is simply a standardized JSON format to represent geographical features, along with their non-spatial attributes (e.g <em>operated_by_name</em> and <em>vehicle_type</em>).</p><p>To convert the geo-data in the screengrab above into the GeoJSON format, I needed to understand the structure of the geo-data itself. If I began to eyeball this text, I would need a new pair of glasses.</p><p>So I resorted to <strong>Python</strong>. I decided, <em>“Let&#39;s read this data on Python as a JSON and then use the inbuilt </em><em>pprint method to display the JSON. Surely then, the structure would be easily visible, right?”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xB0qPm5cVamuDuUH" /><figcaption>Wrong.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aD7Xf3kRSGTNPoFH" /><figcaption>Still wrong.</figcaption></figure><p>This JSON was extremely long-winded and pretty-printing did not allow me to get a sense of the geo-data structure that was there. The other options were to json.dumps() or yaml.dump()the object, and they were not very understandable either, since the route data it contained, a lengthy array of latitudes and longitudes, when displayed would make it difficult to navigate the structure.</p><p>So, I decided to do it the old-school way.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FoO2XEhK1w5Frq%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FoO2XEhK1w5Frq%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FoO2XEhK1w5Frq%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/da63167239c1af40b6835248dcc8dc32/href">https://medium.com/media/da63167239c1af40b6835248dcc8dc32/href</a></iframe><p>I pulled out my notebook and painstakingly figured out the schema of the JSON, matching brackets and indents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OTBpf-uYtmUR98wr" /><figcaption>Note how my writing trails off after a point</figcaption></figure><p>Now I’m used to printing the schema of complex dataframes in Spark using the printSchema() function, so I was a tad bit disappointed that Python did not provide support for that.</p><p>So I did what any engineer would do:</p><blockquote>Write <a href="https://github.com/suryashekharc/print_schema">code</a> that would print the schema.</blockquote><p>I wrote a very generic code that will work with most of the common Python objects — dictionaries, JSONs, and lists. The program will recursively print the structure of the object, emitting details wherever possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0ibUCu0NWdpSmNwd" /><figcaption>Much better, yes?</figcaption></figure><p>This way of printing the schema has many advantages.</p><p>For example, if I want to know how many bus-routes are there in the geo-data, I can just look at the count of the <em>“features”</em> list, 95 in this case. If I want to know what the dictionary “<em>geometry</em>” contains, I know it has keys “<em>type</em>” and “<em>coordinates</em>”, whose values are string and list respectively. I also enabled features like choosing the indent of every level, and if you want continuous vertical lines for a schema.</p><p>I was already checking all this code into <strong>GitHub</strong>, but now I became ambitious.</p><blockquote>I wanted to deploy the project to PyPI so that you too can download the package and use it as a pprint alternative (if you want, no pressure).</blockquote><p>I followed this <a href="https://realpython.com/pypi-publish-python-package/">guide on RealPython</a> on how to publish a package to PyPI. I published version 1.0.0 on PyPI and I could even download it from other machines. It felt almost… too easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IXJTIbI0wZqEfnBf" /><figcaption>JSONOFABITCH</figcaption></figure><p>My hunch was right. Even though I could download it, it was not working as expected on other machines. The import statements were not right (<em>module not callable</em>), I was getting funny errors on Python2, and the PyPI page was not picking up the readme markdown file. Github used to magically read the markdown file, but why was PyPI not being smart enough?</p><p>I sat with my colleague Puneet after work one day, and together we began debugging the issues. Turns out, there were a few more things left to do. PyPI requires the setup.py file to mention a long_description(which we decided to <a href="https://github.com/suryashekharc/print_schema/blob/master/setup.py">read dynamically from the readme file</a>). And, to enable the correct level of importing, we made <a href="https://github.com/suryashekharc/print_schema/blob/master/print_schema/__init__.py">this change</a> to the init file.</p><p>Plus, my code needs Python3 (I used fancy type-hinting in the code that is not supported by Python2), so I added the python_requires field to <a href="https://github.com/suryashekharc/print_schema/blob/master/setup.py">setup.py</a>. We put this entire contraption on <strong>test.pypi</strong> — a sandbox that allows you to test out a package before it is deployed to PyPI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Hupwzku7s22WKcD" /><figcaption>Looks like we’re all set!</figcaption></figure><p>PyPI quite rightly does not allow you to upload the same version of a package twice, so we pushed the new (and functional!) print-schema version 1.0.2. My first contribution to PyPI is now alive and kicking!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qq0mbKAGkP6Wb5qu" /></figure><p>I encourage you to download with pip install print-schema. Here are the <a href="https://github.com/suryashekharc/print_schema">Github link</a> and the <a href="https://pypi.org/project/print-schema/">PyPI link</a>.</p><p>Do let me know what you think of it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbea595bb1c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-new-way-to-visualize-complex-python-objects-bbea595bb1c9">A new way to visualize complex Python objects</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow: The First Act of the Data Engineering Team]]></title>
            <link>https://medium.com/zendrive-tech/airflow-the-first-act-of-the-data-engineering-team-9f2acd07f54a?source=rss-3f2a779a2e4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f2acd07f54a</guid>
            <category><![CDATA[tech-blog]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Surya Chakraborty]]></dc:creator>
            <pubDate>Wed, 11 Dec 2019 06:57:48 GMT</pubDate>
            <atom:updated>2019-12-11T06:57:48.760Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Engineering</h4><h4>For when Cron wasn’t Good Enough.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kq7k2c3oTzqRaBDH_P9sIw.png" /><figcaption>(From the editors: Happy December and Happy Holidays!)</figcaption></figure><p>It was the spring of 2019 when the Data Engineering team was officially created at Zendrive. The charter was simple: manage the <em>petabytes</em> of time-series data available to us.</p><p>There was a lot to do then, and there’s even more to do now.</p><h3>Can’t Be Manual</h3><p>When we sat down to figure out the current challenges in our processes, one in particular stood out. A sheer number of processes would be run<strong> manually</strong> whenever the need arose. So, too often we were deploying resources — personnel and our Spark cluster — to handle the task. It was sapping too many of our resources and too much of our attention.</p><p>Take one of our core technologies, the Hard Braking Detector- which does exactly what it says on the tin. Let’s say we are running a version v5 of the detector on the field, while we have also <strong>simultaneously </strong>developed a v6.</p><p>Now we want to test out how they compare. This process would require you to load up the data for say, 30 days. Then you extract the hard braking numbers for each version, then you compare their performance numbers in different segments: at the trip-level, then at the phone-level, even at the OS-level.</p><p>Each of these aggregate operations is <strong>expensive to compute</strong> when you are dealing with <strong>millions of trips</strong> getting recorded every day.</p><p>So we decided, “<em>Let’s make a dashboard out of this!</em>” We wanted to automate it so that the dashboard would get populated with aggregate stats daily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xr4zqpB1DIMWMYtO" /><figcaption>Like this one.</figcaption></figure><h3>Cron Sucks (in this situation)</h3><p>For automating a recurring job, our first instinct was to run a Cron job. But Cron, as you know, comes with its limitations. Even ignoring the lack of an interface to monitor jobs or the difficulty in debugging a failed Cron job, the main drawback we wanted to avoid is the lack of dependency management. We wanted to adopt a solution that would <em>scale</em> — and that’s where Airflow came in.</p><h3>But… what is Airflow?</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FW4zO9PSNXYMiYefZ4T%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fcat-what-capoo-W4zO9PSNXYMiYefZ4T&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FW4zO9PSNXYMiYefZ4T%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="380" frameborder="0" scrolling="no"><a href="https://medium.com/media/2476a29242989c942f2bd144dd141849/href">https://medium.com/media/2476a29242989c942f2bd144dd141849/href</a></iframe><p>Take it straight from the<a href="https://airflow.apache.org/docs/stable/"> docs</a>: <em>“Airflow is a platform to programmatically author, schedule and monitor workflows. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.”</em></p><p><strong>Basically, it schedules work as it should be.</strong></p><p>Airflow uses directed acyclic graphs (DAGs) to manage workflow orchestration. Tasks and dependencies are defined in Python and then Airflow manages the scheduling and execution.</p><p>In our case, our program has two main steps: compute the numbers, and push the aggregate stats to our BigQuery data warehouse (chosen for ease of visualization via DataStudio). By now, you may realize where the dependency management fits in.</p><p>We need to first <em>finish</em> computing the numbers before injecting them into BigQuery. So we designed a metrics generator and a metrics injector class.</p><p>Since Cron does not care about dependencies, it would be a pain to maintain such a job. If the preceding job finished on time, all would be fine. However if it exceeded its time, things were bound to break.</p><p>That’s where Airflow stepped up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*ZglE7nYYbec7rvt4" /><figcaption>Inspired by<a href="https://www.slideshare.net/mesodiar/intro-to-airflow-good-bye-cron-welcome-scheduled-workflow-management"> a presentation</a> by Burasakorn Sabyeying on Cron vs Airflow</figcaption></figure><p>We wrote the corresponding DAG*:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b29dda493b9876e5d5349c43415c588/href">https://medium.com/media/9b29dda493b9876e5d5349c43415c588/href</a></iframe><p>*(Note that the DAG is in Python. So simple!)</p><p>What if one task fails? Airflow allows you to set the number of times you’ll retry.</p><p>What if there are old tasks to rerun? It lets you backfill.</p><p>Want to integrate it with your Jenkins-powered CI/CD pipeline? There are CLI options to get the state of a job!</p><p>We, in the data engineering team, love Airflow for the powerful functionalities it offers. Currently, we maintain 13 Airflow jobs in production — ranging from DAGs that are used by one of our core algorithms for model building and subsequently model prediction, to DAGs that are used for integration-testing production code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-pztFL1eeXaEI987" /><figcaption>(Text redacted)</figcaption></figure><p>I am quite fond of the dashboard. It allows us to monitor all tasks at a glance, identify issues quickly, firefight in real-time, and fix them before an angry email.</p><p>The best part? More UI, less code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f2acd07f54a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zendrive-tech/airflow-the-first-act-of-the-data-engineering-team-9f2acd07f54a">Airflow: The First Act of the Data Engineering Team</a> was originally published in <a href="https://medium.com/zendrive-tech">Zendrive Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>