<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kalyani Subbiah on Medium]]></title>
        <description><![CDATA[Stories by Kalyani Subbiah on Medium]]></description>
        <link>https://medium.com/@skalyani103?source=rss-1d35f8e1ad96------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vKLIjvh_4h19PvBKh9WakQ.jpeg</url>
            <title>Stories by Kalyani Subbiah on Medium</title>
            <link>https://medium.com/@skalyani103?source=rss-1d35f8e1ad96------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 23:16:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@skalyani103/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Open-Source AI Coding on Local]]></title>
            <link>https://medium.com/@skalyani103/open-source-ai-coding-on-local-dbc143453b07?source=rss-1d35f8e1ad96------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbc143453b07</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[qwen]]></category>
            <category><![CDATA[ai-coding]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[agent-harness]]></category>
            <dc:creator><![CDATA[Kalyani Subbiah]]></dc:creator>
            <pubDate>Sun, 27 Sep 2026 15:58:03 GMT</pubDate>
            <atom:updated>2026-09-27T16:43:11.708Z</atom:updated>
            <content:encoded><![CDATA[<h4>Testing out frameworks on VS Code</h4><h3>Overview</h3><p><em>&lt;This blog is human written&gt;</em></p><p>Word on the street is that open source coding agent harnesses and models are not far behind paid versions like Claude Code and Cursor. I decided to test it out with -</p><p><strong>Setup</strong>: MacBook M4 Pro [24GB RAM] + VS Code + Qwen + Agent Harness</p><h4>Model- [via Ollama]</h4><ol><li>Qwen [9B]</li></ol><p>Gemma 4 is also supposedly good for non-coding stuff.</p><h4>Coding assistants-</h4><ol><li>My own agent harness</li><li>Continue.dev</li><li>Cline</li></ol><p>The IDE I use for personal projects is VS code. I picked a simple repo to connect a device to ChatGPT + execute commands to see how they work with just a few python files + docs instead of something super complex. I feel simplicity is more typical of personal projects.</p><h4>Criteria for evaluation:</h4><ol><li>Give an overview of the repo structure and what the repo does</li><li>Summarize a single specified file in the repo and what it does</li><li>Modify files based on specified command</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z-Y1lixS8SseGRzbsEt8aQ.png" /><figcaption>^I will probably be one of the first victims of the AI apocalypse</figcaption></figure><h3>Qwen + custom harness</h3><p>Downloading a model from Ollama to local was super easy, although deciding which one to use was difficult. I had an overly long conversation with Gemini, and decided to pick <em>Qwen 3.5:9B</em> since my Mac is new and I didn’t want to destroy it. Folks online also suggest <em>Qwen 3.5:14B</em> which also has a quantized version.</p><p>I prompted Gemini to create a custom agent harness script for me, to just read and explain files in the repository- <em>qwen_agent.py</em>. The script allows for chat in terminal like Claude, but without conversation history to keep things simple.</p><p>Despite tweaking the prompt, Qwen kept reading the entire repository contents and spitting out a summary. I tried adjusting the loops and the prompt. I finally threw my hands up in surrender when I started getting answers in Chinese [a fun quirk of Qwen].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcD6-IrYzb_H8mm79y0WIw.png" /><figcaption><em>Et tu Gemini? Even Gemini starting speaking in Chinese to me out of nowhere when Qwen was in the response.</em></figcaption></figure><p>Not so simple after all huh? So there is a product / difficulty in tuning these coding harnesses for open-source models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loKw-nxUhzVp1QyuN2E3Xg.png" /><figcaption><em>My struggle above where Mr. Qwen just keeps calling the project directory and summarizing contents instead of finding/reading the file.</em></figcaption></figure><p>Interestingly, building a custom agent harness to read/write files for the paid version of <em>Claude Sonnet </em>was much easier, probably because they already have solid tool schemas defined for reading/writing files + model with more parameters. So it seems like model sophistication is also super important for building a coding agent, even though harnesses get emphasized.</p><h4><strong>Results</strong></h4><ol><li>Give an overview of the repo structure and what the repo does — <strong>9/10</strong></li><li>Summarize a single specified file in the repo and what it does — <strong>0/10</strong></li><li>Modify a file based on specified command — <strong>Functionality not added</strong></li></ol><h3>Qwen + Continue.dev</h3><p>Continue.dev was cited as a common option, except they do seem to send telemetry data online, so you might want to disable that for privacy. I also had to specify the exact file name for summary using @. However, approving the terminal commands to run got old very fast, and sometimes, I also had to teach it to chunk the files before reading it { It kept trying to just read the top 200 lines of code }</p><h4><strong>Results</strong></h4><ol><li>Give an overview of the repo structure and what the repo does — <strong>7/10</strong></li><li>Summarize a single specified file in the repo and what it does — <strong>6/10</strong></li><li>Modify a file based on specified command — <strong>3/10</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AlI9P4Ocx0lUG1j--1_JLw.png" /></figure><h3>Qwen + Cline</h3><p>Again, super easy to install and set up and I like the UI/window with the token consumption on top. It also allows for running terminal commands automatically with a smart Auto-Approve functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*F_pUNee0Gu9-DcSI7VQCHw.png" /><figcaption>Oops</figcaption></figure><p>Once prompted to look in the subdirectory — it worked nicely-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*Lt9qY9vb2Ra5zqSjJ6nKRg.png" /><figcaption>Now we’re talking</figcaption></figure><p>Not bad for something that is partially open-source ! {At least for now.} Cline has a repo — <a href="https://github.com/cline/cline">https://github.com/cline/cline</a> that I will probably explore with Cline to find out where my harness went wrong.</p><h4><strong>Results</strong></h4><ol><li>Give an overview of the repo structure and what the repo does — <strong>9/10</strong></li><li>Summarize a single specified file in the repo and what it does — <strong>6/10</strong></li><li>Edit/write/modify a file based on specified command — <strong>?/10</strong> [It successfully found an API key in a file, removed it, and moved it to an env file on prompting]. Although I feel this needs more testing.</li></ol><h4><strong>Verdict</strong></h4><p>Claude Code or Cursor is probably still pretty far ahead of free tools/models that can be hosted in local, at least for now. As for Qwen 3.5:9B — maybe a 480B model + GPU can give you more, but you will still have to pay for hosting in cloud to run in-demand or buy a GPU {in which case unless you’re a super heavy user, getting a Claude API will work out better}</p><p>But for personal projects, the free tools out there are not bad. True, I wasted many hours that could be spent working on the actual project on this, but it’s great running something in local that works pretty well without worrying about API costs. It already makes sense to use quantized/distilled models running in local for lighter work right now.</p><h4>Obligatory xkcd</h4><p><a href="https://www.explainxkcd.com/wiki/index.php/1537:_Types">https://www.explainxkcd.com/wiki/index.php/1537:_Types</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbc143453b07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python on AWS Lambda using Docker images]]></title>
            <link>https://medium.com/@skalyani103/python-on-aws-lambda-using-docker-images-5740664c54ca?source=rss-1d35f8e1ad96------2</link>
            <guid isPermaLink="false">https://medium.com/p/5740664c54ca</guid>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Kalyani Subbiah]]></dc:creator>
            <pubDate>Tue, 14 Jun 2022 22:25:20 GMT</pubDate>
            <atom:updated>2022-06-14T22:25:20.809Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS Lambda is a low cost solution for running scripts on an automated basis, whether on a schedule or triggered by some other service. This tutorial will talk about how to implement a AWS Lambda function in python with Docker, by uploading the Docker image to an AWS Elastic Container Registry first. It also briefly shows how to run the Lambda function on a schedule using EventBridge.</p><h4>AWS Lambda</h4><p>When to use AWS Lambda? This AWS documentation on <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html">how to invoke a Lambda function</a> explains how AWS Lambda can fit in a workflow. There are two ways to implement AWS Lambda — 1. Triggered as part of streaming applications, 2. Direct invocation — with EventBridge or when an object is created in S3.</p><p>The answer: If the script has a smaller workload, and needs to be invoked on a schedule or triggered as part of a workflow, Lambda is superior. Further, if the script needs to be run occasionally instead of constantly, Lambda is cheaper and easier to manage than EC2. It comes with a time limit of 15 minutes, which is sufficient for most workflows.</p><h4>Docker</h4><p>Why Docker? Why not use Lambda Layers? Each Lambda layer comes with a 50MB limit, which is usually not enough for multiple python packages, even after they are zipped. The limit for Docker on the other hand, <a href="https://www.google.com/search?q=lambda+docker+limit&amp;oq=lambda+docker+limit&amp;aqs=chrome..69i57j0i512l7j0i457i512j0i512.1838j0j4&amp;sourceid=chrome&amp;ie=UTF-8">is 10GB</a>.</p><p>The drawbacks of Docker: It is unclear on whether Docker installs the packages every time the Lambda function is invoked. This will considerably slow down performance. However, if performance isn’t a major concern, a Docker -&gt; Lambda pipeline is a good solution.</p><h4>Create Lambda handler and Dockerfile</h4><ol><li>It’s important to have all your python package requirements stored in a ‘requirements.txt file’ at the root of your project directory. The best practice for doing this is to create a virtual environment and activate it in the shell:</li></ol><pre>$ python3 -m venv env<br># source env/bin/activate</pre><p>Then, install all your packages and try running your script from the virtual environment. Once it works, run</p><pre>$ python3 -m pip freeze &gt; requirements.txt</pre><p>(It’s good practice to code in a virtual environment from the start)</p><p>2. The code that the Lambda function runs needs to be in a ‘handler’ file in the root of the project directory. The AWS docs on how to create a <a href="https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html">Lambda function handler</a> are helpful.</p><p>3. Install Docker and run the application.</p><p>4. Create a file called ‘Dockerfile’ in the root of your directory and add the following to it</p><pre># lambda base image for Docker from AWS<br>FROM public.ecr.aws/lambda/python:latest</pre><pre># copy all code and lambda handler<br>COPY src ./src<br>COPY lambda_handler.py ./<br>COPY requirements.txt ./</pre><pre># install packages<br>RUN yum install -y gcc-c++ pkgconfig poppler-cpp-devel<br>RUN python3 -m pip install -r requirements.txt</pre><pre># run lambda handler<br>CMD [&quot;lambda_handler.handler&quot;]</pre><h4>Upload Docker image on ECR and Lambda</h4><p>AWS Elastic Container Registry is pretty self-explanatory — it’s a way to upload docker images on AWS to be used by other services.</p><ol><li>First export your AWS account id in the shell or better yet, add it your ~/.bashrc or ~/.bash_profile to avoid having to do this step every time:</li></ol><pre>$ export AWS_ACCOUNT_ID = &lt;account_id&gt;</pre><p>2. Install the AWS CLI and configure with your AWS credentials</p><pre>$ aws configure</pre><p>3. Paste the following to a file called ‘deploy-lambda.sh’:</p><pre>URL_STRING=&quot;.dkr.ecr.us-east-1.amazonaws.com&quot;</pre><pre>CONTAINER_STRING=&quot;&lt;project&gt;&quot;</pre><pre>IMAGE_STRING=&quot;latest&quot;</pre><pre>ECR_IMAGE_URI=&quot;$AWS_ACCOUNT_ID$URL_STRING/$CONTAINER_STRING:$IMAGE_STRING&quot;</pre><pre># log in to ECR</pre><pre>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin &quot;$AWS_ACCOUNT_ID$URL_STRING&quot;</pre><pre># remove previous images to save space<br>docker rmi &quot;$AWS_ACCOUNT_ID$URL_STRING/$CONTAINER_STRING&quot;<br>docker rmi &quot;$CONTAINER_STRING&quot;</pre><pre># build image<br>docker build --tag &quot;$CONTAINER_STRING&quot; .</pre><pre># tag and push to AWS ECR<br>docker tag $CONTAINER_STRING:latest &quot;$ECR_IMAGE_URI&quot;<br>docker push &quot;$ECR_IMAGE_URI&quot;</pre><h4>Create a Lambda Function and Attach Image</h4><ol><li>Create Lambda function and attach your ECR Image with the following settings</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3XfDRqdysFU4OUwld1pqPw.png" /></figure><p>2. Increase the ‘Timeout’ setting in the Configuration tab to 5 minutes or more.</p><p>3. Try testing your Lambda function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j9ujpjr_d6XUhcOnnhHDtw.png" /></figure><p>4. Add the following at the end of the bash script ‘deploy-lambda.sh’ to make updates easier:</p><pre># update Lambda<br>aws lambda update-function-code --function-name $CONTAINER_STRING --image-uri &quot;$ECR_IMAGE_URI&quot;</pre><h4>Add EventBridge Rule</h4><ol><li>Follow the instructions to create an EventBridge Rule for Lambdas from <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html">https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html</a></li><li>You’d have to look up <a href="https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm">Cron expressions</a> to figure out the syntax for the schedule. For ex. to run the script twice a day at 8:00 am and 5:00 pm, use:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*WsnQJeccsU7GiNsCKT-QaQ.png" /></figure><p>Done! The Lambda function will run on a schedule now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5740664c54ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Install and run Airflow on an AWS EC2 instance (Ubuntu)]]></title>
            <link>https://awstip.com/how-to-run-airflow-on-an-aws-ec2-instance-ubuntu-553bdcb1cfd2?source=rss-1d35f8e1ad96------2</link>
            <guid isPermaLink="false">https://medium.com/p/553bdcb1cfd2</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[ec2-instance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Kalyani Subbiah]]></dc:creator>
            <pubDate>Fri, 27 May 2022 20:15:28 GMT</pubDate>
            <atom:updated>2022-06-13T08:48:50.249Z</atom:updated>
            <content:encoded><![CDATA[<p>Why run Airflow on EC2? The reasons could be one or more of the following: processing power without taxing local machines, better speed or more memory. But running Airflow on EC2 is not as easy as it seems. Here’s the easiest way to run Airflow on AWS EC2 for a beginner without complicated setup scripts:</p><h4>I. Create an Ubuntu EC2 instance</h4><ol><li>Launch a new Ubuntu EC2 instance (at the very least a t3.medium instance).</li><li>Create a new Security Group with the following permissions for Inbound Rules: SSH on port 22, HTTP on port 80, TCP on port 8080 all from your IP.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xeo_kMZGyStWfmXwBV62Pw.png" /></figure><p>3. Attach the Security Group to your EC2 instance.</p><p><strong>II. Install and Configure Airflow</strong></p><ol><li>SSH into the instance using a key file OR use EC2 instance connect (at the time of writing EC2 instance connect was buggy for Ubuntu instances).</li><li>Run the following commands (from this <a href="https://linuxpip.org/install-airflow/">source</a>)</li></ol><pre>$ sudo apt install python3-pip</pre><pre>$ sudo apt-get install software-properties-common</pre><pre>$ sudo apt-add-repository universe</pre><pre>$ sudo apt-get update</pre><pre>$ sudo apt-get install python-setuptools</pre><pre>$ sudo apt install python3-pip</pre><pre>$ sudo apt-get install libmysqlclient-dev</pre><pre>$ sudo apt-get install libssl-dev</pre><pre>$ sudo apt-get install libkrb5-dev</pre><p>3. A good practice before running the next two commands is to create a python virtual environment and activate it:</p><pre>$ pip3 install apache-airflow[s3,aws,postgres]</pre><pre>$ pip3 install typing_extensions</pre><p>4. Create a directory called ‘airflow’ in /home/ubuntu/.</p><pre>$ cd ~</pre><pre>$ mkdir airflow</pre><p>4. Add AIRFLOW_HOME to your environment variables in ‘/etc./environment’ (this like ~/.bashrc or ~/.bash_profile is where Ubuntu stores its environment variables for its bash terminal)</p><p>For ex.</p><pre>sudo vim /etc/environment</pre><p>and then add</p><pre>AIRFLOW_HOME = ‘/home/ubuntu/airflow’</pre><p>5. Run the following to initialize the Airflow metadata DB and a user login:</p><pre>$ airflow db init</pre><pre>$ airflow users create — username admin — password &lt;password&gt; — firstname Anonymous — lastname Admin — role Admin — email admin@example.org</pre><h4>III. Running your DAGs</h4><ol><li>Airflow dags need to reside in a folder specified in the ‘dags_folder’ variable, found in the config file: ‘airflow.cfg’. In a new terminal window, copy your dags folder from local to EC2 using the scp command.</li></ol><p>(In local terminal — NOT sshed into EC2):</p><pre>$ scp -i &lt;key_file&gt; -r airflow/dags ubuntu@&lt;ec2-path&gt;:/home/ubuntu/airflow</pre><p>2. Go back to the terminal window where you are SSHed into EC2. Install all python dependencies required for the dag. Hopefully you have them stored in a ‘requirements.txt’ file so all you have to do is activate your virtual environment and run</p><pre>$ pip3 install -r requirements.txt</pre><p>3. Now run:</p><pre>$ airflow webserver</pre><p>4. Open a new terminal window and SSH into the EC2 instance. Run:</p><pre>$ airflow scheduler</pre><p>5. Find your EC2 instance’s Public IP address. Now go to &lt;PublicIP&gt;:8080 on your browser, login and run your dag.</p><p>Notes:</p><ol><li>Some like to run the webserver and scheduler as daemons using the -D flag or system. I prefer to run them separately to monitor for warnings and errors, and to avoid having to kill processes.</li><li>Ideally it would make sense to package and run dags as Docker containers, but this is a good workflow for your first time running Airflow on EC2 to avoid confusion.</li></ol><p>And there you have it! Now you can sit back and let the EC2 instance do the hard work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*KJqDh0IdSfdQCbWbIlXulA.jpeg" /><figcaption>meme credit: <a href="https://securityqueens.co.uk/there-aint-no-party-like-an-ec2-party-securing-your-aws-instance/">https://securityqueens.co.uk/there-aint-no-party-like-an-ec2-party-securing-your-aws-instance/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=553bdcb1cfd2" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/how-to-run-airflow-on-an-aws-ec2-instance-ubuntu-553bdcb1cfd2">How to Install and run Airflow on an AWS EC2 instance (Ubuntu)</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to make an Open-Source Pull Request]]></title>
            <link>https://medium.com/@skalyani103/how-to-make-an-open-source-pull-request-bf273d023639?source=rss-1d35f8e1ad96------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf273d023639</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[version-control]]></category>
            <dc:creator><![CDATA[Kalyani Subbiah]]></dc:creator>
            <pubDate>Sat, 14 May 2022 15:23:14 GMT</pubDate>
            <atom:updated>2022-05-14T15:23:54.533Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently made my first open-source pull request. Here are the <strong>steps</strong>:</p><ol><li>Find an issue — Stumbling across an improvement or bug <strong>organically</strong> while using the package is your best bet.</li><li><strong>Fork</strong> the repo. I forked <a href="https://github.com/apache/airflow">apache/airflow</a> to &lt;github-username&gt;/airflow.</li><li><strong>Clone</strong> the repo to a local directory.</li><li>Type in your required changes, making commits along the way. The code you write or edit should be cohesive in style and substance with the rest of the module/repo. It’s also important to make sure the code is <strong>backwards-compatible</strong> for users unless absolutely unavoidable, in which case you issue a <a href="https://docs.python.org/3/library/exceptions.html#DeprecationWarning"><strong>DeprecationWarning</strong></a>. (In which case, don’t open the door if someone shows up with a pitchfork.)</li><li>Edit or add new <a href="https://docs.pytest.org/en/7.1.x/"><strong>unit tests</strong></a> to check if the changes are working.</li><li>If it’s a significant edit, check the <strong>documentation</strong> to make sure its still compatible.</li><li><a href="https://medium.com/@topspinj/how-to-git-rebase-into-a-forked-repo-c9f05e821c8a"><strong>Rebase</strong> your forked repo</a> with the upstream repo (the open source repo you are editing)</li><li><strong>Push</strong> to your forked repo. Run all your tests again.</li><li>Create a new <strong>pull request</strong> on the GitHub GUI and hit ‘compare across forks’. Add the issue number being addressed to your PR.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCx9ln99yYcQp1P8jKCzdA.png" /></figure><p>10. Keep track of build checks, reviews and comments, and add changes with fresh commits as required.</p><p>11. If and when your PR gets merged:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*pYeUkI6E9QBCUKsRw0Stew.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf273d023639" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where’s my train?]]></title>
            <link>https://medium.com/@skalyani103/wheres-my-train-9c8431c76b7f?source=rss-1d35f8e1ad96------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c8431c76b7f</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[mta]]></category>
            <category><![CDATA[gtfs]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Kalyani Subbiah]]></dc:creator>
            <pubDate>Sun, 13 Feb 2022 15:59:46 GMT</pubDate>
            <atom:updated>2022-02-13T15:59:46.542Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to build an NYC subway tracker app using React and Flask</h3><p>Whenever I leave my apartment, I check google maps for the next train arriving at my stop. There are alternatives! The MTA has a <a href="https://map.mta.info/#@40.7230687,-73.9428871,12.56z">snazzy app</a>, and there is a <a href="https://www.raspberrypi.com/news/nyc-train-sign/">Raspberry Pi project</a> to build a similar board. But wouldn’t it be nice to have a simple web app similar to this?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*kl0LxBhLqIwc4jnYJ-F38w.jpeg" /><figcaption>Image source: <a href="https://nymag.com/news/intelligencer/70983/">https://nymag.com/news/intelligencer/70983/</a></figcaption></figure><h3>MTA Realtime Feeds</h3><p>The MTA has <a href="https://developers.google.com/transit/gtfs-realtime">real-time feeds</a> for all the different lines. These endpoints are available once you register and obtain an API key.</p><p>The format for these feeds is an esoteric standard that the transportation data community uses- called <a href="https://developers.google.com/transit/gtfs">GTFS</a>. GTFS Realtime is the format used by the API to transmit information on trip updates, service alerts and vehicle positions. GTFS in turn uses Google’s standardized version of XML, called a <a href="https://developers.google.com/protocol-buffers">protocol buffer</a>.</p><h3>Flask API</h3><p>Let’s first build a backend Flask API for train arrival times. The backend takes the GTFS feed files from the MTA and processes it into a JSON format that the frontend app can use.</p><p>The flask api is available at: <a href="https://github.com/kalyani-subbiah/mta-api">https://github.com/kalyani-subbiah/mta-api</a></p><p>Here is the feed parser using two packages from Google: google.transit and google.protobuf. It pulls the data from the <a href="https://api.mta.info/#/subwayRealTimeFeeds">MTA Subway Realtime Feeds</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec79dd3db3318d6cc72b613b8c9a84e3/href">https://medium.com/media/ec79dd3db3318d6cc72b613b8c9a84e3/href</a></iframe><p>Next, download a<a href="http://web.mta.info/developers/data/nyct/subway/google_transit.zip"> zip file</a> containing metadata on stops and routes from the <a href="http://web.mta.info/developers/developer-data-terms.html#data">MTA’s Static Data Feeds</a>. We will use two files: named stops.csv and routes.csv.</p><p>In order to clean and structure our data for the API there are three things we need to do:</p><h4>I. Match the stop ids to station names</h4><p>For ex. stop id ‘J12’ is 121 St Station. Some stations have multiple stop ids. Since it’s a one-to-many relationship, we create our own station ids based on the station name and link them to stop ids.</p><p>We usestops.csv to get the station names. This preprocessing is done in the stations.py file in <a href="https://github.com/kalyani-subbiah/mta-api">https://github.com/kalyani-subbiah/mta-api</a>.</p><p>The end result is available in the url /api/stations/.</p><h4>II. Store the arrival times for each station/route</h4><p>The real-time feed contains information on trip updates. Each trip update has a stop id, a route id (A, B, C, 1, 2, 3 etc.) , and an arrival time. We filter the objects that contain the key trip_updates from the feed. Then, preprocess them to extract and sort the arrival times for each station and route.</p><p>This preprocessing is done in the times.py file in <a href="https://github.com/kalyani-subbiah/mta-api">https://github.com/kalyani-subbiah/mta-api</a>. The end result is stored in the route /api/train_times/. Results are available for each station id at /api/train_times/&lt;station_id&gt;.</p><h4>III. Store the colors for each route</h4><p>For ex. route id ‘2’ to color EE352E.</p><p>We use routes.csvto get the route colors. This preprocessing is done in the routes.py file in <a href="https://github.com/kalyani-subbiah/mta-api">https://github.com/kalyani-subbiah/mta-api</a>.</p><p>The end result is available at the url /api/routes/.</p><h3>React Frontend</h3><p>The finished app looks like this and is available at <a href="https://github.com/kalyani-subbiah/subway-times-app">https://github.com/kalyani-subbiah/subway-times-app</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JpsLt0sJCBTxK8BKNt_QbQ.png" /></figure><p>The dropdown lets you select a stop using Select from the package react-select . Once selected, trains are displayed in ascending order of time to arrival.</p><p>The React frontend requests the Flask API for three different pieces of information from each of these three endpoints:</p><ol><li>/api/stations/ — The station names for the dropdown. The <a href="https://reactjs.org/docs/hooks-effect.html">useEffect</a> hook requests this endpoint when the app is first loaded.</li><li>/api/stations/&lt;selected-station&gt;/ — Train routes arriving at the stop and the latest times. Once a station is selected, a callback function getStopInfo is run and this endpoint is requested.</li><li>/api/routes/ — getStopInfo also sends a request to pick a color for each route, for ex. the color blue for the A train.</li></ol><p>To run the app, follow the instructions in both repos and make sure they are running concurrently. (The Flask API will run on Port 5000 and the React app will run on Port 3000 by default.)</p><h3>Future Extensions</h3><p>I recently decided to dive into full stack development to learn the fundamentals of how the web works. I finished 6 parts of the <a href="https://fullstackopen.com/en/">Full Stack Open Course</a> (which is excellent!) and decided to work on a project using my newfound React skills.</p><p>This project took me a few weeks and was a fun exercise in both Flask and React. I still have to ponder best practices for my code and how to take this further. There are three future extensions that I’m planning for this project when I get the time to come back to it:</p><ol><li>Automatically select a station based on a user’s location.</li><li>Find a way to show alerts and delays real time in a better way.</li><li>Log the train arrival times to a DB to produce analysis on delays.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c8431c76b7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>