<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashwin M Murugappan on Medium]]></title>
        <description><![CDATA[Stories by Ashwin M Murugappan on Medium]]></description>
        <link>https://medium.com/@ashwin_muthiah?source=rss-d05ad5d28c5e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8NaqT7IsQknDPRAE45teDw.jpeg</url>
            <title>Stories by Ashwin M Murugappan on Medium</title>
            <link>https://medium.com/@ashwin_muthiah?source=rss-d05ad5d28c5e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:22:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashwin_muthiah/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Scraping NBA Stats is cooler than Michael Jordan]]></title>
            <link>https://medium.com/data-science/how-scraping-nba-stats-is-cooler-than-michael-jordan-49d7562ce3ef?source=rss-d05ad5d28c5e------2</link>
            <guid isPermaLink="false">https://medium.com/p/49d7562ce3ef</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nba]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Ashwin M Murugappan]]></dc:creator>
            <pubDate>Wed, 26 Aug 2020 03:25:52 GMT</pubDate>
            <atom:updated>2020-12-31T14:17:23.046Z</atom:updated>
            <content:encoded><![CDATA[<h4>Finding API endpoints and Client-side web scraping</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rNwXSHbf0dAxa3bA" /><figcaption>Photo by <a href="https://unsplash.com/@valerysysoev?utm_source=medium&amp;utm_medium=referral">Valery Sysoev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>This summer I picked up a new hobby of following the NBA, as a data enthusiast, I wanted to understand how this season of NBA was different on paper from the previous ones as this was played inside the bubble without any fans.</p><p>For acquiring relevant data I started by utilizing the python library beautiful soup. However, to my surprise, the data wasn’t stored on the HTML Source page. After some digging, I discovered that the NBA stats website was built with an AngularJS, which means that the site is rendered client-side instead of server-side.</p><h4>What is Client-side rendering</h4><p>The HTML which is rendered is only a template and it doesn’t hold any data, The Javascript in the server response fetches the data from an API and uses it to create the page client-side.</p><p>Basically, when you view the pages source code, you wouldn’t find the data but just a template of the webpage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ay35LCsJn_9UJr6OuSsHg.png" /><figcaption><a href="https://stats.nba.com/players/traditional/?SeasonType=Regular%20Season&amp;sort=PTS&amp;dir=-1&amp;Season=2019-20">NBA States Website</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pGIksJJN5pAu6wTg2_hVtQ.png" /><figcaption>Ctrl + U takes you to the Page source</figcaption></figure><h3>Let’s Get Started</h3><p>In this article, we will be scraping the NBA stats website for the <a href="https://stats.nba.com/players/traditional/?SeasonType=Regular%20Season&amp;sort=PTS&amp;dir=-1&amp;Season=2019-20">League player’s stats.</a> After hours of researching, I decided to adopt a process that was a lot simpler than Beautiful soup.</p><h4>Finding the API endpoint from the website</h4><p>The first step is to open the webpage you want to scrape on your web browser (preferably google chrome or firefox) and open the developer tools. To do this just right click select inspect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*jG_Fmj6ERa_r_joh1bWceg.jpeg" /><figcaption>Right Click Followed by Inspect</figcaption></figure><p>This would open a panel either to the right or the bottom of the page, Select network and XHR, and reload the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ex45rW_LWZVAG7fu2YehIg.jpeg" /><figcaption>Inspect panel</figcaption></figure><p>Once we reload the page, all the requests from the page will be visible. At this point, you should do some digging to find the request you want. Most probably the endpoint would be named after the webpage you are looking at.</p><p>Since we are looking at the league players’ stats page, the endpoint might be named something similar. Select every option and preview the results to find the correct endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sRbHnULuPeJjQ2EM0CEGkQ.jpeg" /><figcaption>Select and Preview</figcaption></figure><p>Once you find the right endpoint you are all set to move to the next step.</p><h4>Calling the API endpoint to get the data</h4><p>In order to call the API, we would make use of the <a href="https://requests.readthedocs.io/en/master/user/quickstart/#custom-headers">requests</a> python package. In order to do so, We need 3 components as part of the request syntax below.</p><p>The first part is the URL, in our case since we are accessing the league player stats, we can get it from the last step.</p><p>Under the Header tab, select general and copy the first part of the request URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fxWr9AiFQkdXS6Y3p0eKRw.jpeg" /><figcaption>Request URL</figcaption></figure><p>Next, we need the request headers, Which can also be found under the same Header tab but under “Request Headers” Subsection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FWkHO5B8A_0yf9Xwswp_ng.jpeg" /><figcaption>Request Header</figcaption></figure><p>Header as a dictionary</p><p>The final component we need is the parameters, which can be found under the “Query String Parameter” subsection under the Header tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JESvDtNXW7-MsbLGyLSkwQ.jpeg" /><figcaption>Parameters</figcaption></figure><p>Parameters</p><p>Now that we have all three parts, it’s simple to call the API. The response can then be manipulated into a data frame for analysis.</p><p>Get request</p><p>The final request will look something like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDoJGRdnXbEsrI1PwuDmiw.jpeg" /><figcaption>Dataframe</figcaption></figure><h3>Thank You !!</h3><p>Congratulations 👏 !! we have successfully scraped the NBA stats website.</p><p>PS: This process definitely works for <a href="https://stats.nba.com/">stats.nba.com</a>. This might also work for any other website which was built with a client-side web framework using languages such as AngularJS. If the website you are targeting is built with the server-side framework and languages like Django or Ruby on Rails then our friend Beautiful Soup will give you a hand.</p><p>Good luck with your web scraping journey! I hope this post is helpful.</p><p>Feel free to reach out to me on <a href="https://twitter.com/AshwinMuru">Twitter</a> or <a href="https://www.linkedin.com/in/ashwinmuthiah/">Linkedin</a> if you may have any questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49d7562ce3ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-scraping-nba-stats-is-cooler-than-michael-jordan-49d7562ce3ef">How Scraping NBA Stats is cooler than Michael Jordan</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating ETL with AWS Lambda]]></title>
            <link>https://medium.com/data-science/automating-etl-with-aws-lambda-97b9e3404929?source=rss-d05ad5d28c5e------2</link>
            <guid isPermaLink="false">https://medium.com/p/97b9e3404929</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Ashwin M Murugappan]]></dc:creator>
            <pubDate>Tue, 19 May 2020 14:09:32 GMT</pubDate>
            <atom:updated>2020-06-08T21:26:36.266Z</atom:updated>
            <content:encoded><![CDATA[<h4>Package and deploy a serverless ETL pipeline using AWS Lambda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jsA5YGxAxcnBwrS7" /><figcaption>Photo by <a href="https://unsplash.com/@hasanalmasi?utm_source=medium&amp;utm_medium=referral">Hasan Almasi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Introduction</h4><p>The world around us is changing faster than ever before. The amount of data we produce every day is truly mind-boggling. There are 2.5 quintillion bytes of data created each day, which means the data we use today might not be relevant tomorrow. So, we need a way to completely automate our data flow process, the good news is you don&#39;t need to worry about having fancy servers and expensive hardware to do so, AWS lets you do this with the added advantage of high performance and scalability.</p><p>In this post, we will only be using AWS free tier offerings. So you can follow along. Before we jump into the process we need to understand the 3 AWS services we are going to use EC2, AWS Lambda, and Cloudwatch.</p><h4><strong>EC2</strong></h4><p>Amazon <a href="https://aws.amazon.com/ec2/">Elastic Compute Cloud</a> (EC2) is a service that lets you rent virtual machines. We can launch a <a href="https://en.wikipedia.org/wiki/Virtual_machine">Virtual machine</a> with any configuration we want(Operating system, Storage,etc.) for any amount of time we want.</p><h4><strong>Lambda</strong></h4><p><a href="https://aws.amazon.com/lambda/">AWS Lambda</a> is a serverless computing service that runs us code in response to events and automatically manages the underlying computing resources for us. We can use AWS Lambda to extend other AWS services with custom logic or create our back-end services. <strong>AWS Lambda can automatically run code in response to multiple events</strong>, such as HTTP requests via Amazon API Gateway, modifications to objects in Amazon S3 buckets, table updates in Amazon DynamoDB. In our case, it will run the code according to schedule.</p><h4><strong>Cloudwatch</strong></h4><p>As the word suggests, <a href="https://aws.amazon.com/cloudwatch/">CloudWatch</a> collects monitoring and operational data in the form of logs, metrics, and events, providing us with a unified view of AWS resources, applications, and services that run on AWS and on-premises servers. We can use CloudWatch to detect anomalous behavior in our environments, set alarms, visualize logs and metrics side by side, <strong>take automated actions</strong>, troubleshoot issues, and discover insights to keep our applications running smoothly.</p><p>AWS Lambda is the platform where we do the programming to perform ETL, but AWS lambda doesn&#39;t include most packages/Libraries which are used on a daily basis (Pandas, Requests) and the standard <strong>pip</strong> install pandas <strong>won&#39;t work </strong>inside AWS lambda. So there is a need to install packages/libraries <strong>locally or on an EC2</strong> virtual machine, upload it as a <strong>zip file to S3,</strong> and attach it as a <strong>layer</strong> to <strong>Lambda function.</strong></p><p>Here is an illustration of the workflow I’m talking about,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7rBAKZX_FgpT66trDkZDQQ.png" /><figcaption>Workflow</figcaption></figure><h4><strong>Create and launch an EC2 Linux instance with an S3 access role</strong></h4><p>Log into the AWS console. Under services, select EC2 then Launch Instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H53Sw4y5TBWFYXNZJSMCxQ.png" /><figcaption>Select Launch Instance</figcaption></figure><p>Step 1: Select the operating system for our EC2 instance, I selected <strong>Ubuntu Server 18.04 LTS (HVM), </strong>But you can select<strong> </strong>Amazon Linux 2 AMI (HVM)<strong> </strong>or any other environment you are comfortable with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C948jSTeLoPzLVUJ92X4qg.png" /><figcaption>Select Ubuntu Server 18.04 LTS (HVM)</figcaption></figure><p>Step 2: Choose an instance type, select the t2.micro(Its Included in the free tier, we get up to 750 hours of micro instances each month).</p><p>Step 3: Configure instance details, leave everything else to its default except for the <strong>IAM role. </strong>Create a new role with S3 access. Select “Create new IAM role”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYhXDmPoZhhcXCuv_1mcvQ.jpeg" /><figcaption>Create a new IAM role</figcaption></figure><p>We will be redirected to “IAM Roles” page where we can select create a role, followed by EC2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nLG81FtM8-Et5R0xM9RtQ.jpeg" /><figcaption>Select Create Role</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VH25g176XrxacjwCsNt1aA.png" /><figcaption>Select EC2</figcaption></figure><p>Attach the “AmazonS3FullAccess” policy to read and write to S3 buckets and give a name to the role. Hit create a role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16e50ZJU0KwI52GySj6xFQ.jpeg" /><figcaption>Attach S3FullAccess Policy so that we have read &amp; write permissions to S3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a4rFDMOuunjf8P7KbgjoQw.jpeg" /><figcaption>Give Role name</figcaption></figure><p>Head back to the Launch instance page and hit refresh next to the “IAM role”, select the role we just created, and hit review and launch. We can also set storage, tags, and configure security groups if needed, but for now, the defaults are good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9lKkEdH20yB20ZaujElMYA.png" /><figcaption>Select IAM role</figcaption></figure><p>We will be moved to Step 7: Review instance launch - hit launch. We will be prompted to create or select a key pair. Select create a new pair and give it a name, <strong>download the key pair,</strong> and hit launch instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WmGBHutCP5V4gKApHcCCjg.png" /><figcaption>Create a new key pair and Launch Instance</figcaption></figure><p>Once our instance is created select view instance. Then select the Instance we just created and hit connect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bI4ZrZ7VBPon9jaAxqqn9w.png" /><figcaption>Select instance and Hit Connect</figcaption></figure><p>Copy the “Example” code and go the command prompt on your computer to connect to the instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XL6-SodSaBIpNlKVI4ygRw.png" /><figcaption>Copy the Example Code and paste it on your command prompt</figcaption></figure><p>Paste the ssh command as shown in the example, change the path of the key(my_key.pem), and hit enter. Once we are connected to our command prompt it should look something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EoUm9_Ez_dRzEl-MXzwPvg.png" /><figcaption>Command Prompt</figcaption></figure><h4><strong>Install all dependencies(Libraries) we need in the virtual environment, package the dependencies in a zip file and export it to S3</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89dea785fc3c4afbea6cd145f4aa862e/href">https://medium.com/media/89dea785fc3c4afbea6cd145f4aa862e/href</a></iframe><p>Follow the above code and we have successfully exported our packaged zip file to an S3 bucket.</p><h4><strong>Create and Adding Lambda Layers</strong></h4><p>Select Services on the top left corner of the AWS console and navigate to AWS lambda and then to <strong><em>Layers. </em></strong>Select<em> </em><strong><em>Create Layer.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qfLnqoawbikXkyKaB_XP4w.png" /><figcaption>Select Create Layer</figcaption></figure><p>Copy the <strong>object URL</strong> of the zip file we uploaded to the S3 bucket and paste it on to the <strong>“Amazon S3 link URL”</strong> text box. Hit create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e7hbV4cro067ZzeRmCzIBw.png" /><figcaption>Copy the Object URL</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4HvIKIVHlkhLs-7fTp8vg.png" /><figcaption>Paste the S3 link, select Compatible runtimes</figcaption></figure><p>Select add a layer and select the layers we created and the version. Hit add.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4M1L1uE7tEiIrDKv3LfOsA.png" /><figcaption>Adding Layer</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1cKTaUFxrh4cKgcC5YXjTQ.png" /><figcaption>Select the Layer you just created in the last step and Hit add</figcaption></figure><p><strong>Creating a Lambda Function</strong></p><p>On the AWS lambda console, select functions, and hit create functions. Give the function a name and choose the language to use to write the function. Hit create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*At49-xPpgLyH97zWyqUh7g.png" /><figcaption>Creating a Function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6EODtthtH_ZMm_TwOL2EA.png" /><figcaption>Give Function name and Select runtime</figcaption></figure><p>Update your Lambda function to extract, transform, and load the data. Here I’m reading a CSV file using pandas and storing it in an S3 bucket. This is the place where we can perform transformations and decide where we are going to load the data, in our case its an S3 bucket.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58f3a008439e761782db40edf3a8e5d3/href">https://medium.com/media/58f3a008439e761782db40edf3a8e5d3/href</a></iframe><p>Select test, enter a name for the test event and leave the rest to default. Hit save. Hit test again and the code performs our ETL process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhuLRAu3OZUxmIV7Of7ZhQ.png" /><figcaption>Select Test on the top right of your screen and add test even name</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ARJOqfcVN96RHJXEtYr-jw.png" /><figcaption>Give test event a name and leave the rest to default</figcaption></figure><p><strong>One Final Step: Schedule your ETL process</strong></p><p>Select Services on the top left corner of the AWS console and navigate to CloudWatch followed by event and then to rules. Select create a rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GK_fi7l-gTiVTGF-6w5BoQ.png" /><figcaption>services-Cloudwatch — Events — Rules — Create Rule</figcaption></figure><p>Step 1: Select schedule followed by Cron expression, enter the Cron expression (4 * * ? * *)to run the Lambda function every day at 4:00 GMT.</p><p>Learn more about Cron&#39;s expressions <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html#CronExpressions">here</a>.</p><p>Step 2: Enter the name and description. Select create a rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lr_ufj-BOKxXsv9xJ7KyTg.png" /><figcaption>Creating a Rule</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eod3vYadJ6sm9oUXjmwhvw.png" /><figcaption>Give the Rule a name and a description</figcaption></figure><p>Congratulations 👏 !! Our ETL process is automated and will be updated every 24 hours at exactly 4:00GMT.</p><p>PS: Make sure to terminate EC2 along with any Elastic Block Store (EBS) so that you don&#39;t go beyond free tier usage.</p><p>Good luck with your serverless journey! I hope this post is helpful, and feel free to reach out if you may have any questions.</p><p><a href="https://www.linkedin.com/in/ashwinmuthiah/">Ashwin Muthiah Murugappan - Northeastern University - Boston, Massachusetts | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97b9e3404929" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/automating-etl-with-aws-lambda-97b9e3404929">Automating ETL with AWS Lambda</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>