<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aakash Pydi on Medium]]></title>
        <description><![CDATA[Stories by Aakash Pydi on Medium]]></description>
        <link>https://medium.com/@aakashpydi?source=rss-6c7501dfca1b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3IxEziqtmf323CK8SoNqLw.jpeg</url>
            <title>Stories by Aakash Pydi on Medium</title>
            <link>https://medium.com/@aakashpydi?source=rss-6c7501dfca1b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:00:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aakashpydi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A/B Testing Machine Learning Models in Production Using Amazon SageMaker (Discussion)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/a-b-testing-machine-learning-models-in-production-using-amazon-sagemaker-discussion-ee8953043397?source=rss-6c7501dfca1b------2"><img src="https://cdn-images-1.medium.com/max/1181/1*SaDN5foxZD9IxvX9MoXEWA.png" width="1181"></a></p><p class="medium-feed-snippet">Thinking about A/B Testing ML Models in Production with a Potential Real-time Inference ML Workflow</p><p class="medium-feed-link"><a href="https://medium.com/data-science/a-b-testing-machine-learning-models-in-production-using-amazon-sagemaker-discussion-ee8953043397?source=rss-6c7501dfca1b------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/a-b-testing-machine-learning-models-in-production-using-amazon-sagemaker-discussion-ee8953043397?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee8953043397</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Mon, 15 Jun 2020 00:44:24 GMT</pubDate>
            <atom:updated>2020-06-15T00:44:24.918Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Production-Level ETL Pipeline Platform Using Apache Airflow]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/building-a-production-level-etl-pipeline-platform-using-apache-airflow-a4cf34203fbd?source=rss-6c7501dfca1b------2"><img src="https://cdn-images-1.medium.com/max/1097/1*w_SwwTYbpmBB9-IATyqP9A.png" width="1097"></a></p><p class="medium-feed-snippet">Using Apache Airflow to Manage Data Workflows in CernerWorks</p><p class="medium-feed-link"><a href="https://medium.com/data-science/building-a-production-level-etl-pipeline-platform-using-apache-airflow-a4cf34203fbd?source=rss-6c7501dfca1b------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/building-a-production-level-etl-pipeline-platform-using-apache-airflow-a4cf34203fbd?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4cf34203fbd</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sat, 26 Oct 2019 03:08:12 GMT</pubDate>
            <atom:updated>2019-10-26T03:08:12.786Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A Definitive Compilation of Apache Airflow Resources]]></title>
            <link>https://medium.com/data-science/a-definitive-compilation-of-apache-airflow-resources-82bc4980c154?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/82bc4980c154</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sat, 15 Jun 2019 13:13:58 GMT</pubDate>
            <atom:updated>2019-06-22T14:13:03.758Z</atom:updated>
            <content:encoded><![CDATA[<h4>AKA a new data engineer’s reading list to start working on a production level Apache Airflow architecture.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*c9NU5f4LQ_AAeV352szvMw.png" /></figure><h3>Official Resources</h3><ol><li><a href="https://airflow.apache.org/">Airflow Documentation</a></li><li><a href="https://github.com/apache/airflow">Airflow GitHub Repo</a></li><li><a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home">Airflow Wiki Home</a></li><li><a href="https://apache-airflow.slack.com/">Airflow Community Slack Channel</a></li><li><a href="https://twitter.com/ApacheAirflow">Airflow Twitter</a></li><li><a href="https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-3036?filter=allopenissues">Airflow JIRA</a></li></ol><h3>Introduction/Overview</h3><ol><li><a href="https://medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8">‘Airflow: a workflow management platform</a>’ by Maxime Beauchemin (Creator of Apache Airflow). Airbnb Engineering and Data Science. June, 2015.</li><li><a href="https://medium.com/@r39132/apache-airflow-grows-up-c820ee8a8324">‘Apache Airflow Grows Up!</a>’ by Sid Anand (Chief Data Engineer, Paypal. Committer &amp; PMC Member Apache Airflow). Medium. Jan, 2019.</li><li><a href="https://medium.com/@dustinstansbury/understanding-apache-airflows-key-concepts-a96efed52b1a">‘Understanding Apache Airflow’s Key Concepts</a>’ by Dustin Stansbury (Data Scientist, Quizlet). Medium (~ 1.8k +1&#39;s). May, 2017.</li><li>‘<a href="https://blog.insightdatascience.com/airflow-101-start-automating-your-batch-workflows-with-ease-8e7d35387f94">Airflow 101: How to start automating your data pipelines with Airflow</a>’ by Sriram Baskaran (Program Director, Data Engineering Insight Data Science). Medium(~ 1.2k +1&#39;s). Oct, 2018.</li><li>[Video] ‘<a href="https://www.youtube.com/watch?v=dgaoqOZlvEA">Best practices with Airflow- an open source platform for workflows &amp; schedules</a>’ by Maxime Beauchemin (Creator of Apache Airflow).</li><li>[Video] ‘<a href="https://www.youtube.com/watch?v=tcJhSaowzUI">Modern ETL-ing with Python and Airflow (and Spark)</a>’ by Tamara Mendt (Data Engineer, HelloFresh). PyConDE 2017.</li><li>[Video] ‘<a href="https://www.youtube.com/watch?v=cHATHSB_450">A Practical Introduction to Airflow</a>’ by Matt Davis (Data Platform Engineering at Clover). PyData SF 2016.</li><li>[Video] ‘<a href="https://www.youtube.com/watch?v=XJf-f56JbFM">Developing elegant workflows in Python code with Apache Airflow</a>’ by Michael Karzynski (Tech Lead at Intel). EuroPython Conference. July 2017.</li><li>[Video] ‘<a href="https://www.youtube.com/watch?v=60FUHEkcPyY">How I learned to time travel, or, data pipelining and scheduling with Airflow</a>’ by Laura Lorenz (Data &amp; SWE at Industry Dive). PyData DC 2016.</li></ol><h3>Airflow in the Industry</h3><ol><li><a href="https://eng.uber.com/managing-data-workflows-at-scale/">‘Managing Uber’s Data Workflows at Scale’</a> by Alex Kira. Uber Data Engineering. Feb, 2019.</li><li>‘<a href="https://blog.twitter.com/engineering/en_us/topics/insights/2018/ml-workflows.html">Productionizing ML with workflows at Twitter</a>’ by Samuel Ngahane and Devin Goodsell. Twitter Engineering. June, 2018.</li><li>‘<a href="https://engineering.pandora.com/apache-airflow-at-pandora-1d7a844d68ee">Apache Airflow at Pandora</a>’ by Ace Haidrey. Pandora Engineering. Mar, 2018.</li><li>‘<a href="https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8fccff">Running Apache Airflow at Lyft</a>’ by Tao Feng, Andrew Stahlman, and Junda Yang. Lyft Engineering. Dec, 2018.</li><li>‘<a href="https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8">Why Robinhood uses Airflow</a>’ by Vineet Goel. Robinhood Engineering. May, 2017.</li><li>‘<a href="https://medium.com/dailymotion/collaboration-between-data-engineers-data-analysts-and-data-scientists-97c00ab1211f">Collaboration between data engineers, data analysts and data scientists</a>’ by Germain Tangus (Senior Data Engineer, Dailymotion). Dailymotion Engineering. May, 2019.</li><li>‘<a href="https://engineering.sift.com/sift-trains-thousands-models-using-apache-airflow/">How Sift Trains Thousands of Models using Apache Airflow</a>’ by Duy Tran. Sift Engineering. Mar, 2018.</li><li>‘<a href="https://blog.socialcops.com/technology/engineering/airflow-meta-data-engineering-disha/">Airflow, Meta Data Engineering, and a Data Platform for the World’s Largest Democracy</a>’ by Vinayak Mehta. Socialcops Engineering. Aug, 2018.</li><li>‘<a href="https://medium.com/leboncoin-engineering-blog/data-traffic-control-with-apache-airflow-ab8fd3fc8638">Data Traffic Control with Apache Airflow</a>’ by Nicolas Goll Perrier (Data Engineer, leboncoin). leboncoin Engineering Blog. Jan, 2019.</li><li>‘<a href="https://medium.com/snaptravel/airflow-part-2-lessons-learned-793fa3c0841e">Airflow Part 2: Lessons Learned (at SnapTravel)</a>’ by Nehil Jain. SnapTravel Engineering. Jun, 2018.</li><li>‘<a href="https://www.astronomer.io/blog/using-apache-airflow-to-create-data-infrastructure/">Using Apache Airflow to Create Data Infrastructure in the Public Sector</a>’ by Varun Adibhatla and Laurel Brunk. Astronomer.io. Oct, 2017.</li><li>‘<a href="https://wecode.wepay.com/posts/airflow-wepay">Airflow at WePay</a>’ by Chris Riccomini. WePay. Jul, 2016.</li></ol><h3>Airflow Distributed Deployment</h3><ol><li>‘<a href="https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405">How Apache Airflow Distributes Jobs on Celery workers</a>’ by Hugo Lime (Data Scientist, Sicara AI and Big Data). Sicara Engineering. Apr, 2019.</li><li>‘<a href="https://stlong0521.github.io/20161023%20-%20Airflow.html">A Guide On How To Build An Airflow Server/Cluster</a>’ by Tianlon Song (Sr. Software Engineer, Machine Learning &amp; Big Data at Zillow). Oct, 2016.</li></ol><h3>Testing</h3><ol><li>‘<a href="https://medium.com/wbaa/datas-inferno-7-circles-of-data-testing-hell-with-airflow-cef4adff58d8">Data’s Inferno: 7 Circles of Data Testing Hell with Airflow</a>’ by WB Advanced Analytics. Jan, 2018.</li><li>‘<a href="https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c">Testing in Airflow Part 1 — DAG Validation Tests, DAG Definition Tests and Unit Tests</a>’ by Chandu Kavar on Medium ( ~1k +1&#39;s). Aug 2018.</li></ol><h3>Additional Reading</h3><ol><li>‘<a href="https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753">We’re All Using Airflow Wrong and How to Fix It</a>’ by Jessica Laughlin. Bluecore Engineering. Aug, 2018.</li></ol><p>These are the resources I’ve used to understand airflow and the data architecture frameworks associated with it. I should update this post as I continue my work and find more helpful resources. Please feel free to suggest additions to this list.</p><p><strong>For an ultra exhaustive compilation of Airflow resources</strong>, check out the ‘<a href="https://github.com/jghoman/awesome-apache-airflow">Awesome Apache Airflow GitHub Repo</a>’ by Jakob Homan (Data Software Engineer, Lyft. Airflow Committer and PMC Member).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82bc4980c154" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-definitive-compilation-of-apache-airflow-resources-82bc4980c154">A Definitive Compilation of Apache Airflow Resources</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding C-Store: Columnar Databases and Big Data Analytics]]></title>
            <link>https://medium.com/@aakashpydi/understanding-c-store-columnar-databases-and-big-data-analytics-aa669bb60f0?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa669bb60f0</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Mon, 10 Jun 2019 01:09:50 GMT</pubDate>
            <atom:updated>2019-11-02T22:24:48.469Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding C-Store: Columnar Databases and Data Intensive Analytics</h3><p>From what I understand, the first prototype for a columnar database was introduced in a 2005 paper from MIT’s school of computer science and artificial intelligence, called <a href="http://db.csail.mit.edu/projects/cstore/vldb.pdf">C-Store: A Column Oriented Database. (Stonebraker et al). MIT CSAIL.</a></p><p>The key idea from this paper seems to be that, in columnar databases we store each column’s values sequentially in a database, as opposed to storing each row’s values sequentially. Let’s try to visualize this idea.</p><p>Consider the following examples given on <a href="https://docs.aws.amazon.com/redshift/latest/dg/c_columnar_storage_disk_mem_mgmnt.html">Amazon Redshift’s Database Developer Guide</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*xNYKI1azVyfzhXB-cIvKUw.png" /></figure><p>Some points to note regarding the example above:</p><ul><li>Notice that each block has values of different types (string, integer, etc).</li><li>What happens if the row size is greater than a <a href="https://docs.oracle.com/cd/B19306_01/server.102/b14220/logical.htm">block</a>? We will need to read more than a block to read a single row.</li><li>Similarly, what happens if the row size is smaller than a <a href="https://docs.oracle.com/cd/B19306_01/server.102/b14220/logical.htm">block</a>? In this case we will have inefficient use of disk space.</li><li>Note that a data block is the smallest unit of data used by a database. See this link for more information: <a href="https://docs.oracle.com/cd/B19306_01/server.102/b14220/logical.htm">Introduction to Data Blocks. Oracle.</a></li><li>This is a ‘write-optimized’ database as it has a very low cost to write a new record to disk. The DB can easily add a new block with the record’s data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*qi2UPrKFpGDWrgGMwB7F6g.png" /></figure><p>Some points to note regarding the example above:</p><ul><li>Notice that each block has the same data type. This turns out to be a significant advantage as the DB can use <a href="http://db.csail.mit.edu/projects/cstore/abadisigmod06.pdf">compression techniques</a> to significantly reduce the disk usage and I/O.</li><li>This is a ‘read optimized’ database, particularly in a big data analytics context (which is what we use the large datasets in our columnar database for). Consider how most queries only require a few of the column values. So when we have a large dataset, with many rows and columns, only retrieving the columns that we need to execute our analytics query is substantially more efficient than retrieving all the records in the database.</li></ul><p>The original CSAIL paper notes that write optimized database systems are naturally ideal write heavy applications (such as <a href="https://docs.microsoft.com/en-us/azure/architecture/data-guide/relational-data/online-transaction-processing">Online Transaction Processing</a> applications) whereas read optimized database systems are naturally ideal for read heavy applications (such as <a href="https://aws.amazon.com/data-warehouse/">Data Warehouses</a>).</p><p>It should be evident why columnar databases are well suited for big data analytics architectures. They offer (i) significant data compression, and (ii) highly optimized read operations that are typically used for analytics jobs. My Director, Dan Woicke wrote about the value of using a columnar database to our team’s work here: <a href="https://hp.cioreview.com/cxoinsight/cerner-advances-big-data-analytic-capabilities-nid-11200-cid-59.html">Cerner Advances Big Data Analytic Capabilities. Dan Woicke. CIO Review</a>. There are more features that columnar databases offer which you can explore in the additional reading section below.</p><p>Some additional reading,</p><p>(1) <a href="https://medium.com/@ameya_s/c-store-a-columnar-database-1fe7e84d7247">C-Store: A Columnar Database. Ameya. Medium.</a></p><p>(2) <a href="https://medium.com/@mangatmodi/rowise-vs-columnar-database-theory-and-in-practice-53f54c8f6505">Rowwise vs Columnar Database? Theory and in Practice. Modi. Medium.</a></p><p>(3) <a href="https://towardsdatascience.com/the-beauty-of-column-oriented-data-2945c0c9f560">The beauty of column oriented data. Zaks, Maxim. Towards Data Science.</a></p><p>(4) <a href="https://web.cs.ucdavis.edu/~green/courses/ecs165b-s10/Column_Store_Tutorial_VLDB09.pdf">Column Oriented Database Systems. Stavros Harizopoulous. UC Davis Presentation.</a></p><p>(5) <a href="http://www.cs.yale.edu/homes/dna/papers/abadisigmod06.pdf">Integrating Compression and Execution in Column-Oriented Database Systems. Abadi et Al. MIT.</a></p><p>(6) <a href="http://vldb.org/pvldb/vol5/p1790_andrewlamb_vldb2012.pdf">The Vertica Analytic Database: C-Store 7 Years Later. (Lamb et al). Vertica.</a></p><p>(7) <a href="https://www.vertica.com/wp-content/uploads/2018/05/why_all_column_stores_are_not_the_same_wp.pdf">Why All Column Stores Are Not The Same: Twelve Low Level Features That Offer High Value To Analysts. Vertica.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa669bb60f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using a Generative Adversarial Network (GAN) to Create Novel Artistic Images]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/using-a-generative-adversarial-network-gan-to-create-novel-artistic-images-e06c65ff43ad?source=rss-6c7501dfca1b------2"><img src="https://cdn-images-1.medium.com/max/640/0*l-oSEWTt1v0RA0jq.JPG" width="640"></a></p><p class="medium-feed-snippet">In this project I used a Generative Adversarial Network (GAN) architecture to generate NEW artistic images that capture the style of the&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/data-science/using-a-generative-adversarial-network-gan-to-create-novel-artistic-images-e06c65ff43ad?source=rss-6c7501dfca1b------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/using-a-generative-adversarial-network-gan-to-create-novel-artistic-images-e06c65ff43ad?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e06c65ff43ad</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[gans]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 01:20:21 GMT</pubDate>
            <atom:updated>2019-06-17T12:22:21.198Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Real Time Classification Video Stream Using a LeNet5 Convolutional Neural Network]]></title>
            <link>https://medium.com/@aakashpydi/real-time-classification-video-stream-using-a-lenet5-convolutional-neural-network-4164b8608650?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4164b8608650</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 01:18:02 GMT</pubDate>
            <atom:updated>2019-06-09T01:18:02.217Z</atom:updated>
            <content:encoded><![CDATA[<p>In this project I used a LeNet5 convolutional neural network to carry out classifcation for the <a href="https://www.cs.toronto.edu/~kriz/cifar.html">CIFAR100 Dataset</a>. The trained CNN model was then used to carry out real time object classification in a video stream. <a href="https://github.com/aakashpydi/CIFAR100-CNN-Model">(Link to Github Repo of Source Code)</a>. I also included an implementation of a CNN model to carry out classification for the MNIST (handwritten digits) dataset.</p><p>In img2num.py, a LeNet5 convolution neural network is used to carry out classification for the MNIST (handwritten digits) dataset. In img2obj.py, a LeNet5 convolution neural network is used to carry out classification for the CIFAR100 dataset. Note that nn_img2num.py is also included to carry out comparison of the LeNet5 CNN model and the simple NN model from the last homework. (This was done by storing the plot details from img2num.py and then accessing that information from nn_img2num, and then subsequently plotting the required charts). The file lenet5_data.tar has the required plot information from running img2num.py.</p><p>The file saved_cnn_model_save.tar stores the evaluated CIFAR100 LeNet5 CNN model. This allows one to quickly load the model (circumventing the lengthy training time). If one wants to train the model again, simply delete this save file.</p><p>The associated results are given below.</p><h3>Image to Number (MNIST Dataset)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*D2M5wGFg4Q_Kc15D.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*XhOB9wkFvJ-zJj4N.png" /></figure><h4>Comparison Between LeNet5 CNN and Simple NN for MNIST Digit Classification</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*In2Y4rvySve3l0Bs.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*IvPrwMrkZn-YdEKP.png" /></figure><h3>Image to Object (CIFAR100 Dataset)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*N8PX-X4MVElzK55e.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Z8lNuEr8mERqM1_m.png" /></figure><h4>One Object and its Caption</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*vFavNG-bTjp5aAxd.png" /></figure><h4>Continuous Captioning of Images from a Webcam</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*DROjA2-qYU7YQ3i2.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/0*veq6R-BV6QQ4IRFB.png" /></figure><p>Written on April 16, 2018</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4164b8608650" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training My Neural Network with the MNIST (Handwritten Digits) Dataset]]></title>
            <link>https://medium.com/@aakashpydi/training-my-neural-network-with-the-mnist-handwritten-digits-dataset-d56f075221e?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d56f075221e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 01:13:25 GMT</pubDate>
            <atom:updated>2019-06-09T01:13:25.276Z</atom:updated>
            <content:encoded><![CDATA[<p>I trained my neural network with the <a href="http://yann.lecun.com/exdb/mnist/">MNIST dataset</a> in this project. The MNIST dataset consists of handwritten images (60,000 images in the training set and 10,000 images in the test set). <a href="https://github.com/aakashpydi/MNISTNeuralNetwork">(Link to Github Repo of Source Code)</a>.</p><p>The neural network with forward propagation and back-propagation pass is implemented in neural_network.py. The MyImg2Num neural network is defined in my_img2num.py. It relies on the Neural Network class implemented in neural_network.py for its functionality. The NnImg2Num neural network is defined in nn_img2num.py. This neural network uses the torch.nn neural network package for its functionality. Finally both of these neural networks are trained and tested on the MNIST handwritten digits dataset. In order to run the scripts, simply create an instance of the corresponding class and call the train() instance method as shown below.</p><p>nn_mnist = NnImg2Num()</p><p>nn_mnist.train()</p><p>my_mnist = MyImg2Num()</p><p>my_mnist.train()</p><p>The performance charts associated with each of the neural networks are given below.</p><h3>NnImg2Num Neural Network</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*mTA23KsCsrxWzF1E.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Q8ikPBzACcO5Zgkn.png" /></figure><h3>MyImg2Num Neural Network</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*WoN6uMigrx6ry1Bb.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*RMkU6SPC5weV_hKm.png" /></figure><p><em>Written on April 13, 2018</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d56f075221e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Neural Network Backpropagation and Stochastic Gradient Descent]]></title>
            <link>https://medium.com/@aakashpydi/implementing-neural-network-backpropagation-and-stochastic-gradient-descent-9ef91272416?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ef91272416</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 01:11:37 GMT</pubDate>
            <atom:updated>2019-06-09T01:11:37.916Z</atom:updated>
            <content:encoded><![CDATA[<p>I implemented backpropagation and stochastic gradient descent in my neural network for this project. <a href="https://github.com/aakashpydi/BackpropSGDNeuralNetwork">(Link to Github Repo of Source Code)</a>. I tested my implementation using, AND, OR, NOT and XOR networks.</p><p>The neural network with forward propagation and back-propagation pass is implemented in neural_network.py. The functionality of the classes defined in logic_gates.py relies on this neural network. The class representations of the AND, OR, NOT, and XOR gates is given in logic_gates.py. These calsses were modified to derive the weight (theta) values of their corresponding neural network through the use of backword propagation. The training data set was generated in the train() method associated with each class.</p><p>The implemented classes were tested in test.py. The output of executing test.py is given below. Note that, the output of test.py is also attached at the end of this file.</p><h3>Comparison of Theta I handcrafted for HW02 vs the one’s learnt from back-propagation</h3><h3>AND Gate</h3><p>| Handcrafted Weight Layer_0| Derived Backpropagation Weight Layer_0| | — — — — — — — — — :| — — — — — — — — — — — — — :| | -10 | -10.9117 | | 6 | 7.2167 | | 6 | 7.2167|</p><h3>OR Gate</h3><p>| Handcrafted Weight Layer_0| Derived Backpropagation Weight Layer_0| | — — — — — — — — — :| — — — — — — — — — — — — — :| | -1 | -3.3639 | | 2 | 7.2062 | | 2 | 7.2062|</p><h3>NOT Gate</h3><p>| Handcrafted Weight Layer_0| Derived Backpropagation Weight Layer_0| | — — — — — — — — — :| — — — — — — — — — — — — — :| | 1 | 3.3257 | | -2 | -6.8652 |</p><h3>XOR Gate</h3><p>| Handcrafted Weight Layer_0| Derived Backpropagation Weight Layer_0| | — — — — — — — — — :| — — — — — — — — — — — — — :| | -25, -25 | 6.8729, 2.5445 | | -50, 50 | -4.6067, -6.2229 | | 50, -50 | -4.6419, -6.4120|</p><p>handcrafted Weight Layer_1Derived Backpropagation Weight Layer_1–25–4.4001509.336750–9.5169</p><h3>Corresponding Output of the Test Script</h3><h3>AND Gate Output</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/0*M8E6zYDJ7CwrnQn7.png" /></figure><h3>OR Gate Output</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/0*duiPvnjNSerR1d5m.png" /></figure><h3>NOT Gate Output</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/0*pLbXywD1EPYNk8XT.png" /></figure><h3>XOR Gate Output</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*KaV_4haJSgZOS59d.png" /></figure><p><em>Written on April 10, 2018</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ef91272416" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing and Testing a Feed Forward Neural Network]]></title>
            <link>https://medium.com/@aakashpydi/implementing-and-testing-a-feed-forward-neural-network-a2178daf27df?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2178daf27df</guid>
            <category><![CDATA[feed-forward]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-intelligence]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 01:08:34 GMT</pubDate>
            <atom:updated>2019-06-09T01:08:34.599Z</atom:updated>
            <content:encoded><![CDATA[<p>I implemented and tested a feed forward neural network using pytorch in this project. <a href="https://github.com/aakashpydi/FeedForwardNeuralNet">(Link to Github Repo of Source Code)</a>. I tested my implementation using, AND, OR, NOT and XOR networks.</p><p>The forward propagation neural network is implemented in neural_network.py. The functionality of the classes defined in logic_gates.py relies on this neural network. The class representations of the AND, OR, NOT, and XOR gates is given in logic_gates.py.</p><p>The implemented classes were tested in test.py. The output of executing test.py is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/0*gdhwBXv_FHpWYbHd.png" /></figure><p><em>Written on April 8, 2018</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2178daf27df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing 2D Convolution]]></title>
            <link>https://medium.com/@aakashpydi/implementing-2d-convolution-ebad23f1e43c?source=rss-6c7501dfca1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebad23f1e43c</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolution-neural-net]]></category>
            <dc:creator><![CDATA[Aakash Pydi]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 01:05:46 GMT</pubDate>
            <atom:updated>2019-06-09T01:05:46.016Z</atom:updated>
            <content:encoded><![CDATA[<p>I implemented 2D Convolution from scratch in this project. <a href="https://github.com/aakashpydi/Conv2DImplementation">(Link to Github Repo of Source Code)</a> I recorded my output and performance results below.</p><p>The parameters associated with each of the tasks is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*bqMlUzdaW2Eg2udL.JPG" /></figure><h3>PART A</h3><p>The following two input images were used to carry out the 2d convolutions described in task A.</p><p>A Cartoon Image (1280 x 720)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N06Z4i9fadWa2BHr.jpg" /></figure><p>A Checkerboard Image (1920 x 1080)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FLFQjlVAAVvAsw2P.jpg" /></figure><p>The output for part A is given below. The normalized output for part A is included at the end of the file.</p><h4>Task 1</h4><p>The following two images were produced using kernel_1. The checker board result indicates that kernel 1 functions as a form of horizontal edge detector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oT4t1LbKdlW32EXi.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kdpjJJoNTbpKH6rS.png" /></figure><h4>Task 2</h4><p>The following two images were produced using kernel_4.The checker board result indicates that kernel 4 also functions as a form of horizontal edge detector. We also observe that the edge detection is more pronounced in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PahUdmz4m0qf8fsc.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NI1RTFM-wQ3fM2f9.png" /></figure><p>The following two images were produced using kernel_5. The checker board result indicates that kernel 5 functions as a form of vertical edge detector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tZS3gcvmFVQKjQTS.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Eyz7NT65LkQjfky.png" /></figure><h4>Task 3</h4><p>Note that all the output images are smaller than the input image due to stride value being greater than 1.</p><p>The following two images were produced using kernel_1.The checker board result indicates that kernel 1 functions as a form of horizontal edge detector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*rT_2eoUxJ-XEsRWa.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*vOlY-Pn64tJmreLd.png" /></figure><p>The following two images were produced using kernel_2. The checker board result indicates that kernel 2 functions as a form of vertical edge detector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*qHTv3_TCxqi_lSS6.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*KY426kpXhE7q_4Zl.png" /></figure><p>The following two images were produced using kernel_3. This kernel is supposed to be a smoothing filter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*4nx3L8cROD0mlHTZ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*bx-xnu22JIFLKeXZ.png" /></figure><h3>PART B</h3><p>Graphs showing the time taken for convolution as a function of the i Values (where 2^i = number of output channels), for each of the input images is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*xlDCkwRDufoxhscS.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*dEON8HIYgYGRnE_3.png" /></figure><h3>PART C</h3><p>Graphs showing the number of operations in the convolutions as a function of the given kernel sizes, for each of the input images is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*suWC8_HXDD4SqN6Z.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Co4iNeNESYhWfRnn.png" /></figure><h3>Normalized Output for PART A</h3><h4>Task 1</h4><p>The following two images were produced using kernel_1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pf_HqE0OEuF-yG_r.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pkkFYJq5dH6bAoKd.png" /></figure><h4>Task 2</h4><p>The following two images were produced using kernel_4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2KNB1eApP3Wt-6J.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mGzIycizNVC7F140.png" /></figure><p>The following two images were produced using kernel_5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p5rVVBwX_iIu7sUn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hUbm8ySc3dDY1EHY.png" /></figure><h4>Task 3</h4><p>The following two images were produced using kernel_1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*EQD44mddX6AIJ8rq.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*_OZQWr5W3QySXnqs.png" /></figure><p>The following two images were produced using kernel_2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*cGbj37884rXzoX9U.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*pV6CRgvL1oDu_JXL.png" /></figure><p>The following two images were produced using kernel_3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*5vXgaMAsMuGI8Wgm.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*UfhfF83CO8Yd_C4z.png" /></figure><p><em>Written on April 6, 2018</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebad23f1e43c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>