<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harsh Rana on Medium]]></title>
        <description><![CDATA[Stories by Harsh Rana on Medium]]></description>
        <link>https://medium.com/@hr23232323?source=rss-be594c58dc55------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oLkMsRYou8yIWRl2M5FpyA@2x.jpeg</url>
            <title>Stories by Harsh Rana on Medium</title>
            <link>https://medium.com/@hr23232323?source=rss-be594c58dc55------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:39:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hr23232323/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Choose a Winning Mobile App Tech Stack in 2023]]></title>
            <link>https://medium.com/geekculture/how-to-choose-a-winning-mobile-app-tech-stack-in-2023-dcfa52e35d3f?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcfa52e35d3f</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 23:01:52 GMT</pubDate>
            <atom:updated>2023-04-10T23:01:52.735Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn the CERT analysis technique to help you choose a winning tech stack that ensures you ship fast and build to scale.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yh9dRmec-RtMu2OHCC3YTg.png" /><figcaption>Screenshots from <a href="https://getbisque.com/">Bisque</a> (generated by Author using <a href="https://mockrocket.io/">MockRocket</a>)</figcaption></figure><p>Embarking on the journey of mobile app development can be both exciting and overwhelming. With thousands of technologies, frameworks, and libraries available, it’s often challenging to know where to begin. Instead of suffering from analysis paralysis, let’s cut through the BS and refine our decisioning process.</p><p>In this blog post, we’ll explore some popular options for building mobile apps in 2023. We’ll discuss the factors to consider when choosing a tech stack, and share examples of finding the right combination of tools. The goal is to make informed decisions for our mobile app projects.</p><h3>Frontend</h3><p>Deciding between native, cross-platform and hybrid development is the first step in getting started with the frontend, but also one with the most opinions. Engineers online will swear by one or the other, without sharing how to make an informed decision. Here’s a brief outline of each, with some pros and cons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mj2A4Tk7lgDXrz_-4ENRGA.png" /><figcaption>Various Tech stack options for developing a mobile app in 2023 (Author)</figcaption></figure><h4>Native development</h4><p>Native development involves building separate apps for each platform (Android and iOS) using platform-specific programming languages and tools. For Android, this means using Java or Kotlin with Android Studio, while for iOS, you’ll work with Swift or Objective-C in Xcode.</p><h4>Pros</h4><ul><li>Optimal performance and seamless integration with device features.</li><li>Full access to platform-specific APIs.</li><li>Comprehensive documentation and support from platform vendors (Google and Apple).</li></ul><h4>Cons</h4><ul><li>Separate codebases for each platform, leading to increased development time and costs.</li><li>Requires expertise in multiple programming languages and tools.</li></ul><h4><strong>Cross-platform development</strong></h4><p>Cross-platform development enables you to create a single codebase that runs on multiple platforms. There are several frameworks and tools available for cross-platform development, such as React Native, Flutter, and Xamarin.</p><h4>Pros</h4><ul><li>Single codebase for both Android and iOS, reducing development time and costs.</li><li>Easier to maintain and update.</li><li>Access to a wide range of community-developed packages and plugins.</li></ul><h4>Cons</h4><ul><li>Potentially lower performance compared to native apps.</li><li>Limited access to platform-specific APIs and features.</li></ul><h4>Hybrid development</h4><p>Hybrid development involves building mobile apps using web technologies like HTML, CSS, and JavaScript, wrapped inside a native container. Popular hybrid development frameworks include Apache Cordova (formerly PhoneGap), Ionic, and Capacitor.</p><h4><strong>Pros</strong></h4><ul><li>Leverage existing web development skills and resources.</li><li>Single codebase for multiple platforms.</li><li>Access to native device features through plugins.</li></ul><h4>Cons</h4><ul><li>Lower performance compared to native and cross-platform apps.</li><li>Limited access to platform-specific APIs and features.</li><li>Reliance on third-party plugins for native functionality.</li></ul><h3>Backend</h3><p>When it comes to building a mobile app, choosing the right backend can be just (if not more) as important as selecting the frontend technology. The backend is responsible for crucial functionality like processing data, handling authentication, and providing API services to ensure your app runs smoothly and securely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DVIDkGwnEPHTVRqo43e_fA.png" /><figcaption>Various mobile app backend options in 2023 (Author)</figcaption></figure><p>In this section, we’ll discuss a few popular backend options to consider for your app, along with some pros and cons for each.</p><h4>Firebase</h4><p>Firebase is a popular Backend-as-a-Service (BaaS) platform developed by Google. It offers a variety of features and is known for its ease of use, scalability, and real-time data synchronization.</p><h4>Pros</h4><ul><li>Easy to set up and manage, with minimal backend development experience required.</li><li>Real-time data synchronization and extensive built-in features.</li><li>Scalable and hosted by Google, ensuring reliability and performance.</li></ul><h4>Cons</h4><ul><li>Uses a NoSQL database (Firestore), which may not be suitable for projects requiring a traditional relational database.</li><li>Limited query capabilities compared to a traditional RDBMS.</li><li>May become costly as your app scales and requires more resources.</li></ul><h4>Supabase</h4><p><a href="https://supabase.com/">Supabase</a> is an open-source alternative to Firebase and is a good option for developers who prefer working with a traditional RDBMS and appreciate the control and flexibility provided by open-source technology.</p><h4>Pros</h4><ul><li>Uses PostgreSQL, a powerful and flexible RDBMS.</li><li>Real-time data synchronization and built-in features similar to Firebase.</li><li>Free tier provides all features needed for an MVP.</li></ul><h4>Cons</h4><ul><li>Newer platform with a smaller community and fewer resources compared to Firebase.</li><li>Requires more backend development experience compared to Firebase.</li><li>May lack some advanced features or integrations compared to Firebase.</li></ul><h4>Django + PostgreSQL</h4><p>If you’re comfortable with Python and want more control over your backend, this combination provides a powerful and flexible backend solution.</p><h4>Pros</h4><ul><li>Full control over backend architecture and customization.</li><li>Uses the powerful and flexible PostgreSQL RDBMS.</li><li>Wide range of third-party packages and plugins available.</li></ul><h4>Cons</h4><ul><li>Requires more setup, development, and management compared to BaaS platforms.</li><li>Requires proficiency in Python and familiarity with the Django framework.</li><li>Additional hosting and infrastructure management required.</li></ul><h4>Node.js + Express + MongoDB/SQL</h4><p>If you prefer working with JavaScript, and want more control over your backend, this option provides a powerful and flexible backend solution.</p><h4>Pros</h4><ul><li>Flexibility to choose between a NoSQL database (MongoDB) or a traditional SQL database.</li><li>Allows developers to use JavaScript for both frontend and backend development.</li><li>Large community and extensive resources available.</li></ul><h4>Cons</h4><ul><li>Requires more setup, development, and management compared to BaaS platforms.</li><li>Requires proficiency in JavaScript and familiarity with Node.js and Express.</li><li>Additional hosting and infrastructure management required.</li></ul><h3>How to decide — Enter CERT</h3><p>When deciding for yourself, consider the <strong>CERT</strong> method (yes I just made up that acronym, but <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6851171/">acronyms help you remember things better</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rlbSj4GUJ2zGA93nvvhtQ.png" /><figcaption>CERT (Author)</figcaption></figure><h4>1. Community</h4><p>Probably the most important factors for a small team — the community. Opt for technologies with strong community support, extensive documentation, and a rich ecosystem of packages and plugins. This can make it easier to find solutions to common problems and extend your app’s functionality.</p><h4>2. Expertise</h4><p>Evaluate your team’s existing skills and familiarity with programming languages and tools. Choosing a tech stack that aligns with your team’s expertise can accelerate development and improve overall quality. Nothing harder than learning a new tech stack and building something daunting all at the same time (unless you’re the type to love such a challenge).</p><h4>3. Requirements</h4><p>Analyze your app’s specific needs, such as performance, platform-specific features, and UI/UX requirements. Native development is generally recommended for high-performance apps or those requiring deep integration with platform-specific APIs. This is specific to you and your idea, so take stock of what you’re building.</p><h4>4. Timeline</h4><p>Consider the time and resources available for development. Cross-platform and hybrid development can save time and costs by allowing a single codebase for multiple platforms. But this can often come at the cost of performance. There’s always a tradeoff.</p><h3>Case study — How I’m building Bisque</h3><p>First off, what is Bisque?</p><blockquote>Bisque is a Boston-based social networking app for restaurant lovers to share their food experience with their friends.</blockquote><p>You can check out more information (or join the waitlist) <a href="https://getbisque.com/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tc9XzIzM-HGs0WZhtpfsuA.png" /><figcaption>Screenshot from <a href="https://getbisque.com/">Bisque website</a> (Author)</figcaption></figure><h4>How did we decide the tech stack for Bisque?</h4><p>After researching various options, we decided to go with <strong>cross-platform</strong> frontend development using <strong>Flutter, </strong>and a <strong>Supabase</strong> backend<strong> </strong>for Bisque. We used the CERT method to help make our decision.</p><p><strong>Community</strong>: Flutter has a strong community and is maintained by Google. Additionally, the SDK + documentation is great.</p><p><strong>Expertise</strong>: I’m the only engineer building Bisque. I have a fair amount of expertise using React, Javascript, Python etc. so Dart (language used by Flutter) was fairly easy to pick up.</p><p><strong>Requirements</strong>: The frontend of our application is fairly simple, and Flutter allows us to compile the application natively.</p><p><strong>Timeline</strong>: I wanted to build an MVP as soon as possible, even if that meant that we’d have to reconsider our tech stack for building a highly scalable v2.</p><p>Even after the CERT analysis, it finally came down to <strong>React Native</strong> or <strong>Flutter</strong> for the frontend and <strong>Firebase</strong> or <strong>Supabase</strong> for the backend. Here’s how we decided-</p><h4>Flutter vs. React Native</h4><p>We decided to go with Flutter, as it offers a more performant and customizable UI, which is necessary for a social media application (ce<strong>R</strong>t). Additionally, Flutter’s hot-reload feature, extensive widget library, and growing community support made it an attractive choice for building the MVP fast (cer<strong>T</strong>).</p><p>React Native, on the other hand, uses JavaScript and bridges the gap between native components and JavaScript code. While it has a more extensive ecosystem and a larger community, we were concerned about potential performance limitations due to the bridge between javascript and the native platform.</p><h4>Firebase vs. Supabase</h4><p>We were initially considering Firebase due to its popularity, extensive features, and ease of use. However, we were drawn to Supabase because it uses PostgreSQL, while Firebase uses Firestore. I felt more comfortable with a traditional RDBMS for our MVP (c<strong>E</strong>rt). Apart from that, Firebase and Supabase were almost identical in their featureset.</p><p>In the end, we chose <strong>Flutter</strong> for our frontend and <strong>Supabase</strong> for our backend, as this combination aligned with our CERT analysis for each.</p><h3>Wrapping up</h3><p>Choosing the right tech stack for your mobile app development project can be a complex decision. By understanding the various options available, carefully considering your project’s requirements, and making an informed decision using a CERT analysis, we hope to have provided valuable insights to help you make an informed choice for your own mobile app projects.</p><p>Remember, there’s no one-size-fits-all solution, and what works best for one project might not be ideal for another. Keep an open mind, be willing to learn from others, and don’t be afraid to change directions if needed. Good luck on your mobile app development journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcfa52e35d3f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-choose-a-winning-mobile-app-tech-stack-in-2023-dcfa52e35d3f">How to Choose a Winning Mobile App Tech Stack in 2023</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the newest software engineer on our team — ChatGPT]]></title>
            <link>https://medium.com/@hr23232323/introducing-the-newest-software-engineer-on-our-team-chatgpt-1a9182aec5da?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a9182aec5da</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Sun, 08 Jan 2023 15:26:42 GMT</pubDate>
            <atom:updated>2023-01-08T15:26:42.773Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing the newest software engineer on my team — ChatGPT</h3><h4><em>Over the last few weeks, I’ve been using ChatGPT during my day-to-day work as a software engineer. It’s helped me learn concepts faster, cross-check my work and most importantly — collaborate with a real (fake?) coworker.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g6Ebw_iGh5L8GlDl2cHHZg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@austindistel">Austin Distel</a> — <a href="https://unsplash.com/photos/wD1LRb9OeEo">https://unsplash.com/photos/wD1LRb9OeEo</a></figcaption></figure><p>You’ve probably already heard of ChatGPT — a long-form question-answering AI developed by OpenAI. What started out as gimmicky chatbot, is recently grabbing everyone’s attention.</p><p>And for good reason…</p><p>ChatGPT can do a lot. All over the internet, people are using it for many things, including -</p><ul><li><a href="https://becominghuman.ai/how-to-write-a-rap-song-using-ai-in-5-minutes-408bca270ff0">Writing a rap song</a></li><li><a href="https://medium.com/writers-blokke/i-used-chatgpt-to-write-a-blog-post-heres-how-it-went-369a886c4bdc">Writing a blog post</a></li><li><a href="https://medium.com/geekculture/write-a-novel-using-ai-openai-chatgpt-c02be59cd937">Writing a novel</a></li></ul><p>And most recently — for writing code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*-NKdlkovg0_ccmsT.jpg" /><figcaption>Credit — <a href="https://9gag.com/gag/armg5wB">https://9gag.com/gag/armg5wB</a></figcaption></figure><p>Yes, ChatGPT can write code. No, the code isn’t complete enough to replace your job as a software engineer. But yes, knowing how to use this tool will help you stay ahead in the field.</p><p>Over the last few weeks, I’ve been using ChatGPT for quite a few scenarios at work. I’m describing the two most useful (IMHO) below.</p><h4>Quick Notes</h4><p>Note: I’m showing non-real work examples for privacy reasons<br>Note 2: I’ll show excerpts from my conversation and share a link to the full conversation for those of you who are curious</p><h3>Research spikes</h3><p>For those of you who work in an Agile setting, know when you’re asked to work on a <a href="https://www.agile-academy.com/en/agile-dictionary/spike/">spike ticket</a>?</p><p>This could be before the team makes a decision on a new testing package.</p><p>Or, before you’re ready to start building a new subsystem.</p><p>Yeah, those don’t need to take days… Using ChatGPT efficiently, you can easily knock out general research spikes in minutes. Let me show you an example.</p><p>I’ve been working on an application which has quickly increased in complexity. Suddenly, the single tenant application we were building was starting to look more like a multi-tenant application. Instead of writing a bunch of if-else logic all over the app, the team decided we needed a research spike to better understand multi-tenant applications and how specifically to implement one in React.</p><p>Enter ChatGPT -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RFDpb4zTCZDGISjXZzh0zQ.png" /></figure><p>Okay, so far so good. A general purpose high level overview of some of the directions we could go in. Based on my own knowledge, I pick a direction (HOCs), provide some context to the AI and ask leading questions to understand implementation details-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5kabiAFRjCKdZQDP2Xq4dQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgFJ2YNMilo68uravzkFPA.png" /></figure><p>Great! Now I can ask another leading questions to get even more specific code snippets/implementation details -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYRM2Hb0H7CP5QfGn_fQ3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*ViuqSFAo2AOSx-kRu9jqEA.png" /></figure><p>And there you have it! A research spike which would have taken me 15+ google searches, potentially 1–2 hours of reading/researching existing patterns and then back-and-forth’s with other engineers on my team, accomplished in a 15 minute conversation. The full conversation can be read <a href="https://sharegpt.com/c/soNT2HA">here</a>.</p><p>Let’s look at another scenario where augmenting your knowledge with ChatGPT is super helpful.</p><h3>High level system design</h3><p>ChatGPT really shines when starting from scratch. This example is so powerful (and useful for engineers of any experience level) that I went ahead and wrote a whole article on just this use case-</p><p><a href="https://levelup.gitconnected.com/move-over-10x-engineers-its-time-for-the-supercharged-chatgpt-engineers-to-take-over-3ce97c9671e5">Move over 10x engineers — it’s time for the supercharged ChatGPT engineers to take over</a></p><h3>Takeaways</h3><p>ChatGPT is here and it’s amazing. It can effectively augment your software engineering skills, and for junior engineers it can help you learn that skill / get that promotion much faster. Apart from the two scenarios I described in this article, theres a lot more use cases where ChatGPT can help. Some other ones are-</p><ul><li>Exploring alternative approaches</li><li>Code documentation</li><li>Generating unit tests</li><li>Refactoring file paths/patterns</li></ul><p>I’m going to continue writing about this, because this is a once-in-a-lifetime technology that you need to understand and start using. There’s no telling how this tool affects the field of software engineering 5 years in the future, so the best you can do is to leverage it in the early stages and supercharge your skills.</p><p>My next article is going to be about <strong>how (and what specialized software engineering skills)</strong> I’m using ChatGPT to learn, which will help me stay indispensable. If you’d like to know when that article is published, subscribe using the link below.</p><p><em>Thanks for checking out this article! Leave a comment below if you have any questions. If you enjoyed this article and would like to see more content like this, I’m building a mailing list of engineers, entrepreneurs and builders. Sign up for it below.</em></p><p><a href="https://harshrana.substack.com/">techbits - by Harsh Rana</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a9182aec5da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Move over 10x engineers — it’s time for the supercharged ChatGPT engineers to take over]]></title>
            <link>https://levelup.gitconnected.com/move-over-10x-engineers-its-time-for-the-supercharged-chatgpt-engineers-to-take-over-3ce97c9671e5?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ce97c9671e5</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[careers]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Fri, 30 Dec 2022 15:14:44 GMT</pubDate>
            <atom:updated>2022-12-30T15:14:44.544Z</atom:updated>
            <content:encoded><![CDATA[<h3>Move over 10x engineers — it’s time for the supercharged ChatGPT engineers to take over</h3><h4>What started out as a personal experiment, has quickly become a tool in my toolbox as a senior software engineer. Whether you’re a junior engineer or a seasoned vet, you need to learn how to collaborate effectively with ChatGPT</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3_2_zIbugvVUjIIv" /><figcaption>Photo by <a href="https://unsplash.com/@agk42">Alex Knight</a> on <a href="https://unsplash.com/s/photos/Chat-AI">Unsplash</a></figcaption></figure><p>There’s quite a few articles on the internet talking about ChatGPT and the implications it’s going to have on software engineering. Some have chosen to <a href="https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/">point out the gaps in the tool</a> while others have focused on <a href="https://blog.bitsrc.io/i-asked-chat-gpt-to-build-a-to-do-app-have-we-finally-met-our-replacement-ad347ad74c51">trying to make ChatGPT straight-up write code</a>. In this article, I’ll show you one of the best use cases I’ve seen for this awesome tool — as a collaborator for designing software systems.</p><p>But before we start…</p><h3>How should we evaluate the performance of an engineer (or AI) designing a system?</h3><p>In <a href="https://www.linkedin.com/in/alex-xu-a8131b11/">Alex Xu’</a>s book, <a href="https://www.amazon.com/System-Design-Interview-insiders-Second/dp/B08CMF2CQF">System Design Interview</a>, he describes the process of evaluating a candidate’s system design¹ based on how they-</p><ol><li>Analyze a vague problem</li><li>Solve the problem step-by-step</li><li>Explain the idea</li><li>Discuss with others</li><li>Evaluate and optimize the system</li></ol><p>I believe these are great evaluation points, and most engineers would relate to these metrics, when designing systems for their day-to-day jobs. Over the last few weeks, I’ve used these to analyze ChatGPT’s performance and will use the same today when working through an example system design.</p><h3>What’re we asking ChatGPT to design?</h3><p>Today, I’m asking ChatGPT to help me with some backend system design. More specifically, I’m trying to build a <strong>social media</strong> and need to understand what to think about when <strong>designing the database</strong>. In a future article, we’ll try and do the same exercise for the API.</p><p>One point of clarification is that I’m treating ChatGPT just like any other engineer on my team — a collaborator — and not like a replacement for my own thinking/analysis and judgement. The goal is to use this new tool to supercharge my existing skills and be more efficient.</p><p>In the following two sections, I’ll share the conversation I had with ChatGPT and then analyze it.</p><h3>A working session with ChatGPT</h3><p>In the interest of word count, space and your time I won’t copy-paste my full conversation here. I’ll share the most insightful bits and the final SQL DDL generated by ChatGPT. Here’s the <a href="https://shareg.pt/MvmA1bI">full conversation</a> (powered by this awesome tool called <a href="https://sharegpt.com/">ShareGPT</a>) for those of you interested in the process.</p><p>I started with a high level ask -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nLnCm5nGV1TiFitUTGiHYw.png" /></figure><p>And then dove in deeper -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Mjb9iBnTU4Sb43oLDXR_Q.png" /></figure><p>After this, I asked some probing questions like -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QwxifM7Ht_PR_FHEFv1f2w.png" /></figure><p>As you can already see, this isn’t any different than working with any other engineer on your team. You start from the top, dive in as needed and ask probing questions. Finally, you ask them to do some research, make a decision and present their decisioning process to the team. You can see this in action below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_bk6PmZPC1_0Q-AXm0GVA.png" /></figure><p>Wow. ChatGPT presented me with some options and when provided with some constraints was able to help me make a decision (with a well thought though explanation).</p><p>After this, I was satisfied with the state of the design so I asked it to help me design specific tables. An example of this can be seen below (note: the ERD link was bugging, but I didn’t mind)-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ok-EWpRNV6lHVP1suxlu9w.png" /></figure><p>I repeated this for the remainder of the tables and concluded with asking it to help me write the SQL DDL of all the tables we discussed. The final SQL DDL of all the tables I discussed with ChatGPT can be seen below-</p><pre>-- Users table<br>CREATE TABLE users (<br>  id INTEGER PRIMARY KEY AUTOINCREMENT, -- Unique identifier for each user<br>  username TEXT NOT NULL, -- User&#39;s username<br>  password TEXT NOT NULL, -- User&#39;s hashed password<br>  name TEXT NOT NULL, -- User&#39;s full name<br>  profile_picture TEXT, -- Reference to user&#39;s profile picture (file name or URL)<br>  email TEXT NOT NULL, -- User&#39;s email address<br>  location TEXT -- User&#39;s location (optional)<br>);<br><br>-- Relationships table<br>CREATE TABLE relationships (<br>  id INTEGER PRIMARY KEY AUTOINCREMENT, -- Unique identifier for each relationship<br>  user_id_1 INTEGER NOT NULL, -- ID of first user in the relationship<br>  user_id_2 INTEGER NOT NULL, -- ID of second user in the relationship<br>  relationship_type TEXT NOT NULL, -- Type of relationship (e.g. friend, follower)<br>  status TEXT NOT NULL, -- Status of the relationship (e.g. pending, accepted)<br>  created_at DATETIME NOT NULL, -- Date and time the relationship was created<br>  FOREIGN KEY (user_id_1) REFERENCES users (id),<br>  FOREIGN KEY (user_id_2) REFERENCES users (id)<br>);<br><br>-- Posts table<br>CREATE TABLE posts (<br>  id INTEGER PRIMARY KEY AUTOINCREMENT, -- Unique identifier for each post<br>  user_id INTEGER NOT NULL, -- ID of the user who made the post<br>  content TEXT NOT NULL, -- Content of the post (text, image, video)<br>  type TEXT NOT NULL, -- Type of post (text, image, video)<br>  created_at DATETIME NOT NULL, -- Date and time the post was made<br>  updated_at DATETIME, -- Date and time the post was last updated<br>  likes INTEGER DEFAULT 0, -- Number of likes the post has received<br>  shares INTEGER DEFAULT 0, -- Number of times the post has been shared<br>  comments INTEGER DEFAULT 0, -- Number of comments the post has received<br>  FOREIGN KEY (user_id) REFERENCES users (id)<br>);<br><br>-- Comments table<br>CREATE TABLE comments (<br>  id INTEGER PRIMARY KEY AUTOINCREMENT, -- Unique identifier for each comment<br>  post_id INTEGER NOT NULL, -- ID of the post to which the comment belongs<br>  user_id INTEGER NOT NULL, -- ID of the user who made the comment<br>  content TEXT NOT NULL, -- Content of the comment<br>  created_at DATETIME NOT NULL, -- Date and time the comment was made<br>  likes INTEGER DEFAULT 0, -- Number of likes the comment has received<br>  FOREIGN KEY (post_id) REFERENCES posts (id),<br>  FOREIGN KEY (user_id) REFERENCES users (id)<br>);<br><br>-- Likes table<br>CREATE TABLE likes (<br>  id INTEGER PRIMARY KEY AUTOINCREMENT, -- Unique identifier for each like<br>  user_id INTEGER NOT NULL, -- ID of the user who liked the content<br>  post_id INTEGER, -- ID of the post that was liked (optional)<br>  comment_id INTEGER, -- ID of the comment that was liked (optional)<br>  FOREIGN KEY (user_id) REFERENCES users (id),<br>  FOREIGN KEY (post_id) REFERENCES posts (id),<br>  FOREIGN KEY (comment_id) REFERENCES comments (id)<br>);<br><br>-- Analytics table<br>CREATE TABLE analytics (<br>  id INTEGER PRIMARY KEY AUTOINCREMENT, -- Unique identifier for each analytics record<br>  user_id INTEGER NOT NULL, -- ID of the user who performed the action being tracked<br>  action TEXT NOT NULL, -- Type of action being tracked (e.g. view, like, share)<br>  object_type TEXT NOT NULL, -- Type of object on which the action was performed (e.g. post, comment)<br>  object_id INTEGER NOT NULL, -- ID of the object on which the action was performed<br>  timestamp DATETIME NOT NULL, -- Date and time the action was performed<br>  FOREIGN KEY (user_id) REFERENCES users (id)<br>);</pre><p>You can quickly see the value of the above exercise. Instead of starting from a blank page, I was able to leverage ChatGPT to talk me through some of the different aspects of the system and give me a generic set of tables/fields to start working with. From here, I can think about some business-specific logic we may have missed and modify the tables as needed.</p><p>Additionally, I was able to ask questions about different parts of the system, probe on some missed functionality and even ask it to make a decision given the budget of the project. Now that’s awesome!</p><h3>Takeways</h3><p>This isn’t a complete solution by any means. But you can use the same technique to dive into the other parts of the system — user authorization/authentication, performance, scalability, privacy, security etc. At the end of the day, it’s your responsibility as an engineer to combine your learnings from all these conversations, leverage any code ChatGPT produced and actually implement the system.</p><p>ChatGPT is a great tool for software engineers to add to their toolbox. While it’s not complete enough to replace your jobs, it can certainly take over some of the heavy lifting involved in ideating system designs. If someone has previously built the system you’re trying to build (especially if their code has a public footprint), there’s a good chance that ChatGPT can share some insights into it.</p><p>This means that engineers who learn to use this tool will become more efficient and build sh*t faster.</p><p><em>Thanks for checking out this article! Leave a comment below if you have any questions. If you enjoyed this article and would like to see more content like this, I’m building a mailing list of engineers, entrepreneurs and builders. Sign up for it below.</em></p><p><a href="https://harshrana.substack.com/">techbits - by Harsh Rana</a></p><p>[1]: <a href="https://bytebytego.com/courses/system-design-interview/foreword">https://bytebytego.com/courses/system-design-interview/foreword</a></p><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></li></ul><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ce97c9671e5" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/move-over-10x-engineers-its-time-for-the-supercharged-chatgpt-engineers-to-take-over-3ce97c9671e5">Move over 10x engineers — it’s time for the supercharged ChatGPT engineers to take over</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can an algorithm pick a winning NBA Fantasy Draft?]]></title>
            <link>https://medium.com/data-science/can-an-algorithm-pick-a-winning-nba-fantasy-draft-c05342f130f2?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/c05342f130f2</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[nba]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[basketball]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Thu, 03 Sep 2020 18:09:32 GMT</pubDate>
            <atom:updated>2020-09-03T18:15:19.059Z</atom:updated>
            <content:encoded><![CDATA[<h3>Can an Algorithm Pick a Winning NBA Fantasy Draft?</h3><h4>Using data analysis, modeling, and optimization to pick a winning team</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AlI2EGtZ5dESXOYEBtwf4w.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@markusspiske?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Markus Spiske</a> from <a href="https://www.pexels.com/photo/ball-basketball-basketball-court-basketball-hoop-1752757/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p><strong><em>Note from Towards Data Science’s editors:</em></strong><em> While we allow independent authors to publish articles in accordance with our </em><a href="https://towardsdatascience.com/questions-96667b06af5"><em>rules and guidelines</em></a><em>, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our </em><a href="https://towardsdatascience.com/readers-terms-b5d780a700a4"><em>Reader Terms</em></a><em> for details.</em></p><p>I enjoy basketball. It’s a fast-paced competitive game and I’ve enjoyed both playing and watching it for a long time. The NBA is famous for generating <a href="https://stats.nba.com/">very clean data</a>, which has long been used by enthusiasts (like myself) for <a href="https://buckets.peterbeshai.com/app/#/playerView/201935_2015">data visualizations</a>, <a href="https://sports.sites.yale.edu/nba-model-math">modeling</a> and <a href="https://towardsdatascience.com/predict-nba-games-make-money-machine-learning-project-b222b33f70a3">game predictions</a>.</p><p>Recently, I was contacted by <a href="https://www.draftkings.com/lobby">DraftKings</a> regarding an interview for a potential job. As part of my preparations for the same, I started using their platform and competing in mock competitions to get acquainted with the DraftKings (DK) contest process. It was during this time period that I really started getting into the idea of using data to model and predict a winning roster.</p><p>I built the algorithm iteratively, and from scratch- starting with a naive version 1, a more robust version 2 and currently I’m working on a winning version 3.</p><blockquote>I built the algorithm iteratively, and from scratch</blockquote><p>You can follow along my algorithm design journey in the rest of the article.</p><h3>Quick Level Set: Scoring and Rules</h3><p>DK’s rules and scoring for their <strong>NBA classic</strong> fantasy contests are fairly intuitive, even if you have no prior basketball knowledge. In a nutshell, the objective is to:</p><blockquote>Create an <strong>8-player lineup</strong> while staying under the $50,000 salary cap.</blockquote><p>Players get different points for different actions (more details below) and the draft with the most number of points, at the end of all games in a night, wins. Sounds simple enough :)</p><p>The breakdown for different actions that result in positive (or negative) points can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*xPwkBN0CB3clZxLz5CTKxw.png" /><figcaption>NBA Fantasy points breakdown- DraftKings. Photo by Author.</figcaption></figure><p>One last constraint which makes drafting slightly more complicated is player positions. According to DK: <em>Lineups will consist of 8 players and must include players from at least 2 different NBA games.</em></p><p>Further, the 8 players are broken down by positions, which can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*eH1bQDoLyUdv0oxCXXasbg.png" /><figcaption>NBA Fantasy player positions- DraftKings. Photo by Author.</figcaption></figure><p>There you have it! A simple optimization problem with a set of constraints. Sounds like something an algorithms would excel at. <strong>Or would it?</strong></p><h3><strong>Algorithm Version 1- Naive</strong></h3><p>My goal with this algorithm was to build it as fast as possible, with little to no hopes of winning. Mainly because I was interested in setting up a strong foundation, without worrying about building complex logic early in the process. To do this, I downloaded a player dataset from DK and started a Jupyter notebook. If you’re interested, you can find the full raw data <a href="https://github.com/hr23232323/draftkings_nba/blob/master/previous_day_salaries/08-26/DKSalaries.csv">here</a> and my notebook <a href="https://github.com/hr23232323/draftkings_nba/blob/master/optimize-%20V1.ipynb">here</a>.</p><p>Let’s see what our data looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ekv4_BhYTSH1RBTbItkkgQ.png" /><figcaption>Players dataset- DraftKings. Photo by Author.</figcaption></figure><p>Right off the bat, we can tell that for a simple algorithm, given our requirements and constraints, we’ll find the following columns useful: ID, Salary and AvgPointsPerGame (fantasy points). This would allow us to pick the “best” players while staying under the $50,000 salary cap. Sure, without positional information we could have overlaps etc. but that’s an issue for a later version. Remember, version 1 should be the simplest implementation of your product.</p><p>Given this data, our first pass optimization algorithm can be broken up into the following simple steps:</p><ol><li>Randomly select 8 players from the dataset.</li><li>If the sum of the salaries of the players is greater than $50,000: go back to step 1 (too expensive).</li><li>Otherwise, sum the AvgPointsPerGame of each of the players in the roster and compare with a master maximum value. If greater, replace maximum value and roster.</li><li>Unless all possible combinations have been explored, return to step 1. Once no more combinations, return the maximum value and the roster.</li></ol><p>There we have it: a simple naive algorithm that picks 8 players in random that will have the maximum expected fantasy points while staying under the $50,000 salary cap. But this algorithm has a few glaring issues:</p><ul><li>No control regarding the position of the players. Hence the algorithm could generate a roster which consists of &gt;3 of one position (G/F), in which case the roster would be invalid.</li><li>No check on players who are injured or not scheduled to play. This would result in a most definitive loss as all player points are important for a winning draft.</li><li>Lastly, the algorithm is very inefficient. Considering that we need to check each possible roster: for a given number of players n and roster size r, the number of possible rosters would be-</li></ul><blockquote>C( n , r ) = n! / (n — r)! . r!</blockquote><p>To get an appreciation of this complexity, take a look at the table below which shows the number of checks if the total number of available players is 100.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*Vf34X-2VA1OvIVIfD0aRhQ.png" /><figcaption>Time complexity magnitude for first algorithm. Photo by Author.</figcaption></figure><p>It’s safe to assume that our algorithm will take a VERY long time to output a roster of 8 players. But, because this is a first pass algorithm, we‘re happy with what we got. You can see the algorithm in action below, picking the top 5 players for a combined salary of $35,000. Not bad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*mh0qnOjjbcGLglqj6PMxVw.png" /><figcaption>Output from algorithm 1- Top 5 players with the maximum expected points under $35,000 combined salary. Note: first row shows combined expected fantasy points, second row is combined salary and third are the IDs of the player, followed by the names. Photo by Author.</figcaption></figure><p>Because we’re on a mission to build a winning algorithm, let’s talk about version 2 optimizations.</p><h3>Algorithm Version 2- Intermediate Sports Bettor</h3><p>Now, this is where our algorithm goes from being a naive optimizer to an intermediate-level sports bettor. Based on the drawbacks of version 1, and the factorial time complexity, I decided to implement a few <strong>data and algorithm level optimizations</strong>.</p><p>First, I cleaned the data to only include players who’re confirmed to play games. This was an easy way to decrease the total number of available players from ~100 to ~85. This might look like a small increase, but in reality, for a roster of 8 players, our number of checks drastically decreases when the total number of players decreases. The change in the number of checks can be seen below.</p><ul><li>C (100, 8) = 186,087,894,300</li><li>C (85, 8) = 48,124,511,370</li></ul><p><strong>Our total number of operations (or checks) in the algorithm went down by ~75%!</strong></p><p>Next up, I modified the algorithm itself to pick specific positions. Now, instead of picking every possible roster from the total number of players available, the algorithm picks 3 guards from only all available guards, followed by 3 forwards and lastly 1 center. As you can see, the total here is only 7 players and leaves the last pick to the user. This is a quick way to save some additional time on the algorithm as the user can manually find the best remaining player (highest expected points given the salary remaining).</p><p>This was a huge optimization because the number of guards vs the total number of players is ~40 vs 85. The number is similar for forwards and even less for centers. Note, there’s a slight overlap between the players in each category as some players play multiple positions but this was easy to deal with: I removed played who were already picked as Guards, before picking Forwards etc. The performance boost as a result of the above changes can be seen below:</p><ul><li>C (85 , 8) = 48,124,511,370</li><li>C (40 , 3) x C (40 , 3) x C (20, 1) = 1,952,288,000</li></ul><p>This is huge. <strong>Now, the algorithm is conducting almost ~95% fewer operations </strong>and we have the best possible roster broken up by positions and under our salary cap. Let’s test our results!</p><h3>Real World Results</h3><p>If you’ve made it so far, congratulations. You’ve worked through the technical stuff, now it’s time for the results! I tried the algorithm’s pick over the course of three days on DK’s classic multiplier contests. <strong>Each time my entry fee was $1 and the payoff was $3 for the top 30% of the finishers. </strong>You can see the lineups generated by the algorithm and the results below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*aSQueOIBfgeDzxu5HHBctw.png" /><figcaption>Day 1 — Oh no, finished at rank 26 and lost money. DK screenshot by Author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*LFO81o365wp-xiqzpZGE-w.png" /><figcaption>Day 2 — Oh no again, finished at rank 26 and lost another dollar. DK screenshot by Author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*UBEgTYehkx68xYfvoJFvAA.png" /><figcaption>Day 3- Woohoo! Finished at rank 3 and made $3. DK screenshot by Author.</figcaption></figure><p>As you can see from the above results, the real world outcomes of the competition have been good! Out of the three days that I created lineups using the algorithm, we lost twice and won once. Our intermediate-level sports bettor algorithm has done better than I expected, but there’s still a long way to go.</p><p>I noticed few nuances about the results, including that our algorithm (before the v2 optimization) made a mistake on day 1 where an injured player was drafted into the team (P. Beverley) which resulted in a weak draft. This was fixed in version 2 and will not be repeated again. Additionally, once cool thing has been that despite the mixed results, the algorithm has consistently created lineups which get &gt;200 fantasy points, which is pretty high!</p><h3>What’s next?</h3><p>Well, there you have it. So far, I’ve spent $3 on entree fees and made $3 on winnings, for a grand total of $0 change! I have $25 left to spend on this project before my inner alarm bells start ringing, so I clearly need to improve this algorithm. After talking to some of my friends, who know a lot more about basketball than myself, I have a few hypotheses to test out. Some of these include:</p><ul><li>Using additional player data over the last <em>n</em> games. This way the model would have more context, instead of just a snapshot value</li><li>Using prior team match-up data to adjust weights placed on certain games. For example, this could help avoid picking a player in a match-up where (based on previous meets) the player has failed to perform</li><li>Exploring dual optimization strategies</li></ul><p>And more! If you have any ideas about how to improve this project please feel free to reach out to me on <a href="https://www.linkedin.com/in/harshrana1997/">LinkedIn</a> or over email which you can find on my <a href="https://www.harshrana.com/">Website</a>. Additionally, all the data and code for this project can be found on my <a href="https://github.com/hr23232323/draftkings_nba">Github repository</a>, so feel free to clone/fork it and test your own hypotheses! And, as always, any and all feedback is greatly appreciated.</p><p>Stay safe out there everyone and keep building cool stuff.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c05342f130f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/can-an-algorithm-pick-a-winning-nba-fantasy-draft-c05342f130f2">Can an algorithm pick a winning NBA Fantasy Draft?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 2020 US Stock Market Drop was bad]]></title>
            <link>https://medium.com/data-science/understanding-the-2020-us-stock-market-drop-using-data-visualizations-bccd2d91bf9f?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/bccd2d91bf9f</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[stock-market]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Tue, 31 Mar 2020 21:10:26 GMT</pubDate>
            <atom:updated>2020-03-31T22:01:51.040Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding the 2020 US Stock Market Drop using Data Visualizations</h3><h4>Analyzing the severity of the COVID-19 related US stock market sell-off using data visualizations and a historic context.</h4><p>If you don’t know about the US stock market drop that occurred over the last 2 months, then you’re either:</p><ol><li>A college student, OR</li><li>An entry-level worker with no investments and student loans worth a lifetime</li></ol><p>Either way, whether or not you’re invested in the market, you should know about what’s happened over the last few weeks. Depending on where you stand, you’re either lucky or unlucky enough to be witnessing a <a href="https://www.investopedia.com/terms/b/blackswan.asp"><strong>Black Swan</strong> </a>event. Quick level set: a black swan event is defined beautifully by Investopedia as:</p><blockquote>An unpredictable event that is beyond what is normally expected of a situation and has potentially severe consequences. Black swan events are characterized by their extreme rarity, their severe impact, and the widespread insistence they were obvious in hindsight.</blockquote><p>In this article, I’ll be quantifying the impacts of the 2020 US stock market drop and comparing it with other major drops from recent (and not-so-recent) history using data science. <strong>I’ll be gathering data to help build my cases, constructing comparative statistics to give context and building visualizations to communicate important ideas</strong>.</p><p>One last thing, I’m only using publicly available data so all the code for the data visualizations and raw data used in this article is readily available on my <a href="https://github.com/hr23232323/market_drop_analysis">project GitHub</a>. Now, let&#39;s get started!</p><h3>Quantifying the market turmoil over the last few weeks</h3><p>There are a few different datasets we can use to understand the absolute turmoil in the markets. Let&#39;s start by analyzing the standard statistics of the S&amp;P 500 over the last ~3 months (01/02/20–03/20/20). This is what my dataset looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*F2g4fNHWc3c0BOaCrnUeFQ.png" /><figcaption>Standard dataset showcasing open, high, low, close, change% and volume of the S&amp;P 500</figcaption></figure><p>You probably see these descriptors every time you search for a stock on the web; now, let&#39;s do some analyses to understand them better. I’ll start by showing the simple price/timeline chart which already reveals a lot of information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HO4QevzcuVA_zvvJFx92WA.png" /></figure><p>You can clearly see that sometime around the middle of February (~02/19), the market volatility started increasing aggressively. Note that the volatility increase is the focus, not necessarily whether the trend was up or down. Let’s take this one step further by adding a visualization of market volume over time, which showcases just how many transactions were taking place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TtaEKHY1aKzQH6wD_xz0Ig.png" /></figure><p>Now you can also see that market volatility and volume both increased, specially over the last few weeks. At least on the surface, another thing you can see from the graph above is that the first time the volume really skyrocketed was a day or two after the market took the first few big drops.</p><p>Let’s use another visualization to see % change in price and market volume in conjunction. These descriptors use different scales so we’ll use a derived statistic to see abnormalities. A good way to do this is using <strong>Z-score</strong>. Simply put, Z-score is a statistic that gives you an idea of how far from the mean a data point is. Given our sample size of just the last 3 months, the mean is extremely skewed, so I’ll be using the <strong>modified Z-score</strong> which relies on the median as it is less sensitive to outliers and skew. Additionally, I’ll only be using the absolute value of the modified z-score because I’m only interested in the distance from the median, not direction. You can see the results below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZABVEY0Ja4UbY5FZIiC5iA.png" /></figure><p>Now we’ve confirmed that the abnormal % change in the market preceded the abnormal volume. From the graph, you can see that something happened over the weekend of February 22nd which led to the % change shooting up from ~0.5 standard deviation from the median, all the way up to ~2 standard deviations. In a follow up post I’ll be exploring the Coronavirus cases over that weekend to further understand what happened.</p><p>While looking at this timeline from a Z-score perspective is useful, one last visualization which further captures the panic in the market can be seen below, where I simply plot the daily % change in the S&amp;P 500. Notice how the peaks and valleys really explode over the last few weeks and the number of outliers in the box plot of the same data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aCn0MCBaCHxAhlli5cB-jA.png" /></figure><p>We’ve spent the last few minutes analyzing the US market drop in isolation. In the next section, we’ll be comparing this drop with others in recent history to get a better context and really answer the following question:</p><h3>Just how bad was this market drop?</h3><p>Pretty. Bad.</p><p>One of the strategies which is most useful in understanding the behavior of the market at any point, is getting some historic context. In this section, we’ll be comparing the 2020 US stock market drawdown with others from recent history to further analyze the market reaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*l3Bws1ygXLXNjqu212kY8g.png" /></figure><p>To do this, I collected data about the ten most recent 10%+ drawdowns in the market. As you can see, this list captures some 10–20% drawdowns and also the financial crisis of 2007–2008 and the dot com bubble of 2000, which were both much more severe. After this, I collected the daily S%P 500 returns during these time periods. My goal here was to compare, among other statistics, the rate of drop of the 2020 stock market drop with those from over the last 20 years.</p><p>After ideating a few different data visualizations, I decided to use a spaghetti plot to best showcase the severity of the most recent drop. The visualization can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gFjRa8h9ulpSGuC5fIIgMg.png" /></figure><p>As you can see I decided to build the graph to showcase the drop in the value of a $10,000 investment from the start of the drawdown. This created a level ground where analyzing the severity of the drawdown was possible. As you can see from the graph, not only was the 2020 US market drop much more severe in the absolute drop in investment value, but it was also a much faster drop. This can again be seen in the following graph where I’ve separated each of the drawdowns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sc-qg7kbMDzJp_jKmk-FYw.png" /></figure><p>Well, there you have it! The 2020 US market drop was by no means a minor event. Over the course of 30 days, it wiped out almost $8T from the overall market cap of the S&amp;P 500. <strong>To be more specific, that’s more than $8,000,000,000,000.</strong> In the future, I’ll be looking at datasets associated with Coronavirus to further analyze the market behavior.</p><p>I hope you enjoyed reading this article just as much as I enjoyed putting it together. If you have any questions, feedback or comments, feel free to get in touch with me through <a href="https://www.linkedin.com/in/harshrana1997/">LinkedIn</a>, or my <a href="http://www.harshrana.com/">website</a>. Additionally, all the code for the data visualizations and raw data used in this article is readily available on my <a href="https://github.com/hr23232323/market_drop_analysis">project GitHub</a>.</p><p>Stay safe out there everyone and keep building cool stuff.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bccd2d91bf9f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understanding-the-2020-us-stock-market-drop-using-data-visualizations-bccd2d91bf9f">The 2020 US Stock Market Drop was bad</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trump said 1700 Google Engineers are building a Coronavirus assessment tool. I built it in 24 hours]]></title>
            <link>https://levelup.gitconnected.com/trump-said-1700-google-engineers-are-building-a-coronavirus-assessment-tool-i-built-it-in-24-hours-79342d6abe6e?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/79342d6abe6e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[coronavirus]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Fri, 20 Mar 2020 14:51:22 GMT</pubDate>
            <atom:updated>2020-03-20T15:51:28.442Z</atom:updated>
            <content:encoded><![CDATA[<h3>Trump said 1700 Google Engineers would build a Coronavirus assessment website. I built it in 24 hours.</h3><h4><em>President Trump </em><a href="https://www.factcheck.org/2020/03/trump-misrepresents-google-coronavirus-website/"><em>said last week</em></a><em> that Google is working with the United States Government on building a website to help people determine whether they need to get tested for COVID-19 and that “Google has 1700 engineers working on this right now.”</em></h4><h4><em>I built a simple website that does just that, in 24 hours.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KtVhk8csh2I4vkuqMHabFw.png" /><figcaption>Homepage for the Coronavirus assessment website that I built: covidscan.com</figcaption></figure><h3>Motivation</h3><p>I had two main motivations when building this project:</p><ol><li>I wanted to see how complicated it would be to build such an assessment tool, given the value of time in this situation. Of course, I wasn’t trying to build something which would be used by 500+ million people, but it would serve as a proof of concept.</li><li>I had a lot more extra time on my hands given the social distancing prescription being handed out and wanted a productive way to spend this extra time</li></ol><p>Based on these motivations, I wanted to make this website as fast as possible, using a low overhead technology stack. The idea itself was simple enough, so I decided to use vanilla HTML, CSS, and JavaScript. For deployment, I decided to use the trusted services of Netlify. I figured that the website would be lightweight enough and given Netlify’s free tier (which allows up to 100GB bandwidth/month), it would be the perfect choice. Now let’s talk about design.</p><h3>Design</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*cGfGnzC1FRbh50AoaoZKrA.png" /><figcaption>Mockup for the website</figcaption></figure><p>I wanted to create a website that was minimal, to-the-point, and easy to use for both old and young individuals. I determined it was best to create a one-page application in the form of a single questionnaire that can be completed in less than 30 seconds. Participants would be asked to enter as little information as possible and still receive a useful assessment.</p><p>After going through a couple of design iterations, I decided to use a combination of buttons and checkboxes to get all the data as seamlessly as possible. The design can be seen here.</p><p>As you can see, I placed a lot of importance on making the user experience as smooth as possible. The buttons would be used to answer multiple choice questions, while the checkboxes allowed users to select multiple responses that applied to them. A combination of these would enable the website to capture all the personal conditions, health factors, and symptoms needed to give an assessment.</p><p>Additionally, I decided to stick with a simple one-page layout because that would minimize the confusion for older users and make the entire experience as simple as possible.</p><p>Users would see a disclaimer as soon as they get on the page before they even started the questionnaire. This was a precautionary step even though the website did not include any sort of medical advice. With health related websites, one can never be too safe.</p><h3>Execution</h3><p>Building the actual website frontend was fairly straightforward. I used HTML forms to create separate forms for each of the questions. The reason I chose to create multiple forms, rather than just the one, was because it allowed me to create a very simple website with a single question and develop it end-to-end fast. This allowed me to build iteratively and get a minimum viable product on the market as fast as possible, while still being able to add features without any downtime.</p><blockquote>This allowed me to build iteratively and get a minimum viable product on the market as fast as possible, while still being able to add features without any downtime</blockquote><p>For the backend, I decided to utilize a simple Google Sheet to mimic a database. The website would not be capturing much data and thus the time and resources required to get a database up and running was unnecessary. I would like to give a HUGE shout out to <a href="https://medium.com/u/4a2d079fd895">David McCoy</a> for his awesome article which clearly outlines <a href="https://medium.com/@dmccoy/how-to-submit-an-html-form-to-google-sheets-without-google-forms-b833952cc175">how to set up a Google Sheet to capture data from a static website</a>. I only had to implement minor modifications to his tutorial for my use case, but again the entire process was made that much simpler given the abundance of guides by smart people like David.</p><p>In a nutshell, my data pipeline consisted of 3 simple steps:</p><ol><li>Users answer a question</li><li>Backend JavaScript blocks the default submission and instead adds the question (and their answer) to a JS object</li><li>Steps 1 and 2 are repeated until the user answers all questions. Once they’re done, backend JS submits a request using AJAX to store the information in the Google Sheet</li></ol><p>Given the simplicity of the entire project design, it didn’t take me long to get the website up and running!</p><h3>Results</h3><p>You can check out the actual website by going to <a href="http://www.covidscan.com/">www.covidscan.com</a>. You can also see a demo of the final website below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8xZmtnodRn3Vjc2OtUBQMg.gif" /><figcaption>Working demo of the final product</figcaption></figure><p>As promised, the website was kept extremely minimal, easy-to-use and was deployed in under 24 hours. I used Netlify to deploy the website directly through the <a href="https://github.com/hr23232323/corona-test">project Github</a> for free. I also used Google Domains for purchasing the domain name “covidscan.com” for $12. Feel free to share the link with your friends and family who feel like they’re showcasing symptoms. The assessment is completely based on the <a href="https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html">CDC guidelines</a> and can be a helpful way to minimize the load on the healthcare workers.</p><p>I hope you enjoyed reading this article just as much as I enjoyed putting it together. If you have any questions, feedback or comments, feel free to get in touch with me through <a href="https://www.linkedin.com/in/harshrana1997/">LinkedIn</a>, or my <a href="http://www.harshrana.com/">website</a>. Overall, this was a very satisfying way to make use of the extra social distancing time prescribed to the nation during these crazy times, so be on the lookout for more such projects from me.</p><p>Stay safe out there everyone and keep building cool stuff.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79342d6abe6e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/trump-said-1700-google-engineers-are-building-a-coronavirus-assessment-tool-i-built-it-in-24-hours-79342d6abe6e">Trump said 1700 Google Engineers are building a Coronavirus assessment tool. I built it in 24 hours</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How 15 minutes saved me over $100/year in hosting costs for my personal website]]></title>
            <link>https://levelup.gitconnected.com/how-15-minutes-saved-me-over-100-year-in-hosting-costs-for-my-personal-website-b6483b1bd03d?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6483b1bd03d</guid>
            <category><![CDATA[personal-branding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[computer-programming]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Mon, 20 Jan 2020 23:37:01 GMT</pubDate>
            <atom:updated>2020-01-20T23:37:01.260Z</atom:updated>
            <content:encoded><![CDATA[<h4>A strong web presence is easier and cheaper than you thought, let me prove it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1L1QQaVO9frSNAy88poE_g.png" /><figcaption>My personal website from 2014. Ah, the good ol’ days..</figcaption></figure><p>I’ve had a personal website for quite some time. I built the first version in 2014 as a junior in high school. It was a fairly simple website with my face, some details about my school involvements, and aspirations (as you can see above). It wasn’t the most complicated website, but back in 2014 with my limited knowledge of web development and deployment, I chose Bluehost as my web hosting provider.</p><p><strong>This article outlines how I paid WAY more than I needed to host my website from 2014 to 2019, and how I dropped my cost to less than 10% of what it used to be by using Netlify and Google Domains.</strong></p><h3>The (purposefully) complicated world of website hosting</h3><p>Before I talk about the services I used and paid for, let me cut through the complicated world of web hosting and introduce the only major services anyone really needs: <strong>domain manager and hosting provider</strong>.</p><p>The <strong>domain manager</strong> is a platform that allows you to buy new domains. A domain is just another word for website addresses such as “ilikewebdev.com”, which is an awesome name in my opinion. The <strong>hosting provider</strong> is a company which allows you to upload website-related files and serve those pages to users. These companies typically have their servers around the world and allow simple folks, such as you and I, to use their servers for “hosting” our websites.</p><p>Together, these two services are combined and provided by many web hosting companies like Bluehost, GoDaddy etc. Now, while these are the main services you need, there are many add-ons which can make your life a lot easier in different ways. Some of the ones I opted into include:</p><ol><li><strong>Domain Privacy + Protection</strong>: to register a website domain, you need to provide your details such as name, address, phone number, email etc. and all your information is available for publicly unless you buy this add-on. I spent 2 days without this and literally received 100’s of calls and messages, don’t make my rookie mistake.</li><li><strong>HTTPS Certificate</strong>: you can choose to have an unsecured website, but search engines like google tend to rank them lower than ones that have HTTPS enabled. Thankfully most service providers allow you to add this feature for free in 2020, but still something to be aware of.</li></ol><p>Now that you understand the world of web hosting and the services needed to buy a domain and get a website up and running, let me describe the breakdown of my recurring costs.</p><h3>Old cost breakdown</h3><p>I used Bluehost as both my domain manager and hosting provider from 2014 to 2019. Here’s the last bill I ever paid to Bluehost:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3qdvabXzOGTRiYBhSD31Sg.png" /><figcaption>This was SO much money which I never should’ve paid</figcaption></figure><p>As you can see, I was paying an absurd amount of money for the services I was using. In my defense, 17 year-old me didn’t know much better but thankfully 22 year-old me does. I decided to stop using Bluehost and move my website to new service providers altogether.</p><h3>Upgraded hosting tech stack</h3><p>Starting this year, I migrated my domain to be managed by <a href="https://domains.google/">Google Domains</a>. This was the first of two changes I made. There were a few reasons I picked Google Domains:</p><ol><li>Google is one of the most successful company of all time, so there’s a sense of security</li><li>Their costs were comparable — and sometimes less than — other domain managers</li><li>They provide Domain Privacy and Protection at no additional cost!</li></ol><p>So right off the bat, by switching to Google Domains I could stop paying $14.88/year for domain privacy and protection. Migrating my website from Bluehost to Google Domains was fairly simple and took &lt;5 minutes. You can learn more about this process by reading this <a href="https://support.google.com/domains/answer/9003139?hl=en">article</a>.</p><blockquote>By switching to Google Domains I could stop paying $14.88/year for domain privacy and protection</blockquote><p>The next change I made was to my hosting provider. I decided to move to <a href="https://www.netlify.com/">Netlify</a> because of many reasons:</p><ol><li>Netlify has a free tier which is perfect for static personal websites</li><li>They provide support for custom domains and HTTPS integration at no additional costs</li><li>They can be integrated with GitHub, thus providing an extremely developer-friendly deployment and maintenance experience</li></ol><p>Now this was a major game changer. With Bluehost, I was paying $8.99/month or $107.88/year; with Netlify, I pay exactly $0. Again, migrating hosting privileges from Bluehost to Netlify took &lt;10 minutes and the move from a file manager (used by Bluehost and most other hosting providers) to GitHub was a treat.</p><blockquote>With Bluehost, I was paying $8.99/month or $107.88/year; with Netlify, I pay exactly $0.</blockquote><p>You can understand the migration process and learn more about it by watching the following <a href="https://www.youtube.com/watch?v=Q9giWrfIJKk&amp;t=11s">video tutorial</a>.</p><h3>New cost breakdown and comparison</h3><p>Since migrating to Google Domains and Netlify, my server/hosting costs have been completely eliminated because the free tier provided by Netlify is perfect for me! I only pay $12/year for the right to own my own domain and host a website there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*KAgLD-uyQj5xKchk54rZig.png" /><figcaption>My ONLY cost related to my personal website</figcaption></figure><p>Compare this to my earlier costs of $14.88 (privacy protection) + $15.99 (domain) + $107.88 (hosting) = $138.75/year. <strong>This reduction represents a cost saving of $126.75 or in other words, a price reduction of &gt;90%!</strong></p><h3>Ending thoughts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cjkBsm6L4PJZaztryNx5AA.png" /><figcaption>Photo by <a href="https://www.pexels.com/@padrinan?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Miguel Á. Padriñán </a>from <a href="https://www.pexels.com/photo/web-text-1591060/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>It’s 2020 and owning a personal website which talks about you, your achievements and skills is easier and cheaper than ever. $12/year is roughly:</p><p><em>Six coffees</em></p><p>or</p><p><em>Three pints of ice cream</em></p><p>or</p><p><em>One lunch</em></p><p>Hope this article helps you realize that a stronger web presence is closer than you thought. PS: You can check out how my website has upgraded since 2014 by visiting <a href="http://www.harshrana.com/">harshrana.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6483b1bd03d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/how-15-minutes-saved-me-over-100-year-in-hosting-costs-for-my-personal-website-b6483b1bd03d">How 15 minutes saved me over $100/year in hosting costs for my personal website</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Web Application deployment in 5 steps]]></title>
            <link>https://medium.com/data-science/machine-learning-web-application-deployment-in-5-steps-d8eb6a389cdd?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8eb6a389cdd</guid>
            <category><![CDATA[data-science-sprints]]></category>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Wed, 08 Jan 2020 04:33:04 GMT</pubDate>
            <atom:updated>2020-01-08T04:33:04.611Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/data-science-sprints">Data Science Sprints</a></h4><h3>Machine Learning Web Application Deployment in 5 steps</h3><h4>You’ve built a model and an application; now it’s time to let the world see! Learn to deploy an ML web application — for free — using AWS.</h4><p>Let me paint you a picture:</p><p><em>You spent 2 months of your life working on a very exciting data science application. It involved advanced data collection, time-consuming data wrangling, hours of training and endless model micro-optimizations. Once you were finished, you had something that you were truly happy with. But today, 3 months since your blockbuster model was completed, it’s just sitting in a GitHub repository with your research notebooks and semi-documented code. You’ve gained 12 stars and 2 forks but wonder, “is this it? Do only 14 people in this entire world care about this awesome thing that I built? Should I have done something differently? If so, what?”</em></p><p>If you’ve had this experience of creating something wonderful, only to have a handful of people interact with it and even fewer give you feedback, <strong>you’re not alone</strong>. I’ve made this mistake often and I have a lot of friends in data science who’ve also had similar experiences. Now, I don’t want to take away from folks who enjoy working on theoretical problems and thrive in research settings, but I do think there are data scientists out there who need to showcase their work for the world to see. Not only would this help non-technical folks such as recruiters see and understand your work, but it would also open up the door to receiving real-world feedback on projects.</p><p><strong>In this article, I’ll share how to deploy a machine learning web application to an Amazon AWS EC2 instance — without Amazon Beanstalk — to keep our deployment entirely free-tiered</strong>. You can check out a live demo of our end product <a href="http://ec2-54-145-145-48.compute-1.amazonaws.com/"><strong>here</strong></a>.</p><p>While you can use your own models and application to follow along, I’ll be using one of my previous projects where I first built a sentiment analysis classifier and then a flask web application on top of that. You can find all my code for the model in <a href="https://github.com/hr23232323/nlp_medium_p1">this repository</a> and for my web application in <a href="https://github.com/hr23232323/nlp_medium_p2">this one</a>. I highly recommend the flask-based development tech stack as it is the easiest to deploy to AWS and should suffice for any basic application.</p><h3>Deployment Steps</h3><p>Let’s be real, Amazon Web services (AWS) can be super intimidating for someone who’s never worked with web technologies and backend services before. Honestly, I’ve worked professionally as a full-stack web developer and I still had a hard time navigating and getting started with AWS, because it just offers so many different services! Do you need an S3 bucket? What about a Load Manager? You have to use Lambda requests right?</p><p>Quick note: <strong>this guide is NOT meant for production or enterprise-level solutions and should only be used for basic personal projects</strong>. If you are interested in a more production-ready deployment guide, feel free to drop a comment, so I (or other Medium readers) can point you in the right direction. In this article, I’m really focusing on a lean application deployment for personal use. We’ll be using an AWS EC2 instance, which is basically a virtual server. It’s part of the free-tier offered by AWS and thus deploying your project through the steps mentioned below will be absolutely free.</p><p>With those details out of the way, let&#39;s talk about how we’re going to deploy our basic application on a vanilla EC2 instance in 5 simple steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUgEWHJTU6dLzcXXZmAZcA.png" /><figcaption>Our 5 steps to ML web app deployment</figcaption></figure><p>At a very high level, our deployment steps can be seen in the chart above. We’ll start out by setting up an EC2 instance on which our application will run. This’ll involve utilizing a few guides but I’ll help you through this and mention common errors to save you time and frustration. Next, we’ll connect to our instance and migrate our code to it. This is where I’ll share the application we’ll be working with today, walk you through our Flask directory setup and API. After that, we’ll install project dependencies, followed by setting up a local Apache server to run on our EC2 instance which will be used to serve our application on the public port provided by the EC2 server. Let’s get started!</p><h3>EC2 Instance Setup</h3><p>Setting up an EC2 instance on AWS is really not super hard and has been covered by numerous different articles on TDS and beyond, so I will not be wasting your time by going through it again. Instead, I’ll point to the guides which helped me the most and walk through some common errors/questions that show up during setup. Remember, the goal of this step is to:</p><ol><li>Create an AWS account (if you haven’t already)</li><li>Start up an EC2 server using the tutorials and guides I’ll point to below</li><li>Ensure your instance settings are correct</li></ol><p>Here’s the great official guide to help you setup your instance: <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance">AWS official EC2 instance setup guide</a>. You’ll want to make a few key changes to the setup as follows.</p><p>In <em>Step 1: Choose an Amazon Machine Image (AMI)</em>, pick the <strong>Ubuntu Server 16.04 LTS (HVM).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nyhrLFR61_-VhFUI7K1SA.png" /><figcaption>AMI used in this article</figcaption></figure><p>Stick to the free tier option (t2.micro) because it’s got enough computing power for most small-scale personal projects. Instead of choosing the Review and Launch button, click through the different configuration options and stop at <em>Step 6: Configure Security Group. </em>Here, add a new security group with the following settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R94WyWgLak5AvJhVeYpQAQ.png" /><figcaption>The first security group should already exist, add the second with the same settings: HTTP; TCP; 80; Anywhere; default; description (optional)</figcaption></figure><p>This will allow users from anywhere in the world to be able to access a port on your running EC2 instance using HTTP, which is perfect for our web application. Now when you finish this step, review and click the launch button you’ll see the following dialog box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*8ygHPz7Foxek-_sWS2lv6A.png" /><figcaption>Key-pair screen</figcaption></figure><p>Make sure to select “Create a new key pair” (if you don’t already have one), and save this .pem file in a secure location. You’ll need this later on. Congratulations, your EC2 instance is now up and running! It’s time to connect to it and migrate your application.</p><h3>Connect to EC2</h3><p>Now that our instance is up and running, we can connect to it. The connection process will look different based on your OS, so I can’t be of much help for Mac/Linux users, but here’s a <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html">great page</a> with resources for you folks. I’ll be walking primarily through the Windows setup process, and again I’ll point you towards some good resources while mentioning hurdles and extra steps. I highly suggest using PuTTY as your SSH client because of its ease of use and simplicity. The best guide available to walk through the connection sequence is again the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html">AWS official one</a>, so feel free to work through that.</p><p>Once you&#39;ve completed generating your .ppk key from your .pem key and are ready to connect to your instance, the following details will help:</p><pre><strong>Hostname</strong>: ubuntu@<em>your_public_dns_name</em></pre><p>Where your public DNS name can be found on your AWS EC2 homepage by clicking “running instances”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GAQBT86uX1L7Fk9u4Z0EMw.png" /><figcaption>AWS EC2 Running Instances page: use to locate your public DNS</figcaption></figure><p>The rest of the steps covered by the guide for connecting to your EC2 instance are fairly straight-forward, but make sure to save your settings in the PuTTY page so you don’t have to set this up every time. Note: your connection type is SSH and ensure that your .ppk file is connected under Connection &gt; SSH &gt; Auth before saving your settings.</p><p>If everything went smoothly, after this step you’ll be able to access your instance using the PuTTY client. This will be a dark DOS-looking screen from the ’70s and that’s exactly what we’re looking for!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_6E2ljzY2DCiH2Ax4nJbiw.png" /><figcaption>Perfect, you’re inside your EC2 instance!</figcaption></figure><h3>Migrate Application Code to EC2</h3><p>Like I mentioned earlier in the article, we’re going to be deploying <a href="https://towardsdatascience.com/web-app-development-for-data-scientists-48e445e54d7b">a machine learning web application</a> that I built over the past 2 weeks. Before we get into code migration, a little knowledge about the model, application and project structure is necessary. The model is a basic sentiment analysis classifier with incremental learning capabilities. We created a single page application on top of this to help a user interact with our model. We have two Flask API routes which allow the user to enter a review/thought and receive a sentiment analysis, and provide feedback to the model allowing for ongoing learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/0*6ScZZUqANmpGoUJb.png" /><figcaption>Project structure</figcaption></figure><p>The project structure can be seen in the image on the right. All our assets (images, models, etc.) are in different sub-folders inside the same main folder static , our HTML page is inside the templates folder and our app.py file is in the python folder.</p><p>We’ll be using the SCP command to help us transfer our files from our local computer (windows) to our EC2 instance. Before doing that, go ahead and create a directory (I called mine web_app, I would suggest using the same name to avoid any confusion) inside /home/ubuntu where your application will exist using the following command-</p><pre>mkdir web_app</pre><p>So your desired deployment location on your instance is /home/ubuntu/web_app. Now, you’ll have to download and unzip the application mentioned above into your local machine. You can find the project code in this <a href="https://github.com/hr23232323/nlp_medium_p2">GitHub repo</a>. Once you have the project files in your local machine, you can use the following SCP command (from the windows terminal) to move your files from your local machine to your AWS EC2 instance:</p><pre>scp -i <strong>&lt;path/to/pem/key&gt;</strong> -r <strong>&lt;local/path/to/project/folder&gt;</strong> <a href="/service/mailto:ubuntu@ec2-54-235-0-36.compute-1.amazonaws.com">ubuntu@</a><strong>&lt;instance/public/DNS&gt;</strong>:/home/ubuntu/web_app</pre><p>Note: if you encounter an error with the following message “Permissions for ‘private-key’ are too open”, check out this <a href="https://superuser.com/questions/1296024/windows-ssh-permissions-for-private-key-are-too-open">awesome article</a> about getting past that.</p><p>Great! At this point, your project files should now all be in your EC2 instance. Ensure that the folder structure/paths are correct (ex: /home/ubuntu/web_app/python/app.py etc.) and now we can move on to the next step.</p><h3>Dependency Installation</h3><p>Now, this step can be the most confusing for people because python environments, modules and other installations can quickly get complicated and cross-dependent. We’ll keep it simple by breaking this part into two main steps:</p><ol><li>Install pip and other python dependencies</li><li>Install Apache and mod_wsgi on our EC2 instance</li></ol><p>Let&#39;s start by ensuring we’re running Python 3 and installing the correct version of Pip. Your EC2 instance should come pre-installed with python3. To check, use the following command:</p><pre>python3 -V</pre><p>You should see a version greater than 3.5. If that’s the case you can move on to the next step. If not, look through this <a href="https://docs.aws.amazon.com/cli/latest/userguide/install-linux-python.html">official guide</a>. Once you have python3 installed, you’ll need to download and install the correct pip version. To do so, use the following commands (in order):</p><pre>curl -O <a href="https://bootstrap.pypa.io/get-pip.py">https://bootstrap.pypa.io/get-pip.py</a> <br>sudo python3 get-pip.py</pre><p>Now, you should have both python3 and pip installed. Before moving on to the next step run the following command to ensure everything is up-to-date:</p><pre>sudo apt-get update</pre><p>We’ll finish this step off by installing our web app dependencies (Flask and Scikit-learn) using the following commands:</p><pre>sudo pip install Flask<br>sudo pip install scikit-learn</pre><p>Now we can move to the next step: <strong>installing Apache and mod_wsgi</strong>.</p><p>To set up Apache you need two modules: Apache2 and mod_wsgi. Apache2 is our simple web server and mod_wsgi is an Apache module used for Python based web applications. To install the correct versions for python3, use the following commands:</p><pre>sudo apt-get install apache2<br>sudo apt-get install libapache2-mod-wsgi-py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*MJ5Y6cMIlx2-X9aeYQBwaQ.png" /><figcaption>Apache default page. Congrats, your server is up and running!</figcaption></figure><p>At this point, the Apache server should be serving a default page on your AWS EC2 public DNS. To verify this, note the public DNS (ec2-xx-xxx-xx-xxx.compute-x.amazonaws.com/) from your AWS console and visit it in the browser. You should see a screen which looks like the one on the left.</p><p>If you can see the Apache default screen, rejoice because you’re almost there! If not, feel free to use the comment section so myself and others can help you. You can also look through the Apache log file (/var/log/apache2/error.log) to try and debug yourself.</p><p>You’ve set up python3 and pip, installed Apache and it’s required modules and are already serving up a page. Now, all that’s left is to setup Apache and point it towards your project folder and give it permission to access those files.</p><h3>Apache Setup and deployment</h3><p>To get our application running on the server, we need to set some Apache configurations. This involves two steps: set up server configurations/permissions and set up mod_wsgi configuration.</p><p>To set up your server configuration, you need to edit the following file: /etc/apache2/sites-enabled/000-default.conf. This is a restricted file and thus you’ll have to use sudo to edit it. Use the following command:</p><pre>sudo vi /etc/apache2/sites-enabled/000-default.conf</pre><p>Add the following code block right after the line ServerAdmin webmaster@localhost:</p><pre>DocumentRoot /home/ubuntu/web_app</pre><pre>WSGIDaemonProcess flaskapp threads=5 python-home=/usr/local/lib/python3.5/site-packages/ user=ubuntu<br>        WSGIScriptAlias / /home/ubuntu/web_app/static/flaskapp.wsgi</pre><pre>&lt;Directory /home/ubuntu/web_app&gt;<br>            WSGIProcessGroup flaskapp<br>            WSGIApplicationGroup %{GLOBAL}<br>            Require all granted<br>        &lt;/Directory&gt;</pre><pre>&lt;Directory python&gt;<br>            Require all granted<br>        &lt;/Directory&gt;</pre><pre>&lt;Directory static&gt;<br>            Require all granted<br>        &lt;/Directory&gt;</pre><pre>&lt;Directory static/models&gt;<br>            Require all granted<br>        &lt;/Directory&gt;</pre><p>These configuration settings should point Apache towards the location of our web application and give it all the required permissions to interact with our files and models.</p><p>The last step in our Apache setup is configuring mod_wsgi to support our Flask-based application. To do so, you have to create a file named flaskapp.wsgi in the <strong>static folder</strong>. Put the following code in that file:</p><pre>import sys<br>import site<br>site.addsitedir(‘/home/ubuntu/.local/lib/python3.5/site-packages’)<br>sys.path.insert(0, ‘/home/ubuntu/web_app/python’)<br>from app import app as application</pre><p>These are configurations needed by Apache to locate your python dependencies and your project folder(s). Once you’ve made/modified these files and put in the appropriate code, your application is ready to run. Restart your server using the following command to see the changes take effect:</p><pre>sudo apachectl restart</pre><p>Hopefully, at this point, your web application is successfully running on your EC2 public DNS and is ready for the world to interact with! If you see a 403 or 500 error code, try to look through the Apache log file (/var/log/apache2/error.log) for clues about what went wrong. As always, you also use the comment section to share errors and I’ll do my best to help. If everything went smoothly, you should see our app when you visit your public DNS in a browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHL_eV9Y0WPAe_kNrz4gFQ.gif" /></figure><p>You can check out a live demo <a href="http://ec2-54-145-145-48.compute-1.amazonaws.com/"><strong>here</strong></a>.</p><h3>Closing Thoughts</h3><p>I think the internet should be full of real world projects built by people to openly communicate ideas they’re most excited about. Learning how to <a href="https://towardsdatascience.com/building-and-deploying-a-data-science-project-in-two-weeks-3c63f0acdab1">build a model from scratch</a>, <a href="https://towardsdatascience.com/web-app-development-for-data-scientists-48e445e54d7b">designing and developing a web application </a>around it and finally deploying it online for the world to see is a great step towards that goal. I hope some of you get inspired to build and share something of your own and if you do, don’t forget to pay it forward by inspiring others to do the same.</p><p>I hope you enjoyed reading this article just as much as I enjoyed putting it together. If you have any questions, feedback or comments, feel free to get in touch with me through <a href="https://www.linkedin.com/in/harshrana1997/">LinkedIn</a>, or my <a href="http://www.harshrana.com/">website</a>. Additionally, if you did end up building and deploying your own application, please share it with me because I’d love to see it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8eb6a389cdd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-web-application-deployment-in-5-steps-d8eb6a389cdd">Machine Learning Web Application deployment in 5 steps</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web App development for Data Scientists]]></title>
            <link>https://medium.com/data-science/web-app-development-for-data-scientists-48e445e54d7b?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/48e445e54d7b</guid>
            <category><![CDATA[data-science-sprints]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Tue, 17 Dec 2019 15:13:48 GMT</pubDate>
            <atom:updated>2019-12-17T15:13:48.779Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="http://towardsdatascience.com/tagged/data-science-sprints/">Data Science Sprints</a></h4><h4>Building a full-stack Data Science Project in Two weeks to learn Natural Language Processing, Flask and ML Model Deployment!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KmXs_TQHJ5m4x55iDfwcew.gif" /><figcaption>The end product that we’ll be building in this article</figcaption></figure><p>I know some of you are probably thinking: <em>why should a data scientist care about building a web application? </em>The answer to your question is fairly simple: while analyzing datasets, finding patterns and building models are all important skills for a data scientist, it is equally important to communicate ideas and build interactive layers for your models. This can be especially important if you’re thinking about building a consumer facing product or even starting your own business.</p><blockquote>The only person more valuable than a good data scientist, is a good data scientist who also understands engineering and deployment.</blockquote><p>My goal with this project was to build a basic sentiment analysis machine learning model+web application and deploy it in under two weeks to learn three new technologies- natural language processing (NLP), flask and incremental/ongoing learning. In this article, I’ll cover building a flask web application to help users interact with a sentiment analysis model. You can use your own model, or feel free to look utilize this <a href="https://github.com/hr23232323/nlp_medium_p1">repository</a> for one that I built previously. <strong>All my code for this post can be found on my </strong><a href="https://github.com/hr23232323/nlp_medium_p2"><strong>Github</strong></a><strong> so feel free to follow along</strong>.</p><h3>Web Applications refresher in 2 minutes!</h3><p>Web applications can be as simple or as complex as you want them to be. While the architecture of complex applications can confuse even the best of developers, a simple web application isn’t very complicated. In a nutshell, it includes the following pieces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABeqEW-wCzO7Gbbx9a5FNw.png" /><figcaption>Basic Web application layout</figcaption></figure><ol><li><strong>HTML pages</strong>: these are the different webpages that your users can interact with. Many simple applications utilize a single webpage, hence the name single page application (SPA).</li><li><strong>Cascading Style Sheets (CSS)</strong>: CSS is a style sheet language used to alter the appearance of an HTML page. Everything from text color to background is described in CSS.</li><li><strong>JavaScript</strong>: this is an object-oriented programming language which has taken over the web development community. Javascript is primarily used for introducing complex features on a webpage, such as interactivity, request handling etc.</li><li><strong>Web Server</strong>: this is the most confusing part of a web application. A web server can consist of hardware and/or software and its job is to “serve” resources to a user. Thus, when a user wants to see a webpage, the browser requests a resource from the web server over HTTP and receives a response. For our project, we’ll be using a flask server for development.</li></ol><p>Now that you know the different parts of a web application, let me introduce you to Flask which will be the web application framework we’ll be using.</p><h3>Flask for dummies</h3><p>Flask is a lightweight and extremely versatile web application framework. It is built with flexibility and speed in mind, which makes it perfect for smaller applications with little-to-no backend needs. What makes Flask an especially useful tool in the data scientist’s toolkit is that it’s built in Python; which many data scientists are already comfortable with. So what does Flask do?</p><p>Flask takes care of a lot of the environment and project setup involved in a web application. What this essentially means is that you, the developer, can focus on your application while Flask takes care of HTTP, routing, assets etc. I promise all this will become easier to understand as we progress along this article.</p><p>Our Flask application will consist of the following pieces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_jHgN5PGqOPdNV0HlGia-Q.png" /><figcaption>Flask app layout; notice how the components mentioned in the basic web app section fall into place here</figcaption></figure><p>Based on the diagram above, you can clearly see the different functional pieces of our application. The app.py file will serve as the main server logic and help serve files and run any ML logic as needed. The static folder will contain any assets required for our app, from images to our pickeled models. Lastly, the template folder will contain the single HTML web page used in our app.</p><p>Now that you know what makes up a web application and why we’re using flask, let’s talk about how we’ll achieve all this.</p><h3>Development Life Cycle</h3><p>Throughout this project, I’ve tried to follow the following mindset:</p><blockquote>I wanted to do all this in a rapid building/learning mode with a focus on functionality, not on aesthetic.</blockquote><p>Given this, I want to share the application development life cycle that I followed. This will help you understand how to value functionality over everything else and build rapid prototypes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hPSM8e6NnB6PKN77q98qNQ.png" /><figcaption>Development Life Cycle used for this project</figcaption></figure><p>As you can see the main functionality will be built iteratively. This’ll help us always have a working application in place, while we keep adding features. We’ll outline the features needed in our application in the requirement gathering phase. Then, we’ll design the UI/UX for the entire application to avoid worrying about aesthetics later in the development period. After that, we’ll prioritize features and build them in 24 hour sprints. Finally, I’ll be deploying this application to an AWS EC2 instance, which will be covered entirely in another article. Let’s begin!</p><h3>Requirement Gathering</h3><p>Our project involves the user interacting with a machine learning model, so we have to really understand how these interactions will occur. We have an <a href="https://github.com/hr23232323/nlp_medium_p1/blob/master/nlp_model.ipynb">NLP classifier</a> that can predict whether a user is feeling positively or negatively about a subject. So, we need an <strong>input field where the user can enter a review/thoughts</strong> and an <strong>output location where the prediction will be displayed</strong>. Additionally, we also built a <a href="https://github.com/hr23232323/nlp_medium_p1/blob/master/incremental_learning.ipynb">functionality</a> to incrementally train our model. So, <strong>we need to provide the user with a way to provide feedback (correct/incorrect prediction)</strong>. Those are all the interactions in our basic application.</p><p>Given the simplicity of our application, it makes sense to built this as a single page application. So, another requirement for us it that <strong>the input, output and feedback functionalities need to all be in the same page</strong>. This will help keep our project lean and minimal and reduce development time.</p><h3>UI design</h3><p>I like drawing UI designs on a piece of paper before writing any code because we can make changes faster, revamp entire sections and have a concrete design in place for rapid building. Additionally, I also believe that starting out with all different functionalities on different screens, helps putting everything together into a single page application (SPA) without missing any pieces. My UI/UX design progression can be seen here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjAw7GZWqgECQLJvwsKOcw.png" /><figcaption>Design #1</figcaption></figure><p>I started out with three different screens (design #1) each with it’s own functionality. The first screen would be where a user can enter an input and click predict. The second would be where the user sees the output and has the option to mark it as a correct or incorrect prediction. Finally, the last screen would be a confirmation screen so the user knows his feedback was taken and an option to start again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzlCa4djN89vAGVvlA3dtQ.png" /><figcaption>Design #2</figcaption></figure><p>These three screens molded together for my first SPA design (design #2). This was a static design with placeholders where the individual pieces (output, feedback buttons etc.) would become active when needed. I considered it active because none of the elements would hide at any point; they were all always visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X57FI9Q6WDUi9YBkRFfpGA.png" /><figcaption>Design #3 (Final)</figcaption></figure><p>My final design was a dynamic version of the static SPA (design #3), where output and feedback elements would initially be hidden (left side screen) and then appear when the user clicked the prediction button (right side screen). I was pretty happy with this design and comfortable moving forward at this point with the actual development.</p><h3>Iterative Development</h3><p>Before starting development, I created a priority queue of my features needed to finish this project. They can be seen (in order) below:</p><ol><li>Build out the static user interface for the entire application (vanilla HTML+CSS)</li><li>Capture input from user, run it through our model and come up with a prediction, which is displayed in the frontend</li><li>Capture user feedback and appropriately teach the model using the incremental learning functionality built before</li><li>Change the UI from static to dynamic: hide unnecessary elements on startup and show them when required</li></ol><p>At the end of each iteration, I would have a working application with limited functionality. This would help test and debug new features rapidly. Let me share my code and progress through each of these features.</p><h4>Iteration 1: Static User interface</h4><p>Building out a static user interface was fairly simple. If you’ve never developed for the web before, I would highly recommend checking out <a href="https://developer.mozilla.org/en-US/">MDN Web Docs</a> for any and all questions you have. Given the simplicity of our application, our HTML code was minimal and while I do not want to waste your time describing this in depth, I’ll cover it from a high-level. Our UI was broken up into two main &lt;div&gt; tags: one for the top half which contained the instructions, input text-box and prediction button, and the bottom half which contained the prediction output and feedback buttons. Most of the HTML associated with this project should make sense to anyone who’s done web development before, but I do want to point out one important flask specific code:</p><pre>&lt;form id=&quot;input-form&quot; class=&quot;input-form&quot; <br>   action=&quot;<strong>{{ url_for(&#39;predict&#39;)}}</strong>&quot; method=&quot;post&quot;&gt;<br>     ..<br>     ..<br>     ..<br>&lt;/form&gt;</pre><p>The {{url_for(‘predict’)}} attribute in the form tag is a flask method which points to the “predict” route. This association helps us capture user input from the form and process it inside our app.py file where our model is available. You’ll see this in action when we cover flask in the next section.</p><p>Lastly, To style this HTML page, I primarily used CSS’s flexbox display mode to create a minimal layout. So even though none of the buttons worked yet, and the image was unchanging, my static layout at the end of this iteration can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2fddqk0AYZ4U6TE1Ty8jA.png" /><figcaption>Static UI after iteration 1. None of the buttons work yet :)</figcaption></figure><p>I won’t be covering CSS in this article given the audience’s interest, but if you’re interested you can find all my code <a href="https://github.com/hr23232323/nlp_medium_p2"><strong>here</strong></a>.</p><h4>Iteration 2: Capture input and show prediction on Frontend</h4><p>At the end of the previous section, we only had an HTML and CSS file. Now, we’ll start using the app.py file to build our application’s logic and also move from directly displaying the HTML page to serving it using the Flask server. To start serving the HTML page using Flask, we only need a few lines of code:</p><pre>from flask import Flask, render_template, request</pre><pre>app = Flask(__name__, template_folder=&#39;../templates&#39;, static_folder=&#39;../static&#39;)</pre><pre>@app.route(&#39;/&#39;)<br>def home():<br>    return render_template(&#39;index.html&#39;, image_filename=&quot;img/happy.webp&quot;, display_mode=&quot;none&quot;)</pre><pre>@app.route(&#39;/predict&#39;, methods=[&#39;POST&#39;])<br>def predict():<br>    return render_template(&#39;index.html&#39;, image_filename=&quot;img/happy.webp&quot;, display_mode=&quot;none&quot;)</pre><pre>@app.route(&#39;/save_pred&#39;, methods=[&#39;POST&#39;])<br>def save_pred():<br>    return render_template(&#39;index.html&#39;, image_filename=&quot;img/happy.webp&quot;, display_mode=&quot;none&quot;)</pre><pre>if __name__ == &quot;__main__&quot;:<br>    app.run(debug=True)</pre><p>Note: Flask doesn’t require the template_folder and static_folder to be explicitly mentioned, but I like doing this in case I mess up names/paths. Also, at this point, our project folder structure should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*GbL0ln64F3rDgFXGZE6vdA.png" /><figcaption>Project folder structure</figcaption></figure><p>This way all our assets (images, models etc.) are in different sub-folders inside the same main folder static , our HTML page is inside the templates folder and our app.py file is in the python folder.</p><p>At this point, we’re ready to run our application using the Flask server. To do so: navigate to the python folder using your command line and run the command python app.py . Remember that you should’ve already install Flask using pip at this point. More information about that <a href="https://flask.palletsprojects.com/en/1.1.x/installation/">here</a>. Congrats, your application is up and running! Now lets add the prediction logic.</p><p>Notice how we’ve put placeholders for both the predict and the save_pred routes. This is done to ensure that the Flask app deploys without errors. To make our predict function work, we need to capture the user’s input from the textbox, convert it into a bag-of-words vector using our NLP vectorizer (input_transformer.pkl) and run that through our classifier (review_sentiment.pkl). Also note that I’ll be using global variables to store the model input and output, to persist them across the prediction and incremental learning functions. The code for this functionality can be seen below:</p><pre># module needed to load models<br>from sklearn.externals import joblib</pre><pre># load models pickeled earlier<br>input_transformer = joblib.load(open(&#39;../static/models/input_transformer.pkl&#39;, &#39;rb&#39;))<br>model = joblib.load(open(&#39;../static/models/review_sentiment.pkl&#39;, &#39;rb&#39;))</pre><pre># Global variables for persistence across methods (and requests)<br>model_input=&quot;&quot;<br>model_output=&quot;&quot;</pre><pre>@app.route(&#39;/predict&#39;, methods=[&#39;POST&#39;])<br>def predict():</pre><pre>    # retrieve global variables to store input and output<br>    global model_input<br>    global model_output<br>    <br>    # get text from the incoming request (submitted on predict button click)<br>    text = request.form[&#39;input_text&#39;]<br>    <br>    # convert text to model input vector<br>    final_features = input_transformer.transform([text])<br>    <br>    # use classifier&#39;s predict method to get prediction<br>    prediction = model.predict(final_features)<br>    <br>    # store model input and output<br>    model_input = text<br>    model_output = prediction[0]</pre><pre>    return model_output</pre><p>There you have it! Now our app can convert user-inputted text into a vector which can be classified by our pre-trained model.</p><h4>Iteration 3: Incremental Learning</h4><p>The steps we take to make this happen are the following: vectorize input text using the global variable, get the button user clicked on (correct/incorrect), assert “correct” prediction based on the user’s button choice, incrementally train model till it relearns, update model pickle. The code for this can be seen below:</p><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/save_pred&#39;, methods=[&#39;POST&#39;])<br>def save_pred():<br>    # retrieve global variables<br>    global model_input<br>    global model_output</pre><pre>    # vectorize user input<br>    final_features = input_transformer.transform([model_input])<br>    <br>    # get user&#39;s button choice (correct/incorrect)<br>    save_type = request.form[&quot;save_type&quot;]<br>    <br>    # modify global variable if user selected &quot;incorrect&quot;<br>    if(save_type == &#39;incorrect&#39;):<br>        if(model_output == &#39;p&#39;):<br>            model_output = &#39;n&#39;<br>        elif(model_output == &#39;n&#39;):<br>            model_output = &#39;p&#39;<br>        else:<br>            print(&quot;Error: Model output was neither N nor P&quot;)</pre><pre>    # Strengthen weight of particular connection<br>    max_iter = 100<br>    counter = 0<br>    for i in range (0,max_iter):<br>        model.partial_fit(final_features, [model_output])<br>        if(model.predict(final_features) == [model_output]):<br>            counter = i<br>            break<br>    <br>    # Save trained model pickle<br>    joblib.dump(model, &#39;../../review_sentiment.pkl&#39;)<br>       </pre><pre>    # return placeholder (will be updated later)<br>    return &quot;&quot;</pre><p>While the code above achieves all the functionality that we need, it does not provide the user with any sort of feedback once the retraining is complete. To address this, we’ll add a return text variable to either return “<em>the weights were changed, thank you for correcting me!</em>” or “<em>the weights were strengthened, thank you for teaching me!</em>” based on the situation. Additionally, I also believe that it is very important to capture the data that the model is being retrained on to prevent misuse of this functionality. A few lines of code can help us store the input, the corrected output and the number of cycles it took to retrain in a csv file:</p><pre>fields = [model_input, model_output, counter]<br>    with open(&#39;user_teaching_data.csv&#39;, &#39;a&#39;) as file:<br>        writer = csv.writer(file)<br>        writer.writerow(fields)</pre><p><strong>Iteration 4: Dynamic Content</strong></p><p>With our static app complete, this iteration was all about making it dynamic and more user-friendly. You must’ve noticed that all the elements were always visible in our app and the page was reloading each time. While this isn’t a big deal, we can easily use jQuery and AJAX to avoid reloading and also hide elements as needed. This will help give our application a polished look and also transform to positive user experience. Going in-depth into either jQuery or Ajax is beyond the scope of this article, but if you’re interested I highly suggest checking out my <a href="https://github.com/hr23232323/nlp_medium_p2/blob/master/static/scripts/script.js"><strong>code</strong></a>, and this <a href="https://medium.com/free-code-camp/ajax-basics-explained-by-working-at-a-fast-food-restaurant-88d95f5fcb7a">awesome article about AJAX</a>.</p><p>At the end of this iteration, our app was working as expected: it accepted user information, displayed the results from our NLP models, allowed the user to retrain our model and did all this in a minimal and elegant manner. Our fully functioning web app demo can be seen again below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KmXs_TQHJ5m4x55iDfwcew.gif" /><figcaption>Final working demo for web application</figcaption></figure><p>Congrats on getting this far into our two week challenge! Hopefully, by now you’re not only capable of building an NLP model, but also have the necessary skills to build web applications to make those models interactive.</p><h3>Wrap up</h3><p>So far we’ve achieved a lot of cool things: we took our previously trained data science model and built a fully functional web application around it. We spent around six days building our web application by focusing on functionality over aesthetics. This mindset allowed us to build our model and web application all within ~10 days.</p><p>With four more days left in our two week period, <strong>we’ll tackle our biggest challenge yet: model and application deployment on AWS using a vanilla EC2 instance</strong>. I’ll be writing another post to guide you through the deployment process, so feel free to follow <a href="https://towardsdatascience.com/tagged/data-science-sprints">my column on Towards Data Science</a> to get a notification when it’s published!</p><p>I hope you enjoyed reading this article just as much as I enjoyed putting it together. If you have any questions, feedback or comments, feel free to get in touch with me through <a href="https://www.linkedin.com/in/harshrana1997/">LinkedIn</a>, or my <a href="http://www.harshrana.com/">website</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48e445e54d7b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/web-app-development-for-data-scientists-48e445e54d7b">Web App development for Data Scientists</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building and Deploying a Data Science Project in Two weeks]]></title>
            <link>https://medium.com/data-science/building-and-deploying-a-data-science-project-in-two-weeks-3c63f0acdab1?source=rss-be594c58dc55------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c63f0acdab1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science-sprints]]></category>
            <dc:creator><![CDATA[Harsh Rana]]></dc:creator>
            <pubDate>Mon, 09 Dec 2019 04:15:00 GMT</pubDate>
            <atom:updated>2019-12-16T22:07:52.216Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="http://towardsdatascience.com/tagged/data-science-sprints/">Data Science Sprints</a></h4><h4><em>Learn the basics of Natural Language Processing, Flask and ML Model Deployment by building something fun!</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kP3vRc3FqS_v2gXDcYT8xg.png" /><figcaption>End product: a sentiment analysis web application with built-in ongoing learning capacity</figcaption></figure><p>I’m a kinesthetic learner; I learn by doing, building and breaking things. In this article, I’ll share how I built a basic sentiment analysis machine learning model+web application and deployed it in under two weeks to learn three new technologies- natural language processing (NLP), flask and incremental/ongoing learning.</p><h3>Learning Outcomes</h3><p>My primary goals with this undertaking were the following:</p><ol><li>Learn NLP basics and train a sentiment analysis classifier</li><li>Learn how to build a web application using flask</li><li>Learn ML-based web application deployment</li></ol><p>I wanted to do all this in a rapid building/learning mode with a focus on functionality, not on aesthetic. Some of the other topics I ended up exploring and learning along the way include model optimization, incremental/ongoing learning and jQuery. <strong>All Jupyter notebooks and code used in this article can be found on my </strong><a href="https://github.com/hr23232323/nlp_medium_p1"><strong>Github</strong></a><strong>.</strong></p><h3>The Dataset</h3><p>For this project, I needed a dataset which contained text data, but was already labelled and structured. This would enable me to focus on NLP-based preprocessing while skimming over other forms of data preparation. After looking at a few different options, I ended up choosing the <a href="https://www.yelp.com/dataset">yelp reviews dataset</a> because it was pre-labelled (text review and 1–5 star ratings), but the actual text inside the reviews was messy and inconsistent. You can see an example record from the raw JSON dataset below-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JV-512Rli_X2X-VpVKiPBw.png" /><figcaption>Example record from raw JSON data</figcaption></figure><p>For my model, I was only interested in the text review and the 1–5 stars (maybe in the future I could try doing something fun with the other attributes). Additionally, to further simplify my training process and stick to the rapid development mindset, I decided to modify the labels from 1–5 ratings to a positive (4 or 5 stars) or negative (1 or 2 stars) sentiment associated with the textual review. I dropped the reviews with a 3 star rating because their classification would be difficult to validate. A <a href="https://github.com/hr23232323/nlp_medium_p1/blob/master/data_cleaning.ipynb">few lines of code</a> later, I had my data inside a pandas dataframe which can be seen below-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYOjtzj9Jo5P4eA6_Z52rQ.png" /><figcaption>Pandas dataframe with just the text review and positive/negative sentiment</figcaption></figure><p>There are many potential issues with the text. Some characters such as <em>*</em> and <em>\n</em> serve no real purpose in helping understand the text, while others such as <em>$</em> and<em> !</em>, may bring insight into how the reviewer was feeling. Additionally, there are many words/slangs which a human could understand ($8Gs = $8000), but would make little to no sense for a machine. All these and many more scenarios will be tackled in the next section.</p><h3>Natural Language Processing Pipeline</h3><p>To prepare the data for classification, we have to first design a NLP pipeline to prepare the text data. There are several steps involved in an NLP pipeline, some of which can be seen below-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4_f6B5QhqSqanOx7Y8YcQ.png" /><figcaption>Sample NLP Pipeline</figcaption></figure><p>In our case, steps such as part-of-speech tagging and named entity recognition would not be very beneficial, as the task at hand is a sample classification problem. So we’ll focus on techniques which offer the most bang for our buck: tokenization, stop words removal and ngram utilization (discussed below).</p><p>For the first iteration of my NLP model, I decided to implement a simple bag-of-words model using sklearn’s CountVectorizer() method and train it on roughly 100,000 rows. I utilized sklearn’s train_test_split() method to break up my data into training and testing blocks. After trying a few different classifiers such as logistic regression, multinomial naive bayes and linear support vector machine, I found that the logistic regression model worked best for this data. Here’s the confusion matrix of the respective classifiers-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r32uNO0RTLKIq-zujnFa9g.png" /><figcaption>Confusion Matrices for the various classifiers</figcaption></figure><p>Thinking further ahead, I found that the logistic regression model would be unable to utilize incremental learning later in the project, while the linear SVM (powered by sklearn’s SGDClassifier()) would. The accuracy trade-off was small with the logistic regression having an average F1-score of 93%, while the SVM having an average F1-score of 92%. Thus, I decided to proceed with the linear SVM to utilize incremental learning later in the project.</p><p>Additionally, I chose to replace CountVectorizer() with the HashingVectorizer() as it uses a technique called hashing to decrease the size of the input processing model by &gt; 99%. The only major trade-off is the loss of ability to look at explicit words in the model (hashing converts all words to numbers), which wasn’t a big concern for me. You can read more about this model <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html">here</a>.</p><h3>Model Optimization</h3><p>I did a few things to optimize my model. As you can see from the confusion matrix above, my training and testing data both have a lower number of negative reviews, as compared to positive. This resulted in my F1-score for the negative label being 82% and for the positive label being 94%. To mitigate this disparity, I had to balance my dataset by making sure that the number of records corresponding to the two labels was comparable. After balancing my dataset, I proceeded to omit stop words like <em>and, the, a </em>etc. from the bag-of-words model.</p><p>After these two steps my model’s F1-score dropped from 92% to 91%, but the previous disparity between the two labels was smaller. The F1-score of my negative and positive reviews were 91% and 90% respectively, so overall the classifier was performing better. Next, I decided to utilize <strong>ngrams</strong> to further optimize my model.</p><p>Ngrams simply refers to a continuous group of n words which co-occur. For example-</p><p>In the sentence &quot;I love data science&quot;, the 1-grams would be [&quot;I&quot;, &quot;love&quot;, &quot;data&quot;, &quot;science&quot;]. The 2-grams would be [&quot;I love&quot;, &quot;love data&quot;, &quot;data science&quot;] and so on.</p><p>You can even see from the example above why utilizing ngrams would be helpful. The 2-gram “data science” appearing in a sentence would provide more information than just the 1-grams “data” and “science”. Now, back to the optimization: I utilized 1,2 grams for further strengthening my model. The confusion matrices before and after these optimizations can be seen below-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HoQQpB985CcOZ0jnYBDQqw.png" /><figcaption>Before and after the ngram optimization</figcaption></figure><p>As you can see, this was a major improvement as both positive and negative incorrect guesses dropped. The model’s final F1-score moved up to 92%, and the disparity between the two classes was negligible. At this point, the model was ready. Next up, incremental/ongoing learning.</p><h3>Incremental Learning</h3><p>As you all have already seen, our sentiment classification model is far from perfect. This is where the beauty of incremental learning comes in.</p><blockquote>Incremental learning is a machine learning paradigm where the learning process takes place whenever new example(s) emerge and adjusts what has been learned according to the new example(s).¹</blockquote><p>If you think about how human beings learn language, you’ll quickly see that they’re learning incrementally. Every time a parent corrects a child, a teacher corrects a student or two friends correct each other, a form of incremental learning takes place. In this section, we’ll build a similar system for our NLP model so users can “correct” it when it makes a classification error.</p><p>Remember how we chose the SGDClassifier() due to this functionality earlier? The SGDClassifier has a built-in partial_fit() method which will help us implement incremental learning in a few lines of code as follows-</p><pre># Incremental training</pre><pre># example x<br>X_instance = cv.transform([&quot;I like this place, but not much&quot;])  </pre><pre># user determined label<br>y_instance = [&#39;n&#39;]</pre><pre># max iterations of training until the classifier relearns <br>max_iter = 100</pre><pre># loop for a maximum of max_iter times<br># partially fit the classifier over the learning instance<br># stop when classifier relearns (or max_iter is reached)<br>for i in range (0,max_iter):<br>    clf.partial_fit(X_instance, y_instance)<br>    if(clf.predict(X_instance) == y_instance):<br>        break</pre><p>Using this, we can train our model on new data whenever a mistake is made while still maintaining the initial training. An example can be seen below-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*85Tjo8ffcs-F4YLUl2i_aQ.png" /><figcaption>Example of incremental learning</figcaption></figure><h3>Wrap up and part 2</h3><p>So far we’ve achieved quite a bit: we’ve created a basic NLP classifier which can read a piece of text and classify the sentiment into either positive or negative. We’ve optimized our model using NLP techniques such as stop words and ngrams to achieve a baseline F1-score of 92%. And lastly, we’ve written code to incrementally train our classifier, whenever it makes a mistake, so the learning never stops!</p><p>In the next part, we’ll build on top of everything we’ve covered so far and build a basic web application using Flask. In particular, we’ll build APIs to classify user-entered text and incrementally teach/correct our model. This will enable us to interact with our model as a user and further learn about machine learning model deployment!</p><p>I hope you enjoyed reading this article just as much as I enjoyed putting it together. If you have any questions, feedback or comments, feel free to get in touch with me through <a href="https://www.linkedin.com/in/harshrana1997/">LinkedIn</a>, or my <a href="http://www.harshrana.com/">website</a>.</p><p>[1]: Geng X., Smith-Miles K. (2009) Incremental Learning. In: Li S.Z., Jain A. (eds) Encyclopedia of Biometrics. Springer, Boston, MA</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c63f0acdab1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-and-deploying-a-data-science-project-in-two-weeks-3c63f0acdab1">Building and Deploying a Data Science Project in Two weeks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>