<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[tech-that-works - Medium]]></title>
        <description><![CDATA[Tech blog for sharing concepts, ideas, experience and issues faced by the authors while building cool stuff. - Medium]]></description>
        <link>https://medium.com/tech-that-works?source=rss----f3f3b94712c0---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>tech-that-works - Medium</title>
            <link>https://medium.com/tech-that-works?source=rss----f3f3b94712c0---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:37:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/tech-that-works" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Navigating Pune with AI Lens]]></title>
            <link>https://medium.com/tech-that-works/navigating-pune-with-ai-lens-855ce2f60675?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/855ce2f60675</guid>
            <dc:creator><![CDATA[Samrudha Kelkar]]></dc:creator>
            <pubDate>Sat, 07 Oct 2023 09:28:34 GMT</pubDate>
            <atom:updated>2024-02-10T16:20:36.689Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using LLMs to explore Pune’s History</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tMXH2OXnJhuvE2iIPe7evQ.jpeg" /></figure><p>Large Language Models (LLMs) have basically soaked up a ton of information from all the text they’ve been fed. It’s pretty mind-blowing to see how much these AI systems know about our world.</p><p>Recently, I tried out an <a href="https://samrudh.github.io/llm_meets_pune/">experiment</a> to check out how AI views the history of <a href="https://en.wikipedia.org/wiki/Pune">Pune</a>, a city in India with a really long and rich past. Pune’s history spans centuries, and it still proudly showcases its deep-rooted heritage. Different rulers have left their mark on the city over the years, and if you look at the city’s layout, you can actually see how it’s connected to different historical periods and how it’s grown over time.</p><p>I looked at the database of <a href="http://opendata.punecorporation.org/Citizen/CitizenDatasets/Index?categoryId=34">heritage places</a> in Pune generously provided by Government of India and Pune Municipal Corporation through the <a href="https://data.gov.in/">OpenData, a Digital India initiative</a>. Places mentioned in this dataset spans wide range of history from 9th Century to 20th century.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*6XFFvEH6Q7gURGFbn6Gudg.png" /><figcaption>Heritage Places in Pune city</figcaption></figure><p>We decided to take a closer look at each of the sites and tried to find their corresponding Wikipedia pages based on the information provided in the Name column of our dataset. We managed to succeed in 77% of our attempts during this phase, identifying 146 heritage places with Wikipedia pages.</p><p>With that key information in hand, next move was to extract content from these Wikipedia pages and feed it into our LLM AI model. We achieved this using Python and Wikipedia APIs, seamlessly integrating the wealth of knowledge from these pages into our analysis.</p><pre>import wikipedia<br>def get_wiki_summay(x):<br>    summary = &quot;&quot;<br>    try:<br>        summary = wikipedia.summary(x.split(&quot;/wiki/&quot;)[1])<br>    except Exception as e:<br>        pass<br><br>    return summary<br><br><br>heritige_urls_df[&quot;wiki_summary&quot;] = heritige_urls_df[&quot;article_url&quot;].apply(<br>    get_wiki_summay<br>)</pre><p>After obtaining the text, we generated embeddings, which essentially translate the text query into vector representations. AI models, particularly LLMs, boast dense representations for text, encapsulating knowledge in numerical form. In this experiment, we employed two distinct models. The first one leveraged a BERT-based <a href="https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2">sentence encoder</a> from <a href="https://huggingface.co/">HuggingFace</a>, yielding a vector of 384 dimensions. The second model utilized <a href="https://platform.openai.com/overview">OpenAI</a> embeddings through <a href="https://www.langchain.com/">LangChain</a>, yielding embedding vectors with a size of 1586, making them four times as expansive as the sentence encoder’s output.</p><pre>import pandas as pd<br><br>## first model<br>from sentence_transformers import SentenceTransformer<br><br>## second model<br>from langchain.embeddings import OpenAIEmbeddings<br><br><br>def compile_text(x):<br>    text = f&quot;&quot;&quot;Name: {x[&#39;Name&#39;]},  <br>                Address: {x[&#39;Address&#39;]}, <br>                wiki_summary: {x[&#39;wiki_summary&#39;]}<br>            &quot;&quot;&quot;<br><br>    return text<br><br><br>sentences = df.apply(lambda x: compile_text(x), axis=1).tolist()<br><br># Sentence Tranformer model<br>model = SentenceTransformer(r&quot;sentence-transformers/paraphrase-MiniLM-L6-v2&quot;)<br>output = model.encode(<br>    sentences=sentences, show_progress_bar=True, normalize_embeddings=True<br>)<br><br># OpenAI model<br>embeddings = OpenAIEmbeddings(openai_api_key=&quot;your api key&quot;)<br>output = [embeddings.embed_query(x) for x in (sentences)]<br><br><br>df_embedding = pd.DataFrame(output)</pre><p>After obtaining the vectors for each place, we proceeded to classify the data into distinct groups using unsupervised clustering algorithms, with <a href="https://en.wikipedia.org/wiki/K-means%2B%2B">KMeans</a> being our weapon of choice. However, considering the relatively small sample size of just 146 heritage places in this task, we first applied PCA transformation to condense the dimensions. Remarkably, with approximately 30 dimensions for the sentence model and 50 dimensions for the OpenAI model, we managed to capture 70% of the variance, which appeared quite satisfactory for our objectives.</p><p>When it came to determining the cluster size, we faced the classic conundrum of choosing a suitable clustering quality metric. Beyond just achieving the best grouping, we also valued factors like ease of interpretation and performance across both models. Ultimately, we converged on three as the optimal cluster size using the <a href="https://en.wikipedia.org/wiki/Elbow_method_(clustering)">elbow method</a> for the sentence encoder model, while the same size seemed “good enough” for the OpenAI model.</p><p>Now, let’s delve into the exciting part: how AI goes about grouping these fascinating places within the city!</p><p>Below are the clusters formed by sentence encoder model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*pxp9yUfpWyM2kzgIjP2xOw.png" /><figcaption>Heritage Places Clustering Results</figcaption></figure><p>In our analysis, we distinctly identified three clusters that delineate Pune’s historical evolution. The first cluster represents Pune before 1850, a time marked with influence of temples, <em>wada</em>s. The second cluster signifies Pune’s transformation under British rule after 1850, and the third encapsulates modern Pune, characterized by its proliferation of educational and government institutions.</p><p>Cluster two, while rich in ancient temples, also intertwines with the royal heritage of Pune, often invoking associations with terms like “Raj.” Raj Bhavan is thus linked with city’s historical roots. A particularly intriguing heritage site within this cluster is the “<a href="https://map.sahapedia.org/article/Brother-Deshpande%20Memorial%20Church/3403">Brother Deshpande Memorial Church,</a>” named in honor of Ramchandra Pralhad Deshpande. Interestingly, unlike other church locations such as “St. Ornella’s Church” or “St. Matthew’s Tamil Church,” it did not fall into cluster 1. The reason becomes apparent when considering the church’s location in <a href="https://en.wikipedia.org/wiki/Kasba_Peth,_Pune">Kasba Peth</a>, one of the city’s oldest districts and name which sounds Indian. It seems our model was astutely capturing this historical nuance.</p><p>Below, we have the clusters visually represented on a map. In the center, we observe the historical heart of Pune (cluster 2) depicted in orange, with the European influence extending to the eastern side highlighted in blue. On the other hand, the more contemporary heritage sites, such as educational institutions around the Deccan, Law college Road , are marked in green. <a href="https://samrudh.github.io/llm_meets_pune/model1.html">Here</a> you can see the map in detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKNbYzNdwHFqdkbLnKfs7g.png" /></figure><p>One intriguing aspect of the model lies in its capacity to categorize two closely situated locations into distinct clusters, despite their strikingly similar addresses. While the address information was included in the input text, the model’s embeddings effectively differentiated between them based on their respective Wikipedia content. For instance, “<em>Chattushringi Mandir</em>” and “<em>Government Polytechnique College,</em>” both located on opposite sides of the same road, were grouped into separate clusters. <a href="https://samrudh.github.io/llm_meets_pune/model2.html">Here</a>, you can find the results obtained using the OpenAI model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*sYNOJNXJhJ4nTyMAfmp79A.png" /></figure><p>View both the model clusters <a href="https://samrudh.github.io/llm_meets_pune/">here</a>. Feel free to <a href="https://www.linkedin.com/in/samrudha-kelkar-data-science/">connect</a> if you want to discuss more.</p><p><strong>References</strong>:</p><ol><li><a href="https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2">https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2</a></li><li><a href="https://towardsdatascience.com/mastering-customer-segmentation-with-llm-3d9008235f41">https://towardsdatascience.com/mastering-customer-segmentation-with-llm-3d9008235f41</a> (Huge thanks for inspiration!!)</li><li><a href="https://gist.github.com/scionoftech/0f35d5e231be2cf46823d774023268b6">https://gist.github.com/scionoftech/0f35d5e231be2cf46823d774023268b6</a></li><li><a href="http://opendata.punecorporation.org/Citizen/CitizenDatasets/Index?categoryId=34">http://opendata.punecorporation.org/Citizen/CitizenDatasets/Index?categoryId=3</a></li><li>Photo by <a href="https://unsplash.com/@prasad89?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Prasad Bhalerao</a> on <a href="https://unsplash.com/photos/nDWAQ8VyIdI?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=855ce2f60675" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/navigating-pune-with-ai-lens-855ce2f60675">Navigating Pune with AI Lens</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hard Hat Detection: End To End Deep Neural Network]]></title>
            <link>https://medium.com/tech-that-works/hard-hat-detection-end-to-end-deep-neural-network-b6ad3e4bfb18?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/b6ad3e4bfb18</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[work-from-home]]></category>
            <dc:creator><![CDATA[Ashhadul Islam]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 08:24:20 GMT</pubDate>
            <atom:updated>2021-06-17T18:01:37.754Z</atom:updated>
            <content:encoded><![CDATA[<p>From collection, extraction and pre-processing of data to training and deployment of the neural network.</p><p><em>This is written in a hybrid format. It is a tutorial but has a story line. You’ve been warned.</em></p><p><em>Also preferable Operating systems are mac or ubuntu.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*BHQXZGGqCI-zPM3o.jpg" /><figcaption>Lets start where it all begins, a plateful of mouth watering Kachori and Alu ki Sabzi</figcaption></figure><p>This is it, you think, clenching your fist, I need to rope this client in.</p><p>When you had started up your own autonomous camera surveillance company you had no idea that getting clients would be this hard. Surveillance-as-a-Service, hmph, you scoff at yourself.</p><p>Mr Yogi releases a loud burp right on your face. He doesn’t even try to hold it in or redirect it in any other direction. The table is too small, you catch a whiff of his exhaust. It smells of Kachori. Instead of feeling disgusted you start yearning for a plate of hot kachori with steaming aloo ki sabzi. Now you’re disgusted at yourself. At least the guy had breakfast. You on the other hand had to make do with a cup of chai and two biscuits at the small shanty on the way to office.</p><p>“So, did you think about the project”, Yogi asked. Yogi was a senior security manager at this surveillance company. His company was apparently revamping itself and adding lot of AI modules to reduce cost/employees and attract customers.</p><p>“Yeah yeah, we have discussed with our technical team”, you reply, “as soon as we get some sample data from your end, we can start working”</p><p>“Oh really, are you sure you can create a system that detects whether a construction worker is wearing a hard hat or not?”, Yogi sounded doubtful, “From streaming images of the CCTV cameras.” He added.</p><p>Yogis skepticism was not unwarranted. You have already declined some of his more ambitious projects like finding the speed at which people are walking from camera feed, re identifying people when they move from one camera to another. What you have not told him is that you are currently focusing on single frame processing tasks, as in tasks that can be done with just a single frame of image and does not depend on the previous and later frames.</p><p>“You remember our social distancing and mask detection projects”, you try to convince Yogi with some previous wins. Those were all single frame problems. “We already have a framework for this kind of a use case, all we need are images of people wearing hard hats and images of people not wearing them.”</p><p>“Yeah, yeah, sure I will ask my team to get some images. We are also setting up our systems so we are not sure how the final camera stream will look like. But I have some old cameras installed in a couple of construction sites, I will get some images from them.”</p><p>Yogi gets up from the chair, “I’ll send the images over today and I hope that by the end of the week you will be able to give me a demo”</p><p>Today is Monday, if you get the data and they are fine, you can get the team started by Tuesday. “Next week, next Monday”, you reply. Yogi purses his lips, you know he is not happy. “First thing. Monday morning”, Yogi replies. He doesn’t wait for your confirmation, just turns and leaves.</p><p>Later in the day he sends the images, “as discussed”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSxke6rnyP2Jk7HyzcmZyg.png" /><figcaption>“The final images will be of a much better quality”, he adds</figcaption></figure><h4>Data</h4><p>The developers story.</p><p>Well the data that Yogi sent is no good. You have to find other sources of data now. Fortunately, a little bit of googling reveals this.</p><p><a href="https://public.roboflow.com/object-detection/hard-hat-workers">Hard Hat Workers Object Detection Dataset</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xI5irCAZEmDUEG0pvt5yWw.png" /><figcaption>7041 annotated images of people not/wearing hard hats</figcaption></figure><p>You find out that the data is there. You just need to download the images, crop the faces with and without hard hats, keep the faces with hard hats in one folder and the ones without in another.</p><p>Downloading the dataset gives you a zipped folder which on expansion looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*r9tav5-K_rHvLkSX8iNFlg.png" /><figcaption>Test and train folders contain images with faces wearing and not wearing hard hats</figcaption></figure><p>Each of the train and test folders contain an annotations file which is as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7jaT9vx99SoVsISajGjLdg.png" /><figcaption>Annotations file in the train folder</figcaption></figure><p>Each line contains the following things</p><ol><li>Name of the image file</li><li>Multiple quintets(sets of 5) of numbers. Each quintet consists of 4 numbers which give the position of the face, followed by a 1 or 0 defining whether the face has a hard hat on its head.</li></ol><p>You take one line for example</p><blockquote>001446_jpg.rf.00a11ab7cffb0a8fec57e98b2c4663d5.jpg 216,318,265,373,1 5,319,52,376,1 100,325,146,370,1 306,360,347,409,1 133,321,156,369,1 169,174,216,218,0 341,352,368,380,0 291,360,313,399,0 39,358,70,399,0</blockquote><p>The name of the file is 001446_jpg.rf.00a11ab7cffb0a8fec57e98b2c4663d5.jpg</p><p>The file contains 9 faces. Quintet of first face is (216,318,265,373,1). This implies that the face starts from x value 318, y value 216 till x value 373 and y value 265. How did you get to know this? Pure hit and trial. There must have been a better way, but you don’t have time to read the README. The last value in the quintet is 1 which means that the face has a hard hat on. You fire up jupyter notebook to get some value for money.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bee31f79ae66e65295ae214ad0573f7/href">https://medium.com/media/7bee31f79ae66e65295ae214ad0573f7/href</a></iframe><p>And this is what you see in the output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*V2r5Hwtjnf6BhnYx4ooHWA.png" /><figcaption>A group of workers wearing hard hats hauling an unconscious worker</figcaption></figure><p>The annotation works, but you are a little rattled by the image. Probably a case of dehydration you think, must have been the heat. You pick up the AC remote and reduce the temperature. You remember the videos you had seen on how to stay healthy while working from home. You take a large swig from the water bottle. Maybe he wasn’t wearing a hard hat and that is the reason behind his insipid state. All the more reason for you to develop the model. You are elated, you have found your purpose again. You know you’re being sarcastic. You focus back on the code.</p><p>Now, what you basically need to do is write a loop that will go through the annotations file, and for each line, open the image mentioned in the line, crop the faces present in the line and save them in a yes folder or a no folder depending on the last value of each quintet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bbb54e9a21e0f283e73c4c220525722/href">https://medium.com/media/7bbb54e9a21e0f283e73c4c220525722/href</a></iframe><p>At the end of execution you have two folders. One contains faces without hard hats and one contains faces wearing them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*mR2IOyWGAmagUn-3uSTTcg.png" /><figcaption>Output folder after data processing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*Uqmxs3ETObhFMwZJRoPQJw.png" /><figcaption>Faces with and without helmets</figcaption></figure><p>These are the images that you are going to work with. You check the count of images with hard hat and images without. You find that, in the training folder, number of heads with helmet is 15358 while the number of heads without is 4875. Ughh, data imbalance. There could be ways of dealing with this. But once again, time is of the essence. So you decide that you will only use 4875 images with hard hats and all images without, to make it even.</p><p>The images are cropped, focusing only on the face along with the head. This makes your work a little simpler. You decide that you are going to solve this problem in two steps. For any given image, you are first going to use an existing pre-trained model to detect faces. On the faces thus detected, you are going to apply a new hard hat detection model. So your job is now to train a binary classifier using the data present in the no_helmet, yes_helmet folders.</p><h4>Training the hard hat detection model</h4><p>If the data is here, can the model be far behind? You remember that you need to go the washroom. Bio-breaks as they would be called during your office meetings. Ah office, the good times… you start reminiscing about the good old days, until you are brought back to your senses with a mail asking for a status update.</p><p>You decide you are going to use keras, because that requires lesser code than pytorch. More importantly because you only recently learnt pytorch and you’re not that comfortable with it. You import all the libraries and try to decide which model you are going to use.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80576fdb2069c3b6c8e5e783e4287e7f/href">https://medium.com/media/80576fdb2069c3b6c8e5e783e4287e7f/href</a></iframe><p>At first you try to retrain an inception resnetv2 model. You find out that it gives only 60% accuracy in test. Then you try out inceptionv3 — that is better, gives you 79%. Just short of a respectable 80. You know how this is going to look in the slides. You decide to give it one more shot. You go for the VGG16.</p><p>The code for VGG16 is simple, but you need to write a function to get the data loaded from those two folders(yes_helmet and no_helmet).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2761619afb46e77784519038fcbda1a0/href">https://medium.com/media/2761619afb46e77784519038fcbda1a0/href</a></iframe><p>Now that you have the function to load the data, you are ready to train your model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c876c98ef98e63c5ceabd97a930e3215/href">https://medium.com/media/c876c98ef98e63c5ceabd97a930e3215/href</a></iframe><p>It takes a while, but you have access to some GPUs so the wait isn’t that long. You are hopeful because the validation accuracy is looking good. You remember to save the model, re load it and try it on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4133fb73e23e56721c2a7e40da4c49f3/href">https://medium.com/media/4133fb73e23e56721c2a7e40da4c49f3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*X3hbUtlc_vqXyGkUFycCRQ.png" /><figcaption>Test accuracy</figcaption></figure><p>You can barely believe your eyes. 92% accuracy! This is going to look really good in the deck. Master Potter will be pleased. What, you’re not a house-elf enslaved to this work!</p><h4>Tying the hard hat detection model to the face detection model</h4><p>Anyways, you are happy with the progress so far. The end is near. Now you need to first use a face detection model to detect faces in an image and then apply this freshly minted vgg16 model on the face image. There is a gnawing worry in your mind, and it is about to be spilled out.</p><p>First things first, you need a face detecting model that works. You don’t have to look around much, because you have used it before. All hail <em>ageitgey. </em>You check out his <a href="https://github.com/ageitgey/face_recognition">github</a> repo, you know what to do. This repo is all about face recognition. But it does face detection first. So you just need to scoop that face detection part out. You make a note that in order to run the face detection part from ageitgey’s repository, your system will need to have dlib installed. Fortunately the server where you are going to host the code and your laptop, both have been through the ordeal of installing the libraries to support the face_recognition library of ageitgey. However this <a href="https://gist.github.com/ageitgey/629d75c1baac34dfa5ca2a1928a7aeaf">link</a> contains the details in case you ever need to do a fresh install. You remember that it only works in mac or ubuntu.</p><p>You move on and start writing the code. But first you download a couple of images to test the entire flow and put it in a folder called “end_end_test”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkGKTUGihknAkzGb-9ZaCQ.png" /><figcaption>Test images, mix of both wearing and not wearing hard hats</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*odL1FHlS8EfHm8S4x_X5Rw.png" /><figcaption>Another crowd</figcaption></figure><p>You think its going to work out well, but you have this terrible apprehension that something is going to go very very wrong.</p><p>Your realize that your fears are not unfounded when you apply the first step that is face detection and the output that you get is something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*Z-pxnolCmg3oJ_wL-gQlQA.png" /><figcaption>Face detection works as it is supposed to</figcaption></figure><p>You realize that your earlier premise of applying hard hat detection on face detection is not going to work out, because the face detection module is doing what it is supposed to do, figuring out where the faces are. It is not however, accounting for the extra pixels above the head to incorporate the helmet. In this case, all the faces would be classified as not wearing hard hat by your 92% accuracy model because it won’t get to see the hard hat even if it exists.</p><p>At this point only one thing can be done. Time for some dirty coding. You define two variables, head_taller_parameter and chin_taller_parameter. These two parameters are going to be your life savers. Basically what you’re going to do is, once the face co ordinates are detected, you want to crop the image not at the exact co ordinates but only after extending the top and bottom of the image by a bit. Once again you resort to hit and trial to find the optimal proportion of expansion. Finally you get this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*ozadNeWar9pat_gaalj2Vw.png" /><figcaption>There’s the hard hat! (y)</figcaption></figure><p>The values of the head_taller_parameter and chin_taller_parameter are 0.5 and 0.1 respectively. Basically you are extending the face image on the top by 50% and on the bottom by 10% of the actual height of the face image. This gets the helmet in the cropped frame and now you are hopeful that your model will work. You incorporate this “hack” into your code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/834845c968c802b6c085ef4d5d600044/href">https://medium.com/media/834845c968c802b6c085ef4d5d600044/href</a></iframe><p>You run the two step process on each image and put a marker around the faces. Green box if the face has a hard hat, red if it does not.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6113c6b456e1e92a2f12338e880b2071/href">https://medium.com/media/6113c6b456e1e92a2f12338e880b2071/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9Fl6fLXbO4D5rUJZT88YA.png" /><figcaption>6/6 is not a bad start</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*ZGoAtLs1aOufeYx7S7s30g.png" /><figcaption>Third guy is too far away</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bBEMQ5Bu-JJMYwRhsN0m9w.png" /><figcaption>This works too</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-N8Xfl8ZZQP1QMURFuEH9Q.png" /><figcaption>This is also fine</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*-TCc1v7xTjRp839XHEIByw.png" /><figcaption>Not so good</figcaption></figure><p>Now you don’t like to make excuses for your models, but the last result was not that great. It classified some people not wearing hard hats as actually wearing them. You have a feeling you know the reason why, but stating that out loud might be derogatory to the people suffering from hair loss.</p><p>You decide that you are okay with the model and it is ready to go for production. You (barely)clean out the code and put it on <a href="https://github.com/ashhadulislam/hard-hat-detection">github</a>, because whatever comes from open source must go back to open source.</p><p>Then you sit with your team and integrate the model into the system so that it works real time from cctv feed. Now that is a story for another day.</p><h4>Epilogue</h4><p>Building a model in jupyter notebook is one thing, but getting it hosted in a real time cctv surveillance and analytics platform, so that it works without reasonable delay — that is a different beast altogether. And that would not be possible without the folks at hipla, <a href="https://www.linkedin.com/in/haquekazi/">Kazi</a>, <a href="https://www.linkedin.com/in/sudiprooj/">Sudip</a>, <a href="https://www.linkedin.com/in/shubhajit-roy/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAACzjwRMBBbFnRWjBd0SWy9EAdpldMJunP_4">Shubhajit</a>, <a href="https://www.linkedin.com/in/anubhab-ray-98a75b175/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAACmUiLoBeat89HCruqAAjcas-oqId2wjz4w">Anubhab</a> and others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zc0NcYSpKhp1IcrRrWLH5w.png" /><figcaption>Live hard hat detection</figcaption></figure><p>For more details about our capabilities check out this <a href="https://www.youtube.com/watch?v=Cuj2RgHzGLY">youtube video</a> or visit <a href="https://contatrack.ai/">https://contatrack.ai/</a></p><p>Until next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6ad3e4bfb18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/hard-hat-detection-end-to-end-deep-neural-network-b6ad3e4bfb18">Hard Hat Detection: End To End Deep Neural Network</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Converting Templates to React Project]]></title>
            <link>https://medium.com/tech-that-works/converting-templates-to-react-project-b737b0456fb8?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/b737b0456fb8</guid>
            <category><![CDATA[redux]]></category>
            <category><![CDATA[html]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[website]]></category>
            <category><![CDATA[dynamic-web-development]]></category>
            <dc:creator><![CDATA[Prateek Mishra]]></dc:creator>
            <pubDate>Thu, 10 Jun 2021 10:36:00 GMT</pubDate>
            <atom:updated>2021-06-12T10:29:41.256Z</atom:updated>
            <content:encoded><![CDATA[<p>Many times we might have thought of something wonderful but not able to build or create it. This is the case with most of Backend Developers/ UI Developers when they want to build a beautiful UX design.</p><p>As an engineer who is working on backend designs / code or who is A UI engineer, designing the UX might not be your strong suit. To kick start we might look up on ideas and HTML templates available online and mimic the design. But it’s not that easy ? Is it. Breaking up the html template into a framework specific guideline always seems to be difficult.</p><p>Today I am going to demonstrate how can you convert any HTML template into a React Project, divide them up in component and make it a dynamic website. This would help you kick start on your full stack project and help you put together your ideas and code into visualization.</p><p>To get started, let’s understand what is <strong>React</strong></p><blockquote>React is a Java script Library for building user interface. Integrating react with redux (a container for JS apps), you can achieve a standalone User Interface Project and deploy that anywhere.<br>Read More about React visit the <a href="https://reactjs.org/">official website</a>. For Redux <a href="https://redux.js.org/">click here</a></blockquote><p>Let’s download node.js first to build our react application —<a href="https://nodejs.org/en/download/"> Download Here</a></p><p>Once the installation is successful (<a href="https://nodejs.org/en/">Node &gt;= 10.16 and npm &gt;= 5.6</a>), you can open up a terminal and create a new react Project with <a href="https://www.typescriptlang.org/">typescript </a>template</p><blockquote>npx create-react-app myreactproject — template typescript</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SoUo0yVMS6XqRspOT9e62A.png" /><figcaption>Creating new react app (the command has 2 hyphens before template)</figcaption></figure><p>Now Add some base packages, you can use yarn (react guidelines) or npm which ever you like</p><blockquote>npm add typescript <a href="http://twitter.com/types/node">@types/node</a> <a href="http://twitter.com/types/react">@types/react</a> <a href="http://twitter.com/types/react-dom">@types/react-dom</a> <a href="http://twitter.com/types/jest">@types/jest</a> <a href="http://twitter.com/types/react-router">@types/react-router</a> <a href="http://twitter.com/types/react-router-dom">@types/react-router-dom</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bpqTCH-mJSul9-J2SYV2Qw.png" /><figcaption>npm install some base react packages (Might take a while)</figcaption></figure><p>Now let’s check out and clean our project and create a base component folder.</p><p>Use <a href="https://code.visualstudio.com/download">Visual Studio code</a>, you can open it by typing<strong> ‘code .’</strong> in the terminal. Clean up the project by making the following changes</p><blockquote>Create folder: <strong>components/app</strong> inside <strong>src</strong><br> Move: <strong>App.tsx</strong> into <strong>components/app</strong></blockquote><blockquote>delete: public/robots.txt<br> delete: src/App.css<br> delete: src/App.test.tsx <br> delete: src/index.css <br> delete: src/logo.svg<br> modified: public/index.html — Remove Logo references<br> delete: public/logo192.png<br> delete: public/logo512.png<br> deleted: src/setupTests.ts<br> deleted: yarn.lock</blockquote><p>Remove the reference of index.css and logo.svg from App.tsx. Completely clean the project with no demo data. The Project looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nF9L64mliwdYD5kujKl8RA.png" /></figure><p>Do test the application once before getting started.</p><blockquote>Run the project using ‘npm start’</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*9xDWgAW5hKv4aZGU3Y0cCg.png" /><figcaption>This is how it looks like</figcaption></figure><p>This should launch your application to localhost:3000 (default)</p><p>Now that we are done with base react project start, let get some template which we want to convert to react to React Project. You can search and download your own template. For demonstration I will let download the below mentioned template</p><p><a href="https://bootstrapmade.com/iportfolio-bootstrap-portfolio-websites-template/download/">iPortfolio - Bootstrap Portfolio Websites Template</a></p><p>Once you download any template, the first thing to do is to identify the components sections which you would create to convert this to react project</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcSIS7aIw0YglUztPFO5cg.png" /><figcaption>Breaking a template into Components</figcaption></figure><p>As you can see we need to break and identify the components in a html template. In our case they will be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*UyIZ9ydfMadQcgfVOTvtkQ.png" /></figure><p>Based on Side Panel’s Navigation Menu Component Menu Item click, we would load different content in Main Content Component</p><p>Now let’s create the individual components in vscode. For creating component’s you need to create a folder with component name, create a component file(.tsx) and an index.tsx file, as per the folder structure. In this case it should look something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_LgBxhM45bgSqvvthmVGZQ.png" /></figure><p>Currently All files are empty. We have just created a folder structure.</p><p>Now we need to move each section carefully into the related component. Open up the index.html from the temple and start copying sections into respective .component.tsx file</p><p>While copying we need to take care of 2 things.</p><ol><li>We will create functional components inside .componet.tsx file and inside the return we would populate the copied html</li><li>we would need to close each open html element. Example &lt;img&gt; would work in html but in react would throw error, hence you need to close this as &lt;img /&gt;</li><li>class would be replaced by className as per react standards. So when you copy html content into react component, replace the keyword class with className</li><li>All inline style should be created as a class and kept in public/custom.css and you can then import this custom.css in index.html under public</li><li>Copy all static content like images, script, css under public folder inside react project</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBu2E3vEuDzfBcqoBFcjAg.png" /></figure><p>6. Modify the index.html inside public folder with the index.html from the template. This is how the current index.html from public folder looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5z6YYCruloZd468XL52sUQ.png" /><figcaption>Before capture of index.html</figcaption></figure><p>While doing this step keep in mind</p><p>a. After copying replace the entire body section with</p><blockquote>&lt;body&gt; <br> &lt;div id=”root”&gt;&lt;/div&gt; <br> &lt;/body&gt;</blockquote><p>b. Ensure your html header has a link to manifest. This is required by react</p><blockquote>&lt;link rel=”manifest” href=”%PUBLIC_URL%/manifest.json” /&gt;</blockquote><p>c. If you have any scripts under body section, you can keep it there or move to header section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EyMgPiGdT-uDp3Abf9x74w.png" /><figcaption>After capture of index.html</figcaption></figure><p>Now the section &lt;div id=”root”&gt; &lt;/div&gt; is controlled by react, and replaces only the content which is updated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*VaV2DJBNMr1BsH75--R2fg.png" /><figcaption>React updating root element with App Component</figcaption></figure><p>Now that we have populated the index.html file, moved all the assets/css/js and images files to public folder, the only job to do is to move the content per section to related component. This might take a while. Start with the smallest and easiest component. In our case that would be footer component. For each component, you need to</p><ol><li>Create an empty Functional Component and export it</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*U2ow7eqTXW9ZEz15irCImg.png" /></figure><p>2. Update the related index.tsx file to export this Component. You can directly export from the component.tsx itself, but as a good practice we export files via index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*F4cJ-UtfOZ_1teeRkW36OA.png" /></figure><p>3. Update the footer content into the footer.component.tsx ‘s return section</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GBcBPzaFBgCElhjZLUr8A.png" /></figure><p>Remove the comment section while copying and replace ‘class’ by className in the component file. Once moved keep a placeholder under the originalFile to keep track of what you have moved</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CeFr6UP7Sg-NSaaOTlWZA.png" /></figure><p>Like this, we need to move all components one by one into their respective components. Proceed with this or have a reference to the project link below in case you face an issue during this step.</p><p>Let’s take one more example. We would now move the side bar panel and create router to sub components, so that on click of an element, the particular section is loaded on the right hand side</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8LJOoZOLiqf8oGJlguuAQ.png" /></figure><p>Now lets create route for each page using Link attribute</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltp7lTZgxDLLE1AQEFI0gA.png" /></figure><p>Now that we have created routes, we can map different component to these routes using a simple switch in the main Layout component. Before moving to this step ensure you have created all seperate components.</p><p>While combining the components to parent, now you can reference the child components created directly</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6fxqiEvF7yrgYxkZyqtlA.png" /></figure><p>Routers Can be created based on what you want to display on click of elements</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*crDukCZ8A5uN3YWzql26iw.png" /></figure><p>One optimization that needs to be done in the above code is to seperate the common code out. Example Remove the sidepanel outside and keep the tag &lt;main id=”main”&gt; outside the switch. I leave that to you guys</p><p>Once you have completed this intergration, Tadaaa, you are ready to go, just fire up npm start</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rR4Nnbc_aP7hGgIs-6cJA.png" /></figure><p>The entire webpage is now a react project. You can further modularize this project into smaller components. Note that when a component is reloaded only that part of dom is refreshed and hence if you feel a section may change more frequent that another, then you must create them in different components.</p><p>I hope you had fun and would find this useful. You can find the source code to this Project here.</p><p><a href="https://github.com/5-k/HtmlToReactApplication">https://github.com/5-k/HtmlToReactApplication</a></p><p>If you like my content, do support or buy me a coffee :p</p><p><a href="https://www.buymeacoffee.com/PskV9Rt">https://www.buymeacoffee.com/PskV9Rt</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b737b0456fb8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/converting-templates-to-react-project-b737b0456fb8">Converting Templates to React Project</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set up AWS EC2 instance in less than 10 minutes]]></title>
            <link>https://medium.com/tech-that-works/set-up-aws-ec2-instance-in-less-than-10-minutes-3a1ed8e243d2?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/3a1ed8e243d2</guid>
            <category><![CDATA[java-webservices]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Prateek Mishra]]></dc:creator>
            <pubDate>Mon, 07 Jun 2021 16:08:25 GMT</pubDate>
            <atom:updated>2021-06-07T16:08:27.509Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*Ff84x9Y-7PPWStblMhpFHg.png" /></figure><p>Today I will be talking about how to start up your ec2 instance up and running.</p><p>For those of you who are new to Amazon web services, AWS is subsidiary of amazon that provides you on demand cloud computing platforms to individuals and to companies on a paid subscription basis.</p><p>AWS provides you with multiple web services and EC2 (Elastic cloud compute) is one of the most used web service. Using EC2 you can setup your virtual machine on cloud with various plugins or managed services like RDS (database systems), SNS (Notification service), firewall, security etc. You can access you EC2 instance by SSH or can use the public &amp; private keys to access the machine. And since its on cloud its accessible from anywhere and you have your setup done in a matter of minutes.</p><p>Let’ begin on how to setup an EC2 instance. For people who do wish to try AWS there is a free 12 months package available by Amazon. Create a free account.</p><p><a href="https://aws.amazon.com/free/">Free Cloud Computing Services - AWS Free Tier</a></p><p>For creating a EC2, first we need to create a VPC (Virtual Private Network). This requires a a basic knowledge of networking. We need to create a private network and subnets where the instance will be running.</p><p>Chose the Region as per your wish on the top right corner. It is this region your virtual machine will be set up. You can later create a replica of your machine in a different region for disaster recovery.</p><p>In our example, we will be running one private and one public network inside a single availability zone (AZ’s). The EC2 instance will run on the public network (connected to internet gateway) and the private network would have any service running that you want to protect from direct access. In this example we would be adding a RDS (relation database). Any service running on a private subnet would require to connect to a public subnet via a NAT gateway. Have a look at the architecture diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*VASHjSmZMVO1Wsyqv23Sog.png" /></figure><p><strong>Creating a Virtual Private Network</strong></p><ol><li>Go the AWS icon, and click on the service VPC.</li><li>Click on the create VPC button on the top left.</li><li>Select the second option to create a VPC with public and private subnet (You could choose as per your requirement, keep all the secure services on private network that you would want to protect from direct access to the internet)</li><li>You need to provide a CIDR range(Classless Inter domain range), public subnet and private subnet range.</li><li>Give a name to your VPC, Public and private subnet.</li><li>Ensure that the Public and private subnet are in the same availbility zone. If you need to create a high availbility architecture then you should create two more public and private subnet (post the VPC creation) manually. I would cover that in a seperate story.</li><li>Select the option, Select an NAT instance instead. Key pair is optional. You can create a key pair from VPC dashboard later and attach to your VPC. We would create our NAT instance without a key pair.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*00HjXRYc-X2o6q5IhRmJSQ.png" /></figure><p>Once your VPC is created, go to the VPC dashboard and check the details of your machine.</p><ol><li>Under VPC, you would be now seeing a new VPC by the name the you created.</li><li>The subnet tab would now have 2 subnets, the private and the public subnet with the name that you created.</li><li>Route Table would have 2 new tables, one for each of private and public subnet.</li><li>Here under one of the route tables, you would see a table with one subnet associated. This is a public subnet. Under the routes, you would see that all the traffic (0.0.0.0/0) is routed to a newly created internet gateway. You can give the table a name, say public route table.</li><li>The second table, would be a private route table. Under the routes tabs you can see that all the traffic (0.0.0.0/0) is routed to a NAT instance (present on the public subnet). This way any communication from this subnet would go through the NAT instance to the public subnet. From public subnet it goes to the internet gateway.</li><li>You would need to associate your private subnet to this route table. Click on subnet association, and select your subnet CIDR, that you provided when creating VPC.</li><li>Check out the security group tab. A new security group is created while creating a VPC. By default All ports are open. You can control to restrict the open ports. (Lets say only SSH(22) and HTTP(80)). Note that security group is associated to each VPC. Provide a name to this group.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rDBIetg_FgRMIfM4ygbMpg.jpeg" /><figcaption>Private Route Table</figcaption></figure><p>Once your VPC is all set, we can now create a EC2 instance. Click on the aws icon and open the EC2 service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AcYbfZJRC7PdaVBREBliww.jpeg" /><figcaption>Public Route Table</figcaption></figure><p><strong>Creating an EC2 instance</strong></p><pre>1.Click on Launch EC2 instance.</pre><pre>2.Select the type of AMI (Amazon machine Image). There are some choice to select the market place AMI or a community AMI (amazon), or you can customize your own as per your requirement.</pre><pre>3. Once selected, now you need to select the machine hardware type. This is again as per your requirement. In this example, I have selected a general purpose t2 micro. Click on Next: Configure Instance</pre><pre>4. Select the number of instances as 1 and then select MY VPC as your network. Note each EC2 instance is chargeable, so select as per your requirement. </pre><pre>5. Ensure that the selected subnet is a public subnet, else you would not be directly able to access this machine. Click Next: Add storage.</pre><pre>6. By default a 8 GB volume is added, which is marked as delete on termination. You can chose not to delete the data on termination in case you wish to attach this storage(EBS storage only) to a different EC2 instance later. You can add more EBS storage as per your requirement. </pre><pre>7. Click on Review and Launch.</pre><pre>When launching you would be asked to create a new key pair (if you don’t have already). Give your key pair a name and then click on download key pair. Note that this key pair can be downloaded only one time. If you lose this you cannot download the key pair again. You would need to go through the recovery process (in a different story).</pre><pre>8. Click on launch instance.</pre><p>Your instance would take 2 to 3 minutes to launch. Go to the EC2 dashboard and check for the status. Once the status is running you can use that machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1dvtjVPfFxD3jwxcL8NzZQ.jpeg" /></figure><p>That’s it. You just created a compute machine in a matter of minutes. Imagine the burden you would need to do to buy and configure your PC. AWS makes it ready in less than 3 minutes.</p><p>You can try doing ssh to this machine, to test if this machine is working okay. To do so you need to attach an elastic public IP to this machine.</p><ol><li>Click on Elastic IP tab.</li><li>Click on Allocate new address</li></ol><p>You Elastic IP is ready now. Go to the elastic IP dashboard and select the elastic IP created. <br>Click on Actions &gt; Associate address. Select the ec2 instance that you created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbGyPU9y3m1QMByJFJH_Gg.jpeg" /><figcaption>Elastic IP association</figcaption></figure><p>Now Head back to EC2 dashboard. Under the instance summary you would see the public IP associated this instance.</p><p>Do a ssh to test this machine</p><pre>ssh ec2-user@&lt;Elastic-IP&gt; -i &lt;My_Key.pem&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*Q48O8FhCXyTRpZ2uGCpsXw.png" /><figcaption>SSH Machine</figcaption></figure><p>Note: My_Key.pem is the key that you generated while creating ec2 instance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a1ed8e243d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/set-up-aws-ec2-instance-in-less-than-10-minutes-3a1ed8e243d2">Set up AWS EC2 instance in less than 10 minutes</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running RASA 2 bot with GCP]]></title>
            <link>https://medium.com/tech-that-works/running-rasa-2-bot-with-gcp-810abec7d7e0?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/810abec7d7e0</guid>
            <category><![CDATA[chatbot-development]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[app-engine]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[rasa]]></category>
            <dc:creator><![CDATA[Samrudha Kelkar]]></dc:creator>
            <pubDate>Sat, 27 Mar 2021 11:04:31 GMT</pubDate>
            <atom:updated>2021-03-27T18:00:55.085Z</atom:updated>
            <content:encoded><![CDATA[<h3>Running RASA 2 chatbot with GCP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/0*StxDsuJPSSWlJpBG.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*9tTDPV-kKlx5syB8ZxHFQw.png" /></figure><p>Have you built your first rasa bot?</p><p>Is it running only on your machine and you are eager to show it to the world?</p><blockquote>If yes, this is the blog for you!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4fD6qvVzPxROQkVnJj6vg.jpeg" /></figure><p>Long back, when RASA was a new cool thing in the town, I had written about <a href="https://medium.com/tech-that-works/running-rasa-bot-with-aws-ec2-c08ed3b02455">hosting the rasa bot on AWS</a>. Now that there have been significant changes within RASA in the last two years, it&#39;s time for a new post to host the latest bot on a cloud, this time on<em> GCP</em>. Let&#39;s dive straight into the code..</p><p><strong>Get the bot running on local</strong></p><p>RASA has made this step extremely easy with its new CLI-based interface.</p><p>Get a fresh folder, a fresh python virtual environment</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4b0816d16d0dfaea5511e2256be1ef2/href">https://medium.com/media/f4b0816d16d0dfaea5511e2256be1ef2/href</a></iframe><p>This will install the latest open source rasa version. Now just type <em>rasa init </em>to create a project structure of your bot.</p><p>Test your bot by typing <em>rasa shell</em>. This will create a sample bot that comes with a prebuilt NLU and domain. It will interact on basic chitchat like hi, bye.</p><p><strong>Dockerized bot image</strong></p><p>Shell is good enough to test your bot but let&#39;s create a proper Docker image for the bot to deploy it systematically. In the same project folder create an empty file with the name <strong>Dockerfile </strong>(note that it&#39;s not Dockerfile.txt)</p><p>Add below content in the same.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23cfc0c997ec8d39f63bd72a75b6dac5/href">https://medium.com/media/23cfc0c997ec8d39f63bd72a75b6dac5/href</a></iframe><p>We can quickly check if the image is built correctly.</p><blockquote>docker build . -t &lt;rasa bot tag&gt;</blockquote><p>Now create an empty <em>app.yaml</em> file in the same folder and add below content</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/389fd42cc8b924b6d176516cabb39c09/href">https://medium.com/media/389fd42cc8b924b6d176516cabb39c09/href</a></iframe><p>I referred to <a href="https://medium.com/@muhammadhassaanrafique/deploying-rasa-powered-chatbot-to-google-cloud-platform-f4ee2eee5be4">this blogpost</a> to create this YAML file. Note that the extension of the file is not <em>.yml</em>, it&#39;s <em>.y</em><strong><em>a</em></strong><em>ml</em></p><p>Add the entire code now to your Github by creating a fresh repository. <a href="https://docs.github.com/en/github/getting-started-with-github/create-a-repo">Read here</a> if you have never done that.</p><p><strong>Host a bot on GCP</strong></p><p>Now that we have docker image and App engine descriptions for our bot, we will host the same in GCP. We will be using<em> App Engine</em> service by GCP which lets you build scalable apps without worrying much about server management. All configurations can be just specified in app.yaml file and the App Engine will take care of deployments. You can read more about it <a href="https://cloud.google.com/appengine">here</a></p><blockquote>However, just be aware of the cost. Everything simple comes at a cost. My bot which ran for about 2 days continuously came with bill of INR 400 (~5 USD)</blockquote><p>Let&#39;s see how to deploy App Engine now:</p><ol><li>Create a new project in GCP</li><li>Go to App-engine service and enable it for the project</li></ol><p>3. Open a cloud shell symbol in the right top which looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/63/1*bC-4s-LUWQ1GmPwCFM2kEw.png" /></figure><p>4. Open a terminal and clone the repository in which you have committed your bot code.</p><pre>git clone &lt;your_repository_name&gt;</pre><p>(you will be asked your Github username and the password. This way of password authentication will soon be deprecated by Github. However, there is a way-out which I will explain below)</p><p>5 Navigate to the folder of the repository and type</p><pre><strong>gcloud app deploy -v &lt;your_bot_deployment_version&gt;</strong></pre><p>6. This will deploy your bot. The hosted link address can be extracted from <em>gcloud app browse</em></p><p>If you go to the link from the above step, you should see the same message that we saw when we ran Docker on local.</p><pre>Hello from Rasa: 2.4.0</pre><p><strong>Auto build and push the updates:</strong></p><p>It&#39;s not a good idea to login to cloud shell every time you change something in your bot code. To avoid that you can hook a trigger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*0Fn6Zuz8eA0D1-ZIdcPk7Q.png" /></figure><p>7. Search for triggers in the GCP console and click on the <strong>create trigger </strong>button as shown above</p><p>8. Give some name, description, and tag. That&#39;s for your reference.</p><p>9. In the event section click on push to a branch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*5tNyZApxuFEPTjzTqTJQiQ.png" /></figure><p>10. In the Source section click on <strong>connect new repository</strong> part.</p><p>It will open a side tab like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*cCAqeGg_I1fGbjtuT7NaFA.png" /></figure><p>Here select Github</p><p>Authenticate your Github account</p><p>Select the repository which has bot code. I recommend ONLY choosing one repository that has code and not others associated with your account</p><p>10. Since we have already had Dockerfile in the repository, we will select the Dockerfile option in the configuration. GCP also has <em>cloudbuild </em>option for building your app if it is not dockerized already.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*oCJ_rB5FBJHJZ1xyFPu4kQ.png" /></figure><p>11. The last thing is to assign permission to our trigger to redeploy App-engine when you change the code. For that go to the Settings part of your Trigger, and enable App-engine permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*tDPVxkI8NfyexAOhmaSVcA.png" /></figure><p><strong>Interact with the bot service:</strong></p><p>Now that we have built a backend service, we can use any channel to interact with the bot. I will stick to the web-based REST API calls like my <a href="https://medium.com/tech-that-works/running-rasa-bot-with-aws-ec2-c08ed3b02455">last blog</a></p><p>Here is the snippet from my flask app which sends user text to the bot service in GCP and returns back the responses</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e45957bfd17fa0227f62b082eedb805/href">https://medium.com/media/0e45957bfd17fa0227f62b082eedb805/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=810abec7d7e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/running-rasa-2-bot-with-gcp-810abec7d7e0">Running RASA 2 bot with GCP</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Machine learning models in production]]></title>
            <link>https://medium.com/tech-that-works/monitoring-machine-learning-models-in-production-a932dc388515?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/a932dc388515</guid>
            <category><![CDATA[model-monitoring]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aditya Kumar]]></dc:creator>
            <pubDate>Tue, 23 Feb 2021 10:51:23 GMT</pubDate>
            <atom:updated>2021-02-23T10:51:23.346Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R4ZfT6zvjCIY4-EK" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>After deploying many ML models in production, it became evident that there should be an easy and efficient way to monitor the ML models after deployment. This blog post is focused on monitoring the classification models in production.</p><p>Recently, I was working on the text classification problem which will classify the text into one of ~50 categories. Once the model is built and tested, it needs to be deployed as a flask API along with other models. Some text classification models are already deployed as an API that uses python flask to serve the incoming requests which use Gunicorn as a WSGI server and are deployed on Kubernetes clusters and trained models are stored in S3. So, the current architecture looks something like this, and a newly trained model needs to be deployed in this kind of setup</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*i_Wg2uzukiqAIUNZfrjtKg.jpeg" /><figcaption>Classification Model APIs</figcaption></figure><p>To prepare the training data, we have started with some keywords related to categories to tag the data, generate new keywords from existing categories, manually tagging data, etc to prepare the training dataset. Handling cases, where a text might fall into any of closely related categories and also considered already deployed classifiers to tag data in some of the categories on which they were trained on. Random manual check on labels was done to ensure if tagged entries using the above methods are good enough as training data is in the range of few millions.</p><p>Then a model is trained using training data and tested on a held-out set to measure the performance of the classifier, if model performance is acceptable on test data then the model is deployed in production as an API. Now as the model is deployed, then why there is a need for monitoring the model?</p><p><strong>Need for monitoring</strong></p><p>You have already put the classification model in production, which performed very well on your test dataset. But,</p><ul><li>How do you know that the model is performing well on the new data?</li><li>How do you know it is time to retrain the model?</li><li>How to know the effect of data drift and conceptual drift on the model?</li></ul><p>One possible way to detect the performance of the classifier is to take all incoming requests for classification for a certain duration and manually label the data and compare it with model predictions. Do this exercise periodically after a certain period of time to gauge the performance of the model. This is the same thing that we did while training the model and needs to be done periodically. This process seems unintuitive to me and requires a lot of manual effort and periodically monitoring of models which become very cumbersome when you have many classification models running in production.</p><p>There is a need for something simple, quick, and yet very intuitive which gives an idea about how models are performing in the production. Also in the current setup, how models are deployed and used by downstream applications is pretty stable, therefore don’t want to invest too much time in getting a look at available ML tools that provides this functionality out of the box.</p><p><strong>Prediction Distribution of Model</strong></p><p>After some research, having a dashboard that displays the plot for the prediction distribution of the incoming requests seems very intuitive to me and will also answer some of the questions, like:</p><ul><li>How a model is performing against each category?</li><li>Does prediction distribution follow a similar pattern as training data?</li><li>Is the model biased towards any category i.e. model is predicting some class very often?</li><li>Is the model failing to predict any category?</li><li>Is there a need to retrain the model?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GAHlC4mg9QMu0OqvfWkFog.png" /><figcaption>Distribution of predicted classes</figcaption></figure><p><strong>What else can be tracked?</strong></p><p>A new model is trained to predict the text into one of the 52 categories and uses the BERT-base-cased model, so to deploy that in production and staging we have to increase the resources significantly in comparison to previous models so that model can run smoothly on CPU.</p><p>Generally, when it comes to deployment, there are two environments Staging/UAT and Prod, and there is a significant difference in these environments in terms of resources allocated to the application like memory, CPU time. The idea is to allocate more resources to the application in production so that it can serve its purpose without any issue. In our case also, in production number of workers running are 2X times as of staging environment, hence resources needed are also doubled. Therefore we want to know that do we really need the increased resources in production?</p><p>That&#39;s why we want to track the number of API calls which will eventually answer a few questions like</p><ul><li>Is there any need to increase or decrease the resources in production?</li><li>Can the whole system be deployed as batch inference in case the number of API calls is less?</li><li>Are previously deployed models still being used by downstream applications and if so, then how frequent?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZA161TxbxH8qpdN-pvPtHA.png" /><figcaption>Total API Calls</figcaption></figure><p>There might be some more metrics we can measure like, the responsiveness of APIs. But here, the main focus was to know how the model was performing in predicting the categories and keep the effort very simple to track down these metrics.</p><p>Here is the sample code for generating divs for Bar plot.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc4afd6e1640abd61ee5d326e85a95cb/href">https://medium.com/media/bc4afd6e1640abd61ee5d326e85a95cb/href</a></iframe><p>Below, HTML template can be rendered from the Flask module by passing the HTML divs generated by the above code.</p><pre>@app.route(&quot;/monitoring&quot;, methods=[&#39;GET&#39;])<br>def monitor():<br>    divs = generate_div(predict_dist)<br>    return render_template(&#39;monitor.html&#39;, div1=Markup(divs[0]),   div2=Markup(divs[1]), div3=Markup(divs[2]))</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c08e288826579251cac12a1f96c973b4/href">https://medium.com/media/c08e288826579251cac12a1f96c973b4/href</a></iframe><p>Please let me know if you like the post, or have some suggestions/concerns and feel free to reach out to me on <a href="https://www.linkedin.com/in/aditya00kumar/">LinkedIn</a>.</p><p>References:</p><ol><li><a href="https://christophergs.com/machine%20learning/2020/03/14/how-to-monitor-machine-learning-models/">https://christophergs.com/machine%20learning/2020/03/14/how-to-monitor-machine-learning-models/</a></li><li><a href="https://mlinproduction.com/value-propositions-ml-monitoring-system/">https://mlinproduction.com/value-propositions-ml-monitoring-system/</a></li><li><a href="https://www.explorium.ai/blog/understanding-and-handling-data-and-concept-drift/">https://www.explorium.ai/blog/understanding-and-handling-data-and-concept-drift/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a932dc388515" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/monitoring-machine-learning-models-in-production-a932dc388515">Monitoring Machine learning models in production</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Flighting in Angular]]></title>
            <link>https://medium.com/tech-that-works/feature-flighting-in-angular-c1bcac89f696?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1bcac89f696</guid>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[featureflighting]]></category>
            <category><![CDATA[flighting]]></category>
            <category><![CDATA[js]]></category>
            <category><![CDATA[features]]></category>
            <dc:creator><![CDATA[Prateek Mishra]]></dc:creator>
            <pubDate>Thu, 11 Feb 2021 15:14:05 GMT</pubDate>
            <atom:updated>2021-05-27T05:35:53.323Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>In today’s fast-paced, feature-driven markets, it’s important to continuously deliver value and receive feedback on features quickly and continuously. Partnering with end users to get early versions of features vetted out is valuable.</blockquote><p>Are you planning to continuously integrate features into your application while they’re under development? You probably have a few questions, such as:</p><ul><li>How can you toggle features to hide, disable, or enable features at run-time?</li><li>How can you revert a change deployed to production without rolling back your release?</li><li>How can you present users with variants of a feature, to determine which one performs better?</li></ul><p>We might want some features to be available to developers only and we might have a set of beta users who we want to expose the feature to. In this case we need a framework which grows above the *ngIf from Angular.</p><p>The importance of such framework is that when we want to remove the flighting and make it available to general public it should not be a hurdle to do.</p><p>For this we would have to create a backend and a frontend that would be readily used. In this I would be discussing on the UI part of this framework.<br>For backend, I would be using Azure App Configuration exposed over an api which would be called from UI to get the feature information. I will cover that in a separate post.</p><p>Traditionally you can use the *ngIf or [hidden] or even redirections. However the key is to expose a directive and use a Angular Service for reusability. Plus we need to efficient enough to add a caching layer so that we don’t keep on calling the API and save some network cost.</p><blockquote>First we create a model class on taking the inputs and the kind of flighting we want to have -</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*v0YaUWaIa1pgjP7g0CRIWg.png" /><figcaption>Models required</figcaption></figure><blockquote>Now we would be requiring a Angular Service that uses these models and decides the flighting</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*io4YG5oghw6Z51zBAgPypQ.png" /><figcaption>Service</figcaption></figure><p>In this injectable service, we are essentially using calling an external service (refer the backend POST) and we are storing the flighting information in cache (session storage). The code is available in github.</p><blockquote>Now we create a directive to use this service.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*ivbfCQUW72ISnBsSCtAKTA.png" /><figcaption>Directive</figcaption></figure><p>Once we have these three in place we are good to know. You can keep them in a shared angular module and export the directive and the service.</p><p>The good point here is that unlike traditional way, you don’t need to inject the service in every component you wish to use. You can just create a model and use the directive directly in the html and the binding would be handled by Angular.</p><blockquote><strong>Implementation in code-</strong></blockquote><p>Now lets say you have a component in UI that you wish to feature flight to some users. For other’s that option should not appear, in that case you can create a model like explained below.</p><p>Declare a variable in your TS file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*OrwSMw1PlvkVXChueZRDXg.png" /></figure><p>And then populate the model in ngInit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*gWx_K_TGBx0p137DDLfXAQ.png" /></figure><p>Here you are mentioning that if the feature is not flighting then the attibute this is applied on should be hidden. The key is passed to the feature type and the value is the user email. In case you wish to expose it per user you can follow this way. But in case there is a common attribute let’s say country code and you wish to expose a feature to an entire country then you need to pass the feature key would become by country code and you need to handle the same in the directive.</p><p>And in the html you would need to add this attribute on the element you want to toggle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*NOp7xKNb4nHODM1pKx_e9Q.png" /></figure><blockquote>And that’s it.</blockquote><p>Once you build the reusable component (service, directive and models), you can use the code extensively by just proving a model input and binding the html element with the model.</p><h4>There can we different use cases, such as</h4><ol><li>Hide if flighted</li><li>Show if flighted</li><li>Redirect to some unauthorized page if flighted.</li></ol><p>All of such conditions can be a part of the service and then you can use the feature by just toggling the model across components.</p><p>I would create a separate post on the API service that would fetch the flighting inputs.</p><blockquote>Do give me a thumbs up in case you find this useful.</blockquote><p>Here are the link to the code base -</p><p><a href="https://github.com/5-k/feature-flighting-angular">https://github.com/5-k/feature-flighting-angular</a></p><p>Cheers</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1bcac89f696" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/feature-flighting-in-angular-c1bcac89f696">Feature Flighting in Angular</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Kinesis Data Streams using Python — Part 1]]></title>
            <link>https://medium.com/tech-that-works/aws-kinesis-data-streams-using-python-part-1-197a37552fe6?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/197a37552fe6</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ashhadul Islam]]></dc:creator>
            <pubDate>Tue, 18 Aug 2020 09:16:57 GMT</pubDate>
            <atom:updated>2020-08-19T14:27:42.050Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Kinesis Data Streams using Python — Part 1</h3><p>Writing into AWS Kinesis Data Stream using Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOt7QYpXb-7DHjcpYzMThg.png" /><figcaption>How It Works — Amazon Kinesis Data Stream(Picture from <a href="https://aws.amazon.com/kinesis/">aws</a>)</figcaption></figure><p>With streaming data widely available, you may often need to store the large volume of incoming data before storing it in the database. AWS Kinesis is a platform for Ingesting and storing data streams before they can be subjected to further processing. Using Python writing into the data stream and reading from it is easier than what you’d think. Let’s find out.</p><h4>Part 1 — Setting up kinesis data stream in aws</h4><p><strong>Console Work</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*he7b0llQjc6aoAFB8CDgZA.png" /><figcaption>Choose Kinesis service from AWS Management Console</figcaption></figure><p>At the AWS management console, search for kinesis and choose the option as shown in the image above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hyG5hLXp4j7EKpAJ7F8Sg.png" /></figure><p>I already have a data stream so it shows total data streams as 1 for me. For you it might be 0. Let us get our data stream up and running. Click on create <strong>data stream</strong>.</p><p>Next give a name to the stream and assign the number of shards that you want. More the number of shards, more data kinesis will be able to process simultaneously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*4-_JtSyIp-KP_F9SWzVHgA.png" /><figcaption>1 shard sufficient for now</figcaption></figure><p>Click on create stream and you can see your data stream pop up in the console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*EJ-HieP3zoD6lvfE2KmvHA.png" /><figcaption>Takes a while for the status to become Active</figcaption></figure><h4>Part 2 —Writing into kinesis using python</h4><p>Now that we have kinesis set up, let us try and write data into the streams.</p><p>Let us keep it simple. Randomly generate some data in the form of dictionaries, convert them to json and send the data to kinesis.</p><p>First let us import the libraries</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a51d4b6e08ebac801f82d0e43e9e299/href">https://medium.com/media/4a51d4b6e08ebac801f82d0e43e9e299/href</a></iframe><p>Now we need to create the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee001696af8402330ba2b4d559ed01b7/href">https://medium.com/media/ee001696af8402330ba2b4d559ed01b7/href</a></iframe><p>The code snippet above shows how data dictionary can be created randomly and then converted into json. We need to convert the dictionary into json as this is necessary to put the data in the kinesis data stream.</p><p>Now that we are generating continuous stream of random data, let us put it in the stream.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9cc92399fd70fd3f1528cf31c3381ddd/href">https://medium.com/media/9cc92399fd70fd3f1528cf31c3381ddd/href</a></iframe><p>This line of code will be inside the while loop and called for each data point that is being created.</p><p>Here is the code in action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*xeMZ6mIc5FEo5ulS0XbbdQ.gif" /><figcaption>Random data being generated and put inside the kinesis data stream</figcaption></figure><p>Here is the entire code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c465d56c89009c45494c541dc5daaa46/href">https://medium.com/media/c465d56c89009c45494c541dc5daaa46/href</a></iframe><p>You can find the code base in Github <a href="https://github.com/ashhadulislam/GarudaHacks/blob/master/test_codes/kinesis_py_code/write_kine_random.py">here</a>. If you are looking for a little more convoluted example you can check another <a href="https://github.com/ashhadulislam/GarudaHacks/blob/master/test_codes/kinesis_py_code/write_kine_random.py">example</a> where I simulate three health devices and generate random data here.</p><p>This was all about writing data into kinesis data stream.<br>In the <a href="https://medium.com/@ashhadulislam/aws-kinesis-data-streams-using-python-part-2-a85d325f8abe">next part</a> we’ll talk about how to read the data from kinesis data stream.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=197a37552fe6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/aws-kinesis-data-streams-using-python-part-1-197a37552fe6">AWS Kinesis Data Streams using Python — Part 1</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a song playlist from YouTube comments using NLP]]></title>
            <link>https://medium.com/tech-that-works/create-a-song-playlist-from-youtube-comments-using-nlp-a810a6198bac?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/a810a6198bac</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <dc:creator><![CDATA[Samrudha Kelkar]]></dc:creator>
            <pubDate>Mon, 06 Apr 2020 06:57:00 GMT</pubDate>
            <atom:updated>2020-04-11T20:06:14.132Z</atom:updated>
            <content:encoded><![CDATA[<p>Where do you listen to your music? Mostly on some apps? And what you listen to? Songs of your favorite artists, albums, creating your own playlist. More often than not, you get hooked to powerful recommendations given by all popular music streaming apps.</p><blockquote>What if people take their time and effort to suggest you “songs” that you might like. Yes, What if they <strong>type </strong>those songs for you?</blockquote><p>Here I do not mean creepily tracking their music playlist. So last weekend I came across this <a href="https://www.youtube.com/watch?v=YkFotcZnVQ8&amp;list=RDYkFotcZnVQ8&amp;start_radio=1">YouTube music reactions channel</a> where the YouTuber(<a href="http://shop.depressiontoexpression.com/">his support page</a>) listen to Indian Bollywood songs and react to it. Interestingly, people who love those Indian songs suggest him more songs. Remember these are the heartfelt recommendations users really really want to get featured in reaction videos. And most of these recommendations are simply amazing. <em>So I thought why not use NLP algorithms and extract all these beautiful recommended songs</em>.</p><p>If we can do that, that will be one incredibly awesome playlist to have. Here is my experiment to get the playlist from youtube comments</p><p><strong>A] Get the damn data:</strong> ML on youtube comments is something I always wanted to do. I came across <a href="https://github.com/egbertbouman/youtube-comment-downloader">this </a>repository which does the task of using <em>googleapi </em>for you and gives you a nice new line delimited output file.</p><p>Data looks something like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BwlQOBHZjGyT7KDgkfrhBg.png" /></figure><p><strong>B] Let’s do some NLP:</strong> NLP tasks mainly involve tokenizing the comments, entity extractions, and POS tagging. We can create the pattern of those comments using the output of the above pipeline.</p><p>I explored Stanza, a cool new NLP library in the town. I was thinking of using it for a custom NER pipeline where my entities will be “song” and “artist”. But I found the annotating is a pretty tough job here since most of the comments are in <strong>Hinglish, </strong>(a mixture of Hindi-English). Also frankly, the documentation of training your own NER for stanza is not that great yet. The data directory structure is hard to crack and not worth the pain for my tiny little data set (2100 total records)</p><p>Interestingly, Spacy has open-sourced <a href="https://github.com/explosion/spacy-stanza">spacy-stanza</a>. It gives the capability of doing familiar spacy stuff while using stanza models under the wrapper. You have to just import the stanza pipeline.</p><pre>import stanza<br>from spacy_stanza import StanzaLanguage<br><br>snlp = stanza.Pipeline(lang=&quot;en&quot;)<br>nlp = StanzaLanguage(snlp)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4QT0XKi5NBx2RLxfxH-WDg.png" /></figure><p><strong>c] Algo: </strong>The output from POS tagger and resolved entities can then be used as features to run your song extraction algorithm. Here I used spacy’s inbuilt matcher which allows you to create rules which run on the spacy doc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8501c605610e0576868ea0ab700db5af/href">https://medium.com/media/8501c605610e0576868ea0ab700db5af/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*uJ9zfh8Jsevqdnk7ORamwQ.png" /></figure><p>We see that multiple matches can be found but they share the same match_id. that’s the beauty of Spacy. Simple string matching might solve this task, but using a spacy doc and matcher let us filter duplicates easily using match_id. I kept the longer spans of detected texts in filtering duplicates.</p><p>Interested in getting the recommended singers list? We can always use the tag <em>PERSON </em>of the entity tagger to get all the singers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdeb1827fe31c67b756fbf7f0aef4428/href">https://medium.com/media/bdeb1827fe31c67b756fbf7f0aef4428/href</a></iframe><p><strong>Results:</strong></p><p>The video that I used to extract comments was by the popular Bollywood Indian Music singer <em>Arjit Singh. Here</em> is the list of the few suggested songs that I could extract :</p><blockquote>Ayat by arijit singh, Chaap Tilak by Rahat Fateh<br>Sau dard hai by Sonu nigam, Ayat by arijit<br>aankhon ke Sagar by shafqat amanat<br>song Kalank, teri mitti by B praak<br>song Mai Rahoon, Tum ko, Ek dil, jaage Hain by ar Rahman<br>Sang Dhol Baje by Shreya Ghoshal<br>teri mitti’ by B, HARAM BY ATIF ASLAM, Ayaat song by arijit<br>KAHI by SONU NIGAM<br>AAYAT by Arjit Singh<br>OwnMusic song by Deep Chaubey<br>Teri mitti by B Praak<br>Kastoori’ By Indian Ocean</blockquote><p>Most of these songs are by Arjit Singh. Others look really close to the song in the video in terms of music quality, melody, genre, etc. Totally I could extract <em>255 good songs </em>(with few duplicates) from close to 2k comments!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a810a6198bac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/create-a-song-playlist-from-youtube-comments-using-nlp-a810a6198bac">Create a song playlist from YouTube comments using NLP</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wilson Lower bound Score and Bayesian Approximation for K star scale rating to Rate products]]></title>
            <link>https://medium.com/tech-that-works/wilson-lower-bound-score-and-bayesian-approximation-for-k-star-scale-rating-to-rate-products-c67ec6e30060?source=rss----f3f3b94712c0---4</link>
            <guid isPermaLink="false">https://medium.com/p/c67ec6e30060</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[rankings]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Aditya Kumar]]></dc:creator>
            <pubDate>Thu, 09 Jan 2020 07:47:29 GMT</pubDate>
            <atom:updated>2020-01-27T18:18:41.350Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G1b-Lb9me1oNgp_B" /><figcaption>Photo by <a href="https://unsplash.com/@charlesdeluvio?utm_source=medium&amp;utm_medium=referral">Charles</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a maintainer of an online community, which is having a lot of products where user gives a rating to products based on their experience, then it is definite that at some point you have to find an answer to questions like, how you are going to show the product on the page based on filters i.e. like highest voted or lowest voted, etc.? Or How can you rate a product based on upvotes and downvotes? How you can give a score to a product which is rated on a K scale by users?</p><p>There are some ways you can find a score and rate products:</p><ol><li>Score = Average rating of products</li><li>Score = Positive rating - Negative rating</li><li>Score = Proportion of Positive ratings</li></ol><p>Evan Miller’s famous blog <a href="http://www.evanmiller.org/how-not-to-sort-by-average-rating.html">How not to sort</a> explains why the above two scores are not good ways to rate the product or sort a product.</p><p>Lower bound of <strong>Wilson score confidence interval for a Bernoulli parameter </strong>provides a way to sort a product based on positive and negative ratings.</p><p>The idea here is to treat the existing set of user ratings as a statistical sampling of a hypothetical set of user ratings from all users and then use this score. In other words, what user community would think about upvoting a product with 95% confidence given that we have an existing rating for this product with a sample (subset from the whole community) user ratings.</p><p>Therefore if we know what a sample population thinks i.e. user reviews for a product, you can use this to estimate the preferences of the whole community.</p><p>If there are X positive votes and Y negative votes for a product and we want to understand how popular the product will be across the whole community. We can estimate that with 95% confidence between wilson_lower_bound_score and wilson_upper_bound_score% of users will upvote this product using the Wilson Score of Confidence interval.</p><h3>Wilson Score</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*a55XGo_ZIv6lFGn23nCQeQ.png" /><figcaption>Wilson Lower Bound Score</figcaption></figure><pre>where p̂ ​= (# of positive ratings)/(Total ratings)<br>      n = Total ratings<br>      Z_(α/2)=  Quantile of the standard normal distribution</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af400d3fd0aa09010cb4ecb977141497/href">https://medium.com/media/af400d3fd0aa09010cb4ecb977141497/href</a></iframe><p>Wilson Confidence Interval considers binomial distribution for score calculation i.e. it considers only positive and negative ratings. If your product is rated on a 5 scale rating, then we can convert ratings {1–3} into negative and {4,5} to positive rating and can calculate Wilson score.</p><p>Here are some examples:</p><ol><li>If a product is rated across each category uniformly [10, 10, 10, 10, 10], i.e. 10 votes for rating {1–5}, then wilson_lower_bound(20,50,.95), avg_rating([10, 10, 10, 10, 10]) =&gt; (0.2760838973025655, 3.0)</li><li>A product receives only one rating i.e. positive and one product receives 10 positive and 2 negative ratings: in that case value of product having more ratings should be greater wilson_lower_bound(1,1,0.95) &lt; wilson_lower_bound(10,12,0.95) , which is true.</li><li>A product having ratings A: (209 up and 50 down votes) and B: (118 up and 25 down) wilson_lower_bound(209,259,0.95) &lt; wilson_lower_bound(118,143,0.95)</li><li>Suppose one product receives [5, 10, 20, 0, 0] ratings, then wilson_lower_bound(0,35,0.95) = 0. If any product does not have any positive ratings associated with it then the Wilson score is zero.</li><li>Wilson Score can not be applied to a new product which is yet to receive any rating. Above implementation gives score equal to zero if there is no rating associated with the product i.e. wilson_lower_bound(0,0,0.95) = 0</li></ol><p>Wilson score gives us the zero value for both the product which does not receive any positive user rating and to the product which is new and yet to receive any rating, which essentially does not make any sense as this implies <strong>no user rated product</strong> is the same as a <strong>product having lower ratings</strong>. Also, it is not clear how tight the lower bound is i.e., how far it deviates away from the “real” proportion of thumb-ups [1]. It does not seem intuitive to convert items rated on the five-star scale to convert to upvotes and downvotes for calculating scores to follow a binomial distribution.</p><p><strong>Bayesian Approximation</strong> provides a way to give a score to a product when they are rated on the K star scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*vJh2ZKQLIC0UmlKsZhPdsg.png" /><figcaption>Bayesian Approximation for K scale rating</figcaption></figure><p>where s_k = k i.e. 1 point, 2 points, ….N point Scale</p><p>N = total ratings with n_k ratings for k scale</p><p>The above expression provides the lower bound of a normal approximation to a Bayesian credible interval for the average rating. For more mathematical details please check [4].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b35904031ddb68b4e7d8ed125275b6f/href">https://medium.com/media/3b35904031ddb68b4e7d8ed125275b6f/href</a></iframe><p>Let’s look at some test samples</p><ol><li>bayesian_rating_products([0, 0, 0, 0, 1]) = 2.2290</li><li>bayesian_rating_products([0, 2, 0, 10, 0]) = 2.9921</li><li>bayesian_rating_products([5, 10, 20, 0, 0]) = 2.2349</li><li>bayesian_rating_products([10, 10, 10, 10, 10]) = 2.6296</li></ol><p>On comparing 1 and 2, we can observe the second product should have a higher score and that is the case here. Also in point 3, unlike Wilson score it provides a score to the product which does not have a positive rating and still that score is greater than the first product which seems reasonable to me.</p><p>Bayesian Approximation does not consider only upvotes unlike Wilson&#39;s score but considers ratings across the K scale and proves to be better in this scenario.</p><p>Please let me know your views and suggestions in the comments.</p><p><strong>References:</strong></p><ol><li><a href="http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2011.pdf">How to Count Thumb-Ups and Thumb-Downs: User-Rating based Ranking of Items from an Axiomatic Perspective</a></li><li><a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36265.pdf">Star Quality: Aggregating Reviews to Rank Products and Merchants</a></li><li><a href="http://www.evanmiller.org/how-not-to-sort-by-average-rating.html">http://www.evanmiller.org/how-not-to-sort-by-average-rating.html</a></li><li><a href="http://www.evanmiller.org/ranking-items-with-star-ratings.html">http://www.evanmiller.org/ranking-items-with-star-ratings.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c67ec6e30060" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-that-works/wilson-lower-bound-score-and-bayesian-approximation-for-k-star-scale-rating-to-rate-products-c67ec6e30060">Wilson Lower bound Score and Bayesian Approximation for K star scale rating to Rate products</a> was originally published in <a href="https://medium.com/tech-that-works">tech-that-works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>