<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinay Kudari on Medium]]></title>
        <description><![CDATA[Stories by Vinay Kudari on Medium]]></description>
        <link>https://medium.com/@vinaykudari?source=rss-3fbe4f6ebe31------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k-yUJOFrDwj7VJApzVaadA.jpeg</url>
            <title>Stories by Vinay Kudari on Medium</title>
            <link>https://medium.com/@vinaykudari?source=rss-3fbe4f6ebe31------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:31:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinaykudari/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build any deep-learning image classifier under 15 lines of code using fastai v2]]></title>
            <link>https://medium.com/data-science/build-any-deep-learning-image-classifier-under-15-lines-of-code-using-fastai-v2-123c81c13b?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/123c81c13b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-recognition]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Mon, 14 Sep 2020 17:31:07 GMT</pubDate>
            <atom:updated>2020-09-19T13:17:11.367Z</atom:updated>
            <content:encoded><![CDATA[<p>Fastai is a high level deep learning library built on top of PyTorch. <a href="https://medium.com/u/34ab754f8c5e">Jeremy Howard</a> has recently introduced a newer version of the library along with a very handy, beginner-friendly <a href="https://github.com/fastai/fastbook">book</a> and a <a href="https://course.fast.ai/">course</a>. I was pretty surprised at its abstraction level, which helps you to create state-of-the-art models in minutes, without having to worry about the math behind it.</p><p>This article was written with total beginners in mind, and you would be able to follow it even if you have little coding experience. After reading this article you will be in a position to write a piece of code from scratch which can identify your favorite superhero or even recognize the animal by the sound it makes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d4-BpeeMiu-BqXcbGFvs3A.png" /><figcaption>Credits: <a href="https://medium.com/u/f4f82be9e332">Vaishnavi Chityala</a></figcaption></figure><p>Here’s a snapshot of the outcome of the mask classifier I’ve trained using little training data, few lines of code, and a few minutes of training on a GCP cluster <a href="https://course.fast.ai/start_gcp.html"><em>Click here to setup your own FastAI GPU VM for *</em><strong><em>free</em></strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*X0kaw32fQPUMgTKuAqCuqg.png" /><figcaption>Try with your own images <a href="https://findmask.ml/">here</a></figcaption></figure><p>In order to achieve this result firstly, we need to find faces in the image also known as <strong>localization</strong>, then <strong>classify</strong> each face and draw a colored bounding box according to the category (<em>green: with_mask, red: no_mask, yellow: mask_worn_improperly</em>) it belongs. Today lets understand the second part of the problem <strong>multi-class</strong> image classification</p><p>Project code is available <a href="http://bit.ly/mask-classifier-nb"><em>here</em></a></p><p>I will explain some basic concepts of deep learning and computer vision while we write some code. I highly recommend you to <strong>run</strong> the code line by line on a jupyter notebook as we understand the ideas behind the abstract functions</p><h3>Processing the data</h3><p>The library is split up into modules, we primarily have tabular, text, and vision. Our problem for today will involve vision so let’s import all the functions which we are gonna need from the vision library</p><pre><strong>In [1]: </strong>from fastai.vision.all import *</pre><p>Just like how we learn to identify objects by observing images our computer to needs data to recognize images. For mask detection I’ve curated a <strong>labeled</strong> dataset collected from <a href="https://www.kaggle.com/andrewmvd/face-mask-detection">kaggle</a> and other sources, you may download it from <a href="http://bit.ly/face-mask-dataset"><em>here</em></a></p><p>We store the path where our dataset resides. Path returns apathlib object which can be used to perform <a href="https://medium.com/swlh/five-most-useful-pathlib-operations-77f9c96790b3">some</a> file operations very easily</p><pre><strong>In [2]:</strong> DATASET_PATH = Path(&#39;/home/kavi/datasets&#39;)</pre><p>Before we <strong>train </strong>our model (teach our algorithm to recognize images) we first need to tell it a few things</p><ul><li>What is the expected input and output? What is the problem domain?</li><li>Where is the data located and how it is labeled?</li><li>How much data do we want to keep it aside to evaluate the performance of the model?</li><li>Do we need to transform the data? If so how?</li></ul><p>Fastai has this super flexible function called DataBlock which take inputs to the above questions and prepares a template</p><pre><strong>In [3]: </strong>mask_datablock = DataBlock(<br>           get_items=get_image_files,<br>           get_y=parent_label,<br>           blocks=(ImageBlock, CategoryBlock), <br>           item_tfms=RandomResizedCrop(224, min_scale=0.3),<br>           splitter=RandomSplitter(valid_pct=0.2, seed=100),<br>           batch_tfms=aug_transforms(mult=2)<br>        )</pre><ul><li>get_image_files function grabs all the image file locations recursively in the given path and returns them, this way we tell fastai where to get_items</li><li>In our dataset I’ve placed the images in separate folders named according to the category, parent_label function returns the parent directory of the file according to the path</li></ul><p><strong><em>eg</em> : </strong>parent_label(&#39;/home/kavi/datasets/with_mask/1.jpg&#39;) =&gt; &#39;with_mask&#39;</p><p>You may write your <a href="https://forums.fast.ai/t/fastai-v2-recipes-tips-and-tricks-wiki/64486">own function</a> according to how your data is labeled</p><ul><li>After knowing the file paths of the input images and target labels we need to <strong>preprocess</strong> the data based on the type of problem. Sample pre-processing steps for images includes creating images from file paths using Pillow and converting them into a tensor</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*7hfZb4b1Lyb9UxHCkYjNXQ.jpeg" /><figcaption>Image is taken from <a href="https://lisaong.github.io/mldds-courseware/01_GettingStarted/numpy-tensor-slicing.slides.html">here</a></figcaption></figure><h4>How an image is represented in our PC?</h4><p>Every image is a matrix of pixel intensities. Each value ranges from 0 –255. 0 being the darkest and 255 being the brightest intensity of respective channel</p><p>A color image is a 3 layered matrix / 3rd order tensor. Each layer comprises <strong>R</strong>ed, <strong>G</strong>reen, <strong>B</strong>lue intensities whereas a black and white picture is a 1-D matrix</p><p>We pass a tuple (input_transformation, output_transformation) of type <a href="https://docs.fast.ai/vision.data#TransformBlocks-for-vision">TransformBlock</a> to blocks In our problem we need to predict a category for an image hence we pass (ImageBlock, CategoryBlock) If suppose you want to predict the age of a person based on his picture you would need to pass (ImageBlock, RegressionBlock)</p><ul><li>Deep learning models work best when all the images are of the same size, also they learn quicker when the resolution is lower. We will tell how to rescale the images by passing a suitable resize function to item_tfms and this function will be applied to each image</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*-ADRO6xAXMczdiExQSBotw.png" /><figcaption>Image resized with ResizeMethod.Squish, ResizeMethod.Pad and ResizeMethod.Crop methods</figcaption></figure><p>Fastai provides various resizing methods: Crop, Pad or Squish. Each of them is somewhat problematic. Sometimes we might even end up losing some critical information just like in the third image after the center crop</p><p>We can use RandomResizedCrop to overcome this issue. Here we randomly select a part of the picture and train the model for a couple of <strong>epochs</strong> (One complete pass through all the images in the dataset) covering most regions of every picture. min_scale determines how much of the image to select at minimum each time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*lCBKcw7A8EoyIdCiltu07A.png" /><figcaption>Image resized using RandomResizedCrop(224, min_scale=0.3)</figcaption></figure><ul><li>A model accuracy assessment on the training dataset would result in a biassed score and may lead to poor performance on unseen data. We need to tell our DataBlock API to set aside some of the pre-processed data to evaluate the performance of the model. The data which the algorithm sees is called <strong>training</strong> data and the data kept aside is called <strong>validation</strong> data. Often the datasets will have validation set defined, but in our case we don&#39;t have one so we need to pass a split function to splitter</li></ul><p>Fastai has a <a href="https://docs.fast.ai/data.transforms#Split">couple</a> of split functions, lets use RandomSplitter for today’s problem valid_pct will determine what fraction of training data needs to <strong>set aside</strong> and seed will make sure that the same random images are set aside always</p><ul><li>Having a diverse data set is crucial to the performance of any deep learning model. So what do you do if you don’t have the appropriate amount of data? We generate new data based on existing data this process is called <strong>data augmentation</strong></li></ul><blockquote><strong><em>Data augmentation</em></strong> <em>refers</em> to creating random variations of our input data, such that they appear different, but do not actually change the meaning of the data.<strong> <em>— Fastbook</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*qePL7mj2ja1cnCMczlaAkg.png" /><figcaption>Single teddy image augmented using aug_transforms()</figcaption></figure><p>The above images are generated from a single picture of teddy. In the real world, we often have to make predictions on unseen data, the model will perform poorly if it just <strong>memorizes</strong> the training data (<strong>overfitting</strong>), rather it should understand the data. Augmenting the data is proven helpful in improving the performance of the models in many cases</p><p>In fastai we have a predefined function aug_transforms which performs some default image transforms such as flipping, altering the brightness, skewing, and few <a href="https://docs.fast.ai/vision.augment#aug_transforms">others</a>. We pass this function to batch_tfms and the interesting thing to note is that these transforms are performed on a GPU (if available)</p><p>DataBlock holds the list instructions which will be performed on our dataset. This acts as a blueprint to create a DataLoader which takes our dataset path and apply the pre-processing transforms on the images as defined by the DataBlock object and load them into a GPU. After loading the data batch_tfms will be applied on the <strong>batch</strong>. The default batch size is 64, you may increase/decrease based on your GPU memory by passing bs=n to dataloders function</p><blockquote><strong>Tip:</strong> <strong>!nvidia-smi</strong> command can be executed in your notebook anytime to know your GPU usage details. You may restart the kernel to free the memory</blockquote><pre><strong>In[10]:</strong> dls = mask_datablock.dataloaders(DATASET_PATH)</pre><p>dls is a dataloader object which holds training and validation data. You may have a look at our transformed data using dls.show_batch()</p><h3>Training our mask-classifier model</h3><p>A model is a set of values also known as <strong>weights</strong> that can be used to identify patterns. Pattern recognition is everywhere, in living beings it&#39;s a cognitive process that happens in the brain without us being consciously aware of it. Much as how we learn to recognize colors, objects, and letters as a child by staring at them. Training a model is determining the <strong>right</strong> set of weights which can solve a <strong>particular</strong> problem, in our case to classify an image into 3 categories: <em>with_mask, without_mask, mask_weared_incorrect</em></p><h4>How to train models quickly?</h4><p>As a grown-up adult, we can learn to recognize objects almost instantly this is because we have been <strong>learning</strong> patterns all the way since we were born. Initially, we have learned to recognize colors, then simple objects like balls, flowers. Later after few years, we were able to recognize persons and other complex objects. Similarly in machine learning, we have pre-trained models that have been trained to solve a <strong>similar problem</strong> and can be modified to solve our problem</p><blockquote>Using a pretrained model for a task different to what it was originally trained for is known as <strong><em>transfer learning</em></strong><em> — </em><strong><em>Fastbook</em></strong></blockquote><p>resnet34 is one such model trained on the <strong>ImageNet </strong>dataset which contains around 1.3 million images and can classify into 1000’s of image categories. In our problem, we have only 3 categories so we use the initial layers that are useful to recognize basic patterns such as lines, corners, simple shapes and then retrain on the final layers</p><pre><strong>In[11]:</strong> learn = cnn_learner(dls, resnet34, metrics=error_rate)</pre><p>Fastai offers a cnn_learner function that is particularly useful in the training of computer vision models. Which takes DataLoader object dls, pre-trained model resnet34 (here 34 mean we have 34 layers), and a metric error_rate which calculates the percentage of images that are <strong>classified incorrectly</strong> on <strong>validation</strong> data</p><blockquote>A <strong><em>metric</em></strong> is a function that measures the quality of the model’s predictions using the <strong>validation set <em>— Fastbook</em></strong></blockquote><h4>How does transfer learning work?</h4><p>Initially, we replace the final layer of our pretrained model with one or more new layers with randomized weights this part is known as <strong>head. </strong>We update the weights of the <strong>head </strong>using the <strong>back-propagation </strong>algorithm which we will learn in a different article</p><p>Fastai provides a method fine_tune which performs the task of tuning the pre-trained model to solve our specific problem using the data we have curated</p><pre><strong>In[12]:</strong> learn.fine_tune(4)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*qB8CaMNaE8mjTW9EsHrsjA.png" /><figcaption>Output of the fine_tune function</figcaption></figure><p>We pass a number to fine_tune which tells how many <strong>epochs</strong> (the number of times we fully go through the dataset) you need to train. This is something you need to play with, there is no hard and fast rule. It depends on your problem, dataset, and the time you want to spend on training. You may run the function multiple times with different epochs</p><p>Tips from Jeremy and my learnings:</p><ul><li>Training for large no of epochs can lead to <strong>overfitting</strong>, which may lead to poor performance on unseen data. If the validation loss continues to increase in consecutive epochs it means that our model is memorizing the training data and we need to stop training</li><li>We can improve performance by training the model with <strong>different resolutions</strong>. Eg. Train with 224x224 and later with 112x112 pixel images</li><li>Data <strong>augmentation</strong> will help prevent overfitting to some extent</li></ul><h3>Using the Model for Inference</h3><p>Now we have our trained mask classifier which can classify whether a person is wearing a mask properly, improperly, or not wearing a mask from the picture of his face. We can export this model and use it to predict the class elsewhere</p><pre><strong>In[13]:</strong> learn.export()<br><strong>In[14]:</strong> learn.predict(&#39;path/to/your/image.jpg&#39;)</pre><p>We can load this model using load_learner(&#39;path/to/model/export.pkl&#39;)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*4zfZ10u7InzDGmmEO5c3iw.png" /><figcaption>Screenshot of web app</figcaption></figure><p>I went on to make a REST API exposing this model to the internet and with the help of my friends: <a href="https://www.behance.net/chityalava5022">Vaishnavi</a> and <a href="https://www.linkedin.com/in/jaswanth-sai-sattenapalli-b74766128/">Jaswanth</a>, made a web application which takes an input image and draws bounding boxes according to the category the face belongs and also the count of face categories. Please feel free to drop your feedback there which will help improve the model. The web app is live at <a href="https://findmask.ml/"><strong>https://findmask.ml</strong></a></p><p>Article on how did I build a Deep Learning REST API is coming soon :)</p><h3>Conclusion</h3><p>Now you are in spot to build your image classifier, you may also use this technique to classify sounds by converting them into spectrograms or any suitable image form. If you are starting to learn deep-learning I highly recommend you to take the fastai v2 <a href="https://course.fast.ai/">course</a> and join the <a href="https://www.meetup.com/Machine-Learning-Tokyo/events/273130729/">study group</a> organised by <a href="https://medium.com/u/a08b1e531f9c">Sanyam Bhutani</a> and <a href="https://medium.com/u/144e88eb73a2">MLT</a> where we read and discuss each chapter of <a href="https://github.com/fastai/fastbook">Fastbook</a> everyweek.</p><p>Feel free to reach me for any feedback or if you are facing any issue, happy to help. I’m active at <a href="https://twitter.com/kudarivinay?lang=en">Twitter</a> and <a href="https://www.linkedin.com/in/vinaykudari/">LinkedIn</a></p><h4>Related Interesting Articles</h4><ul><li><a href="https://muellerzr.github.io/fastblog/">https://muellerzr.github.io/fastblog/</a></li><li><a href="https://towardsdatascience.com/10-new-things-i-learnt-from-fast-ai-v3-4d79c1f07e33">https://towardsdatascience.com/10-new-things-i-learnt-from-fast-ai-v3-4d79c1f07e33</a></li></ul><h4>My Other Stories</h4><ul><li><a href="https://towardsdatascience.com/sqlalchemy-python-tutorial-79a577141a91">SQLAlchemy — Python Tutorial</a></li><li><a href="https://medium.com/@vinaykudari/protecting-your-internet-identity-4d8ef368caa7">How to protect your internet identity?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=123c81c13b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/build-any-deep-learning-image-classifier-under-15-lines-of-code-using-fastai-v2-123c81c13b">Build any deep-learning image classifier under 15 lines of code using fastai v2</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Protecting Your Internet Identity]]></title>
            <link>https://medium.com/@vinaykudari/protecting-your-internet-identity-4d8ef368caa7?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d8ef368caa7</guid>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[vpn]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Wed, 05 Sep 2018 18:33:08 GMT</pubDate>
            <atom:updated>2018-09-05T18:33:08.729Z</atom:updated>
            <content:encoded><![CDATA[<p>Nothing in this world comes free, If you think the internet giants are providing you with free services then you are wrong. You are actually paying more than what you expected to pay for ‘those’ services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ioMKgzbq3E4P6hDItydYdw.jpeg" /></figure><p>Let’s look at the 10 simple tweaks you could implement to protect your privacy to maximum extent while you are browsing the web.</p><ol><li><strong>Free VPN service </strong>is a myth it boasts of giving you internet privacy but it actually works by selling your data. If you really care about your privacy you could use <a href="https://www.torproject.org/"><strong>Tor</strong></a><strong> </strong>which is free and most secure VPN till date or simply <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-16-04"><strong>setup</strong></a> your own VPN service in a remote server.</li><li>Never leave <strong>DNS settings to automatic</strong>, doing this your ISP can trace all your web history even when you are in incognito mode. DNS is a sort of “phone book” that translates the domain you enter (www.google.com) in your browser into a computer readable IP. Instead <a href="https://gizmodo.com/yes-you-should-still-change-your-dns-settings-for-bett-1828216684"><strong>set your DNS settings</strong></a> manually to privacy first DNS servers like Clouldflare, OpenDNS.</li><li>Use a opensource Ad Blocker like <a href="https://www.ublock.org/"><strong>uBlock</strong></a><strong> </strong>which blocks unwanted ads, trackers and also speeds up your browsing experience. If you can spend some and money you could <a href="https://pimylifeup.com/raspberry-pi-pi-hole/"><strong>set up pihole</strong></a><strong> </strong>(opensource network wide ad blocker) on a Raspberry Pi which can block almost all the advertisements all over the network. You might me wondering what trackers are, did you ever notice that the items you had browsed before keep on following you via social media? It is the work of internet trackers.</li><li>Internet trackers work by seeding your internet browser with something called a cookie, which keep on tracking the websites you visit and selling the data to ecommerce companies like Amazon, Flipkart. You can <a href="https://www.howtogeek.com/241006/how-to-block-third-party-cookies-in-every-web-browser/"><strong>block 3rd party cookies</strong></a> on your browsers to avoid tracking to some extent.</li><li>It is better to switch to more privacy oriented search engine and browser like <a href="https://duckduckgo.com/"><strong>DuckDuckGo</strong></a><strong> and </strong><a href="https://www.mozilla.org/en-US/firefox/new/"><strong>Firefox</strong></a><strong> </strong>respectively and always try to use <a href="https://opensource.com/resources/what-open-source"><strong>opensource</strong></a> software&#39;s which are built by public.</li><li>Maintain separate browsers, like one for financial transactions other for watching videos or normal surfing over the web. You can also <a href="https://www.veracode.com/blog/2013/03/browser-security-settings-for-chrome-firefox-and-internet-explorer"><strong>tweak some of browser settings</strong></a> to improve security. Try to avoid using public Wi-Fi especially when doing internet transactions.</li><li>Try to <strong>avoid using oAuth : Sign up via Facebook/Google </strong>and use <a href="https://searchsecurity.techtarget.com/definition/two-factor-authentication"><strong>Two Factor authentication</strong></a> where ever possible<strong>, </strong>It is really convenient sometimes but you are actually giving up more information than required, notice what information is getting shared with the party.</li><li>Look out spam/malware links you would generally come across in mails or social media, Never <strong>click on a link</strong> you did not expect to receive even though it looks appealing, also limit your social media audience to the most possible extent. Always check for <strong>HTTPS</strong> in the URL bar, HTTPS indicates that the web page you are on is secure and can be trusted .</li><li>If you are using google account make sure you<strong> </strong><a href="https://www.eff.org/deeplinks/2015/11/guide-google-account-privacy-settings-students"><strong>turn off your browsing history</strong></a><strong> </strong>also<strong> </strong><a href="https://myaccount.google.com/shared-endorsements?pli=1"><strong>turn off shared endorsements settings</strong></a><strong> </strong>which could leak your personal data like your picture to advertising companies.</li><li>Be aware of your activity in social media sometimes those fun quizzes can squeeze your personal information like those you have <strong>answered in security questions</strong> during any signup.</li></ol><p>Thanks for reading, please do share 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d8ef368caa7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQLAlchemy — Python Tutorial]]></title>
            <link>https://medium.com/data-science/sqlalchemy-python-tutorial-79a577141a91?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/79a577141a91</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[sqlalchemy]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Thu, 23 Aug 2018 14:46:10 GMT</pubDate>
            <atom:updated>2020-09-15T15:40:39.780Z</atom:updated>
            <content:encoded><![CDATA[<p>We often encounter data as Relational Databases. To work with them we generally would need to write raw SQL queries, pass them to the database engine and parse the returned results as a normal array of records.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*D1TpfUof-8amlhN1E2A6JA.png" /></figure><p>SQLAlchemy provides a nice “Pythonic” way of interacting with databases. So rather than dealing with the differences between specific dialects of traditional SQL such as MySQL or PostgreSQL or Oracle, you can leverage the Pythonic framework of SQLAlchemy to streamline your workflow and more efficiently query your data.</p><h4>Other stories on datascience can be found <a href="https://medium.com/hacking-datascience">here</a></h4><h4>Installing The Package</h4><pre>pip install sqlalchemy</pre><h4>Connecting to a database</h4><p>To start interacting with the database we first we need to establish a connection.</p><pre>import sqlalchemy as db<br>engine = db.create_engine(&#39;dialect+driver://user:pass<strong>@host</strong>:port/db&#39;)</pre><p>Some examples of connecting to various databases can be found <a href="http://docs.sqlalchemy.org/en/latest/core/engines.html#postgresql"><strong>here</strong></a></p><h4>Viewing Table Details</h4><p>SQLAlchemy can be used to automatically load tables from a database using something called reflection. Reflection is the process of reading the database and building the metadata based on that information.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1596c2dcabfd778f53716b57049399d/href">https://medium.com/media/d1596c2dcabfd778f53716b57049399d/href</a></iframe><h4>Querying</h4><p>Table and MetaData have already been imported. The metadata is available as metadata.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6aeec7df852e29063adf0d33f23d3a9/href">https://medium.com/media/a6aeec7df852e29063adf0d33f23d3a9/href</a></iframe><p>ResultProxy:<em> The object returned by the </em><em>.execute() method. It can be used in a variety of ways to get the data returned by the query.</em></p><p>ResultSet: <em>The actual data asked for in the query when using a fetch method such as </em><em>.fetchall() on a ResultProxy.</em></p><p><strong>Dealing with Large ResultSet</strong></p><p>We use .fetchmany() to load optimal no of rows and overcome memory issues in case of large datasets</p><pre>while flag:<br>    partial_results = ResultProxy.fetchmany(50)<br>    if(partial_results == []): <br>	flag = False<br>    //<br>	code<br>   //<br>ResultProxy.close()</pre><p><strong>Convert to dataframe</strong></p><pre>df = pd.DataFrame(ResultSet)<br>df.columns = ResultSet[0].keys()</pre><p><strong>Filtering data</strong></p><p>Lets see some examples of raw SQLite Queries and queries using SQLAlchemy.</p><p><strong>where</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census <br>WHERE sex = F</pre><pre><strong>SQLAlchemy :<br></strong>db.select([census]).<em>where</em>(census.columns.sex == &#39;F&#39;)</pre><p><strong>in</strong></p><pre><strong>SQL :</strong><br>SELECT state, sex<br>FROM census<br>WHERE state IN (Texas, New York)</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census.columns.state, census.columns.sex]).where(census.columns.state.<em>in_</em>([&#39;Texas&#39;, &#39;New York&#39;]))</pre><p><strong>and, or, not</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census<br>WHERE state = &#39;California&#39; AND NOT sex = &#39;M&#39;</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census]).where(db.<em>and_</em>(census.columns.state == &#39;California&#39;, census.columns.sex != &#39;M&#39;))</pre><p><strong>order by</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census<br>ORDER BY State DESC, pop2000</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census]).<em>order_by</em>(db.desc(census.columns.state), census.columns.pop2000)</pre><p><strong>functions</strong></p><pre><strong>SQL :</strong><br>SELECT SUM(pop2008)<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([db.<em>func.sum</em>(census.columns.pop2008)])</pre><p>other functions include <em>avg, count, min, max</em>…</p><p><strong>group by</strong></p><pre><strong>SQL :</strong><br>SELECT SUM(pop2008) as pop2008, sex<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([db.func.sum(census.columns.pop2008).label(&#39;pop2008&#39;), census.columns.sex]).<em>group_by</em>(census.columns.sex)</pre><p><strong>distinct</strong></p><pre><strong>SQL :</strong><br>SELECT DISTINCT state<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census.columns.state.<em>distinct</em>()])</pre><p><strong>case &amp; cast</strong></p><p>The case() expression accepts a list of conditions to match and the column to return if the condition matches, followed by an else_ if none of the conditions match.</p><p>cast() function to convert an expression to a particular type</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f38ad030d8ac051b01d25bc7748ec61/href">https://medium.com/media/1f38ad030d8ac051b01d25bc7748ec61/href</a></iframe><p>We use .scalar to the result when the result contains only single value</p><p><strong>joins</strong></p><p>If you have two tables that already have an established relationship, you can automatically use that relationship by just adding the columns we want from each table to the select statement.</p><pre>select([census.columns.pop2008, state_fact.columns.abbreviation])</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9544fd229798f505a9a389d0f3c5b2ff/href">https://medium.com/media/9544fd229798f505a9a389d0f3c5b2ff/href</a></iframe><h4>Creating and Inserting Data into Tables</h4><p>By passing the database which is not present, to the engine then sqlalchemy automatically creates a new database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fba55fcaa5cb24716ce7adfb8bd44ff/href">https://medium.com/media/4fba55fcaa5cb24716ce7adfb8bd44ff/href</a></iframe><h4>Updating data in Databases</h4><pre>db.update(table_name).values(attribute = new_value).where(condition)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5475d8dded9400c47473ead446276a9c/href">https://medium.com/media/5475d8dded9400c47473ead446276a9c/href</a></iframe><h4>Delete Table</h4><pre>db.delete(table_name).where(condition)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d73cbe89f9ebaf7b686318db8a49b8d/href">https://medium.com/media/0d73cbe89f9ebaf7b686318db8a49b8d/href</a></iframe><h4>Dropping a Table</h4><pre>table_name.drop(engine) #drops a single table</pre><pre>metadata.drop_all(engine) #drops all the tables in the database</pre><p>IPython Notebook’s and other assets of this story can be found <a href="https://github.com/vinaykudari/hacking-datascience/tree/master/notebooks/sqlalchemy"><strong>here</strong></a></p><p>See ya again 😃</p><p><strong>My other stories</strong></p><ul><li><a href="https://towardsdatascience.com/build-any-deep-learning-image-classifier-under-15-lines-of-code-using-fastai-v2-123c81c13b">Build any deep-learning classifier under 15 lines of code</a></li><li><a href="https://medium.com/@vinaykudari/protecting-your-internet-identity-4d8ef368caa7">How to protect your internet identity?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79a577141a91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sqlalchemy-python-tutorial-79a577141a91">SQLAlchemy — Python Tutorial</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQLAlchemy — Python Tutorial]]></title>
            <link>https://medium.com/hacking-datascience/sqlalchemy-python-tutorial-abcc2ec77b57?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/abcc2ec77b57</guid>
            <category><![CDATA[sqlalchemy]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Thu, 23 Aug 2018 02:13:26 GMT</pubDate>
            <atom:updated>2018-08-23T02:59:32.659Z</atom:updated>
            <content:encoded><![CDATA[<p>We often encounter data as Relational Databases. To work with them we generally would need to write raw SQL queries, pass them to the database engine and parse the returned results as a normal array of records.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*D1TpfUof-8amlhN1E2A6JA.png" /></figure><p>SQLAlchemy provides a nice “Pythonic” way of interacting with databases. So rather than dealing with the differences between specific dialects of traditional SQL such as MySQL or PostgreSQL or Oracle, you can leverage the Pythonic framework of SQLAlchemy to streamline your workflow and more efficiently query your data.</p><h4>Installing The Package</h4><pre>pip install sqlalchemy</pre><h4><strong>Connecting to a database</strong></h4><p>To start interacting with the database we first we need to establish a connection.</p><pre>import sqlalchemy as db<br>engine = db.create_engine(&#39;dialect+driver://user:pass<strong>@host</strong>:port/db&#39;)</pre><p>Some examples of connecting to various databases can be found <a href="http://docs.sqlalchemy.org/en/latest/core/engines.html#postgresql"><strong>here</strong></a></p><h4>Viewing Table Details</h4><p>SQLAlchemy can be used to automatically load tables from a database using something called reflection. Reflection is the process of reading the database and building the metadata based on that information.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1596c2dcabfd778f53716b57049399d/href">https://medium.com/media/d1596c2dcabfd778f53716b57049399d/href</a></iframe><h4>Querying</h4><p>Table and MetaData have already been imported. The metadata is available as metadata.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6aeec7df852e29063adf0d33f23d3a9/href">https://medium.com/media/a6aeec7df852e29063adf0d33f23d3a9/href</a></iframe><p>ResultProxy:<em> The object returned by the </em><em>.execute() method. It can be used in a variety of ways to get the data returned by the query.</em></p><p>ResultSet: <em>The actual data asked for in the query when using a fetch method such as </em><em>.fetchall() on a ResultProxy.</em></p><p><strong>Dealing with Large ResultSet</strong></p><p>We use .fetchmany() to load optimal no of rows and overcome memory issues in case of large datasets</p><pre>while flag:<br>    partial_results = ResultProxy.fetchmany(50)<br>    if(partial_results == []): <br>	flag = False<br>    //<br>	code<br>   //<br>ResultProxy.close()</pre><p><strong>Convert to dataframe</strong></p><pre>df = pd.DataFrame(ResultSet)<br>df.columns = ResultSet[0].keys()</pre><p><strong>Filtering data</strong></p><p>Lets see some examples of raw SQLite Queries and queries using SQLAlchemy.</p><p><strong>where</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census <br>WHERE sex = F</pre><pre><strong>SQLAlchemy :<br></strong>db.select([census]).<em>where</em>(census.columns.sex == &#39;F&#39;)</pre><p><strong>in</strong></p><pre><strong>SQL :</strong><br>SELECT state, sex<br>FROM census<br>WHERE state IN (Texas, New York)</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census.columns.state, census.columns.sex]).where(census.columns.state.<em>in_</em>([&#39;Texas&#39;, &#39;New York&#39;]))</pre><p><strong>and, or, not</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census<br>WHERE state = &#39;California&#39; AND NOT sex = &#39;M&#39;</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census]).where(db.<em>and_</em>(census.columns.state == &#39;California&#39;, census.columns.sex != &#39;M&#39;))</pre><p><strong>order by</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census<br>ORDER BY State DESC, pop2000</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census]).<em>order_by</em>(db.desc(census.columns.state), census.columns.pop2000)</pre><p><strong>functions</strong></p><pre><strong>SQL :</strong><br>SELECT SUM(pop2008)<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([db.<em>func.sum</em>(census.columns.pop2008)])</pre><p>other functions include <em>avg, count, min, max</em>…</p><p><strong>group by</strong></p><pre><strong>SQL :</strong><br>SELECT SUM(pop2008) as pop2008, sex<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([db.func.sum(census.columns.pop2008).label(&#39;pop2008&#39;), census.columns.sex]).<em>group_by</em>(census.columns.sex)</pre><p><strong>distinct</strong></p><pre><strong>SQL :</strong><br>SELECT DISTINCT state<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census.columns.state.<em>distinct</em>()])</pre><p><strong>case &amp; cast</strong></p><p>The case() expression accepts a list of conditions to match and the column to return if the condition matches, followed by an else_ if none of the conditions match.</p><p>cast() function to convert an expression to a particular type</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f38ad030d8ac051b01d25bc7748ec61/href">https://medium.com/media/1f38ad030d8ac051b01d25bc7748ec61/href</a></iframe><p>We use .scalar to the result when the result contains only single value</p><p><strong>joins</strong></p><p>If you have two tables that already have an established relationship, you can automatically use that relationship by just adding the columns we want from each table to the select statement.</p><pre>select([census.columns.pop2008, state_fact.columns.abbreviation])</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9544fd229798f505a9a389d0f3c5b2ff/href">https://medium.com/media/9544fd229798f505a9a389d0f3c5b2ff/href</a></iframe><h4>Creating and Inserting Data into Tables</h4><p>By passing the database which is not present, to the engine then sqlalchemy automatically creates a new database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fba55fcaa5cb24716ce7adfb8bd44ff/href">https://medium.com/media/4fba55fcaa5cb24716ce7adfb8bd44ff/href</a></iframe><h4>Updating data in Databases</h4><pre>db.update(table_name).values(attribute = new_value).where(condition)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5475d8dded9400c47473ead446276a9c/href">https://medium.com/media/5475d8dded9400c47473ead446276a9c/href</a></iframe><h4>Delete Table</h4><pre>db.delete(table_name).where(condition)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d73cbe89f9ebaf7b686318db8a49b8d/href">https://medium.com/media/0d73cbe89f9ebaf7b686318db8a49b8d/href</a></iframe><h4>Dropping a Table</h4><pre>table_name.drop(engine) #drops a single table</pre><pre>metadata.drop_all(engine) #drops all the tables in the database</pre><p>IPython Notebook’s and other assets of this story can be found <a href="https://github.com/vinaykudari/hacking-datascience/tree/master/notebooks/sqlalchemy"><strong>here</strong></a></p><p>See ya again 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abcc2ec77b57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/sqlalchemy-python-tutorial-abcc2ec77b57">SQLAlchemy — Python Tutorial</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Probability and Statistics]]></title>
            <link>https://medium.com/hacking-datascience/introduction-to-probability-and-statistics-9c8ed6f6a6bc?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c8ed6f6a6bc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Mon, 06 Aug 2018 07:56:15 GMT</pubDate>
            <atom:updated>2018-08-06T09:39:58.765Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*Uu7Bc_2baQ6WojJkRreukw.png" /></figure><p>Why Probability and Statistics?</p><p>We live in a very uncertain world, we are unsure about our next minute. In this kind of world probability and statistics provide a rational way to deal with uncertainty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*Rsws0afjeqJ5e7fzmMJaTg.png" /><figcaption>The fastest route may not be the shortest.</figcaption></figure><p>It is certain to find the fastest route between point a and point b, it can be found my measuring the distance between them, while it is uncertain to find the fastest route as it depends on various factors such as traffic at that moment, time of the day and also the day.</p><p><strong>Probability Theory</strong></p><p>We all know that the probability of getting head and tail are equal to 1/2, So what does this actually mean? If suppose we toss a coin 1000 times we I’ll have the heads approximately <strong>about</strong> 500 times. Let me simulate this using python <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/statistics-for-ds/Intro%20to%20Prob%20and%20Stats.ipynb"><strong>code</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*h7AnOu1CNhVhEQcRO1U6_w.png" /><figcaption>histogram of generate_counts()</figcaption></figure><p>Head is counted as +1 and tail as -1 so if three times head and 2 times tail showed up then the sum is 1, if the no of tail and heads are same, the sum is o, from the histogram we can say that sum is ~0 for maximum no of times.</p><pre>                  sum = x1 + x2 + x3 + ...... xk</pre><p>After doing some stimulations of summing k numbers, <strong>x</strong>i = 1 with probability 1/2 and <strong>x</strong>i = -1 with probability 1/2 we see that the sum is almost always in the range <strong>[-4√k, +4√k] </strong>as k tends to a large number sum/k tends to 0.</p><p><strong>What is Statistics?</strong></p><p>Statistics is about analyzing the real-world data and drawing conclusions. For example if you would like to predict which party is going to win elections one way is to ask all potential voters about their opinion, but it would be extremely expensive instead we could select random unbiased group and take their opinions, now this exactly equivalent to coin problem.</p><p><strong>Frequentist Statistics</strong> draw conclusions from repeating the event multiple number of times under same conditions and comparing the frequency of each outcome, just like flipping a coin. The result entirely depends on the number of times the event is repeated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*ViNoy99m6Y0q3xtYbXho9w.png" /></figure><p><strong>Bayesian Statistics</strong> comes into the play where we cannot perform multiple experiments under same set of conditions, like in diagnosing the patient. He might have unique set of medical conditions which the doctor might not have encountered in the past.</p><p>Homework Notebook can be found <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/statistics-for-ds/HW/HW_Topic1.ipynb"><strong>here</strong></a>. 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c8ed6f6a6bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/introduction-to-probability-and-statistics-9c8ed6f6a6bc">Introduction to Probability and Statistics</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[hacking-pandas]]></title>
            <link>https://medium.com/hacking-datascience/hacking-pandas-4af7f10e3a31?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/4af7f10e3a31</guid>
            <category><![CDATA[group-by]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[melt]]></category>
            <category><![CDATA[pivot]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Fri, 27 Jul 2018 22:06:17 GMT</pubDate>
            <atom:updated>2018-08-07T21:12:33.332Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is a powerful python package which makes importing, cleaning, analyzing, and exporting the data easier. In this story I’d like to write down some of the key functions which I found them useful.</p><p>Will keep on appending new stuff to the story.</p><p><strong>#Update 08/08/18 : Categorical Data, Grouping.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*6WvQ6gUxIRNMv65AobkB-Q.png" /></figure><h4>Importing the data</h4><p>The first part of any data science problem is to import data, now lets see how few hacks can make importing much easier and cleaner.</p><p><strong>syntax</strong></p><pre>pd.read_csv(file_name,argument1, argument2,...)</pre><p><strong>key arguments</strong></p><p>delimiter = character : <strong>delimiter</strong> is a sequence of one or more characters used to specify the boundary between separate, independent regions. The <strong>character</strong> can be &#39; &#39; or &#39;,&#39; or a tab delimiter &#39;\t&#39; .</p><p>header = n : header implies the start of the data, If header argument is not passed it is assumed to take the 0th row, therefore column names are inferred from the first line of the file.</p><p>index_col = n : columns to use as row labels, If a sequence is given, a <em>MultiIndex</em> will be used.</p><p>use_cols = [ ]: list can have column numbers or name, only those mentioned in the list will be selected.</p><p>nrows = n : no of rows that are to be read.</p><p>comment = character : character indicates the beginning of the line which should not be parsed, it can be &#39;#&#39; or &#39;\\&#39;</p><p>na_values = list : additional string list to recognise as NaN value.</p><p>parse_dates = True : parses dates into <strong>yyyy-mm-dd hh:mm:ss</strong> format.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/623c1f4d102358cd528fa83f53b36dfe/href">https://medium.com/media/623c1f4d102358cd528fa83f53b36dfe/href</a></iframe><h4>Cleaning and analyzing the data</h4><p>Data almost never comes in the form which can be used for analyzing or training, The most common problems in raw data include missing values, inconsistent data, duplicate records.</p><p>Tabular data can be organized in many ways some are suitable for reporting and some are suitable for analysis, The process of turning an untidy data-set into a form which is suitable for data analysis is know as <strong>data</strong> <strong>tidying</strong>.</p><p><strong>The principles of tidy data are</strong></p><ol><li>columns represent unique variables.</li><li>rows represent individual observations.</li></ol><p>Lets explore some functions which make data cleaning easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*x66pKHSARDDODBMGnIrNOw.png" /></figure><p><strong>Melting</strong></p><p>Melting is a way of tidying the data, the columns are turned into rows making data good for analysis.</p><p><strong>syntax</strong></p><pre>pd.melt(df, argument1, argument2,....)</pre><p><strong>key arguments</strong></p><p>id_vars = [ ] : represents columns you do not want to melt</p><p>value_vars = [ ] : represents columns you do wish to melt into rows if this argument is not provided all the columns not set in id_vars will be melted.</p><p><strong>Pivoting</strong></p><p>Exactly opposite to melting, turning every unique value into columns is called pivoting. Making the data report friendly.</p><p><strong>syntax</strong></p><pre>df.pivot(argument1, argument2, ...)</pre><p><strong>key arguments</strong></p><p>index = [ ] : represents columns you do not want to pivot</p><p>columns = [ ] : represents columns you want to pivot</p><p>values = [ ] : values to be used when the column is pivoted</p><p>aggfunc = np.ufunc : in case of duplicate rows the values are computed as per the ufunc mean, max, min..</p><p><strong>why?</strong></p><p>makes data cleaning easier for better data analysis.</p><h4>Date Time Series</h4><p>It is one of most common series data, I have found the pandas package to be extremely useful for time series manipulation and analysis.</p><p><strong>functions</strong></p><h4>pd.to_datetime(series, format=time_format)</h4><p>Passing list of strings and specifying the format in which the strings are present will parse the strings into datetime objects.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37f3d1e175333ee7af8a087df9087ee0/href">https://medium.com/media/37f3d1e175333ee7af8a087df9087ee0/href</a></iframe><p><strong>why?</strong></p><p>datetime object is powerful and we can get the individual elements of the date <strong>Eg. datetime_object.dt.hour( )</strong></p><h4><strong>df.reindex(list, method=func)</strong></h4><p>reindexing a dataframe df according to the indices given in the list, if the indices doesn&#39;t match then the values are filled according to optional logic func, it can be ffill or bfill they represent forward fill and backward fill.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ace6b2dc68bb3bd7ac75af785425c330/href">https://medium.com/media/ace6b2dc68bb3bd7ac75af785425c330/href</a></iframe><p><strong>why?</strong></p><p>when we need to compare two dataframes it would be better if we have the same indexing order.</p><h4><strong>pd.resample(rule).func( )</strong></h4><p>Resampling involves changing the frequency of your observations, the records are up-sampled or down-sampled according to the <strong><em>rule</em></strong>, Most common rules in the table below. This can be chained with any statistical functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*MD1HrXj73EWIMYm9vXeZyQ.png" /></figure><p>downsampling is reducing the frequency of records example turning daily data to weekly data in contrast up sampling is increasing the frequency, while we compute the later we often come up with many missing values we can overcome this by interpolating the values.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7126616c34daf9319ff16c9abb614fca/href">https://medium.com/media/7126616c34daf9319ff16c9abb614fca/href</a></iframe><p><strong>why?</strong></p><p>to provide additional structure or insight into the learning problem for supervised learning models</p><p><strong>#Update 08–08–18</strong></p><h4>Categorical Data</h4><p>Often in real world data each column consists of repetitive values, like gender, country. We can convert the series data type to category type using</p><p><strong>syntax</strong></p><p>df[column_name].astype(&#39;category&#39;)</p><p><strong>why?</strong></p><p>Save memory especially in case of large datasets also speed up operations like groupby.</p><h4>Grouping</h4><p>It involves <strong>splitting</strong> the dataframe by the column name into groups and <strong>applying</strong> a function on each group in the column and <strong>merging</strong> all the results.</p><p><strong>syntax</strong></p><pre>df.groupby(column_name) <strong><em>or</em></strong> df.groupby(condition)</pre><p>groupby return an object on which we apply a function and finally combine the results</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d751e14e8854ec8c6775e0709767a743/href">https://medium.com/media/d751e14e8854ec8c6775e0709767a743/href</a></iframe><h4><strong>functions</strong></h4><p><strong>aggregation</strong> function returns a single aggregated value for each group</p><p><strong>syntax</strong></p><pre>df.groupby(column_name).agg([func1, func2, ...])</pre><p>If we want to apply different functions to different columns we could pass a dictionary to agg( ) instead of a list.</p><p>df.groupby(column_name).agg({column1:func1, column2:func2})</p><p><strong>transformation </strong>function applies the function to each group and returns the same size data frame where each group has same aggregated value.</p><p><strong>syntax</strong></p><pre>df.groupby(column_name).transform(function)</pre><p><strong>filter</strong> function filters the dataframe on some Boolean condition</p><p><strong>syntax</strong></p><pre>df.groupby(column_name).filter(function)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cbe5480486a78d5b73382b4d62fd754/href">https://medium.com/media/4cbe5480486a78d5b73382b4d62fd754/href</a></iframe><p><strong>visualization</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*fcGsJmbYsUpyFQHL98LVvg.png" /><figcaption>Thanks to pbpython :)</figcaption></figure><p><strong>why?</strong></p><p>helps to properly analyse the data also fill missing values appropriately using transform.</p><p>See ya soon 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af7f10e3a31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/hacking-pandas-4af7f10e3a31">hacking-pandas</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[hacking-numpy]]></title>
            <link>https://medium.com/hacking-datascience/hacking-numpy-29ed0a6ec8ef?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/29ed0a6ec8ef</guid>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Tue, 24 Jul 2018 11:44:50 GMT</pubDate>
            <atom:updated>2018-08-23T02:17:54.031Z</atom:updated>
            <content:encoded><![CDATA[<p>The core python packages like pandas, scikit-learn are built on top of numpy, So in this story I’m going to list some key functions of numpy package which made my computations much easier/efficient.</p><p>Will keep on appending new stuff to the story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*sESyI6dq-_QYutdNlg9zpg.png" /></figure><p>Get the ipython notebook of this story from <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/hacking-numpy.ipynb"><strong>here</strong></a>.</p><pre>import numpy as np #importing the package</pre><h4><strong>np.random.seed( )</strong></h4><p>generating random number’s is one of the major task of any data science problem, numpy does a very good job in this regard. There will be times when we want numpy to generate same set of random values on every invocation, In this scenario we could set a seed <strong><em>instead of copying</em></strong> the random array.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b89de97ef0b01da362f6306d247e79e5/href">https://medium.com/media/b89de97ef0b01da362f6306d247e79e5/href</a></iframe><p><strong>why?</strong></p><p>when we want to check the performance of our model we often need to test on same data to get the accurate results, using seed will help save memory.</p><h4>np.reshape(array, shape) / array.reshape(shape)</h4><p>as the name says it resizes the given ndarray where <em>shape = (rows, columns).</em></p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/362ebfa4dd6205c86e3ba115b3bcc3ae/href">https://medium.com/media/362ebfa4dd6205c86e3ba115b3bcc3ae/href</a></iframe><p><strong>why?</strong></p><p>reshape is especially useful when we want to reshape matrices.</p><h4>np.nditer( )</h4><p>it is an efficient multi dimension iterator object, it does iterate over every element in a ndarray.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b769e1bc63585871e34ef9f81605e61/href">https://medium.com/media/7b769e1bc63585871e34ef9f81605e61/href</a></iframe><p><strong>why?</strong></p><p>efficiently iterate over multi dimensional arrays.</p><h4>np.flatnonzero( )</h4><p>returns the indices which are non zero in the input array.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4473e7a4cb18c5c5bc2b131338224cb/href">https://medium.com/media/e4473e7a4cb18c5c5bc2b131338224cb/href</a></iframe><p><strong>why?</strong></p><p>selecting specific sequence in an array obeying required condition.</p><h4><strong>np.argsort( )</strong></h4><p>returns the index of each element of the input array after sorting.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c558d9c7e1b77d40ee59ab1f8f4c050e/href">https://medium.com/media/c558d9c7e1b77d40ee59ab1f8f4c050e/href</a></iframe><p><strong>why?</strong></p><p>useful when we need to sort an array according to another array.</p><h4>np.ufunc.at(input_array, indices, second_operand_ufunc)</h4><p>applies ufunc(Universal Functions) at particular indices, the list of ufunc are available <a href="https://goo.gl/7V2ZTo">here</a></p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10f0e129914dd4fb3c5360afd4910569/href">https://medium.com/media/10f0e129914dd4fb3c5360afd4910569/href</a></iframe><p><strong>why?</strong></p><p>it can used to apply mathematical operations to a part of array satisfying certain condition.</p><p>See you again 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29ed0a6ec8ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/hacking-numpy-29ed0a6ec8ef">hacking-numpy</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python hacks for data-science.]]></title>
            <link>https://medium.com/hacking-datascience/python-hacks-for-data-science-9009a3d7e6e5?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/9009a3d7e6e5</guid>
            <category><![CDATA[reduce]]></category>
            <category><![CDATA[generator]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[maps]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 22:51:23 GMT</pubDate>
            <atom:updated>2018-08-15T19:59:50.659Z</atom:updated>
            <content:encoded><![CDATA[<p>In this story I’ll show how some tasks can be made easier/efficient using few built-in python tools, I’m considering Python 3.6.5 in all my stories henceforth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*OrwbmUm3yD7VJQu4ebJqSA.png" /></figure><p>Get the ipython notebook of this story from <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/python_hacks_for_datascience.ipynb"><strong>here</strong></a></p><h4>Generators</h4><p>Before getting to know about generators let’s first understand iterators, So what is an iterator?</p><blockquote><em>Iterator in python is simply an object that can be iterated upon.</em></blockquote><p>next(iterator) will return one value at a time, when we reach the end there is no data to return then an exception will be raised. If an object can return an iterator then that object is know an <strong>iterable</strong>, few examples are <em>lists, dictionaries, tuples, </em>and<em> strings</em>.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/634705524f02ad70ec542280da229260/href">https://medium.com/media/634705524f02ad70ec542280da229260/href</a></iframe><p>So now what is a generator?</p><blockquote>Generators are iterator objects, they get executed only when we need them.</blockquote><p>When a generator function is called it returns an generator object without even executing the function, when we apply<strong> next( )</strong> on that object the function starts executing until it reaches <strong><em>yield</em></strong> statement, and the yielded value is returned.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8394cdf3525ef1029075e93475fbf7f4/href">https://medium.com/media/8394cdf3525ef1029075e93475fbf7f4/href</a></iframe><p><strong>why?</strong></p><p><strong>To save memory space </strong>: When we are dealing with very large datasets which cannot be loaded into memory all at ones, we can use generators.</p><h4>zip( )</h4><blockquote>zip( ) takes iterables as arguments and returns a zip object which contain list of tuples.</blockquote><p><strong>syntax</strong></p><pre>zip(iterable1, iterable2, …)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5db7ff014789ae44264b3c392f511752/href">https://medium.com/media/5db7ff014789ae44264b3c392f511752/href</a></iframe><p>to unzip the zipped object we can use <strong>zip(*zipped_object)</strong></p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46d8a5c0ceb98c3bf21fefa2b7a23d6d/href">https://medium.com/media/46d8a5c0ceb98c3bf21fefa2b7a23d6d/href</a></iframe><h4>List Comprehension</h4><blockquote>They are used to create new lists quickly from other iterables.</blockquote><p><strong>syntax</strong></p><pre>[expression <em>condition</em> loop <em>condition</em>]</pre><p><strong>note</strong> : condition’s are optional</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18b32567dab380cb1ae2d9bec7632d47/href">https://medium.com/media/18b32567dab380cb1ae2d9bec7632d47/href</a></iframe><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c0035bea9a911dde4ae7758bf122c76/href">https://medium.com/media/6c0035bea9a911dde4ae7758bf122c76/href</a></iframe><p><strong>why?</strong></p><p><strong>compact and faster : </strong>title says it so<strong> 😛</strong></p><h4><strong>Lambda function</strong></h4><blockquote>Lambda function doesn&#39;t have any name and sometimes called as an <em>anonymous</em> function, they can take arguments similar to a normal function but can only have a single expression, when it is called the <strong><em>expression</em></strong> is evaluated and the result is returned which can be assigned to any variable.</blockquote><p><strong>syntax</strong></p><pre>lambda input-arguments : expression</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21842b6122606027f9b365a367743947/href">https://medium.com/media/21842b6122606027f9b365a367743947/href</a></iframe><p><strong>why lambda function?</strong></p><p><strong>ease of use</strong> : when the function isn&#39;t complex we could use lambda function instead of a normal function.</p><h4>map( )</h4><blockquote>map( ) takes a function and a sequence as arguments, It returns a map object which contain the function call results.</blockquote><p>map object can be converted to list using <strong>list(map-object)</strong></p><p><strong>syntax</strong></p><pre>map(function, iterable1, iterable2, …) </pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b619a6f7359a327a565317922f9745c/href">https://medium.com/media/3b619a6f7359a327a565317922f9745c/href</a></iframe><h4><strong>filter( )</strong></h4><blockquote>filter() takes a function and a sequence similar to a map function and filters the given sequence with the help of the input function that tests each element in the sequence to be true or not and returns a filter object.</blockquote><p><strong>syntax</strong></p><pre>filter(function, iterable)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1815488d738fb51267adf129a3c99215/href">https://medium.com/media/1815488d738fb51267adf129a3c99215/href</a></iframe><p><strong>why?</strong></p><p><strong>super fast execution</strong> : given multiple sequence arguments, it sends items taken form sequences in parallel as distinct arguments to the function.</p><h4>reduce( )</h4><blockquote>reduce( ) takes a function and a sequence as arguments, when the reduce function is called the first two items of the sequence are passed on to the input function then the computed result along with the next item in sequence is again passed on to the function, this will repeat until the end of the sequence and the final result is returned.</blockquote><p><strong>note : </strong><em>reduce( ) has to be imported from functools</em></p><p><strong>syntax</strong></p><pre>functools.reduce(function, iterable)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15296d601cd0e19eb25ef30ada17d82f/href">https://medium.com/media/15296d601cd0e19eb25ef30ada17d82f/href</a></iframe><p><strong>why?</strong></p><p><strong>fast computation</strong> : way faster than using loops.</p><h4>glob( )</h4><blockquote>Returns an iterable of names matching the given pattern.</blockquote><p><strong>syntax</strong></p><pre>files = glob(&#39;sales*.csv&#39;)</pre><p>returns a list which contains all the filenames like sales_1.csv, sales_2.csv….</p><p><strong>why?</strong></p><p>This is very useful when we have data in many files, we could read the list of files using a for loop.</p><p>See you again. 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9009a3d7e6e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/python-hacks-for-data-science-9009a3d7e6e5">Python hacks for data-science.</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started…]]></title>
            <link>https://medium.com/hacking-datascience/getting-started-c159a5e8ec1d?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/c159a5e8ec1d</guid>
            <category><![CDATA[about]]></category>
            <category><![CDATA[course]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[summary]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 22:50:01 GMT</pubDate>
            <atom:updated>2018-08-06T07:30:13.609Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey 😃 Bored of watching video lectures? Tired of doing courses which have almost the similar content, <em>introduction to …..</em> stuff? or need a bit revision of the key concepts? Welcome, you are at the right place. I have been in your shoes sometime back, so I had thought to pen down the summary of hours of video lectures, days of discussions, and months of practice. So here it is <strong>hacking-datascience </strong>for you<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ukaiQ5iA-tg1SafPVSUag.jpeg" /></figure><p>I was introduced to machine learning by my aunt during the 7th semester of my undergrad college, since then <em>(8 months until today)</em> I have been taking online courses, reading books, also taking advice from experts. I’ll try to summarise my learning and highlight the key concepts which are often missed in majority of the intro courses. I’ll keep on adding stuff to the stories as I encounter.</p><p>Feel free to <a href="https://www.linkedin.com/in/vinaykudari/"><strong>contact</strong></a> me for any queries, suggestions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c159a5e8ec1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/getting-started-c159a5e8ec1d">Getting started…</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What can data do ?]]></title>
            <link>https://medium.com/@vinaykudari/what-can-data-do-a3b1f23ddaa5?source=rss-3fbe4f6ebe31------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3b1f23ddaa5</guid>
            <category><![CDATA[cancer-research]]></category>
            <category><![CDATA[cancer]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Sun, 24 Jun 2018 23:38:22 GMT</pubDate>
            <atom:updated>2018-06-25T06:03:36.361Z</atom:updated>
            <content:encoded><![CDATA[<p>The main purpose of writing this blog is to know your opinion on this idea, Can this really help people? Is this idea feasible?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*FcCH_KuJiiVCBfAWQpU1IQ.jpeg" /></figure><p>Being a technical guy I sometimes used feel sad for not choosing medicine as my career path, but now this isn&#39;t a regret any more slowly the medical field is being backed up by technology. There’s a whole new world opening up for data scientists in this regard, the data era in medicine will be defined by a shift from intuition and opinion to data. We can collect more data in a day now than we could in a year not too long ago. Collecting data and applying it to solve healthcare problems will transform the cost and effectiveness of the medicine.</p><blockquote>‘In the next 10 years, data science and software will do more for medicine than all of the medical sciences together’</blockquote><p>So what is data science? It is a set of various tools used to discover hidden patterns from the large pool of data, which help individuals and organizations to make proper decisions. Recently if you have observed the train ticket reservation platform (IRCTC) we can see the confirmation probability of our ticket even before we make our booking, also now a computer can detect if a tumour is cancerous or non-cancerous just by looking at the scan, these are few of many use cases where data analysis is leveraged.</p><h4>UDI — Unique Diagnostic ID.</h4><p>Generally, we don&#39;t keep track of our medical reports, I feel that if there’s a software that can keep all our medical reports/conditions digitally and process them to find useful pattern can save lives.</p><p>Let me give you an example, suppose a person ‘A’ had gone through few medical conditions before he was diagnosed with a disease ‘X’, Now lets say if the person ‘B’ is also going through a similar pattern that ‘A’ had gone through, we can find this by analysing medical reports. Then we could inform him to get checked for ‘X’ even before the disease spreads. I think this will help, as large section of people aren’t aware of the symptoms/per-conditions of all the diseases.</p><blockquote>‘Early cancer diagnosis saves lives, cuts treatment costs’</blockquote><p>All the medical details related to the person are linked with the UDI which can be further linked to national ID (AADHAR). With a simple mobile application, he/she can get all his past medical records, This ID can be used to know the patient past medical history, necessary medical details, because of this continuity of medical care can be achieved, leading to redundancy of tests and proper care. Also, this can be used in case of emergencies where patient past medical conditions/details are to be known to perform further operations.</p><p>There are many applications out there which keep track of individual medical reports but there isn&#39;t any central database of all the health data which can be analysed to help patients.</p><p>Kindly let me know your views on this. I can be reached at this address: vinay.kudari30@gmail.com also I’m available at my LinkedIn: <a href="https://www.linkedin.com/in/vinay-kudari-6536a7bb">https://www.linkedin.com/in/vinay-kudari-6536a7bb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b1f23ddaa5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>