<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yashowardhan Shinde on Medium]]></title>
        <description><![CDATA[Stories by Yashowardhan Shinde on Medium]]></description>
        <link>https://medium.com/@yashowardhanshinde?source=rss-9e97580d4265------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*mmlCn8RehLrDq_IuofB6SQ.jpeg</url>
            <title>Stories by Yashowardhan Shinde on Medium</title>
            <link>https://medium.com/@yashowardhanshinde?source=rss-9e97580d4265------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:34:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yashowardhanshinde/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Label Text Data Using LLMs?]]></title>
            <link>https://medium.com/geekculture/how-to-label-text-data-using-llms-f0ffc3fcd168?source=rss-9e97580d4265------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0ffc3fcd168</guid>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[data-labeling]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Yashowardhan Shinde]]></dc:creator>
            <pubDate>Fri, 05 May 2023 16:01:37 GMT</pubDate>
            <atom:updated>2023-05-05T16:01:37.197Z</atom:updated>
            <content:encoded><![CDATA[<p>In recent years, the field of machine learning has seen exponential growth in the amount of available data. With the rise of big data, many organizations are collecting massive amounts of unstructured data such as text, images, and videos. One of the biggest challenges in utilizing this data is labeling it, which involves manually annotating the data with tags or categories that can be used for training machine learning models.</p><p>However, manual labeling can be a time-consuming and expensive process, and can often lead to errors and inconsistencies in the labeling. This is where the power of natural language processing (NLP) and language models like ChatGPT can be harnessed.</p><p>In this blog post, we will explore how to use LLMs (Large Language Models) for labeling text data. We will explore 2 simple approaches that can be used for text labeling and we will also look at a sample code for the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R-AcECd5RIT2_9vT.png" /></figure><h4>Contents:</h4><ol><li>Zero-Shot Approach</li><li>Few-Shot Approach</li><li>Code Implementation</li></ol><h3>Zero-Shot Approach</h3><p>In this approach, you directly ask the model to classify the sample. In the context of any LLM labeling text for sentiment analysis can be very efficient using this method. The prompt for this approach will look like the one shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Qrel2Nozu-IgVpXh5rTmg.png" /><figcaption>Sample Prompt for Zero-Shot Approach — Sentiment Classification</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXmCYPn7m-Ede80ltFcgFw.png" /><figcaption>Sample Prompt for Few-Shot Approach — General Text Classification</figcaption></figure><p>In the second example, an additional prompt is added after the main sentence in order to provide a problem definition which helps the LLM to understand the human intention and criteria for the required answer.</p><h3>Few-Shot Approach</h3><p>Few-shot learning is a method where a model is presented with a small set of high-quality examples consisting of an input and desired output on the target task. This allows the model to better understand the human intention and criteria for the required answers. As a result, few-shot learning often leads to better performance compared to zero-shot learning. Some limitations with this approach are, it requires more token consumption and may exceed the context length limit when input and output text is lengthy. Therefore, few-shot learning offers improved accuracy, but at the cost of increased computational resources and the possibility of exceeding context length limitations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyMjJzedpaSsjY2vokVI_A.png" /><figcaption>Sample Prompt for Few-Shot Approach — Sentiment Classification</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tuilIYvfknmzSZW23fnKQA.png" /><figcaption>Sample Prompt for Few-Shot Approach — General Text Classification</figcaption></figure><p>Here, in the second example, the context samples given to the model have 2 categories: Sports and Politics which is enough for the LLM to understand the scope of the classification. But this can be made more informative by adding a problem definition prompt like: “With context to the above examples classify the following example into the Sports and Politics category.”</p><blockquote>Many studies looked into how to construct in-context examples to maximize the performance and observed that <strong>choice of prompt format, training examples, and the order of the examples can lead to dramatically different performance</strong>, from near random guess to near SoTA. — <a href="https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/">link</a></blockquote><p>As the Few-Shot approach is known to perform really well in most cases where the given samples have good quality and diversity, this makes me think if this approach can potentially help us in avoiding model fine-tuning for simple to moderate classification tasks, where a limited amount of good-quality samples are enough for an LLM to understand the context of the task.</p><p>To get a deeper insight into how the prompt can be engineered and how to select a diverse set of samples for a few-shot approach you can refer to this really informative article by <a href="https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/">Weng, Lilian</a>.</p><h3>Code Implementation</h3><p>To use ChatGPT for labeling text you can use the following code. We will be using the API offered by OpenAI for GPT-3 and Hugging Face model T5 offered by Google.</p><h4>OpenAI GPT-3 API:</h4><pre>import openai<br><br># Get the OpenAI API key by signing up on OpenAI.<br>openai.api_key = &quot;&quot;<br><br># Zero-Shot Approach<br># Directly asking the model to label the text.<br>def generate_text_labels(text, categories):<br>    labels = []<br>    text_label_mapping = {}<br><br>    # String of categories in which you want to classify the text.<br>    category_str = &quot;, &quot;.join(map(str, categories))<br>    <br>    # Sample Prompt<br>    # Example: I am happy today; Classify this sentence as Positive, Negative or Neutral in one word.<br>    for i in range(len(text)):<br>        response = openai.ChatCompletion.create(<br>                  model=&quot;gpt-3.5-turbo&quot;,<br>                  messages=[<br>                        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;{text[i]}; Classify this sentence as {category_str} in one word.&quot;},<br>                        # OR - For simple sentiment classification:<br>                        # {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Text: {text[i]} \nSentiment in one word:&quot;},<br>                    ]<br>                )<br>        label = response.choices[0][&quot;message&quot;][&quot;content&quot;].strip(&quot;.&quot;)<br>        labels.append(label)<br>        text_label_mapping[text[i]] = label<br>    <br>    return labels, text_label_mapping<br><br># Few-Shot Learning<br># context = [(&quot;Sentence&quot;, &quot;Category/Sentiment&quot;)]<br>def generate_text_labels_context(text, context):<br>    labels = []<br>    text_label_mapping = {}<br>    <br>    # Examples to help model understand the task and context<br>    context_string = str()<br>    for i in range(len(text)):<br>        context_string += f&quot;Text: {context[i][0]}\nSentiment: {context[i][1]}\n&quot;<br>        # OR<br>        # context_string += f&quot;Text: {context[i][0]}\nCategory: {context[i][1]}\n&quot;<br><br>    # Sample Prompt<br>    &quot;&quot;&quot;<br>    Text: A<br>    Category/Sentiment: X<br>    Text: B<br>    Category/Sentiment: Y<br>    Text: C<br>    Category/Sentiment:<br>    &quot;&quot;&quot;<br>    for i in range(len(text)):<br>        response = openai.ChatCompletion.create(<br>                  model=&quot;gpt-3.5-turbo&quot;,<br>                  messages=[<br>                        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;{context_string} \nText: {text[i]} \nSentiment:&quot;},<br>                        # OR<br>                        # {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;{context_string} \nText: {text[i]} \nCategory:&quot;},<br>                    ]<br>                )<br>        labels.append(response.choices[0][&quot;message&quot;][&quot;content&quot;].strip(&quot;.&quot;))<br>        text_label_mapping[text[i]] = response.choices[0][&quot;message&quot;][&quot;content&quot;].strip(&quot;.&quot;)<br>    <br>    return labels, text_label_mapping</pre><h4>Hugging Face — Google T5:</h4><pre>from transformers import T5Tokenizer, T5ForConditionalGeneration<br><br>tokenizer = T5Tokenizer.from_pretrained(&quot;google/flan-t5-base&quot;)<br>model = T5ForConditionalGeneration.from_pretrained(&quot;google/flan-t5-base&quot;)<br><br># Zero-Shot Approach<br># Directly asking the model to label the text.<br>def generate_text_labels(text, categories):<br>    labels = []<br>    text_label_mapping = {}<br><br>    # String of categories in which you want to classify the text.<br>    category_str = &quot;, &quot;.join(map(str, categories))<br>    <br>    # Sample Prompt<br>    # Example: I am happy today; Classify this sentence as Positive, Negative or Neutral in one word.<br>    for i in range(len(text)):<br>        input_text = f&quot;{text[i]}; Classify this sentence as {category_str} in one word.&quot;<br>        input_ids = tokenizer(input_text, return_tensors=&quot;pt&quot;).input_ids<br><br>        outputs = model.generate(input_ids)<br>        label = tokenizer.decode(outputs[0])<br>        labels.append(label)<br>        text_label_mapping[text[i]] = label<br>    <br>    return labels, text_label_mapping<br><br># Few-Shot Learning<br># context = [(&quot;Sentence&quot;, &quot;Category/Sentiment&quot;)]<br>def generate_text_labels_context(text, context):<br>    labels = []<br>    text_label_mapping = {}<br>    <br>    # Examples to help model understand the task and context<br>    context_string = str()<br>    for i in range(len(text)):<br>        context_string += f&quot;Text: {context[i][0]}\nSentiment: {context[i][1]}\n&quot;<br>        # OR<br>        # context_string += f&quot;Text: {context[i][0]}\nCategory: {context[i][1]}\n&quot;<br><br>    # Sample Prompt<br>    &quot;&quot;&quot;<br>    Text: A<br>    Category/Sentiment: X<br>    Text: B<br>    Category/Sentiment: Y<br>    Text: C<br>    Category/Sentiment:<br>    &quot;&quot;&quot;<br>    for i in range(len(text)):<br>        input_text = f&quot;{context_string} \nBased on the above examples determine the sentiment of the following sentence. \nText: {text[i]} \nSentiment:&quot;<br>        # OR<br>        # input_text = f&quot;{context_string} \nBased on the above examples determine the category of the following sentence. \nText: {text[i]} \nCategory:&quot;<br>        input_ids = tokenizer(input_text, return_tensors=&quot;pt&quot;).input_ids<br><br>        outputs = model.generate(input_ids)<br>        label = tokenizer.decode(outputs[0])<br>        labels.append(label)<br>        text_label_mapping[text[i]] = label<br>    <br>    return labels, text_label_mapping</pre><p>Based on the output of the LLM the prompt can be modified to get more accurate results. In some cases, the output of the model might be correct but it wouldn’t be in the required format. In this case, you might have to do some basic output processing to get the results in the required format.</p><h3>Conclusion</h3><p>In conclusion, LLMs have proven to be effective tools for labeling text data. The two main approaches for using LLMs for text labeling are Zero-Shot and Few-Shot learning. The article also covered the implementation of both approaches using popular frameworks such as Hugging Face’s transformers library and OpenAI’s GPT-3 API. We can leverage these frameworks to quickly label large amounts of text data accurately, saving us valuable time and resources.</p><p>I hope this blog will help you streamline and automate many text data labeling tasks. With the availability of pre-trained models and easy-to-use APIs, we can apply these techniques to a wide range of applications (text summarization, NER, etc.) and domains, making it an essential tool for any data scientist or machine learning practitioner. If you have enjoyed reading this blog, do consider following me on Medium for similar content and connecting with me on <a href="https://www.linkedin.com/in/yashowardhan-shinde-1636971b1/">LinkedIn</a>!</p><p><a href="https://www.analyticsvidhya.com/blog/author/yasho_191/">Check out my other articles here!</a></p><h3>References</h3><ul><li><a href="https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/">Prompt Engineering</a></li><li><a href="https://huggingface.co/google/flan-t5-base">google/flan-t5-base · Hugging Face</a></li><li><a href="https://platform.openai.com/docs/api-reference/chat/create">OpenAI Platform</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0ffc3fcd168" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-label-text-data-using-llms-f0ffc3fcd168">How to Label Text Data Using LLMs?</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Data Augmentation using Keras ImageDataGenerator]]></title>
            <link>https://medium.com/geekculture/custom-data-augmentation-using-keras-imagedatagenerator-7cfd58e54171?source=rss-9e97580d4265------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cfd58e54171</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[augmentation]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Yashowardhan Shinde]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 22:02:33 GMT</pubDate>
            <atom:updated>2022-11-08T22:02:33.033Z</atom:updated>
            <content:encoded><![CDATA[<p>We often run into trouble when training a neural network because we don’t have enough data to train it. And as we all know, most machine learning models, and deep learning models, in particular, are only as good as the training data they are given. Now we can try to collect more data, but collecting data is a time-consuming and tedious task. A solution to this problem is data augmentation.</p><p><strong>The contents of this blog are as follows:</strong></p><ol><li>What is Data Augmentation?</li><li>Keras ImageDataGenerator</li><li>Steps to Create Custom Transformations</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tn5OQOMmetl1XxBWsEiF8w.png" /></figure><h3>What is Data Augmentation?</h3><p>Augmentation is a technique used to increase the size of your dataset by applying different transformations to the existing data and creating new samples using them. As seen in Figure 1, a single image of a dog can be augmented using transformations like rotation and cropping. The aim of augmentation is not just to increase the number of samples in the dataset but also to create variations in the dataset, which will prevent overfitting and help the model generalize better on unseen images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_LOOgcnxzk1wfOTiMlwCg.png" /><figcaption>Figure 1. Augmented Images of a Dog</figcaption></figure><h3>Keras ImageDataGenerator</h3><p>An easy way of augmenting data without creating a large overhead is by using the Keras ImageDataGenerator. The ImageDataGenerator generates batches of tensor image data with real-time data augmentation. It ensures that our model will receive variation in data at each epoch, which helps prevent overfitting. At the same time, the ImageDataGenerator uses less memory as it loads all the images in batches rather than all at once.</p><p>ImageDataGenerator also provides a wide range of transformations:</p><ol><li>Rotation</li><li>Brightness</li><li>Shear</li><li>Zoom</li><li>Horizontal and Vertical Flip</li></ol><p>But what if you want to add more transformations? How can you keep all the awesome features of the ImageDataGenerator and add your custom transformations to it? So, in this blog, I will be explaining to you how you can add your custom transformations to ImageDataGenerator.</p><h3>Steps to Create Custom Transformations</h3><p>The most efficient way of creating your custom transformations is by creating a Custom Image Data Generator class that inherits from the original ImageDataGenerator class. This class can be easily extended to add new augmentations in the future. Let us get into the code now.</p><p>We will be implementing 2 custom transformations: vertical and horizontal motion blur. This will help us understand how multiple transformations can be implemented.</p><h4>Step 0: Importing Libraries</h4><pre>import cv2<br>import numpy as np<br>from tensorflow.keras.preprocessing.image import ImageDataGenerator<br>import matplotlib.pyplot as plt</pre><h4>Step 1: Declare CustomDataGenerator class</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4ef7958bb4099282168dbf1321646b0/href">https://medium.com/media/e4ef7958bb4099282168dbf1321646b0/href</a></iframe><p>As you can see, we extend ImageDataGenerator so that we can specify our parameters in the <strong>__init__() </strong>method. We declare the <strong>processing_function</strong> as <strong>actions</strong> where we will call our transformations horizontal and vertical motion blur. Now let&#39;s define our parameters in the <strong>__init__()</strong> function.</p><h4>Step 2: __init__() function</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6eba02163fbd0bf1499b8ec584e07e78/href">https://medium.com/media/6eba02163fbd0bf1499b8ec584e07e78/href</a></iframe><p>We have declared two parameters here, <strong>v_kernel_size</strong> and <strong>h_kernel_size,</strong> for the vertical motion blur and horizontal motion blur, respectively.</p><h4>Step 3: Writing the Transformation Logic</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/998fd32dd5fb2d3874ec29d4e64bd608/href">https://medium.com/media/998fd32dd5fb2d3874ec29d4e64bd608/href</a></iframe><p>This is not the most efficient way of implementing these transformations, but for the sake of implementing multiple transformations, we have created two different functions for motion blur. Note that this transformation can be done using a single function as well. You can check out this implementation of <a href="https://www.geeksforgeeks.org/opencv-motion-blur-in-python/#:~:text=Horizontal%3A,vector%20to%20create%20the%20filter.">motion blur.</a></p><h4>Step 4: Action Function Logic</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6632192ab46629da3bb44d8c92435626/href">https://medium.com/media/6632192ab46629da3bb44d8c92435626/href</a></iframe><p>Here, we call these our transformations. We need to randomly select one of the transformations for which we create a list of our transformations and generate random values in the range of 0 to 1, corresponding to each transformation. We then select the transformation with the highest value.</p><h4>Step 5: Putting all the functions together</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/879eb963ed349acdaeef19a52430c300/href">https://medium.com/media/879eb963ed349acdaeef19a52430c300/href</a></iframe><p>This is what our CustomDataGenerator will look like. Now it&#39;s time to execute this data generator and see what the output looks like. Note that we can still use the transformations present in the ImageDataGenerator as we are inheriting them into our custom data generator.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f29420a703454bdc12061334a596b34a/href">https://medium.com/media/f29420a703454bdc12061334a596b34a/href</a></iframe><p>The output of this script can be seen in Figure 8. The point to note here is that we had also included a default parameter of the ImageDataGenerator class <strong>horizontal_flip</strong> which can be seen in the subplot 1, 2, and 9. Our custom transformations are applied to subplots 3, 4, 5, 6, 7, and 8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*xg5Hx9a7tDIAaOTzTRzzbg.png" /><figcaption>Figure 8. Output of CustomDataGenerator</figcaption></figure><h3>Conclusion</h3><p>We have finally built a CustomDataGenerator with our custom transformations and that too, with only a few lines of code. You can try to implement other transformations, like blocking random patches and adding Gaussian noise. I hope this blog will help you create custom augmentations using Keras ImageDataGenerator without any problems. If you have enjoyed reading this blog, do consider following me on Medium for similar content and connecting with me on <a href="https://www.linkedin.com/in/yashowardhan-shinde-1636971b1/">LinkedIn</a>!</p><p><a href="https://www.analyticsvidhya.com/blog/author/yasho_191/">Check out my other articles here!</a></p><p>References:</p><ul><li><a href="https://www.analyticsvidhya.com/blog/2020/08/image-augmentation-on-the-fly-using-keras-imagedatagenerator/">Image Augmentation Keras | Keras ImageDataGenerator</a></li><li><a href="https://stepup.ai/custom_data_augmentation_keras/">Tutorial: Custom Data Augmentation in Jeras</a></li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator">tf.keras.preprocessing.image.ImageDataGenerator | TensorFlow v2.14.0</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cfd58e54171" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/custom-data-augmentation-using-keras-imagedatagenerator-7cfd58e54171">Custom Data Augmentation using Keras ImageDataGenerator</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing Tensorflow on Apple Silicon]]></title>
            <link>https://medium.com/geekculture/installing-tensorflow-on-apple-silicon-84a28050d784?source=rss-9e97580d4265------2</link>
            <guid isPermaLink="false">https://medium.com/p/84a28050d784</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[apple-silicon]]></category>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Yashowardhan Shinde]]></dc:creator>
            <pubDate>Thu, 20 Oct 2022 17:01:45 GMT</pubDate>
            <atom:updated>2022-10-20T17:01:45.593Z</atom:updated>
            <content:encoded><![CDATA[<p>Although a lot of content is present about the installation of Tensorflow on the new ARM-powered Mac, I still struggled to set up my Tensorflow environment on the Macbook Air M1. So I&#39;m writing this blog to walk you through an easy installation guide that never fails me and can help you set up your Tensorflow environment within minutes every time. So let&#39;s get started!</p><p>The contents of this blog are as follows:</p><ol><li>Can you use TensorFlow with GPU support on Apple Silicon?</li><li>Requirements to set up Tensorflow with GPU support on Mac</li><li>Step-by-step guide for installing TensorFlow with GPU support.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcyBZprtuGJFw3qbW-D0PQ.png" /></figure><h3>Can you use TensorFlow GPU on Apple Silicon?</h3><p>Early on, I did not know that you could take advantage of the GPU cores in the new ARM chips introduced by Apple. I came across the official Apple guide for installing Tensorflow GPU on a Mac powered by the new Apple silicon, which they call <a href="https://developer.apple.com/metal/tensorflow-plugin/">TensorFlow-Metal</a>. So yes, you can use TensorFlow with GPU support on Apple Silicon.</p><h3>Requirements to set up Tensorflow with GPU support on Mac</h3><p>So the basic requirement for installing TensorFlow-Metal is Miniforge. You can download Miniforge easily using the following steps:</p><ol><li>Download the Miniforge File from <a href="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh">here</a>.</li><li>Use the commands as shown below:</li></ol><pre>chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh<br>sh ~/Downloads/Miniforge3-MacOSX-arm64.sh<br>source ~/miniforge3/bin/activate</pre><p>If this doesn&#39;t work for some reason, you can refer to this video by <a href="https://www.youtube.com/watch?v=w2qlou7n7MA">Dr. Jeff Heaton</a> on YouTube. This video is well-made and explains the installation of Miniforge and Anaconda on Mac M1 in depth. Once you have Miniforge installed, you are good to go. I would highly recommend watching this video once.</p><h3>Step-by-step guide for installing TensorFlow</h3><p>Start by creating a new CONDA virtual environment and activating it. Use the commands below to do so.</p><pre>conda create -n tf python=3.9</pre><pre>conda activate tf</pre><p>Now we need to install the TensorFlow-Metal packages one by one. These include TensorFlow-deps, TensorFlow-macOS, and TensorFlow-metal. To install these, you can run the following commands:</p><pre>conda install -c apple tensorflow-deps</pre><pre>python -m pip install tensorflow-macos==2.9</pre><pre>python -m pip install tensorflow-metal</pre><p>Once this is done, you are good to go. You can check if your installation was successful by using the following commands. You should be able to see the TensorFlow version and a GPU device when you use the <strong><em>tf.config.experimental.list_physical_devices(&#39;GPU&#39;)</em></strong> command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OmzcND9eJXafUHe1LKkz6A.png" /><figcaption>Figure 1. Successful Installation Result</figcaption></figure><p>But now comes the tricky part. Many people will get an error at this stage, and 99% of the time, this is mainly because of the mismatch between the required and the installed NumPy version. The error could look like the one shown in Figure 2. The problem is that each TensorFlow version is built with a specific NumPy version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CfMxnxpsM7ulmVqgX6Spag.png" /><figcaption>Figure 2. The most common error after installation is completed</figcaption></figure><p>No need to worry. This error can be easily fixed by uninstalling NumPy and reinstalling the correct version.</p><pre>conda remove numpy</pre><pre>conda install -c conda-forge numpy=x.xx</pre><p>Now there should be no problems with your TensorFlow, and you are finally done with the installation. Using the code snippet shown earlier in the blog, Figure 1, you can ensure that everything works properly. I hope this guide helps you and you will never have trouble installing TensorFlow again. Consider following me on Medium and connecting with me on <a href="https://www.linkedin.com/in/yashowardhan-shinde-1636971b1/">LinkedIn</a>!</p><p><a href="https://www.analyticsvidhya.com/blog/author/yasho_191/">Check out my other articles here!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84a28050d784" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/installing-tensorflow-on-apple-silicon-84a28050d784">Installing Tensorflow on Apple Silicon</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is skewness in data? How to fix skewed data in python?]]></title>
            <link>https://yashowardhanshinde.medium.com/what-is-skewness-in-data-how-to-fix-skewed-data-in-python-a792e98c0fa6?source=rss-9e97580d4265------2</link>
            <guid isPermaLink="false">https://medium.com/p/a792e98c0fa6</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Yashowardhan Shinde]]></dc:creator>
            <pubDate>Fri, 27 Aug 2021 10:29:16 GMT</pubDate>
            <atom:updated>2021-08-27T10:29:16.027Z</atom:updated>
            <content:encoded><![CDATA[<p>We all have come across the terms outliers or skewness while cleaning or preprocessing data for a project. So, in this article, we will be discussing the following points:</p><ol><li>What is skewness?</li><li>How to identify skewness in data?</li><li>Why is skewness a problem?</li><li>How do we fix this skewness in python?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDQ07qinK-LcifyYkrRt7g.jpeg" /></figure><h3>What is skewness?</h3><p>Skewness is an asymmetry in the distribution of data as it does not show any kind of symmetry in continuous data. Skewed data can be of 2 types. Right-Skewed data is also called as Positively-Skewed data and, Left-Skewed data is called as Negatively-Skewed data.</p><h3>How to identify if your data is skewed or not?</h3><p>There are 2 main methods to identify skewness in the data. The first is the Observational method and, the second is the Statistical method.</p><h3>1. Observational Method</h3><p>Identification of skewness can be done easily by plotting a histogram and observing a few characteristics.</p><p>For a normal distribution i.e a distribution with no skewness the histogram should look like Fig(1) here, both the ends (tail regions) taper equally with a peek at the centre of the distribution like a “Bell-Curve”. For data with no skewness the <strong>Mean = Median = Mode</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*V3wUT_rqJm6ly5vYgk64Dg.png" /><figcaption>Normal Distribution Fig(1).</figcaption></figure><p>For distribution with left-skewness or negative skewness, the histogram should look like Fig(2) here the only the left part of the distribution tapers with the peak shifted towards the right-hand side. For Left-Skewed data the <strong>Mode &gt; Median &gt; Mean</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*oWh214Dm0mbDdoGAlVw_gg.png" /><figcaption>Left-Skewed Data Fig(2).</figcaption></figure><p>For distribution with right-skewness or positive-skewness the histogram should look like Fig(3) here the only the right part of the distribution tapers with the peak shifted towards the left-hand side. For Right-Skewed data the <strong>Mean &gt; Median &gt; Mode.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*oDRXZugvjxbDlxqJEwmJzA.png" /><figcaption>Right-Skewed Data Fig(3).</figcaption></figure><h3>2. Statistical Method</h3><p>Now, the method we studied above is the observational method. We can also calculate the skewness of our data using statistics as well. So the formula to calculate skewness is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/0*KWtcXmyR5_kBcvF_.jpg" /></figure><p><a href="https://media.geeksforgeeks.org/wp-content/uploads/skewness.jpg">Skewness Formula</a></p><p>The acceptable value of skewness is considered to be between -3 and +3. If the value of skewness falls on either side of this range then it indicates that the skewness must be fixed.</p><h3>Why is skewness a problem?</h3><p>Most of the statistical models do not work when the data is skewed. The reason behind this is that the tapering ends or the tail region of the skewed data distributions are the outliers in the data and it is known to us that outliers can severely damage the performance of a statistical model. The best example of this being regression models that show bad results when trained over skewed data.</p><h3>How do we fix this skewness?</h3><p>Now, as we know that skewness is bad for our data and many machine learning algorithms prefer or perform better when numerical data has a normal distribution we need a way to remove this skewness, to do so in python most frequently used methods are <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer"><strong>PowerTransformer</strong></a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer"><strong>QuantileTransformer</strong></a>. Both these methods are used to get a normal or Gaussian-like distribution. Now let us see how to use these methods with an example:</p><p>For the example below the dataset considered is the <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud"><strong>European Credit Card</strong></a> dataset which you can find on Kaggle and I have considered a few random columns with skewness to perform the transformation. The code for implementing the transformers is as following:</p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import numpy as np<br>import pandas as pd <br>from sklearn.preprocessing import PowerTransformer, QuantileTransformer<br>cols1 = [&quot;V1&quot;, &quot;V3&quot;, &quot;V4&quot;]<br>def test_transformers(columns):<br>    pt = PowerTransformer()<br>    qt = QuantileTransformer(n_quantiles=500, output_distribution=&#39;normal&#39;)<br>    fig = plt.figure(figsize=(20,30))<br>    j = 1<br>    for i in columns:<br>        array = np.array(df[i]).reshape(-1, 1)<br>        y = pt.fit_transform(array)<br>        x = qt.fit_transform(array)<br>        plt.subplot(3,3,j)<br>        sns.histplot(array, bins = 50, kde = True)<br>        plt.title(f&quot;Original Distribution for {i}&quot;)<br>        plt.subplot(3,3,j+1)<br>        sns.histplot(x, bins = 50, kde = True)<br>        plt.title(f&quot;Quantile Transform for {i}&quot;)<br>        plt.subplot(3,3,j+2)<br>        sns.histplot(y, bins = 50, kde = True)<br>        plt.title(f&quot;Power Transform for {i}&quot;)<br>        j += 3<br>test_transformers(cols1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kBvO-aZ_2p_itnbtvDBY1A.png" /><figcaption>PowerTransformer and QuantileTransformer applied on Data</figcaption></figure><p>It can be seen that QuantileTransformer has performed better than PowerTransformer in terms of converting the skewed distribution into a normal distribution, but you should always select the method which works best with your data. It’s observed that QuantileTransformer works better on larger datasets than PowerTransformer.</p><p>Other methods can be used to reduce the skewness as well. Like for left-skewed data squares, cubes or, higher power transforms can be performed and, for right-skewed data, we can use the Square-root, Cube-root or Log transforms. To know more about these, you can refer to this <a href="https://www.kaggle.com/getting-started/110134">article</a>.</p><h3>Conclusion</h3><p>We have now learned what skewness is. What are the types of skewness? How to identify skewness by observation and using statistics? How to fix the skewness in our data? If you liked this blog don’t forget to share it with your friends. In case you have any doubts or suggestions, do reply in the comments. Please feel free to contact me via mail and connect with me via Linked In. I hope you enjoyed reading this blog. Happy Learning!</p><p>Mail ID: yashowardhanshinde@gmail.com | <a href="https://www.linkedin.com/in/yashowardhan-shinde-1636971b1/">LinkedIn</a></p><p><a href="https://www.analyticsvidhya.com/blog/author/yasho_191/">Check out my previous articles here!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a792e98c0fa6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to find the correlation between continuous variables and visualise it using python?]]></title>
            <link>https://medium.com/geekculture/how-to-find-the-correlation-between-continuous-variables-and-visualise-it-using-python-7faf5b028ae0?source=rss-9e97580d4265------2</link>
            <guid isPermaLink="false">https://medium.com/p/7faf5b028ae0</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[correlation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Yashowardhan Shinde]]></dc:creator>
            <pubDate>Tue, 08 Jun 2021 04:32:24 GMT</pubDate>
            <atom:updated>2021-06-08T04:32:24.110Z</atom:updated>
            <content:encoded><![CDATA[<p>Feature selection is an important step in building a good machine learning model, one of the technique that helps us in selecting these features is checking the correlation between different features of the dataset. In this article we will discuss the following:</p><ol><li>What is Pearson Correlation Coefficient?</li><li>How to find the correlation between continuous variables in a dataset?</li><li>How to visualise this correlation?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*acND2Kvqlq860Xa8z5zBEw.png" /></figure><h4><strong>What is Pearson Correlation Coefficient?</strong></h4><p>The <strong>Pearson Correlation Coefficient</strong> is basically used to find out the strength of the linear relation between two continuous variables, it is represented using <strong>r</strong>. The mathematical formula to calculate the correlation coefficient is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_3Y6j6tZWekQjAn6dMNrWg.png" /><figcaption>Pearson Correlation Formula (Fig 1)</figcaption></figure><p>The value of this correlation coefficient ranges from <strong>-1 to 1</strong>, where 1 signifies positive correlation Fig(2), 0 represents no correlation Fig(3) and -1 represents negative correlation between the two variables Fig(4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*dLeIBhU_bTrUKbQZynxkgg.png" /><figcaption>Different Types of Correlations</figcaption></figure><p>It is important to note that <strong>Pearson Correlation only works on continuous data</strong> and not on categorical data, for categorical data we have to use a test called <a href="https://en.wikipedia.org/wiki/Chi-squared_test"><strong>Chi-Square Test</strong></a> which we will talk about in later blogs.</p><h4><strong>How to find the correlation between continuous variables in a dataset?</strong></h4><p>Now that you have an idea of what Pearson Correlation Coefficient is we can now move on to actually finding out this coefficient value in python. To find this correlation coefficient there are two ways either using <strong>Pandas </strong>or using <strong>SciPy</strong> libraries, we will have a look at both the methods.</p><p>For the example I have considered 4 data columns which are randomly initialised and will be finding out the correlation between all the four columns. Below is the code Implementation of finding out correlation coefficients:</p><h4><strong>Using SciPy:</strong></h4><p>It is preferred to use <a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.pearsonr.html"><strong>stats.pearsonr()</strong></a> when you want to compute correlation for lesser columns.</p><pre>import scipy.stats as stats<br>import numpy as np</pre><pre># initialise random data points<br>data_1 = np.random.randn(1, 20)<br>data_2 = np.random.randn(1, 20)</pre><pre># The stats.pearsonr() returns 2 values the correlation coefficient and the p-value<br>r, p_value = stats.pearsonr(data_1[0], data_2[0])<br>print(f&quot;Correlation coefficient of data_1 and data_2 = {r}&quot;)</pre><h4><strong>Using Pandas Dataframe:</strong></h4><p>Pandas has a built-in function called <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html"><strong>.corr()</strong></a> to find out Pearson correlation so we will use the same.</p><pre>import numpy as np<br>import pandas as pd</pre><pre># initialise random data points<br>data_1 = np.random.randn(1, 20)<br>data_2 = np.random.randn(1, 20)<br>data_3 = np.random.randn(1, 20)<br>data_4 = np.random.randn(1, 20)</pre><pre>data = {&quot;Column 1&quot;:data_1[0], <br>        &quot;Column 2&quot;:data_2[0], <br>        &quot;Column 3&quot;:data_3[0], <br>        &quot;Column 4&quot;:data_4[0]}</pre><pre># Initialise a dataframe with all the columns<br>df = pd.DataFrame(data)</pre><pre># .corr() can calculate &#39;pearson&#39;, &#39;kendall&#39;, &#39;spearman&#39; correlations.<br>print(df.corr(method=&#39;pearson&#39;))</pre><p>I would suggest you to try implementing these codes yourself for better understanding.</p><h4><strong>How to visualise this correlation using Heatmaps?</strong></h4><p>Now that we have found out these correlation values it would be great if we would be able to visualise the same so that we can interpret the results easily. To do so we will use the <strong>Seaborn</strong> Library and plot a <strong>Heat-map</strong>. Every box of this heat-map will be a representation of the correlation coefficient between the corresponding columns in the grid.</p><pre>import seaborn as sns<br>import matplotlib.pyplot as plt</pre><pre>fig = plt.figure(figsize=(15,10))<br>sns.heatmap(df.corr(method=&#39;pearson&#39;), annot = True, cmap=&quot;Blues&quot;)<br>plt.title(&quot;Correlation Heatmap&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*W6faQZ7cMBXwheShL2dMtw.png" /><figcaption>Fig 5.</figcaption></figure><p>The diagonal of the heat map will always show <strong>value = 1</strong> as it gives the correlation of the column with itself which should be 1 considering the columns are same. So now based on these values, we can find out the columns that are highly correlated and make a decision about the features to be selected.</p><p>But the heat-map doesn’t just tell us about the correlation between the independent and target variable it also shows if there is any significant correlation between the independent variables themselves, this is called <strong>multi-collinearity</strong> which is also an important part for feature selection during Multiple Linear Regression and we will discuss in the upcoming blogs. Till then you can read about this topic <a href="https://en.wikipedia.org/wiki/Multicollinearity"><strong>here</strong></a>.</p><h4><strong>Conclusion</strong></h4><p>We have now understood what correlation is. How to find out correlation in python? and How to visualise it for better understanding of the results. That’s all for this one! Stay tuned for the upcoming blogs and if you have any suggestions or doubts please comment them down below or mail me. Also do connect with me on <a href="https://www.linkedin.com/in/yashowardhan-shinde-1636971b1/"><strong>LinkedIn</strong></a><strong>. </strong>Happy learning to all!</p><p><strong>Mail ID: yashowardhanshinde@gmail.com</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7faf5b028ae0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-find-the-correlation-between-continuous-variables-and-visualise-it-using-python-7faf5b028ae0">How to find the correlation between continuous variables and visualise it using python?</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>