<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SelectStar on Medium]]></title>
        <description><![CDATA[Stories by SelectStar on Medium]]></description>
        <link>https://medium.com/@selectstar-ai?source=rss-423a8f65a8f3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vGvx3q2I81-NcZgkMYqWcg.jpeg</url>
            <title>Stories by SelectStar on Medium</title>
            <link>https://medium.com/@selectstar-ai?source=rss-423a8f65a8f3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:00:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@selectstar-ai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Easiest Way to Create a Simple Recommendation Service]]></title>
            <link>https://selectstar-ai.medium.com/easiest-way-to-create-a-simple-recommendation-service-c81d37dfbe08?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c81d37dfbe08</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[crowdsourcing]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Mon, 14 Dec 2020 14:40:56 GMT</pubDate>
            <atom:updated>2020-12-14T14:40:56.852Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we are going to discuss what <em>Recommendation Systems </em>are. We will talk about how to create a recommendation system with very little difficulty, but before that, we will talk about the different Machine Learning-based techniques used in implementing these systems. There are two most common algorithms used for the recommendation system: Content-based filtering and Collaborative filtering.</p><p>A basic example of a recommendation system from real life can be online shopping through Amazon or other e-commerce. When you purchase something, let’s say a phone, you will get a recommendation to buy a headphone or a screen protector to go with your phone. Another example can be Netflix’s recommendation engine. If you watch a particular genre of movies on Netflix e.g. action movies, you will soon see the recommendations of the same or a genre similar to the movie that you previously watched. The reason Netflix is able to understand viewers’ preferences so well and is also able to predict the next movie or show that the viewers would like to watch is achieved through the magic of a recommendation engine. By the end of this tutorial, you will not only be able to create a recommendation system on your own, but you will also be able to understand the techniques and the logic behind these systems. So without further ado, let’s get started!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*d1L3PQOzyy9TCGzgzexmgw.png" /><figcaption>Today, recommendations are prevalent all over our daily lives.</figcaption></figure><h3>What is a Recommendation System?</h3><p>To explain what a recommendation system truly is, we will make an analogy between the behaviors of human beings and those of these recommendation systems. Imagine a friend who is familiar with your taste in movies and always recommends movies that you end up watching again and again. Recommendation systems work very much in the same manner. They are like your friends, who know your tastes, likes, and dislikes and make recommendations of things to you accordingly. The question is, how do these recommendation systems know so much about you? Well, recommendation systems <em>basically filter data by using different algorithms and then recommend the most relevant items to you</em>. They first understand the past behavior of a user, based on his/her activity, and on the basis of that, they recommend products that the user is likely to buy or use. The next item that you buy on Amazon, the next show that you watch on Netflix, the next video that you watch on YouTube, the next friend that you add on Facebook, and the next song that you listen to on Spotify, all work on the same principles of recommendation systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DaW6jCmPmXKT4Ikf.png" /><figcaption>Netflix might know so much more than you can imagine.</figcaption></figure><h3>Techniques used in Recommendation Systems</h3><p>The most common types of recommendation systems are <em>Content-Based </em>and <em>Collaborative Based </em>Filtering recommendation systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*KZl97_Qtk4ozsV1ULSFYcg.png" /></figure><p><strong>Content-Based Filtering<br></strong>In Content-Based Filtering, items are recommended to users on the basis of being similar to what the user likes, and that is known by the user’s previous actions or explicit feedback given by him/her. We will try to understand this technique through an example. Let’s say that we have a user by the name of user X. User X has watched a movie M1 which belonged to the genre of action. She gave this movie a rating of 5 (out of 5). She watched another movie M2, which belonged to the genre of action as well, and gave it a rating of 4. From this, we can understand that user X enjoys watching action movies and we came to this conclusion because of the high ratings that she has given to these movies. Let’s say the third movie, M3 has recently been released and is of the same genre i.e. action. It also has some similarities with the previous movies e.g. the lead actor, director, etc. Now, the recommendation system will suggest this movie to user X on the basis of User 1’s profile.</p><blockquote>Main idea: Recommend items to customer X which are similar to previous items rated highly by X.</blockquote><p>For each item, i.e. a movie, website, blog, etc., an item profile is created and this profile becomes a set of feature vectors. For a movie, such feature vectors would consist of the writer, actor, director, etc. Important features are extracted from these item profiles and are used to build the User Profile which is the weighted average of rated item profiles.</p><p><strong>Collaborative Based Filtering</strong><br>Collaborative Based Filtering utilizes the quality judgments of others who are similar to the target user. Consider an example where we have 2 users, X and Y. X watches a movie M1 and gives it a rating of 5. User Y watches the same movie and gives it a rating of 4. User X watches movie M2 which is coincidentally also seen by user Y and both give it a rating of 4. User X watches a third movie M3 and gives it a rating of 5. Now, since the ratings of the two movies, M1 and M2, given by X and Y are so similar to each other, it is highly likely that the movie M3 (rated as 5 by user X) will also be liked by the user Y. The recommendation system will therefore recommend the movie M3 for user Y to watch.</p><blockquote>Main idea: Consider user X and find a set N of other users whose ratings are similar to X’s ratings. Then, estimate X’s ratings based on ratings of users in N.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bUx33q2-YpzkyDY1.png" /></figure><h3>Building your own Recommendation System using a Simple Machine Learning</h3><p>Now that we have talked about the two main techniques used in recommendation systems, we will discuss a simple way of creating one on your own using Machine Learning on Google Compute Engine. You can make use of open source technologies to create a basic recommendation system on the cloud platform.<br>A recommendation system consists of 4 phases: 1) collection, 2) storage, 3) analysis, and 4) recommendation. Its architecture consists of a front-end, a storage component, and Machine Learning. In the front-end, you can see a page with top recommendations by deploying a simple application on <a href="https://cloud.google.com/appengine/">Google App Engine</a>. This will help you to build a scalable web platform. For storage, <a href="https://cloud.google.com/sql/">Google Cloud SQL</a> integrates well with <a href="http://spark.apache.org/mllib/">MLlib</a> which is a Machine Learning library. Lastly, you can use <a href="https://cloud.google.com/dataproc/">Google Cloud Dataproc</a> to deploy and run MLlib based scripts. For more on this, visit <a href="https://cloud.google.com/blog/products/gcp/how-to-build-your-own-recommendation-engine-using-machine-learning-on-google-compute-engine">How to build your own recommendation engine using machine learning on Google Compute Engine</a></p><h3>Conclusion</h3><p>To sum it all up, we started this tutorial by discussing what recommendation systems and what they are used for. We talked about some real-life examples where these systems are used e.g. by Facebook to suggest friends, by Netflix to recommend movies or shows, and by YouTube to recommend videos, etc. Then we discussed the techniques of Content-based filtering and Collaborative filtering that enable a recommendation system to do what it is supposed to do i.e. to make recommendations to users. We saw how the two techniques differed from each other, where one technique i.e. Content-based filtering, made suggestions on the basis of the user’s profile whereas the other technique, Collaborative filtering, harnessed the judgment of other similar users to make recommendations to the user in question. Lastly, we briefly touched upon how one can build his/her own recommendation system using Machine Learning on Google Compute Engine.</p><h3>Recommendation System needs lots and lots of data!</h3><p>To provide the perfect recommendation, your recommendation system needs lots and lots of data with high quality. Most times, it is very difficult for an individual or small- or medium-sized companies to collect such large quantity data while maintaining high quality. Therefore, it is often more efficient to find another service that does laborious works for you. We could be your perfect solution!</p><p>Here at <a href="https://www.selectstar.ai/"><em>Selectstar</em></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity on time. Moreover, our in-house managers double-check the quality of the collected or processed data. If you need data? If you need preprocessed data? Let us know!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*pcPgOXOWU2GzbXCU.png" /><figcaption><a href="https://selectstar.ai/">selectstar.ai</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c81d37dfbe08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fairness? Ethics? Why is Diverse Data Important for Your A.I. Models?]]></title>
            <link>https://selectstar-ai.medium.com/fairness-ethics-why-is-diverse-data-important-for-your-a-i-models-89a564aefbf4?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/89a564aefbf4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[diversity]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Sat, 21 Nov 2020 04:44:36 GMT</pubDate>
            <atom:updated>2020-11-21T04:44:36.835Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we will be discussing the importance of having diversity in your Machine Learning (ML) or Artificial Intelligence (AI) datasets. We are all well aware that to effectively use an ML or AI model to solve a specific problem, it is crucial to have high-quality training data for the model itself. No matter how efficient or accurate the model is, if it is provided with and trained upon a poor quality dataset, it will never produce the desired or correct output. One important quality-related attribute of any dataset, regardless of the problem, is diversity. In this tutorial, we will talk about why diverse data is important for your models and what are the different ways in which you can introduce diversity and variability in your datasets. So, without wasting any time, let’s begin!</p><h3>Why is Diversity in a Dataset Important?</h3><p>We talked about having diverse data in one’s dataset, but what exactly is diversity and why do we even need it? Diversity is basically the variety that you have in your data. At times, while trying to solve a problem through ML or AI models, the data that we collect can be too huge in quantity and that can have a severe impact on the performance of the model. What to do then? Well, you can definitely opt to cut down the data so that your model can process it faster, but what about all that valuable information that you will lose if you remove important data from your dataset? Surely that will reduce the accuracy of your model. The main question that now arises is, how can we find a middle ground where your dataset is of a suitable size for the model to process it in a reasonable amount of time and that the information that the dataset contains is variable enough to tackle the full range of cases that the intended system will have to confront? The answer is simple, <strong>DIVERSITY</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*q1pEm2a23h9YBwn9.jpg" /><figcaption>Diverse Ethnicity is Crucial for Face Recognition</figcaption></figure><p>At times when the datasets are too large, the only way to do anything useful with them is to extract much smaller subsets from them and analyze those subsets instead. The subsets, however, need to be diverse enough so that the model can learn to handle and deal with all the different causes of the problem that it is trying to solve. Using diverse subsets is much more practical as compared to using a dataset with say a million data points, as that can be impossible to use on a desktop computer. Take the example of the face recognition and classification model. If this model is trained on a dataset of images showing different faces of people, and for each person, there are images taken from different angles, under changed lighting conditions, from varying distances from the camera lens, and in contrasting backgrounds, etc. Then the model is most likely to classify the faces more accurately as compared to being trained on a dataset that has thousands of similar types of images. In short, representative and diverse datasets are more likely to provide useful insights as compared to those who do not cover all facets of the problem at hand.</p><h3>How to Introduce Diversity in a Dataset?</h3><p>Diversity in a dataset can be achieved in quite a number of ways. If you are collecting the data purely from somewhere, you can include diverse data items in it by gathering relevant data from various different resources as compared to just a single resource. Also, keeping in mind the context of the problem that the model needs to solve, helps in the process of elimination of the different resources from which you can gather data and only leaves a handful of genuine data sources. There are many sources for open datasets that you can utilize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0zV2KLnsfAte91Ou.jpeg" /></figure><p>If you are gathering the data items on your own, e.g. you are taking pictures for an image classification problem, you can make sure that your dataset is as diverse and variable as possible by:</p><ul><li>Taking pictures at different angles.</li><li>Taking pictures under different lighting conditions.</li><li>Taking pictures at varying the distance of the camera lens from the object in question.</li><li>Varying the object size and shape if possible and then taking pictures.</li><li>Changing the background of the object in question and then taking pictures.</li><li>In the case of a colored object, taking pictures consisting of different colors.</li></ul><p>The same concepts of diversity can apply to datasets consisting of data of a different type and nature as well.</p><p>If you want to include diversity in a subset of a large dataset, one way can be to create a similarity matrix which is basically a huge table consisting of points and that maps every point in the dataset against every other point. The point of intersection of the row representing one data item and the column representing another constitutes the points’ similarity score on some standard measure. However, this method of dealing with matrices can be quite a time consuming and resource-intensive since we are talking about practically a million data items in a matrix. You can opt for different algorithms to include variability in your subsets e.g. <a href="https://news.mit.edu/2020/automating-search-entirely-new-curiosity-algorithms-0428">MIT Researcher’s Algorithm</a>. In this algorithm, a small subset from a much larger dataset is chosen at random and the algorithm then selects one point inside the subset and another outside it randomly as well. It then chooses any one of three simple operations i.e. swapping of the points, adding the point outside the subset to the subset, or deleting the point inside the subset on the basis of a number of factors which include the size of the large set, the size of the subset itself, etc. This process continues till the subset is diverse enough to meet a certain measurable level.</p><h3>Fairness and Ethics</h3><p>As mentioned above, one way to have diverse data is to collect it from different resources. However, while doing so, it is important to keep fairness, ethics, values, and good morals in mind. If you are collecting data from e.g. a website, it is important for you to first ask for permission from the owner of the data before utilizing it for your work or personal use. You can formally do so by dropping the concerned an email or contacting him/her in any other way if possible instead of getting access to it without formal consent. Also, you should also provide references to the different sources from where you gathered the data in your formal documentation or anywhere else where you can.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M5Oyxw6dV7BhbHwD.png" /><figcaption>Make sure your algorithm is introduced with various colors and its overlaps to be fair and ethical</figcaption></figure><h3>Need <strong>Diversity</strong> in Your Dataset? Let us HELP you out!</h3><p>Given the nature of its process, crowdsourcing is a very efficient way to deliver diversity in your data. Here at <a href="https://www.selectstar.ai/"><em>Selectstar</em></a>, we <strong>crowdsource</strong> our tasks to diverse users located globally to ensure quality and quantity simultaneously. Moreover, our in-house managers double-check the quality of the collected or processed data!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*0Lq7kU4_3K1pP0LL.png" /></figure><p>Creating and maintaining diversity in your dataset is not an easy task. Thinking about and maintaining all the things mentioned above is quite a burden. Especially, for small- to medium-sized companies, managing human resources and technical specialties are very challenging. Therefore, it is often more efficient to find another service that does laborious works (including both collection and preprocessing) for you. For that, we could be your perfect solution! Check us out at <a href="http://selectstar.ai/">selectstar.ai</a> for more information! Let us be your HELP!</p><h3>Conclusion</h3><p>To sum it all up, in this tutorial, we started off by talking about how important it is to have a dataset that meets a certain standard of quality and one very important constituent of a good quality dataset is diversity. Generally, a good dataset is composed of plentiful training data. Diversity of the training data ensures that it can provide more discriminative information to the model so that it can accurately predict results. We then discussed ways in which one can introduce diversity in his/her dataset e.g. by collecting data from different sources, using different algorithms to derive a diverse subset from a large set of data etc. Lastly, we touched upon the ethics and code of conduct that should be adopted while introducing variability in one’s dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89a564aefbf4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Different Types of Neural Networks — CNN & RNN]]></title>
            <link>https://selectstar-ai.medium.com/different-types-of-neural-networks-cnn-rnn-a91b27babfa3?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a91b27babfa3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Mon, 26 Oct 2020 07:08:19 GMT</pubDate>
            <atom:updated>2020-10-26T07:08:19.773Z</atom:updated>
            <content:encoded><![CDATA[<h3>Different Types of Neural Networks — CNN &amp; RNN</h3><h3>Introduction</h3><p>In this tutorial, we are going to talk about what Neural Networks are, how they function, and what are the different types of neural networks in general. Although there are different categories of neural networks, each having different topology and architecture, the underlying concept of every type is the same — i.e. <strong>being similar in action and structure to the human brain</strong>. We will be focusing on two types of neural networks, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) in this tutorial. Moreover, our goal is to tell you about the different types of datasets that are available online and which you can utilize for your own CNNs and RNNs projects. So, let’s begin!</p><h3>What is a Neural Network?</h3><p>Neural Networks form the basis of Deep Learning which in turn is a subfield of Machine Learning. Like we said before, in these networks, the algorithms are inspired by the human brain. Basically, a neural network takes in data <em>(as input)</em>, trains itself to recognize the patterns within the data <em>(learning)</em>, and then uses this learning (learned weights) to predict outputs for a new set of data which is similar in nature to the input data <em>(output).</em></p><p>Neural networks are made up of layers of neurons where each neuron acts as the core processing unit of the network. The <em>Simple Neural Network </em>has roughly three layers:</p><ol><li>The <strong>input layer </strong>receives the input data.</li><li>The <strong>hidden layer</strong> exists between the input and the output layers and performs the computation required by the network.</li><li>The <strong>output layer</strong> predicts the final output.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ioXDEsRRCewBSaBo.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AmAax6OMMfTlaLLV.png" /><figcaption>Simple visualized differences between <strong>Simple Neural Network </strong>and <strong>Deep Neural Network</strong></figcaption></figure><p>Neural networks can be more complex and this complexity is added by the addition of more hidden layers. A neural network that is made up of more than three layers i.e. has one input layer, several hidden layers, and one output layer is known as a <em>Deep Neural Network</em>. These networks are what support and underpin the idea and concepts of Deep Learning where the model basically trains itself to process and predict from data.</p><p>Then, how does a Deep Neural Network work? One quick way to understand the working of a deep neural network can be done using the example of handwritten digit recognition, MNIST tutorial from our previous article (<a href="https://medium.com/@selectstar.ai/what-is-mnist-and-why-is-it-important-e9a269edbad5">click here to learn more</a>!). We won’t go into detail about this tutorial here, but be sure to check it out!</p><h3>What are the Different Types of Neural Networks?</h3><p>Like we said before, there are a whole bunch of neural networks that differ by topology and structure and are used for different purposes. Some common examples include Perceptrons, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Hopfield Network, and so on. In this tutorial, we will be shedding light on <em>CNNs </em>and <em>RNNs</em>.</p><h4><strong>Convolutional Neural Networks (CNNs):</strong></h4><p>CNNs are quite different from other deep neural networks as they treat data as spatial. They are commonly applied to image processing problems as they are able to detect patterns in images, but can also be used for other types of input like audio. CNNs have the following layers:</p><p>- Convolution<br> - Activation Layer (typically use ReLU)<br> - Pooling<br> - Fully Connected</p><p>CNNs are typically used to compare images piece by piece. The pieces that a CNN looks for are called features. CNNs are able to detect similarities between different images much better as compared to whole image matching schemes because their technique is to find rough feature matches in roughly the same position in two or more images.</p><p>What sets a CNN apart from other deep neural networks is the fact that in CNNs, input data is fed through convolutional layers. In these layers, instead of having neurons being connected to every neuron in the previous layer, a neuron is instead only connected to other neurons that are close to it i.e., each neuron only concerns itself with neighboring neurons. This connectivity pattern is inspired by the organization of the animal visual cortex and greatly simplifies the connections in a network and enables it to uphold the spatial aspect of a dataset.</p><p>In the ReLU layer, every negative value from the filtered images is replaced by 0 to avoid these values from summing up to 0. The ReLU transform function activates a neuron only if the input is above a certain value i.e., if the input is below 0, the output is 0. Next is the Pooling layer in which the image stack is shrunk to a smaller size. The final layer is the fully connected layer where the actual classification happens. Here, the filtered and shrunken images are put together into a single list and the predictions are made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zh8xWtK4g69YHzBb.jpg" /><figcaption>Simple visualization of Convolutional Neural Network structure</figcaption></figure><h4><strong>Recurrent Neural Networks (RNNs):</strong></h4><p>RNNs are a type of neural network which are designed to recognize patterns in sequences of data e.g. in text, handwriting, spoken words, etc. Apart from language modeling and translation, RNNs are also used in speech recognition, image captioning, etc.</p><p>RNNs are unique in the sense that in RNNs, every neuron in a hidden layer receives an input with a specific delay in time. This is particularly useful for situations where previous information is needed in current iterations to make a decision. For example, if you are trying to predict the next word in a sentence, you first need to know the previously used words. To access previous information, RNNs contain loops that allow the previous information to persist. You can think of an RNN as multiple copies of the same network, where each copy is passing a message/information to the next document.</p><p>In RNNs, the model size does not increase with the increased input size and can process inputs and share any lengths and weights across time.<br>RNNs have some drawbacks as well. Firstly, the computational speed is slow because the model takes historical information into account. Also, RNNs cannot remember information from a long time ago and cannot consider any future input for the current state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sQjhyo0pTZ-D_suH.png" /><figcaption>Simple visualization of Recurrent Neural Network structure</figcaption></figure><h3>Training Datasets for Deep Learning</h3><p>There are a lot of training datasets that you can find online and can make use of in your Deep Learning tasks and projects be it image datasets, natural language processing datasets, audio/speech datasets, etc. Some examples of such datasets include:</p><ol><li><a href="http://yann.lecun.com/exdb/mnist/">MNIST Dataset</a> can be used for handwritten digit recognition. It consists of 60,000 training samples and 10,000 test samples of handwritten digits.</li><li><a href="https://cocodataset.org/#home">MS COCO</a> is large-scale object detection, segmentation, and captioning dataset.</li><li><a href="http://ai.stanford.edu/~amaas/data/sentiment/">IMDB Reviews</a> is a dataset regarding movies and is used for binary sentiment classification.</li><li><a href="https://wordnet.princeton.edu/">WordNet</a> which is a large dataset containing English synsets which are groups of synonyms that describe different concepts</li><li><a href="http://millionsongdataset.com/">Million Song Dataset</a> is a huge audio dataset containing popular music tracks.</li></ol><p>Apart from finding training datasets online, you can only go for ready to use datasets offered by Machine Learning libraries like <a href="https://www.tensorflow.org/">TensorFlow</a> and <a href="https://scikit-learn.org/">Scikit-learn</a>. Such libraries allow you to download preprocessed datasets and load them directly into your programs for use.</p><h3>What if you can’t find a suitable dataset from the previous list?</h3><p>Finding the proper dataset just suitable for your needs is quite difficult, and sometimes not even possible. It is not only difficult to gather, but also difficult to have it pre-processed for your specific needs. Often, it is more efficient to find another service that does these laborious works for you. For this, we could be your perfect solution!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*8jfpQNixWRLTmYCy.png" /><figcaption><a href="https://selectstar.ai/">selectstar.ai</a></figcaption></figure><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity on time. Moreover, our in-house managers double-check the quality of the collected or processed data. Let it be your professional dataset or academic dataset. We are here to help!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a91b27babfa3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is My Dataset Good? Checking the Quality of your Collected Data]]></title>
            <link>https://selectstar-ai.medium.com/is-my-dataset-good-checking-the-quality-of-your-collected-data-a0087f276d9?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0087f276d9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[diversity-in-tech]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Sat, 10 Oct 2020 06:05:20 GMT</pubDate>
            <atom:updated>2020-10-10T06:05:20.231Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we will be discussing the different ways to check the quality of your dataset and determine whether is good or bad. We are all well aware of the fact that in order to effectively use a Machine Learning (ML) or AI model, it is imperative to have high-quality training data for the model itself. No matter how efficient or accurate the model is, if it is provided with and trained upon a poorly constructed dataset, it will never produce the desired or correct output. The goal of this tutorial is to discuss and devise a number of ways in which one can ensure that the data that he/she has collected for different ML and AI projects are of high quality. We will also be suggesting a number of techniques that you can use at an individual or organizational level to drastically improve the quality of your datasets. So let’s begin!</p><h3>Why Check the Quality of a Dataset?</h3><p>As we discussed before, having a good quality dataset is of utmost importance in achieving the highest possible accuracy of your model. Moreover, it is paramount that the dataset is processed in such a way that the model can make complete sense of the information that is provided. In short, if you do not provide the right data to your ML or AI models, you will not only get wrong but also, at times disastrous results. Consider the example of Autonomous Vehicle driving. If a Computer Vision algorithm from your vehicle is trained with incomplete or poor data, it will not only produce incorrect outputs but can also put human lives at stake. Now can you get the idea of how important it is to have a good quality dataset? Good. Now let’s see how we can keep a dataset’s quality in check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/0*3gnFW5hLlqO75NdL.jpg" /></figure><h3>General Methods for Checking a Dataset’s Quality</h3><p>The methods to check a dataset’s quality varies from project to project as ML and AI projects can be very different from each other. Thus, quality assessments are customizable according to the specific needs of the project itself and to meet certain goals and objectives. The following methods, where applicable, can be used to assess the quality of one’s data:</p><ol><li><strong>Setting a Benchmark<br></strong>You can measure the accuracy of your dataset by <em>comparing it with the dataset which is already carefully annotated</em>. This way of setting a benchmark through a vetted example and comparing your collected data with it can help you to see exactly how much of your data differs or deviates from the benchmark. You measure how much a set of annotations from a group or individual meet or miss the benchmark. You can even use different mathematical formulas to get an exact number or percentage.</li><li><strong>Consensus Method<br></strong>The Consensus Method, also known as the Overlap Method, is a way of checking the consistency of data within a group; it is quite common in assessing a dataset’s quality. In this method, the sum of the total number of consistent or agreeing data annotations is divided by the total number of annotations to calculate the accuracy.</li><li><strong>Audit Method<br></strong>Here, you get the <em>data labels reviewed by an external party of experts</em>. In this method, usually, the auditors keep on continuing the process of reviewing data levels until a certain level of accuracy is met. It can be highly accurate and specifically useful; yet, it is extremely expensive in both time and budgets.</li><li><strong>Data Monitoring<br></strong>In this method of quality assurance, the project management team constantly monitors the data on a monthly, weekly, or at times on a daily basis to check its quality. Again, it is quite expensive to manage such a team.</li><li><strong>Hybrid Approach</strong><br>This is a multi-layered approach towards quality assurance in which multiple different methods like the ones mentioned above are applied altogether to measure data quality.</li></ol><h3>Suggestions to Improve Data Quality</h3><p>To improve the quality of your dataset on a smaller or even a larger level, you can adopt different ways and techniques. On a smaller or individual scale, you can focus on the following steps:</p><ul><li><strong>Planning</strong><br>Before actually collecting the data itself, it is very important for you to <em>understand and assess the context of the problem that needs to be solved</em> and then decide upon the best possible route to build a dataset for the problem in hand. For instance, if the problem that you are trying to solve is quite common, says a handwritten digit classification problem, you can easily find many sources for open datasets e.g. MNIST in this case.</li><li><strong>Gathering Data<br></strong>While gathering data, it is important for you to <em>consider diversity in the dataset</em>. For example, if you are collecting images for image dataset needed to solve a classification problem, you have to make sure that your training dataset is as varying and diverse as possible by taking pictures from different angles, under different conditions, changing the object sizes, the distance of the camera lens from the object, etc. The same concepts of diversity can apply to other types of data as well.</li><li><strong>Data Filtering</strong><br>To have a dataset of high quality, it is important for you to <em>filter your dataset</em>. In this process, you get rid your dataset of duplicate data e.g. in the case of image datasets, you should delete duplicate images and poor quality data i.e. images that are low in resolution, too small in size, etc. This manual pruning is at times inevitable to improve the quality of your dataset.</li><li><strong>Some other factors<br></strong>To ensure that your dataset is of high quality, you need to consider a number of other factors as well. These include <em>choosing the right amount of data</em>. As a general rule, for Machine Learning projects, the size of the dataset should be at least 10 times the number of features per class. In the case of Deep Learning projects, the size should be at least 100 times the number of features per class. The data should also be <em>well balanced </em>i.e. the sample quantities should be equally distributed among classes. The samples should be representative of real situations where you are going to apply your model. Lastly, the samples should <em>have as maximum variety and diversity </em>as they possibly can.</li></ul><p>On a larger scale i.e. at an organizational level, to ensure high-quality data, you need to actively build diversity into the data teams by defining goals, metrics, and roadmaps and algorithms used to develop datasets. You should also use the right tools to do so, as, without proper tools, you cannot do a good job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*uOlJM3NgA0YWbhUn.jpg" /></figure><h3>Conclusion</h3><p>In this tutorial, we started off by talking about how important it is to have a dataset that meets a certain standard of quality, as without setting a boundary for quality, your model is severely handicapped and cannot do its job properly. We then discussed the different practices that you can follow to test the quality of your dataset for different ML or AI learning projects. These include methods like setting a benchmark to compare your data items to, using the consensus method, the audit method, continuous data monitoring, and a combinational approach of the aforementioned methods. Lastly, we suggested different ways in which one can improve data quality by focusing on the planning phase of data collection, filtering the collected data, choosing the right amount of data and including diversity in the dataset, etc. In a nutshell, keeping a check on your dataset quality and continuously looking for and implementing ways to improve is what makes you effectively solve a problem through ML or AI.</p><h3>Need the Best Quality Dataset? Let us HELP you out!</h3><p>Creating and maintaining the best quality dataset is not an easy task. Thinking about and maintaining all the things mentioned above is quite a burden. Especially, for a small- to medium-sized companies, managing human resources and technical specialties are very challenging. Therefore, it is often more efficient to find another service that does laborious works (including both collection and preprocessing) for you. For that, we could be your perfect solution!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*a8Lpc0ljevgpPAX0.png" /><figcaption><a href="https://selectstar.ai/contact">selectstar.ai</a></figcaption></figure><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity simultaneously. Moreover, our in-house managers double-check the quality of the collected or processed data! Check us out at <a href="http://selectstar.ai/">selectstar.ai</a> for more information! Let us be your HELP!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0087f276d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Captcha, and how does it collect data?]]></title>
            <link>https://selectstar-ai.medium.com/what-is-captcha-and-how-does-it-collect-data-8f6e2ce60416?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f6e2ce60416</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[captcha]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Mon, 28 Sep 2020 09:52:33 GMT</pubDate>
            <atom:updated>2020-09-28T09:52:33.255Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we are going to take a look at <strong>CAPTCHA</strong>s and how they work. Many of you may or may not be familiar with the term ‘<strong>CAPTCHA</strong>’ itself, but would have definitely come across it at some point while surfing the internet. Let me remind you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*HB7NCNNdveegFSsp.png" /><figcaption>Now, it looks familiar, isn’t it?</figcaption></figure><p>You MUST have seen an image similar to this while trying to create an account on a website, posting a comment on a blog, or while purchasing something online. Tests similar to the image above are called <strong>CAPTCHA</strong>s, short for <em>Completely Automated Public Turing Tests to Tell Computers and Humans Apart</em>. So in this tutorial, we will be discussing what <strong>CAPTCHA</strong>s are, the main concepts behind these tests, and how they work in general. We will talk about the different types of <strong>CAPTCHA</strong>s, as well as some of the applications of <strong>CAPTCHA</strong>s, an immensely important one being data collection for Machine learning or Artificial Intelligence projects. So without further ado, let’s get started!</p><h3>What is a CAPTCHA, and Why is it needed?</h3><p>Basically, <strong>CAPTCHA </strong>is a test that is supposed to differentiate between human beings and computers based on traits or characteristics that are possessed by only humans but not by machines. While performing an activity such as signing up on a website, you are required to type in a sequence of characters, letters, or digits that are given or select a particular set of images, e.g., that of cats, to prove that you are actually a human and not a machine. This is because machines cannot make out a distorted or puzzled text while humans can. For the visually impaired, there is Audio <strong>CAPTCHA</strong>, which includes garbled spoken letters that cannot be identified or understood by machines. All of this is done to prevent bots or automated computer programs from getting unauthorized access to an application. <strong>CAPTCHA</strong> stops from adding spans and malicious URLs to an application’s database, hampering an application’s performance, or endangering its security in any other way. In short, the goal is to stop anyone from trying to exploit any weaknesses or shortcomings of a system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yO8qyu49TBgmTuEo.png" /></figure><h3>Different types of CAPTCHAs</h3><p>There are several different types of CAPTCHAs. We will introduce the six most popular ones here.</p><p><strong>1. Character or Text Recognition</strong></p><p>This is the most common type of <strong>CAPTCHA </strong>where you are given a sequence of characters with a slight variation or distortion, and you are supposed to type them in the given input field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*Qb_HRisf8FeOmCdZ.jpg" /><figcaption>Example of character or text recognition</figcaption></figure><p><strong>2. Image Recognition</strong></p><p>Image recognition <strong>CAPTCHA </strong>consists of different forms of image tests which can include naming images, distinguishing images from a set of different images, as well as identifying an odd image out of the set. This type of <strong>CAPTCHA </strong>makes use of the bots’ weaknesses in solving image recognition related problems which is a human’s forte.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/0*AfbMPF8sL-Ff3rN9.png" /><figcaption>Example of image recognition</figcaption></figure><p><strong>3. Audio CAPTCHA</strong></p><p>As mentioned before, this type of <strong>CAPTCHA </strong>is for the visually impaired. It mostly consists of audio of spoken letters or numbers which are slightly distorted in order to puzzle a bot but easy for a human to comprehend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/0*Jn9WKLKUHUEuiKnh.png" /><figcaption>Several forms of CAPTCHA support the audio type as well</figcaption></figure><p><strong>4. Social media login</strong></p><p>While trying to register or create an account on a website, you are sometimes asked to sign up using your social media accounts e.g., Facebook account, Gmail account, etc. This is done because bots do not have social media accounts, so this method prevents them from registering. It also saves the authentic user’s time as he/she does not need to create an account from scratch and can instead use an already created social media account. As the accounts are genuine, this method of registering helps to increase a site’s security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*EgXQo3CVZf5LyjAD.jpg" /><figcaption>Example of social media login</figcaption></figure><p><strong>5. Math problem</strong></p><p>A form appears with a simple math problem, requiring you to solve for the answer. Most of the questions involve simple arithmetics e.g. 8+2, 11–9, etc. These can be quite tricky for robots to solve because it has to perform not only image recognition but also a multiple semantic analysis of numbers and symbols to solve the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*Uvk3ii5ul1bHlruc.jpg" /></figure><p><strong>6. Time-based CAPTCHA</strong></p><p>Recording the amount of time taken to solve a form and differentiating between a bot and a human on the basis of that time is another type of <strong>CAPTCHA</strong>. Bots mostly fill out all the fields in a form immediately whereas humans naturally take some time to enter the information.</p><h3>How does CAPTCHA work?</h3><p><strong>CAPTCHA </strong>works on the basis of analyzing the variations or differences between humans and automated computer programs. <strong>CAPTCHA </strong>is supposed to be 3 things:</p><ol><li>Easy for a human to solve</li><li>Hard for a bot to solve or understand</li><li>Easy for a tester machine to create and grade</li></ol><p>The different types of <strong>CAPTCHA</strong>s as mentioned above give tasks to solve, which are challenging for computers but not so much for humans. A <strong>CAPTCHA</strong> test makes use of humans’ strengths in the domains of invariant recognition, segmentation, and context. We all know that humans can identify characters even if they are in different shapes or forms, arranged in a different pattern, etc. Even if a character is overlapping with another, a human being can segment the characters and understand them in their proper context, which a computer finds quite tough to perform simultaneously.</p><h3>Data Collection and CAPTCHA</h3><p>Apart from differentiating between a human and a computer, a <strong>CAPTCHA </strong>can also help to gather data to train machine learning models. You could very well consider it a crowdsourcing technique for data collection and annotation. Think about it, the information gathered from e.g. image recognition <strong>CAPTCHA </strong>is a way of using human intelligence to annotate a dataset. When a user selects all the images which contain or show a cat, he or she is in fact helping to build an annotated dataset of cat images. The same is the case for text recognition as well. Thus, <strong>CAPTCHA</strong>s are not only effective in ensuring the security of a website but are also a useful method to create fully annotated datasets for Machine Learning or AI models.</p><h3>Other Types of Data Collection Methods?</h3><p>To train the industry-level algorithms, companies need data collection &amp; annotation, and they are often very challenging. Moreover, it is difficult to control the quality within a company, especially your company is a small- or medium-sized company. Therefore, it is often more efficient to find another service that does laborious works for you. We could be your perfect solution!</p><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity on time. Moreover, our in-house managers double-check the quality of the collected or processed data. If you need data? If you need preprocessed data? Let us know!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*62hcCvj-y6BWlWXx.png" /><figcaption><a href="https://selectstar.ai/">selectstar.ai</a></figcaption></figure><h3>Advantages of CAPTCHA</h3><ol><li>Data collection and annotation to create large annotated datasets for Machine Learning models.</li><li>Prevention of fake registrations on a website by allowing Social Media logins. To prevent fake registrations, big websites like Facebook and Gmail are integrated with <strong>CAPTCHA</strong>.</li><li>Prevention of spam comments by only allowing humans to post a comment. This is done to prevent spammers to wrongfully raise their websites’ search engine ranks by bombarding their websites’ comments section and leaving fake reviews of products.</li><li>Increasing the security of online purchasing or shopping by ensuring that the buyers are only humans. At times, competitors of business may use invalid names, emails, shipping addresses to order your products so that you waste your time and money to deliver those products.</li></ol><h3>Conclusion</h3><p>To sum it all up, we started off this tutorial by discussing what <strong>CAPTCHA</strong>s really are and the reasons why they are needed i.e. to tell a human user apart from a bot and in a nutshell, to prevent someone from exploiting a site or application’s weaknesses, etc. We talked about how a <strong>CAPTCHA </strong>plays with a human’s strengths and a computer’s weaknesses, we then touched upon the different types of <strong>CAPTCHA</strong>s like text, audio, image, etc. We also learned that besides ensuring a site’s security, the information gathered through <strong>CAPTCHA</strong>s can be an effective way of involving human intellect to create a full-fledged dataset for AI or ML. Lastly, we talked about the different advantages or usefulness of <strong>CAPTCHA</strong>s overall.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f6e2ce60416" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Try Out Simple Image Classification (at home!!)]]></title>
            <link>https://selectstar-ai.medium.com/try-out-simple-image-classification-at-home-87e95c1820ce?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/87e95c1820ce</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[crowdsourcing]]></category>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Fri, 04 Sep 2020 07:09:18 GMT</pubDate>
            <atom:updated>2020-09-04T07:09:18.870Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we will talk about image classification, using one of the most used datasets, THE <a href="http://yann.lecun.com/exdb/mnist/"><strong>MNIST</strong></a>. We will write a Python code to do so and will also discuss ways in which we can improve the classification accuracy!</p><p>Image classification, in general, is the process of finding patterns in images based on contextual information contained within the images. Based on that information or features, the model learns to recognize what an image represents in general. One such example is of Handwritten Digit Recognition. Most of the time, it is relatively easy for a human being to understand what someone else has written, but the same does not hold for a machine. A machine needs to ‘<strong><em>learn</em></strong>’ what information is contained in an image of a handwritten text so that it can later identify that text. This identification in images is made based on the pixel values. For handwritten digit classification, the first significant source of data that comes to mind is the MNIST dataset, as it is the most popular dataset for this domain. So without further ado, let’s learn how to implement handwritten digit classification and also discover ways in which the accuracy of the classification can be improved!</p><h3>About MNIST dataset</h3><p>Modified National Institute of Standards and Technology, known as MNIST in short, is a dataset containing handwritten images of digits from 0 to 9 (made by the guru, Professor <a href="http://yann.lecun.com/"><em>Yann LeCun</em></a>, and his colleagues). To be more precise, the dataset contains a total of 70,000 square, grayscale images of dimensions 28 x 28 pixels each. Each image comprises 784 features, and each feature corresponds to a pixel’s intensity, which is from 0 to 255. Out of the total 70,000 images in the dataset, 60,000 are for training, and the remaining 10,000 images are for testing. The total number of classes is 10, signifying the output i.e., the digits from 0 to 9 inclusive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9SK3g4d3AQtlgFYN.png" /><figcaption>THE MNIST database of handwritten digits</figcaption></figure><h3>Prerequisites</h3><p>Before you move ahead with this tutorial, you need to have basic programming knowledge in any language, preferably in Python 3, as well as some understanding of Machine Learning and Deep Learning concepts. Our code will be using <a href="https://colab.research.google.com/notebooks/intro.ipynb">Google Colab</a>, but you can work on any code editor of your choice. We will be using Keras.</p><h3>Python Code for Handwritten Digit Classification</h3><p>Step 1: Import essential libraries and packages</p><pre>import tensorflow<br>import tensorflow.keras<br>from keras import models<br>from keras import layers<br>from keras.utils import normalize<br>from keras.utils import to_categorical<br>import matplotlib.pyplot as plt</pre><p>Step 2: Load the MNIST dataset</p><p>Since the MNIST dataset is hugely popular, it can easily be easily accessed from different sources. Even <a href="https://www.tensorflow.org/">TensorFlow</a> and <a href="https://keras.io/">Keras</a> allow us to download it through their respective API directly. Thus, we will download it through Keras’ API.</p><pre>mnist_dataset = tensorflow.keras.datasets.mnist<br># Split the data into training and test sets.<br>(train_X, train_Y), (test_X, test_Y) = mnist_dataset.load_data()<br># train_X represents the pixel values (feautures) of the 28 x 28 image.<br>print(train_X[3])<br>plt.imshow(train_X[3],cmap=plt.cm.binary)<br>plt.show()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MnQ3WJ491B-8Mh_r.png" /></figure><p>Step 3: Build the Model</p><p>We have decided to go for a 4-layered model i.e., having two hidden layers and an input and output layer. The rectifier activation function is chosen for the neurons in the hidden layers.</p><pre># 4-layered model<br>model = models.Sequential()<br># Hidden layer containing 512 units. Rectifier linear function is used as the activation function. <br>model.add(layers.Dense(512, activation=&#39;relu&#39;, input_shape=(28 * 28, )))<br>model.add(layers.Dense(128, activation=tensorflow.nn.relu))<br># Dropout is a regularization technique that is used to reduce overfitting.<br>model.add(layers.Dropout(0.2))<br># Output layer which is 10-way softmax layer, which returns an array of 10 probability scores displaying the probabability of image of the digit belonging to which of the 10 classes.<br>model.add(layers.Dense(10, activation=&#39;softmax&#39;))</pre><p>Step 4: Normalize the data i.e., change the current data range from 0 to 255 to 0 and 1 inclusive.</p><p>It is considered a good practice to scale input values when it comes to neural networks. Thus we will normalize our data.</p><pre>train_X = normalize(train_X, axis=1)<br>test_X = normalize(test_X, axis=1)<br>print(train_X[3])<br>print(&quot; &quot;)<br>print(&#39;Shape of train_X: &#39;, train_X.shape)<br>print(&#39;Number of images in train_x: &#39;, train_X.shape[0])<br>print(&#39;Number of images in test_X: &#39;, test_X.shape[0])</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bH-UGJWdM8AwxiPd.png" /></figure><p>Step 5: Flatten each 28 x 28 image to 1 x 784 image</p><p>The training dataset is structured to be a 3-dimensional array. To be able to use the dataset in Keras API, we need 4-dimensional NumPy arrays. Therefore, we need to flatten or reduce the images so that they represent a vector of pixels. In this case, since the image is 28 x 28, there will be 784-pixel values.</p><pre>train_X = train_X.reshape(train_X.shape[0], -1)<br>test_X= test_X.reshape(test_X.shape[0], -1)<br># Encode labels<br>train_Y = to_categorical(train_Y)<br>test_Y  = to_categorical(test_Y)<br>print(&#39;Shape of train_X: &#39;, train_X.shape)<br>print(&#39;Number of images in train_x: &#39;, train_X.shape[0])<br>print(&#39;Number of images in test_X: &#39;, test_X.shape[0])</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*EtE8XFkbzspzri4O.png" /></figure><p>Step 6: Train the model</p><p>We will now fit the model.</p><pre>model.fit(train_X, train_Y, epochs=7)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v8BX6agvMPAlyeqE.png" /></figure><p>Step 7: Evaluate the model accuracy on test data</p><p>We will now evaluate the accuracy of the model.</p><pre>_, test_accuracy = model.evaluate(test_X, test_Y, verbose=0)<br>print(f&quot;\nTest Accuracy : {test_accuracy * 100} %&quot;)</pre><p>Output:</p><p>97.85% seems quite great? Isn’t it? No, never satisfy! In the next section, we will discuss how to improve it even more!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*cJJY8bpyJYuiN1aH.png" /></figure><h3>Improving Accuracy of Image Classification with MNIST</h3><p>We say that the accuracy of our model is 97.85%. The question that arises is, can we still improve the accuracy? The answer, yes, we can! We can improve the accuracy of the image classification by altering our model as well as our dataset. For a start, we can increase the number of layers in our neural network. The added layers can enable a neural network to learn a more complex classification function that can possibly improve classification performance and accuracy. Changing and trying out different kernel sizes and activation functions can also lead to better results.<br>Where the dataset itself is concerned, we can firstly expand our dataset, and this is the easiest solution. The sizes of images in the dataset can also tamper with i.e., increasing or decreasing their sizes according to need. Another fundamental way in which the image classification accuracy can significantly increase is by Data Augmentation. This is basically the practice of adding synthetic data to your dataset by flipping images, adding noise and other distortions, etc. One way is that we can shift each image by one pixel in all of the 4 directions (left, right, up, down) i.e. by [-1,0], [1,0], [0,1], [0, -1] respectively. We can also shift the images diagonally and include these shifted images in the dataset. In this way, not only will we have more data, but also more variation in that data, which can help to boost the accuracy of classification.</p><h3>Conclusion</h3><p>To sum it all up, we started off by discussing what image classification in general is and how it is carried out by a model i.e., based on the pixel values in an image. We also talked about the very popular MNIST dataset, which is basically a dataset of handwritten digits and is used commonly in the handwritten digit classification problem. We thoroughly discussed the details and particulars of this dataset and went through each step of handwritten digit classification in Python. Lastly, we talked about how we can improve the test accuracy by introducing some changes in the model as well as in the dataset, for example, by creating an augmented dataset.</p><h3>Do you need a dataset like MNIST?</h3><p>Having various types of datasets like MNIST can be helpful for your personal studying, especially under current lock-down environment. However, datasets like MNIST is challenging to compose and are not accessible easily; because it is challenging not only to gather, but also to have it pre-processed just for your specific needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*Gnt95CeoqO88biWl.png" /><figcaption><a href="https://www.selectstar.ai/">selectstar.ai</a></figcaption></figure><p><a href="https://www.selectstar.ai/"><strong><em>SelectStar </em></strong></a>has been working with big firms to establish such open datasets, like <a href="https://korquad.github.io/">KorQuAD </a>sets (The Korean Question Answering Dataset). Here, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity on time. Moreover, our in-house managers double-check the quality of the collected or processed data. Let it be your professional dataset or academic dataset!</p><p>Just focus on your works. <strong><em>SelectStar</em></strong> got your back!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87e95c1820ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Problems of Online Image Crawling]]></title>
            <link>https://selectstar-ai.medium.com/problems-of-online-image-crawling-66fe548df4f7?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/66fe548df4f7</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[crawling]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Fri, 24 Jul 2020 12:37:53 GMT</pubDate>
            <atom:updated>2020-07-24T12:37:53.756Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we are going to talk about online image crawling and some of the common problems that we come across while doing so. Online image crawling is basically a way of gathering large amounts of images through different websites. In short, we scrape images from different web pages, store them in a folder, drive or any desired location, and can use those images for different purposes, like for building an image dataset for a machine learning model. So in this tutorial, we will be discussing two ways of online image crawling, one through a Google Chrome extension named <a href="https://chrome.google.com/webstore/detail/fatkun-batch-download-ima/nnjjahlikiabnchcpehcpkdeckfgnohf?hl=en">Fatkun Batch Download Image</a> and another by writing Python script to scrape images from a web page. The main topic that we will be going through is the common problems incurred while scraping images from the web. So let’s get started!</p><h3>Scraping Images from the Web</h3><h4>Google Chrome’s Fatkun Batch Download Image Extension</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XAFWD-F7_7qb7y9b.png" /></figure><p>Fatkun Batch Image download is a simple and useful image download extension. As its name suggests, it allows you to download images in a batch from a website and use them for different purposes. You can apply search filters, select and deselect images, opt for a particular tab or all tabs for image download, rename images in batch, and can also vary image sizes before downloading them. For more about this extension i.e. for installing it, setting it up, and using it, you can read the ‘Downloading images through the Fatkun Batch Download Image extension’ section in our <a href="https://mc.ai/creating-the-best-quality-image-dataset/">Creating the Best Quality Image Dataset</a> article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*99rGiCnuXnIoXrX4.png" /></figure><p>Drawbacks of this extension:</p><ul><li>Tedious when it comes to collecting images for large datasets.</li><li>Images collected are thumbnails and not original size images.</li></ul><h4>Image Crawling in Python</h4><p>There are multiple Python packages and libraries that can help you to scrape images from a website. These include Beautiful Soup, Selenium, Scrapy, etc. We will be talking about <a href="https://scrapy.org/">Scrapy</a> which is basically a framework written in Python and designed for web scraping. It can also be used to extract data using APIs or just as a general-purpose web crawler. For more details on this framework and for full-fledged implementation of it for large-scale image scraping for datasets, you can have a look at our <a href="https://mc.ai/fuel-up-the-deep-learning-custom-dataset-creation-with-web-scraping/">Custom Dataset Creation with Web Scraping</a> article.</p><h4>Problems of Online Image Crawling</h4><p>Online image crawling or scraping is an important aspect for gathering relevant images from different websites especially when it comes to building high-quality image datasets for different machine learning models to be trained on. However, scraping images from the web is not without its challenges. The context of the internet changes with each passing day, making it complicated and at times almost impossible to successfully gather images from different websites. Even if you succeed in doing so, the performance of the web scraper may be seriously compromised. So it’s extremely important to consider certain factors before diving into the scraping process itself. Let’s have a look at some of the common problems that one could come across while scraping images online:</p><p>1. Access to scraping:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*kQ61dF9AOc-b5Kqe.png" /></figure><p>Before you plan on scraping images or other content from a particular website, it is important for you to ensure that the target website permits scraping. This is because many applications restrict access through their robots.txt file. A robots.txt file basically tells search engine crawlers which pages or files the crawler can or cannot request from your site. In case, if access for image crawling has been denied to you, you can through a formal or informal route, contact the owner of the website, explain your situation and request him/her for access. If that does not work out, you can look for other websites with similar content and hope for the best.</p><p>2. Anti-scraping policies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*5zQ6jpbn8aNkgMRV.png" /></figure><p>Another common problem which you can come across while trying to scrape images from a website is its anti-scraping policy. For instance, to prevent the scraping of their content, many websites make use of IP blocking. This is a way of preventing someone from scraping the content of your website by banning the scraper’s IP address or restricting it to slow down the scraping process. This occurs when the target website detects a high number of requests coming from the same IP address, which is usually the case in online image crawling since mostly a large number of images are being scraped at one time. The website considers it as a malicious activity and retorts to IP blocking. One fine solution to this problem is <a href="https://www.octoparse.com/tutorial/octoparse-cloud-service">Octoparse Cloud Service</a> which uses multiple IP addresses to scrape one website at the same time and thereby prevents IP blocking.</p><p>3. Diversity in website structures and layouts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r6tpDGscXmqCPYRm.png" /></figure><p>Web scrapers have some generic limitations that cannot be overcome. Each web scraper is tailored to perfectly suit one particular website and cannot be used for every other website to scrape images. This is because websites have different structures, characteristics. formats and layouts. So there is no general web scraper which fits all websites, rather particular ones for each website. It is also common for not only websites but also webpages to have different structures and layouts from each other. This is because web page designers design different web pages according to their own tastes and standards. This also makes online image crawling an extremely tedious task considering the fact that you have to make changes to your web crawler in accordance with different websites or web pages in order to scrape images from them.</p><p>4. Continuously changing website content:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DeVvAesqa-qd6abl.png" /></figure><p>Many websites update their content continuously by adding new features, removing unnecessary features, making certain layout and design changes, etc. This is done to improve user experience but can greatly affect the performance of the web scraper. Since each web scraper is specific to a website, any change in that website in turn is a call for changes in the implementation of the web scraper as well. Even if the website change is extremely minor, it might require you to adjust the scraper accordingly and that can be challenging at times. Octoparse Cloud Service again helps in visualizing these changing structures so that the image crawler can be altered accordingly.</p><p>5. Bad Image Quality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n2fDAl8DrkNSo2-n.jpg" /></figure><p>Image quality is an extremely important factor especially when we are talking about building image datasets. The quality of your scraped images can be seriously compromised due to any technical weaknesses or shortcomings of your web scraper. Therefore, it is pivotal for you to choose a high-quality web scraper with a lot of good features to do the job.</p><p>6. Slow loading of Website:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*8nKepMvYK7toFtBJ.jpg" /></figure><p>Websites may respond very slowly or even fail to load because they may be receiving too many access requests which can be a serious problem in online image crawling.</p><p>7. Geographical restrictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/0*82IH33bKfTzRbQQb.jpg" /></figure><p>At times, the biggest hindrance to online image crawling can be your location, since some websites may not be accessible or may not permit scrapping of their content in certain regions or countries.</p><h3>Conclusion</h3><p>To sum it all up, we started off by discussing what online image crawling is and then briefly touched upon how it can be done through Fatkun Batch Image Download extension and by using Python’s Scrapy framework. We discussed the common problems of online image crawling in general like the bad quality of data, anti-scraping policies etc. We suggested solutions for some of them as well and learned that while some of the problems were easily solvable, others were not.</p><h3>Outsourcing your labors</h3><p>Online image crawling is often not enough for companies to train their industry-level algorithms. Moreover, it is difficult to control the quality within a company, especially your company is a small- or medium-sized company. Therefore, it is often more efficient to find another service that does laborious works for you. We could be your perfect solution!</p><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity on time. Moreover, our in-house managers double-check the quality of the collected or processed data. If you need data? If you need preprocessed data? Let us know!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*kgSzo-agjNozE_Zv.png" /><figcaption><a href="http://selectstar.ai/">selectstar.ai</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66fe548df4f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Diversity? Accuracy? Important Properties of your Dataset]]></title>
            <link>https://selectstar-ai.medium.com/diversity-accuracy-important-properties-of-your-dataset-e8b3072b29d6?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8b3072b29d6</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Mon, 20 Jul 2020 04:10:28 GMT</pubDate>
            <atom:updated>2020-07-20T04:10:28.597Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we are going to talk about some of the important qualities that differentiate a good dataset from a bad one. We know that a dataset is basically a collection of data. It can consist of tables where each column in a table represents a particular variable in question and each row represents a value of that variable. A dataset can also consist of various documents or files. Regardless of the format, having a good quality dataset is extremely important because it is directly linked to the sustainability of your algorithms like the Machine Learning model i.e. the model will fail to serve its purpose if the quality of the dataset is poor or not up to the mark. So in this tutorial, we will be discussing some of the characteristics that a good dataset should possess. So let’s get started!</p><h3>Prerequisites</h3><p>Before you go ahead, please note that there are a few prerequisites for this tutorial. To follow the code samples given, you should have some basic programming knowledge in any language (preferably in Python). You should also be familiar with basic machine learning concepts. We will be using <a href="https://colab.research.google.com/notebooks/intro.ipynb">Google Colab</a> for writing the code in our examples but you can work on any code editor of your liking.</p><h3>Characteristics of a Good Quality Dataset</h3><p>A dataset is of high quality if it fulfills its purpose and satisfies its requirements of use by the application or the client. A good machine learning model is of no use if it is trained on poor quality data. Therefore, it is vital to have a dataset of high quality. A good quality dataset should ideally have the following propoerties:</p><h4>1. High Accuracy:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xg4bDCqTanCZUcsI1XPWBQ.png" /></figure><p>Accuracy refers to the correctness of the data. To be more precise, it refers to how close a value is to the actual value that represents the problem correctly. To analyze the accuracy of the information in your dataset, you need to ask yourself whether the information correctly reflects upon the concerned situation or problem or not. For example, if you are dealing with a dataset of houses and some of the house sizes are given in square centimeters or if some of the house prices are negative, then that information is most likely to be inaccurate.</p><h4>2. Reliability:</h4><p>Reliability is an important data attribute and it basically ensures that the different dataset values or datums do not contradict each other and that overall, the information that the dataset contains can be trusted; it discusses the qualitative aspects of your dataset. A model trained on reliable data is more likely to make correct predictions than one trained on unreliable data. When measuring or determining the reliability of your dataset, you should make sure that it does not contain:</p><p><strong>Duplicated values</strong></p><p>As they are repeated values, they hold no importance as they do not provide any new information in the dataset and should be removed. You can remove duplicate values in Python using Pandas’ drop_duplicates function. Consider the example below which contains information about students in some universities and has some duplicates.</p><pre># FirstName LastName     Sex  Age Degree  Graduation<br>0     Jamie   Fallon    Male   20     SE        2019<br>1      Erin   Silver  Female   23     EE        2020<br>2      Phil   Rhodes    Male   19     ME        2021<br>3     Jamie   Fallon    Male   20     SE        2019<br>4      Erin   Silver  Female   23     EE        2020<br>5     Jamie   Fallon    Male   22     SE        2020 </pre><p>To remove duplicate rows:</p><p><strong>Python Code:</strong></p><pre>new_info = info.drop_duplicates()<br>print(new_info)</pre><p><strong>Output:</strong></p><pre># FirstName LastName     Sex  Age Degree  Graduation<br>0     Jamie   Fallon    Male   20     SE        2019<br>1      Erin   Silver  Female   23     EE        2020<br>2      Phil   Rhodes    Male   19     ME        2021<br>5     Jamie   Fallon    Male   22     SE        2020</pre><p>To remove duplicates on the basis of columns, specify a subset or column that should be unique. In our example, there are 2 Jamie Fallons. If we want to remove the record of one who graduates later, we could do so by using the following code:</p><p><strong>Python Code:</strong></p><pre>info = info.sort_values(&#39;Graduation&#39;, ascending=True)<br>info = info.drop_duplicates(subset=&#39;FirstName&#39;, keep=&#39;first&#39;)<br>print(info)</pre><p><strong>Output:</strong></p><pre># FirstName LastName     Sex  Age Degree  Graduation<br>0     Jamie   Fallon    Male   20     SE        2019<br>1      Erin   Silver  Female   23     EE        2020<br>2      Phil   Rhodes    Male   19     ME        2021</pre><p><strong>Missed or omitted values</strong></p><p>Should also be removed or recoded i.e. represented differently. You can remove omitted values in Python using Pandas’ drop_dropna function. Consider the example below with some missed values:</p><pre>FirstName LastName     Sex  Age Degree Graduation<br>   0      Erin   Silver     NaN   23     EE       2020<br>   1      Phil   Rhodes    Male   19     ME       2021<br>   2     Helen    David  Female   23     EE       2020<br>   3     Jamie   Fallon    Male   22     SE        NaT</pre><p><strong>Python Code:</strong></p><pre>new_info= info.dropna()<br>print(new_info)</pre><p><strong>Output:</strong></p><pre>FirstName LastName     Sex  Age Degree Graduation<br>1      Phil   Rhodes    Male   19     ME       2021<br>2     Helen    David  Female   23     EE       2020</pre><p><strong>Special characters, punctuations, stop words</strong></p><p>In the case of textual data, special characters, punctuation marks and stop words like ‘a’, ‘is’, ‘and’, ‘the’ etc. do not add any meaning to the text. The model does not understand the grammar of the text, rather the nouns and the adjectives used. Thus, they should be removed from your textual data. This is called text pre-processing and for this, you need to:</p><ol><li><strong>Make the necessary downloads and imports.</strong></li></ol><pre>import nltk<br>nltk.download(&#39;punkt&#39;)<br>nltk.download(&#39;stopwords&#39;)<br>from nltk.tokenize import word_tokenize<br>import string<br>from nltk.corpus import stopwords</pre><p>2. <strong>Split the words in the text file into tokens or list items. Consider this text file:</strong></p><p><strong>Python Code:</strong></p><pre>text_file = &#39;text_file.txt&#39;<br>tokens = word_tokenize(text)<br>print(tokens)</pre><p><strong>Output:</strong></p><pre>[&#39;This&#39;, &#39;is&#39;, &#39;@&#39;, &#39;@&#39;, &#39;@&#39;, &#39;@&#39;, &#39;@&#39;, &#39;a&#39;, &#39;random&#39;, &#39;text&#39;, &#39;,&#39;, &#39;file&#39;, &#39;thAt&#39;, &#39;CONTAINS&#39;, &#39;(&#39;, &#39;punctuation&#39;, &#39;)&#39;, &#39;marks&#39;, &#39;,&#39;, &#39;special&#39;, &#39;#&#39;, &#39;characters&#39;, &#39;!&#39;, &#39;!&#39;, &#39;!&#39;, &quot;&#39;&#39;&quot;, &#39;``&#39;, &#39;and&#39;, &#39;spaces&#39;, &#39;is&#39;, &#39;for&#39;, &#39;the&#39;, &#39;purpose&#39;, &#39;of&#39;, &#39;understanding&#39;, &#39;how&#39;, &#39;important^^^&#39;, &#39;text&#39;, &#39;preprocessing&#39;, &#39;is&#39;, &#39;fOr&#39;, &#39;the&#39;, &#39;&amp;&#39;, &#39;quality&#39;, &#39;of&#39;, &#39;a&#39;, &#39;dataset&#39;, &#39;.&#39;, &#39;A&#39;, &#39;high&#39;, &#39;quality&#39;, &#39;dataset&#39;, &#39;has&#39;, &#39;preprocessed&#39;, &#39;&amp;&#39;, &#39;clean&#39;, &#39;text..&#39;, &#39;!&#39;, &#39;!&#39;]</pre><p><strong>3. Remove all punctuation marks and special characters from the list.</strong></p><p><strong>Python Code:</strong></p><pre>intermediate_step = str.maketrans(&#39;&#39;, &#39;&#39;, string.punctuation)<br>punctuation_removed = [a.translate(intermediate_step) for a in tokens]<br>print(punctuation_removed)</pre><p><strong>Output:</strong></p><pre>[&#39;This&#39;, &#39;is&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;a&#39;, &#39;random&#39;, &#39;text&#39;, &#39;&#39;, &#39;file&#39;, &#39;thAt&#39;, &#39;CONTAINS&#39;, &#39;&#39;, &#39;punctuation&#39;, &#39;&#39;, &#39;marks&#39;, &#39;&#39;, &#39;special&#39;, &#39;&#39;, &#39;characters&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;and&#39;, &#39;spaces&#39;, &#39;is&#39;, &#39;for&#39;, &#39;the&#39;, &#39;purpose&#39;, &#39;of&#39;, &#39;understanding&#39;, &#39;how&#39;, &#39;important&#39;, &#39;text&#39;, &#39;preprocessing&#39;, &#39;is&#39;, &#39;fOr&#39;, &#39;the&#39;, &#39;&#39;, &#39;quality&#39;, &#39;of&#39;, &#39;a&#39;, &#39;dataset&#39;, &#39;&#39;, &#39;A&#39;, &#39;high&#39;, &#39;quality&#39;, &#39;dataset&#39;, &#39;has&#39;, &#39;preprocessed&#39;, &#39;&#39;, &#39;clean&#39;, &#39;text&#39;, &#39;&#39;, &#39;&#39;]</pre><p><strong>4. Remove all non-alphabets from the list of tokens.</strong></p><p><strong>Python Code:</strong></p><pre>only_alphabets = [a for a in punctuation_removed if a.isalpha()]<br>print(only_alphabets</pre><p><strong>Output:</strong></p><pre>[&#39;This&#39;, &#39;is&#39;, &#39;a&#39;, &#39;random&#39;, &#39;text&#39;, &#39;file&#39;, &#39;thAt&#39;, &#39;CONTAINS&#39;, &#39;punctuation&#39;, &#39;marks&#39;, &#39;special&#39;, &#39;characters&#39;, &#39;and&#39;, &#39;spaces&#39;, &#39;is&#39;, &#39;for&#39;, &#39;the&#39;, &#39;purpose&#39;, &#39;of&#39;, &#39;understanding&#39;, &#39;how&#39;, &#39;important&#39;, &#39;text&#39;, &#39;preprocessing&#39;, &#39;is&#39;, &#39;fOr&#39;, &#39;the&#39;, &#39;quality&#39;, &#39;of&#39;, &#39;a&#39;, &#39;dataset&#39;, &#39;A&#39;, &#39;high&#39;, &#39;quality&#39;, &#39;dataset&#39;, &#39;has&#39;, &#39;preprocessed&#39;, &#39;clean&#39;, &#39;text&#39;]</pre><p><strong>5. Convert all words to lower-case for consistency.</strong></p><p><strong>Python Code:</strong></p><pre>lowercase_tokens = [a.lower() for a in only_alphabets]<br>print(lowercase_tokens)</pre><p><strong>Output:</strong></p><pre>[&#39;this&#39;, &#39;is&#39;, &#39;a&#39;, &#39;random&#39;, &#39;text&#39;, &#39;file&#39;, &#39;that&#39;, &#39;contains&#39;, &#39;punctuation&#39;, &#39;marks&#39;, &#39;special&#39;, &#39;characters&#39;, &#39;and&#39;, &#39;spaces&#39;, &#39;is&#39;, &#39;for&#39;, &#39;the&#39;, &#39;purpose&#39;, &#39;of&#39;, &#39;understanding&#39;, &#39;how&#39;, &#39;important&#39;, &#39;text&#39;, &#39;preprocessing&#39;, &#39;is&#39;, &#39;for&#39;, &#39;the&#39;, &#39;quality&#39;, &#39;of&#39;, &#39;a&#39;, &#39;dataset&#39;, &#39;a&#39;, &#39;high&#39;, &#39;quality&#39;, &#39;dataset&#39;, &#39;has&#39;, &#39;preprocessed&#39;, &#39;clean&#39;, &#39;text&#39;]</pre><p><strong>6. Remove all stop words from the list.</strong></p><p><strong>Python Code:</strong></p><pre>stop_words = set(stopwords.words(&#39;english&#39;))<br>stop_words_removed = [w for w in words if not w in stop_words]<br>print(stop_words_removed)</pre><p><strong>Output:</strong></p><pre>[&#39;random&#39;, &#39;text&#39;, &#39;file&#39;, &#39;contains&#39;, &#39;punctuation&#39;, &#39;marks&#39;, &#39;special&#39;, &#39;characters&#39;, &#39;spaces&#39;, &#39;purpose&#39;, &#39;understanding&#39;, &#39;important&#39;, &#39;text&#39;, &#39;preprocessing&#39;, &#39;quality&#39;, &#39;dataset&#39;, &#39;high&#39;, &#39;quality&#39;, &#39;dataset&#39;, &#39;preprocessed&#39;, &#39;clean&#39;, &#39;text&#39;]</pre><h4>3. Consistency in Feature Representation:</h4><p>Consistency in feature representation is an important characteristic of a good dataset as it ensures data compatibility. For this, you need to:</p><ul><li>Convert non-numeric features to numeric e.g. to perform matrix multiplication, the data needs to be numeric as such operations cannot be performed on strings.</li><li>Resize inputs to a fixed size especially in the case of image models as they require images in their dataset to be of the same size. This can be done through Python Imaging Library known as PIL in short by using its image resize function.</li><li>Normalize numeric features i.e. change the values of numeric columns to a scale such that the differences between the ranges remain unchanged. The actual range of values is converted to a standard range of values, typically between -1 to 0 or 0 to +1 or -1 to +1. This helps models to perform better and increases overall accuracy.</li></ul><h4>4. Right Dataset Size:</h4><p>The right quantity or size of the dataset in an extremely important characteristic that in turn affects its overall quality. No matter how efficient your model is, the dataset size can become a bottleneck in terms of its accuracy. There is no hard and fast rule about the size of the dataset, as it is specific to the type of problem that you are trying to solve, so size is mostly based on good judgment and should be sufficient to yield expected performance outcomes. You can however ensure that your dataset is at least an order of magnitude more than the trainable parameters. You can also split the training and testing sets in the ratio 80/20 or 70/30 or use an alternative approach like K-fold cross-validation.</p><h4>5. Diversity in Dataset:</h4><p>Diversity is a critical factor when it comes to making a good quality dataset. Consider the example of an image recognition and classification model. If this model is trained on a dataset of images showing different breed of dogs, and for each breed, there are images taken from different angles, under changed lighting conditions, from varying distances, in contrasting backgrounds and showing their tails, paws, etc. differently, then the model is most likely to classify the dogs more accurately as compared to if it is trained on a model which has similar images. In short, non-representative or non-diverse datasets are unlikely to provide useful insights as compared to those who cover all facets of the problem in question.</p><h4>6. Completeness:</h4><p>Completeness refers to how comprehensive the dataset is. This is an important attribute of measuring data quality as it makes sure that every important and relevant piece of information is put into the dataset for the model to train on. If the information is incomplete, the data may become unusable.</p><h4>7. Up to date Data:</h4><p>Having up to date data in your dataset is an important data quality characteristic because if the data is not up to date, it may not be applicable to the current scenario or problem that the model is intended to solve. Take the example of a house price prediction model that predicts house prices based on their sizes in square feet. If the dataset for this contains house prices from the 1960s, it may not be applicable to predict house prices for the year 2020.</p><h4>8. Relevance:</h4><p>This refers to how important or relevant the data is to the concerned problem. If you gather a dataset that contains some irrelevant or unrelated information to the problem that you’re trying to solve, then you will not attain the desired results and waste your time. Your dataset should strictly contain relevant information and should meet the requirements for the intended use.</p><h3>Conclusion</h3><p>To sum it all up, we discussed how important it is for a model to be fed a high-quality dataset as it drives the quality of the overall machine learning model. We pointed out the main characteristics of a good quality dataset such that it is accurate, complete, reliable, up to date, diverse, and relevant. We also discussed the steps required in text pre-processing. With these aforementioned factors, we can make certain that a high-performance machine learning dataset is built and that we are able to reap the benefits of a robust and accurate machine learning model that has learned from such a superior quality training dataset.</p><h3>Need the Best Quality Dataset? Let us HELP!</h3><p>Creating and maintaining the best quality dataset is not an easy task. Especially, for a small- to medium-sized companies, managing human resources and technical specialties are very challenging. Therefore, it is often more efficient to find another service that does laborious works (including both collection and preprocessing) for you. For that, we could be your perfect solution!</p><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity simultaneously. Moreover, our in-house managers double-check the quality of the collected or processed data! Check us out at <a href="http://selectstar.ai/">selectstar.ai</a> for more information! Let us be your HELP!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*i-tmiQPsFcBSXKWP.png" /><figcaption><a href="https://en.selectstar.ai/contact">selectstar.ai</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8b3072b29d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Crawling Everything: Case of Social Media]]></title>
            <link>https://selectstar-ai.medium.com/data-crawling-everything-case-of-social-media-109e47f567d7?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/109e47f567d7</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[crawling]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 07:52:37 GMT</pubDate>
            <atom:updated>2020-07-09T07:52:37.695Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we are going to talk about data scraping and how it can be used for various purposes. Data scraping is the process of importing information or data from a website and displaying it in a spreadsheet or a local file on your computer. We will be exploring how one can scrape data from social media sites, particularly Twitter. We will also learn the same for Reddit using its official API. Lastly, we will learn how to generally scrape the content in web pages and pictures from different web pages as well. So, let’s get started!</p><h3>Prerequisites</h3><p>Before you go ahead, please note that there are a few prerequisites for this tutorial. You should have some prior basic knowledge of Machine Learning, as well as basic programming knowledge in any language (preferably in Python). We will be using Jupyter Notebook for writing our code. If you do not already have it installed, visit <a href="https://jupyter.org/install.html">Jupyter Notebook</a> or work on any other code editor of your liking.</p><h3>1. Scraping tweets from Twitter using Twint</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/0*M39HOoQ1r9cLtshE.jpg" /></figure><p>There are a number of ways to scrape tweets from Twitter. You can do so using the Twitter API but a shortcoming of this is that it limits the number of tweets that can be scraped. Manually scraping the tweets is also one option but requires unnecessary time and effort. This is why we will be using Twint to collect our tweets from Twitter. <strong>Twint </strong>is a tool that allows you to scrape tweets on different basis e.g. the tweets of a particular user, tweets containing a particular keyword, tweets that are tweeted after or within a certain time, etc.</p><h3>Installations</h3><p>You can install <strong>Twint </strong>by typing the following command in your terminal</p><pre>pip install twint</pre><h3>Scraping Twitter tweets using Twint</h3><h4>Scraping tweets of a particular user</h4><pre>import twint<br>    config = twint.Config()<br>    # Search tweets tweeted by user &#39;BarackObama&#39;<br>    config.Username = &quot;BarackObama&quot;<br>    # Limit search results to 20<br>    config.Limit = 20<br>    # Return tweets that were published after Jan 1st, 2020<br>    config.Since = &quot;2020-01-1 20:30:15&quot;<br>    # Formatting the tweets<br>    config.Format = &quot;Tweet Id {id}, tweeted at {time}, {date}, by {username} says: {tweet}&quot;<br>    # Storing tweets in a csv file<br>    config.Store_csv = True<br>    config.Output = &quot;Barack Obama&quot;</pre><pre>    twint.run.Search(config)</pre><p><strong>Output:</strong></p><pre>Tweet Id 1261004586359422979, tweeted at 18:44:56, 2020-05-14, by BarackObama says: Vote.<br>    Tweet Id 1260955716644470784, tweeted at 15:30:44, 2020-05-14, by BarackObama says: Michelle and I want to do our part to give all you parents a break today, so we’re reading “The Word Collector” for @chipublib. It’s a fun book that vividly illustrates the transformative power of words––and we hope you enjoy it as much as we did. pic.twitter.com/ADYbL6Dzg4<br>    Tweet Id 1260707691900612615, tweeted at 23:05:11, 2020-05-13, by BarackObama says: Despite all the time that’s been lost, we can still make real progress against the virus, protect people from the economic fallout, and more safely approach something closer to normal if we start making better policy decisions now. https://www.vox.com/2020/5/13/21248157/testing-quarantine-masks-stimulus …<br>    ....</pre><h4>Scraping tweets with a particular keyword</h4><pre>import twint<br>    # Configure<br>    config = twint.Config()<br>    # Search tweets that mention Taylor Swift<br>    config.Search = &quot;taylor swift&quot;<br>    # Limit search results to 10<br>    config.Limit = 20<br>    # Return tweets that were published after Jan 1st, 2020<br>    config.Since = &quot;2020-01-1 20:30:15&quot;<br>    # Formatting the tweets<br>    config.Format = &quot;Tweet Id {id}, tweeted at {time}, {date}, by {username} says: {tweet}&quot;<br>    # Storing tweets in a csv file<br>    config.Store_csv = True<br>    config.Output = &quot;Taylor Swift&quot;</pre><pre>    twint.run.Search(config)</pre><p><strong>Output:</strong></p><pre>Tweet Id 1261267734861619201, tweeted at 12:10:35, 2020-05-15, by CVirginie4 says: In 2 days 😍😍💖@taylorswift13 @taylornation13 💖#TaylorSwift #Lover #TaylorSwiftCityOfLover https://twitter.com/cvirginie4/status/1259162021238657024 …<br>    Tweet Id 1261267727961821184, tweeted at 12:10:34, 2020-05-15, by allebahsia says: i met somebody while it was raining and since then everything has changed wow taylor swift the soundtrack to my life huh  https://twitter.com/MissAmericHANA/status/1260628201237147648 …<br>    Tweet Id 1261267719145447424, tweeted at 12:10:32, 2020-05-15, by KenHatesU says: Or &quot;Pancake&quot;<br>    Tweet Id 1261267707250577409, tweeted at 12:10:29, 2020-05-15, by Butera_Maraj says: Vodk il a percé au states jusqu’à il côtoie Taylor Swift  https://twitter.com/taylorswift13/status/1261054225867472897 …</pre><h3>Scraping Reddit using Reddit API</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WLNk1mMKeIk9CvfB.jpg" /></figure><p>We will be scraping donation requests made on Reddit by using the official Reddit API. To access it, you need to:</p><ol><li>Go to the official <a href="https://www.reddit.com/">Reddit</a> website</li><li>Log into your Reddit account or create a new one</li><li>Go to User Settings</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/0*GhiKyZOOEutpu6Vf.png" /></figure><p>4. Go to Privacy and Security</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bXJJ5lJBZZk9IoMa.png" /></figure><p>5. Go to App authorization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e4jLIZoz4DYWroBo.png" /></figure><p>6. Click on ‘are you a developer? create an app’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1bhpVOAeabH8KCYe.png" /></figure><p>7. Create a name for your application and fill in the other relevant credentials. In redirect URL, put the URL of your localhost.</p><p>8, Click on ‘create app’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b5SkO5YSgRtX0aop.png" /></figure><p>9. Copy the characters underneath ‘personal use script’ and next to ‘secret’ and save them in a file or notepad. You will be needing them to gain access to the API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9QDt-H7vvRy0wY_R.png" /></figure><h4>Installations</h4><p>We will be using a Python framework named <strong>Praw </strong>to easily use the Reddit API. To install it, run the following command in your terminal:</p><pre>pip install praw</pre><h4>Python Code</h4><pre>import praw  <br>    import pandas as pd  <br>    import numpy as np</pre><pre>    # Fill in your own credentials for client_id, client_secret and user_agent. Characters in&#39;Personal use script&#39; make your client_id, those in &#39;secret&#39; make client_secret and user_agent is the name of your application.<br>    reddit = praw.Reddit(client_id = &#39;&#39;,  <br>                         client_secret = &#39;&#39;, <br>                         user_agent = &#39;&#39;) <br>    # Get posts from the subreddits related to donations <br>    hot_post_1 = reddit.subreddit (&#39;donate&#39;).hot(limit = 10) <br>    hot_post_2 = reddit.subreddit (&#39;Assistance&#39;).hot(limit = 10) # Offers<br>    hot_post_3 = reddit.subreddit (&#39;Charity&#39;).hot(limit = 10) <br>    hot_post_4 = reddit.subreddit (&#39;Donation&#39;).hot(limit = 10) <br>    hot_post_5 = reddit.subreddit (&#39;gofundme&#39;).hot(limit = 10) # lots of categories<br>    hot_post_6 = reddit.subreddit (&#39;RandomKindness&#39;).hot(limit = 10) <br>    hot_post_7 = reddit.subreddit (&#39;donationrequest&#39;).hot(limit = 10 )</pre><pre>    # Saving donation posts in an empty list<br>    posts = []<br>    for post in hot_post_1:<br>        posts.append ([post.title, post.score, post.id, post.subreddit, post.url, post.num_comments, post.selftext, post.created])<br>        <br>    for post in hot_post_2:<br>        posts.append ([post.title, post.score, post.id, post.subreddit, post.url, post.num_comments, post.selftext, post.created])<br>    for post in hot_post_3:<br>        posts.append ([post.title, post.score, post.id, post.subreddit, post.url, post.num_comments, post.selftext, post.created])</pre><pre>    for post in hot_post_4:<br>        posts.append ([post.title, post.score, post.id, post.subreddit, post.url, post.num_comments, post.selftext, post.created])<br>        <br>    for post in hot_post_5:<br>        posts.append ([post.title, post.score, post.id, post.subreddit, post.url, post.num_comments, post.selftext, post.created])<br>        <br>    for post in hot_post_6:<br>        posts.append ([post.title, post.score, post.id, post.subreddit, post.url, post.num_comments, post.selftext, post.created])<br>        <br>    posts = pd.DataFrame (posts, columns = [&#39;title&#39;, &#39;score&#39;, &#39;id&#39;, &#39;subreddit&#39;, &#39;url&#39;, &#39;num_comments&#39;, &#39;body&#39;, &#39;created&#39;])<br>    #posts<br>    df = pd.DataFrame (data = posts)<br>    dataframe = df.to_csv (r&#39;donations.csv&#39;, index = False)<br>    # Data Processing<br>    df = pd.read_csv (&#39;donations.csv&#39;)<br>    df = df.drop ([&#39;id&#39;, &#39;subreddit&#39;, &#39;num_comments&#39;, &#39;url&#39;, &#39;created&#39;],1)<br>    df = df[[&#39;title&#39;, &#39;score&#39;,&#39;body&#39;]]<br>    print (df.head ())<br>    print(df.shape)<br>    # Saving donation posts to a csv file<br>    dataframe = df.to_csv (r&#39;donations.csv&#39;, index = False)</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0jUJ9n6z9ChVVMmF.png" /></figure><h3>3. Scraping contents of a web page</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CXVSBQ04nESiRYTF.png" /></figure><p>We will be scraping the text content of a Wikipedia page about <a href="https://en.wikipedia.org/wiki/Coronavirus">Reddit</a> using a simple and powerful Python library named BeautifulSoup. It is also important for you to be familiar with some of the basics of <a href="https://html.com/">HTML</a> for web scraping. First, right-click and open your browser’s inspector to inspect the webpage. Hover your cursor on the desired section whose content you want to scrape, and you should be able to see a blue box surrounding it. If you click it, the related HTML will be selected in the browser console. The section that we wish to scrape is a div that contains the entire text within the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ekA6YVZ7IjL8ERlp.png" /></figure><h4>Installations</h4><p>To install BeautifulSoup, run the following command in your terminal:</p><pre>pip install BeautifulSoup4</pre><h4>Python Code</h4><pre># import libraries<br>    import urllib<br>    from bs4 import BeautifulSoup<br>    # specify url of webpage whose content you need to scrape<br>    url = &quot;https://en.wikipedia.org/wiki/Coronavirus&quot;<br>    request = urllib.request.Request (url)<br>    # query the website and return the html of the webpage<br>    response = urllib.request.urlopen (request)</pre><pre>    # parse the html using beautiful soup <br>    var = BeautifulSoup (response,&#39;html.parser&#39;)</pre><pre>    # Take out the &lt;div&gt; and get its value<br>    text_box = var.find (&#39;div&#39;, attrs = {&#39;id&#39;: &#39;bodyContent&#39;})<br>    text = text_box.text.strip () <br>    print (text)</pre><p><strong>Output:</strong></p><pre>From Wikipedia, the free encyclopedia</pre><pre>Jump to navigation<br>Jump to search<br>This article is about the group of viruses. For the ongoing disease involved in the COVID-19 pandemic, see Coronavirus disease 2019. For the virus that causes this disease, see Severe acute respiratory syndrome coronavirus 2.</pre><pre>Subfamily of viruses in the family Coronaviridae</pre><pre>Orthocoronavirinae</pre><pre>Transmission electron micrograph (TEM) of avian infectious bronchitis virus</pre><pre>Illustration of the morphology of coronaviruses; the club-shaped viral spike peplomers, colored red, create the look of a corona surrounding the virion when observed with an electron microscope.</pre><h3>4. Scraping images</h3><p>We will be scraping images in batch through the Fatkun Batch Download Image extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNx6SfbNRbFK4mcM.png" /></figure><h4>Prerequisites</h4><p>You will be needing <a href="https://www.google.com/chrome/?brand=CHBD&amp;gclsrc=aw.ds&amp;&amp;gclid=EAIaIQobChMI_LejjK-Y6AIVw9DeCh0MowmOEAAYASAAEgJsZvD_BwE">Google Chrome</a> Browser along with <a href="https://chrome.google.com/webstore/detail/fatkun-batch-download-ima/nnjjahlikiabnchcpehcpkdeckfgnohf/related?hl=en">Fatkun Batch Download Image</a> extension.</p><h4>Steps:</h4><ol><li>After you are finished with the installation, search for the website and the pictures that you want to download</li><li>Click on the extension’s icon</li><li>Now an extension will get opened which would display a new tab showing all images that have been detected by it. All the pictures that appear on the extension’s tab by default have opted for the purpose of download. After making the choice, click on ‘save image’.</li><li>The extension would now provide you with the warning and will ask where to save the file before it is being downloaded and you have to give the confirmation for each image.</li><li>The extension would create for you a new folder based on the title of the website and there you could download all the desired images. You could even click on ‘more options’ so that with the aid of link you could simply filter the images, rename and sort them as per size.</li></ol><h3>Conclusion</h3><p>To sum it all up, we started off by getting an introduction to data scraping or data crawling. We applied it in 4 different ways, i.e. how to extract tweets from Twitter without using its API, how to scrape posts from subreddits using the official Reddit API. We learned to scrape content from web pages using Python’s BeautifulSoup library and lastly, we learned how to download images in batch using Google’s Fatkun Batch Image Download extension.</p><h3>If Crawling is hard? Let us HELP!</h3><p>While crawling presents easy access to many web-based data collections, most times, such data also accompanies heavy noises and contaminations to be used as a dataset right away. Therefore, companies or researchers need to devote heavy efforts in quality controlling; having enough human resources is always a great challenge. Therefore, it is often more efficient to find another service that does laborious works (including both collection and preprocessing) for you. For that, we could be your perfect solution!</p><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity simultaneously. Moreover, our in-house managers double-check the quality of the collected or processed data! Check us out at <a href="http://selectstar.ai/">selectstar.ai</a> for more information!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*n7S6vIlWIcFMt8gi.png" /><figcaption><a href="http://selectstar.ai/">selectstar.ai</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=109e47f567d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Easiest Way to Create Your Chatbot Buddy!]]></title>
            <link>https://selectstar-ai.medium.com/the-easiest-way-to-create-your-chatbot-buddy-164d016b564c?source=rss-423a8f65a8f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/164d016b564c</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[chatbot-design]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-labeling]]></category>
            <dc:creator><![CDATA[SelectStar]]></dc:creator>
            <pubDate>Thu, 02 Jul 2020 05:22:16 GMT</pubDate>
            <atom:updated>2020-07-02T05:22:16.140Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this tutorial, we are going to talk about how you can create your own Chatbot buddy easily. A Chatbot is basically a computer program that conducts a conversation between the user and the computer through auditory or conversational methods. In short, it acts as a real-world conversational partner. So, in this tutorial, we will be creating a very simple Chatbot application that will tell you a little about Lung cancer. So, let’s get started!</p><h3>Prerequisites</h3><p>Before you go ahead, please note that there are a few prerequisites for this tutorial. You should have some prior basic knowledge of Machine Learning, as well as basic programming knowledge in any language (preferably in Python). You must also have some knowledge about NLTK which is a leading platform for building Python programs to work with human language data. Apart from this, the article is pretty beginner-friendly and easy to understand. We will also be using <a href="https://colab.research.google.com/notebooks/intro.ipynb">Google Colab</a> for writing our code but you can write it on any code editor of your choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g74t0rrcO1M5g_T0.png" /><figcaption>image source from <a href="https://www.fingent.com/blog/capitalizing-on-ai-chatbots-will-redefine-your-business-heres-how/">here</a></figcaption></figure><h3>Chatbot</h3><p>A chatbot is a part of the software in various applications such as virtual assistants (Siri, Alexa, Google Assistant, etc.), website function, or other networks that try to converse with users to understand their needs and then assist them to perform a particular task. Chatbots are categorized into two main types: Self Learning and Rule-Based chatbots.</p><ol><li>Self-Learning chatbots use machine learning and artificial intelligence algorithms to recognize the characteristics of the inputs that they get from the users and use them later on. These characteristics are obtained from the trained models and usually in the forms of high-dimensional vectors. They are more efficient than Rule-Based chatbots.</li><li>Rule-Based chatbots follow certain rules on which the responses are based. A rule-based approach is a very simple and nice approach to start with, but it can fail to handle the complex questions of a user.</li></ol><h3>Our Application</h3><p>We will be making a chatbot buddy which will act as a doctor and will answer your questions about lung cancer. It is basically a Self-learning chatbot and will respond to the user questions by selecting the best response that is most similar to the question that the user has asked. The application takes text about <a href="https://www.mayoclinic.org/diseases-conditions/lung-cancer/symptoms-causes/syc-20374620">Lung Cancer</a> and uses it to converse with the user and answer his or her queries about the disease. The knowledge of the chatbot is limited to the content of the website.</p><h3>Python Implementation</h3><pre>#Installing essential packages<br>pip3 install nltk<br>pip3 install newspaper3k<br># Importing relevant libraries<br>from newspaper import Article<br>import random<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.metrics.pairwise import cosine_similarity<br>import nltk<br>import numpy as np<br>from random import choice<br>import string</pre><pre># Downloading NLTK packages<br>nltk.download (&#39;punkt&#39;)<br>nltk.download (&#39;wordnet&#39;)</pre><pre># Geting the paper/article and extracting its text<br>paper = Article (&#39;https://www.mayoclinic.org/diseases-conditions/lung-cancer/symptoms-causes/syc-20374620&#39;)<br>paper.download()<br>paper.parse()<br>paper.nlp()<br>paper_text = paper.text<br>#print(paper_text)</pre><pre>text = paper_text</pre><pre># Converting the text in the article into a list of individual words or sentences<br>tokens = nltk.sent_tokenize (text)<br>#print(tokens)</pre><pre># Removing punctuation in the text by creating a dictionary<br>no_punctuation = dict( (ord (punct), None) for punct in string.punctuation  )</pre><pre># Function that returns lemmatized words after punctuation removal. Words are in lower case<br># lemmatize : sort so as to group together inflected or variant forms of the same word<br>def lemmatizeWord (text):<br>    return nltk.word_tokenize(text.lower().translate(no_punctuation))<br>#print(lemmatizeWord(text))</pre><pre>def reply (user_reply):<br>     # User&#39;s question<br>     user_reply = user_reply.lower ()</pre><pre>     # Chatbot&#39;s reply to the user&#39;s question<br>     chatbot_reply = &#39;&#39;</pre><pre>     # Appends the question of the user to the list of tokens.<br>     tokens.append (user_reply)</pre><pre>     # Measures the frequency and rarity of a word. Is statistical   measure used to evaluate how important a word is to a document.    Importance is proportional to the number of times a word is repeated in a document or text.<br>     TfidfVector = TfidfVectorizer (tokenizer = lemmatizeWord, stop_words = &quot;english&quot;)</pre><pre>     # Converting text to matrix form<br>     tfidf = TfidfVector.fit_transform (tokens)</pre><pre>     # Gets similarity score between the user’s question and our      text in the article<br>     scores = cosine_similarity (tfidf[-1], tfidf)</pre><pre>     # Get sindex of most similar text to the user&#39;s question<br>     index = scores.argsort ()[0][-2]</pre><pre>     # Reduces dimensionality of scores (make one list) by flattening<br>     flat = scores.flatten ()</pre><pre>     # Sorts in ascending order<br>     flat.sort ()<br>     best_score = flat[-2]<br>     # print(best_score)<br>     # for no text similar to the user&#39;s question, best_score will be 0<br>     if (best_score == 0):<br>          chatbot_reply = chatbot_reply + &quot;Sorry, I do not  understand what you are saying.&quot;<br>     else:<br>          chatbot_reply = chatbot_reply + tokens[index]</pre><pre>     # Remove user&#39;s response from tokens list<br>     tokens.remove (user_reply)</pre><pre>     # print(chatbot_response)<br>     return chatbot_reply<br></pre><pre>def main ():<br>     var = 1<br>     print (&quot;Chatbot: Hi, I am your doctor chatbot buddy. I specialise in Lung cancer (sort of).What is your name?. PS, if you wish to quit this conversation, type &#39;quit&#39;.&quot;)<br>     user_name = input (&quot;User: &quot;)</pre><pre>     if user_name == &quot;quit&quot;:<br>          var = 0<br>          print (&quot;Chatbot: Oh thats ok. We can talk later&quot;)</pre><pre>     else:<br>          print(&quot;Chatbot: Well hello &quot;+ user_name + &quot;. What do you want to know about Lung cancer?&quot;)</pre><pre>     while (var == 1):<br>          user_reply = (input (user_name + &quot;: &quot;+ &quot; &quot;))<br>          user_reply = user_reply.lower ()<br>          if (user_reply != &quot;quit&quot;):<br>               if (user_reply == &quot;thanks&quot; or user_reply == &quot;thank you&quot;):<br>                    var = 0<br>                    print (&quot;Chatbot: You are welcome.&quot;)</pre><pre>               else:<br>                    print (&quot;Chatbot: &quot;+ reply (user_reply))</pre><pre>          else:<br>               var = 0<br>               print (&quot;Chatbot: Oh thats ok. We can talk later.&quot;)</pre><pre>main ()</pre><p><strong>Output:</strong><br><strong>Chatbot: </strong><em>Hi, I am your doctor chatbot buddy. I specialise in Lung cancer (sort of).What is your name?. PS, if you wish to quit this conversation, type &#39;quit&#39;. User: Junaid Chatbot: Well hello Junaid. What do you want to know about Lung cancer? Junaid: I suffer from shortness of breath? Do you think I have lung cancer? Chatbot: Fluid accumulating in the chest can cause shortness of breath. Junaid: So what causes lung cancer? Chatbot: How smoking causes lung cancer Doctors believe smoking causes lung cancer by damaging the cells that line the lungs. Junaid: Nooo :3 Chatbot: Sorry, I do not understand what you are saying. Junaid: quit Chatbot: Oh thats ok. We can talk later.</em></p><h3>Applications of Chatbots</h3><p>Chatbots have become popular in the past few years and businesses have discovered innovative ways to put them to use. Basically, chatbots can help users to:</p><ul><li>Book flights/tickets for travel</li><li>Order food</li><li>Do Market Research</li><li>Make product inquiries</li><li>Have health care knowledge</li><li>Have companionship</li></ul><h3>The chatbot needs lots and lots of text!</h3><p>For your chatbot to perform the best quality outputs, you need lots and lots of text data with high quality. Most times, it is very difficult for an individual or small- or medium-sized companies to collect such large quantity data while maintaining high quality. Therefore, it is often more efficient to find another service that does laborious works for you. We could be your perfect solution!</p><p>Here at <a href="https://www.selectstar.ai/"><strong><em>Selectstar</em></strong></a>, we crowdsource our tasks to diverse users located globally to ensure the quality and quantity on time. Moreover, our in-house managers double-check the quality of the collected or processed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /><figcaption><a href="http://selectstar.ai/">selectstar.ai</a></figcaption></figure><h3>Conclusion</h3><p>Chatbots have are high in demand and are used extensively in the world nowadays. Every business needs it to automate their customer service/ support department, at least to some extent. If you visit a Facebook page, there’s a 90% chance that if a page is well-established, they would have a messenger chatbot, which is equipped to provide you any generic information. Where the chatbot fails, you get diverted to a human for support instead. This lowers the company’s need for employees for their Customer Support Team, hence increasing their profits as well. This was just one example, you’ll find chatbots being used by pretty much every big enterprise, to automatically answer 24/7 common customer queries.</p><p>To sum it all up, we started off by getting an introduction of what chatbots are and talked about the two main types which are Self-learning and Rule-based. We made a small medical application about lung cancer using the Self-learning based approach in Python. Lastly, we discussed some of the other common applications of Chatbots and how businesses today are creatively using chatbots to attract customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=164d016b564c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>