<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joseph (Iosif) Mushailov on Medium]]></title>
        <description><![CDATA[Stories by Joseph (Iosif) Mushailov on Medium]]></description>
        <link>https://medium.com/@jmushailov?source=rss-34ee111f24af------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*unzRD5OECF62F8adV-ucfA.jpeg</url>
            <title>Stories by Joseph (Iosif) Mushailov on Medium</title>
            <link>https://medium.com/@jmushailov?source=rss-34ee111f24af------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:11:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jmushailov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LSTM Framework For Univariate Time-Series Prediction]]></title>
            <link>https://medium.com/data-science/lstm-framework-for-univariate-time-series-prediction-d9e7252699e?source=rss-34ee111f24af------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9e7252699e</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Joseph (Iosif) Mushailov]]></dc:creator>
            <pubDate>Tue, 30 Mar 2021 01:25:28 GMT</pubDate>
            <atom:updated>2021-03-31T23:38:41.635Z</atom:updated>
            <content:encoded><![CDATA[<h4>Full Code Template &amp; Walkthrough</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BQawDwiMyzUITjUlXeE3Q.jpeg" /><figcaption><strong>Credit: </strong><a href="https://unsplash.com/@marcojodoin"><strong>Marc Jodoin</strong></a></figcaption></figure><p>LSTM methodology, while introduced in the late 90’s, has only recently become a viable and powerful forecasting technique. Classical forecasting methods like ARIMA and HWES are still popular and powerful but they lack the overall generalizability that memory-based models like LSTM offer.</p><p>The main objective of this article is to lead you through building a working LSTM model. However, while this article’s goal isn’t necessarily to compare new and classic modeling techniques, I will discuss some advantages and disadvantages of classical vs. RNN-based techniques in the conclusion.</p><p>The full code is provided below. Given you have the dataset and the necessary classes imported, the results should be entirely reproducible.</p><h3>What Are LSTMs?</h3><p>LSTM (Long Short-Term Memory) is a Recurrent Neural Network (RNN) based architecture that is widely used in natural language processing and time series forecasting. <a href="https://www.youtube.com/watch?v=WCUNPb-5EYI">Brandon Rohrer’s video</a> offers a great, intuitive introduction.</p><p>The LSTM rectifies a huge issue that recurrent neural networks suffer from: short-memory. Using a series of ‘gates,’ each with its own RNN, the LSTM manages to keep, forget or ignore data points based on a probabilistic model.</p><p>LSTMs also help solve exploding and vanishing gradient problems. In simple terms, these problems are a result of repeated weight adjustments as a neural network trains. With repeated epochs, gradients become larger or smaller, and with each adjustment, it becomes easier for the network’s gradients to compound in either direction. This compounding either makes the gradients way too large or way too small. While exploding and vanishing gradients are huge downsides of using traditional RNN’s, LSTM architecture severely mitigates these issues.</p><p>After a prediction is made, it is fed back into the model to predict the next value in the sequence. With each prediction, some error is introduced into the model. To avoid exploding gradients, values are ‘squashed’ via (typically) sigmoid &amp; tanh activation functions prior to gate entrance &amp; output. Below is a diagram of LSTM architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ujYN_Jpow9k2LCoK.png" /><figcaption>LSTM architecture. Image from <a href="https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:LSTM_cell.svg">Wikipedia</a> under <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a> License.</figcaption></figure><h3><strong>TL;DR — Just Give Me The Code</strong></h3><h4>Execution Script</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56938af41105a86f9eac1e6f59272cfb/href">https://medium.com/media/56938af41105a86f9eac1e6f59272cfb/href</a></iframe><p><strong>Methods for Data Manipulation</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de2063601af22ca6ed0081d678027ba3/href">https://medium.com/media/de2063601af22ca6ed0081d678027ba3/href</a></iframe><p><strong>Methods for Time Series</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/251ccec58ab6c150981000ac2c1d5c2a/href">https://medium.com/media/251ccec58ab6c150981000ac2c1d5c2a/href</a></iframe><p><strong>LSTM Methods</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12583f3727ab148602fd4fd91fc7d8a8/href">https://medium.com/media/12583f3727ab148602fd4fd91fc7d8a8/href</a></iframe><h3>Code &amp; Data Walkthrough — Data Prep</h3><p>The data is the US/EU exchange rate from 2010 to present, not seasonally adjusted. You can pull the data from the St. Louis Federal Reserve <a href="https://fred.stlouisfed.org/series/DEXUSEU">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*jAyycPWvC8mQ2fhfOlGdgQ.png" /><figcaption>The Data</figcaption></figure><p>Before building the model, we create a series and check for stationarity. While stationarity is not an explicit assumption of LSTM, it does help immensely in controlling error. A non-stationary series will introduce more error in predictions and force errors to compound faster.</p><p>We filter out one ‘sequence length’ of data points for later validation. In this case, 60 points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*MQdhucg9A4FYe4tUMUFqgg.png" /><figcaption>Our Dickey-Fuller — Shows Stationarity</figcaption></figure><p>The data format required for a LSTM is 3 dimensional, with a moving window.</p><ul><li>So the first data point will be the first 60 days of data.</li><li>The second data point is the first 61 days of data but not including the first.</li><li>The third data point is the first 62 days of data but not including the first and second.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*5myR0DUG4Dg2c0UqXY5DQw.png" /><figcaption>Our Training Data Frame</figcaption></figure><p>The last major step of prep is to scale the data. Here we use a simple minmax scaler. Our sequence length is 60 days for this part of the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*eTqZfbP112qINju4RMHSBg.png" /></figure><h3>Code Walkthrough — Building the LSTM</h3><p>The next part of the analysis is pretty straightforward code-wise. We first create a learning rate scheduler. This scheduler will monitor the validation loss and modify the learning rate on plateaus.</p><p>We create a method to reset all of the weights in case we want to re-train with different parameters (the method is unused in my code but it’s there if you need it).</p><p>We then build an LSTM with 2 layers, each with 100 nodes and build an output function with a sigmoid activation.</p><p>The history object stores model loss over epoch, which can be plotted to evaluate whether an adjustment is needed in the training process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*GgD-S9tbhSDRMqwxgSd01g.png" /><figcaption>Model Loss by Epoch</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*HssCWptilW2SU3XUDscYVg.png" /><figcaption>Code snippet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*VJxkvpRhZkkfBzsBePLTVg.png" /><figcaption>Training the Model</figcaption></figure><h3>Code Walkthrough — Predicting the Future</h3><p>The class provided allows us to both test model accuracy and to predict the future.</p><p>The method below tests our model against the validation data frame we previously set aside. The results are pretty promising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*virHeWJVeVqmOiGczm7wUA.png" /><figcaption>Our Comparison Method</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*b6d82dCfENfpnDGBTf3O2w.png" /><figcaption>Our Comparison</figcaption></figure><p>If we are happy with the accuracy, we can then predict the future. Below is a quick demonstration of how an LSTM predicts. You can see how some error is introduced into the model at each time step, but since each data point is a series of historical points, the error is lessened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*iM-6lIm9hOxknR6ykqVd9w.png" /><figcaption>Network Feeding Prediction Back Into the Model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*Z5y04la32bFMPc6aW9LtHg.png" /><figcaption>Reverse the MinMax &amp; Predict</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*2GCfRYyVvEyXFY04X6AXXQ.png" /><figcaption>The Predicted Points (Orange)</figcaption></figure><h3>Conclusion</h3><p>Since this article is mainly about building an LSTM, I didn’t discuss many advantages / disadvantages of using an LSTM over classical methods. I’d like to offer some guidelines in this conclusion:</p><p><strong>Non-Technical Considerations</strong></p><ol><li>It is very important to weigh the costs and benefits of using a complex model vs. a simpler model. A slight boost in accuracy may not be worth the time, effort and loss of interpretability introduced by LSTMs.</li><li>If time-series forecasting were easy, stock markets would be solved! There is an inherent element of noise in all time-series data that we cannot feasibly capture, regardless of how great our model is.</li><li>Don’t ignore intuition. Time series models implicitly assume that previous time periods dictate future time periods. This may not always be the case. If your results don’t make sense intuitively, don’t be afraid to ditch the model!</li></ol><p><strong>Technical Considerations</strong></p><ol><li>ARIMA (and MA-based models in general) are designed for time series data while RNN-based models are designed for sequence data. Because of this distinction, it’s harder to build RNN-based models out-of-the-box.</li><li>ARIMA models are highly parameterized and due to this, they don’t generalize well. Using a parameterized ARIMA on a new dataset may not return accurate results. RNN-based models are non-parametric and are more generalizable.</li><li>Depending on window size, data, and desired prediction time, LSTM models can be <em>very </em>computationally expensive. Sometimes they’re not feasible without powerful cloud computing.</li><li>It’s good practice to have a ‘no-skill’ model to compare results to. A good start would be to compare the model results to a model predicting <em>only</em> mean for each time step over the period (horizontal line).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9e7252699e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/lstm-framework-for-univariate-time-series-prediction-d9e7252699e">LSTM Framework For Univariate Time-Series Prediction</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delivering Immediate Business Value With NLP & R]]></title>
            <link>https://medium.com/data-science/delivering-immediate-business-value-with-nlp-r-c51020d72ce4?source=rss-34ee111f24af------2</link>
            <guid isPermaLink="false">https://medium.com/p/c51020d72ce4</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[code-template]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[Joseph (Iosif) Mushailov]]></dc:creator>
            <pubDate>Sun, 01 Nov 2020 18:16:14 GMT</pubDate>
            <atom:updated>2020-11-04T16:17:07.247Z</atom:updated>
            <content:encoded><![CDATA[<h4>Let’s Face It — We Can’t Just Sit There and Train Neural Networks All Day</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*PpG8g77JBsQOnHyf" /><figcaption><strong>Photo: </strong><a href="https://unsplash.com/@florenciaviadana"><strong>Florencia Viadana</strong></a><strong>, Unsplash</strong></figcaption></figure><p>As data scientists, our jobs are to deliver tangible, bottom-line results to the business. While I’d love to train neural networks all day, it’s critical that we build solid relationships with our business units and find ways to deliver easily understandable and quantifiable value.</p><p>Unless you’re working for a big tech company, chances are that your team has plenty of use cases for analytics. In this short tutorial, I’m going to give you the code and overview of how you can leverage basic NLP (Natural Language Processing) to deliver real, communicable, valuable analytics.</p><p>For tutorials on gathering requisite text data from the internet (and some warnings) check out my articles on web crawling:</p><p>Part 1 for complete beginners can be found <a href="https://medium.com/analytics-vidhya/automated-browsers-scraping-and-crawling-easier-than-it-looks-af37bd765bca"><strong>here</strong></a><strong> </strong>and part 2, where we take a more object oriented and reusable approach can be found <a href="https://medium.com/analytics-vidhya/automated-browsers-scraping-and-crawling-part-2-cc9e2149a64"><strong>here</strong></a>.</p><h3>TL;DR — Just Give Me The Code</h3><p>Dataset can be found <a href="https://archive.ics.uci.edu/ml/datasets/Drug+Review+Dataset+%28Druglib.com%29"><strong>here</strong></a><strong> </strong>(there’s a link in the code as well).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac3d7a96209800842b4e590e6aae8755/href">https://medium.com/media/ac3d7a96209800842b4e590e6aae8755/href</a></iframe><h3>Let’s Break it Down: Corpus</h3><p>The corpus is the core data structure we will work with for NLP. Since we’re passing a character vector to the Corpus() function, we need to specify that the source is a vector source. Using the inspect() method will allow you to take a look at the newly created Corpus.</p><p>Following the creation of the corpus, we clean up the text data using some common methods. Our cleaner() function does some basic manipulations like removing numbers, but it also removes all stop words, which are words that have little to no meaning on their own. Popular stop words are “the” and “a.”</p><p>Note that there may be some information loss when removing stop words. This is where analysis becomes more art than science. You may opt to just remove basic ones, you may decide to remove industry specific terms, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*AzAd8vO479j55m99gJOB2w.png" /><figcaption>Original review of product (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*HaJXWaSd-R5qhWZf-ixfew.png" /><figcaption>With stop words removed. Note that “it did not” is no longer in the sentence (Image by Author)</figcaption></figure><h3>TDM &amp; Word Cloud</h3><p>When we created our Corpus, R basically coded every review as a separate text document. A Term Document Matrix (TDM) takes a corpus and makes a vectorized frequency table of <strong>every single word</strong>. Here’s an example of 3 dummy reviews made into a TDM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*TmfqN-zzAo2YmUfTUeVgfg.png" /><figcaption>Example of How a TDM is Made (Image by Author)</figcaption></figure><p>From this TDM, we can create a matrix, sort it by row sums (word frequency) then visualize the results using a word cloud. This is the first part of our analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*S9umT3pStqFQ8H3Ikp5ceQ.png" /><figcaption>Word Cloud of Drug Reviews (Image by Author)</figcaption></figure><h3>Word Correlations</h3><p>The word cloud is a good start, but it is a very basic starting point. We can enhance our analysis by evaluating correlations between words. Word correlations leverage the same methodology that R’s base cor() function uses, with Pearson correlation being the default. Here’s the Pearson correlation formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*Mniw00slTXxxC71WXFa0IA.png" /><figcaption>Pearson Correlation Coefficient (r) Formula,</figcaption></figure><p>Since our TDM is vectorized, R can perform these comparisons efficiently. In our master code, we need to enter a term of interest and a lower correlation bound. Don’t be surprised if you see most word correlations falling under ~15%. In fact, 15% could be a significant correlation because of the sheer size of our datasets. Even with our tiny (by ML standards) dataset, our TDM is very large (23M elements — 3107 reviews x 750 words).</p><p>Let’s look at a correlation graph between the word “Pain” and the rest of our TDM, imposing a lower correlation bound of 18%:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIeLwBTguU97wp4T2fJMqw.png" /><figcaption>Correlation Graph Between TDM and “Pain”, 18% Correlation Lower Bound (Image by Author)</figcaption></figure><p>The graph shows us a few insights right away. For example, we see “nerve” has a high correlation with pain with a correlation of 22%. This can have direct, immediate business impact; we have efficiently summarized over 3,000 reviews and pulled a common ailment.</p><h3>Sentiment Analysis</h3><p>While there are many complex ways to perform sentiment analysis, we will focus on reasonably performing, out of the box methods in this article. For a detailed overview of sentiment analysis, check out this <a href="https://en.wikipedia.org/wiki/Sentiment_analysis">Wikipedia page</a>. For a quick startup, we essentially leverage huge lists of pre-labeled negative / positive words from Microsoft’s sentiment lexicon. We left join the lexicon with our list of words and then just calculate ratios of positive:negative words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*1bKsZS-_DyqEIDymLMTe_Q.png" /><figcaption>Snippet From Our Merged List (Image by Author)</figcaption></figure><h3>N-Grams</h3><p>But wait, there’s more! While we can take a closer look into individual word frequencies, this is ostensibly a bar graph / table form of what we visualized in our word cloud. Single words also suffer from missing information. Similar to what we saw in our discussion of stop words, we can lose important qualifiers in a sentence with this approach. For example, we can have “fan” be one of our top words, but maybe half of the sentences also include “I was not a…”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B0to5YzYNY3aoyWebng8Ug.png" /><figcaption>Image by Author</figcaption></figure><p>We can counter this loss of information with N-gram analysis. An N-gram is basically “N” words that co-occur in our dataset. All we need to do in this code template is change the “n” in our method to the size we want. Here are the results after running a Bigram and Trigram analysis and cleaning our datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*JNBIZDVk4SIdhYrcImRQvQ.png" /><figcaption>Almost Instantly, We Get a Good Idea of The Popular Topics (Image by Author)</figcaption></figure><h3>Great, How Do I Sell It?</h3><p>There are a few great advantages of this code template, both technical and non-technical:</p><ol><li><strong>Combat Bias &amp; Salience: </strong>One of the most important advantages of having a programmatic approach is removing the human element. It’s common for companies to have a point person reviewing reported drug side effects, internal surveys, etc. This is a problem. People are biased, whether they know it or not. Subconscious bias can lead to one type of comment remaining salient or directly influencing the delivery of results. A program like this can be easily scaled and easily mass-shared (see #4), eliminating the siloed bias/salience effects.</li><li><strong>Who Doesn’t Like Word Clouds? </strong>The resulting visuals are easily understandable, they look nice, and are a great way to build business-side confidence in your analytics.</li><li><strong>Speed: </strong>Sure, it’s not completely optimized, but this code is <em>fast</em>. Start-to- finish, this program runs in 6.25 seconds on my PC. Granted, my PC is a bit faster than most work machines, but not by that much. A person physically combing through these 3100 reviews could take days.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*gM9OSwzD9ZjKNH-iZHfN4A.png" /><figcaption>Timing the Script (Image by Author)</figcaption></figure><p>4. <strong>No Analytics Infrastructure? No Problem! </strong>This template can be recycled for any text analysis with little to no user interaction. Infrastructure helps but isn’t really necessary. If we wanted to do a ‘soft deploy’ of this type of code:</p><ul><li>Add whichever export conditions you want to see (graphs, n-grams, etc.)</li><li>Create a .bat file that references this script</li><li>Schedule a windows process to run the executable every “X” days as data is updated, or just click the .bat</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c51020d72ce4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/delivering-immediate-business-value-with-nlp-r-c51020d72ce4">Delivering Immediate Business Value With NLP &amp; R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Programmatically Finding Your Neighborhood’s Best Avocado Toast]]></title>
            <link>https://levelup.gitconnected.com/programmatically-finding-your-neighborhoods-best-avocado-toast-820aeea44a32?source=rss-34ee111f24af------2</link>
            <guid isPermaLink="false">https://medium.com/p/820aeea44a32</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Joseph (Iosif) Mushailov]]></dc:creator>
            <pubDate>Tue, 15 Sep 2020 21:31:30 GMT</pubDate>
            <atom:updated>2020-11-02T04:06:09.364Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Yelp’s API and R to Query Yelp’s Database in Real Time</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GrXsvlx_pMGeMzJG" /><figcaption>Credit: Ben Kolde, Unsplash.com</figcaption></figure><p>Using company APIs is a great way to automate data extraction. While it is possible to scrape websites for data, (see my articles on web scraping) APIs offer a simpler, and often more legal, way to get data at-scale.</p><p><strong>The Complete Basics — What is an API?</strong></p><p>An API stands for “Application Programming Interface,” which in (very) simple terms, is a piece of software that allows two applications to communicate. App 1 will send requests to the App 2, and App 2 will return results. Different APIs will have different restrictions on access to data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*iS8f0Y962mgiAuO1-zhvlA.png" /><figcaption>Think of the way you interact with a website. This isn’t far off from the basics of RESTful APIs</figcaption></figure><p><strong>The Complete Basics —Types of Web Service APIs</strong></p><p>There are many different types of web APIs but we’ll focus on introducing 4 major web service APIs.</p><ol><li><strong>JSON Protocol: </strong>Server is queried and JSON-formatted results are returned. If you’re using something like Python/R you’ll need to parse the JSON to get a usable data format.</li><li><strong>XML Protocol: </strong>Basically the same as JSON, except results are delivered in a specific XML format. Pretty simple protocol.</li><li><strong>SOAP Protocol: </strong>Similar to XML except it’s a lot more complex. While it still delivers results in XML, it uses a proprietary XML format.</li><li><strong>*REST: </strong>While REST isn’t technically a protocol, it is often compared to SOAP. REST is more lightweight and (generally) has better performance. It is also easy to work with for beginners and is what we will work with in this article.</li></ol><p><strong>The Complete Basics — HTTP Methods to Use With REST APIs</strong></p><p><strong>GET: </strong>Retrieves resources and information. GET requests never change the underlying resource, so they are viewed as ‘safe methods.’ As you can expect, most APIs allow for GET requests.</p><p><strong>POST: </strong>Creates a new resource. We use the POST method to create our client credentials to access the API.</p><p><strong>PUT: </strong>Updates an existing resource. If you’re using a company’s API, chances are you won’t have any access to PUT resources.</p><p><strong>DELETE: </strong>As the method implies, this is used to delete resources. As with PUT, you likely won’t have access to this.</p><p><strong>PATCH: </strong>Like PUT, updates an existing resource. Unlike PUT, PATCH partially edits the resource. This method has a few problems associated with it and is not universally used. You likely won’t be using PATCH, especially for just querying.</p><p>More detailed documentation can be found on the RESTful official website: <a href="https://restfulapi.net/http-methods/">https://restfulapi.net/http-methods/</a></p><h3><strong>TL;DR: Just Give Me the Code</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ddc0bcff79929902bafc0042f33dc6a/href">https://medium.com/media/7ddc0bcff79929902bafc0042f33dc6a/href</a></iframe><h3>Breaking The Function Down</h3><p>The function looks a bit intimidating, but it’s quite simple when broken down.</p><ol><li>It takes as a parameter the number of pages you would like to query. By default the Yelp API only allows 50 results per page maximum. So if we want to get multiple pages, we need to explicitly tell our program to continue.</li><li>Before we define anything, we want a master dataframe where we will store all of our results.</li></ol><pre># Defining the Function to search and parse<br>YelpScr &lt;- function(offset=offsetP){<br>  <br>  # Define our storage dataframe where we will store results<br>  storage = data.frame()</pre><p>3. To query Yelp, we will essentially need to create a URL. Given parameters that we will pass, this next function constructs a URL. The URL will look something like: “https://yelp.api.com/v3/businesses/search/query=…”</p><pre>for(i in 1:length(offsetP)){  <br>    <br>    url &lt;- modify_url(https://medium.com/yelp,%20path%20=%20c(&quot;v3&quot;,%20&quot;businesses&quot;,%20&quot;search&quot;),<br>                     query = list(term = term, location = location, <br>                                   limit = limit,<br>                                   radius = radius,<br>          # For next pages -&gt; offset 51 for pg 2, 101 for pg 3...<br>                                   offset = offsetP[i]<br>                      ))</pre><p>4. We pass a GET request, essentially requesting data and providing our client ID to authenticate with the server.</p><pre># Our GET request, querying the server<br>res &lt;- GET(url, add_headers(&#39;Authorization&#39; = paste(&quot;bearer&quot;, client_key)))<br>    <br># Getting our results<br>results &lt;- content(res)</pre><p>5. We create a local method that can parse the results into a list and then into an easily readable tibble.</p><pre># Parsing our results<br>yelp_httr_parse &lt;- function(x) {<br>      <br>      # Defining the variables we want<br>      parse_list &lt;- list(id = x$id, <br>                         name = x$name, <br>                         rating = x$rating, <br>                         review_count = x$review_count, <br>                         latitude = x$coordinates$latitude, <br>                         longitude = x$coordinates$longitude, <br>                         address1 = x$location$address1, <br>                         city = x$location$city, <br>                         state = x$location$state, <br>                         zip_code = x$location$zip_code,<br>                         distance = x$distance,<br>                         phone = x$phone,<br>                         url = x$url<br>      )<br>      <br># Quick lapply to clean the data up<br>parse_list &lt;- lapply(parse_list, FUN = function(x) ifelse(is.null(x), &quot;&quot;, x))<br>      <br>      # Making our data into a tibble for easy access<br>      df &lt;- tibble(id=parse_list$id,<br>                   name=parse_list$name, <br>                   rating = parse_list$rating, <br>                   review_count = parse_list$review_count, <br>                   latitude=parse_list$latitude, <br>                   longitude = parse_list$longitude, <br>                   address1 = parse_list$address1, <br>                   city = parse_list$city, <br>                   state = parse_list$state, <br>                   zip_code = x$location$zip_code,<br>                   distance = x$distance,<br>                   phone = x$phone,<br>                   url = x$url)<br>      df<br>    }</pre><p>6. We apply the local method we created to our results and then rbind (append row-wise) all data structures created by our method.</p><pre># Results list<br>results_list &lt;- lapply(results$businesses, FUN = yelp_httr_parse)<br>    <br>payload &lt;- do.call(&quot;rbind&quot;, results_list)</pre><p>7. We make another local function: after every page is queried, we show some of the metadata associated with the page. Below is what this function looks like when it’s running.</p><pre># Show analysis on every pageg<br>ratingsCalc &lt;- function(payload){<br>      x &lt;- payload$rating * payload$review_count <br>      x2 &lt;- round( (sum(x) / sum(payload$review_count)), 2)</pre><pre>print(paste0(&quot;Average Yelp rating for stores within a radius of about &quot;, radius, &quot; feet of &quot;, location, <br>&quot;, using the search term ***&quot;, term,  &quot;*** &quot;, &quot;is: &quot;, x2, &quot; stars. This is calculated using a total of &quot;, sum(payload$review_count), &quot; reviews&quot;))<br>}</pre><pre># Show the messages as it queries Yelp<br>    msg &lt;- ratingsCalc(payload)<br>    strsplit(msg, &quot;\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*_A6LSUZjFzhDu6Z_k5YRQg.png" /><figcaption>Our metadata being shown. Avocado was misspelled. My bad. :/</figcaption></figure><p>8. After every page is parsed, we store it row-wise in our storage dataframe. We also include an exit condition for empty pages and return the dataframe.</p><pre># Store the results in our dataframe<br>storage = rbind(storage, payload)</pre><pre>if(sum(payload$review_count)==0){ break}<br>  }<br>  # Return our final storage DF<br>  return(storage)<br>  }</pre><h3>Time to run it!</h3><p>9. We pass our client ID and client key (can be acquired for free from Yelp’s developer portal). We need to make sure that we <strong>never</strong> pass our credentials explicitly. There are a few ways we can pass our credentials. The 3 best ways are outlined in the comments: Use the <strong>Keyring package, </strong>opt to store it in a text file, open, read, and close the file, or force user input.</p><pre># ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #<br>#                 Running Our Code<br># ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #</pre><pre># We want to avoid explicitly having our client id/key in our code. Instead we can do 3 things:<br>  # 1) Open a .txt with our id/key in rows 1/2, respectively<br>  # 2) Use the R Keyring library to save your keys <br>  # 3) Force user to input id/key manually. This becomes too much of a headache once you start using the script often since it&#39;s a manual touch every time</pre><pre>#  ~~~~~~~~~~~~~~~~~~~~~~~ PASSING CREDENTIALS<br># Opening text file, reading the keys, closing<br>f=open(&#39;C:/path/of/your/credentials.txt&#39;,&quot;r&quot;)<br>lines=f.readlines()<br>client_id=lines[0]<br>client_key=lines[1]<br>f.close()</pre><pre># Alternatively, ask for a password if you run this script rarely.<br>client_id &lt;- rstudioapi::askForPassword()<br>client_key &lt;- rstudioapi::askForPassword()<br>#  ~~~~~~~~~~~~~~~~~~~~~~~ PASSING CREDENTIALS</pre><pre># Opening text file, reading the keys, closing<br>f=open(&#39;C:/path/of/your/credentials.txt&#39;,&quot;r&quot;)<br>lines=f.readlines()<br>client_id=lines[0]<br>client_key=lines[1]<br>f.close()</pre><p>10. Posting and generating our access token so we can actually query the API.</p><pre># POST command to access API<br>res &lt;- POST(&quot;<a href="https://api.yelp.com/oauth2/token">https://api.yelp.com/oauth2/token</a>&quot;,<br>            body = list(grant_type = &quot;client_credentials&quot;,<br>                        client_id = client_id))</pre><pre># Generate our token<br>token &lt;- content(res)$access_token</pre><p>11. Defining our search parameters. Full documentation on the limitations of the search parameters can be found on Yelp’s website:</p><p><a href="https://www.yelp.com/developers/documentation/v3/business_search">https://www.yelp.com/developers/documentation/v3/business_search</a></p><pre># Search parameters<br>yelp &lt;- &quot;<a href="https://api.yelp.com/">https://api.yelp.com</a>&quot; # Base URL<br>term &lt;- &quot;Avocado Toast&quot; # Search term<br>location &lt;- &quot;10013&quot; # Manhattan Zip Code<br>categories &lt;- NULL # Optional filter, looking at specific category of place... ex: bar, park, etc.<br>limit &lt;- 50 # Biggest limit of results Yelp returns per page<br>radius &lt;- 16000 # Radius, in meters from location. Max is 40,000 (25 miles)</pre><pre># Number of pages you want to query. 50 = 1 page of results. So 600 is 600/50 = 12 pages of results<br>  # The function automatically stops if it reaches an empty page<br>offsetP &lt;- seq(1,601, 50)</pre><p>12. Putting everything together! Running the function and exporting it wherever you’d like as a CSV.</p><pre># This will query our database with our function!<br>storage &lt;- YelpScr(offsetP)</pre><pre># Export it<br>save_path = &quot;C:/users/path/to/where/you/want/it&quot;<br>nameOfExport &lt;- paste0(term,&quot;_&quot;, radius, &quot;_meter_radius_of_&quot;,location)<br>write.table(storage, paste0(save_path, nameOfExport, &quot;.csv&quot;), row.names = FALSE)</pre><h3>And We’re Done! Well, Unless You Want it in Excel…</h3><p>If you’re an Excel person, you have a couple more quick steps. The file format we get out is a .csv, but it can be very easily transformed into a readable Excel file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*vIDra03chnKvDTvdlGhMPA.png" /><figcaption>The output when opened in Excel</figcaption></figure><p>13. Highlight column A → Data Tab → “Text to Columns” → “Next” → Delimit on <strong>Space</strong> → “Finish”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*kWcirUywjq7ZDwSnImNbiA.png" /><figcaption>Delimiting on Space</figcaption></figure><p>Now we have an easily readable Excel file we can save as a .xlsx. From here you can filter, sort, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*5cHFk1jorQCVN7LUR45KEQ.png" /></figure><h3>Legal Considerations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*98CuhTI_4jMDTWCO.png" /></figure><p>Like with my scraping articles, I want to stress reading the Terms of Service prior to using this data for anything. Often companies offer a paid API for commercial use and these free APIs are only for personal use. You can find Yelp’s ToS here:</p><p><a href="https://www.yelp.com/developers/api_terms">https://www.yelp.com/developers/api_terms</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=820aeea44a32" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/programmatically-finding-your-neighborhoods-best-avocado-toast-820aeea44a32">Programmatically Finding Your Neighborhood’s Best Avocado Toast</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Browsers, Scraping and Crawling — Part 2]]></title>
            <link>https://medium.com/analytics-vidhya/automated-browsers-scraping-and-crawling-part-2-cc9e2149a64?source=rss-34ee111f24af------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc9e2149a64</guid>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[browser-automation]]></category>
            <dc:creator><![CDATA[Joseph (Iosif) Mushailov]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 00:41:41 GMT</pubDate>
            <atom:updated>2020-08-01T16:20:11.542Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automated Browsers, Scraping and Crawling — Part 2</h3><h4>A reusable, object-oriented way of automating a browser</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*-GB6inSvQlitoTcU" /><figcaption><strong>Photo: Safar Safarov, Unsplash</strong></figcaption></figure><p>If you are a complete beginner, I strongly urge you to check out <a href="https://medium.com/swlh/automated-browsers-scraping-and-crawling-easier-than-it-looks-af37bd765bca"><strong>part 1</strong></a> of this tutorial first. The beginning of this tutorial will be geared toward beginners, so I will be covering Python class and method basics, but it is only going to cover the bare basics. You can find the TL;DR right below it.</p><p><strong>Why not reuse the code from part 1?</strong></p><p>As you may have noticed, there is a lot of manual input in the previous process. We would need to copy-paste a lot of code into a new Python file or retype it all. A rule of thumb I use: if you ever find yourself needing to duplicate code, try to automate the process as much as possible.</p><h3><strong>Classes and Methods</strong></h3><p>In order to automate, it is crucial to understand what classes and methods are. Let me give you an example: Say we want to be able to create ‘dogs’ in your code easily and efficiently. We need a dog <strong>class</strong> — a blueprint that will contain the essential characteristics that a dog usually has. Within the dog class we first define our <strong>constructor</strong>, which is where we define all attributes and objects that belongs to all dogs we create (this isn’t exactly what a constructor method does but for an initial conceptual understanding we will work off this). The constructor (in Python it appears as __init__) is a <strong>method, </strong>which is something that executes within classes. Methods are not restricted to constructors; a method is just an umbrella term for specific functionality that our class can perform.</p><p>Our __init__ method takes all the default arguments that we feel ALL dogs must be given, which in this case, are breed and name. Our bark method takes our passed name and species, and prints a sentence given those attributes.<strong> </strong>Here is a heavily commented example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23533f402da51b5555efd62e1e10ea68/href">https://medium.com/media/23533f402da51b5555efd62e1e10ea68/href</a></iframe><h3>Just a recap…</h3><ul><li>The <strong>class </strong>is a blueprint of a Dog</li><li>The<strong> __init__</strong> <strong>constructor </strong>inside the class defines all of the attributes that belong to ALL methods of that class.</li><li>The bark method, which takes the attributes of the __init__ method only (so the breed and the name), prints out the class dog name and class dog breed when called.</li></ul><p>Again, this is very simplified, but it’s the view you’ll need to understand the code we write for headless browsing.</p><h3><strong>TL;DR — Just Give Me the Code</strong></h3><h3>Locator Class</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc7d3fa60bab746b057daf551322408f/href">https://medium.com/media/cc7d3fa60bab746b057daf551322408f/href</a></iframe><h3>Navigator Class</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ccda13a3f2066f83e0870cd2f2c9641/href">https://medium.com/media/0ccda13a3f2066f83e0870cd2f2c9641/href</a></iframe><h3><strong>Utilization</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8535b5095bfe2145f4b5adf4d12ad294/href">https://medium.com/media/8535b5095bfe2145f4b5adf4d12ad294/href</a></iframe><h3>Let’s Break it Down: Locator Class</h3><p>The __init__ constructor takes one parameter; the driver object we created. We see that the web element is also included in the constructor, but it is already assigned to “None.” We don’t need to provide it when we instantiate an object of this class. The reason it’s in the constructor is because we will overwrite it later, and we want to be able to use this element everywhere in the class. Once we overwrite it, it will belong to the entire class and all methods within the class can use it.</p><pre>def __init__(self, driver):<br>    self.driver = driver<br>    # We will overwrite the element later<br>    self.element = None</pre><p>The next couple of methods find our web elements for us. The first takes the XPath and the second takes the CSS. We use an <strong>Expected Condition, </strong>part of selenium.webdriver.support. We will cover these in a little more detail later.</p><p>Note that once we locate the element, we assign it to the class’ element object. This way we know it is now accessible by any method within the class.</p><pre># Finding an element via XPATH</pre><pre>def locate_xpath(self, seconds, val):<br><br>      locator = (By.XPATH, val)<br><br>      ele = WebDriverWait(self.driver,    seconds).until(EC.visibility_of_all_elements_located(locator = locator))<br><br>     self.element = ele<br></pre><pre># Doing it with CSS instead</pre><pre> def locate_css(self, seconds, val):<br><br>      locator = (By.CSS_SELECTOR, val)<br><br>      ele = WebDriverWait(self.driver,  seconds).until(EC.visibility_of_all_elements_located(locator = locator))<br><br>      self.element = ele</pre><p>The previous two methods found multiple elements while this one finds a single element. This is useful for clicking and typing into input boxes.</p><pre>def find_single_xpath(self, seconds, val):<br>      locator = (By.XPATH, val)<br>      ele = WebDriverWait(self.driver,  seconds).until(EC.visibility_of_element_located(locator = locator))<br>      self.element = ele</pre><p>These methods are relatively self explanatory. One clicks, one types in, and one pulls the elements on the web-page. We see that append_list is by default none in the scrape() method. If we pass a list, it will append for us. Otherwise it’ll just return the list.</p><pre>def scrape(self, append_list = None):<br>     scraped_list = [x.text for x in self.element]<br>     if append_list is not None:<br>     scraped_list = append_list + scraped_list<br>     return(scraped_list)<br> <br> def click(self):<br>     self.element.click()<br> <br> def typein(self, text):<br>     self.element.sendKeys(text)</pre><h3>Navigator Class</h3><p>This one is a lot simpler than our locator class. Our __init__ only contains the driver, so only the driver we pass is available to all methods. The back() and fwd() methods allow us to pass the number of pages we want to move.</p><pre> def __init__ (self, driver):<br>     self.driver = driver<br> <br><br> def get_pg(self, url):<br>     self.driver.get(url)</pre><pre><br>    def back(self, num_pg):<br>        for i in range(1,num_pg+1):<br>            self.driver.back()</pre><pre> <br>    def fwd(self, num_pg):<br>        for i in range(1,num_pg+1):<br>            self.driver.forward()</pre><p>This method allows us to minimize, maximize or full-screen our browser window. Can be useful to track changes if needed or get the browser off the screen if that isn’t necessary.</p><pre> def change_window(self, how):<br>     if how == “min”:<br>          self.driver.minimize_window()<br>     elif how == “max”:<br>          self.driver.maximize_window()<br>     elif how == “full”:<br>          self.driver.fullscreen_window()</pre><h3>Using the Classes</h3><p>Using the classes is relatively simple once they’re made. We made sure to make all of the methods within each class easily understandable so anyone looking at your code would immediately be able to tell what each method does.</p><p>What we’re doing here is creating an object of the navigator and locator classes. Once we do this we can leverage all the reusable functionality we wrote into our classes</p><pre># Creating our navigator object<br>nav = navigator.navigator(driver)</pre><pre># Creating our locator object<br>loc = locator.locator(driver)</pre><pre># Now we can easily move to our website<br>nav.get_pg(&quot;https://www.google.com&quot;)</pre><pre># And then locate whichever xpath we want<br>loc.locate_xpath(10, &quot;your/xpath/here&quot;)</pre><h3>Expected Conditions</h3><p>With time.sleep(), we have a static, unchanging wait period between each iteration of the algorithm. If we throttle it, say 10 seconds each pass and need to run the algorithm hundreds of times, this can result in huge wait periods. Selenium offers a great way to slow down your web browser or make it optimally efficient without making it static.</p><p>With expected conditions we can make the driver wait exactly as long as it needs to wait and no longer… once the element is loaded, for example, the driver proceeds with its task. You can get all of the documentation <a href="https://www.selenium.dev/selenium/docs/api/py/webdriver_support/selenium.webdriver.support.expected_conditions.html">here</a> but here are some useful expected conditions:</p><pre># Executes the script once the exact element is found. NOTE: this does mean that it is loaded; only found!<br><strong>visibility_of_element_located</strong></pre><pre># Executes the script once the exact element is clickable<br><strong>element_to_be_clickable</strong></pre><pre># Executes the script when at least one element is present. Once all are loaded, it can return all of the elements at once<br><strong>presence_of_all_elements_located</strong></pre><h3>Easier Ways to Locate</h3><p>There are many ways to find your desired elements. The most time consuming way would be to manually compose the XPath or CSS like I described in part 1. This can be useful for some advanced situations when you layer on contains(), starts-with(), etc., but in most beginner cases you can do it much quicker. If you want to delve into the more complex ways to do it, check out this article on <a href="https://www.swtestacademy.com/xpath-selenium/"><strong>Swtestacademy.com</strong></a><strong>.</strong></p><ol><li><strong>Use </strong><a href="https://selectorgadget.com/"><strong>SelectorGadget</strong></a><strong>: </strong>This plugin is fantastic for quick CSS Selection and XPath generation. Simply launch the plugin and click on the desired element. It’ll highlight all similar elements and give you the option to get the XPath.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4EIY19B7YJIl-YFIPKNXkQ.png" /><figcaption>The plugin showing us the .price_color web element</figcaption></figure><p>2. While you’re inspecting a particular element, just go to <strong>Copy → Copy XPath</strong>. Just remember to test this in your console before exporting to Python! You may not always get what you’re looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ho1s7t09oijj_z0OPAaFLA.png" /></figure><h3>Before You Go…</h3><p>Like I mentioned in part 1, please read the ToS and the Robots.txt prior to doing any of this. The benefits of automated browsers are huge, but please automate responsibly.</p><p>Thank you for all of the positive feedback on part 1 and happy browsing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc9e2149a64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/automated-browsers-scraping-and-crawling-part-2-cc9e2149a64">Automated Browsers, Scraping and Crawling — Part 2</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Browsers, Scraping and Crawling — It’s Easier Than it Looks]]></title>
            <link>https://medium.com/analytics-vidhya/automated-browsers-scraping-and-crawling-easier-than-it-looks-af37bd765bca?source=rss-34ee111f24af------2</link>
            <guid isPermaLink="false">https://medium.com/p/af37bd765bca</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[xpath]]></category>
            <dc:creator><![CDATA[Joseph (Iosif) Mushailov]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 19:42:00 GMT</pubDate>
            <atom:updated>2020-08-01T16:20:56.056Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automated Browsers, Scraping and Crawling — It’s Easier Than it Looks</h3><h4>All you need is introductory knowledge of Python and some trial and error</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*kpcbTDTBpDvftaEs" /><figcaption>Photo: <strong>Arif Riyanto, Unsplash</strong></figcaption></figure><p>I noticed there aren’t too many resources on web crawling that are geared toward total beginners, so I decided to make one. Part 1 will be for complete beginners then we’ll take a more object oriented approach to it in Part 2 (which will be a separate article).</p><p><strong>The Complete Basics — What is Web Crawling?</strong></p><p>Web crawling is writing a program to navigate an internet browser. An automated browser (also called a headless browser) can be used to do things a human could do, just faster and better. A couple applications:</p><ol><li><strong>Stress testing a website.</strong> It is possible to run an automated browser to test new elements added to a website. This often requires testing some functionality thousands or (sometimes) millions of times.</li><li><strong>Alert systems. </strong>Having the browser consistently test a web page and alert a development team if the page goes down.</li><li><strong>Web scraping.</strong> Essentially means pulling information off of web pages. We’re going to focus a lot on this along with all of the legal nuances involved.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*-_EqWQTqKgb54SNVU_AfUw.png" /><figcaption>Some of the most popular headless browsers. In order of appearance: Firefox, Chrome, PhantomJS, HTMLUnit &amp; Zombie.JS. We’ll be working with ChromeDriver for this tutorial.</figcaption></figure><p><strong>Setting Up ChromeDriver</strong></p><p>You can search up “ChromeDriver” or just head to the website below:</p><blockquote><a href="https://chromedriver.chromium.org/"><strong>https://chromedriver.chromium.org/</strong></a></blockquote><p>I would recommend downloading the most recent <strong>stable</strong> release of ChromeDriver. Once you’ve downloaded it, check that your actual chrome browser is up to date and matches the driver you downloaded (There may be a slight discrepancy between the two but if the base of the version matches you should be fine. If you don’t have Google Chrome installed, you should also just install that):</p><blockquote>Google Chrome Settings (Click the triple dot on the top-right and click settings) → “About Chrome” → Version #</blockquote><p>Unzip your ChromeDriver and put it <strong>somewhere you will remember.</strong> I personally prefer putting it into my Python Scripts folder (since that’s usually already in PATH). If you put it somewhere else you can simply add it to PATH later. Alternatively you can pass the path in your Python script. Here’s how to add it to PATH:</p><blockquote>Windows → type “Edit System Environment Variables” → “Environment Variables” → PATH → Edit → Put the path you stored your ChromeDriver in. ex: “C:\Users\Joe\Documents\HeadlessBrowsing”</blockquote><h3>TL;DR — Just Give Me the Code</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c885a405fbe3376275e19e1088617a9d/href">https://medium.com/media/c885a405fbe3376275e19e1088617a9d/href</a></iframe><p><strong>For those of you still with me, let’s start line-by-line!</strong></p><p>Let’s open your preferred IDE and start coding:</p><pre>from selenium import webdriver<br>driver = webdriver.Chrome()</pre><p><strong>Or if you’d prefer to pass the path explicitly:</strong></p><pre>from selenium import webdriver<br>driver = webdriver.Chrome(“C:\\Users\\Joe\\Documents\\HeadlessBrowsing\\chromedriver.exe”)</pre><p>If you see this blank web browser, great! Everything is good to go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*U-irP96ea-RJLKsxVdC9ow.png" /><figcaption>The browser should be empty. We’ll navigate to a new site below</figcaption></figure><p>Let’s make our browser go to a website. We’ll use this brilliant website made for scraping practice:</p><pre>driver.get(“<a href="http://books.toscrape.com/">http://books.toscrape.com/</a>”)</pre><p>Once there, press “F12” on your keyboard to inspect the JavaScript &amp; HTML behind the page. This is going to be important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sGXqHmhRkxxQoxZ7NGI7g.png" /></figure><p>To scrape, we want to find either the <strong>CSS </strong>or the <strong>XPATH </strong>of the element. Part 2 will cover this more but for now we just want to get this script working.</p><ol><li>Right click any item price</li><li>Click “Inspect” (Click it again if your code does not match mine below)</li><li>Examine what the navigator highlights for you</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*A3ySluN9eAgxnOPot_t2nw.png" /><figcaption>The <strong>class</strong> of our element is called “price_color”</figcaption></figure><p>4. Switch over to the “Console” in your web browser element inspector and let’s work out our <strong>Pointer </strong>object. We want our algorithm to only target the price in this case, so here’s the syntax:</p><pre># Type this for XPATH. Note the syntax. To pinpoint xpaths we always use $x() in the class. We start with a ‘p’ element whose class is exactly ‘price_color’ </pre><pre>$x(“//p[<a href="http://twitter.com/class">@class</a> = ‘price_color’]”)</pre><pre># This is for CSS. It’s similar to XPATH but the syntax varies a bit. We start with $$ instead of x$</pre><pre>$$(“p[class=’price_color’]”)</pre><p>If we press enter after each of these we see the console returning that they do indeed exist, and there’s 20 of them, which is the exact number of prices on the page. We’ve confirmed these are valid. <strong>These are also valid pointers we can copy-paste into Python once we remove the “$x()”!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*4BBFfqQHifvQ_fe21Qvc4Q.png" /></figure><h3>Our code in Python</h3><pre>from selenium import webdriver<br>driver = webdriver.Chrome()<br>driver.get(“<a href="http://books.toscrape.com/">http://books.toscrape.com/</a>&quot;)</pre><pre># Let’s work with the XPATH<br>prices_xpath = “//p[<a href="http://twitter.com/class">@class</a> = ‘price_color’]”</pre><pre># Method to find multiple elements by their XPATH<br>prices = driver.find_elements_by_xpath(prices_xpath)</pre><pre># Simple list comprehension so we can see the results<br>prices_list = [x.text for x in prices]</pre><pre># Visualize our results<br>print(prices_list)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*ShJTjYrtAon0jIKlbMLzxQ.png" /><figcaption>Here’s what we should see in our Python console</figcaption></figure><p><strong>Harder Elements</strong></p><p>Try doing the same process we outlined above, but with the book names. Notice the difference. This time it’s not one “product_price” element for all the books; each book is unique with its own unique index.</p><p>We can note that the indices decrement by 1 each time and just put the book title into the index. It’s possible to make a <strong>loop</strong> to make a list of id’s and we can pull them that way.</p><p>This is, however, very inefficient; we want a dynamic approach. This is where the power of XPATH comes in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*exw7c13su26nKKQY6jVLPg.png" /><figcaption>Compare this title to the “Soumission” title. They’re quite different.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*JgmCSegWng4oO0LlNumALQ.png" /></figure><p><strong>Using XPATH to make generic paths</strong></p><p>Take a look at the hierarchy of tags leading up to our title (You can click the element selector and hover over elements to see a visual representation of this).</p><blockquote>section → div→ ol → li → article → h3 → a</blockquote><p>We can make a relatively robust path to our names simply by examining the tags in the hierarchy! Let’s take a look at the indentations to track the hierarchy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*Gnbe7sjFLYEoZp5w0wAlWQ.png" /></figure><p>Testing the hierarchy in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*sKPXnC5aOZ413rXv3GVy4Q.png" /><figcaption>It works! We got 20 hits. Let’s transfer to Python</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*MW2V9aAusKnlAAPF-qZbog.png" /><figcaption>Here are what we expect our results to look like</figcaption></figure><p><strong>Progressing to the next page and updating</strong></p><p>We can progress to the next page in two ways:</p><ol><li>Repeat what we’ve been doing above (but we remove the ‘s’ from ‘elements’ since we are no longer looking for multiple objects; just one):</li></ol><pre>next_button_xpath = ‘//div/ul/li/a’<br>next_button = driver.find_element_by_xpath(next_button_xpath)<br>next_button.click()</pre><p>2. Make a simple concatenation loop. Each page changes a single variable: the page number. Noticing this we can .get() every time. We automate our program to look at all the pages</p><pre>import time</pre><pre># Start the for loop and modify the URL each loop. Let&#39;s just scrape the first 3 pages.</pre><pre>for i in range(1,4):<br>    # Progress to the new page<br>    driver.get(&quot;<a href="http://books.toscrape.com/catalogue/page-%7B%7D.html">http://books.toscrape.com/catalogue/page-  {}.html</a>&quot;.format(str(i)))<br>    <br>    # Find our new products and prices<br>    new_prices = driver.find_elements_by_xpath(prices_xpath)<br>    new_products = driver.find_elements_by_xpath(products_xpath)<br>    <br>    # Concatenate our lists<br>    prices_list = prices_list + [x.text for x in new_prices]<br>    products_list = products_list + [x.text for x in new_products]<br>    <br>    # Good etiquette is to slow down the scraper to not put a lot of strain on the server. Let&#39;s make it wait 10 seconds between each scrape<br>    time.sleep(10)</pre><p><strong>And finally… make it pretty!</strong></p><pre>import pandas as pd</pre><pre># Simply pass a dictionary of our lists to our pandas DataFrame<br>scraped_database = pd.DataFrame({“Book Name”: prod_list, “Book Price”:prices_list})</pre><pre># Closing our browser<br>driver.quit()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*GkxZ5mBijkZoOgQB4WUYxw.png" /><figcaption>And there’s our database!</figcaption></figure><h3>But wait, there’s more!</h3><p>I’ll be doing a part 2 where I discuss the advanced features of scraping such as making your browser wait for elements to load, a better approach to time.sleep(), error conditions, more efficient locating, and object oriented approaches to making this process a LOT better using custom classes and methods.</p><p>Before I end this article I wanted to bring up two very important points. When you automate browsers please keep these two considerations in mind:</p><h4>Always, always, always read the robots.txt of a website as well as their terms of service. Many websites completely prohibit scraping of their proprietary data. You must take this into consideration. For more info:</h4><blockquote><a href="https://support.google.com/webmasters/answer/6062608?hl=en">https://support.google.com/webmasters/answer/6062608?hl=en</a></blockquote><h4>If a website does allow scraping, always slow down your scraper! Be nice to their servers and build a time.sleep() condition in. Or, as you’ll see in part 2, a WebDriverWait() in.</h4><p>Thanks for reading and happy headless browsing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af37bd765bca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/automated-browsers-scraping-and-crawling-easier-than-it-looks-af37bd765bca">Automated Browsers, Scraping and Crawling — It’s Easier Than it Looks</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>