<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arthur de Castro Pires on Medium]]></title>
        <description><![CDATA[Stories by Arthur de Castro Pires on Medium]]></description>
        <link>https://medium.com/@piresarthur?source=rss-76ba6986e243------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*25Tln3uL40mhdY0bP8nYFQ.jpeg</url>
            <title>Stories by Arthur de Castro Pires on Medium</title>
            <link>https://medium.com/@piresarthur?source=rss-76ba6986e243------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:03:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@piresarthur/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Web Scrapers with Node.Js — Part 2]]></title>
            <link>https://medium.com/@piresarthur/web-scrapers-with-node-js-part-2-1e8016599c3?source=rss-76ba6986e243------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e8016599c3</guid>
            <category><![CDATA[site-crawlers]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[web-scraping-tools]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-scrapping-strategy]]></category>
            <dc:creator><![CDATA[Arthur de Castro Pires]]></dc:creator>
            <pubDate>Sat, 11 Jul 2020 15:57:42 GMT</pubDate>
            <atom:updated>2020-07-11T15:57:42.665Z</atom:updated>
            <content:encoded><![CDATA[<h3>Web Scrapers with Node.Js — Part 2</h3><p>Hey, this is Arthur again!</p><p>Thanks for coming back, or reaching out for the first time! Well, if this is the first time, check <a href="https://medium.com/@piresarthur/web-scrapers-with-node-js-part-1-112af61ee801">part One</a> to understand what this is all about.</p><p>Let&#39;s get into business, shall we? In this post I will talk about three possible ways of mixing different technologies to scrape information from websites and when is best to use each of them! I&#39;m very excited!! Don&#39;t forget to give me feedback on the end!</p><h3>What technologies do I use?</h3><p>I generally have these three possible technologies to create the scrapers:</p><ol><li><a href="http://Puppeteer v4.0.1pptr.dev">puppeteer</a> alone doing all the steps;</li><li><a href="https://github.com/axios/axios">axios</a> (to make the HTTP requests)+ <a href="http://cheerio.js.org/">cheerio</a> (to parse HTML from websites or JSON for API);</li><li>Mixing puppeteer with axios + cheerio.</li></ol><h3>When do I use each of them?</h3><p>I know we all love to see some code to make the examples more understandable, so here it goes…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxUPGcdy9cg74187mco%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FxUPGcdy9cg74187mco%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FxUPGcdy9cg74187mco%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="348" frameborder="0" scrolling="no"><a href="https://medium.com/media/edd0a974207b436a901425ab94a0c94c/href">https://medium.com/media/edd0a974207b436a901425ab94a0c94c/href</a></iframe><p>Well not yet, but I promise I will show some wonderful code to you in a minute!</p><p>In <a href="https://medium.com/@piresarthur/web-scrapers-with-node-js-part-1-112af61ee801">part One</a> I listed some of the steps I currently use to analyze the websites before creating the robots <em>(I like calling them robots, please don&#39;t judge me over technicalities) </em>and one of the steps was checking if they were a <strong>dynamic</strong> or <strong>static</strong> website. This discovery will guide your decision to choose from options 1–3 above.</p><h3>Static Websites != Dynamic Websites</h3><p>Well, quickly explaining the differences over them is this:</p><p><strong>Static websites</strong> contain all the information directly inserted on the HTML, there&#39;s no need to fetch external data after the page has already loaded, it&#39;s common to have this type of website when the server sends HTML as response as opposed to JSON (usually with APIs as backend), those are called server-side rendered pages.</p><p><strong>Dynamic websites</strong> are different, they have the basic HTML structure to load the page and the content is loaded over time with requests made to the server. This scenario is very common in modern web applications sometimes called Single Page Applications (SPA for short), that use JavaScript frameworks such as <a href="https://angular.io/docs">Angular</a> or libraries such as <a href="https://reactjs.org/">React.js</a>. Basically, it needs a browser to render the first HTML piece and make further requests for the complementary data to present the user.</p><h3>So what?</h3><p>So now that you have refreshed your memory to remember/learn the difference between these two types of website, let me explain how this information guides your decision.</p><p><strong>Puppeteer </strong>is a headless browser, this means that it has an API to allow you to control a browser programmatically. Yeah!! That&#39;s right! Now you can show off to your friends saying that your browser is alive. But the main concept to retain here is that it walks, moves and behaves like a <strong>browser</strong>, so when dealing with <strong>dynamic websites,</strong> <strong>puppeteer</strong> is king 👑!</p><p><strong>Axios </strong>is a &quot;Promise based HTTP client for the browser and node.js&quot;, meaning that it is responsible for making HTTP requests to some URL/Endpoint in order to receive a response with virtually any format, it can be HTML, JSON, XML, etc.</p><p><strong>Cheerio</strong> aims to be the <a href="https://www.w3schools.com/whatis/whatis_jquery.asp">jQuery</a> for the server, so it handles DOM manipulation. It is capable of inserting, selecting, modifying any HTML element, and a number of other functions that really help when creating the scrapers.</p><p>Now that those 3 guys are briefly explained, let me tell you about the scenarios I&#39;ve faced so far:</p><ol><li><strong>Website completely dynamic</strong></li></ol><p>Some websites are completely dynamic so the only way out is using some sort of browser emulator to make the robot look like a browser and receive the js-rendered content. Well, guess what do I have to use in order to create a scraper for this scenario? Yep... You guessed it right! <strong>Puppeteer</strong> for salvation!</p><p><strong>2. Website completely static</strong></p><p>These are the best websites on the point of view of the scraper! On one request the server sends the requester the whole HTML document with everything in its place, ready for us to scrape ❤️.</p><p>Since using HTTP requests are far less time, compute and memory consuming, it makes the robot really fast on extracting the data. Well I kinda gave you the hint for this one, so let me be pragmatic, for this scenario the best option I found is using <strong>axios + cheerio</strong>. Together they are amazing!</p><p>Axios handles the data download/request, and cheerio parses the response, which is HTML in string format, into an object that can be manipulated using common DOM functions (in cheerio/jQuery style) and CSS selectors.</p><p><strong>3. Weird mixed websites</strong></p><p>Hey, on this one, I don&#39;t mean weird in a bad way, but in an exotic sense (These days is more common than I might think, exotic is my personal opinion). Parts of them renders directly on the first call to the server and specific data is gathered over time. To illustrate better this scenario, imagine an online store:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F5wG597QP1fjG7xIfte%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F5wG597QP1fjG7xIfte%2Fsource.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F5wG597QP1fjG7xIfte%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="228" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebb60e9cc64a27c851b97d68acba9959/href">https://medium.com/media/ebb60e9cc64a27c851b97d68acba9959/href</a></iframe><p>The first time the page loads, it has all the links for the products, everything looks good, but when I try to use <strong>axios</strong> to make the request, I discover that the product&#39;s prices are not showing up 🤔🤔🤔 WTF!? right… Yep, that&#39;s it, just the prices loading with a different AJAX request.</p><h4><strong>Why mix axios + cheerio with puppeteer?</strong></h4><p>I didn&#39;t want to point out this explicitly but comparing <strong>puppeteer</strong> and <strong>axios</strong> is not right, both are totally different tools for totally different purposes. In this web scrapers scenario, they are being used for the same output, that&#39;s the only reason I&#39;m going to compare them, alright?</p><p>Puppeteer is heavier than axios in terms of getting responses from websites, the latter only has to request the information with an HTTP call, puppeteer has to boot up a browser, run setups, open a browser tab and then request the website page, is obvious that it will be slower, and that&#39;s exactly why not to use<strong> </strong>only<strong> </strong>puppeteer when they can work together.</p><p>This approach is easily achievable using the strategy I&#39;m going to talk about in Part 3 (Three). But **spoiler alert** what it takes is to split each part of the scraper into separate parts, using the best solution for each of them.</p><h3>What&#39;s next?</h3><p>Cool people!! Thanks for getting this far! In part 3 (three) I will talk about the strategy I&#39;ve been using in terms of code and structure to help me create those scrapers in a fast and standardized way! Check it out!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e8016599c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scrapers with Node.Js — Part 1]]></title>
            <link>https://medium.com/@piresarthur/web-scrapers-with-node-js-part-1-112af61ee801?source=rss-76ba6986e243------2</link>
            <guid isPermaLink="false">https://medium.com/p/112af61ee801</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-scrapping-strategy]]></category>
            <category><![CDATA[site-crawlers]]></category>
            <category><![CDATA[web-scrapping-tips]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Arthur de Castro Pires]]></dc:creator>
            <pubDate>Sat, 11 Jul 2020 14:45:53 GMT</pubDate>
            <atom:updated>2020-07-11T15:58:24.549Z</atom:updated>
            <content:encoded><![CDATA[<h3>Web Scrapers with Node.Js — Part 1</h3><p>Hey, this is Arthur and this is my first story ever written outside my handbook of internal learning, so bear with me and explode me with constructive feedback, would you?</p><p>On this post, I would like to show you how I&#39;ve been creating web scrapers over the past year using Node.Js as backend.</p><p>The part 1 (one) will talk about the 2 types of web scrapers I currently develop.</p><h3>Important notice:</h3><p>This article is not meant to teach you how to create scrapers (I will create one article just for this, explaining some of the possible ways), it&#39;s meant to show what I&#39;ve learned in terms of strategy, architecture, code organization and reusability.</p><h3>Prior knowledge necessary:</h3><ul><li>I have to assume that you already know what web scrapers are and what they do;</li><li>I have to assume that you already know how to code in JavaScript;</li><li>I have to assume that you already know how to create a project and install dependencies with npm.</li></ul><h3>What&#39;s the idea:</h3><p>Whenever I receive the task from our clients to create a new scraper I usually execute the same steps:</p><ol><li>Take a look on the website on a regular user perspective;</li><li>Check if the website has all the information the client is requesting;</li><li>Check if the website is a static website or a dynamic one (this will determine the tech stack to use, more on this later);</li><li>Create the file with the basic structure for the scraper;</li><li>Start gathering the element selectors related to the data I need;</li><li>Understand how the pagination works and how to determine if the crawler reached the last page (I call this &quot;stopping strategy&quot;).</li><li>Finally, code the script! 🥰</li></ol><p>With the exception of the 7th item, those are the &quot;analogic&quot; steps I take in order to create a new scraper.</p><h3>Types of Web Scraper:</h3><p>I like to break down into 2 groups of web scrapers:</p><ol><li><strong>General web scrapers:</strong></li></ol><p>These &quot;guys&quot; tend to scrape ALL the website data you are looking for, and then on a later step, you would process and filter the data you need.</p><p>This type of scraper is better suited for small websites, that don&#39;t have millions of data points to be extracted. Marketplaces such as Amazon.com, eBay, etc wouldn&#39;t be a good idea to use this type of crawler because it could take forever to get all the products they have available (considering you don&#39;t have a massive infrastructure)</p><ol><li><strong>Specific web scrappers:</strong></li></ol><p>These guys are more targetted and have guidance when looking for data to extract, they will download only the necessary data for the task to be completed.</p><p>The best example would be using the search bar on the website to search for the products you really need, and then getting the data from the search results.</p><h3><strong>What&#39;s better?</strong></h3><p>There&#39;s no answer to this question… As it&#39;s very common in technology, it all <strong><em>depends.</em></strong><em> </em>The best type of scraper will be determined based on the timeframe you have to deliver the results, on the size of the website you&#39;re accessing, etc.</p><h3>What&#39;s next?</h3><p>In <a href="https://medium.com/@piresarthur/web-scrapers-with-node-js-part-2-1e8016599c3">part 2</a> (two) I will talk about the technologies I use and how I mix them, and also I will show the fundamental strategy I&#39;ve been using to develop +30 scrapers in less than 1 month (alone).</p><p>Part Two is <a href="https://medium.com/@piresarthur/web-scrapers-with-node-js-part-2-1e8016599c3">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=112af61ee801" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>