<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ondra Urban on Medium]]></title>
        <description><![CDATA[Stories by Ondra Urban on Medium]]></description>
        <link>https://medium.com/@ondraurban?source=rss-e99088a1a720------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vrciCeznwCo-ua1PvaGAIA.jpeg</url>
            <title>Stories by Ondra Urban on Medium</title>
            <link>https://medium.com/@ondraurban?source=rss-e99088a1a720------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:04:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ondraurban/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to scrape the web with Playwright]]></title>
            <link>https://medium.com/@ondraurban/how-to-scrape-the-web-with-playwright-ece1ced75f73?source=rss-e99088a1a720------2</link>
            <guid isPermaLink="false">https://medium.com/p/ece1ced75f73</guid>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[playwrights]]></category>
            <category><![CDATA[puppeteer]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Ondra Urban]]></dc:creator>
            <pubDate>Mon, 03 Aug 2020 14:47:43 GMT</pubDate>
            <atom:updated>2020-08-05T13:31:11.704Z</atom:updated>
            <content:encoded><![CDATA[<h4>Playwright is a rising star in the web scraping and automation space. If you thought Puppeteer was powerful, Playwright will blow your mind.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cMjXDI8ebHqNA8EVjo_KlQ.jpeg" /><figcaption>Playwright logo</figcaption></figure><p><a href="https://github.com/microsoft/playwright">Playwright</a> is a browser automation library very similar to <a href="https://github.com/puppeteer/puppeteer">Puppeteer</a>. Both allow you to control a web browser with only a few lines of code. The possibilities are endless. From automating mundane tasks and testing web applications to data mining.</p><p>With Playwright you can run Firefox and Safari (WebKit), not only Chromium based browsers. It will also save you time, because Playwright automates away repetitive code, such as waiting for buttons to appear in the page.</p><blockquote>You don’t need to be familiar with Playwright, Puppeteer or web scraping to enjoy this tutorial, but knowledge of HTML, CSS and JavaScript is expected.</blockquote><p>In this tutorial you’ll learn how to:</p><ol><li><strong>Start a browser with Playwright</strong></li><li><strong>Click buttons and wait for actions</strong></li><li><strong>Extract data from a website</strong></li></ol><h3>The Project</h3><p>To showcase the basics of Playwright, we will create a simple scraper that extracts data about <a href="https://github.com/topics">GitHub Topics</a>. You’ll be able to select a topic and the scraper will return information about repositories tagged with this topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*gNQkSVS-2f4tk11yVIW4Cw.png" /><figcaption>The page for JavaScript GitHub Topic</figcaption></figure><p>We will use Playwright to start a browser, open the GitHub topic page, click the <em>Load more</em> button to display more repositories, and then extract the following information:</p><ul><li>Owner</li><li>Name</li><li>URL</li><li>Number of stars</li><li>Description</li><li>List of repository topics</li></ul><h3>Installation</h3><p>To use Playwright you’ll need <a href="https://nodejs.org/">Node.js</a> version higher than 10 and a package manager. We’ll use npm, which comes preinstalled with Node.js. You can confirm their existence on your machine by running:</p><pre>node -v &amp;&amp; npm -v</pre><p>If you’re missing either Node.js or NPM, visit the <a href="https://docs.npmjs.com/downloading-and-installing-node-js-and-npm">installation tutorial</a> to get started.</p><p>Now that we know our environment checks out, let’s create a new project and install Playwright.</p><pre>mkdir playwright-scraper &amp;&amp; cd playwright-scraper<br>npm init -y<br>npm i playwright</pre><blockquote>The first time you install Playwright, it will download browser binaries, so the installation may take a bit longer.</blockquote><h3>Building a scraper</h3><p>Creating a scraper with Playwright is surprisingly easy, even if you have no previous scraping experience. If you understand JavaScript and CSS, it will be a piece of cake.</p><p>In your project folder, create a file called scraper.js (or choose any other name) and open it in your favorite code editor. First, we will confirm that Playwright is correctly installed and working by running a simple script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7566eded0da9ca7a10674ba51646365f/href">https://medium.com/media/7566eded0da9ca7a10674ba51646365f/href</a></iframe><p>Now run it using your code editor or by executing the following command in your project folder.</p><pre>node scraper.js</pre><p>If you saw a Chromium window open and the GitHub Topics page successfully loaded, congratulations, you just robotized your web browser with Playwright!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-g0d8utMzc4sGG7ACHOt7Q.png" /><figcaption>JavaScript GitHub topic</figcaption></figure><h4>Loading more repositories</h4><p>When you first open the topic page, the number of displayed repositories is limited to 30. You can load more by clicking the <em>Load more…</em> button at the bottom of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XgVRjVfZ7f-H4GGgAclBaQ.png" /></figure><p>There are two things we need to tell Playwright to load more repositories:</p><ol><li><strong>Click</strong> the <em>Load more…</em> button.</li><li><strong>Wait</strong> for the repositories to load.</li></ol><p>Clicking buttons is extremely easy with Playwright. By prefixing text= to a string you’re looking for, Playwright will find the element that includes this string and click it. It will also wait for the element to appear if it’s not rendered on the page yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3c8mpVdheQFEGZDRhew-0A.png" /><figcaption>Clicking a button</figcaption></figure><p>This is a huge improvement over Puppeteer and it makes Playwright lovely to work with.</p><p>After clicking, we need to wait for the repositories to load. If we didn’t, the scraper could finish before the new repositories show up on the page and we would miss that data. <a href="https://playwright.dev/#version=v1.2.1&amp;path=docs%2Fapi.md&amp;q=pagewaitforfunctionpagefunction-arg-options">page.waitForFunction()</a> allows you to execute a function inside the browser and wait until the function returns true .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qX1qjuy6E4mpFQDVIjHq9g.png" /><figcaption>Waiting for</figcaption></figure><p>To find that article.border selector, we used browser Dev Tools, which you can open in most browsers by right-clicking anywhere on the page and selecting <strong>Inspect</strong>. It means: Select the &lt;article&gt; tag with the border class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4qVh-PhMS3TVLYvpT-J-Q.png" /><figcaption>Chrome Dev Tools</figcaption></figure><p>Let’s plug this into our code and do a test run.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa753bee4d031619d59a54786b9d7d04/href">https://medium.com/media/fa753bee4d031619d59a54786b9d7d04/href</a></iframe><p>If you watch the run, you’ll see that the browser first scrolls down and clicks the <em>Load more…</em> button, which changes the text into <em>Loading more</em>. After a second or two, you’ll see the next batch of 30 repositories appear. Great job!</p><h4>Extracting data</h4><p>Now that we know how to load more repositories, we will extract the data we want. To do this, we’ll use the <a href="https://playwright.dev/#version=v1.2.1&amp;path=docs%2Fapi.md&amp;q=pageevalselector-pagefunction-arg-1">page.$$eval</a> function. It tells the browser to find certain elements and then execute a JavaScript function with those elements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LjAgVv43-2_Z9xd_BDiQXQ.png" /><figcaption>Extracting data from page</figcaption></figure><p>It works like this: page.$$evalfinds our repositories and executes the provided function in the browser. We get repoCards which is an Array of all the repo elements. The return value of the function becomes the return value of the page.$$eval call. Thanks to Playwright, you can pull data out of the browser and save them to a variable in Node.js. Magic!</p><p>If you’re struggling to understand the extraction code itself, be sure to check out <a href="https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors">this guide on working with CSS selectors</a> and <a href="https://javascript.info/searching-elements-dom#querySelectorAll">this tutorial on using those selectors to find HTML elements</a>.</p><p>And here’s the code with extraction included. When you run it, you’ll see 60 repositories with their information printed to the console.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5613717f2ba4b1d3e15b3a1d285a090b/href">https://medium.com/media/5613717f2ba4b1d3e15b3a1d285a090b/href</a></iframe><h3>Conclusion</h3><p>In this tutorial we learned how to start a browser with <a href="https://playwright.dev/">Playwright</a>, and control its actions with some of Playwright’s most useful functions: page.click() to emulate mouse clicks, page.waitForFunction() to wait for things to happen and page.$$eval() to extract data from a browser page.</p><p>But we’ve only scratched the surface of what’s possible with Playwright. You can log into websites, fill forms, intercept network communication, and most importantly, use almost any browser in existence. Where will you take this project next? How about turning it into a command-line interface (CLI) tool that takes a topic and number of repositories on input and outputs a file with the repositories? You can do it now. Happy scraping!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ece1ced75f73" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>