<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lisandramelo on Medium]]></title>
        <description><![CDATA[Stories by Lisandramelo on Medium]]></description>
        <link>https://medium.com/@lisandramelo34?source=rss-83410997397f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*27uv4oZMkc0fwgIVmz5DTA.jpeg</url>
            <title>Stories by Lisandramelo on Medium</title>
            <link>https://medium.com/@lisandramelo34?source=rss-83410997397f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:45:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lisandramelo34/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Football Players Network: A Practical Introduction to Graph Theory]]></title>
            <link>https://medium.com/@lisandramelo34/building-a-football-players-network-a-practical-introduction-to-graph-theory-98a46a22d564?source=rss-83410997397f------2</link>
            <guid isPermaLink="false">https://medium.com/p/98a46a22d564</guid>
            <category><![CDATA[graph-theory]]></category>
            <category><![CDATA[football-analytics]]></category>
            <category><![CDATA[football]]></category>
            <category><![CDATA[transfermarkt]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Lisandramelo]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 16:06:46 GMT</pubDate>
            <atom:updated>2024-09-29T16:06:46.300Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>What is a Graph?</strong></h3><p>In 1736, mathematician Leonard Euler tackled a famous problem known as the <strong>Seven Bridges of Königsberg</strong>. The problem arose in Königsberg, a city now called Kaliningrad (formerly in Germany), where two islands were connected by seven bridges to two mainland areas.¹ A representation of the city and the bridges is shown by the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*5-dN--hOYNUj27UlGrK2oQ.jpeg" /><figcaption>Wikipédia</figcaption></figure><p>The question posed was whether one could find a path through the city that crossed every bridge exactly once and returned to the starting point. Euler proved that no such path exists, and in doing so, laid the foundations of <strong>graph theory</strong>.¹</p><p>Graph theory, a branch of mathematics concerned with networks of points (vertices) connected by lines (edges), provides powerful tools for modeling relationships. A <strong>graph</strong> is a data structure highly useful for representing and solving real-world problems.² It has an intuitive visual representation and includes important algorithms and properties for analyzing complex data.</p><p>A graph consists of two main elements: <strong>nodes</strong> (or vertices) and <strong>edges</strong>. The nodes represent individual entities, and the edges represent the relationships or connections between them. In the case of the Seven Bridges of Königsberg problem, the land areas are represented as nodes, while the bridges are represented as edges.</p><p>Here’s the graph representation of Euler’s famous problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/1*GcNLu_x6vZcKpPHhc2xguA.jpeg" /><figcaption>Graph Theory, Konigsberg Problem¹</figcaption></figure><p>You can see in the representation, each part of the city as a circle — node — and each bridge as a line — edge — connecting the respective part of the land.</p><p>In this tutorial, we will apply these concepts to the world of football. We’ll scrape data on football players, build a graph representing relationships between players and their teammates, and visualize it. Our primary tool will be Python, and we’ll use libraries like Selenium for web scraping, NetworkX for graph building, and Plotly for visualization.</p><h3><strong>Building a Graph of Football Players’ Teammates</strong></h3><p>In the context of football, we can use graphs to model relationships between players based on their teams. Each football player (node) is connected to their teammates (other nodes) by edges. The more teammates a player has, the more connected they are in this graph. This can help identify key social players within a team or league.</p><h4><strong>Step 1: Import Necessary Libraries</strong></h4><p>In this tutorial, we’ll employ <strong>Selenium</strong> and <strong>BeautifulSoup</strong> for web scraping.</p><pre># 1. Data Manipulation<br>import pandas as pd # For handling and analyzing structured data<br><br># 2. Graph Construction<br>import networkx as nx # For creating and analyzing network graphs<br><br># 3. Data Visualization<br>import plotly.graph_objs as go # For interactive graph visualization<br><br># 4. Web Scraping<br>from selenium import webdriver # Automates browser interaction<br>from selenium.webdriver.chrome.service import Service # For managing the Chrome service<br>from bs4 import BeautifulSoup # For parsing HTML data<br><br># 5. Utility Libraries<br>import time # For adding delays in scraping<br>import re # For regular expressions, used to clean and extract data</pre><h4><strong>Step 2: Define Constants</strong></h4><p>Next, we’ll define a few constants. For example, `NUM_PLAYERS` specifies the number of players we’ll scrape data for, and `NUM_TEAMMATES` defines how many teammates to retrieve for each player.</p><pre>NUM_PLAYERS = 75 # Modify this value to scrape more players<br>NUM_TEAMMATES = 5 # Number of teammates to scrape for each player</pre><p>Feel free to adjust these values based on your needs. The more players you scrape, the richer your final graph will be, but it will also take more time.</p><h4><strong>Step 3: Set Up Selenium WebDriver</strong></h4><p>Selenium automates interactions with web pages, such as clicking buttons, entering data, and navigating across multiple pages. This makes it ideal for scraping dynamic websites like Transfermarkt, which we’ll use to gather player data.</p><p>To use Selenium it is important to set up the WebDriver in your Python script, ensuring the path to your WebDriver is correctly specified.</p><pre># Configuring Chrome options (e.g., headless mode can be set here if needed)<br>chrome_options = webdriver.chrome.options.Options()<br><br># Path to the Chrome WebDriver executable<br>chrome_driver = &quot;C:\\your\\path\\to\\the\\executable\\chromedriver.exe&quot;<br><br># Initializing the WebDriver service with the executable path<br>service_to_pass = Service(executable_path=chrome_driver)<br><br># Launching the WebDriver with the service and options<br>wd = webdriver.Chrome(service=service_to_pass, options=chrome_options)</pre><h4><strong>Step 4: Access the Webpage</strong></h4><p>Now that Selenium is set up, we’ll use it to access player data from Transfermarkt. Specifically, we’ll extract player names, their respective teams, and URLs to their player profiles.</p><pre># Define the URL for the most valuable players on Transfermarkt<br>url = &#39;https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop?land_id=0&amp;ausrichtung=alle&amp;spielerposition_id=alle&amp;altersklasse=alle&amp;jahrgang=0&amp;kontinent_id=0&amp;plus=1&#39;<br># Open the webpage using the Selenium webdriver<br>wd.get(url)</pre><h4><strong>Step 5: Scraping Player Data</strong></h4><p>Now we going to define two functions that will be neccessary in our scrapping code.</p><p>The first one is used to extract the player information from a web page of the most valuable players and then add the extracted information to a list of players data.</p><pre>def scrape_page(html):<br>    &quot;&quot;&quot;<br>    Extracts player information from a web page of the most valuable players. <br>    The information of each player is added to the players_data list.<br><br>    Args:<br>        html (str): The page source that will be scraped.<br><br>    Returns:<br>        None: Appends player information to the global players_data list.<br>    &quot;&quot;&quot;<br>    soup = BeautifulSoup(html, &#39;html.parser&#39;)<br>    <br>    # Find all player rows in the table (rows with &#39;odd&#39; and &#39;even&#39; class)<br>    player_rows = soup.find_all(&#39;tr&#39;, class_=[&#39;odd&#39;, &#39;even&#39;])<br>    <br>    # Extract player information from each row<br>    for player in player_rows:<br>        name = player.find_all(&#39;td&#39;)[1].find(&#39;img&#39;).get(&#39;alt&#39;)  # Player&#39;s name<br>        player_link = &#39;https://www.transfermarkt.com&#39; + player.find(&#39;td&#39;, class_=&#39;hauptlink&#39;).find(&#39;a&#39;)[&#39;href&#39;]  # Player page link<br>        <br>        # Add player data to the global list (uncomment if needed: rank, age, club, market value)<br>        players_data.append({<br>            &#39;Name&#39;: name,<br>            &#39;Player Page&#39;: player_link,<br>        })</pre><p>The next one is used to get the address of the next page of players in the Pagination.</p><pre>def change_page(html):<br>    &quot;&quot;&quot;<br>    Extracts the URL of the next page from a paginated web page.<br><br>    Args:<br>        html (str): The HTML source of the current page.<br><br>    Returns:<br>        link (str): The URL of the next page to be scraped.<br>    &quot;&quot;&quot;<br>    soup = BeautifulSoup(html, &#39;html.parser&#39;)<br><br>    # Find the &#39;Next Page&#39; button and extract the link<br>    next_button = soup.find(&#39;li&#39;, class_=&#39;tm-pagination__list-item tm-pagination__list-item--icon-next-page&#39;)<br>    link = &#39;https://www.transfermarkt.com&#39; + next_button.find(&#39;a&#39;)[&#39;href&#39;]<br>    <br>    return link</pre><p>With the two functions, we can gather the players using the iteration below.</p><pre># Initialize an empty list to store player data<br>players_data = []<br># Pagination: Continue scraping until you gather 250 players or run out of pages<br>players_count = 0<br><br>while players_count &lt; NUM_PLAYERS:<br>    time.sleep(2)  # Wait for the page to fully load<br>    <br>    # Scrape data from the current page<br>    try:<br>        scrape_page(wd.page_source)  # Scrape player data from the current page<br>        players_count = len(players_data)  # Update the player count<br>        <br>        # Move to the next page using the &#39;change_page&#39; function<br>        url = change_page(wd.page_source)<br>        wd.get(url)<br>    except:<br>        print(&#39;There was a problem accessing the data or no more players were found&#39;)<br>        break</pre><p>Now we can convert the list with data into a Dataframe.</p><pre># Convert the players data list into a DataFrame for future analysis<br>df_players = pd.DataFrame.from_dict(players_data[:NUM_PLAYERS])<br>df_players</pre><h4><strong>Step 6: Scraping Teammates’ Information and Build Graph</strong></h4><p>After gathering data on individual players, we’ll scrape information on their teammates. This is critical for building our graph, as teammates will form the connections (edges) between players.</p><p>For that, we will need a function to get the teammates page for each player.</p><pre>def teamates_page(df_row):<br>    &quot;&quot;&quot;<br>    Modifies a player&#39;s profile URL to navigate to the page listing their most frequently played teammates.<br><br>    Args:<br>        df_row (Series): A row from the DataFrame containing the player&#39;s profile URL.<br><br>    Returns:<br>        url (str): The modified URL pointing to the player&#39;s teammates page.<br>    &quot;&quot;&quot;<br>    # Replace &quot;profil&quot; with &quot;gemeinsameSpiele&quot; in the player&#39;s profile URL to get the teammates page URL<br>    return df_row[&#39;Player Page&#39;].replace(&quot;profil&quot;, &quot;gemeinsameSpiele&quot;)</pre><p>We can apply the function to our Dataframe to populate a new column with the teammates page for each player.</p><pre># Add a new column to the DataFrame with URLs to each player&#39;s teammates page<br>df_players[&#39;Teammates Page&#39;] = df_players.apply(teamates_page, axis=1)</pre><p>Now we can move on to accessing the teammate page and add that information to construct our graph.</p><p>For that we will need another function to scrap the data.</p><pre>def scrape_teammates(html):<br>    &quot;&quot;&quot;<br>    Extracts players information from a web page of a player&#39;s most played teammates. <br>    The information of each player is added to the players_data list.<br><br>    Args:<br>        html (str): The page source that will be scraped.<br><br>    Returns:<br>        teamates_vet (DataFrame): A DataFrame of all the teammates on the page.<br>    &quot;&quot;&quot;<br>    soup = BeautifulSoup(html, &#39;html.parser&#39;)<br>    <br>    # Find the table containing teammate data<br>    teammate_table = soup.find(&#39;table&#39;, {&#39;class&#39;: &#39;items&#39;})<br>    teamates_vet = []  # List to store teammate data<br>    count_runs = 0  # Counter to handle iterations over the rows<br><br>    for tm in teammate_table.find_all(&#39;tr&#39;)[1:]:  # Skip the header row<br>        if count_runs % 3 == 0:  # Process every third row<br>            try:<br>                # Extract teammate name and profile link<br>                teammate_name = tm.find(&#39;img&#39;).get(&#39;alt&#39;)<br>                player_link = &#39;https://www.transfermarkt.com&#39; + tm.find(&#39;td&#39;, class_=&#39;hauptlink&#39;).find(&#39;a&#39;)[&#39;href&#39;]<br>                <br>                # Add teammate data to global players_data and local teamates_vet<br>                players_data.append({<br>                    &#39;Name&#39;: teammate_name,<br>                    &#39;Player Page&#39;: player_link,<br>                })<br>                teamates_vet.append({<br>                    &#39;Name&#39;: teammate_name,<br>                    &#39;Player Page&#39;: player_link,<br>                })<br>            except:<br>                # Handle any errors in data extraction gracefully<br>                print(&quot;There was a problem gathering data from the row below&quot;)<br>                print(tm)<br>        <br>        count_runs += 1  # Increment the counter<br><br>    # Return the collected teammate data as a DataFrame<br>    return pd.DataFrame(teamates_vet)</pre><p>Alongside a function to add the information of each player and their relationship to a graph.</p><pre># 3.8 Add a Player and Their Teammates to the Graph<br><br>def add_to_graph(player, teammates):<br>    &quot;&quot;&quot;<br>    Adds a player and their teammates to a NetworkX graph (G), creating nodes and edges.<br><br>    Args:<br>        player (dict): A dictionary containing player data to be added as a node.<br>        teammates (DataFrame): A DataFrame containing teammate data to be added as nodes and connected to the player.<br><br>    Returns:<br>        None: Modifies the global graph G in place.<br>    &quot;&quot;&quot;<br>    # Add the player as a node if not already in the graph<br>    if player[&#39;Player Page&#39;] not in G:<br>        G.add_node(player[&#39;Player Page&#39;], data=player)<br><br>    # Iterate through the teammates DataFrame<br>    for index, row in teammates.iterrows():<br>        # Add the teammate as a node if not already in the graph<br>        if row[&#39;Player Page&#39;] not in G:<br>            G.add_node(row[&#39;Player Page&#39;], data=row)<br>        <br>        # Add an edge between the player and teammate if no existing edge is found<br>        if not G.has_edge(player[&#39;Player Page&#39;], row[&#39;Player Page&#39;]) and not G.has_edge(row[&#39;Player Page&#39;], player[&#39;Player Page&#39;]):<br>            G.add_edge(player[&#39;Player Page&#39;], row[&#39;Player Page&#39;])</pre><p>The function above is responsible to constructing the graph, it uses the players’ and teammates’ data to create nodes and edges in a NetworkX graph. Each player and their teammates are represented as nodes, while the relationships between them are represented as edges.</p><p>The next step involves iterating over the list of players and applying the scraping function to extract each player’s teammates, then adding the data to the graph.</p><pre># Initialize an empty graph to store players and their relationships<br>G = nx.Graph()<br><br># Iterate through each player in the DataFrame<br>for index, row in df_players.iterrows():<br>    url = row[&#39;Teammates Page&#39;]<br>    wd.get(url)  # Open the player&#39;s teammates page<br>    row = row.drop(labels=&#39;Teammates Page&#39;)  # Drop the teammates page URL from the current row for cleaner data<br>    <br>    # Initialize variables to store teammate data<br>    teammate_count = 0<br>    teammates = pd.DataFrame(columns=[&#39;Name&#39;, &#39;Player Page&#39;])  # Empty DataFrame to store teammates<br>    <br>    # Loop to gather up to 25 teammates from each player&#39;s page<br>    while teammate_count &lt; NUM_TEAMMATES:<br>        # Scrape teammate data from the current page<br>        page_teammates = scrape_teammates(wd.page_source)<br>        <br>        # Concatenate the current page&#39;s teammates to the overall list<br>        teammates = pd.concat([teammates, page_teammates[:NUM_TEAMMATES]])<br>        teammate_count = len(teammates)<br>        <br>        # Try to navigate to the next page of teammates<br>        try:<br>            url = change_page(wd.page_source)<br>            wd.get(url)<br>        except:<br>            print(&#39;There was a problem accessing the data or no more players were found&#39;)<br>            break<br>    <br>    # Print the scraped teammates and player data for debugging<br>    print(teammates)<br>    print(row)<br>    <br>    # Add the player and their teammates to the graph<br>    add_to_graph(row, teammates)</pre><p>This code will iterate through each player, collect their most-played-with teammates, and add both the player and their teammates to the graph, building the network incrementally.</p><h4><strong>Step 7: Visualizing the Football Network</strong></h4><p>Once the graph is constructed, the next step is to visualize the network of relationships between football players. We will use the Plotly library for interactive graph visualization. But one thing that can help with the readability of the graph befor plotting it, is to instead of showing the players pages as the node name, we can show their name. For that we use the function below.</p><pre># 3.5 Extract Player Name from a Transfermarkt Profile URL<br><br>def extract_player_name(url):<br>    &quot;&quot;&quot;<br>    Extracts the player&#39;s name from a Transfermarkt profile URL and formats it.<br>    <br>    Args:<br>        url (str): The URL from which the player&#39;s name will be extracted.<br><br>    Returns:<br>        player_name (str): The player&#39;s name with hyphens replaced by spaces and proper capitalization.<br>        If no match is found, the original URL is returned.<br>    &quot;&quot;&quot;<br>    # Use regex to find the player name part of the URL<br>    match = re.search(r&#39;transfermarkt\.com/([^/]+)/&#39;, url)<br>    <br>    if match:<br>        # Extract the name and replace hyphens with spaces<br>        player_name = match.group(1).replace(&quot;-&quot;, &quot; &quot;)<br>        <br>        # Capitalize the first letter of each word (e.g., &#39;john-doe&#39; becomes &#39;John Doe&#39;)<br>        player_name = player_name.title()<br>        return player_name<br>    <br>    # Return the original URL if no match is found<br>    return url</pre><p>Finally, we’ll use Plotly to visualize the graph. Plotly allows for interactive, web-based visualizations, perfect for exploring complex networks like this one.</p><pre># Position nodes in the graph using a spring layout<br>pos = nx.spring_layout(G)<br><br># Extract edge positions<br>edge_x = []<br>edge_y = []<br>for edge in G.edges():<br>    x0, y0 = pos[edge[0]]<br>    x1, y1 = pos[edge[1]]<br>    edge_x.append(x0)<br>    edge_x.append(x1)<br>    edge_x.append(None)  # For separating edges in the plot<br>    edge_y.append(y0)<br>    edge_y.append(y1)<br>    edge_y.append(None)<br><br># Create the edge trace (lines)<br>edge_trace = go.Scatter(<br>    x=edge_x, y=edge_y,<br>    line=dict(width=1, color=&#39;black&#39;),<br>    hoverinfo=&#39;none&#39;,<br>    mode=&#39;lines&#39;<br>)<br><br># Extract node positions<br>node_x = []<br>node_y = []<br>for node in G.nodes():<br>    x, y = pos[node]<br>    node_x.append(x)<br>    node_y.append(y)<br><br># Node adjacency and hover text<br>node_adjacencies = []<br>node_text = []<br>for node, adjacencies in G.adjacency():<br>    num_connections = len(adjacencies)  # Number of connections for the node<br>    node_adjacencies.append(num_connections)<br>    node_text.append(f&quot;Node {extract_player_name(node)} has {num_connections} connections&quot;)<br><br># Create the node trace (points)<br>node_trace = go.Scatter(<br>    x=node_x, y=node_y,<br>    mode=&#39;markers&#39;,<br>    hoverinfo=&#39;text&#39;,<br>    marker=dict(<br>        showscale=True,<br>        colorscale=&#39;viridis&#39;,<br>        size=10,<br>        color=node_adjacencies,<br>        colorbar=dict(<br>            thickness=15,<br>            title=&#39;Node Connections&#39;,<br>            xanchor=&#39;left&#39;,<br>            titleside=&#39;right&#39;<br>        ),<br>    )<br>)<br><br># Assign node hover text<br>node_trace.text = node_text<br><br># Create the plot<br>fig = go.Figure(data=[edge_trace, node_trace],<br>                layout=go.Layout(<br>                    title=&#39;Interactive Network Graph&#39;,<br>                    titlefont_size=16,<br>                    showlegend=False,<br>                    hovermode=&#39;closest&#39;,<br>                    margin=dict(b=0, l=0, r=0, t=40),<br>                    annotations=[dict(<br>                        text=&quot;Network visualization using NetworkX and Plotly&quot;,<br>                        showarrow=False,<br>                        xref=&quot;paper&quot;, yref=&quot;paper&quot;,<br>                        x=0.005, y=-0.002<br>                    )],<br>                    xaxis=dict(showgrid=False, zeroline=False),<br>                    yaxis=dict(showgrid=False, zeroline=False)<br>                )<br>)<br><br># Display the graph<br>fig.show()</pre><p>In the visualization, each node represents a player, and the edges show teammate relationships. Larger nodes indicate players with more connections. You can hover over each node to view player details, and the graph layout (spring layout) ensures that players with more connections are placed closer together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bn-Es_z5jMfG8CAJhQ8iyQ.jpeg" /><figcaption>Graph visualization for the parameters set inicially</figcaption></figure><p>The resulting graph provides a visual summary of player relationships. Players with many teammates will appear as larger, more central nodes, while isolated players or those with fewer connections will be on the periphery. This type of visualization can highlight key players who serve as the “connectors” within their teams or leagues.</p><h4><strong>Conclusion and Next Steps</strong></h4><p>In this tutorial, we’ve built a graph using data scraped from Transfermarkt and visualized the relationships between football players and their teammates. This exercise provides a foundation for more advanced analysis, such as identifying key players using graph algorithms like centrality or even expanding the dataset to include players more players and more teammates for each player.</p><h4>Github Repository</h4><p>If you are looking for the Project’s Repo you can access it here: <a href="https://github.com/veronicamars73/Football-Players-Network">https://github.com/veronicamars73/Football-Players-Network</a></p><h4><strong>References</strong></h4><p>¹ George, Betsy. “Graph Theory, Konigsberg Problem”. In <em>*Encyclopedia of GIS*</em>, eds. Shashi Shekar and Hui Xiong, SpringerScience+Business Media, LLC., 2008, pp. 409–412.</p><p>² Goldbarg, Marco Cesar; Goldbarg, Elizabeth. “Grafos — Conceitos, Algoritmos e Aplicações”. 1o ed, Elsevier Editora Ltda, 2012.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98a46a22d564" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Data from WhoScore: A WebScraping Project with Selenium]]></title>
            <link>https://medium.com/@lisandramelo34/getting-data-from-whoscore-a-webscraping-project-with-selenium-60441eb9a952?source=rss-83410997397f------2</link>
            <guid isPermaLink="false">https://medium.com/p/60441eb9a952</guid>
            <category><![CDATA[football]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Lisandramelo]]></dc:creator>
            <pubDate>Fri, 07 Jun 2024 10:26:42 GMT</pubDate>
            <atom:updated>2024-06-07T10:26:42.202Z</atom:updated>
            <content:encoded><![CDATA[<p>A while ago, I wrote a <a href="https://dev.to/lisandramelo/extracting-data-from-transfermarkt-an-introduction-to-webscraping-%202i1c">post on dev.to about Web Scraping with Python, Beautiful Soup and Requests</a>. While that post provided an overview of the data scraping process for most websites, in some cases this approach is not sufficient. Some websites are set to prevent automated access for data scraping. Usually, these websites aim to avoid robots that can cause server overloads or users who can obtain information and use it without due credit.</p><p>Despite these protections, the use of automation tools on websites can be essential for creating automations, testing, and data analysis solutions for web applications. Therefore, learning about these tools is essential for developing, testing, and analyzing websites with anti-crawler protections.</p><p>In this tutorial, we will explore how to use Selenium to access and extract data from websites that have more advanced protection mechanisms. Selenium is a browser automation tool that can simulate human interaction with web pages, allowing you to bypass some restrictions imposed on traditional web scraping scripts.</p><p><strong>Library Requirements and Installation</strong></p><p>Before we start producing the code itself, we will need to ensure that we have all the necessary tools. Ensure that you meet the following requirements:</p><p>- <a href="https://www.python.org/downloads/">Python installed on your machine</a>;</p><p>- Selenium Library for Python;</p><p>- BeautifulSoup Library for Python;</p><p>- Pandas Library for Python;</p><p>- A Browser WebDriver.</p><p><strong>Installation of Required Tools</strong></p><p><strong>Selenium Installation</strong></p><p>The Selenium library for Python is a tool for automating interactions with web browsers. Selenium allows you to write Python scripts that perform actions in a browser, such as clicking buttons, filling out forms, navigating between pages, and extracting data from websites with anti-crawler protections.</p><p>To install the <a href="https://pypi.org/project/selenium/">Selenium</a> library, you can use <a href="https://pypi.org/project/pip/">pip</a>.</p><pre>pip install selenium</pre><p><strong>BeautifulSoup Installation</strong></p><p>The BeautifulSoup library is a tool for extracting data from HTML and XML files. It makes browsing, searching, and modifying HTML and XML documents simple and effective.</p><p>To install the <a href="https://pypi.org/project/beautifulsoup4/">Beautiful Soup</a> library, you can also use <a href="https://pypi.org/project/pip/">pip</a>.</p><pre>pip install beautifulsoup4</pre><p><strong>Pandas Installation</strong></p><p>The Pandas library provides high-performance data structures and data manipulation functions, making data analysis processes more efficient and intuitive.</p><p>To install the <a href="https://pypi.org/project/pandas/">Pandas</a> library, you can also use <a href="https://pypi.org/project/pip/">pip</a>.</p><pre>pip install pandas</pre><p><strong>WebDriver Download</strong></p><p>The Selenium tool uses WebDrivers to perform automation activities. A WebDriver is a tool used to automate testing in web browsers. It allows developers and testers to control a browser (such as Chrome, Firefox, or Safari) programmatically, simulating real user interaction. One of the most popular WebDrivers is Selenium WebDriver, which supports several browsers and programming languages, such as Python, Java, and C#.</p><p><a href="https://developer.chrome.com/docs/chromedriver?hl=pt-br">ChromeDriver</a> is a specific component of Selenium WebDriver that allows you to control a Google Chrome browser. It serves as a bridge between Selenium WebDriver and the browser, enabling automated tests to be run in Chrome.</p><p>To download it, go to the <a href="https://googlechromelabs.github.io/chrome-for-testing/">Google Chrome download site</a> and select the version of <em>chromedriver </em>compatible with your Operating System.</p><figure><img alt="The Chrome page of downloads Chrome Driver Stable showing the Versions Available" src="https://cdn-images-1.medium.com/max/1024/1*5AVbP4qB7rD6vjcuqt3ydg.png" /><figcaption>Chrome Driver Stable Versions Available</figcaption></figure><p>After downloading the corresponding compressed file, extract the files and store the location of the `chromedriver.exe` file, as it will be used later on the project.</p><p><strong>Coding</strong></p><p>Our first step in coding the project will be to import the libraries we will use. To do this, use the following code snippet. The snippet imports BeautifulSoup, the Selenium library, and the Pandas library.</p><pre>from bs4 import BeautifulSoup<br>from selenium.webdriver.chrome.service import Service<br>from selenium import webdriver<br>import pandas as pd</pre><p>With the imported libraries, we can configure our web driver to access internet pages. For configuration, the <a href="https://www.selenium.dev/documentation/webdriver/browsers/chrome/">Selenium WebDriver</a> constructor needs a <a href="https://www.selenium.dev/documentation/webdriver/drivers/service/">Service</a>, which is used to configure and manage the WebDriver service for Chrome, such as specifying the path to the ChromeDriver executable and defining additional arguments; and settings, or options, for the Chrome browser instance. Therefore, in the section below, we are setting and instantiating the object responsible for obtaining and manipulating the page.</p><pre>chrome_options = webdriver.chrome.options.Options()<br>chrome_driver = &quot;file/path/of/chromedriver.exe&quot;<br>service_to_pass = Service(executable_path=chrome_driver)<br>wd = webdriver.Chrome(service=service_to_pass, options=chrome_options)</pre><p>Now, we will execute the action to obtain data from the webpage we choose. To do this, we will use the <a href="https://www.selenium.dev/documentation/webdriver/interactions/navigation/">get()</a> method from the created object. The get method is responsible for opening the website. Later, we will use the WebDriver object property called <a href="https://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webdriver.WebDriver.page_source">page_source</a>, which returns the source code (content) of the webpage.</p><p>Moreover, you will need to chose the address of the page you want to access. In the project, I used the website <a href="https://www.whoscored.com/">www.whoscored.com</a>, specifically its page about <a href="https://www.whoscored.com/Statistics">statistics</a>. This website has anti-crawler protection and, thus, using it as a test, we can see the effectiveness of the tool.</p><figure><img alt="A screenshot of the web page we are using to scrap" src="https://cdn-images-1.medium.com/max/1024/1*YPEFs-1rZ6ejc6QwlopMPg.png" /><figcaption>Webpage used in the Tutorial</figcaption></figure><pre>URL_BASE = &quot;https://www.whoscored.com/Statistics&quot;<br>wd.get(URL_BASE)<br>soup_file = wd.page_source</pre><p>After this section, we now have access to the entire HTML code of the webpage. We may use this information for testing, analysis or necessary transformations. Additionally, we can use a WebDriver to fill out forms, click on buttons, or navigate between pages. For the current project, I will only propose cleaning the unstructured information present on the page and transforming it into structured data.</p><p>For this, we will use the previously imported Pandas and BeautifulSoup libraries. If you have any trouble following the code below, I recommend reading my tutorial <a href="https://dev.to/lisandramelo/extracting-data-from-transfermarkt-an-introduction-to-webscraping-2i1c">Introduction to WebScraping</a>, as it introduces each of the functions and methods used below.</p><p>The first part of the processing consists of passing the source code through BeautifulSoup’s HTML parser.</p><pre>soup_page = BeautifulSoup(soup_file, &quot;html.parser&quot;)</pre><p>Now, we will fetch the data we want. In this project, we will obtain data from the table highlighted below. These are summarized data with the 20 best clubs according to the grades assigned by the website.</p><figure><img alt="A screenshot of the main table scraped in the article" src="https://cdn-images-1.medium.com/max/1024/1*otORbP7UCis_KgFGuvSkKA.png" /><figcaption><em>Table used for Scraping</em></figcaption></figure><p>First, we will access the chosen table in HTML by its ID.</p><pre>main_table = soup_page.find(&#39;div&#39;, {&#39;id&#39;: &#39;top-team-stats-summary&#39;})<br>team_sum_stats_table = main_table.find(&#39;table&#39;, {&#39;id&#39;: &#39;top-team-stats-summary-grid&#39;})</pre><p>Now, within the table, we will get the existing column names.</p><pre>team_sum_stats_header = team_sum_stats_table.find_all(&#39;th&#39;)<br>header_columns = [column_name.text for column_name in team_sum_stats_header]</pre><p>Note that the code above uses <a href="https://www.w3schools.com/python/python_lists_comprehension.asp">List Comprehensions</a>. This type of resource uses a cleaner and simpler syntax to create lists from other lists. Hence, the code above is equivalent to the one proposed below.</p><pre>team_sum_stats_header = team_sum_stats_table.find_all(&#39;th&#39;)<br>header_columns = []<br>for column_name in team_sum_stats_header:<br>  header_columns.append(column_name.text)</pre><p>Now, let’s get the cell data from our table. To do it, use the code below..</p><pre>team_sum_stats_body = team_sum_stats_table.find(&#39;tbody&#39;).find_all(&#39;tr&#39;)<br>teams_stats = [[cell_value.text for cell_value in row.find_all(&#39;td&#39;)] for row in team_sum_stats_body]</pre><p>This time, the code has two nested list comprehensions. Maybe it seems complex, but in reality, the proposed code does the same as the code below.</p><pre>teams_stats = []<br>for row in team_sum_stats_body:<br>  cells = row.find_all(&#39;td&#39;)<br>  row_values = []<br>  for cell_value in cells:<br>    row_values.append(cell_value.text)<br>    teams_stats.append(row_values)</pre><p>Now, we already have our columns and rows of values. We can then create our DataFrame with structured information that was present on the website. To do this, use the code below.</p><pre>df_sum = pd.DataFrame(teams_stats, columns=header_columns)<br>print(df_sum.head())</pre><p>The code will return the first five records in the website table like the image below.</p><figure><img alt="A screenshot of the program output" src="https://cdn-images-1.medium.com/max/964/1*frp7jfw3GpR0y-odN399vw.png" /><figcaption><em>Program Output</em></figcaption></figure><p><strong>Repository</strong></p><p>The complete project code is available in my <a href="https://github.com/veronicamars73/Getting-WhoScored-Data">github repository</a>.</p><p><strong>Final Comments</strong></p><p>I hope the tutorial helps in some way. I encourage you to implement your own versions and I’m happy to help in any way I can.</p><p>I leave my email<em> lisandramelo34@gmail.com</em> and my profile on <a href="https://www.linkedin.com/in/melo-lisandra">LinkedIn</a> if you want to get in touch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60441eb9a952" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>