<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Héric Libong on Medium]]></title>
        <description><![CDATA[Stories by Héric Libong on Medium]]></description>
        <link>https://medium.com/@rico69?source=rss-3205a7458701------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*NiE05pq0pEhKM6jI.jpg</url>
            <title>Stories by Héric Libong on Medium</title>
            <link>https://medium.com/@rico69?source=rss-3205a7458701------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:52:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rico69/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scraping Fifa men’s ranking with Scrapy and hidden API]]></title>
            <link>https://medium.com/@rico69/scraping-fifa-mens-ranking-with-scrapy-and-hidden-api-7799570b7737?source=rss-3205a7458701------2</link>
            <guid isPermaLink="false">https://medium.com/p/7799570b7737</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[scrapy]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Héric Libong]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 10:18:17 GMT</pubDate>
            <atom:updated>2024-07-19T10:06:58.871Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zmeMYNh_ww8Rmzu3nt3Lqw.jpeg" /></figure><p><strong>Collect the 1992–2023 Fifa rankings in seconds using the internal API of the Fifa website</strong></p><blockquote>PS : This article was written in April 2023. Since then, the Fifa website has made a few changes to these urls. That said, the approach and methods remain the same. The link at the end of the article takes you to the repo with the full code.</blockquote><p>After the publication of the latest FIFA ranking on April 6th, I visited the association’s website to examine their procedures and potentially obtain the historical ranking since its creation in 1992. While I had access to their procedures, I could not find any public API or an easy way to download the data from their tables. Consequently, I decided to collect the data myself.</p><h4>Find Hidden API</h4><p>When it comes to extracting data from a dynamic website, my first approach is to search for a hidden API. Not only does this allow me to avoid getting blocked, but it also enables me to quickly obtain structured data without relying on long and complex code. Although not all sites have a hidden API, finding one that allows access to the data can greatly simplify the task.</p><p>So, I immediately opened the browser inspector and… Bingo! After refreshing the page, I found an <a href="https://www.fifa.com/api/ranking-overview?locale=en&amp;dateId=id13974">API URL</a> in the XHR tab, which returns the complete current ranking data in a JSON format list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYNFmszp_yBQlLlPyAtMqg.png" /></figure><p>If we perform the same process by selecting a different date from the drop-down menu, we observe that the page consistently returns the same type of data. Our objective, therefore, will be to retrieve all of these pages. But for now, let’s focus on how to retrieve the latest ranking data using Scrapy.</p><h4>Get data with Scrapy</h4><p>have chosen to use Scrapy because it is my preferred scraping library and it provides a powerful and flexible framework for web scraping tasks. For this project, it can be used outside of its environment, from a simple file, but to speed up the process, we will create a Scrapy project with the default template spider package :</p><pre>pip install scrapy</pre><pre>scrapy startproject &lt;projectname&gt;</pre><p>In the <em>spiders</em> folder :</p><pre>scrapy genspider &lt;spidername&gt; site.com</pre><p>The first objective is to extract the data from the HTTP response and return it as a list, as we see it on the API endpoint link. Since the URL returns JSON data, we need to:</p><ul><li>Import the <em>json</em> library</li><li>Enter the correct domain name: <strong>fifa.com</strong></li><li>Enter the endpoint URL into the <em>starts_url</em> variable</li><li>In the <em>parse</em> function, load the content of the HTTP response into the data variable as a Python dictionary</li><li>Loop through the rankings of all teams and return the data.</li></ul><pre>import scrapy<br>import json  #The URL returns data in JSON format.<br><br><br>class MenrankingSpider(scrapy.Spider):<br>    name = &#39;menranking&#39;<br>    allowed_domains = [&#39;fifa.com&#39;]<br>    start_urls = [&#39;https://www.fifa.com/api/ranking-overview?locale=en&amp;dateId=id13974&#39;]<br><br>    def parse(self, response):<br>        data = json.loads(response.body) #loads the contents of the HTTP response into the `data` variable as a Python dictionary.<br>        for ranking_data in data[&#39;rankings&#39;]:<br>            yield ranking_data</pre><p>In the terminal :</p><pre>scrapy runspider menranking.py</pre><p>The code instantly returns all the data from the endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybBVWKyHRj1Z6i1RTGfdgQ.png" /></figure><p>From there, we can select the data we want to keep for exporting in different formats. For me, I only need:</p><ul><li>the country name of the team</li><li>the total points</li><li>the previous points</li><li>the rank</li><li>the previous rank</li><li>the flag link</li><li>the country URL</li><li>the confederation.</li></ul><pre>import scrapy<br>import json  #The URL returns data in JSON format.<br><br>class MenrankingSpider(scrapy.Spider):<br>    name = &#39;menranking&#39;<br>    allowed_domains = [&#39;fifa.com&#39;]<br>    start_urls = [&#39;https://www.fifa.com/api/ranking-overview?locale=en&amp;dateId=id13974&#39;]<br><br>    def parse(self, response):<br>        data = json.loads(response.body) #loads the contents of the HTTP response into the `data` variable as a Python dictionary.<br>        base_url = &#39;https://www.fifa.com&#39;<br>        for ranking_data in data[&#39;rankings&#39;]:<br>            yield {<br>                &#39;country&#39;: ranking_data[&#39;rankingItem&#39;][&#39;name&#39;],<br>                &#39;rank&#39;: ranking_data[&#39;rankingItem&#39;][&#39;rank&#39;],<br>                &#39;previousRank&#39; : ranking_data[&#39;rankingItem&#39;][&#39;previousRank&#39;],<br>                &#39;totalPoints&#39;: ranking_data[&#39;rankingItem&#39;][&#39;totalPoints&#39;],<br>                &#39;previousPoints&#39;: ranking_data[&#39;previousPoints&#39;],<br>                &#39;flagUrl&#39;: ranking_data[&#39;rankingItem&#39;][&#39;flag&#39;][&#39;src&#39;],<br>                &#39;countryUrl&#39;: base_url + ranking_data[&#39;rankingItem&#39;][&#39;countryURL&#39;],<br>                &#39;conf&#39;: ranking_data[&#39;tag&#39;][&#39;text&#39;] <br>            }</pre><p>Again, the data is returned instantly. However, we notice that we do not have the publication date of the ranking. We will see later how to fix this.</p><h4>Get all data from all pages</h4><p>The main challenge when extracting data from a hidden API is to find the keys that allow you to navigate from one page to another. These are usually found in the metadata, at the beginning or sometimes at the end of the list of data returned by the endpoint, and vary depending on the API.</p><p>It can be a page number, the total number of pages, an ID number, or even a boolean indicating that there is a next page.</p><p>The issue here is that there are no metadata available. Moreover, as we mentioned earlier, <strong>no date</strong> is specified. Thus, at first glance, we do not have a key that links the ranking dates to the IDs, and hence to the different data pages.</p><p>My solution — there may be others — consisted of examining the page’s source code. In the middle of the code, it was found that the corresponding dates and identifiers were all a list of dictionaries containing a key-value pair with “id” for the ID number, and “text” for the date.</p><pre>&quot;dates&quot;:[{&quot;id&quot;:&quot;id13974&quot;,&quot;text&quot;:&quot;6 Apr 2023&quot;},{&quot;id&quot;:&quot;id13869&quot;,&quot;text&quot;:&quot;22 Dec 2022&quot;}...{&quot;id&quot;:&quot;id2&quot;,&quot;text&quot;:&quot;8 Aug 1993&quot;},{&quot;id&quot;:&quot;id1&quot;,&quot;text&quot;:&quot;31 Dec 1992&quot;}]</pre><p>We just need to extract this list, turn it into a JSON content, and write a function to send it to the API URL. We can first test this by writing the function in a separate file, for example: test.py.</p><pre>import scrapy<br>import json<br><br>class MySpider(scrapy.Spider):<br>    name = &#39;myspider&#39;<br>    start_urls = [&#39;https://www.fifa.com/fifa-world-ranking/men?dateId=id13974&#39;]<br><br>    def parse(self, response):<br>        #extracts content of the first script tag in the response that contains the string `&quot;dates&quot;` with all the json content of the page.<br>        script_content = response.xpath(&#39;//script[contains(., &quot;dates&quot;)]/text()&#39;).extract_first()<br>        #transform it into a Python dictionary<br>        date_list = json.loads(script_content)<br>        #extract the list of dates from the dictionary<br>        date_list = date_list[&#39;props&#39;][&#39;pageProps&#39;][&#39;pageData&#39;][&#39;ranking&#39;][&#39;dates&#39;]<br>        <br>        print(date_list)</pre><pre>scrapy runspider test.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MfUUJiDmceZuNbNCdFMI_w.png" /></figure><p>In order to extract the list of dates, I used <a href="https://jsonformatter.curiousconcept.com/#">json formatter</a>, a free online tool that makes it easy to quickly identify the correct keys. Simply copy and paste the entire JSON content into the window and click the “Process” button.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_wN8yRC2stkTnqjrXFjHA.png" /><figcaption>jsonformatter</figcaption></figure><p>We can now integrate this function and modify our code to retrieve all API pages and therefore all FIFA rankings since its creation in 1992. Let’s proceed step by step:</p><ul><li>Import <strong>datetime</strong> module, which will be used in later steps to work with the date format in the page&#39;s source code.</li><li>Change the starting URL to the current men’s ranking page, which will be used to extract the date and ID information needed to scrape all rankings data (Note that we can choose any page because the data we are interested in is the dates and the identification number of the source code).</li></ul><pre>import scrapy<br>import json<br>from datetime import datetime<br><br><br>class MenrankingSpider(scrapy.Spider):<br>    name = &#39;menranking&#39;<br>    allowed_domains = [&#39;fifa.com&#39;]<br>    # changing the starting url by using the one of the ranking page and not the one of the API<br>    start_urls = [&#39;https://www.fifa.com/fifa-world-ranking/men?dateId=id13974&#39;]</pre><ul><li>Add our method for extracting the list of dates and code source IDs to the parse function</li><li>Create a loop with the list of <strong>dates and ID</strong>s by iterating over the API URL.</li><li>Format the date as <strong>‘yyyy-mm-dd’</strong>.</li><li>Send the request with the date as a parameter.</li></ul><pre>def parse(self, response):<br>        script_content = response.xpath(&#39;//script[contains(., &quot;dates&quot;)]/text()&#39;).extract_first()<br>        date_list = json.loads(script_content)<br>        date_list = date_list[&#39;props&#39;][&#39;pageProps&#39;][&#39;pageData&#39;][&#39;ranking&#39;][&#39;dates&#39;]<br>        for item_id in date_list:<br>            url = f&quot;https://www.fifa.com/api/ranking-overview?locale=en&amp;dateId={item_id[&#39;id&#39;]}&quot;<br>            date_text = item_id[&#39;text&#39;].replace(&#39;Sept&#39;, &#39;sep&#39;) # Set the date to the correct format<br>            date_obj = datetime.strptime(date_text, &#39;%d %b %Y&#39;)<br>            date = date_obj.strftime(&#39;%Y-%m-%d&#39;)<br>            yield scrapy.Request(url=url, callback=self.parse_ranking_data, meta={&#39;url&#39;: url,&#39;date&#39;: date})</pre><p>Finally, we create a new function <strong>parse_ranking_data</strong> which is a modified version of the initial <strong>parse </strong>function, with the addition of the date as metadata for each request.”</p><pre>def parse_ranking_data(self, response):<br>        data = json.loads(response.body) <br>        base_url = &#39;https://www.fifa.com&#39;<br>        for ranking_data in data[&#39;rankings&#39;]:<br>            yield {<br>                &#39;date&#39;: response.meta[&#39;date&#39;],<br>                &#39;country&#39;: ranking_data[&#39;rankingItem&#39;][&#39;name&#39;],<br>                &#39;rank&#39;: ranking_data[&#39;rankingItem&#39;][&#39;rank&#39;],<br>                &#39;previousRank&#39; : ranking_data[&#39;rankingItem&#39;][&#39;previousRank&#39;],<br>                &#39;totalPoints&#39;: ranking_data[&#39;rankingItem&#39;][&#39;totalPoints&#39;],<br>                &#39;previousPoints&#39;: ranking_data[&#39;previousPoints&#39;],<br>                &#39;flagUrl&#39;: ranking_data[&#39;rankingItem&#39;][&#39;flag&#39;][&#39;src&#39;],<br>                &#39;countryUrl&#39;: base_url + ranking_data[&#39;rankingItem&#39;][&#39;countryURL&#39;],<br>                &#39;conf&#39;: ranking_data[&#39;tag&#39;][&#39;text&#39;] <br>            }</pre><p>Now, we can use scrapy’s export functions to get the data in a json file.</p><pre>scrapy runspider menranking.py -o data.json</pre><p>or in csv :</p><pre>scrapy runspider menranking.py -o data.csv</pre><p>20 secondes later …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ziJb-h8INZ_Id1GAEtl4yA.png" /></figure><p><strong>In conclusion</strong>, fetching data through a hidden API can be a convenient, efficient, and fast way to extract data. However, it’s important to note that each API is unique and can have its own quirks and drawbacks. As our example of retrieving data from the FIFA ranking shows, it is necessary to adapt to each API, and not to neglect any track in order to make the most of it and obtain the desired data.</p><p>For full code click <a href="https://github.com/hericlibong/HIDDEN_API_FIFA_RANKING_MEN_SCRAPING/tree/main">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7799570b7737" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>