<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ricardo Heredia on Medium]]></title>
        <description><![CDATA[Stories by Ricardo Heredia on Medium]]></description>
        <link>https://medium.com/@ricardoheredia94?source=rss-77acb6cb16d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TR0eTyON1Fj-2jBCrlM9ew@2x.jpeg</url>
            <title>Stories by Ricardo Heredia on Medium</title>
            <link>https://medium.com/@ricardoheredia94?source=rss-77acb6cb16d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:04:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ricardoheredia94/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Tactical DNA of Carabobo FC: A Game Model Analysis⚽️]]></title>
            <link>https://ricardoheredia94.medium.com/the-tactical-dna-of-carabobo-fc-a-game-model-analysis-%EF%B8%8F-a47e7a7da4bc?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a47e7a7da4bc</guid>
            <category><![CDATA[video-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[soccer-analytics]]></category>
            <category><![CDATA[scouting]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 21:47:40 GMT</pubDate>
            <atom:updated>2025-01-26T23:11:21.655Z</atom:updated>
            <content:encoded><![CDATA[<p>As 2024 kicked off, I made a decision that would help me deepen my understanding of football: enrolling in the Scouting 360 course by Objetivo Analista. My motivation was that I wanted to dive deeper into the intricacies of professional scouting and explore how it intersects with data, my primary field of interest.</p><p>One of the first lessons I learned was: regardless of whether you’re an analyst, scout, or data scout, mastering the basic concepts of football is non-negotiable. Thanks to the foundations, we can truly understand both players and the tactical environments they operate in.</p><p>Among these fundamental concepts, the ‘game model’ stands out as particularly key. In this article, I’ll break down this concept through an analysis of Carabobo FC, the first football division Venezuelan team from my hometown that has proven tactical flexibility and high competitiveness in the season 2024.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*olwje0VzQjZjF_RxwcKpkA.png" /><figcaption>Diego Merino (Spain), first Carabobo FC coach to play a final with the team.</figcaption></figure><h4><strong>Game Model Concept</strong></h4><p>The game model itself is defined as a team’s unique functional organization across all phases of play, creating a collective identity. It serves as the map that guides the game, brought to life through the interaction of players’ capabilities.</p><h4>Team Information</h4><p>In order to understand Carabobo FC’s game model, I looked up for a recent match in the Liga FUTVE YouTube channel and picked the Final of the tournament played on December, 8th 2024. The match ended up with Deportivo Táchira winning 2–4 in the penalty shootout.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbqOzFPFbf2g%3Fstart%3D8286%26feature%3Doembed%26start%3D8286&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbqOzFPFbf2g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbqOzFPFbf2g%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d844a0b74bdd42ad3a5d31631b862ddc/href">https://medium.com/media/d844a0b74bdd42ad3a5d31631b862ddc/href</a></iframe><ul><li>The Coach: Diego Merino, Spanish ex-footballer with experience in teams like Rayo Vallecano Juvenil A and Rayo B or CD Toledo.</li><li>Stadium: Misael Delgado, Valencia, Carabobo, Venezuela.</li><li>Most used lineup: 4–3–3</li></ul><h4>Analysis by phases of the game</h4><p><strong>Offensive Phase</strong></p><p>The coach identifies his game model as “chameleonic”, with variants to play out from the back:</p><ul><li><strong>Directly</strong> with long ball from goalkeeper Bruera or from one of the center backs and/or full backs, offering verticality to reach the opponent’s field as quickly as possible and generate dangerous situations.</li><li>Through<strong> indirect/combinational</strong> attack, prioritizing in this phase, the construction from Pernía or Guaramato (the left and right back), who look for immediate support from Pérez and González (the pivots), in charge of the distribution and effective progression of the ball towards the opposite field, with the objective of making the center forward to finish the play.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-LtoOT9LJ1fRXKG3tSFmQ.png" /><figcaption><em>Offensive Phase: González (5) as ball possessor, looks for ball off load with the wingers to make progress to finishing zone.</em></figcaption></figure><p><strong>Attack/Defense Transition</strong></p><p>They execute a quick coordinated retreat, if any of the fullbacks moves up to the opponent’s half and loses the ball at that height, the winger on the same side quickly returns to cover the free space left by the fullback, preventing the opponent from developing play through that external lane and maintaining the block’s balance.</p><p>Similarly, even when not in covering situations, the wingers integrate into the block to form a 4–5–1 or 4–1–4–1 formation, aiming to effectively defend both the external lanes through the wingers’ commitment, and the central lane through the positioning of the defensive midfielder and pivots. The objective in this phase is to quickly recover the ball and break out at speed using the wingers or a long pass to the center forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFg-2lya1hKZQq_TiMe_Fg.png" /><figcaption>Attack/Defense Transition: This situation occurs after losing the ball in midfield, the wingers Tortolero and Hernández guards the outer lane of the block, and together with the midfielders, shorten the space between the defensive line and the midfield line.</figcaption></figure><p><strong>Defensive Phase</strong></p><p>They place a medium-high block, depending on the height at which the opponent is playing, with up to 5 or 6 players pressing the opponent’s ball and possible receivers. The block pressure is characterized by marking the possible receivers and the ball holder.</p><p>The central defenders are positioned behind the half-court line to prevent the opponent’s counterattack. In situations where the opponent intends to play out from the goalkeeper with the central defenders, the left winger Hernandez and the center forward Balza on the border area to block possible passing lines.</p><p>In defense, the team can switch to 4–4–2 to carry out high block pressure in a coordinated manner, with the clear intention of recovering the ball in the opponent’s half, making the opponent’s play uncomfortable. This forces the opponent to play long balls so that, thanks to the aerial dominance of the Carabobo central defenders Aponte and Lujano, win those duels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_53_qMP61fUQUbkPziJuew.png" /><figcaption>Left: High pressure by Hernandez and Balza to prevent Tachira’s goalkeeper from playing with center backs or full backs. Right: Six Carabobo players are pressing in a high block to cut off the start of the opponent’s initial phase of play.</figcaption></figure><p><strong>Defensive/Attacking Transition</strong></p><p>When they recover the ball near their own area, Carabobo starts mainly with a long ball from Bruera, seeking to overcome the rival midfield line. The objective is to generate aerial duels, taking advantage of the physical abilities of Balza or Hernández to dominate aerial duels and win the second play.</p><p>Depending on the ball recovery zone on the field, the central defenders (Aponte/Lujano) or full backs (Guaramato/Pernía) initiate the game by connecting with the pivots González and Pérez, the main organizers of the game. Jimmy Congo, even though his main role is defensive, also participates in the starting line with short and simple passes to unmarked full bakcs or wingers.</p><p>As an additional variant, the Granate team executes quick counterattacks where the number 9 goes back to receive and connect with the wingers, who are in charge of progressing the game through the flanks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPxRyxJiqSZeokI1Yaz3sw.png" /><figcaption>Left: After the center backs recover the ball, they play short with Jimmy Congo who looks for support from right back Guaramato. Right: To accelerate the counterattack, Balza moves back from his position of 9 to receive the ball in the midfield and unload with left winger Hernández.</figcaption></figure><h3><strong>Conclusions on the Carabobo FC game model</strong></h3><p><strong>Main tactical model</strong></p><ul><li>Main system 4–3–3 with variants to 4–1–4–1 and 4–4–2</li><li>Chameleonic” approach that adapts according to the context of the game and the opponent</li><li>Maintains a balance between possession and direct play</li></ul><p><strong>Key roles</strong></p><ul><li>Pivots as organizers of the game</li><li>Wingers with responsibility in both attack and defense</li><li>Forward as a reference for direct play</li></ul><p><strong>Identity signs</strong></p><ul><li>Solid defensive organization</li><li>Effective transitions in both directions</li><li>Tactical adaptability while maintaining basic principles</li><li>Balance between possession and verticality</li></ul><p><strong>Fundamental Principles</strong></p><p><strong>Defense:</strong> fundamental pillar of the team, establishes a medium-high block with zonal defense through coordinated pressure and organized coverages. Defensive participation of wingers.</p><p><strong>Attack:</strong></p><ul><li>Combination play: they build the game from the center and wingers with the support of the inside pivots, in charge of the creation and development of the team’s game.</li><li>Direct play: goalkeeper or central defenders make long passes to the center-forward overcoming opponents lines.</li><li>Fast transitions through flanks taking advantage of the speed and conduction of the wingers.</li></ul><h3>Key players profiles</h3><p>I’ll mention whom I consider most important players in the Carabobo game model:</p><ol><li><strong>Lucas Bruera — Goalkeeper:</strong></li></ol><ul><li>A goalkeeper that dominates the aerial game, showing security in his starts and an excellent anticipation capacity.</li><li>His positioning under the posts and reading of the game allow him to maintain a high percentage of effectiveness in his interventions, being fundamental in key moments (best player of the Granate team in the match, he saved a penalty in the shootout and made key savings during the 90 minutes).</li><li>Tactical aspect: key player in the construction from the back, with the ability to distribute both short and long, adapting to the needs of the team.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*f1_tU7Onh43Xkd_6BJZ3iA.png" /><figcaption>Data from Sofascore.</figcaption></figure><p>2. <strong>Juan Camilo Pérez — Interior Midfielder:</strong></p><ul><li>A more attacking left pivot, with the ability to give the last pass to finish the play, he complements González well in the creation of play, being him more of a finisher.</li><li>In his heat map we can see the great physical display he enjoys, even reaching 3/4 of the pitch most of the time but without neglecting his defensive work to maintain the balance in the block during the withdrawal in attack/defense transitions and defensive phase in general.</li><li>In this game he had a considerably high passing success rate and a large number of touches, demonstrating that from his feet is generated much of the danger of his team.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*LbvCHJYCb2XWrBJIC3XeIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*gxi41cMY-5gNdtdxVRd9uQ.png" /><figcaption>Data from Sofascore.</figcaption></figure><p>3. <strong>Edson Tortolero — Right Winger</strong></p><ul><li>He is a left winger who plays on the right flank, with great ability to drive the ball inside, taking advantage of his ball handling either by finishing outside the area or filtering balls into the small area for the center-forward.</li><li>He stands out as one of, if not the most dangerous of Carabobo’s attackers (he is ranked 1st on his team by Sofascore in the last season).</li><li>He is both a receiver and a passer of the ball, being key in the development and finishing phases of the game.</li><li>Although his role is to attack, his defensive commitment stands out, both to cover the fullback during attack/defense transitions, as well as to be part of the block of 4 in the midfield, protecting the outer flank.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*8Ajr4x1aalyjDUE1vBtMBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*q1maT1kLhs_rhIno8Zh5VA.png" /><figcaption>Data from Sofascore.</figcaption></figure><p>4. <strong>Carlos Lujano — Center Back</strong></p><ul><li>Captain of the team, a strong central defender who dominates various registers and is an important part of the defensive backline.</li><li>He stands out for his versatility both in the air and with his feet, especially in passing accuracy and ability to play long balls, which is important for one of the variants of Carabobo’s style of play.</li><li>Between him and Aponte, they form a tandem to give the team an efficient ball delivery, also adding solidity and organization to the defensive line.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*lwqFUNF1UaAfFd_gDJLqUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*cdda6gva_MTfncHPrr_o5Q.png" /><figcaption>Data from Sofascore.</figcaption></figure><p>Under Diego Merino, Carabobo FC has developed a game model that shows how a team can be both organized and flexible. Their ‘chameleonic’ style is their identity, we can see it in action through how smoothly they change formations and adapt their play. The team knows how to defend as a solid unit while still being dangerous in attack, with players like Tortolero or Pérez showing they can handle multiple roles effectively.</p><p>Making it to the 2024 final shows that this approach work, that you can stay true to your playing style while being able to adapt when needed. This analysis has helped me understand how a clear game model shapes everything that happens on the field.</p><p>Thanks for reading till the end, hope you’ve enjoyed this report and have gained valuable insights about one of the best teams in the Venezuela first division.</p><p>If you’re interested in looking at more of my content about data science/ML applied to scouting and football analytics you can follow me through:</p><p>X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass">x.com</a></p><p>Bluesky: <a href="https://bsky.app/profile/thenetpass.bsky.social">https://bsky.app/profile/thenetpass.bsky.social</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a47e7a7da4bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data-Driven Scouting Report: Shortlisting 5 Midfielders and 5 Forwards from EFL Championship ⚽️]]></title>
            <link>https://ricardoheredia94.medium.com/data-driven-scouting-report-shortlisting-5-midfielders-and-5-forwards-from-efl-championship-%EF%B8%8F-971fca3e2256?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/971fca3e2256</guid>
            <category><![CDATA[recruitment-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scouting-reports]]></category>
            <category><![CDATA[soccer-analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Sun, 15 Dec 2024 00:48:41 GMT</pubDate>
            <atom:updated>2024-12-15T20:20:40.538Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*cBgNIoZRAcUMjzsy4JJf1g.jpeg" /><figcaption>Championship logo</figcaption></figure><p>As part of the series of articles I’ve been writing about lately, which are all part of a data-driven scouting project to shortlist midfielders and forwards as hypothetical signings for Sheffield United for the upcoming winter transfer window. In this one, I’ll describe the results derived from the K-Means model I’ve trained with the EFL Championship data collected until match day 19.</p><p>For the final results, I decided to use the Championship data to scout the players, since I’ve had issues scraping some variables from League One and League Two, will try fixing them in the future. Thus, the players radar charts are all from players currently playing in the Championship.</p><h3><strong>The Selected Clusters</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*y0Jd6f_341tGR9Hz8E6QCA@2x.jpeg" /><figcaption>Resultant clusters from the K-Means model using Champion ship data.</figcaption></figure><p>I followed the same process used for clustering the players from the three leagues:</p><p><a href="https://ricardoheredia94.medium.com/using-unsupervised-modeling-for-finding-top-midfielders-and-forwards-in-championship-league-one-6d92662837fe">Using Unsupervised Modeling for Finding Top Midfielders and Forwards in Championship, League One…</a></p><p>The results were similar:</p><ul><li><strong>Cluster 0</strong>: represents versatile, “box-to-box” midfielders that fit the mixed profile I aimed for. These players have good registers in metrics like: progressive passes, progressive carries, middle third touches, tackles, interceptions or recoveries.</li><li><strong>Cluster 2</strong>: highlights forwards or more creative midfielders suited for offensive roles. The prominent metrics of these players are: goals, goals-creating actions, attacking third touches, progressive carries, take-ons, crosses, to name a few.</li></ul><h3>Shortlisted Players</h3><p>After obtaining the clusters, I needed to visualize the performance of the players somehow, so I used the radar charts inspired from the mplsoccer documentation:</p><p><a href="https://mplsoccer.readthedocs.io/en/latest/gallery/radar/plot_radar.html">Radar Charts - mplsoccer 1.4.0 documentation</a></p><blockquote>mplsoccer is a Python library for plotting soccer/football charts in Matplotlib, it has many options for visualizing different situations.</blockquote><p>All the metrics are calculated as per 90 and then ranked by percentiles to make a comparison between players and their peers.</p><h3><strong>Top 5 Shortlisted Midfielders</strong></h3><p><strong>Alex Mowatt</strong></p><ul><li><strong>Age:</strong> 29 years old</li><li><strong>Team:</strong> West Bromwich Albion</li><li><strong>Country:</strong> England (ENG)</li><li><strong>Minutes Played: </strong>833</li><li><strong>Market Value:</strong> €3,500,000</li><li><strong>Key Performance Characteristics: </strong>well-rounded central midfielder with particular strengths in midfield control and ball progression. His high involvement in the middle third, combined with above-average goal-scoring ability and strong ball recovery skills, suggests he excels as a box-to-box midfielder. He has balanced skill set and consistent performance across multiple metrics.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FbI6DVWMFvQGhE9FBIzH8w.png" /></figure><p><strong>Jayson Molumby</strong></p><ul><li><strong>Age:</strong> 25 years old</li><li><strong>Team:</strong> West Bromwich Albion</li><li><strong>Country:</strong> Ireland (IRL)</li><li><strong>Minutes Played: </strong>495</li><li><strong>Market Value:</strong> €3,000,000</li><li><strong>Key Performance Characteristics: </strong>dynamic box-to-box midfielder strengths in middle third touches, pass completion, progressive carries and take-ons. In defensive dutties he shows prowess regarding recoveries mid third tackles.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7NN4Ho5NpqxuC6DCeAbTw.jpeg" /></figure><p><strong>Joe Williams</strong></p><ul><li><strong>Age:</strong> 28 years old</li><li><strong>Team:</strong> Bristol City</li><li><strong>Country:</strong> England (ENG)</li><li><strong>Minutes Played:</strong> 819</li><li><strong>Market Value:</strong> €1,200,000</li><li><strong>Key Performance Characteristics</strong>: energetic box-to-box midfielder with creative tendencies, evident with progressive passes, completed passes in penalty area, involvement in midfield transitions with midfield third touches and solid defensive contributions with mid third tackles and recovery rates.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AoIRlJx4i1yb_VkstnqCzQ.jpeg" /></figure><p><strong>Ben Osborn</strong></p><ul><li><strong>Age:</strong> 30 years old</li><li><strong>Team:</strong> Derby County</li><li><strong>Country:</strong> England (ENG)</li><li><strong>Minutes Played: </strong>631</li><li><strong>Market Value:</strong> €900,000</li><li><strong>Key Performance Characteristics: </strong>proficient midfielder with strong creative tendencies, evidencing good progressive passes, crossing ability, mid third touches and tackles. He is involved in goal-creating actions and shot-creating actions, demonstrating attacking influence in the midfield.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hv2Q0lFEFG1jN_r3H-lODA.jpeg" /></figure><p><strong>Lucas Andersen</strong></p><ul><li><strong>Age:</strong> 30 years old</li><li><strong>Team:</strong> Queens Park Rangers (QPR)</li><li><strong>Country:</strong> Denmark (DEN)</li><li><strong>Minutes Played: </strong>557</li><li><strong>Market Value:</strong> €500,000</li><li><strong>Key Performance Characteristics: </strong>a player with more attacking midfielder abilities showing strong creative tendencies, evidenced by his ability to manage the ball, through high rates of pass completion %, PPA passes, attacking third touches and progressive receiving, along with goal-shot creation chances. The defensive registers are lower but still shows influence in defensive tasks such as interceptions and middle third tackles.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_PIaQYYdJw8iyBfylJdJUQ.jpeg" /></figure><h3>Top 5 Shortlisted Forwards</h3><p><strong>Ben Doak</strong></p><ul><li><strong>Age:</strong> 19 years old</li><li><strong>Team:</strong> Middlesbrough</li><li><strong>Country:</strong> Scotland (SCO)</li><li><strong>Minutes Played: </strong>738</li><li><strong>Market Value:</strong> €10,000,000</li><li><strong>Key Performance Characteristics: </strong>as one of the highest-valued young talents in the league explosive attacking winger, his metrics suggest a he’s a dynamic attacking player who excels in creating and finishing chances, with progressive carries, assists, GCA and mid-high rate on successful take-ons with a potential 1v1 ability. Despite not being a classic high goal-scorer he contributes in an important way to creating the chances.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KmBEyxF-3DUifaOoZiCjdA.jpeg" /></figure><p><strong>Romain Esse</strong></p><ul><li><strong>Age:</strong> 19 years old</li><li><strong>Team:</strong> Milwall</li><li><strong>Country:</strong> England (ENG)</li><li><strong>Minutes Played: </strong>528</li><li><strong>Market Value:</strong> €3,000,000</li><li><strong>Key Performance Characteristics: </strong>attacking player strong in attacking and defensive phases. His standout metrics include being in the 95th percentile for take-ons, 91st percentile for both middle third tackles and recoveries, 88th percentile for goals per 90, shots on target % and . This shows involvement in both phase, showing he represents a promising young talent with room for improvement in other metrics.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEvqm7qWLpM-b4h_ickpnQ.jpeg" /></figure><p><strong>Bachir Belloumi</strong></p><ul><li><strong>Age:</strong> 22 years old</li><li><strong>Team:</strong> Hull City</li><li><strong>Country:</strong> Algeria (ALG)</li><li><strong>Minutes Played: </strong>523</li><li><strong>Market Value:</strong> €5,000,000</li><li><strong>Key Performance Characteristics: </strong>player with creative and playmaking abilities, appears as a high performer in passes into penalty area (PPA passes), shots on target %, crosses and aerial duel won % probably skilled in second-ball winnings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUyXM_ArYSy1vSCHgCf7cA.jpeg" /></figure><p><strong>Ronald</strong></p><ul><li><strong>Age:</strong> 23 years old</li><li><strong>Team:</strong> Swansea City</li><li><strong>Country:</strong> Brazil (BRA)</li><li><strong>Minutes Played:</strong> 605</li><li><strong>Market Value:</strong> €1,800,000</li><li><strong>Key Performance Characteristics: </strong>player that<strong> </strong>profiles as a creative and dynamic forward with strong progressive abilities, high rates of attacking-third touches and both progressive receives and carries. More involved in assists, taking successful take-ones and attacking the middle third than goal scoring. Excels in creating opportunities and advancing the ball, making him a possible valuable asset in offensive transitions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wx6D_eUIKMYsvzASA9Epng.jpeg" /></figure><p><strong>Nathaniel Mendez-Laing</strong></p><ul><li><strong>Age:</strong> 32 years old</li><li><strong>Team:</strong> Derby County</li><li><strong>Country:</strong> Guatemala (GUA)</li><li><strong>Minutes Played:</strong> 443</li><li><strong>Market Value:</strong> €400,000</li><li><strong>Key Performance Characteristics: </strong>veteran player with highly creative and dynamic abilities. Though, he’s not topped as one of the highest goal scorers, he excels in other areas of attacking such as: pass completion, crossings, assists and shot-creating actions. Additionally, he shows presence in middle and attacking-third touches but also setting passes into the penalty area. High recoveries and aerial duels won % rates also deserves a note.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KdKwBf8B_XBxbZzUhqX2A.jpeg" /></figure><p>After fulfilling the main task of selecting the players for hypothetical signings, is important to note that this methodology is just a complement to video scouting, where deeper viewing and assessment is needed to check what the data says about a player, making it a well-rounded process that gathers all of the pieces to make importante signing decisions for a squad.</p><p>Shortlisting players this way helps to narrow the scope of “who and where to look” helping recruitment departments and sporting directors make informed decisions backed by data.</p><p>There are a lot more metrics that can be included in such analyses, even eliminating variables that could be noisy for the analysis, but the idea of the whole project was to show how to display a methodology for:</p><ul><li>Obtaining data from <strong>free resources</strong> by your own, setting an <strong>ETL</strong> custom proccess leveraging <strong>Python</strong> programming best practices to automate this process.</li><li><strong>Cleaning and transforming </strong>the merged data in order to organize a well-structured dataset for <strong>feature engineering.</strong></li><li>Using <strong>unsupervised machine learning</strong> to help group players with similar characteristics.</li><li>Create insightful <strong>visualizations</strong> using aN out-of-the-box frontend framework like<strong> Streamlit</strong>, to show the information in a concise and interactive way to interested stakeholders like other recruitment experts peers and sporting directors.</li><li>Improve and back decision making processes involving player recruitment with a <strong>data-driven approach</strong>.</li></ul><p>I’m glad if you read all the way till here, thanks for taking your time to do so. More improvements will come for this project, so hope you stay tuned and can get insightful ideas for your own recruitment analyses.</p><p>If you’re interested in looking at more of my content about data science/ML applied to scouting and football analytics you can follow me through:</p><p>X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass">x.com</a></p><p>Bluesky: <a href="https://bsky.app/profile/thenetpass.bsky.social">https://bsky.app/profile/thenetpass.bsky.social</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><p>And if you have any comments, suggestions or want to do any exchange don’t hesitate on talking to me!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=971fca3e2256" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Unsupervised Modeling for Finding Top Midfielders and Forwards in Championship, League One…]]></title>
            <link>https://ricardoheredia94.medium.com/using-unsupervised-modeling-for-finding-top-midfielders-and-forwards-in-championship-league-one-6d92662837fe?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d92662837fe</guid>
            <category><![CDATA[football-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scouting]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Sat, 07 Dec 2024 20:40:25 GMT</pubDate>
            <atom:updated>2024-12-07T20:40:25.885Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Unsupervised Modeling for Finding Top Midfielders and Forwards in Championship, League One and League Two 🏴󠁧󠁢󠁥󠁮󠁧󠁿⚽️📊</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*cd6RvsWL28Oj2ArstaHA5g.png" /><figcaption>K-Means Clustering Scatter Plot example.</figcaption></figure><h3>What’s Unsupervised Modeling?</h3><p>When we talk about unsupervised modeling we refer to a type of machine learning where the algorithm works with data that has no predefined labels or target outputs. The goal is to identify patterns, structures, or relationships within the data itself. Unlike supervised learning, which uses labeled data to train a model, unsupervised modeling focuses on uncovering hidden insights, such as grouping similar data points (clustering) or reducing dimensionality (PCA).</p><p>It’s commonly used for tasks like customer segmentation, anomaly detection, and data exploration when you don’t have clear labels but want to make sense of the data’s underlying structure.</p><p>This time, I’m using the K-Means model for clustering, if you use Python you can use it out-of-the-box from the <a href="https://scikit-learn.org/1.5/modules/generated/sklearn.cluster.KMeans.html">sklearn</a> library, which is super helpful for machine learning purposes.</p><h3>Choosing the Metrics and Why I Chose these</h3><p>In order to feed the model with data, I selected a set of metrics from midfielders and forwards from England’s tier 2, 3 and 4. I wanted to find midfielders with a mixed profile, who could defend well by tackling, intercepting or clearing the ball and attacking decently, doing passes (short, long, crosses) and carrying the ball forward progressively by generating danger; what in football terms would be called a “box-to-box” like Fede Valverde from Real Madrid. On forwards I wanted to find a profile more like Haaland and also tactically good with the ball, so I selected metrics like goals, shots, passes, touches on middle and final 3rd, aerial duels won %, relevant metrics for I wanted to look at.</p><p>Back when I watched the games of Sheffield United to get acquainted with the style of play, their tactical disposal and the positions I personally thought they would need to improve, I got an idea of what I wanted to scout, so this is why I focused on midfielders and forwards and why these metrics where the ones selected from FBRef.</p><h3>Modeling Procedure</h3><p>The goal of using this model is to group these players, generate labels and focus on the specific clusters where the top performer players are. After that, I wanted to filter them and make the shortlist of my 5 midfielders and 5 forwards for a potential and hypothetical signing for the club.</p><p>The process of the workflow goes like this: <strong>l</strong>oad, scale, evaluate, cluster, and visualize the data for meaningful insights.</p><p>I’ve set two Python files that do the work:</p><ul><li><strong>unsupervised_modeling.py</strong>: it stores reusable functions like data loading, scaling, evaluating K-Means, and visualizing clusters.</li></ul><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import joblib<br>from sklearn.cluster import KMeans<br>from sklearn.metrics import silhouette_score, calinski_harabasz_score, davies_bouldin_score<br>from sklearn.decomposition import PCA<br>from sklearn.manifold import TSNE<br>import matplotlib.cm as cm<br>from sklearn.preprocessing import StandardScaler<br><br># Function to load data from pickle or CSV files<br>def load_data(ruta_cat: str = None, ruta_num: str = None, ruta_df: str = None) -&gt; tuple:<br>    &quot;&quot;&quot;<br>    Load files in pickle or CSV format for categorical, numerical, or general data.<br>    <br>    Parameters:<br>    - ruta_cat (str, optional): Path to the categorical file in pickle or CSV format.<br>    - ruta_num (str, optional): Path to the numerical file in pickle or CSV format.<br>    - ruta_df (str, optional): Path to a general file in pickle or CSV format.<br>    <br>    Returns:<br>    tuple: Loaded DataFrames. Returns None if no path is specified.<br>    &quot;&quot;&quot;<br>    cat, num, df_general = None, None, None<br><br>    # Load categorical data if path is provided<br>    if ruta_cat is not None:<br>        ruta_cat = str(ruta_cat)  # Convert path to string<br>        if ruta_cat.endswith(&#39;.pkl&#39;):<br>            cat = pd.read_pickle(ruta_cat)<br>        elif ruta_cat.endswith(&#39;.csv&#39;):<br>            cat = pd.read_csv(ruta_cat)<br>        else:<br>            print(f&quot;Unsupported file format for &#39;ruta_cat&#39;: {ruta_cat}&quot;)<br><br>    # Load numerical data if path is provided<br>    if ruta_num is not None:<br>        ruta_num = str(ruta_num)  # Convert path to string<br>        if ruta_num.endswith(&#39;.pkl&#39;):<br>            num = pd.read_pickle(ruta_num)<br>        elif ruta_num.endswith(&#39;.csv&#39;):<br>            num = pd.read_csv(ruta_num)<br>        else:<br>            print(f&quot;Unsupported file format for &#39;ruta_num&#39;: {ruta_num}&quot;)<br><br>    # Load general data if path is provided<br>    if ruta_df is not None:<br>        ruta_df = str(ruta_df)  # Convert path to string<br>        if ruta_df.endswith(&#39;.pkl&#39;):<br>            df_general = pd.read_pickle(ruta_df)<br>        elif ruta_df.endswith(&#39;.csv&#39;):<br>            df_general = pd.read_csv(ruta_df)<br>        else:<br>            print(f&quot;Unsupported file format for &#39;ruta_df&#39;: {ruta_df}&quot;)<br>    <br>    return cat, num, df_general<br><br># Function to scale variables using StandardScaler<br>def scale_variables(df, variables_to_scale):<br>    scaler = StandardScaler()<br>    df_scaled = pd.DataFrame(<br>        scaler.fit_transform(df[variables_to_scale]), <br>        columns=variables_to_scale<br>    )<br>    return df_scaled, scaler<br><br># Function to evaluate K-Means and obtain metrics<br>def evaluate_kmeans(df, solutions, random_state=42, n_init=10):<br>    &quot;&quot;&quot;<br>    Evaluates multiple KMeans models with different numbers of clusters and returns metrics for each solution.<br><br>    Parameters:<br>    - df (pd.DataFrame): DataFrame with the data to use.<br>    - solutions (list): List of cluster numbers to try.<br>    - random_state (int): Seed for reproducibility.<br>    - n_init (int): Number of initializations.<br><br>    Returns:<br>    - pd.DataFrame: DataFrame with calculated metrics for each number of clusters.<br>    &quot;&quot;&quot;<br>    elbow, silhouette, calins, davies = [], [], [], [] <br><br>    for solution in solutions:<br>        cluster = KMeans(random_state=random_state, n_clusters=solution, n_init=n_init)<br>        cluster.fit(df)<br>        elbow.append(cluster.inertia_)<br>        silhouette.append(silhouette_score(df, cluster.labels_))<br>        calins.append(calinski_harabasz_score(df, cluster.labels_))<br>        davies.append(davies_bouldin_score(df, cluster.labels_))<br>    <br>    metrics = pd.DataFrame({<br>        &#39;Solution&#39;: solutions,<br>        &#39;Elbow&#39;: elbow,<br>        &#39;Silhouette&#39;: silhouette,<br>        &#39;Calinski-Harabasz&#39;: calins,<br>        &#39;Davies-Bouldin&#39;: davies<br>    }).set_index(&#39;Solution&#39;)<br>    <br>    # Plot metrics<br>    metrics.plot(subplots=True, figsize=(12, 8), layout=(2, 2), sharex=False)<br>    plt.show()<br><br>    return metrics<br><br># Function to train and save KMeans<br>def train_save_kmeans(df, n_clusters, save_path, random_state=42, n_init=10):<br>    &quot;&quot;&quot;<br>    Trains a KMeans model with the specified number of clusters and saves the model to the provided path.<br><br>    Parameters:<br>    - df (pd.DataFrame): DataFrame with the data to use.<br>    - n_clusters (int): Number of clusters for the KMeans model.<br>    - save_path (str): Path where the trained model will be saved.<br>    - random_state (int): Seed for reproducibility.<br>    - n_init (int): Number of initializations.<br><br>    Returns:<br>    - KMeans: Trained KMeans model.<br>    &quot;&quot;&quot;<br>    # Train the model with the specified number of clusters<br>    cluster_model = KMeans(n_clusters=n_clusters, n_init=n_init, random_state=random_state)<br>    cluster_model.fit(df)<br><br>    # Save the model to the specified path<br>    joblib.dump(cluster_model, save_path)<br>    print(f&#39;KMeans model with {n_clusters} clusters saved to: {save_path}&#39;)<br><br>    return cluster_model<br><br># Function to assign clusters to DataFrame<br>def assign_clusters(df, cluster):<br>    df[&#39;cluster&#39;] = cluster.predict(df)<br>    return df<br><br># Function to save the DataFrame with assigned clusters<br>def save_dataframe(df, save_path):<br>    df.to_csv(save_path, index=False)</pre><ul><li><strong>kmeans_pipeline.py</strong>: the main execution file that ties everything together.</li></ul><pre>import sys<br>import logging<br>from pathlib import Path<br>import pandas as pd<br>from unsupervised_modeling import scale_variables, evaluate_kmeans, train_save_kmeans, assign_clusters<br>from paths import TRANSFORMED_DATA_DIR, MODELS_DIR<br>from constants import VARIABLES_TO_SCALE<br><br># Configure logging<br>logging.basicConfig(<br>    level=logging.INFO,<br>    format=&#39;%(asctime)s - %(levelname)s - %(message)s&#39;,<br>    handlers=[<br>        logging.FileHandler(&quot;kmeans_pipeline.log&quot;),<br>        logging.StreamHandler()<br>    ]<br>)<br><br># Main function to execute the K-Means pipeline<br>def main():<br>    # Define paths<br>    input_path = TRANSFORMED_DATA_DIR / &#39;df_training.csv&#39;<br>    model_save_path = MODELS_DIR / &#39;kmeans_model_3_clusters_updated.pkl&#39;<br>    labeled_results_path = TRANSFORMED_DATA_DIR / &#39;df_training_scaled_labeled.csv&#39;<br>    unscaled_results_path = TRANSFORMED_DATA_DIR / &#39;df_training_unscaled_labeled.csv&#39;<br><br>    try:<br>        # Step 1: Load dataset<br>        logging.info(&quot;Loading dataset...&quot;)<br>        df_training = pd.read_csv(input_path)<br>        logging.info(f&quot;Dataset loaded successfully with shape: {df_training.shape}&quot;)<br><br>        # Step 2: Prepare data for K-Means<br>        logging.info(&quot;Preparing data for K-Means...&quot;)<br>        df_training_kmeans = df_training[VARIABLES_TO_SCALE]<br>        df_training_scaled, scaler = scale_variables(df_training_kmeans, VARIABLES_TO_SCALE)<br>        logging.info(&quot;Data scaled successfully.&quot;)<br><br>        # Step 3: Evaluate potential solutions<br>        logging.info(&quot;Evaluating K-Means solutions...&quot;)<br>        solutions = list(range(3, 7))<br>        metrics = evaluate_kmeans(df_training_scaled, solutions)<br>        logging.info(f&quot;K-Means evaluation metrics:\n{metrics}&quot;)<br><br>        # Step 4: Train and save the K-Means model<br>        n_clusters = 3<br>        logging.info(f&quot;Training K-Means with {n_clusters} clusters...&quot;)<br>        kmeans_model = train_save_kmeans(df_training_scaled, n_clusters, model_save_path)<br>        logging.info(f&quot;K-Means model saved to {model_save_path}&quot;)<br><br>        # Step 5: Assign clusters to the DataFrame<br>        logging.info(&quot;Assigning clusters to data...&quot;)<br>        df_with_clusters = assign_clusters(df_training_scaled, kmeans_model)<br><br>        # Save the labeled scaled dataset<br>        logging.info(f&quot;Saving scaled dataset with clusters to {labeled_results_path}...&quot;)<br>        df_with_clusters.to_csv(labeled_results_path, index=False)<br><br>        # Step 6: Add clusters to the original dataset<br>        logging.info(&quot;Adding clusters to the original dataset...&quot;)<br>        df_training[&#39;cluster&#39;] = df_with_clusters[&#39;cluster&#39;]<br><br>        # Save the labeled original dataset<br>        logging.info(f&quot;Saving original dataset with clusters to {unscaled_results_path}...&quot;)<br>        df_training.to_csv(unscaled_results_path, index=False)<br><br>        logging.info(&quot;K-Means pipeline completed successfully!&quot;)<br><br>    except Exception as e:<br>        logging.error(f&quot;An error occurred: {e}&quot;, exc_info=True)<br><br># Entry point for the script<br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p><strong>Step by step</strong></p><p>Let’s break down this whole bunch of code that could definitely be boring at a first glance but is crucial to have an understanding of the process. An important part the execution is the ‘logging’ a module that allows fine-grained control of the code. I found it useful to log any possible error when training the model and executing its steps.</p><ol><li>Loading the data: the training data that resulted from the merge of the FBRef and Transfermarkt data is loaded and organized for scaling before training.</li><li>Scaling data: normalize numerical features using StandardScaler. Clustering algorithms like K-Means work better on scaled data since it removes biases from varying feature magnitudes.</li><li>Evaluate K-Means solutions:</li></ol><p><strong>How to choose the number of clusters (k)</strong></p><p>Choosing the optimal number of clusters for K-Means is challenging because it must be specified before running the algorithm. Since the optimal value of k is not known in advance, the approach is to test several plausible values and evaluate the results using different metrics.</p><p><strong>a) Elbow Method</strong>:</p><ul><li>Measures the sum of squared errors (inertia) within clusters for each kk.</li><li>Look for the “elbow point” where the rate of error decrease slows, indicating the best kk.</li></ul><p><strong>b) Silhouette Method</strong>:</p><ul><li>Compares the average distance of a record to points in its own cluster with the average distance to points in the nearest cluster.</li><li>Score range: −1 to 1 -&gt; Near 1: Clusters are well-separated. Around 0: Some clusters overlap. Negative: Misassigned points or outliers are present. Implemented in sklearn.metrics.silhouette_score.</li></ul><p><strong>c) Calinski-Harabasz Index</strong>:</p><ul><li>Ratio of inter-cluster dispersion to intra-cluster dispersion.</li><li>Higher values indicate better clustering.</li><li>Implemented in sklearn.metrics.calinski_harabasz_score.</li></ul><p><strong>d) Davies-Bouldin Index</strong>:</p><ul><li>Measures the average similarity between each cluster and its nearest cluster. Implemented in sklearn.metrics.davies_bouldin_score.</li><li>Score range: 0 to infinity:</li><li>Closer to 0: Better clustering.</li></ul><p>I explicitly indicate that I want it to use multiple cluster solutions between a range of options (K=3, 4, 5, 6) in order to find the optimal number of clusters that best represents my data. This function (evaluate_kmeans()) computes the metrics named before.</p><p>After that I plot these metrics for easier interpretation and to visually choose the appropriate cluster point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*5zNUN0_qRxWWi2KiYaQjRQ.png" /><figcaption>Metrics subplots to choose the cluster solution.</figcaption></figure><p>4. Train and save the K-Means model: save the trained model for reuse using joblib.</p><p>5. Assign Clusters: assign the cluster labels to each data point (players) in the dataset and then analyze them by identifying the groups.</p><p>6. Plot a line plot visualization: the idea is to have a big picture of how the players are grouped based on the defined metrics, with the goal of identifying the midfielders and forwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__TRxwe_Dm5fjICSurxSIg.png" /><figcaption>Players line plots.</figcaption></figure><h3>Final Results</h3><p>In the end, the model identified two critical clusters:</p><ul><li><strong>Cluster 0</strong>: represents versatile, “box-to-box” midfielders that fit the mixed profile I aimed for.</li><li><strong>Cluster 2</strong>: highlights forwards or more creative midfielders suited for offensive roles.</li></ul><p>This workflow demonstrates how I use clustering to provide a data-driven analysis that can inform scouting decisions, narrowing down the search for potential signings and bringing player evaluations based on different metrics.</p><p>In the upcoming article, I will show how to visualize these players performance metrics through radar charts.</p><p>For now, thanks for reading and staying till the end, hope you’ve gain valuable knowledge.</p><p>You can follow my content through:</p><p>X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass">x.com</a></p><p>Bluesky: <a href="https://bsky.app/profile/thenetpass.bsky.social">https://bsky.app/profile/thenetpass.bsky.social</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d92662837fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cleaning and Transforming Data for Player Recruitment: My Personal Approach to Data Cleaning.]]></title>
            <link>https://ricardoheredia94.medium.com/cleaning-and-transforming-data-for-player-recruitment-my-personal-approach-to-data-cleaning-a91e50b30d34?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a91e50b30d34</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[footballscouting]]></category>
            <category><![CDATA[football-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Fri, 06 Dec 2024 12:18:17 GMT</pubDate>
            <atom:updated>2024-12-06T12:18:17.608Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m1mO9VAOO85SOBEU" /><figcaption>Photo by <a href="https://unsplash.com/@hellorich?utm_source=medium&amp;utm_medium=referral">Richard Boyle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Nowadays, data has become an essential tool for identifying talent and improving team performance. It serves as a complement to video scouting, which, even in this era of AI and technology, remains the primary method for gathering critical player insights. However, data adds value by helping you answer “where and who to look at,” narrowing the scope of analysis. The real challenge lies in collecting it properly and unlocking its true potential: cleaning and transforming it for meaningful analysis. In this article, I’ll share my personal approach to cleaning and preparing football data from FBRef and Transfermarkt as part of my scouting project.</p><p>You can read the process I went through in order to obtain this data in my previous articles:</p><ul><li><a href="https://ricardoheredia94.medium.com/scraping-transfermarkt-with-python-data-driven-player-recruitment-using-englands-championship-8ad18b9103fe">Scraping Transfermarkt with Python: Data-Driven Player Recruitment using England’s Championship…</a></li><li><a href="https://ricardoheredia94.medium.com/scraping-fbref-for-data-driven-scouting-and-enhanced-player-profiling-464acad83270">Scraping FBRef for Data-Driven Scouting and Enhanced Player Profiling</a></li></ul><p>The goal was turning raw data from Championship, League One and League Two into a structured dataset ready to use for clustering.</p><p>For data cleaning purposes it is well known that Pandas is a great library to help with cleansing tasks (<a href="https://pandas.pydata.org/">https://pandas.pydata.org/</a>). There are also other libraries like Polars (<a href="https://docs.pola.rs/#license">https://pola.rs/</a>) which is supposedly faster than Pandas. At the end, you can do it with the tool you prefer, as long as you feel comfortable, and know how to proceed.</p><p>I used Pandas and the approach I took is the following:</p><ul><li>Correcting variable names with the ‘janitor’ package initially: it automatically corrects all the variables and keeps them in lower case.</li><li>Checking for data types consistency.</li><li>Checking for null values.</li><li>Checking and correcting the players position names like -&gt; ‘MF’: ‘Midfielder’, ‘DF’: ‘Defender’, ‘FW’: ‘Forward’, ‘GK’: ‘Goalkeeper’.</li><li>Check and correct the index and subindexes to only have one level in the dataframe.</li></ul><p>After checking all of that I applied the following in :</p><ul><li>Filtered by positions: midfielders and center forwards only.</li><li>Filtered players based on playtime: players who played over 400 minutes.</li><li>Normalized metrics per 90 minutes: it ensures a fair comparison of player performance by accounting for differences in playing time. This way all the data would be normalized and ready to be scaled before using the clustering algorithm.</li><li>Generating new <strong>features</strong>: by doing the per 90 normalization, new variables are created and this is a central part of a machine learning project, creating new features to enrich your modeling. Compared to other projects where you generate thousands of features, I created just a few new metrics upon the ones I already had. This helps narrow the scope of what I actually want to model (more on these variables in my next article about the clustering process).</li></ul><p><em>On a side note: I had created a second positions column named ‘position_2_stats_standard’ because sometimes FBRef adds two positions to a player, like you can be [MF, DF] (midfielder and defender), but I realized I only needed the first position (so if you were MF, DF I’d stay with MF position), and I decided to drop that column and keep the original.</em></p><p>I ended up creating a function in a Python file to make it extensible for the three leagues, but when I was prototyping in a Jupyter notebook I played around with a method of cleaning data with Pandas that I really love which is called “chaining”. In a nutshell, chaining is a method from Pandas used as a way of stringing together multiple operations on a DataFrame or Series in your code. It’s like building a recipe step by step, each method adds a new layer of transformation, keeping your workflow clean and readable. I got acquainted with this thanks to the amazing book of Matt Harrison <em>Effective Pandas: Patterns for Data Manipulation </em>(highly recommend to improve your skills on Pandas).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqQZIBUddRr7RWcgiWRc6Q.png" /><figcaption>Function I used to clean the data from the three leagues.</figcaption></figure><p>The result would be a DataFrame like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdJKY0-BPr5ICN3eQxxjjQ.png" /><figcaption>DataFrame result.</figcaption></figure><p>So far, I’ve described is the process of cleaning the FBRef data, but since my goal is to merge this data with the scraped one from Transfermarkt I needed, to clean the latter and then do the merge. It looked different to the FBRef data (let’s call it the combined leagues data) since what I have here is the players’ general information like team, age, season, country, position and value.</p><p>From these variables, I only used the ‘value’ variable to preprocess and merge with the FBRef combined leagues data, due to the fact that I already had the players name, age, country, position and team already there so it made sense only to include the ‘value’ column. At the end, what I want, is to shortlist midfielders and forwards that perform well and are affordable for the club.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u5PSUnlM8-nXCV9M8_7Ayg.png" /><figcaption>Transfermarkt data from the three leagues.</figcaption></figure><p>And again, I used a function that executes a Pandas chain by:</p><ul><li>Dropping null values from the ‘value’ column.</li><li>Removing the euro symbol.</li><li>Replacing the values from ‘value’ by scientific notation according to the amount (thousands and millions).</li><li>Convert these data types into numeric values.</li><li>And doing some sort of ‘normalization’ between FBRef combined leagues data and Transfermarkt by normalizing player names for consistent merging.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EeebcIu7eJoaCCUxC7NE6w.png" /><figcaption>Cleaning Tranfermarkt data before merging with FBRef to ensure consistency.</figcaption></figure><p>The final output would be a merged DataFrame which I called ‘df_training’, ready to use for K-Means clustering, a process which I’ll go through into detail on my next article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*269BQHw0rdbD3sTeQQqJhA.png" /><figcaption>Final DataFrame example. Note that I added the ‘value’ column and if you don’t see the columns normalized by 90 is because I took this screenshot from a DataFrame result I was testing in my developlement notebook, I added that functionality in the “production” code to upload it to the remote GitHub repo.</figcaption></figure><p>Up until here, I’ve shown, on a high level, the process I usually go through for cleaning data and generating new features. I do want to emphasize the importance of cleaning your data, or at least trying to in a first instance, for analyses purposes. It saves you from headaches and helps you have a better a model and better results, especially when analyzing player performance.</p><p>Thanks for reading and staying till the end, I hope you gain valuable insights and I’ll see you in my next article.</p><p>Always happy to chat!</p><p>You can follow me at:</p><p>X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass">x.com</a></p><p>Bluesky: <a href="https://bsky.app/profile/thenetpass.bsky.social">https://bsky.app/profile/thenetpass.bsky.social</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a91e50b30d34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping FBRef for Data-Driven Scouting and Enhanced Player Profiling]]></title>
            <link>https://ricardoheredia94.medium.com/scraping-fbref-for-data-driven-scouting-and-enhanced-player-profiling-464acad83270?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/464acad83270</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fbref]]></category>
            <category><![CDATA[soccer]]></category>
            <category><![CDATA[scouting]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Sat, 19 Oct 2024 12:59:05 GMT</pubDate>
            <atom:updated>2024-10-21T09:58:32.886Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*rmUYiJKTbn2ljJTxaWIssA.png" /><figcaption>FBRef logo.</figcaption></figure><h3>Why FBref?</h3><p>In today’s world where data in sports has become important for deciding which players are well-suited for a team. Obtaining insightful and detailed metrics has become crucial in order to profile a specific type of player or players for scouting them and eventually signing them or even transfer them to another team.</p><p>Places like FBRef help a lot with that. In this popular web in the soccer analytics world, you can obtain advanced metrics of squads and players from several leagues, such as passing accuracy, defensive stats, and shot-creating actions that can help you build a solid idea of how a player performs.</p><p>That’s why this time, I wanted to scrape FBRef in order to obtain different kind of stats from the three leagues I’ve selected to work with: Championship, League One and League Two, and thus, combine them with the data I already got from Transfermarkt, which I described how to do it in my previous article. You can check it in the link below:</p><p><a href="https://ricardoheredia94.medium.com/scraping-transfermarkt-with-python-data-driven-player-recruitment-using-englands-championship-8ad18b9103fe">Scraping Transfermarkt with Python: Data-Driven Player Recruitment using England’s Championship…</a></p><p>This approach aims to build more detailed player profiles, allowing me to better assess midfielders and strikers suited for a hypothetical signing for Sheffield United.</p><p>In this article, I’ll describe my personal approach on I scraping the data I needed from FBRef, how preprocessed it and how I plan to integrate it with the Transfermarkt data, in order to cluster and identify the top five players for each position from the mentioned leagues.</p><h3>How I did it</h3><p>I basically employed a first-manual-then-automatic approach with Python to do this and using only the <em>requests</em> and <em>pandas </em>packages. I got inspired in a couple of options I found in YouTube and Medium if you want to check them out too:</p><p><a href="https://medium.com/@ricardoandreom/how-to-scrape-and-personalize-data-from-fbref-with-python-a-guide-to-unlocking-football-insights-7e623607afca">How to scrape and personalize data from FBref with Python: A guide to unlocking Football Insights</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FI0TB9m_hyCo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DI0TB9m_hyCo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FI0TB9m_hyCo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcbf561d782ac7d44179ef2f22f1a6e4/href">https://medium.com/media/dcbf561d782ac7d44179ef2f22f1a6e4/href</a></iframe><p>This helped me make my own idea and look into a more personal approach, since I’m obtaining data from lesser known leagues like League One and Two. So, trying to keep it as simple as I could, and thinking that since I’m basically doing the same process three times for three not-so-different leagues, I proceded scraping the Championship, that way I would only do it faster and automated for the other two.</p><ol><li>First, I manually checked the page of the Championship league for the 2024–2025 season and went tab to tab to get familiar with the metrics I actually needed (they’re a lot, and most of the time you don’t need that many). Since what I’m interested in are players, specially midfielders and forwards I only used data from all of the tabs except Goalkeepers.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IarhyLFMi4oSbFSQ-eLkVw.png" /><figcaption>Squad &amp; players tabs.</figcaption></figure><p>2. A little bit of web scraping techniques were needed here because in these tabs you have two tables, the Squad stats, which come first and then the Players table, so when you look at the HTML, this table is commented out like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uci6EDh3Ujqklfv5c6nE4Q.png" /><figcaption>Source code view of the HTML code.</figcaption></figure><p>You need to figure out a way to access it and get the data needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VvanQMcakNt9lK12RTQxoQ.png" /><figcaption>Player’s table stats.</figcaption></figure><p>So, a function that could get that was a good choice:</p><pre>def get_clean_table(url, table_id):<br>    &quot;&quot;&quot;<br>    Fetches and cleans a table from fbref.com given a URL and a table ID.<br><br>    Args:<br>        url (str): The URL of the webpage containing the table.<br>        table_id (str): The &#39;id&#39; attribute of the HTML table.<br><br>    Returns:<br>        pd.DataFrame: The cleaned and ready-to-use DataFrame.<br>    &quot;&quot;&quot;<br>    # Make the request and get the HTML content<br>    response = requests.get(url)<br>    html = response.text.replace(&#39;&lt;!--&#39;, &#39;&#39;).replace(&#39;--&gt;&#39;, &#39;&#39;)<br><br>    # Read the specific HTML table<br>    df = pd.read_html(html, attrs={&#39;id&#39;: table_id})[0]<br><br>    # Clean the columns<br>    df = clean_columns(df)<br><br>    # Fill missing values and reset the index<br>    df = df.fillna(0).reset_index(drop=True)<br><br>    return df</pre><p>2. After obtaining the table, there was cleaning stuff to do regarding mutlti-index and columns names so I could get a decent result:</p><pre>def clean_columns(df):<br>    &quot;&quot;&quot;<br>    Cleans and flattens the columns of a DataFrame with multi-level indices.<br><br>    Args:<br>        df (pd.DataFrame): The original DataFrame with multi-level column indices.<br><br>    Returns:<br>        pd.DataFrame: The DataFrame with flattened and cleaned columns.<br>    &quot;&quot;&quot;<br>    # Flatten the columns by joining the levels<br>    df.columns = [&#39; &#39;.join(col).strip() if isinstance(col, tuple) else col for col in df.columns]<br><br>    # Rename the columns if necessary<br>    new_columns = []<br>    for col in df.columns:<br>        if &#39;level_0&#39; in col:<br>            new_col = col.split()[-1]<br>        else:<br>            new_col = col<br>        new_columns.append(new_col)<br>    df.columns = new_columns<br><br>    return df<br><br>def process_dataframe(df):<br>    &quot;&quot;&quot;<br>    Applies additional processing to a DataFrame.<br><br>    Args:<br>        df (pd.DataFrame): The DataFrame to be processed.<br><br>    Returns:<br>        pd.DataFrame: The processed DataFrame.<br>    &quot;&quot;&quot;<br>    # Check for the existence of columns before applying operations<br>    if &#39;Age&#39; in df.columns:<br>        df[&#39;Age&#39;] = df[&#39;Age&#39;].astype(str).str[:2]<br>    else:<br>        df[&#39;Age&#39;] = None<br><br>    if &#39;Pos&#39; in df.columns:<br>        df[&#39;Position_2&#39;] = df[&#39;Pos&#39;].astype(str).str[3:]<br>        df[&#39;Position&#39;] = df[&#39;Pos&#39;].astype(str).str[:2]<br>    else:<br>        df[&#39;Position_2&#39;] = None<br>        df[&#39;Position&#39;] = None<br><br>    if &#39;Nation&#39; in df.columns:<br>        df[&#39;Nation&#39;] = df[&#39;Nation&#39;].astype(str).str.split(&#39; &#39;).str.get(1)<br>    else:<br>        df[&#39;Nation&#39;] = None<br><br>    # Remove columns if they exist<br>    df = df.drop(columns=[&#39;Rk&#39;, &#39;Matches&#39;], errors=&#39;ignore&#39;)<br><br>    # Replace abbreviations with full names<br>    position_mapping = {&#39;MF&#39;: &#39;Midfielder&#39;, &#39;DF&#39;: &#39;Defender&#39;, &#39;FW&#39;: &#39;Forward&#39;, &#39;GK&#39;: &#39;Goalkeeper&#39;}<br>    df[&#39;Position&#39;] = df[&#39;Position&#39;].replace(position_mapping)<br>    df[&#39;Position_2&#39;] = df[&#39;Position_2&#39;].replace(position_mapping)<br><br>    return df</pre><p>3. The manual part was selecting each URL from each tab I wanted to gather info from and then turning them into dataframes so I could use that data to do some preprocessing and then merging and getting this huge dataset of FBRef metrics.</p><pre># URLs and table IDs<br>tables_info = [<br>    (&#39;https://fbref.com/en/comps/10/stats/Championship-Stats&#39;, &#39;stats_standard&#39;),<br>    (&#39;https://fbref.com/en/comps/10/shooting/Championship-Stats&#39;, &#39;stats_shooting&#39;),<br>    (&#39;https://fbref.com/en/comps/10/passing/Championship-Stats&#39;, &#39;stats_passing&#39;),<br>    (&#39;https://fbref.com/en/comps/10/passing_types/Championship-Stats&#39;, &#39;stats_passing_types&#39;),<br>    (&#39;https://fbref.com/en/comps/10/gca/Championship-Stats&#39;, &#39;stats_gca&#39;),<br>    (&#39;https://fbref.com/en/comps/10/defense/Championship-Stats&#39;, &#39;stats_defense&#39;),<br>    (&#39;https://fbref.com/en/comps/10/possession/Championship-Stats&#39;, &#39;stats_possession&#39;),<br>    (&#39;https://fbref.com/en/comps/10/playingtime/Championship-Stats&#39;, &#39;stats_playing_time&#39;),<br>    (&#39;https://fbref.com/en/comps/10/misc/Championship-Stats&#39;, &#39;stats_misc&#39;),<br>]<br><br># Dictionary to store the processed DataFrames<br>dataframes = {}<br><br># Iterate over each URL and table ID<br>for url, table_id in tables_info:<br>    # Fetch and clean the table<br>    df = get_clean_table(url, table_id)<br>    # Apply the additional processing<br>    df = process_dataframe(df)<br>    # Store the DataFrame in the dictionary using the table ID as the key<br>    dataframes[table_id] = df</pre><p>The result was something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKKZ1DFhoZWWwcSFLEbR3w.png" /><figcaption>Players standard stats.</figcaption></figure><p>4. I wanted to select the columns of metrics so I did a dictionary that included key-value pairs of the lists of variables I wanted to obtain:</p><pre>columns_to_retain = {<br>    &#39;stats_standard&#39;: [&#39;Player&#39;, &#39;Nation&#39;, &#39;Squad&#39;, &#39;Age&#39;, &#39;Born&#39;, &#39;Playing Time 90s&#39;, &#39;Position_2&#39;, &#39;Position&#39;],<br>    &#39;stats_passing_types&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;Pass Types TB&#39;,&#39;Pass Types Sw&#39;, <br>                            &#39;Pass Types Crs&#39;, &#39;Outcomes Cmp&#39;, &#39;Outcomes Blocks&#39;],<br>    &#39;stats_passing&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;Total Cmp&#39;, &#39;Total Att&#39;, <br>       &#39;Total Cmp%&#39;, &#39;Short Cmp&#39;, &#39;Short Att&#39;, <br>       &#39;Short Cmp%&#39;, &#39;Medium Cmp&#39;, &#39;Medium Att&#39;,<br>       &#39;Medium Cmp%&#39;, &#39;Long Cmp&#39;, &#39;Long Att&#39;, <br>       &#39;Long Cmp%&#39;, &#39;Ast&#39;, &#39;1/3&#39;, &#39;PPA&#39;, &#39;PrgP&#39;],<br>    &#39;stats_playing_time&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;Playing Time 90s&#39;, &#39;Playing Time MP&#39;,&#39;Playing Time Min&#39;],<br>    &#39;stats_shooting&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;Standard Gls&#39;, &#39;Standard Sh&#39;, &#39;Standard SoT&#39;, &#39;Standard SoT%&#39;,<br>       &#39;Standard Sh/90&#39;, &#39;Standard SoT/90&#39;],<br>    &#39;stats_gca&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;SCA SCA&#39;,&#39;GCA GCA&#39;],<br>    &#39;stats_defense&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;Tackles Tkl&#39;,<br>       &#39;Tackles TklW&#39;, &#39;Tackles Def 3rd&#39;, &#39;Tackles Mid 3rd&#39;, &#39;Tackles Att 3rd&#39;, &#39;Blocks Blocks&#39;, &#39;Blocks Sh&#39;, &#39;Blocks Pass&#39;, &#39;Int&#39;, &#39;Clr&#39;, &#39;Err&#39;],<br>    &#39;stats_possession&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;Touches Touches&#39;, &#39;Touches Mid 3rd&#39;, &#39;Touches Att 3rd&#39;, &#39;Touches Att Pen&#39;,<br>       &#39;Take-Ons Att&#39;, &#39;Take-Ons Succ&#39;, &#39;Take-Ons Succ%&#39;, &#39;Carries Carries&#39;, &#39;Carries PrgC&#39;, &#39;Carries 1/3&#39;, &#39;Carries CPA&#39;,<br>       &#39;Carries Mis&#39;, &#39;Carries Dis&#39;, &#39;Receiving Rec&#39;, &#39;Receiving PrgR&#39;],<br>    &#39;stats_misc&#39;: [&#39;Player&#39;, &#39;Squad&#39;, &#39;90s&#39;, &#39;Performance Off&#39;,<br>       &#39;Performance Crs&#39;, &#39;Performance Int&#39;, &#39;Performance TklW&#39;,<br>       &#39;Performance Recov&#39;, &#39;Aerial Duels Won&#39;, &#39;Aerial Duels Lost&#39;,<br>       &#39;Aerial Duels Won%&#39;]<br>}</pre><p>5. Merging the dataframes by using a function and obtaining a final merged dataframe:</p><pre>def merge_dataframes(dataframes_dict, columns_dict):<br>    &quot;&quot;&quot;<br>    Merges the dataframes from `dataframes_dict` by filtering the required columns from `columns_dict`.<br><br>    Args:<br>        dataframes_dict (dict): Dictionary of processed dataframes.<br>        columns_dict (dict): Dictionary with the columns to retain for each dataframe.<br><br>    Returns:<br>        pd.DataFrame: Merged DataFrame with the specified columns.<br>    &quot;&quot;&quot;<br>    filtered_dfs = []<br>    <br>    # Filter the columns to retain from each dataframe, only if the columns exist<br>    for key, df in dataframes_dict.items():<br>        if key in columns_dict:<br>            print(f&quot;Columns available in &#39;{key}&#39;: {df.columns.tolist()}&quot;)  # Debug: print available columns<br>            columns_to_retain = [col for col in columns_dict[key] if col in df.columns]<br>            print(f&quot;Columns to retain for &#39;{key}&#39;: {columns_to_retain}&quot;)  # Debug: print columns to retain<br>            df_filtered = df[columns_to_retain].copy()<br>            df_filtered.columns = [f&quot;{col}_{key}&quot; if col not in [&#39;Player&#39;, &#39;Squad&#39;] else col for col in df_filtered.columns]<br>            filtered_dfs.append(df_filtered)<br>    <br>    # Concatenate all dataframes on &#39;Player&#39; and &#39;Squad&#39;<br>    merged_df = pd.concat(filtered_dfs, axis=1)<br>    <br>    # Remove duplicate columns for &#39;Player&#39; and &#39;Squad&#39;<br>    merged_df = merged_df.loc[:, ~merged_df.columns.duplicated()]<br>    <br>    return merged_df<br><br><br>merged_dataframe = merge_dataframes(dataframes, columns_to_retain)</pre><p>And here we go! The table of metrics I was looking for. I forgot to mention that I also use the package <em>janitor </em>which is super helpful to auomatically clean the variables names you can install it with <a href="https://pypi.org/project/pyjanitor/"><strong>pip install pyjanitor</strong></a><strong> </strong>and you’ll see how cool it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uroVdSi6CuhOngbbaxRxNA.png" /><figcaption>Final dataframe output.</figcaption></figure><p>As you can see there are a lot of NaN values in the final dataframe that needs a little bit more cleaning, but the rough part of obtaining the data is pretty much done and ready to be used in an automated way to obtain League One and League Two data.</p><p>With that being said, I hope this is helpful for someone trying to scrape FBRef, and if you read all the way down here, please feel free to give me your opinions on this, how would you do it differently, what would you improve, etc.</p><p>Always a pleasure to connect!</p><p>Sports Analytics X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass">x.com</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=464acad83270" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping Transfermarkt with Python: Data-Driven Player Recruitment using England’s Championship…]]></title>
            <link>https://ricardoheredia94.medium.com/scraping-transfermarkt-with-python-data-driven-player-recruitment-using-englands-championship-8ad18b9103fe?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ad18b9103fe</guid>
            <category><![CDATA[football-analytics]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[scouting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Sat, 05 Oct 2024 09:17:22 GMT</pubDate>
            <atom:updated>2024-10-05T09:17:22.093Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scraping Transfermarkt with Python: Data-Driven Player Recruitment using England’s Championship, League Two, and League One data.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CuxNcqEdYO8Pufow" /><figcaption>Transfermarkt logo.</figcaption></figure><p>Web scraping has always intrigued me. In my opinion, is one of the best skills you can have in the data science/machine learning toolbox. It is as challenging and creative as you can imagine, useful for when you want to build your own custom databases for personal projects and you don’t have access to paid providers.</p><h3>Motivation Behind the Project</h3><p>The goal of the project is to identify a central midfielder with a mixed profile (good in both defense and attack) and a striker, focusing on finding five players for each position who fit the desired characteristics (based on different metrics from FBRef and team’s playing style) and that also have an accessible cost for the Sheffield United, team I randomly chose to carry on with this scouting project. These positions to be reinforced, in a hypothetical upcoming transfer window, has been selected after video analyzing three matches from Sheffield and identifying positions that could benefit from these signings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZliFi8OzY3sk0DMxzOIY7g.png" /><figcaption>Team logo.</figcaption></figure><p>To achieve this, I needed to gather extensive data from Transfermarkt. For those unfamiliar with this web page, it is a place that holds players and team data such as age, country, contract, positions, and market values, a variety of data points helpful to identify potential players, which, for a sporting director and a scouting department is critical in order to make signing decisions.</p><p>To show you the rest of the project’s scope, after identifying the potential target players, I’m planning to use a clustering algorithm, specifically K-Means, to use a broad set of features (along with metrics of these players obtained from FBRef ) and group players with the desired profile from the three leagues.</p><p>From there, I’d get a final shortlist of players for each position. And, to enrich the analysis, I’m also planning to gather event-level data from WhoScored (data provided by Opta), to evaluate recent matches performance of the shortlisted players.</p><p>So, to do this efficiently, I needed an automated way to extract player information directly from Transfermarkt, which led me to build a web scraper.</p><blockquote>Note: a huge shoutout to this project from Pol Marin, from which I was inspired by, and made me want to replicate the clustering idea and extend his approach. <a href="https://towardsdatascience.com/using-clustering-algorithms-for-player-recruitment-98208d3a6cb4">https://towardsdatascience.com/using-clustering-algorithms-for-player-recruitment-98208d3a6cb4</a></blockquote><h3>Designing the Scraper: Building Process and Core Components</h3><p>In order to build the scraper, I first needed to understand what and how I was going to do it. So I wanted to see if somebody else did it first and fortunately, someone did. These two Medium articles were my go-to references to help me go through this task:</p><blockquote><a href="https://blog.devgenius.io/get-to-know-the-greatest-football-league-in-the-world-with-python-and-transfermarkt-25a7cfa6b136">https://blog.devgenius.io/get-to-know-the-greatest-football-league-in-the-world-with-python-and-transfermarkt-25a7cfa6b136</a></blockquote><blockquote><a href="https://medium.com/@conalhenderson/how-to-build-a-custom-web-scraper-to-extract-premier-league-player-market-data-3b8e5378cca2">https://medium.com/@conalhenderson/how-to-build-a-custom-web-scraper-to-extract-premier-league-player-market-data-3b8e5378cca2</a></blockquote><p>I specially followed Conal’s comprehensive tutorial to build my own custom web scraper, as he shows how to do it exactly but with Liverpool’s and Premier League data, so it was a matter of going through it, running the code and understanding the process that happens under the hood when scraping Transfermarkt.</p><p>It was challenging, honestly. But after a few days I managed to replicate the exact same process for my goal: obtaining England’s tier 2, 3 and 4 data. The bright side, is that I learned a new way of web scraping and improved capabilities on Python coding, which is necessary for when you want your code to be more readable and scalable at the same time your project grows and develops.</p><p>I won’t get into each detail of the code though if you’re looking to do some web scraping, I highly recommend following the tutorial to better understand step by step where everything comes from and the final output, since it requires some attention if you don’t know German (all of Transfermarkt source code is in German). I’ll focus on the core parts I included in my pipeline.</p><p>So, to make the scraper as modular and efficient as possible, and using Python flexible capabilities, I designed it with some key components:</p><ol><li><strong>Abstract Parser Class</strong>: I used an Abstract Base Class (Parser) as a template for all specific parsers which helped obtain different types of player data. As you’ll see in the image I focused on getting these data points from all the players: names, ages, countries, market values, and positions (plus the team and the season), because these were the ones I could get, somehow, I couldn’t manage to obtain other datapoints from the HTML source (got to look into that later), but I believe these were the most helpful for my project goal.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UcaTjArnuEVCDCUVcMlaHA.png" /><figcaption>Parsers to obtain the desired data.</figcaption></figure><p>By breaking the parsing into smaller, independent classes, you can maintain a cleaner code and made modifications easier in the feature without affecting other components.</p><p><strong>2. Dataclasses for Structured Data</strong>: To organize my code and ensure type safety, I used Python’s dataclass to create a Team class. This helped me encapsulate relevant information such as the team ID and name, and it was an efficient way to pass data around the scraper.</p><p><strong>3. The Scraper Class</strong>: This was the core of the project. The Scraper class took a team and a list of parsers and then extracted data from Transfermarkt. The structure of the URL allowed for programmatic generation by passing in the team ID and name, making the whole process automated and highly scalable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zUMJ5xsFZ8-8OTSFJaIzMg.png" /><figcaption>Scraper class.</figcaption></figure><p>This part of the code shows the main Scraper class I mentioned, which fetches data for a given team and year, and allows you to scale the scraping across different teams by utilizing a uniform URL structure. It uses BeautifulSoup to retrieve the HTML and <strong>httpx </strong>with custom headers to mimic a browser. Also, using ‘time.sleep(5)’ is critical to use to avoid overwhelming the server, getting kicked out and blocked in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZunERz-BHS8mPu-xm5jaGQ.png" /><figcaption>Scraper function that called the Scraper class and shows how the parsers are used and outputs a CSV with all the info.</figcaption></figure><h3>The Result: Structured CSV file ready to use for scouting analysis</h3><p>The final dataset included the information I meant to scrape: player names, ages, countries, positions, and market values, which I saved into CSV files for further analysis.</p><p>Additionally, I uploaded the data to Supabase in different SQL tables for each league to make it easily accessible for querying and integration with other tools afterwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3o_7yq0MzQKAW8s5h1NJCQ.png" /><figcaption>Championship players table data.</figcaption></figure><p>This allows me to have a central structured storage and to query this data quickly, be it inside Supabase or using a connection to that database and clean the data with Pandas in another development environment.</p><h3>Lessons learned and next steps</h3><p>In this phase of the project I learned a lot about handling the intricacies and best practices of web scraping, but specially I learned about improving the usability of my code, breaking it into modular pieces using classes and parsers, making it easier to extend the scraper and add new features as needed.</p><p>This is just the beginning. As next steps:</p><ul><li>Extend the scraping pipeline for FBRef players metrics and WhoScored event-level data for short-listed players.</li><li>Applying the clustering to identify players that fit the profile for midfielders and strikers.</li><li>Shortlist players across the Championship, League One, and League Two and</li><li>Make a descriptive scouting report with visualizations to show results.</li></ul><p>My intention is to release Medium articles for the corresponding phases of projects to keep showing the progress and storing the code in a GitHub repo.</p><p>If you reached here, thanks for reading . If you’re interested in scraping or football analytics, I’d love to hear your thoughts or discuss any challenges you’ve faced as well.</p><p>Get in touch over here:</p><p>Sports Analytics X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass?source=post_page-----42efff231524--------------------------------">x.com</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ad18b9103fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding the Top Player Performer in EURO 2024 with SQL]]></title>
            <link>https://ricardoheredia94.medium.com/finding-the-top-player-performer-in-euro-2024-with-sql-42efff231524?source=rss-77acb6cb16d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/42efff231524</guid>
            <category><![CDATA[supabase]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[football-analytics]]></category>
            <dc:creator><![CDATA[Ricardo Heredia]]></dc:creator>
            <pubDate>Thu, 12 Sep 2024 20:27:29 GMT</pubDate>
            <atom:updated>2024-09-13T06:49:36.950Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*K0SSwbvpDy-yThR_sJa-EQ.jpeg" /><figcaption>EURO 2024</figcaption></figure><p>Not so long ago when I started in the world of data, one of the main technologies I found out about was <strong>SQL (sequel or s-q-l)</strong>, a more than popular go-tool to pool, organize and analyze data from databases.</p><p>Until recently, I didn’t realize the importance it had and how great it can be to have it in the toolboox of skills to work with data. After putting up with lots of messy CSV files scattered around my folders, I had the urge to use something more robust to organize my data.</p><p>Due to the fact that I’m constantly learning and I love analyzing football/soccer data, I took the course of <strong>SQL &amp; Databases for Sports Analytics from McKay Johns</strong> to brush up my SQL skills and finally start implementing a methodology I could initially stick to, in order to organize my own databases for further analyses and developing valuable insights.</p><p>Huge shoutout to the course, it was great (link at the end) and what I liked the most, and it’s the main point of this article, was the <strong>Final Project, </strong>where you get to design a database, analyze it and report its insights using the <strong>EURO 2024</strong> StatsBomb free data. You do this in an amazing open-source cloud-based platform called <strong>Supabase</strong> which focuses on <strong>PostgreSQL</strong> as its database.</p><p>The project consisted in the following phases:</p><ol><li><strong>Database design &amp; creation</strong></li><li><strong>Query development</strong></li><li><strong>Stored Procedures</strong></li><li><strong>Advanced queries &amp; final reporting</strong></li></ol><p>In each phase there were specific subtasks you had to complete, covering all the theory taught in the program.</p><p>For example, in the database design phase you had to use <strong>Creately, </strong>a visual collaboration platform that allows you to create mind maps or diagrams like entity-relationship diagrams (ERDs) to make the databases used in the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*_OYYbJUlhqRRMPDqYgRwUw.png" /><figcaption>Example ERD diagram made in Creately.</figcaption></figure><p>After creating the database, you had to upsert the variables and the data in it from a CSV file using Supabase interface. Then with all set, use the SQL editor to the make initial queries, get acquainted with the data and find preliminary insights. The hardest part for me was Phase 4, where you needed to build a final comprehensive query in order to determine the “Player of the Tournament” based on some statistics like:</p><ul><li>Pass Accuracy</li><li>Expected Goals (xG)</li><li>Total Goals Scored</li></ul><p>Finally once you made this calculations, you had to assign a rating for each stat. This was a good opportunity to implement concepts like: <strong>aggregate functions</strong>, <strong>CTEs</strong>, <strong>subqueries</strong>, <strong>JOIN</strong>, <strong>CASE statements</strong> and <strong>having clauses</strong>.</p><p>Below I’m sharing the final query and how I structured everything step by step:</p><ul><li>The queries for each stat in order to visualize what I was making.</li><li>Then the logic of the CTEs and JOINs, including the logic for the rating calculation at the end.</li><li>The final result shown in the image after the code block.</li></ul><pre>with pass_accuracy as (<br>  select player_id,<br>  sum (<br>    case<br>      when pass_outcome is null then 1.0 -- this is so the pass is count as float, null is because is successful<br>      else 0<br>    end <br>    ) / count(*) pass_accuracy<br>  from project_match_event<br>  where type = &#39;Pass&#39;<br>  group by player_id<br>  having count(*) &gt; 20 <br>),<br>goals as (<br>  select player_id, count(*) as goals_scored<br>  from project_match_event<br>  where shot_outcome = &#39;Goal&#39;<br>  group by player_id<br>),<br>total_xg as (<br>  select player_id, sum(shot_statsbomb_xg) as total_xg<br>  from project_match_event<br>  group by player_id<br>),<br>avg_tournament_rating as (<br>  select p.name as player_name, t.name as team_name, p.position, pa.pass_accuracy,<br>  case<br>    when pa.pass_accuracy &gt;= .9 then 10<br>    when pa.pass_accuracy between .7 and .9 then 5<br>    else 0<br>  end as pass_accuracy_rating,<br>  g.goals_scored,<br>  case<br>    when g.goals_scored &gt;= 2 then 10<br>    when g.goals_scored = 1 then 5<br>    else 0<br>  end as goals_rating,<br>  xg.total_xg,<br>  case<br>    when xg.total_xg &gt;= 1.5 then 10<br>    when xg.total_xg between 0.8 and 1.5 then 5<br>    else 0<br>  end as total_xg_rating<br>  from project_player p<br>  left join project_team t on p.team_id = t.team_id<br>  left join pass_accuracy pa on p.player_id = pa.player_id<br>  left join goals g on p.player_id = g.player_id<br>  left join total_xg xg on p.player_id = xg.player_id<br>)<br>select team_name, player_name, position, goals_scored, pass_accuracy, total_xg,<br>        (pass_accuracy_rating + goals_rating + total_xg_rating) / 3.0 as avg_tournament_rating<br>from avg_tournament_rating<br>order by avg_tournament_rating desc;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZmwRjJNVolPOAbw_4YdX6A.png" /><figcaption>Query report: Bernardo Silva is ranked as the “MVP” of the tournament.</figcaption></figure><p>Of course, this isn’t the only way to tackle this queries since there aren’t 100% correct answers, though there’s room for improvement, you can also try:</p><ul><li>Excluding goals from penalty shootouts.</li><li>Including more metrics that covers defensive metrics as well, since these were mainly for attackers and by doing this it’d be a more global comparison.</li><li>Using different ranges when making the CASE statements.</li></ul><p>I have to say I’m satisfied with the process of doing this project. I’ve learnt a lot by getting my hands dirty on databases creation, dedicating time to understand each concept at a high level and also leveraging the aid of ChatGPT to help me dissect concepts and understand the errors I’ve made along the way.</p><p>I hope you liked this use case of SQL and whether you do the course, use SQL in your day to day or have any comment about it don’t hesitate to get in touch with me over here:</p><p>Sports Analytics X/Twitter account 👇🏻</p><p><a href="https://x.com/thenetpass">x.com</a></p><p>👉🏻 LinkedIn profile: <a href="https://www.linkedin.com/in/ricardo-heredia-9b97b0181/">https://www.linkedin.com/in/ricardo-heredia-9b97b0181/</a></p><p>PS: Below are the links to the SQL course and the access to StatsBomb free data.</p><ul><li><a href="https://mckay-s-site.thinkific.com/courses/sql-sports">Best SQL and Databases Course</a></li><li><a href="https://statsbomb.com/news/statsbomb-release-free-euro-2024-data/">StatsBomb Release Free Euro 2024 Data - StatsBomb | Data Champions</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42efff231524" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>