<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Ansted on Medium]]></title>
        <description><![CDATA[Stories by Daniel Ansted on Medium]]></description>
        <link>https://medium.com/@danielansted?source=rss-1d1fa7d2f998------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_Yr7uuKKq0RuaL3i</url>
            <title>Stories by Daniel Ansted on Medium</title>
            <link>https://medium.com/@danielansted?source=rss-1d1fa7d2f998------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:20:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielansted/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solving A Child’s Game: First Orchard]]></title>
            <link>https://medium.com/@danielansted/solving-a-childs-game-first-orchard-eb8101484188?source=rss-1d1fa7d2f998------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb8101484188</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[board-games]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Daniel Ansted]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 09:12:01 GMT</pubDate>
            <atom:updated>2025-10-21T09:12:01.518Z</atom:updated>
            <content:encoded><![CDATA[<p>Disclaimer: This article describes my method in solving the child’s game First orchard. I have not packaged this for easy download, but it is available on my GitHub.</p><p>It is intended to be only a mathematical analysis and if you have a toddler at home, I highly recommend you purchasing a physical copy. I have put the instructions into my own words as the original instructions are copyrighted.</p><p>Intro:</p><p>First you can find all the code here: <a href="https://github.com/Noob1701/FirstOrchard">https://github.com/Noob1701/FirstOrchard</a>. A good intro to the project code is the montecarlo.ipynb file and the README.</p><p>Note: I am analyzing First Orchard not the similar game Orchard also published by HABA. You can find visuals of the physical game here <a href="https://www.habausa.com/products/my-very-first-games-first-orchard">https://www.habausa.com/products/my-very-first-games-first-orchard</a></p><p>My goal of this project was to determine the expected win rate of First Orchard. First, I programmed the game in Python, then I programmed and ran simulations on a few possible strategies, finally I solved it mathematically and checked this solution against the simulations to ensure accuracy.</p><p>Ultimately rounding to two decimal places, using the best strategy when a wild is rolled leads you to a 63.20% win chance, using the worst strategy gives a 55.5% chance of winning. Essentially you can effect the outcome ~7.7% of the time.</p><p>Knowing hard numbers like this might help the board game designer make a product better suited to their intended audience (in this case toddlers). I’ll give these thoughts at the end.</p><p>Description of the game:</p><p>There four fruits of four different colors red, blue, green, and yellow, making 16 fruits total. Each of these colors are represented on one side of a 6 sided die and when a color is rolled a fruit is placed into the basket.</p><p>On another side is a raven which moves the raven up the raven track. The last side is a wild, allowing the player to choose any color of fruit to place in the basket. This is the only decision in the game.</p><p>You lose the game if the raven reaches the end of the raven track (5 spaces) and you win the game if all 16 of the fruits are placed into the basket.</p><p>Programming the game:</p><p>As you can guess this was simple enough and the full code can be found at my GitHub. I decided on a class for the die, raven track, and fruit inventory (how many of each fruit) with the associated functions. These are then combined into a single GameState class.</p><p>Strategies and Simulations:</p><p>These strategies were only a few lines of code each and simple enough using list comprehension. I chose three strategies to test at this phase. Remember that there is only one real decision in the game, what to do when a wild is rolled.</p><p>So I chose to run simulations on always choosing the fruit with the least remaining (assuming at least one left), the most remaining, and a random fruit on a wild roll.</p><p>In my latest runs of these simulations I got the following percent chance wins: Always choosing the fruit color with with the least remaining ≈ 55.37%; choosing the fruit color with the most remaining ≈ 63.16%; and choosing a random fruit color each time ≈ 59.66%.</p><p>This gives us very good evidence that choosing from the fruit color with the most remaining is the best strategy. And if this were a more complex game we could analyze this further using inferential statistics. However, this game is simple enough that it can be solved directly bypassing the need for inferential statistics.</p><p>Solving the Game:</p><p>This was the complex part. The cliff notes of the solution is to create a list of every possible resulting game state from an initial game state of interest (fruit inventory and raven track) with a given consistent strategy and recording whether the end was a win or a loss and dividing the number of wins by the number of total possible game states from the initial state of interest. (This only works because each die roll has the same probability). In short, find the total number of possible wins and divide that total number possible game states.</p><p>Mathematicians would call this method a Markov Decision Process or an Absorbing Markov Chain. Normally, I was interested in the odds of winning given perfect play which is choosing from the fruit with the most left (if there are multiple choosing from one of those).</p><p>I don’t think it would be helpful to share all the code for the solution but I will share the final function for solving a single game state (without comparison) and explain the purpose of the functions referenced</p><pre>game_state = GameState()<br>    fruit_dict = {i + 3: fruit_count[i] for i in range(len(fruit_count))}<br>    _set_state(game_state, fruit_dict, raven_track)<br>    end_game_check = _win_perc_return_logic(game_state)<br><br>    if end_game_check is not None:  # game is over<br>        return end_game_check<br>    moves = _decrement_logic(game_state, strat)<br>    win = 0.0<br>    loss = 0.0<br>    for move in moves:<br>        win_instance, loss_instance = win_perc(<br>            move.fruit_inventory.fruit_values, move.raven_track.spaces, strat<br>        )<br>        win += win_instance<br>        loss += loss_instance<br><br>    return round(win / len(moves), 3), round(loss / len(moves), 3)</pre><p>_set_state sets up the initial condition the user is interested in. _win_perc_return_logic simply records a win or a loss or determines if the loop should continue. _decrement_logic records all possible game states in a list according to a consistent strategy for the wild rolls.</p><p>Inconsequential branches are pruned for this analysis (i.e. rolling a color when there are none of that fruit remaining). And then each possible resulting game state is recursively fed back through the win_perc function until all wins/losses are recorded.</p><p>Note: the random ‘solver’ technically not a solver, as it still relies on the random package. The other strategies (e.g. choosing from the fruit type that has the most remaining) do not rely on the random package and are true solvers.</p><p>The random solver not truly being a solver is something I may fix in an updated version, particularly if I follow through with the suggested analysis at the end of this article.</p><p>Testing:</p><p>The solver was tested against the simulations on all mathematically distinct combinations. I did not use inferential statistics to verify how close the simulations should be to the Markov Decision solution. But looking through the logs I did not think that this was necessary as everything was very close.</p><p>The most interesting thing I did in this testing was to give very good evidence (just short of a full proof) that choosing a fruit that has the most remaining always results in the best chance of winning.</p><p>Pygame version:</p><p>If you download my code you can play a visual version of the game in main.py. This not packaged so you have run it as you would a Python file. I usually go to the main.py file and run with debugging. See a screenshot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HV7P9_hnH5Y9ujnBB1GGlA.png" /><figcaption>After clicking on the fruit of your choice</figcaption></figure><p>This is a screenshot after I rolled a wild and I have chosen a fruit of my choice. The four black squares on the side is the visual representation of the raven track and the circles are the visual representation of the fruit inventory.</p><p>This screen tells me several things. First, that I made a non-optimal choice. Second, it tells me why my choice was not optimal; I should have chosen a fruit with the most remaining. Then it gives me my current odds (assuming optimal play going forward). It also gives the odds of winning I would have had, if I had made the optimal choice. Finally it tells me the difference in odds between my choice and the optimal choice.</p><p>I would simply click on the text “Click on a circle of your choice” to make the next roll (Again, this is the situation I had chosen the fruit).</p><p>There are improvements I could make to the pygame version, but I am content with how it is for this project for now. This was a trial run for a simple solver. And the meat of the project really is in the solution to the game.</p><p>Game Design Thoughts:</p><p>So after doing all of this analysis how might a boardgame designer use this information about the win rate? Randomly choosing a fruit has approximately 59.7% chance of winning. This seems quite realistic play. A toddler is likely not going to use a consistent strategy, and even the adult might not know or they might not care about using the best strategy when playing with a toddler.</p><p>There does not seem to be any hard research available as to the ideal win percent for a cooperative game such as this for toddlers. And perhaps this is an initial step that needs to be done. However, I personally feel like this is a just a little too low of a win percent and would advocate something higher.</p><p>One suggestion I might have to increase the odds of winning is to add one more raven space. This would increase the odds of winning using the random strategy to an estimated 73.8%. (This comes from the montecarlo.ipynb file in the repo).</p><p>The next logical step for this project would be to create a heat map of the win percentages based on a selection of raven spaces and number of starting fruits, especially using the random strategy. If there is enough interest I may do this, but I am content with this analysis at present.</p><p>Until we have more information about the appropriate win rate for a cooperative game for toddlers, such analysis would only give us various options and we wouldn’t know what win rate we should be aiming for.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb8101484188" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Candidates Tournament Analysis Part 3: Data Cleaning]]></title>
            <link>https://medium.com/@danielansted/candidates-tournament-analysis-part-3-data-cleaning-78497f4570f8?source=rss-1d1fa7d2f998------2</link>
            <guid isPermaLink="false">https://medium.com/p/78497f4570f8</guid>
            <category><![CDATA[chess]]></category>
            <category><![CDATA[data-cleaning-with-python]]></category>
            <dc:creator><![CDATA[Daniel Ansted]]></dc:creator>
            <pubDate>Sun, 07 Apr 2024 12:45:30 GMT</pubDate>
            <atom:updated>2024-04-07T12:45:30.348Z</atom:updated>
            <content:encoded><![CDATA[<p>So you have PGNs loaded and if you are working with data from master level players and not your own solo games you will probably notice that the names of the players are inconsistent. Here are a few variations I got: Ding Liren vs Liren Ding and Hi Nakamura vs. Hikaru Nakamura.</p><p>Also in my PGNs. there were unknown dates, especially days and months. I even had two games whose results were *’s (Korchnoi and Hubner apparently didn’t finish two games in my candidates dataset). I marked these as abandonned.</p><p>Thus, even though you may have gotten your source data from chessbase or chess.com. You cannot assume it’s clean and ready for use. Some of this is basic data cleaning and you would have to know and/or research this with domain knowledge to determine a or the correct name. E.g. Liren Ding should be Ding Liren.</p><p>This is the problem that I will focus on in this article, as this is the most novel data cleaning problem I had from the PGNs. Here was my solution to identify names needing processing. The underlying assumption is that they would share some part of a whole name.</p><pre># Initialize a set to store unique pairs of matching names<br>unique_matches_b = set()<br><br># Iterate over each row in the DataFrame<br>for index, row in candidates_df.iterrows():<br>    name_parts = row[&#39;Black&#39;].split()<br>    first_name = name_parts[0]<br>    middle_name = &#39; &#39;.join(name_parts[1:-1]) if len(name_parts) &gt; 2 else None<br>    last_name = name_parts[-1]<br>    <br>    # Check if any part of the name matches any part of other names elsewhere in the column<br>    for other_index, other_row in candidates_df.iterrows():<br>        other_name_parts = other_row[&#39;Black&#39;].split()<br>        other_first_name = other_name_parts[0]<br>        other_middle_name = &#39; &#39;.join(other_name_parts[1:-1]) if len(other_name_parts) &gt; 2 else None<br>        other_last_name = other_name_parts[-1]<br>        <br>        # Check if the first or middle name of one entry matches a last name of another entry<br>        if ((first_name == other_last_name or<br>             (middle_name and middle_name == other_last_name)) and<br>            (index != other_index)):<br>            <br>            # Check if the names are not exactly the same<br>            if row[&#39;Black&#39;] != other_row[&#39;Black&#39;]:<br>                # Add the pair of matching names to the set<br>                unique_matches_b.add(tuple(sorted([row[&#39;Black&#39;], other_row[&#39;Black&#39;]])))</pre><p>In short the code checks each match to see if it is partial or complete. If they are an exact match, this is not an issue that would obviously be the same player. In short it will give me a set of names like Ding Liren vs. Liren Ding.</p><p>Another problem I will discuss here is Hi Nakamura vs. Hikaru Nakamura. There were a few options here, but I went with storing and keeping the longest name. That did lead to some awkwardness with longer names, but if so desired you can change individual ones separately. (e.g. Robert James Fischer vs. Bobby Fischer). I kept these personally.</p><p>As the other problems are not particularly novel and are common data cleaning tasks I will end the article here. The short of this story is you can’t trust the headers of PGNs, it is a source of data just like the rest. The game moves should be accurate though, and in any case would be difficult to verify unless you got another source to compare.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78497f4570f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Candidates Tournament Analysis Part 2: Extracting the Opening Name from ECO]]></title>
            <link>https://medium.com/@danielansted/candidates-tournament-analysis-part-2-extracting-the-opening-name-from-eco-13687e0e53b1?source=rss-1d1fa7d2f998------2</link>
            <guid isPermaLink="false">https://medium.com/p/13687e0e53b1</guid>
            <category><![CDATA[chess]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-manipulation]]></category>
            <dc:creator><![CDATA[Daniel Ansted]]></dc:creator>
            <pubDate>Sat, 23 Mar 2024 15:04:36 GMT</pubDate>
            <atom:updated>2024-03-23T15:04:36.114Z</atom:updated>
            <content:encoded><![CDATA[<p>In the first part of this series I showed how to load multiple PGNs (Portable Game Notation) that have multiple chess games. In this article, I will discuss extracting opening information from the ECO code available in the resultant DataFrame using a separate mapping DataFrame (available in the Kaggle link at the end of this article).</p><p>The ECO Code is a short form of a long opening name. Each ECO code is one letter and two numbers A-E and numbers 00–99, making a total of 500 named openings. The each letter has a meaning by which the openings are classified.</p><p>A is uncommon/flank; B is Semi-Open other than French; C is Open and French; D is Closed and Semi-Closed; E is Indian. After this similar openings are typically grouped together so that A75 is a similar opening to A76. Thus, it is a straight forward task to get what I call an opening family from this first letter.</p><p>It should be noted, however, that some openings with the same short name can be found in different families depending on move order.</p><pre>open_fam_list = []<br>for i in range(len(candidates_df)):<br>    if &#39;A&#39; in candidates_df[&#39;ECO&#39;][i]:<br>        open_fam_list.append(&#39;Flank&#39;)<br>    elif &#39;B&#39; in candidates_df[&#39;ECO&#39;][i]:<br>        open_fam_list.append(&#39;Semi-Open, Not French&#39;)<br>    elif &#39;C&#39; in candidates_df[&#39;ECO&#39;][i]:<br>        open_fam_list.append(&#39;Open and French&#39;)<br>    elif &#39;D&#39; in candidates_df[&#39;ECO&#39;][i]:<br>        open_fam_list.append(&#39;Closed/Semi-Closed&#39;)<br>    elif &#39;E&#39; in candidates_df[&#39;ECO&#39;][i]:<br>        open_fam_list.append(&#39;Indian&#39;)<br><br>candidates_df[&#39;Opening_Family&#39;] = open_fam_list</pre><p>It is also a simple task to get a full opening name from the ECO code; however, the full codes tend to reference very specific openings which are often not helpful for aggregation. Here are a few examples: Benoni, Classical with …a6 and 10…Bg4; and Sicilian, Richter-Rauzer Attack, 7…a6 Defense, 11.Bxf6.</p><p>Using the full opening name, you would come to the erroneous conclusion that the English is the most common opening in the Candidates Tournament. It can be useful if you are trying to track a specific opening such as the infamous Ruy Lopez, Berlin Defense. To get the full opening name, it is simply a 1:1 mapping between ECO code and full opening name.</p><pre>opening_list = [] #initializing list <br>for i in range(len(candidates_df)): #loop through DataFrame that has game info<br>    for j in range(len(open_df)): #loop through DataFrame containing mapping<br>        if candidates_df[&#39;ECO&#39;][i] == open_df[&#39;ECO Code&#39;][j]: #find matches<br>           opening_list.append(open_df[&#39;Name&#39;][j]) #Append name of opening from       <br>                                                   #mapping DataFrame</pre><p>The short opening name requires some interpretation, but usually I chose to use the opening name up to the comma. In the examples above this would be Benoni and Sicilian. The full gambit (pun intended) of decisions is available along with the dataset at <a href="https://www.kaggle.com/datasets/danielansted/candidates-chess-tournament-data-1950-2022?rvi=1">https://www.kaggle.com/datasets/danielansted/candidates-chess-tournament-data-1950-2022?rvi=1</a></p><p>I added this short opening name to the mapping DataFrame so the code follows the same logic as the full opening name. If you would prefer your own custom opening name convention, you can simply add those to the mapping file available on the Kaggle dataset.</p><p>This is a brief introduction to ECO Codes and how to extract the opening names from them. The next article in this series will go over some basic data cleaning. The headers from these PGNs are inconsistent and sometimes incomplete.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13687e0e53b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Candidates Tournament Analysis Part 1: Getting Chess Data From PGNs]]></title>
            <link>https://medium.com/@danielansted/candidates-tournament-analysis-part-1-getting-chess-data-from-pgns-8546cd991f5c?source=rss-1d1fa7d2f998------2</link>
            <guid isPermaLink="false">https://medium.com/p/8546cd991f5c</guid>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[chess]]></category>
            <category><![CDATA[loading-data]]></category>
            <dc:creator><![CDATA[Daniel Ansted]]></dc:creator>
            <pubDate>Tue, 19 Mar 2024 13:27:52 GMT</pubDate>
            <atom:updated>2024-04-05T13:37:54.848Z</atom:updated>
            <content:encoded><![CDATA[<p>This is series created by myself with the aid of visuals in future articles by <a href="https://medium.com/u/16bba73278c6">Mythilyram</a>.</p><p>This article will show how I obtained and loaded chess tournament data pertaining to the Candidates tournaments into a DataFrame. This method can be used for any chess event or player that you can find collections for. I did not use programmatic web scraping, but with proper permissions and access you can. The resulting dataset will be available on Kaggle.com shortly.</p><p>I used a paid subscription to chessgames.com. It’s interface is outdated and has ads, but the advantage here is that they have PGNs that you can download in bulk easily.</p><p>There are plenty of options, chess.com for instance allows you to bulk download many master players games in bulk. Also many people like to use chessbase.com. I chose to download and analyze PGNs pertaining to historical Candidates Tournaments as the next one starts April 2.</p><p>First, what exactly is a PGN? Portable game notation (PGN) is a text file that has headers in square brackets that give information about the game (e.g. location, players, and etc…). The headers are essentially a key value pair, words not in quotes are the key and the quoted words are the value. After these headers the PGN contains the moves of the game in standard chess notation. Full example headers below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*iSP833YtBHZ5xJPH1Y2vXg.png" /></figure><p>This standard format allows the Python library chess.pgn to accurately read these games along with its metadata.</p><p>A PGN file is typically restricted to one game; however, that is not a strict rule and in fact I found many of the Candidates Tournaments in a single PGN. This saved a very large amount of time on data collection.</p><p>Once I collected all the Candidates games since 1950, I had two challenges. First how to read a PGN with multiple games and second how to load multiple games into a DataFrame in the same code. Below is the solution to these problems:</p><pre>game_data = []<br>for filename in os.listdir(folder_path):<br>  file_path = os.path.join(folder_path, filename)<br> <br>if filename.endswith(&#39;.pgn&#39;): # Check if the file is a PGN file<br>  with open(file_path) as pgn_file:<br>  while True:<br># Read each game from the PGN file, solves the problem of multiple games <br># in one PGN<br>game = chess.pgn.read_game(pgn_file)<br>  if game is None:<br>  break<br> <br> # Extract game infor from headers and then the moves of the game<br> game_info = {<br> &#39;Event&#39;: game.headers.get(&#39;Event&#39;, &#39;Unknown&#39;),<br> &#39;Site&#39;: game.headers.get(&#39;Site&#39;, &#39;Unknown&#39;),<br> &#39;Date&#39;: game.headers.get(&#39;Date&#39;, &#39;Unknown&#39;),<br> &#39;Round&#39;: game.headers.get(&#39;Round&#39;, &#39;Unknown&#39;),<br> &#39;White&#39;: game.headers.get(&#39;White&#39;, &#39;Unknown&#39;),<br> &#39;Black&#39;: game.headers.get(&#39;Black&#39;, &#39;Unknown&#39;),<br> &#39;Result&#39;: game.headers.get(&#39;Result&#39;, &#39;Unknown&#39;),<br> &#39;ECO&#39;: game.headers.get(&#39;ECO&#39;, &#39;Unknown&#39;),<br> &#39;Moves&#39;: game.mainline_moves()<br> }<br><br># Convert Moves column to list of moves<br>for game_info in game_data:<br>    game_info[&#39;Moves&#39;] = game_info[&#39;Moves&#39;].split()<br><br>game_data.append(game_info)<br>candidates_df = pd.DataFrame(game_data)</pre><p>If you follow this method you should have all games in a DataFrame that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_SNbPNSyLjGFWhqCnK5neg.png" /></figure><p>The next article in this series will show how I extracted the opening name from the ECO (Encyclopedia of Chess Openings) column present in this DataFrame.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8546cd991f5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Features for NLP Classification Tasks]]></title>
            <link>https://medium.com/@danielansted/new-features-for-nlp-binary-classification-tasks-bbc8f5f2506c?source=rss-1d1fa7d2f998------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbc8f5f2506c</guid>
            <category><![CDATA[binary-classification]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Daniel Ansted]]></dc:creator>
            <pubDate>Fri, 22 Sep 2023 00:24:54 GMT</pubDate>
            <atom:updated>2023-09-22T00:36:48.651Z</atom:updated>
            <content:encoded><![CDATA[<p>I had this thought that fake and real news had very different writing styles. So for my first (mostly) independent Data Science project I decided I would build a model that used readability scores as features.</p><p><strong>When can readability scores be used as features in Machine Learning?</strong></p><p>Only when there is a large amount of text that you are attempting to classify (e.g. fake vs. real news or predicting the genre of book).</p><p><strong>What are readability scores?</strong></p><p>Many readability scores, score a text based on linguistic features such as syllables per word and words per sentence. The Flesch Reading ease below is one example of a readability score:</p><p>206.835 — (1.015 * <strong>words / sentences</strong>) — 84.6 * (<strong>syllables / words</strong>))</p><p>More words per sentences and more syllables per word increase the score.</p><p>Another style of readability score features counting specific words or types of words to compute the score.</p><p>For instance, the Gunning-Fog formula essentially states that text that has many 3+ syllable words are more difficult to understand. And the Dale-Chall Score has a specific word list that it references to obtain a score.</p><p>Generally readability scores are used as an objective measure of how easy it is to understand a text. However, as a machine learning feature for classifying large amounts of text, we only care if it can help distinguish between categories.</p><p>We are not using it for it’s intended purpose. And there is no theoretical reason that these scores couldn’t be used for a similar purpose on non-English language texts.</p><p><strong>How to implement readability scores in a Python Notebook?</strong></p><ol><li>pip install <a href="https://pypi.org/project/textstat/">textstat</a></li></ol><p>2. Import text stat into your notebook</p><p>3. Choose your scorers</p><p>4. Choose the text you want scored</p><p>Once you have done this simply call the scorer per the textstat documentation like below:</p><p>textstat.flesch_reading_ease(your_text_data_goes_here)</p><p>In Pandas you can do this in bulk with multiple readability scores in a list and analyze the column with the text.</p><p>def read_score(scorer, column_name):<br> sclist = []<br> for i in range(len(Your_DataFrame)):<br> scvar = scorer(Your_DataFrame[‘Your Text Column’][i])<br> sclist.append(scvar)<br> Your_DataFrame[‘Scores_will_appear_here’] = sclist</p><p>for i in range(len(list_of_chosen_scorers)):<br> read_score(list_of_chosen_scorers[i], list_of_column_names[i]) — 1 column per readability score.</p><p><strong>Results of using textstat</strong></p><p>Obviously your results might differ, but let me share the highlights of my use.</p><p>In my fake news classification task (github can be found <a href="https://github.com/Noob1701/FakeNews">here</a>). The bulk of the work of the features were TF-IDF vectorizations of the top 500 words.</p><p>I checked feature importances of Logistic Regression and Random Forest Classifier. In the Logistic Regression the readability scores dominated most of them being in the top 10.</p><p>This model did perform slightly worse than the final Random Forest model which was a more modest success. In this readability scores were vastly overrepresented in the top 10% of features used.</p><p>Conclusion:</p><p>Readability scores such as the ones in the textstat Python library might be of use in NLP classification tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbc8f5f2506c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>