In order to perform my analysis, I created an actor-by-genre data set using raw data for all the movies at IMDb. To then cluster actors based upon the genres in which they appeared, I utilized the K-Means clustering technique. Ultimately, my objective was to see if there existed naturally occurring career archetypes among actors (i.e., how do actors naturally separate) based upon their respective distribution of genres.

The Question and Stakeholder

The question I wanted to answer was:

Can actors be grouped into meaningful career archetypes based on the distribution of genres in which they have appeared?

A possible stakeholder for this question would be the content strategy and/or casting group at a streaming service. For a company to develop new film or original streaming content, it will have to decide on what actors are likely to fit well with their various project types (e.g., by genre).

For instance, an actor who falls into a strong action-type cluster may be naturally suited for action franchises; whereas, an actor whose films fall into a wider range of clusters (comedy, drama, romance) may be able to help projects that cross multiple genres.

It should also be noted that the purpose of this model is NOT to say an actor can only excel in one kind of role. Rather, this model provides a data-based look at an actor’s previous work, and therefore, the model could assist decision making in areas of casting, content creation and marketing.

The Data

I constructed my dataset from a raw IMDb movie archive containing movies released between 2000 and 2022.

The raw records include information such as:

• IMDb movie ID
• title
• year
• runtime
• genres
• actors
• rating information

Rather than use a pre-assembled actor-by-genre matrix; I processed my own raw movie data, to create this actor-by-genre matrix. For every movie I recorded the actors and genres for movies. Next, I calculated how many times each actor had been involved in each genre. The resulting actor-by-genre matrix was comprised of 33,609 actors. Actors were displayed as rows in the table; genres were displayed as columns and each value in the cells represented how often that actor had acted in a movie classified under the same genre.

Some examples of genre features included: Comedy, Drama, Romance, Action, Crime, Horror, Thriller, Science Fiction (Sci-Fi), Fantasy, Adventure, Documentary, Biographical and Family.

Constructing the Dataset

I used Python’s json library to read the raw IMDb records one movie at a time. For each movie, I extracted its genre labels and actor IDs. I then created a dictionary that counted the number of appearances each actor had in every genre. After processing the entire dataset, I converted this dictionary into a Pandas DataFrame and saved the resulting matrix as a CSV file. This means the dataset used for clustering was created directly from the raw movie records rather than taken from an already-prepared actor-by-genre table.

Preparing the Features

I began reading the raw, unprocessed IMDb data (one movie at a time) using python’s built-in JSON package.

Next, I pulled out the genres for each movie and the ID’s for each of the actor(s) on those movie(s). Then, I constructed a dictionary to keep track of how many times each actor appeared in each genre.

Once I processed all of the movies, I took the previously-created dictionary and turned it into a pandas dataframe. I then wrote the normalized actor-by-genre feature matrix to a csv file.

Therefore, the dataset that I used to perform clustering was created directly from the raw movie data instead of being based on an already-constructed table of actor-by-genre.

Creating Features

Using the raw number of movie appearances to create features leads to scaling issues. For instance, when an actor has 100 movie appearances they will typically have many orders-of-magnitude larger genre counts than an actor who has only 10 movie appearances. Even though these two actors may have proportional amounts of their career appearing in certain genres, the actor with the greater amount of film experience will be given much greater weight during clustering.

In order to prevent this type of weighting issue, I employed row-normalization at the level of rows. Specifically, each actor’s genre count was divided by the total of his or her genre counts. Once normalized, each actor’s row added up to roughly 1.

As such, the features are indicative of the percentage of each actor’s career spent in specific genres as opposed to the absolute number of films in which he or she appeared.

Measuring Similarity

I applied k-means cluster analysis that utilizes euclidean distance to allocate individual data points (actors) into a group (cluster).

Each actor is assigned a set of coordinates in a multi-dimensional feature space defined by one feature per genre. The proximity of two or more actors is directly related to similarity in their relative portion of films within a given genre.

This methodology is an appropriate choice for addressing the type of question I am asking since it allows me to examine similarities/differences in “shapes” (i.e., distribution across genres) of the various actors’ film portfolios rather than comparing the sheer number of total films produced.

Selecting the Number of Clusters

K-Means requires selecting the number of clusters, or k, before fitting the model.

To choose a reasonable value, I used the elbow method. I trained K-Means models for values of k from 2 through 15 and recorded the inertia for each model.

Inertia measures how far observations are from the center of the cluster to which they are assigned. Lower inertia indicates tighter clusters.

The inertia decreased quickly at the lower values of k and then began decreasing more gradually. There was not one perfectly sharp elbow, but the curve showed diminishing improvement around the range of 6 to 8 clusters.

I selected k = 8 because it provided a reasonable balance between reducing within-cluster variation and maintaining groups that could still be meaningfully interpreted.

I also used a fixed random state and multiple K-Means initializations so that the results could be reproduced more consistently.

Cluster Sizes

After fitting the final K-Means model, all 33,609 actors were assigned to one of the eight clusters.

The cluster sizes varied substantially. The largest cluster contained 8,022 actors, while the smallest contained 975.

This suggests that some career patterns are much more common in the dataset than others.

What Do the Clusters Represent?

To understand each cluster, I examined the cluster centroids. The centroid shows the average normalized genre profile of the actors assigned to that cluster.

I also inspected actual actors belonging to each group.

Cluster 0: Comedy, Drama, and Romance

Cluster 0 is centered around comedy, but it is not a pure comedy group.

Comedy represents approximately 37.96% of the average cluster profile, followed by drama at 21.30% and romance at 12.61%.

Examples include Ryan Reynolds and Anne Hathaway.

This cluster appears to represent actors with relatively broad careers that frequently combine comedy with dramatic and romantic roles.

Cluster 1: Drama Specialists

Cluster 1 is one of the most specialized groups.

Drama accounts for approximately 99.79% of the average cluster profile.

Examples include T.C. Stallings and Robert Shepherd.

This group represents actors whose movie appearances in this dataset are almost entirely concentrated in dramatic films.

Cluster 2: Action, Drama, and Crime

Cluster 2 is strongly action-oriented.

Action makes up approximately 42.80% of the cluster center, followed by drama at 12.55% and crime at 11.05%.

Examples include Nicolas Cage and Jason Statham.

This cluster also contained several other actors strongly associated with action films, including Bruce Willis, Samuel L. Jackson, Liam Neeson, Dwayne Johnson, Mark Wahlberg, and Danny Trejo.

This makes Cluster 2 one of the most intuitive groups in the analysis.

Cluster 3: Horror Specialists

Cluster 3 is another highly specialized group.

Horror represents approximately 97.50% of the average genre profile.

Examples include Shawn C. Phillips and Ari Lehman.

This cluster appears to represent actors whose movie careers in the dataset are overwhelmingly focused on horror.

Cluster 4: Horror, Thriller, and Sci-Fi Mix

Cluster 4 is also associated with horror, but it is very different from Cluster 3.

Horror makes up only approximately 17.76% of the average profile, followed by thriller at 15.12% and Sci-Fi at 8.94%.

Examples include Johnny Depp and Eric Roberts.

This group appears to represent actors who have substantial involvement in darker or genre-based films, but whose careers are much more diverse than the horror specialists in Cluster 3.

Cluster 5: Comedy Specialists

Cluster 5 is the comedy equivalent of the drama and horror specialist groups.

Comedy accounts for approximately 99.32% of the average cluster profile.

Examples include Jay Karales and Alyson Hannigan.

Unlike Cluster 0, which combines comedy with drama and romance, Cluster 5 represents actors whose observed movie appearances are almost entirely concentrated in comedy.

Cluster 6: Thriller Specialists

Cluster 6 is the smallest cluster in the analysis, with 975 actors.

Thriller accounts for approximately 98.71% of the average cluster profile.

Examples include Christopher Backus and Laura Wiggins.

This group appears to contain actors whose careers in the dataset are overwhelmingly concentrated in thriller films.

Cluster 7: Drama, Thriller, and Romance

Cluster 7 is the largest cluster, containing 8,022 actors.

Drama accounts for approximately 39.07% of the average profile, followed by thriller at 10.95% and romance at 7.90%.

Examples include Morgan Freeman and James Franco.

This group appears to represent broad dramatic careers rather than actors who specialize almost completely in one genre.

What Does the Analysis Tell Us?

The most intriguing results came from how the actor clustering differentiated two forms of specialization for actors by genre.

In addition to Cluster 0 (Comedy) and Cluster 5 (Comedy), although both are associated with Comedy, there are clearly different levels of career specialization.

Cluster 5 has no other genre represented; it is almost entirely Comedy. In contrast, Cluster 0 is composed primarily of Comedy and also includes Drama and Romance.

Analogous differences appear in Horror.

Cluster 3 has almost all of its members who have had careers that are almost entirely Horror. However, Cluster 4 has a far greater diversity of horror and thriller and sci-fi and other genres.

The same difference is apparent in Drama.

Cluster 1 is largely made up of actors whose careers are almost exclusively in Drama. While Cluster 7 represents a more mixed group of actors whose careers include both Drama and Thriller and Romance and others.

Thus, these findings suggest that this clustering is distinguishing not just what an actor’s primary or most frequent genre is, but also whether an actor specializes in one genre or is more generalist in his/her career path in related genre areas.

This can potentially provide value to a streaming service in making decisions about whether a particular production requires an actor with a well-established identity in a particular genre vs. an actor with experience across multiple genres.

Validation

To ensure that my input data was accurate and the output of the clustering was valid, I conducted a number of validation tests.

Firstly, I validated that the raw IMDb .csv file loaded properly and that the resulting dataset included 33,609 actors.

Secondly, I tested the L1 Normalization. After normalizing the data, each row representing an actor would sum to roughly 1.0.

Thirdly, I validated that the K-Means Model produced exactly eight distinct groups. This corresponds to the “k” value I chose in the original model specification.

Fourthly, I confirmed that all actors in the normalized dataset received a cluster assignment.

Number of Actors in normalized dataset: 33,609

Number of Actors assigned to Clusters: 33,609

Finally, I examined the cluster assignments for a sample of actors assigned to each cluster and their respective centroid values. For example, seeing actors like Nicolas Cage and Jason Statham clustered together in an Action-Heavy cluster supported my interpretation of Cluster 2. Similarly, the highly concentrated nature of the centroid values for Pure Drama, Pure Comedy, Pure Horror and Pure Thrillers supports my reasoning for how I interpreted the specialist clusters.

I utilized ChatGPT as an AI Assistant throughout my project to assist me in organizing my workflow, explaining various aspects of Clustering, troubleshooting errors in my python scripts, and structuring my written Analysis.

While I relied upon the output of the AI assistant, I independently executed all code utilizing my own notebooks, manually reviewed the actual cluster center(s) and actor(s) for each cluster to verify their accuracy, verified the calculated normalization values themselves, reviewed the Elbow Plot generated by scikit learn to validate that I had chosen a suitable number of clusters (K) for my problem space, and compared my written analyses directly to the true outputs from the Jupyter Notebook.

Limitations

To begin with, the dataset used here only includes movies made since 2000 up to the year 2022. Therefore, the data may not accurately reflect an actor’s full body of work especially if the actor has been working professionally prior to 2000.

Additionally, IMDb categorizes movies into genres. The problem is that many movies contain characteristics of two or more genres. Therefore, one actor appearing in a single movie can be counted towards more than one genre.

Another limitation is that this model counts each time an actor appears in a movie equally (regardless of how large their part) or whether they appear at all. The model does not account for differences in roles (i.e., lead vs. supporting) and/or whether the appearance was a cameo or just a quick shot. Also, this model does not use other factors that are likely important for casting decisions. Examples of these other factors are:

• Box office revenue
• Streaming views
• Awards
• Salary paid
• Audience ratings
• Popularity of the movie
• Critical reception of the movie

It is possible that some countries or film-making regions were represented more frequently in the data; likewise, possibly some production methods or actors were represented more often than others.

K-means models assume that clusters will have well-defined centers based on Euclidean distances. However, real-acting careers tend to blend together in ways that don’t lend themselves to being placed neatly into 8 separate groups.

Therefore, k=8 is simply an arbitrary value chosen for purposes of demonstrating the effectiveness of clustering. Choosing this number did not demonstrate the existence of exactly eight true types of acting careers.

Conclusion

The previous section demonstrated that applying cluster analysis to the actor-by-genre data produced evidence of several identifiable trends in movie careers.

Some actors fall into “genre-specific” clusters where they virtually exclusively act in one genre, such as drama, comedy, horror or thriller.

On the other hand, other actors belong to larger clusters combining a variety of genres such as comedy/drama/romance or sci-fi/thriller/horror.

However, perhaps the most important finding from this research is that even though actors acting primarily in the same general genre area, their individual career paths can be quite different. For example, some are specialists in one genre while others act in a wide range of genres.

Using a technique like this could help content strategists for a streaming service or casting teams make better informed comparisons about how different actors have acted in the past and potentially find suitable candidates for future projects either based on having a narrow specialty or broad experience across genres.

In no case will cluster analysis supplant human intuition when making casting decisions. Instead, it could serve as an additional perspective based on historical genre-based data.

Code and Data

The Python code used to construct the dataset, normalize the actor genre profiles, select k, perform K-Means clustering, validate the results, and generate the figures is available in my GitHub repository:

https://github.com/dmkam1205/INST414-Actor-Genre-Clustering

The raw IMDb movie archive was used to construct the actor-by-genre dataset analyzed in this project.

What Can Movie Genres Tell Us About an Actor’s Career? was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Journeymen are often overlooked in the NBA. After researching in my previous post, I found that they’re often among the most connected players in the NBA across franchises. I still want to dive deeper into what kind of players are classified as journeymen. Is it classified randomly, or by a player’s role or position?

Stakeholders

So, the question now becomes: Which types of NBA players are most likely to become journeymen? Front offices must be able to understand which roles are easy to replace and which ones are worth paying for long term. In the offseason, general managers have to assess their teams and see who should receive a short, cheap contract and who gets a multi-year contract. If a particular player is continuously passed around the league, teams may be able to find a similar player to save money. Executives will have the opportunity to use this information to best understand where they should spend or save their money.

Journeymen are not the worst players in the league. They tend to be in the middle of a roster. About 25% of rotation shooters and backup big men are journeymen compared to 19% of all players, while starters and star players rarely are.

Data Collection

I used Basketball Reference to help collect the data that I needed. I utilized the “Player Per Game” tables from the 2015–16 to 2025–26 NBA seasons. I scraped for fields such as season, player, team, games played, position, points, assists, rebounds, minutes, three-point attempts, steals, and blocks. All of these stats are necessary because they are used to describe how a player plays and their main impact on the court.

I collected my data through web scraping. I used requests to download every season, then used BeautifulSoup to help retrieve that information that I needed and pandas to save it into a CSV file. The finished file had 7595 rows across 1554 individuals.

I had to ensure that the data was fully cleaned. Some players were traded to two or three teams in one season. Therefore, I had to combine the statistics from each team. I had to remove 785 rows to make sure that no duplicates existed. Afterwards, I combined each player’s seasons into career averages, which were weighted by games played, so a short five-game stint would not be calculated into the equation. I decided to only focus on players who played at least 50 games. Players who only played a few games may have averages that are misleading. As a result, we were left with 986 players, with 188 of them being journeymen

Features

Each player was evaluated through seven main statistics, which were points, assists, rebounds, three-point attempts, steals, blocks, and minutes. Points and minutes both show how valuable a player is on offense. Assists help to identify who the best playmakers are. Rebounds and blocks tend to be geared towards big men, while three-point attempts are aimed primarily towards shooters on the perimeter. Steals can also be evaluated to highlight defensive activity.

I decided not to factor in the number of teams played. If I decided to do this, only journeymen would be grouped in the data. I would not have the opportunity to learn what type of player they are. I’ve been able to analyze and note which players tend to move around more.

Euclidean distance was very helpful in finding the similarities between players. Before I used this, I made sure that every stat was standardized. If I didn’t do this, points would easily overpower other categories. This would result in groups mainly being identified based on scoring.

Choosing K Value

K-Means clustering was utilized to help group players together. I initially tested k values between 2 and 10 to see how many groups I should use and examine. I used the elbow method to estimate how far specific players are from the center of their group. With the elbow method, as the k value goes up, the number goes down. I would need to find where adding more groups would no longer be beneficial. The silhouette method was used to see how players fit in their group compared to the next group. The higher the score, the better.

When classifying groups, players were divided based on those who play a lot versus those who play a little. After running the analysis, the chart started to bend around k = 4 to 5. At k = 5, the score only dropped slightly from 0.274 at k = 4 to 0.256. Increasing the k value continued to lower the score, and smaller groups were harder to explain. I noticed that at k = 5, the groups tend to match actual basketball roles.

Clusters

PCA allowed me to squeee the seven stats I analyzed into two so that it can be representable in the chart. The x axis is related to overall production and minutes, white the y axis separates big men from perimeter players. This analyze about 81% of the variation in the data.

Primary Scorers ( 130 Players )

If a team needs to score, these are the main options. They tend to have the most points (17.2), assists (4.6), minutes (30.8), and three-point attempts (5.1) compared to other groups. The individuals in this category include players such as DeMar DeRozan, Russell Westbrook, and Tobias Harris. Even though Nikola Jokic is a center, he is also listed in this category due to how often he passes.

Starting Big Men ( 63 Players )

This group is only focused on centers. They lead the league in rebounds ( 8.3 ) and blocks (1.3) while playing starter minutes. Examples of individuals in this category are Rudy Gobert, Nikola Vucevic, Jonas Valanciunas, and Brook Lopez. Similar to scorers, these groups of individuals also change teams slowly.

Rotation Shooters ( 305 Players )

These players tend to be the “spark plug” for their team. They tend to play real minutes (22.8) and take a lot of threes (3.6), but are usually not the first immediate option. Individuals in this category are Kentavious Caldwell-Pope, Jordan Clarkson, Harrison Barnes, and Tyus Jones. About 1 in 4 journeyment are usually found in this category.

Backup Big Men ( 153 Players)

These players are centers and power forwards who come off the bench to protect the rim in short minutes (17.3). Individuals in this category include players such as Bismack Biyombo, Dwight Powell, Kevon Looney, and Mason Plumlee. This group tends to have the highest journeyman rate at about 25.5%.

Deep Bench ( 335 Players )

This is the largest group, but players in this category play the fewest minutes ( 13.7 ) and have the fewest points ( 4.9 ). Individuals in this category include players such as Aaron Holiday, Georges Niang, Doug McDermott, and Pat Connaughton. Only 12% are journeymen. The average career span for individuals in this group is 3.7 seasons.

Journeymen are mainly found in the middle of the roster. Rotation Shooters (24.9%) and Backup Big Men ( 25.5%) become journeymen more often compared to the 19% average in the league. On the flip side, Primary Scorers and Starting Big Men tend to stay where they are. They tend to change teams at only 0.41 per season. Deep Bench players tend to move around the fastest but tend to leave the league before touching five teams.

It can be very difficult to find stars and quality starting centers. Teams work very hard to maintain those types of players. Shooters and backup bigs are useful, but they are usually in high demand, so it is easy to swap one out for another.

Information like this can help front offices determine how they should spend their money. They should avoid overpaying for shooters and backup big men. There will always be a backup big and shooter available on a short, cheap deal. It is better to spend long-term money on stars and starting centers since they are not common to find.

Validation

To validate the input, I made sure that every season had all 30 teams after cleaning out the traded players’ “2TM” and “3TM” rows. I also checked that there were no missing values or duplicate rows in the data. I used Dennis Schroder to make sure that the 11 teams he played for in my data matched his real career on Basketball Reference. Additionally, I incorporated print statements within the code to check how many players were left after cleaning, which ended up being 986.

To validate the output, I looked at whether the groups actually made sense from a basketball point of view. The Starting Big Men were mostly centers, the Backup Big Men were mostly centers and power forwards, and the scorers and shooters were mostly guards and wings. The players in each group also fit together. For example, Gobert and Vucevic landed in the same group, and so did DeRozan and Westbrook. Moreover, I ran K-Means with 10 different starting points and a set random seed so the results would come out the same every time.

There were a few players who landed in odd places. Giannis was placed with the Starting Big Men because of his rebounds and blocks, and Jokic was placed with the Primary Scorers because of how much he passes. Both of them are unique players who don’t really fit into one role, so I treated them as exceptions.

I utilized AI to help write and debug my clustering code and to help draft parts of this post. Before using anything, I ran the code myself and cross-referenced the results with the data to make sure everything made sense.

Limitations

Since I decided to only use seven basic stats, the groups do not take into account defense or shooting percentage. Two players could play completely different on defense and still end up in the same group. Additionally, I used career averages, so if a player started off on the bench and later became a starter, he is only placed by his overall average.

Since I decided to use only the past 11 seasons, some players may not have had enough time to play for five teams. This may cause bias towards veterans, especially when it comes to the Deep Bench group. The 50-game minimum may also cause bias since players who were only in the league for a short time were left out of our research. My data also does not show why a player changed teams. A trade, a player getting waived, or a player signing with a new team are all counted the same.

Lastly, players were matched by name, so if two players had the same name, their stats would be combined. Choosing k = 5 was also partly my own decision, since the silhouette score was actually higher with fewer groups.

Code and Data

Full code can be found here: https://github.com/thecoderkid351/kinds-of-nba-journeymen

Datasets can be found here: 2025–26 NBA Player Stats: Per Game | Basketball-Reference.com ( Only link for the 25–26 season but the same webpage was used to retrieve previous years through web scraping )

What Kinds of NBA Players Become Journeymen? was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

While certain hereditary health conditions are tied to specific dog breeds, many breeds share overlapping genetic vulnerabilities despite looking completely different. The one question to ask is “How can we cluster different dog breeds into distinct risk categories based on their shared genetic health conditions and median lifespans.”

The primary stakeholders for this analysis are pet insurance underwriters and veterinary health policymakers. By understanding which breeds naturally group together based on health risks, these stakeholders can make data-driven decisions. For pet insurance companies, these clusters can inform how they define risk tiers and price insurance premiums accurately. For policymakers, identifying clusters of vulnerable breeds can guide how they allocate funding for preventative genetic screenings across different breed groups.

The Data:

To answer this question, I used the open Breed Health Score API, which compiles health statistics, median lifespans, and documented hereditary medical conditions across 155 dog breeds.

Relevant Fields:
- slug (String): The unique identifier/name for each breed
- conditions (list of strings): The hereditary medical conditions associated with the breed
- median_lifespan_years (float/int): The average lifespan of the breed.

This data is relevant because it provides both the categorical genetic vulnerabilities and continuous lifespan metrics needed to build a comprehensive risk profile for each breed. I collected this dataset by using Python’s requests library to query the API endpoint, converting the resulting JSON response into a structured pandas DataFrame.

Similarity Metric
For this analysis, I used Euclidean distance as the similarity metric. By calculating the distance between the numerical feature arrays of different breeds, the clustering algorithm (K-means) can effectively group breeds that have identical or highly overlapping arrays of genetic conditions and similar lifespans into the same risk tier.

Choosing ‘k’:

To determine the optimal number of clusters (k), I utilized the Elbow Method. I ran the K-Means algorithm for k values ranging from 2 to 10 and plotted the inertia (within-cluster sum of squares).

I plotted the within-cluster sum of squares (inertia) for k values ranging from 2 to 10. Observing the elbow curve, there is a distinct inflection point at k = 4. After this point, the reduction in inertia begins to flatten out, indicating that adding a 5th cluster would provide diminishing returns. Therefore, I selected 4 distinct risk tiers for my analysis.

Cluster Analysis:

After grouping the data into 4 clusters, distinct risk and lifespan profiles emerged among the breeds:

• Cluster 0 (Highest Lifespan/ Low-Risk): This group represents breeds with the longest average lifespans, averaging 13.82 years. These breeds likely share robust genetics and fewer genetics and fewer severe hereditary conditions. Examples in this cluster include the American Cocker Spaniel and the Bedlington Terrier.
• Cluster 2 (Long-Lived / Moderate-Risk): This group represents breeds that live long lives but may share a slightly higher baseline of genetic conditions compared to Cluster 0. They have an average lifespan of 13.29 years and contain the largest number of breeds within it. Breeds in this group include the Australian Cattle Dog and the Australian Shepherd.
• Cluster 1 (Average Lifespan / Elevated-Risk): This cluster contains 47 breeds with a moderately reduced average lifespan of 11.79 years. These breeds likely share overlapping genetic vulnerabilities that reduce longevity. Examples include the Afghan Hound and the Akita.
• Cluster 3 (Shortest Lifespan / High-Risk): This is the smallest but most critical group of 20 breeds, featuring a drastically lower average lifespan of just 9.48 years. This cluster represents the highest-risk tier, likely heavily predisposed to severe genetic conditions. Breeds here include the Anatolian Shepherd and the Bernese Mountain Dog.

Data Validation:

After querying the Breed Health Score API, I verified that the initial response contained 155 dog breeds. When transforming the categorical “conditions” list into a numerical matrix, I checked the final matrix shape to ensure no breeds were dropped on accident. I also checked for missing values by ensuring that any missing values were replaced with 0 so it wouldn’t affect the algorithm.

I did a manual check on the final clusters by looking at specific breed names and did quick research on them. For example, I verified that famously unhealthy breeds, like the French bulldog, were correctly assigned to high-risk/short-lifespan clusters, while generally healthy breeds, like the poodle, were placed in a low-risk cluster.

Limitations and Bias:

The provided API only has data for 155 recognized purebreds. This completely excludes mixed dog breeds, which make up a massive portion of the pet population. Mixed breeds often benefit from ‘hybrid vigor,’ meaning they generally suffer from fewer hereditary conditions, so excluding them leaves out a massive portion of low-risk dogs.

In addition, by using Euclidean distance, the K-Means algorithm treats every medical condition equally. A severe, life-threatening genetic disease, like heart failure, is weighted the same as a non-fatal issue like skin allergies. This might unfairly push breeds with multiple minor issues into a higher-risk cluster than a breed with one fatal issue.

Lastly, the analysis assumes that median lifespan is heavily dictated by hereditary conditions. However, a dog’s lifespan is massively influenced by external factors not seen in this dataset, such as quality of veterinary care, diet, and the physical size of the dog.

API Reference:

Dog breed health API (free JSON) | Breed Health Score

GitHub Link:

GitHub - Nat821/module_3_inst414

Module 3: Finding the Pack: Clustering Various Dog Breeds by Health Risks was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the modern day, the major numbers thrown around for comparing any single few countries are along the lines of GDP, population and land area. The hole I see in them though are that they don’t account for the interconnectedness of the world today. A country like Democratic Republic of the Congo, known for their role in supplying the majority of the world’s cobalt, doesn’t rank highly on the world’s GDP. However, take them out of the equation and we might have our next crisis in the well-developed countries.

What I want to investigate are how a country’s airports tell the story. How many does this country have; how widely are they spread; how many people and square kilometers does it roughly cover for. Britain and France are both densely populated, highly educated, with long histories of flight. But do they look alike from the sky? Does a more sparsely populated country like Canada resemble its European roots, or more like an island nation?

The Problem

Airports can be counted in two ways. Airports per unit of land area is a question of geographic access: how densely is a country covered? Airports per capita is a question of supply and demand: how many people does each airport have to serve? Thus, I combined them together into my driving question:

Which countries have similar aviation infrastructure profiles, and where does a country’s airport supply differ from what its size and population would suggest?

Clustering is a good fit because it works on many dimensions at once and doesn’t require me to decide in advance what the groups are. Countries that land in a cluster that surprises us, such as a wealthy, dense nation grouped with remote island states, are the findings worth investigating.

The Stakeholder

The person asking this question is an aviation policy analyst at a national transport ministry or a development bank. Before commissioning detailed local studies, which are expensive and focused on individual airports, they need to know which countries are genuinely comparable. A clustering of airport data would help inform on two decisions:

1. Who to benchmark against: how a country structures its airport system (how many airports it sustains, how many have scheduled service) compared with the countries it actually resembles, rather than its neighbors.
2. Where to look more closely at: a country that sits oddly within its cluster, or whose supply differs from what its size and population would suggest, is a candidate for a deeper local study. This doesn’t say where to build. It says where the detailed analysis is worth the cost.

The Data

This analysis combines the three tables from a Kaggle dataset, Global Flights, Airports & Airlines, originally pulled from OurAirports and OpenFlights. I additionally combined this with two country-level files for population and land area both from The World Bank.

• Airports: one row per airfield, with type (large, small, heliport, closed, and so on), coordinates, continent, ISO country code, and whether it has scheduled commercial service. This is the core of the question, since it lets me count airports per country and separate commercial airports from private strips and heliports.
• Routes: one row per airline route, with origin and destination airports, coordinates, and Haversine great-circle distance. This adds how a country is connected: how many routes, how far they go, and how international they are.
• Population and land area by country: raw numbers for population and land area (km²) by country

The airports and routes files label countries with two-letter codes, while the World Bank files use three-letter codes, so I converted between them with a lookup table. I used population and land area for 2023.

How the Data Was Collected

I didn’t scrape anything, instead opting to download the aviation tables as CSVs from the Kaggle dataset and load with pandas. One thing to note though is that OpenFlights, the original source of part of my dataset, route data is no longer maintained. Thus, the routes reflect a snapshot rather than today’s network.

Measuring Similarity

Each country is described by eight numbers, and two countries are “similar” when those numbers are close together. I measured closeness with Euclidean distance, the straight-line distance between two countries in feature space, which is what K-Means uses.

Open airfields per 1,000 km²:

• geographic coverage of all airfields

Open airfields per million people:

• airfields relative to population

Scheduled airports per 1,000 km²:

• coverage by airports with airline service

Scheduled airports per million people:

• airline-served airports relative to population

Share of airfields with scheduled service:

• how commercial the system is

Number of outbound routes (log):

• overall connectivity

International share of routes:

• domestic or outward-facing

Mean route distance (km):

• short hops or long-haul

Because Euclidean distance lets large numbers dominate small ones, I prepared the features in two steps. The four ratios and the route count are heavily skewed, so I took their logarithm. I also clustered only the 159 countries which had at least 1 million people. The tiny territories made the ratios extremely skewed. For example: one airstrip on a 10,000-person island would outrank every large country.

After filtering out closed airfields, heliports, and other, 48,880 open airfields remained, about 200 fewer than the unfiltered total, because they sit in territories without World Bank data.

Choosing k

K-Means requires deciding the number of clusters in advance, so I tested every k from 2 to 10 using the elbow plot.

Obviously, there isn’t an extremely sharp elbow demonstrated by the graph. The plot falls from 1,006 at k=2 to 417 at k=10, and the improvement from each added cluster shrinks somewhat gradually. The curve bends gently somewhere around k=5 to 7. From here, I defered to visual testing.

At k=5, the United States was grouped with Brazil and Russia. At k=6, Brazil and Russia moved into a cluster with India and China, where airports are spread thin across huge territories, while the US stays with Canada, Australia, and the Nordic countries, which have many airfields per person and mostly domestic routes.

What the Clusters Represent

The clusters are rather messy overlapping each other but here is what I would determine each cluster to roughly describe:

Cluster 0: Few airports, nearly all commercial

(Nigeria, Bangladesh, Ethiopia, Japan, Egypt, …): Few airfields per person, but 42% have airline service, the highest of any cluster. These are populous countries where almost every airport exists to carry airline passengers rather than private flyers.

Cluster 1: Thinly connected

(Sudan, Uganda, Iraq, Afghanistan, Ukraine, …): The smallest route networks, nearly all of it international. This cluster is the least connected group in the data. I can’t say from this dataset whether that reflects conflict, income, or incomplete 2014-era route data.

Cluster 2: Dispersed and domestic

(e.g., United States, Canada, Australia, Sweden, Norway, Papua New Guinea, …): By far the most airfields per person (about 40 per million), and the only cluster where under half of routes are international. These countries have large or remote territories and rely on air travel within their borders.

Cluster 3: Huge and thin

(e.g., India, China, Indonesia, Brazil, Russia, …): The fewest airfields per km², with mostly domestic routes. Their airports are spread thin over large areas.

Cluster 4: Dense general aviation and international links

(e.g., Germany, United Kingdom, France, Italy, Mexico, Kenya, …): The densest airfield coverage among large countries, but only 7% of airfields have airline service, so most are small private or general-aviation fields.

Cluster 5: City and island states

(e.g., Singapore, Hong Kong, Bahrain, Mauritius, …): The densest coverage, only international routes, and the longest average routes

Conclusions

The six clusters answer the first half of my question: they are groups of countries with similar aviation infrastructure profiles, defined by how many airfields they have per person and per square kilometer, how commercial those airfields are, and how connected and international their routes are.

The second half, where a country’s airport supply differs from what its size and population would suggest, shows up as surprises in who clusters together. Britain and France do look alike from the sky: both fall in Cluster 4 with Germany, Italy, and Spain. But that cluster also contains Mexico, South Africa, and Kenya, so similar airfield density and route structure do not track income. Canada resembles neither its European peers nor an island state instead clustering with the United States, Australia, and Norway, which suggests that a large, sparsely settled territory shapes a country’s airport system more than its wealth does. And Japan falls with Nigeria and Bangladesh in Cluster 0 because a far larger share of their airfields (30%, 53% and 47%) carry airline service than in the general-aviation-heavy systems of Cluster 4, where it is 3 to 11%

For the analyst in my introduction, these clusters are candidate peer groups. A ministry weighing its airport network could benchmark against Cluster 4’s members rather than its geographic neighbors, and a country that sits at the edge of its cluster is a candidate for a closer local study. These findings are screening tools, not instructions: this analysis cannot see passenger volumes or local demand.

Validation and AI Use

I built the code in part with ClaudeAI, working one step at a time. At each step, I described the goal, Claude wrote a section of code, and I would run it in my own notebook, and I returned the output back for review. I treated the AI’s code as a draft to verify rather than something to trust, and checking it caught several problems.

Some of these problems were difficult to tackle such as joining the World Bank country files to the airports file. One uses three-letter country codes and the other uses two-letter codes, so I eventually used a lookup table to convert between them and printed the 51 rows that didn’t match to confirm they were regional aggregates like “World” and “High income” rather than real countries.

Another issue was Nambia being described with the two-letter code “NA” which pandas would read in as a null value. Settling these issues would have been extremely difficult by hand.

I validated the outputs by checking totals against the raw file, confirming the top-ten rankings looked plausible, reading the actual countries in each cluster, and rerunning K-Means with five different random seeds. Four of five reruns agreed with my main result (72–91% agreement), and one agreed only 46%, so the clusters are reasonably but not perfectly stable. The cluster interpretations in this post are my own reading of those members and medians, and further informed by Claude’s suggestions.

Limitations

During this, I noted several limiations least of which were airport size and traffic, dated routes, missing countries, and a lack of income or economic data

Airport size and traffic: Every airfield counts as one, so a gravel strip and Heathrow are weighted the same here. No passenger numbers or flight frequencies are in the data, so I can’t say how busy any airport is.

Dated routes: OpenFlights is no longer maintained, so route features reflect an older snapshot while population and land area are from 2023. Countries whose networks have changed since then may be misplaced.

Missing countries: Clustering only countries with at least 1 million people removed 56 places, mostly island states. Taiwan and Kosovo have no World Bank data and are absent entirely.

No income or economic data: I can’t test whether wealth explains why certain countries cluster together exactly.

A potential sources of bias in the datasets may emerge from how OurAirports and OpenFlights are maintained where small airfields won’t be listed or possibly overlisted even. In the base sheet there are many more different kinds of aircraft fields including that for helicopters and seaplane bases.

Code

• GitHub - paintedpoint/Fantastic-Countries-and-How-to-Differentiate-Them: Which countries have similar aviation infrastructure profiles, and where does a country's airport supply differ from what its size and population would suggest?
• Global Flights, Airports & Airlines
• World Bank Open Data
• World Bank Open Data

Fantastic Countries and How to Differentiate Them was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

The code used to complete this article’s analysis can be accessed publicly through this link: https://github.com/smclean2/NFL-Passing-Archetypes-Repository

Original Dataset Link

Although the code provided should load the data used throughout this article directly into your local workspace, a copy of the original dataset is preserved on GitHub in case of future issues, and may be publicly accessed through this link: https://github.com/smclean2/NFL-Passing-Archetypes-Repository/blob/main/raw_ngs_receiving_2023_2025.csv

Question and Stakeholders

Through the approach of data clustering, one can gaze into the vast web of NFL pass-catchers and answer the question of which pass-catchers belong to which categories of playstyle, or “archetypes” as this article will refer to such categories as. Such a question is of most direct importance to Defensive Coordinators (DCs) of NFL organizations, who are tasked with developing gameplans prior to each contest. These gameplans are expected to be structured in such a way that their teams athlete’s on the defensive side of the roster are in the best possible position on the field, as is accommodating to their particular strengths and weaknesses as defenders, to be as able as possible to prevent opposing offensive threats from taking control of the contest. Even if certain defenders are recognized as top-of-the-league talents at their particular position group, they can often be exploited by specialized offensive threats on the opposing roster whose archetype is tailored towards attacking their more niche weaknesses as defenders; in other words, it is not always practical to task the “best overall” defenders one-against-one with the opposing offenses “best overall” threats, because if these defenders are unequipped to contend with particular archetypes on opposing offenses, they can become non-factors in the contest rather quickly even if their overall talent is undeniable. By identifying which players on opposing offenses embody particular archetypes via the approach of data clustering, these DCs can assign the most appropriate defenders on their roster to match these players appropriately, and ensure that they cannot be exploited so easily throughout the contest.

The Relevant Data

To be able to cluster NFL pass-catchers into particular archetypes, the supporting data must be descriptive of the pass-catchers themselves, rather than being descriptive of the overarching offensive dynamics of the teams which they play for. Traditional box-score metrics (yards, receptions, touchdowns, etc.) become poor data fields in this context, as they largely reflect how reliant their overarching offense is on high-volume passing or how effective said offense is at progressing near the endzone where a touchdown can be scored reasonably. Because of this, the most ideal supporting data is the NFL’s Next Gen Stats dataset, which is a statistical collection that records advanced metrics on nearly every player from every position on a week-to-week, season-to-season basis. Because pass-catchers are of particular interest to this article, the receiving-specific variant of the Next Gen Stats dataset is best equipped for answering the question of interest. This particular dataset contains fields like average depth of target (how many yards down the football field the targeted receiver on a passing play was from the footballs position at the start of the play, averaged over all targets during that contest), yards after catch above expectation (the difference of the probabilistic calculation for how many yards a receiving player was expected to gain at the exact moment they caught the football on a passing play, vs how many they actually ended up gaining), and cushion (how many yards away the covering defender was aligned from the targeted receiver on a passing play just before the play began). This dataset also contains various other supporting fields, such as player IDs, team abbreviations, player names, number of times targeted, and traditional box-score metrics as outlined earlier. Some additional advanced fields like separation (how many yards away was a receiving player from the nearest defender at the moment of the catch on a passing play) are also included, but are not referenced throughout the clustering process for certain reasons (more on that later).

Collecting The Data

In order to actually collect and filter through the NFL’s Next Gen Stats dataset, I made use of a combination of Nflreadpy and Pandas (two Python-based packages) to load this data into an .ipynb workbook, with Nflreadpy handling the initial data loading whilst Pandas was used for filtering the data further to eliminate undesirable observations (e.g., low sample size pass-catchers) and to restructure the data into a clean, one-observation-per-player structure rather than the week-by-week structure which it embodies by default from the Next Gen Stats dataset. Nflreadpy is a package allowing Python users to work with various kinds of NFL data by loading it into their workspace from an exterior, constantly-upkept repository on GitHub which handles data warehousing (the Next Gen Stats dataset is but one amongst many which they offer), whilst Pandas is simply there for working with data frames in general. Also noteworthy is that Nflreadpy is built atop Polars (an efficiency-emphasizing alternative to Pandas), hence why imported data is converted to Pandas format very early on in the workbook (default import is in Polars format). You can see this in the code snippet below, which shows how this data was loaded into the workspace. It is taking a subset of the many years of recorded data (for this article’s sake, that subset is defined as starting with the opening of the 2023 NFL Season and ending with the conclusion of the 2025 NFL Season). Pandas filtered out the non-essential fields with regards to the data clustering process over the course of the code succeeding this example, and it is also worth noting that a number of other libraries were used to enable certain visualizations (ex: matplotlib). After the data was limited to certain fields (more on what those are later), the sklearn library was utilized to normalize all cell values and work through the k-means algorithm to ultimately produce the clustering output.

import nflreadpy as nfl
import pandas as pd
raw_ngs = nfl.load_nextgen_stats([2023, 2024, 2025], stat_type='receiving')
df_ngs = raw_ngs.to_pandas() if hasattr(raw_ngs, 'to_pandas') else raw_ngs.copy()

Similarity In This Context

In order to execute the clustering process to ultimately group all the pass-catchers in this dataset into their true “archetype”, similarity between each pass-catcher has been defined as the Euclidean distance between them, based upon their standardized values across three features (all values are scaled via z-score transformation to allow the calculation to consider small numbers like YAC+ and large numbers like ADOT with equal weight). These features are ADOT (average depth of target), YAC+ (yards after catch above expectation), and cushion (note: these features were explained in “The Relevant Data” section of this post). Many different prospective feature sets were attempted to be implemented into this model, but ultimately, these three features were found to be the most distinctly independent from each other, and the most independent from team offensive dynamics that might distort the model. For instance, targets per game (how many targets a pass-catcher sees come their way throughout a typical contest) was attempted to be implemented, but was ultimately removed because it cannot be considered independently attributable to a pass-catcher so much as it is attributable to the total volume of pass attempts of that pass-catchers respective team. In another case, separation (mentioned earlier) was also attempted in this feature set, but was ultimately removed because there was a strong, negative linear relationship found between this feature and ADOT (pass-catchers with high ADOT almost always had low separation), meaning that allowing it to remain in the feature set would ultimately mean that two features in a four-feature set would practically be recording the same trait in the pass-catchers, causing a “double-counting” of that particular trait in the model.

Determining A Cluster Count

To arrive at a reasonable number of distinct pass-catching archetypes, or “k clusters”, a range of potentially plausible values of k were fed into the “k-means” algorithm via the sklearn library to produce clusters of pass-catchers. Each of these prospective values of k were evaluated based on how cohesive their respective clustering outputs were in terms of their resulting inertia (sum of the squared Euclidean distances between each pass-catcher and the centroid of their assigned cluster). These evaluations were plotted on a curve such that the x-axis showed each prospective k, and the y-axis showed each subsequent inertia value for such k (the “elbow” method). Specifically, all values from k=2 to k=8 were considered plausible, and evaluated as such. The resulting plot (see below) showed that k=5 was, roughly, the most efficient selection for producing reasonably low inertia in the resulting clustering assignment (less than 150 total inertia) whilst avoiding a k-selection that would increase the computational complexity of the clustering process without adding practical value in terms of significantly higher cluster cohesion (lower inertia).

What These Clusters Mean

After looking at the examples of the players within each of the 5 resulting clusters, and comparing metrics on the centroids of each cluster (how high was each particular cluster in average player ADOT, Cushion, and YAC+), I came up with the following labels for each cluster, and descriptions of what they are likely indicating.

Cluster 0 | Proximity Specialists

Players assigned to this cluster are mainly skilled at working in the middle of the field, close to the passer of the football. They are targeted at significantly low yardage depths, running efficient short-distance routes that keep them close in proximity to the passer at all times. They face very high cushion distance before the play begins (the highest on average among all clusters), as defenders tend to “respect” their ability to separate on routes and instead focus on limiting their yardage rather than forcing incompletions. They tend to generate only a moderate amount of after-catch yardage in the high-traffic chaos that surrounds the area of the passer. They operate as “safety valves” in a sense, for passers to immediately throw to for short gains when deeper options prove infeasible to reach. Examples of such specialists included Seattle Seahawks WR Jaxon Smith-Njigba, and Detroit Lions WR Amon-Ra St. Brown.

Cluster 1 | Physical Threats

Players within this cluster are primarily utilized for their ability to dominate their opposing matchups through sheer physicality. They make positive gains for the offense via intermediate-distance routes (moderate ADOT in this cluster centroid indicates as such), rather than explicitly through deep or short routes. More significantly, they see very minimal cushion in their typical matchup (very low cushion before the play begins is referred to as “press coverage” in NFL terms). These players face such press coverage more frequently than all other archetypes (average cushion in this cluster is the lowest among all clusters), and do not tend to generate much significant after-catch yardage due to the highly physical, close-distance of their matchups. Two instances in this cluster were Los Angeles Rams WR Davante Adams, and Arizona Cardinals TE Trey McBride.

Cluster 2 | True Deep Threats

Players within this cluster are primarily utilized for their ability to get open and make catches deep down the football field, regardless of where they line up before the play. They are targeted at very high yardage depths (highest average ADOT among all clusters), and engage in high-cushion matchups where defenders line up far from them before the snap to avoid being easily separated from on deep routes for massive-yardage catches. They rarely manage to gain significant manage to gain significant yardage after the catch, since the deepest area of the defensive backfield is typically populated by defenders who play the “safety” position, who are specifically tasked with quickly tracking down and tackling receivers making deep catches, if reaching them in time to break up the catch is not possible. Examples within this cluster include Philadelphia Eagles WR DeVonta Smith, and Atlanta Falcons WR Drake London.

Cluster 3 | Boundary Specialists

Players within this cluster are mainly skilled at working on the “boundary” of the field (the leftmost or rightmost sideline of the field of play). They face high amounts of press coverage (second smallest typical cushion yardage among all clusters), and are targeted at a relatively large depth of yardage down the field as is traditional for boundary receivers (second largest ADOT among all clusters). Unlike true deep threats however, boundary receivers operate close enough to the edge of the playing field to be able to consistently sneak past safeties, and generate high amounts of after-catch yardage despite running deeper routes and facing frequent press coverage (YAC+ is second highest among all clusters). Often, these players are considered as the “alpha” pass-catcher of their respective offense. Players given this cluster assignment include Cincinnati Bengals WR Ja’Marr Chase, and Minnesota Vikings WR Justin Jefferson.

Cluster 4 | Elusive Threats

Players assigned to this cluster are utilized for their ability to elusively evade defenders by outrunning them in open space, physically shoving them aside, or simply making them miss. Facing relatively moderate cushion, these receivers are deliberately targeted at very low depths of yardage (dead-last in typical ADOT among all clusters), and generate massive swathes of post-catch yardage (highest YAC+ among all clusters). Offenses intentionally ensure that when these players are targeted, they will have as much open space available to them around the catch point as possible, ensuring they have ample room to evade defenders and generate huge chunks of post-catch yardage. Players assigned to this cluster are grouped with peers like Los Angeles Rams WR Puka Nacua, and San Francisco 49ers WR Deebo Samuel.

Overall Insights

Ultimately, this clustering process concludes the line of inquiry into which NFL pass-catchers belong to which playing archetypes, by identifying five distinct playing archetypes (deep threats, boundary specialists, proximity specialists, after-catch specialists, and press-beating pass-catchers) and assigning each significant pass-catcher in recorded NFL Next Gen Stats data to one of these archetype labels. Based on how closely each pass-catcher resembles the “prototypical” quantitative profile of a particular archetype in terms of their pass targets typical target depth, post-catch opportunity, and original coverage scenario (cushion), they are data-backed as operating in the particular skillsets described above. NFL DCs, with the insight of these particular assignments, may observe the archetypical profile of pass-catchers they are scheduled to play against, and assign matchups accordingly. For true deep threats like Drake London, a DC will want to prioritize matching them up with a defender with experience in coverage without relying on press, and high-performing top speed to avoid being separated from vertically over the course of the play. With physical threats like Davante Adams however, a DC will want to match them with a defender with great physical measurables across the board (acceleration, strength, height and length) to be able to compete with them in highly frequent press coverage and other high-physicality scenarios. DCs would ideally counter boundary specialists like Justin Jefferson with defenders explicitly experienced in boundary coverage, meaning they know how to operate in the left and right “deep thirds” of the field and tackle receivers quickly at the catch point without guaranteed help from safeties. Contrastingly, a DC would find it beneficial to counter an elusive threat like Deebo Samuel with a defender who is explicitly experienced in tracking down and tackling the ballcarrier in the open field, and can match up in moderately distant, cushioned positioning across the pass-catcher before the play whilst not allowing them to break away from them and gain heaps of post-catch yardage. Lastly, DCs would find it ideal to counter a proximity specialist like Amon-Ra St. Brown with a defender who excels in pure short-range coverage, having experience in recognizing and guarding the full range of short-range “safety” routes closely enough to be in position to quickly tackle them after the catch or even force an incompletion, whilst not relying on press coverage to close that distance. Finally, you can see in the figures provided below what these cluster centroids ended up looking like, as well as a full list of each clusters assigned pass-catchers.

Validating Data

To ensure the input data for this analysis was valid, I downloaded a CSV copy of the original dataset at each step of the data pipeline (from initial import to eventual standarization and feeding into the k-means algorithim), and manually viewed the contents via Microsoft Excel. Because the NFL’s Next Gen Stats dataset, by default, comes in a week-by-week format (each player gets one row of data populated per contest they participate in), one particular step of the data transformation I was keeping careful note of as I validated the contents of these CSVs was that the dataset was being properly aggregated into a one-summary-per-player format (the average values of YAC+, ADOT, and cushion, for any average target seen the players way during the recorded 2023–2025 time span). This proved essential, because multiple early iterations of the code were not aggregating this properly, which led to outputs that saw the same player appear multiple times (each week they appeared in got incorrectly treated like a separate data entity). Additionally, when the Next Gen Stats dataset is imported such that it includes at least a full season of data, it will (in its default form) keep an additional row of data for each player (not a weekly entry) which summarizes their data over an entire season, if that player saw some sufficient number of targets. This is another thing that led to issues in early iterations of the code, because including those particular rows as part of the overarching aggregations of all rows for the player led to their total targets being “double-counted” (players had roughly 2 times the number of targets from 2023–2025 that they actually did). I was able to catch this early on, because as I viewed the CSV contents in the later steps of the data transformation, I was also comparing the recorded 2023–2025 target totals for random players in those CSVs side-by-side with actual 2023–2025 sums of those same players as seen on ProFootballReference.com. For instance, Keenan Allen (seen in the “Proximity Specialists” cluster) has 393 targets across all his recorded contents from 2023–2025 according to ProFootballReference.com, so when I saw nearly double that in the CSV, I figured something was off. This exact issue is why you can observe the “df_ngs[‘week’] != 0” in the code snippet below, because this is how the issue was patched in the present version of the code (none of those “yearly summary” rows aforementioned are included in the dataset as it goes down the transformation pipeline, ensured by eliminating the “week 0” label that they were assigned within the CSVs I was observing). Once I got to a point where the CSV files looked like they were being aggregated properly, and the output tables were not showing players multiple times or showing them with unreasonably high target totals (do note, earlier iterations of the output tables did display target totals for this exact purpose of output validation, even though the final ones seen in this article do not for the sake of looking less cluttered), I generally concluded that the input data resulting outputs were reasonable and correct enough to not cause panic. Additionally noteworthy is that Google Gemini’s AI models assisted me with developing a significant portion of the code. Accordingly, to be certain that its outputs were correct, I executed a cell-by-cell re-run of the code each time I introduced new AI-generated lines of code into the workspace, to be certain that these additions were not introducing any bugs that might prevent the code from being compliable. In the codes final state as seen on GitHub, everything runs cell-by-cell without compilation errors, which is my foremost “proof-of-concept” that it’s outputs are functioning as intended, granted that they are supporting the smooth compilation of this article’s overarching outputs (the five distinct clustering assignments, the centroid descriptions, etc.).

# Filter to keep regular season games AND exclude full-season summary rows (week == 0)
df_ngs_reg = df_ngs[(df_ngs['season_type'] == 'REG') & (df_ngs['week'] != 0)].copy()

Analytical Limitations

One of the primary limitations of this analysis lies in the set of features that measure similarity between data points. Because of the qualitative, subjective nature of American Football in general, it is very common that not-immediately-obvious relationships exist between particular prospective features collected on players. This proved true in this case, because as discussed earlier, additional features were attempted to be added to the feature set (such as separation), but ultimately proved to have strong positive or negative relationships with other features, hence why they are missing in this overall analysis. It is not ideal to have two features in a small set which both are “making the same point”, or quantifying the same trait, when the goal is to assign broader categorical labels to pass-catchers, so there were significant limits on which features could be used while remaining indepdendent from each other (co-linearity tests were, at one point. ran on the 3 features which were utlimately used, and they proved that they are indeed independent from each other). No one trait should be “double-weighed” when assigning the broad label of a playing archetype, which sums up this limitation. Additionally, some useful features are missing from this feature set which could have been independent from those 3 which were utilized. For instance, height and weight measurables are not provided in the NFL’s Next Gen Stats dataset, but they could have been particularly insightful for assigning pass-catchers into particular clusters, such as the “Physical Threat” archetype. It is also worth considering that the quality of a pass-catchers quarterback (passer of the football) might introduce some bias into this data. To elaborate, it would seem obvious that not all quarterbacks are of the same quality as each other, and likewise, that the quality of quarterback play can vary widely between one pass-catchers quarterback and another. In the context of this analysis, this could mean that particular pass-catchers might be entirely capable of playing within the profile of a different archetype than the one they are labeled as embodying, but they instead are utilized in a different role as part of an offenses larger effort to “make the most” of the quarterback play that they are dealt with. For instance, it is entirely plausible that a pass-catcher labeled as a “proximity specialist” is capable of operating as a “true deep threat” on a different teams offense, but that they are instead utilized in a more short-to-intermediate role because the quarterback of the offense lacks the adequate physical arm strength to deliver accurate throws deeper down the field. As such, it should be considered that there might be some bias for pass-catchers on teams with relatively strong quarterback play to be dealt a more accurate archetype label, relative to those pass-catchers on teams with relatively weak quarterback play.

Modern NFL Pass-Catching Archetypes: A Clustering Approach was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

THE DATA
I used 4,560 posts from 8 groups in the 20 Newsgroups dataset, which you can find here: https://scikit-learn.org/stable/datasets/real_world.html#the-20-newsgroups-text-dataset. Each row has the text of a post and the newsgroup it was posted in, like sci.space or rec.autos. I clustered on the text only and used the newsgroup names afterward to check the results. It works well for this question because it looks like a site with many forums.

COLLECTING THE DATA
I loaded the data in Google Colab with scikit-learn’s fetch_20newsgroups function. I removed headers, signatures, and quoted replies so the clusters depend on what people actually wrote.

FEATURES AND SIMILARITY
I turned each post into numbers using TF-IDF, which gives more weight to words that are special to a post. Then I used K-Means, which groups posts that are close together by Euclidean distance.

CHOOSING K
I tried k from 2 to 12 and plotted the results in Figure 1. The elbow plot had no clear bend, and the silhouette score peaked at k=4, but every score was below 0.01, so the clusters overlap a lot. Since the moderator already has 8 forums, I picked k=8 so I could compare my clusters to the real ones.

WHAT THE CLUSTERS REPRESENT

Cluster 0: The Catch-All
This cluster has 1,984 posts. Its top words are just, like, don, think, and know, and it holds most of sci.space plus a little of every other group. These seem to be posts without strong topic words. One example is about comet orbital elements, and another is a blank post with no text at all. Posts like these have no strong topic words, which is why they ended up here.

Cluster 1: Armenian and Turkish History
This cluster has 118 posts, and all of them came from talk.politics.mideast. One example talks about the Armenian genocide, and another accuses someone of murders.

Cluster 2: Computer Graphics Help
This cluster has 469 posts, mostly from comp.graphics. One person suggests a viewer called VPIC60, and another asks how to view .tga files made in POVRAY.

Cluster 3: Cars
This cluster has 301 posts, and 293 of them came from rec.autos. One example is about a 92 Z28 with a 350 engine, and another is about how much profit a Saturn dealer makes.

Cluster 4: Heated Political Debate
This cluster has 499 posts. Its top words mix israel and arab with gun and guns, and the posts are split almost evenly between talk.politics.guns and talk.politics.mideast. One example is about Arab and Israeli media, and another is about federal marshals coming for people’s guns.

Cluster 5: Belief and Moral Debate
This cluster has 545 posts, with 269 from soc.religion.christian and 208 from alt.atheism. Its top words include religion, church, believe, and bible. The posts argue about right and wrong more than about specific religious facts. One example argues about whether holding people against their will is wrong, and another gives advice about keeping a just friends relationship.

Cluster 6: Hockey
This cluster has 411 posts, and 410 came from rec.sport.hockey. One example asks whether a team captain has ever been traded, and another complains about a Kings and Flames TV broadcast.

Cluster 7: Christian Faith
This cluster has 233 posts, with 173 from soc.religion.christian and 57 from alt.atheism. One example is a theist replying to a challenge about faith and reason, and another asks whether you can say God just exists.

ANSWER TO THE QUESTION
Hockey, cars, and computer graphics each formed their own clusters, so those forums should stay separate. The two political forums got mixed together in one cluster, and the two religion forums were mixed across two clusters. That suggests the moderator could merge them or at least moderate them the same way, but it could also just mean the posts share an argumentative style. Most of the space posts ended up in the catch-all cluster instead of their own.

HOW I CHECKED MY WORK
I made sure the data loaded and that the numbers in the cluster table added up to the number of posts. I read example posts from every cluster to see if the top words matched, and I compared the clusters to the real newsgroups in Figure 2. That showed me that cluster 0 was a catch-all and included a blank post. I used Claude to help write the code. To check it, I ran each step myself in Colab and read the output for every cluster.

LIMITATIONS
The silhouette scores were very low, so the data doesn’t strongly support k=8. Cluster 0 mixes many different topics, so it doesn’t describe one thing. The data is from the 1990s and covers only 8 English-language groups, so it may not match a modern site. TF-IDF only counts words and doesn’t understand meaning, which makes short or casual posts hard to place. I also didn’t remove blank or very short posts, and they probably ended up in cluster 0. Finally, the mixed clusters might just come from similar writing styles instead of similar topics.

CODE
https://github.com/EsromYohT/Module_3_Assignment

WHICH ONLINE FORUMS BLUR TOGETHER? CLUSTERING 20 NEWSGROUPS was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

With the new NBA season starting, I’ve been paying attention to how teams are performing so far, especially with all the injuries happening around the league.

That got me thinking about how we compare NBA teams. Usually, we look at wins, losses, and standings, but I wanted to see what would happen if I compared teams using their actual performance stats instead.

Because the 2026–2027 season has just started, I used data from the completed 2025–2026 season. I looked at all 30 teams and used Python to group them by offensive rating, defensive rating, and pace.

My main question was: What NBA teams have similar stat profiles, and what can those similarities tell us?

I was curious whether teams I wouldn’t normally compare would end up in the same group and whether any of the results would surprise me.

Who Could Benefit From This Information?

One group that I believe could benefit from this information is the NBA front office and the analysts who work within. They help teams evaluate performance, study opponents, and make decisions about their rosters.

Let's say two teams have similar offensive and defensive stats; an analyst could compare their players and strategies to see what they’re doing differently. This could help a team identify areas for improvement or decide which opponents to study more closely.

Although the clusters themselves wouldn’t tell a front office exactly which players to sign or what changes to make, they could show where to focus the attention.

The Data and How I Collected It

For this project, I had fun with it. I used a dataset containing stats for all 30 NBA teams from the 2025–2026 season. I chose the completed season because the current season is still too early to get a full picture of how teams are performing, but I can surely say a lot of players are dropping like flies this season! So many injuries!

The dataset included a lot of information, but I chose to focus on these three:

Offensive rating — Points a team scores per 100 possessions.

Defensive rating — How many points a team allows per 100 possessions.

Pace — The estimated number of possessions a team plays per game.

I chose these stats because I wanted to compare teams based on their offensive and defensive efficiency, along with how quickly they play.

I downloaded the file as a CSV file from BoxScore Lab and loaded it into Jupyter Notebook.

How I Grouped the Teams

To find similarities between NBA teams, I used a machine learning technique called K-Means clustering. Instead of grouping teams by standings or conference, K-Means groups them by how similar their statistics are.

I used three features: offensive rating, defensive rating, and pace. Before clustering, I standardized the values so that each statistic would have a fair influence on the results. K-Means then used Euclidean distance to measure how close teams were.

One of the biggest decisions was figuring out how many clusters to create.

I tested different numbers of clusters, from 2 through 8, and compared their silhouette scores. A higher silhouette score generally means teams fit better within their assigned groups.

Interestingly, six clusters produced the highest score, approximately 0.374. However, one of those clusters contained only a single team.

I chose five clusters, which had a silhouette score of approximately 0.363. Although the score was slightly lower, every cluster contained at least three teams, making the results more useful for comparing teams.

With five clusters, I could identify meaningful similarities without separating the teams into too many small groups.

My Findings and Discovery

After running K-Means, all 30 NBA teams were divided into five clusters based on their offensive rating, defensive rating, and pace.

What surprised me was that teams didn’t necessarily end up together because of their standings or popularity. Some teams had similar statistical profiles even though I wouldn’t normally think to compare them.

Cluster 0: Struggling on Both Ends

Brooklyn Nets, Milwaukee Bucks, and Sacramento Kings.

These three teams had relatively low offensive efficiency and struggled defensively compared with the other clusters. They also played at a slower pace.

Cluster 1: Strong Offense, Slower Pace

Boston Celtics, Cleveland Cavaliers, Denver Nuggets, Los Angeles Lakers, New York Knicks, and five other teams.

This was the largest cluster, containing 10 teams. These teams generally had strong offensive ratings and played at a slower pace, although their defensive performances varied. And if you watched the NBA Finals, the Knicks definitely dominated the game offensively.

Cluster 2: Fast-Paced and Balanced

Atlanta Hawks, Golden State Warriors, Miami Heat, Minnesota Timberwolves, Orlando Magic, Philadelphia 76ers, and Portland Trail Blazers.

These seven teams played at a faster pace while maintaining more balanced offensive and defensive efficiency.

Cluster 3: Playing Fast, but Struggling

Chicago Bulls, Dallas Mavericks, Indiana Pacers, Memphis Grizzlies, New Orleans Pelicans, Utah Jazz, and Washington Wizards.

This cluster had the fastest average pace, but its offensive and defensive efficiency was relatively weak. It was interesting to see that playing faster didn’t necessarily mean performing better.

Cluster 4: Strong on Both Ends

Detroit Pistons, Oklahoma City Thunder, and San Antonio Spurs.

This group stood out because it had the strongest combination of offensive and defensive efficiency. These teams were effective on both ends of the court while also playing at a fairly quick pace. The Spurs were also a very young team, but wow did they ball out. I loved watching them play. The talent and potential with the starting roster was unreal.

Looking at these groups, one thing became clear: teams can play at similar speeds and still have very different levels of efficiency. Pace alone doesn’t tell the whole story.

What Can NBA Teams Learn From This?

One thing that stood out to me was how teams with similar playing speeds could have different results.

For example, Clusters 2 and 3 both played at a faster pace, but Cluster 2 had better offensive and defensive efficiency. If I were an NBA analyst, I’d want to investigate what those teams were doing differently. Are they making better shot selections? Taking care of the ball? Playing better defense?

Another interesting finding was Detroit, Oklahoma City, and San Antonio ending up in the same cluster. Since they had the strongest combination of offensive and defensive efficiency, teams looking to improve on both ends could use them as benchmarks.

I recommend front-office analysts use these clusters to decide which teams deserve a closer look and which areas of performance to investigate first. They could then examine player statistics, game footage, and other information before making roster or strategy decisions.

The clustering doesn’t explain why teams perform differently, but it gives analysts a starting point for figuring that out.

Validation and AI Use

Before interpreting the clusters, I wanted to make sure my data and results were accurate.

I used Python to confirm the dataset included all 30 NBA teams, had no missing values, and contained no duplicate teams. I also confirmed that K-Means produced five clusters and that every cluster contained at least three teams.

To better understand the results, I reviewed the average offensive rating, defensive rating, and pace for each cluster. I also created a scatterplot to visualize the groups and added team labels to make the chart easier to read.

I used ChatGPT to help write and refine parts of my Python code, understand the clustering process, and interpret the results. However, I ran the analysis myself in Jupyter Notebook and checked the actual outputs rather than relying only on AI-generated explanations.

These checks helped me feel more confident that the findings reflected the data I was working with. It was a fun learning experience.

Limitations of My Analysis

Although I found some interesting similarities between NBA teams, a few limitations remain.

First, I only used three statistics: offensive rating, defensive rating, and pace. Basketball is much more complicated than those three numbers. Injuries, individual player performances, coaching decisions, and team chemistry can all affect how a team performs.

I also used statistics from the entire 2025–26 season, which means the results don’t show how teams improved or struggled at different points throughout the year.

Another limitation is that K-Means groups teams based on the features I selected. Using different statistics or a different number of clusters could produce different results.

Finally, just because two teams ended up in the same cluster doesn’t mean they play the same way. It simply means their statistical profiles were similar based on the measurements I used.

Overall, I see these clusters as a starting point for further analysis rather than a complete picture of each team.

Last Thoughts and GitHub Repository

This project showed me that NBA teams can have more in common than their standings suggest. I enjoyed discovering unexpected similarities while learning how clustering can reveal patterns in basketball.

I uploaded my Python notebook, dataset, results, and clustering visualization to GitHub so others can review my code and see how I completed my analysis.

GitHub Repository: https://github.com/devantet/nba_clustering

30 Teams: Who Plays Like Who in the NBA? was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

This made me curious about whether online shoppers have different browsing patterns and whether these patterns are related to their purchasing decisions.

So in this report I will be focusing on:

Can online shopping sessions be divided into different groups based on browsing behavior, and do these groups have different purchase rates?

This analysis would be useful for an e-commerce marketing manager. Understanding how visitors interact with a shopping website could help a manager decide which parts of the customer experience need improvement. The goal of this article is not to predict whether someone would make a purchase. Instead, I wanted to discover groups based on browsing behavior and compare their purchasing outcomes afterward.

Data Collection

UCI Machine Learning Repository

I used the Online Shoppers Purchasing Intention Dataset from the UCI Machine Learning Repository. The dataset contains 12,330 online shopping sessions with 18 columns, including browsing characteristics and a purchasing output called Revenue.

Some of the variables include:

I chose this dataset because it contains information about what visitors do while shopping online. It also allows me to compare browsing patterns with actual purchasing outcomes.

Data Cleaning

I downloaded the dataset from the UCI Machine Learning Repository as a ZIP file and used Python’s pandas library to read the CSV file.I first checked the dataset’s shape, column names, data types, missing values, and summary statistics.

The dataset contained 12,330 rows and 18 columns. I found no missing values, so I did not need to fill in or remove any missing data. I also found 125 identical rows. However, the dataset does not include unique session ID, so I could not determine whether these represented duplicate entries or separate sessions with identical browsing behavior. Therefore I kept the original records in my main analysis. Next I checked for invalid values, such as negative page counts, negative durations, and bounce rate or exit rate values outside the range of 0 to 1. I did not find any invalid values using these checks.

I also created a histogram to see how many product related pages visitors viewed during their shopping sessions. The graph explain that browsing activity was not evenly distributed. Some sessions included only a small number of product pages, while others involved much more browsing.

Feature Selection and Measuring Similarity

This data contain 18 columns but I selected 6 numerical features for clustering:

• ProductRelated
• ProductRelated_Duration
• Administrative
• Informational
• BounceRates
• ExitRates

I chose these variables because they describe how visitors interact with the website, including how many pages they visit, how much time they spend browsing, and the exit related characteristics of the pages they visit. I did not include Revenue or PageValues because it is connected to purchaing transactions and would influence the cluster group.

Before finding K values, I used a log1p transformation on the page count and duration features to reduce the influence of unusually large values. I also used StandardScaler to standardize each features so that variables with larger ranges would not influence the clustering result.

For this analysis, I used Euclidean distance to measure similarity. This means that sessions with smaller distances between their standardized feature values are considered more similar.

Choosing the Number of Clusters

To choose the number of clusters, k, I tested values of k from 2 to 8 and used two methods to evaluate the results: the Elbow Method and Silhouette Score.

The Elbow Method looks at how the total within cluster squared distance, called inertia, changes when the number of clusters increases. I used an elbow plot to test whether adding more clusters continued to improve the model substantially.

I also used Silhouette Score to evaluate how well the clusters were separated. A higher score generally indicates that observations are closer to their own clusters and farther from other clusters.

The highest Silhouette Score was at k = 2 (0.5968), suggesting that two clusters had the strongest separation according to this metric. However, I decided to use k = 3 for my main analysis because it allowed me to identify more detailed browsing behavior patterns. Although k = 3 had a lower Silhouette Score, it provided three groups that were easier to interpret in terms of different levels of browsing engagement. Therefore, I chose k = 3 to balance clustering performance with practical interpretation.

Interpreting K-Means Results

After selecting k = 3, I applied K-Means clustering to all 12,330 shopping sessions using the six standardized features. Which divded the sessions into three clusters.

Cluster 0: Highly Engaged Visitors

Contained 4,224 sessions. Vistors in this group explored more product-related pages and spent more time browsing products than the other groups. With a median number of product pages visited 44, and product browsing duration 1,626 seconds. This cluster also has the highest purchase rate of 24.22%.

This group is considered as highly engaged shopping sessions because the visitors appeared to spend more time exploring products.Two example sessions from this cluster were:

• Session at row index 26: 32 product pages visited and approximately 1,135 seconds of product browsing.
• Session at row index 29: 45 product pages visited and approximately 1,583 seconds of product browsing.

Cluster 1: Casual Browsers

Contained the largest session, 7,202. The median number of product related pages visited was 12, and the median browsing duration was 375 seconds. With a purchase rate of 12.23%.

Compared with Cluster 0, these sessions generally involved less product browsing and shorter browsing durations. Two examples were:

• Session at row index 1: 2 product pages visited and 64 seconds of product browsing.
• Session at row index 4: 10 product pages visited and 627.5 seconds of product browsing.

Cluster 2: Brief Visitors

Contained 904 sessions and had the lowest purchase rate at only 0.44%. The median number of product related pages visited was just 1, and the median product browsing duration was 0 seconds. This cluster also had higher median bounce rate and exit rate metrics than the other two groups. Two examples were:

• Session at row index 0: 1 product page visited, 0 seconds of product browsing, and no purchase.
• Session at row index 2: 1 product page visited, 0 seconds of product browsing, and no purchase.

Analysis

One of the most interesting findings was the difference in purchase rates across the three clusters.

Cluster 0’s purchase rate was almost twice of Cluster 1, while Cluster 2 had a much lower purchase rate than either group. These results suggest an association between browsing engagement and purchasing outcomes. However, this analysis does not conclude that visiting more pages or spending more time on the website causes someone to make a purchase. Other factors, including visitor intentions, product availability, prices, and traffic sources, may also influence purchasing decisions.

Based on my analysis, I would conclude that online shopping sessions can be grouped based on browsing behavior. I was able to identify three groups with different browsing patterns and purchase rates. The results showed that highly engaged sessions had the highest purchase rate, moderate engagement sessions had a lower purchase rate, and quick-exit sessions had the lowest purchase rate.

For an e-commerce marketing manager, these findings could help identify areas for further investigation. For example, the manager could look into why some visitors leave after viewing only a few products or explore ways to make product pages easier to navigate. Since the moderate engagement group contains the largest number of sessions, understanding the browsing behavior of this group could also be an important priority.

Validation

To make sure my analysis was reasonable, I first checked for missing values and invalid numerical values before clustering. The dataset had no missing values, and the selected numerical features were suitable. Second, I used both the Elbow Method and Silhouette Score to test different choices of k instead of selecting the number of clusters completely at random.

I also used ChatGPT to help me understand Python functions, explain clustering concepts, and organize my analysis. To check the AI assisted explanations and results, I ran the code in Jupyter Notebooks and compared the findings with the actual outputs in my notebook.

Limitations

There are several limitations to this project. First, the dataset was donated in 2018, so the browsing patterns may not fully reflect current online shopping behavior. Second, I only used six numerical features for clustering. Other information, such as visitor type, month, traffic source, browser, and region, might also provide additional context. Third, I kept the 125 identical rows because the dataset did not contain unique session id, but they could be representing separate sessions.

Lastly, the clustering only identifies patterns and associations in the data. Even though sessions with higher browsing engagement had higher purchase rates, this does not prove that browsing more pages causes visitors to make purchases. Other factors could also influence their decisions.

Conclusion

Through this project, I learned how K-Means clustering can be used to find patterns in online shopping behavior. I also learned that choosing the number of clusters is not always straightforward. Even though k = 2 had the highest Silhouette Score, I chose k = 3 because it allowed me to explore more detailed differences in browsing behavior.

Overall, this project showed me how clustering can be a useful tool for understanding patterns in data and how those patterns can be connected to real business decisions.

GitHub Code

GitHub - yaru2027-gif/online-shopping-clustering: K-Means clustering analysis of online shopping behavior using the UCI Online Shoppers Purchasing Intention Dataset.

Exploring Online Shopping Behavior Using K-Means Clustering was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

But do health care agencies with high ratings have similar quality metrics?

The stakeholder for this research question is a family member choosing a home health care agency.

The ideal dataset to answer this question would have current data on healthcare agencies from every state. Historical data would also be helpful to view past trends. It would include an overall quality rating for each healthcare agency, as well as a variety of other performance indicators, like preventive care, quality of life, and access to care.

Data Collection

To answer this question, I used the “Home Health Care Agency Ratings” CSV dataset from Kaggle, which has information on all Home Health Agencies registered with Medicare. It includes each agency’s address, phone number, type of ownership and quality measure ratings. For this analysis, I mainly focused on the quality measure ratings.

This was a large dataset, with 11,366 columns and 64 rows. Twenty-one of the columns encompassed different measures of quality at the agency. In order to decide which columns I wanted to analyze, I first checked the percentage of missing data in each column.

I decided to use columns with less than 25% missing data in order to have enough data coverage to inform my analysis. I settled on these nine columns:

1. How often the home health team began their patients’ care in a timely manner
2. How often the home health team taught patients (or their family caregivers) about their drugs
3. How often the home health team checked patients for depression
4. How often the home health team made sure that their patients have received a pneumococcal vaccine (pneumonia shot).
5. How often physician-recommended actions to address medication issues were completely timely
6. How often the home health team checked patients’ risk of falling
7. How often the home health team determined whether patients received a flu shot for the current flu season
8. How often patients got better at bathing
9. How often patients got better at walking or moving around

These columns could be grouped into several quality metrics: timely care (1, 5), medication education (2), preventative care(4, 7), safety assessments (3, 6), and patient improvement (8,9). For this analysis, I wanted to focus on metrics that take into account patient outcomes and care.

I excluded two columns that had less than 25% missing data: “How much Medicare spends on an episode of care at this agency, compared to Medicare spending across all agencies nationally” and “How often patients developed new or worsened pressure ulcers.” The former column didn’t measure agency quality through patient care and the later was removed for consistency among the metrics, since a higher rate would show an adverse outcome.

I intentionally did not include Quality of Patient Care Star Rating when clustering because I wanted to compare the clustered groups with their overall ratings.

Creating Clusters

I used K-Means to cluster the data because it allowed me to identify similarities across the nine performance measures. In order to clean the data, I filled the missing values with the median of each numeric column using the scikit-learn library.

I then used Euclidean distance on the scaled columns to measure similarity. This would measure how far apart agencies are across the nine performance metrics and group agencies with similar profiles.

In order to choose the value of k, I used the elbow method as well as measured the silhouette scores for k=2 through k=8.

I selected k=2 because it had the highest silhouette score and the inertia reduced more sharply for bigger clusters.

Results

Below is a table of the mean for each quality measure for each cluster:

Overall, Cluster 1 had a higher average across each cluster quality measure than Cluster 0. The biggest difference was between receiving a pneumonia shot, receiving a flu shot, and bathing improvement. There was also a large difference in timely medication response and movement improvement.
Screening for the risk of falling had the smallest difference between the two clusters.

In order to compare the clusters’ quality measures with the Quality of Patient Care Star Rating, I also calculated the median and mean star rating:

The mean and median star ratings for each cluster align with the means for each quality measure.

Cluster 0 had lower averages across all nine quality measures, especially in vaccination and functional improvement, and a lower mean and median star rating.

Conversely, Cluster 1 had higher averages across the board. Agencies in Cluster 1 had especially high averages in timely care, medication education, screening, preventive care, and functional improvement compared to Cluster 0.

Comparing Cluster Profiles

In order to visualize the nine quality measures in two dimensions, I used a Principal Component Analysis scatter plot:

Although the scatterplot only captures about half of the total variance in the nine quality measures, it’s clear that Cluster 0 has a lot more variability than Cluster 1. This is supported by the below examples of Cluster 0 and Cluster 1:

Cluster 0 Examples:

This example shows the variation in quality measures agencies have in Cluster 0. For instance, Allied Home Health Corp has a 98.7% for the pneumonia shot measure and just 2.6% for timely medication actions.
This suggests that agencies in Cluster 0 have different strengths and weaknesses, instead of performing poorly in every category.

As seen in the examples above, agencies in this category can still have a star rating of 5, even if they have some very low quality measure values.

Cluster 1 Examples:

The agencies in the Cluster 1 example have an overall higher rating in every category, and less variation between each quality measure.
These examples show that a high star rating of an agency does not necessarily mean overall high quality measures.

Conclusion

My analysis suggests that agencies with a higher overall star rating also tend to have higher average scores for preventive care (vaccines), functional improvement (bathing and walking improvement), and timely care (medication actions).

This can be seen in the differences between pneumonia vaccination, flu shot assessment, and improvement in bathing and walking. Timely care and medication-related actions also had a lesser impact.

AI Assitance and Validation

To make sure the data was accurate, I checked the example agencies with the CSV file to make sure that they matched.

I used AI assistance in several parts of this assignment, especially in cleaning the data and to graph the clusters on a scatter plot. I also used AI to debug my code.

Limitations

The main limitation in this analysis is missing data. For example, not every agency has a rating in each cluster, so not every agency is captured. Cluster 1 is also a lot larger than Cluster 0, so these groups are probably not natural in the data.

GitHub

The notebook and saved data are available at https://github.com/isabellacarrero-baptista/inst414_module_3.git

What makes a high quality home health care agency? was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.

A growing body of research in mental health interventions supports the use of music therapy as a means to improve diverse populations’ wellbeing. For individual patients with cancer and heart conditions, music recordings may be effective for reducing anxiety, while children and adults in hospital settings can benefit from general therapeutic treatment (Aldridge, 1994). More specifically, and for the application of this case study, group music therapy sessions are another form of treatment that can be applied across many different medical settings for different conditions (Aldridge, 1994). That being said, the discussion of group music therapy yields questions about broader applications to individual patients. Group music therapy sessions that use “receptive taped music” (Aldridge, 1994, p. 204) may rely on one genre or playlist that is not universally effective as a therapeutic treatment for all clients in the group.

Measures

Therefore, determining distinct genre profiles among music listeners may lend itself well to the broader study of music therapy as a mental health intervention by allowing us to see which music genres emerge among different clients.

My research questions map onto this accordingly: (1) What genre profiles exist among people who listen to music and (2) Which genres define each profile?

These will help stakeholders such as music therapists make decisions about (1) how to design group music therapy sessions in terms of grouping clients with similar listening habits together (i.e., personalized music therapy in a group setting) and (2) which genres to build intake playlists around.

In order to find the answers to making these decisions, clustering will illustrate the potential groups of clients with similar listening behaviors (i.e., each cluster represents a candidate group of clients). The number of profiles (k-value), size of each profile, and what makes them distinct will demonstrate this. Additionally, the defining genres of each cluster (i.e., the genres its members favor the most in comparison to the average respondent) will indicate which genres the intake playlists should be based upon.

Music & Mental Health Dataset

University of Washington’s Catherine Rasgaitis (2022) published the “Music & Mental Health Survey Results” dataset on Kaggle, which was used as the dataset for this study. The original dataset contained 736 respondents and 33 columns collected via Google Forms in 2022. The fields used for clustering were the listening frequencies for 16 genres. To describe the clusters, the fields used were as follows: age, hours per day, anxiety/depression/insomnia/OCD, and whether music improves their mental health. Together, these fields used for clustering will inform the research questions by linking the genres respondents listen to with the self-reported measures of their mental health, which are what music therapists need to build effective group music therapy sessions.

Data Collection

To begin the data collection process, I downloaded the “Music & Mental Health Survey Results” CSV from Kaggle. In terms of libraries, I used pandas for loading, cleaning, and tables; numpy for distances; matplotlib for figures; and scikit-learn for k-Means. In my cleaning process, I converted each frequency label to a number (Never = 0, rarely = 1, sometimes = 2, very frequently = 3). All respondents answered every genre question, so none were removed.

Similarity Measures

To measure similarity between data points, I used Euclidean distance as used by k-Means. Respondents are represented by 16 features (i.e., one for each genre).

My first attempt with k = 2 clustered the raw 0–3 frequencies and did not answer my question. The two clusters were almost equal in size but differed in how much people listened to each genre rather than what genres they listened to. Euclidean distance on raw frequencies treated heavy listeners and light listeners as very different, even in cases where their taste had similar shapes. Ultimately, that split would not be useful for music therapists to build genre-based client groups or playlists.

In order to measure music taste instead, I subtracted each respondent’s average frequency from their 16 answers so that a positive value would now mean a respondent favors that genre more than their usual listening, while a negative value would mean that they favor it less. These centered values are represented as each respondent’s genre mix, which are the features used for the clustering below.

Selecting k

I used the elbow method and performance in a downstream task to choose my number of clusters.

For the former, I ran k-Means for each value of k from 2–20 then recorded the inertia for each. A lower inertia corresponded to respondents being closer to their cluster’s center; since inertia decreases as k increases, I sought to find the point where adding an additional cluster stopped making a difference.

Figure 1 illustrates this relationship: The inertia declines gradually. I calculated the percent decrease in inertia from adding the clusters in order to read the elbow more precisely. When I moved from 5 to 6 clusters, it improved the inertia, but the improvement dropped at k = 7, so the curve leveled off after k = 6. In conclusion, adding more clusters after k = 6 would no longer have a meaningful effect on cohesion.

In application, I needed the clusters to work as therapy groups, so I checked the smallest cluster size at each value of k. At k = 6, the smallest cluster had 104 respondents; larger k-values generated smaller groups, so I selected k = 6 because the elbow leveled off while each group was still large enough to support a group music therapy program.

Cluster Descriptions

To describe the clusters, I first examined the average listening frequency of each genre in order (see Figure 2) to establish a baseline for each cluster’s defining genres, which were the genres that the cluster’s members favored more than the average respondent did. This was measured as the “lift” of their genre mix compared to the overall average. Additionally, I identified each cluster’s clustroid as the most representative member of that group.

Cluster 0 was defined by classical (+1.35), jazz (+0.51), and gospel (+0.47) music. Respondent r384, the clustroid, listed classical as their favorite genre and listened to it very frequently.

Cluster 1 had 115 respondents and was defined by country (+1.12), folk (+1.03), and rock (+0.51). Accordingly, the clustroid’s favorite genre was rock.

Cluster 2 had a larger sample size of 120 respondents, and the clustroid’s favorite genre was EDM, which they listened to very frequently along with rock. The cluster’s defining genres were then EDM (+1.12), video game music (+0.92), and lofi (+0.40).

Cluster 3, the hip hop (+0.97), rap (+0.96), and R&B (+0.62) cluster, contained 143 respondents. Alternatively, the most representative respondent’s favorite genre was pop, but they listened to hip hop, pop, and R&B very frequently.

Cluster 4 was defined by metal (+1.56), rock (+0.74), and video game music (+0.26), which represents a bit of overlap with clusters 1 and 2. The clustroid listed metal as their favorite genre, which they listened to very frequently.

Cluster 5 was heavily marked by kpop (+1.54), R&B (+0.72), and hip hop (+0.45). This cluster’s clustroid listed pop as their favorite genre, although they listened to that and EDM very frequently.

Although the clusters were all distinct and had different median ages and favorite genres, they are not completely separate from one another. A handful of respondents listened to certain genres very frequently that were outside of their cluster’s defining genres. This may suggest that listener tastes have overlap between clusters. In application, music therapists can use these tastes as a guide to listening tendencies rather than adhere to them as strict categories.

Results

Out of all 736 respondents, the data analysis found 6 different genre profiles that may inform music therapists’ decisions in building group sessions and creating playlists based on these genres.

For the former decision, each cluster represents a group of music therapy patients with similar listening behaviors. Each of the 6 groups contain 104–146 respondents, which means that each profile is large enough to support its own group sessions. Additionally, to describe each cluster, I used fields that were not used to build the clusters themselves (i.e., age, daily listening hours, self-reported anxiety, depression, insomnia, OCD, and the share of respondents who said music improves their mental health) (see Table 2).

The second decision was informed by each cluster’s defining genres because they indicated how playlists may be tailored to specific listener profiles. For instance, a new patient whose listening habits align with the Electronic & Focus profile might begin their session with a playlist containing genres such as EDM or Lofi. This process of identifying clients’ listening profiles and matching them to a number of genres allows music therapists to create personalized playlists that can still map onto an entire group before the session begins.

The different profiles suggest ways for music therapists to make their group therapy sessions more adaptive to the individual patient. Electronic & Focus reported the highest average anxiety, depression, and OCD scores of any cluster, which would make it a strong choice for a sort of pilot group. Metal & Heavy Rock had the highest average insomnia score, as well as the lowest score of respondents saying it improved their mental health; music therapists might use these findings by acknowledging that not every type of genre is suitable for therapeutic care despite listeners’ frequency of listening to a certain genre. The profile with the youngest group of listeners (median = 18 years old) was Kpop & Pop, which may tell music therapists that these genres could work well as therapeutic care for college students or young adults. Classical & Jazz had the lowest average anxiety and depression scores but was listened to the least by respondents. This may demonstrate that despite what listeners prefer to listen to in recreational or everyday activities, certain genres they may overlook could be effective in helping relieve the burdens of mental health conditions.

That being said, none of these findings indicate causal relationships and therefore should only be used as starting points for music therapists to consider when planning group therapy sessions; no single genre causes or reduces symptoms of any mental health condition.

Data Validation

For my data validation, I ran 5 checks on my code. My first check sought to validate the input data by confirming that all 16 genre columns were in the code and each frequency label was converted and mapped onto a number 0–3. All respondents in the original data sample answered every genre question, so none were dropped from the dataset. I then ran a toy test using the following 6 points: (0, 0), (1, 2), (2, 1), (4, 1), (5, 0), and (5, 3). k-Means grouped the first 3 points and the next 3 points into separate clusters, with centroids at (1, 1) and ~(4.7, 1.3). My bookkeeping test confirmed that all respondents were assigned to one cluster, all cluster sizes summed to the total number of respondents, and each clustroid was a member of its own cluster. Next, I re-ran the clustering with 3 random seeds and then compared the results with my original clustering; across all 3 runs, the same groups were found, which means that the clusters are stable and not arbitrary. My final check measured each cluster’s cohesion through its radius, diameter, average distance between members, and density (see Table 3). Each cluster had similar radii and average distances.

Limitations

While this analysis identified potential genre profiles for music therapists to reference as a framework for building group therapy sessions, several limitations affect the overall application of the case study.

First and foremost, the sample may not accurately represent the general population of music therapy patients. As aforementioned in the introduction, multiple populations can benefit from a number of different music therapy techniques, such as cardiac patients, cancer patients, and both children and adults (Aldridge, 1994). The majority of respondents used in this dataset were in their late teens — early 20s, which is likely due to Rasgaitis’ (2022) survey being distributed on social media sites that are used more prevalently within that age group. Therefore, the external validity of the study was reduced as the findings may not map onto clients such as older hospital patients.

Additionally, the mental health measures were self-reported by respondents, not clinically assigned, which means that these measures were not professionally diagnosed and cannot indicate cause and effect. In short, the findings do not demonstrate whether certain genres a group listens to directly helps or hinders mental health.

In terms of the technical analysis, the clusters overlap, which means that each group contains some respondents with different music tastes. When considered with the elbow curve, this may demonstrate that listener tastes formed tendencies, not distinctly separated groups. This also shows that choosing k = 6 involved a certain degree of judgment.

The analysis also showed that centering each respondent’s answers left out some information regarding how much they listen to music overall. For instance, a respondent who listened to music far less than another much more active listener could be placed in the same cluster if they had the same shape for their tastes. This would then affect how a therapist would create their groups, even if the music therapist might normally separate the two listeners without this information.

Github Link

https://github.com/kjmhawes/INST414-Clustering-Analysis-Module-3.git

References

Aldridge, D. (1994). An overview of music therapy research. Complementary Therapies in Medicine, 2(4), 204–216. https://doi.org/10.1016/0965-2299(94)90021-3

Kmeans. Scikit-learn. https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans

Rasgaitis, C. (2022). Music & mental health survey results. Kaggle. https://www.kaggle.com/datasets/catherinerasgaitis/mxmh-survey-results

Clustering Music Listener Profiles to Build Group Music Therapy Sessions was originally published in INST414: Data Science Techniques on Medium, where people are continuing the conversation by highlighting and responding to this story.