Frameworks and templates should not simply be looked to for pre-baked, ready-to-go answers.

Their value lies in their ability to inspire and/or stimulate us:

• to remember what we have forgotten
• to make new connections and apply what we know towards a vision;
• to seek out ways to fill gaps in our knowledge

I recently was speaking with a designer and a researcher who were struggling to convince their team to engage in extensive, in-depth qualitative interviews. The cross-functional team was hesitant because of the time investment. Part of our chat went something like the following.

I asked — “why do you think you should dive into a lot of qualitative interviews?”

They responded — “because we need to do some needfinding to understand the users, it’s the beginning of the project…”

I asked — “you believe you don’t understand the users enough? Do you have a sense for what problems you might be solving for? Are they the right problems? Are there no explicit problems to solve?

Silence. For a bit. Then we had a great conversation about what the team knew and didn’t know, blindspots, the reason the new project had been created, what success meant, etc.

The duo was not completely wrong in their instinct to jump into interviews. At the same time, our back and forth conversation served as a good reminder of something important. Good research methods (and their correct execution) are essential to problem solving, but trying to implement them without first thinking critically about the challenge at hand and how best to build towards relevant findings can result in a reluctant team, or worse, a waste of time.

It is important to think critically about why we research at all and the phase of the product we are researching.

Sometimes thinking critically is not as easy as it sounds when dealing with the pressures of a fast-paced environment where “we don’t have time for deep research” and “we need to ship yesterday” mindsets prevail. I have several thoughts on the deep research vs. delivery speed debate, but I will save that for another post. Here, I share some frameworks that can help support the conversations and critical thinking that guide decisions about research approaches.

The frameworks present a variety of approaches (sometimes we forget what is out there). More importantly, they nudge one to consider a research approach within the context of product phases, needs, goals and timeframes. Basic usage of the frameworks could be to simply use them as reminders of when a given approach might be useful. Better usage would be to leverage them to jump start critical questions like “What do I want to know?”, “How will I know when I’ve learned it?”, “What decisions or actions will this knowledge enable?”. With these questions in hand, one can again look to the frameworks to reflect on which approaches, or which elements of them, will best lead to achieving the goal at hand. I created these frameworks some time ago and have used different versions of them over time.

This first framework presents research approaches across a fidelity spectrum while also matching these approaches to a given product’s maturity. Product maturity (vertical axis, left) is relatively straightforward — as a product / service spends more time in the market and collects more users, its goals, and thus its research needs, often change as well. Fidelity (horizontal axis, top) refers to the technical implementation complexity of a given research approach. More than just a synonym for “execution time”, implementation complexity conveys the point that some research approaches require more advanced artifacts, sophisticated coding / sensing tools, or quantitative modeling, all of which stretch complexity. I don’t mean to imply that “Card-sorting, Co-creation” always take less time than “Resonance tests with Max-diff, Conjoint analysis”, but while the former can be implemented with tools in hand, the latter more likely requires coding or purchasing dynamic surveys.

The resulting framework offers a matrix that stimulates reflection about where one is in the product cycle, one’s technical capacity and, above all, what one’s research goals are and how they might be reached.

This second framework puts a twist on the first by explicitly introducing the element of time constraints. A former design professor of mine used to assign projects with an initial timeframe but before the end of the class he would cut this timeframe in half. It was a forcing mechanism that pushed me and other students to think about what was essential to the project and how to go about obtaining optimal impact and learning. It was a great exercise. I used it as inspiration for this second framework which presents a number of approaches to address two research goals while considering time constraints and product maturity. While it can serve as a “quick reference menu” in a time crunch, I mainly appreciate how it encourages a reflection of how different approaches can expand or condense into others to reach a given goal when time is a factor.

The above content is not earth-shattering, but I have found it extremely useful when leading teams or in my own work. In my experience, it engages multidisciplinary teams and enhances the conceptualization, planning, and communication of design research. The reality is that in different contexts, for different reasons, a clean line from goals/questions to approach does not always exist. Instead, something of an iterative dance takes hold — from goals to approach and from approach to implied goals — and it moves the researchers (and their teams) towards crystalizing what it is they really want to accomplish and how. In the case of the duo mentioned above, it stimulated rich thinking around their main questions and goals and reminded them of a simple but helpful point: deep thinking about key research goals and questions should come first and shape one’s chosen research approach, but it never hurts to have some approaches in mind or on the table.

Frameworks for Design Research in Diverse Scenarios was originally published in 3rd corner on Medium, where people are continuing the conversation by highlighting and responding to this story.

In search of clusters and patterns within team data from the world's greatest tournament using unsupervised machine learning

As a design researcher with experience applying qualitative and quantitative approaches, I like blending the worlds of design and data science where I can. Over the last six months I decided to return to studying Python. I am not a refined nor expert programmer by any means, but I enjoy learning Python to better collaborate with data scientists and to perform my own personal and professional analyses. In this article I share the thinking and coding process I used to run an unsupervised clustering algorithm on teams from the 2018 World Cup.

Football (soccer, in the USA where I grew up) has always been a part of my life — I played organized soccer from kindergarten until graduating from college. Every four years the World Cup produces my favorite 30 days of sport full of interesting data. So when I found a dataset for the 2018 World Cup, I was eager to dive in and test my evolution as a Python beginner. At a high level, I wanted to perform a clustering analysis using average team stats as variables. What would I find? Perhaps teams would cluster by “style of play”, by region, or something else. I used Google Colab, a Google Research product that is a hosted Jupyter notebook service allowing anybody to write and execute arbitrary Python code through a browser. It’s dope.

The dataset I used was from a cool person called DH. I downloaded the csv and then uploaded it into Colab to get started. Improvement Note: arguably a more efficient way to do this would be to insert the url of the file directly into my Python script with a df =pd.read_csv(url).

Outline of Objective

1. explore data; clean and preprocess features needed for clustering
2. run k-means clustering algorithm to see what clusters emerge (note to reader: jump here if you just want to see clustering output + analysis)
3. create 2x2 visualizations to see how different teams/clusters compare

(1) Exploring, Cleaning, Preprocessing

🗃 (1A) Importing Data, Initial Looks and Adjustments

First, I import the data.

#import the necessary libraries and the data
import numpy as np
import seaborn as sns
from google.colab import files
import io
import pandas as pd
data = files.upload()
import warnings
warnings.filterwarnings(‘ignore’)

The above prompts me to upload the csv file and I do so with the line of code displayed below. This pulls the data in as a pandas dataframe (think of a dataframe as an excel table).

df=pd.read_csv(“world_cup_2018_stats.csv”)

To look at the first and last rows of the dataframe(df) I use the .head() and .tail() functions.

What I see:

• each game is represented by 2 rows of stats — one row for each team that played (thus, there are 128 rows for the 64 World Cup games)
• “Group” column shows at what tournament stage a given game occurred

What I do:

• I like having the option to investigate aggregate differences between tournament stages, so I create a simple dataframe which shows each team and the most advanced tournament stage that team reached
• I filter the original dataframe based on stage (“Group” column) to see which teams got to which stages (Group, Round of 16, Quarterfinals, Semifinals/Final).
• I create a new dataframe matching each team to its latest stage

Here is the code I used for filtering:

dfrsixteen = df.loc[df[‘Group’] == “Round of 16”,:]
dfquarter = df.loc[df[‘Group’] == “Quarter-finals”,:]
dfsemi = df.loc[df[‘Group’] == “Semi-finals”,:]

I then use what this filtering code produced to create a new dataframe, dfround, for potential use later on.

👾 (1B) Exploring the Main Dataset

I use df.info() to see what data types I have. I do this because k-means clustering only works with numeric data and I want to see if I need to convert any data types.

The variables I will want to use are integers. "Pen Shootouts" are floats and "Group", "Team", "WDL" are objects. I will remove these for clustering.

Then I use df.describe().transpose() to see a summary of the distribution of all numeric data.

In general, I see expressive standard deviations, given each variable's max and min, as well as relevant quartile values. For many variables there is a nice difference between min and max values — this is good. What would not be good is if there was not much distance between max and min values (as would be the case if a variable only had values like 1,2,3). Because k-means is a distance-based algorithm, it is preferable to have variables which have expressive differences (not 1,2,3) and are continuous.

To decide which variables I'll want to isolate for clustering later on I ask three questions:

• which variables will best differentiate teams?
• which variables are more descriptive of how a team plays?
• which variables have an expressive range of values?

I settle on the following list of variables:
‘Goals For’, ‘Goals Against’, ‘Attempts’, ‘On-Target’,’Blocked’, ‘Woodwork’, ‘Corners’, ‘Offsides’, ‘Ball possession %’, ‘Pass Accuracy %’, ‘Passes’,
‘Distance Covered km’, ‘Balls recovered’, ‘Tackles’, ‘Blocks’, ‘Clearances’, ‘Yellow cards’, ‘Red Cards’, ‘Fouls Committed’

🔩 (1C) Dataframe Manipulation

I choose to create a team-based view of the data in order to more clearly compare teams. Essentially, I want to calculate a given team’s average game stats based on all the games that said team played. The downside to this: since not all teams played the same amount of games, the average stats of teams that went further in the Cup will consider more data. But since I already don’t have that much data to work with and since I don't want to overcomplicate things, I decide to accept that inefficiency.

I create a new dataframe with a team-based view using the following code.

df_teamsort1 = df.sort_values([“Team”], ascending=True)

Improvement Note: with variable names it is usually preferred to not use numbers, so "df_team_sorted" might have been better than what I chose above.

This team-based view is interesting not only because it moves me closer to comparing and clustering teams, but also because, if I want, I can see what each team does over the course of the Cup.

Next, I officially select the variables that I want to use for clustering.

#selecting variable for clustering (minus Group, Team, WDL)

df_teamsort1_redux = df_teamsort1[[‘Group’,’Team’, ‘WDL’,’Goals For’, ‘Goals Against’, ‘Attempts’, ‘On-Target’,’Blocked’, ‘Woodwork’, ‘Corners’, ‘Offsides’, ‘Ball possession %’,’Pass Accuracy %’, ‘Passes’, ‘Distance Covered km’,’Balls recovered’, ‘Tackles’,’Blocks’, ‘Clearances’, ‘Yellow cards’,’Red Cards’, ‘Fouls Committed’,]]

I need to rename certain columns so they are amenable to potential manipulation later on. For example, if I want to filter or execute operations on a column like “Pass Accuracy %” Python may very well give me syntax errors because of the spaces and symbols. So, I change “Pass Accuracy %” to “Pass_Accuracy_pct”.

#renaming columns

df_teamsort1_redux.columns = [‘Group’, ‘Team’, ‘WDL’, ‘Goals_For’, ‘Goals_Against’, ‘Attempts’, ‘On_Target’,’Blocked’, ‘Woodwork’, ‘Corners’, ‘Offsides’, ‘Possession_pct’,’Pass_Accuracy_pct’, ‘Passes’, ‘Distance_km’, ‘Balls_recovered’,’Tackles’, ‘Blocks’, ‘Clearances’, ‘Yellows’, ‘Reds’, ‘Fouls’]

Improvement Note: this line of code is long and hardcoded parameters can block future code flexibility and expansion. I could have used a list to solve this with something like the following

variables_of_interest = ['Group', ….]
df_teamsort1_redux = [variables_of_interest]

I also perform some operations / add new columns that will be useful for clustering. First, I create a “Goals_delta” variable which gives me, per game, goals a team scored minus goals a team allowed. This tells more of the story per game (and per team) than simply how many goals a team scored. Second, I create a “On_target_pct” variable, which is nice for k-means because percentages are continuous variables. I set this variable's type to integer (intuitively this should be a float, but for some odd reason the first time I executed the code I got a type error, one that went away after converting its type to integer. Anyway, onward).

df_teamsort1_redux[‘Goals_delta’] = 
(df_teamsort1_redux[‘Goals_For’] - df_teamsort1_redux[‘Goals_Against’])
 
df_teamsort1_redux[‘On_target_pct’] = 
(df_teamsort1_redux[‘On_Target’] / df_teamsort1_redux[‘Attempts’])*100
 
df_teamsort1_redux[‘On_target_pct’] = df_teamsort1_redux[‘On_target_pct’].astype(int)

🍩😐 (1D) Normalizing Data

The next steps are noteworthy because I aggregate and normalize the data, making operations less painful going forward.

First, I take the variables (columns) that I want to use for clustering and turn them into one variable, columnslist_normalized. This is efficient because I can drop this one variable into future functions as well as use it to alter all the cluster-relevant columns in just one place.

columnslist_normalized = [‘Goals_For’, ‘Goals_Against’, ‘Attempts’,
‘On_Target’, ‘Blocked’, ‘Woodwork’, ‘Corners’, ‘Offsides’,
‘Possession_pct’, ‘Pass_Accuracy_pct’, ‘Passes’, ‘Distance_km’,
‘Balls_recovered’, ‘Tackles’, ‘Blocks’, ‘Clearances’, ‘Yellows’, ‘Reds’,‘Fouls’, ‘Goals_delta’, ‘On_target_pct’]

I put this new variable, columnslist_normalized, to use when I create df_teamavg, a new dataframe that will house average (mean) stats for each team. Here, I use the groupby function in order to pull all stats for a given team into one row.

df_teamavg = df_teamsort1_redux.groupby(‘Team’, as_index=False)

[columnslist_normalized].mean()

Process note: I create new dataframes (or copies) throughout my work as it helps me keep track of the progression of my data transformations. For me, it makes step-by-step issue diagnosis easier.

What I see:

• the variables have different ranges and scales

What I want to do:

• remove this variation and make the variables more comparable. This calls for normalization, the act of rescaling data so that all values are within a new, comparable range of 0–1.

This post helps to explain why normalization is relevant for k-means - it helps avoid implicit, unwanted variable weighting which could impact clustering.

To execute normalization I use the scikit-learn class MinMaxScaler. When one instantiates a class like MinMaxScaler, said class transforms into an object.

SAY WHAT?

Think of scikit-learn as a “library” with a bunch of “books” (classes). When these "books" are “opened” (instantiated into objects) one can put "ideas and info" (data) into them. Then one can do cool things with said data. So, we are going to pull MinMaxScaler from scikit-learn, instantiate MinMaxScaler and then put some data into MinMaxScaler so that it (MinMaxScaler) can do its thing with said data.

When we put a variable (the data from a column) into MinMaxScaler, it takes each value in that column, subtracts the min value of the column and then divides by the range.

y = (x - min) /(max - min)

From my research, a favorable trait of MinMaxScaler is that although it scales the values to the 0–1 range, it doesn't mess with the variable itself too much, meaning it preserves the shape of the original distribution.

Below, I show that once MinMaxScaler is defined, I call the fit_transform() function and use it on my main dataset (x) to create a transformed version of my main dataset (x_scaled).

#import preprocessing, object housing MinMaxScaler
from sklearn import preprocessing 

#take desired columns from df_teamavg, convert the df to a numpy array with .values method and store it as a variable, x
x = df_teamavg[columnslist_normalized].values 
 
#define a MinMaxScaler instance with default hyperparameters and store it as a variable, scaler
scaler = preprocessing.MinMaxScaler()
 
#call fit_transform() function and pass it to my dataset, x, to create a transformed version of the dataset, which will be x_scaled
x_scaled = scaler.fit_transform(x)

#convert dataset x_scaled, still a numpy array, back to a pandas DataFrame, X_norm, so I can keep rocking with it 
X_norm = pd.DataFrame(x_scaled)

#make a copy of original df_teamavg because I dont want to “back-modify” it 
df_teamavg_normalized = df_teamavg.copy() 

#take X_norm DataFrame (which has all the nice normalized data) and “pass” its data to the recently-made df_teamavg_normalized, only inserting the numeric data 
df_teamavg_normalized[columnslist_normalized] = X_norm

#since X_norm and df_teamavg_normalized[columnslist_normalized] have the same dimensions they fit together
#I regain the Team column via df_teamavg_normalized

Normalized Output:

Improvement Note: a friend and expert data scientist told me that the above bit of code is somewhat complex and arguably should be a function — one with inputs (the original DataFrame and columns to be transformed) and an output (the new DataFrame). I havent done that yet, clearly, but i'll get there :)

(2) Running the K-means Algorithm

👷🏾 (2A) The Coding of K-means

Now I run the k-means algorithm on the normalized data. First, a quick attempt at explaining some of the terminology and code below (thanks to a friend's "cookie" analogy).

kmeans = an object, think of it as a “cookie with certain magical traits”
KMeans = a class, think of it as a “cookie maker” or a “factory of different types of kmeans cookies”

import sklearn.cluster as cluster

#make a kmeans “cookie” from the KMeans “factory” and instantiate it/give it attributes of 4 clusters (n_clusters=k)
kmeans = cluster.KMeans(n_clusters=4,init=’k-means++’,max_iter=100,random_state=22)
 
#fit my input data to the kmeans model
kmeans = kmeans.fit(df_teamavg_normalized[columnslist_normalized]) 

#get cluster labels based on how the model groups my data (my teams, based on their average stats)
labels = kmeans.predict(df_teamavg_normalized[columnslist_normalized])

I could have also done the above by writing the following code,

kmeans = cluster.KMeans(n_clusters=4).fit(df_teamavg_normalized[columnslist_normalized])

, but I wanted to break it into step-by-step parts to make sure I was following what was going on and so that I could explain it to others more easily.

Important Note on my choosing 4 clusters: there are several methods available to estimate the optimal number of clusters. The elbow and silhouette methods are most common. While not perfect, they are good a starting point and sanity check when used in conjunction with subject matter knowledge and project goals. I have executed both methods in other situations, but I elected not to get into the details here as it was not my main focus. Here, I chose 4 clusters because I felt that more that 4 would make analysis confusing and spread the data too thin. Additionally, 4 is half the number of tournament groups (8) and also potentially forces interesting pattern finding among the 6 FIFA continental zones that participate in the tournament.

I now add a column called “Cluster” to my dataframe. “Cluster” is going to populate the dataframe with the clusters that the kmeans algorithm assigned to each team (recall "labels" from the code snippet above). For visualization purposes I use the .pop() function to move the cluster labels to the start (the left) of the dataframe, next to the “Team” column. Then I sort by cluster.

df_teamavg_normalized[‘Cluster’] = kmeans.labels_

first_column = df_teamavg_normalized.pop(‘Cluster’)

#insert column using insert(position,column_name,first_column) 
df_teamavg_normalized.insert(1, ‘Cluster’, first_column)

df_teamavg_normalized = df_teamavg_normalized.sort_values([‘Cluster’], ascending=True)

A view of the output: the normalized average stats of each team and their assigned k-means clusters

A last adjustment before analyzing the clusters - I want to be able to see each team, and its cluster, connected to the the non-normalized data (which is easier to interpret) in addition to the normalized data. The normalized data served its purpose in that it made clustering easier, but analyzing differences between clusters and drawing conclusions will be easier with the original, non-normalized data. To do this, I create a new, small dataframe that I can "attach" to the front of the non-normalized stats. This new "attachable" dataframe is called df_team_clusters.

I "attach", or merge, the above dataframe (df_team_clusters) to my earlier dataframe of non-normalized stats (df_teamavg) to create df_teamavg_withclusters

df_teamavg_withclusters = 
df_team_clusters.merge(df_teamavg, on=”Team”, how = ‘inner’)

df_teamavg_withclusters contains teams, their clusters and their average, non-normalized stats.

In order to get a look at the average stats of each cluster I use grouby.()

Cluster_StatsActual = df_teamavg_withclusters.groupby(‘Cluster’, as_index=False).mean()

Cluster_StatsActual

Below I show (1) the clusters and their respective teams after some formatting in excel, and (2) the excel-modified output of Cluster_StatsActual, showing average stats per game for each cluster.

🔮 (2B) Analyzing the Output

Before visualizing the data, I want to address the clusters produced.

Cluster 0 - "target practice" teams that clearly struggled most in the Cup:

• only one of its member teams, Denmark, got out of the Group stage
• performed worst in stats related to goals (Attempts, On Target %, Corners)
• always “on the defensive”, recording the most clearances, fouls, blocks

Cluster 1 - “powerhouses” with different journeys , unified by key stats:

• score high in the “attractive football” categories — logging the most Possession, Passes, Attempts, and Corners
• ran (distance_km) quite a bit, so we cant say they weren’t trying
• didn't recover the ball very much (this could be because they had it more) and they were not as efficient with their chances as Cluster 2
• do a lot of things better than field, but not enough to get to the Semis

Cluster 2 - “get it done” teams that advanced most and did well overall

• didn't turn heads with “attractive football”, but put the ball in the net
• ran hard, fouled hard, and recovered the ball well
• on_Target_% and Goals_delta stats were tops - they were mad efficient
• makes me wonder if Goals_delta was an overpowering variable

Cluster 3 - “good” teams that, by the stats, didn't do anything remarkable

• achieved 2nd or 3rd place in all categories, which is not…"bad"
• do not seem to like running
• give up the most goals

There were a few things in the data that peaked my curiosity, made me wonder about the impact of outliers.
1) Who is pulling the distance_km stat up so high for Cluster 2?
2) What were Sweden, Russia doing so well to get that far in the Cup?
3) Did France, Croatia, Belgium really foul that much?

To quickly try to gain insight into these questions I filtered the dataframe to look at specific teams and compare them to the aggregate data. I used the following code to filter, for example.

df_teamavg_withclusters[(df_teamavg_withclusters.Team == ‘Sweden’)| (df_teamavg_withclusters.Team == ‘Denmark’)|(df_teamavg_withclusters.Team == ‘Croatia’)|(df_teamavg_withclusters.Team == ‘France’)|(df_teamavg_withclusters.Team == ‘Russia’)]

Some answers to my above questions:
1) Russia and Croatia ran their assess off, averaging 125km and 118km per game, respectively. Russia, the host, punched above their weight throughoutt the Cup (likely due to being hopped up on excitement etc). Croatia just seemed to roll like that - they ran all the time, as was evidenced by their multiple come from behind wins. They really were the hard-working, cinderella of the tourney. France, the champion, actually ran much less that others, with 101km per game, on average.

2) Sweden really showed nothing too impressive… oh wait, they nearly had the highest average On_Target_% and were shot blocking machines. Russia was a typical Cluster 2 team and their hard, steady play worked out for them.

3) Yes, France, Croatia, and Belgium (3 of the 4 semi-finalists) fouled a lot, between 13 and 16 time per game. That is higher than I would have imagined for such skilled teams.

(3) 2x2 Visualization Comparisons

📈 (3A) Importing Viz Libraries and Plotting

Now, on to what I enjoy even more than clustering - the visualization of clusters and their related data. Here, I write code to visualize each team (tied to their clusters by color) in 2-by-2 plots.

First, I import the necessary libraries.

import seaborn as sns
import matplotlib.pyplot as plt

The code below contains the parameters needed to produce a readable visualization of clusters grouped by color. It also contains a for loop / iterator that allows me to print the name of each team near its respective data point.

The first question I was trying to answer was “Does Aggression pay off”?
In other words, do teams that foul more score more goals?

#viz parameters 
plt.figure(figsize=(20,12))
sns.scatterplot(data=df_teamavg_withclusters,x=’Fouls’,y=’Goals_For’, hue=’Cluster’, legend=True, alpha=1, palette=”bright”, s=50)

#for loop to print the name of each team near dots
for i in range(df_teamavg_withclusters.shape[0]):
plt.text(x=df_teamavg_withclusters.Fouls[i]+0.09,y=df_teamavg_withclusters.Goals_For[i],s=df_teamavg_withclusters.Team[i],fontdict=dict(color=’black’, alpha=1, size=10))
 
#set x limit
plt.xlim(df_teamavg_withclusters.Fouls.min()-0.1,df_teamavg_withclusters.Fouls.max()+1)

#title
plt.title(‘“Converted Agression” — Goals For x Fouls (Actual)’, fontsize=20) 

#x label
plt.xlabel(‘Fouls’, fontsize=15) 

#y label
plt.ylabel(‘Goals_For’, fontsize=15) 
plt.show()

There is no strong correlation between fouls and goals. But we can see that Russia clearly outperformed with respect to “converted aggression”. Croatia did as well, to a degree. Morocco and Korea were just hacking for fun it seems, while England and Spain were nice guys.

My second question was “Does running (distance_km) pay off in the form of a good Goals_delta stat?” In other words, are the teams running the most outsourcing their opponents? (I wont insert the code here — it is the same as above except for the new parameters related to this question).

There seems to be a slight, slight positive correlation… but it's a stretch. Another potential stretch — this particular 2x2 sort of shows me the clusters grouped together (greens near greens, reds near reds).

What this definitely does show that intrigues me is that teams like France, Belgium, and Brazil did not run an absurd amount but were still able to outscore teams. On the other hand, it seems Russia, Croatia and England really "worked hard for their money". Remember that song?

My third question was “Does passing perfection tie to shooting perfection?” In other words, if a team has great pass accuracy are most of their shots on target?

Again, it seems this 2x2 sort of shows me the clusters grouped together. What is noteworthy here is something I mentioned earlier. Fucking Sweden! They are an outlier of sorts — they cannot pass for shit, but man their shots are on target. Peru surprised me too! Croatia, interestingly enough, was not an overly accurate team — Modrić tried his best, but he couldn’t do it alone.

After seeing these 3 visualizations I can see why France got it done. They are always among the top performers.

The last question that interested me was "did the amount teams ran change over the course of the Cup?" Remember when waaay back in the beginning of this article I connected each team to its furthest stage in the tournament? Now, I join that data (“Round”) with the cluster data in order to create a new dataframe, df_withclusters_rounds. With this I will be able to visualize a bloxplot showing me each round of the Cup and how much the teams that got to that round were running. Each dot represents a team at its furthers stage and the color of that dot represents their cluster.

plt.figure(figsize=(20,5))
 
#define data to populate boxplot and formatting
ax = sns.boxplot(data=df_withclusters_rounds, x=’Round’, y=’Distance_km’, orient=’v’, color=’lightgray’, showfliers=False)
plt.setp(ax.artists, alpha=0.5)
 
#Add in points to show each observation
sns.stripplot(x=’Round’, y=’Distance_km’, data=df_withclusters_rounds, jitter=True, size=6, linewidth=0, hue = ‘Cluster’, alpha=0.7)
 
ax.axes.set_title(‘Distance per Round with ClusterDots’, fontsize=30)
ax.set_xlabel(‘Round’,fontsize=20)
ax.set_ylabel(‘Distance_km’,fontsize=20)
 
#Define where to place the legend
ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

plt.show()

Nothing too earth-shattering here, but let's finish it out. In the Group stage and the Round of 16 teams run about the same amount. Seems like in the Quarterfinals they really take their foot off the gas (this surprised me). However, maybe a better interpretation is that teams just condensed to the same amount of running. See that green dot way up high in the Quarterfinals column? It has got to be crazy running Russia or Croatia, both had overtime drama. In the Semis and Finals the median and the upper limit pops up — perhaps because teams are so close to total victory that they run until they drop.

Wow. Well, that was a mouthful.

I hope this, one of my early attempts to put Python into practice with sports data, was at least a little bit interesting to someone. If you made it this far, I hope you enjoyed it. Until next time.

World Cup 2018 Clustering Analysis was originally published in 3rd corner on Medium, where people are continuing the conversation by highlighting and responding to this story.

Early in 2020 I was explaining to a client my vision for a new podcast I planned on launching — Under Crossing. The basic idea was to look at events and phenomena in the world around us and interprete them using different lenses (human-centered design and data analytics, Brazilian and North American, science-based and mystical, etc). I explained that it was natural for me to think this way and that this is the type of thinking that inspired 3rd Corner.

The client asked if I would present the idea of the podcast and the inspiration for 3rd Corner to teams at the company who didnt know me and were not working with me. She just wanted me to spread this type of thinking at the company.

Presentation

You can see the presentation here

Reach Out and Stay Tuned

If you are interested in learning more, reach out at contact@3rdcorner.studio

And watch for future posts about 3rd Corner thinking, doing, learning and presentations.

Design and Data Processes — 3rd Corner View was originally published in 3rd corner on Medium, where people are continuing the conversation by highlighting and responding to this story.

Joining two approaches

Over the past decade, two disciplines have grown in popularity and relevance — data analytics and human-centered design.

Their rise was largely tied to the fact that both helped us understand and unravel problems in new ways. This was, and is, a positive thing.

The less positive thing is that we created separate worlds for these two disciplines. We often viewed data analytics and human-centered design as separate approaches dealing with two different types of data, the quantitative and qualitative, respectively.

Quantitative data, that which:

• answers questions like “What are they doing?”, “How much?”, “How many?”, “How often?”.
• comes in the form of numbers, can be verified upon review and can be evaluated using mathematical techniques and statistical analysis.
• is collected via surveys, databases, Customer Relationship Management (CRM) data and behavioral analytics.
• has “consistency of measurement” — when the question is “how many apples do you have?”, the difference between 6 and 8 is easy and always the same.

Qualitative data, that which

• answers questions like “Why are they doing it?”, “How?”, “What type?”, “What for?”.
• is non-numerical and collected via interviews, contextual observations, focus groups, etc.
• approximates and characterizes how people feel about themselves and the properties of phenomena around them.
• often lacks “consistency of measurement” — when the question is “on a scale of 1–10, how much pain are you in?”, the difference between 6 and 8 is not so clear and rarely the same.

When tackling a problem, data analysts and data scientists often take a quantitative approach to build hypotheses, refine a point-of-view and offer solutions. This involves data preparation and exploration, model creation and insight generation, and finally, refinement, validation and embedding of actionable analyses.

Human-centered designers frequently look to do the same thing via a qualitative approach (the double diamond is a common reference). This takes the form of user research, insights and principles synthesis, point-of-view construction, ideation and prototyping, and finally pre-launch refinement of a service or experience.

Yes, data analysts/scientists and the human-centered designers represent two different disciplines. But at a closer look, the essence of their approaches is quite similar.

Both aim to organize, structure and present data so that it becomes useful information. Both explore context, develop hypotheses and points-of-view, and refine solutions to solve complex problems.

Yet, at large corporations and even startups, it is all too common for data analysts/scientists and human-centered designers to treat each other like service providers, to work divided, and to not collaborate holistically and consistently.

I do not want to imply that analysts or designers always choose one approach and ignore the other. I do not want to imply that I am the first person to propose a mixed quant-qual approach.

But I do want to propose the following: to build digital products, data analysts/scientists and designers need to leverage each other’s processes and work as a unit. This unit should research, empathize, hypothesize, synthesize, think of variables and model approaches, sketch and prototype together.

Many companies do not successfully balance and blend mindsets from the quantitative and qualitative worlds. Many more do not even try. But marrying these two mindsets is critical, even necessary, for us to appreciate and make calibrated decisions in our increasingly nuanced and data-rich existence.

The Whats. The Whys.
The numbers. The feelings.
Two types of data.
Together they improve how we approach daily activities, challenges, and conclusions.

This was the inspiration for 3rd Corner.

3rd Corner & the road ahead

In August 2019, I started 3rd Corner, a consultancy that helps clients blend the lenses of data analytics, data science and human-centered design to see and solve challenges with a new focus. We work with designers, data scientists and strategists to deliver this.

Understanding people and the world around them became a primary interest of mine while studying anthropology in my last two years of college. This interest not only pushed me to live in West Africa, Europe and Latin America but it also colored my professional pursuits. Years working with financial strategy and quantitative analysis, and then human-centered research and design strategy, helped me develop different tools with which to perceive the data all around me. The more opportunities I had to combine these tools, the more I saw the potential to blend experience design, data analytics and data science.

When orchestrated well, this blend brings the rigor, the creativity and the balance that is essential to our work at 3rd Corner.

The Work

Some of the challenges we have worked on / are working on in our first year (client specifics are currently confidential):

• We are leading a quantified user insights initiative for a platform that aims to democratize investing. We are implementing a process that quantifies user behavior, characteristics, and journeys as well as identifies actionable data in user clusters. We are setting the groundwork for multidisciplinary data science and design teams to leverage quant and qual data to prioritize value and problem-solving efforts.
• We collaborated with an Artificial Intelligence (AI) firm to help a medical network use Natural Language Processing (NLP) to extract meaningful information from language data. We recognized that while AI is a powerful tool, alone it cannot determine which human problems to tackle. Thus, to best leverage NLP, we designed data ideation workshops and mapped data workflows to strategically enhance doctor, administrative and patient experiences.
• We are coaching product designers and data analysts at a financial services company to establish data-driven Design Research Operations. At its core, this involves developing quant/qual research tools relevant to all phases of the product lifecycle — discovery, development and delivery. We believe this will enhance how exploratory and strategic research improve company-wide product decisions.
• We are leading a fashion design company in the implementation of agile-inspired routines. Our goal is to improve how data flows from collection-to-use across analytics and design squads, as well as how experiment with a focus on iterative measurement.
• We worked with a sustainability-focused agency to help them strategize the prototyping of new business models as well as how to creatively think about new metrics and data collection.
• We advised a digital media company aiming to enhance their qualitative research by collecting new types of data via quantitative methods.

Why Now

We realize that companies vary in their ability to blend the efforts of designers, data analysts and data scientists. But this does not stop us from helping clients understand why joining these disciplines is relevant now.

1. It turns out that opposites attract. Over the last decade, human-centered designers and data scientists and analysts have become the popular kids at school. Unfortunately they are often on different, if not opposite, sides of problem-solving and decision processes. But by bringing them together into collaboration units, designers and data scientists and analysts can unlock new insights born out of leveraging their differences: data-informed ideation, machine learning segmentation and outlier-inspired research, prototypes that consume and produce analyzable data, etc.

2. Alone, traditional needs-focused UX work is less and less differentiating. Good User Experience (UX) patterns are no longer secrets; the “design thinker” explosion means in-depth interviews and empathy maps are the norm. As a result, we should not expect differentiation to solely come from how emotionally in-touch, contextual or beautiful our digital services are.

Because many of these services are (at least partially) “fueled” by data that users provide (via direct responses, trigger events, inference of behaviors, etc.) it is increasingly critical to tie the data to the user in actionable ways. Differentiation will come from how we combine contextual empathy work with an understanding of quantitative variables that approximate human sentiment and behavior. This way our products can manifest an understanding of how the data implies intentions, aspirations and desires of individuals.

3. The abundance of quantitative data demands more considered, empathetic treatment. “More data, more problems” becomes our reality if we are careless. Technical capacity to track and ingest rich, large sets of data is only worth so much if we are not considerate about how we select, transform and display it. Consistent collaboration between analysts, scientists and designers helps us ask better questions to filter and frame this data, aligning powerful systems with human needs.

This collaboration also helps us design products with “data journeys” better suited to receive, share, and leverage users' data in ways that solve functional needs as well as respect privacy and transparency.

4. We are increasingly aware of the real complexity of challenges. Whether it is increasing employment for low-income workers via an app or educating young parents about child development via a service, the challenges we face are nuanced. Individuals have more access to opinions, emotions, facts, and numbers than ever before. Implicitly, many individuals expect “problem-solvers” to synthesize these multiple sources and perspectives.

If data analysts, data scientists and human-centered designers work as a unit, we get closer to reaching the expectation for more holistic solutions to complex challenges.

We look forward to the challenges ahead.

Reach Out and Stay Tuned

If you are interested in learning more, reach out at contact@3rdcorner.studio

And watch for future posts about how we think, what we are doing, and what we are learning.

A sincere thanks is owed to several people for their support, collaboration, feedback, and edits: Talita Di Iorio, Noah Norman, Dave Zaretsky, Will Heidrich, Muamer Cisija, Rafael Fernandes.

Towards unifying Data Analytics & Experience Design — a 3rd Corner introduction was originally published in 3rd corner on Medium, where people are continuing the conversation by highlighting and responding to this story.