Word Associations in the Large Language Model Era
Liina Vesilind, Raiko Marrandi, Raido Rehepapp, Rimmo Rõõm
If one hears the word bank, one can immediately think of words like money, notes, bills, ATM, etc. This phenomenon is known as word association and it shows that people, upon hearing a specific word, automatically generate connections and contexts between different words (Word Associations Network, n.d.).
Associations are the longest-lasting theory in psychology, having been around since at least Ancient Greece when Aristotle presented his theory of recollection in “De memoria” (Aristotle & Ross, 1906). Jüri Allik summarized Aristotle’s theory as the following: the elements of the psyche — thoughts, imaginations, memory traces etc. — have a natural tendency to connect or join due to their properties of mutual similarity-difference, subordination and appearing together (History of Psychology lecture, February 1, 2021).
In this blogpost we will be looking at how large language models, such as OpenAI GPT-3.5 and GPT-4, perform in the Kent-Rosonoff Free Association Test compared to humans. As word associations are directly related to a person’s age, life experiences, language proficiency, and more, it is very interesting to see how the results may differ on average, if at all, from the large language models. Furthermore, does knowledge of the test direct answers to converge to standard human answers and does prompting in a creative direction disperse them?
Kent-Rosonoff Free Association Test is one of the most popular word-association tests, first introduced by US psychologist Grace H. Kent and Aaron J. Rosanoff in 1910. The test consists of 100 nouns, verbs, and adjectives (Oxford Reference, n.d.). In the original study by Kent and Rosanoff (1910) 738 people were tested from various age groups, locations, and occupations. Originally used to study insanity and mental disorders, it is now, however, more often used to study the strength of connection between mental objects (Nelson & Brooks, 1974), language development (Marcus, Fernandes, & Johnson, 2012), psychopathology (Kent & Rosanoff, 1910), the subconscious in psychoanalysis (Rabeyron & Massicotte, 2020) and creativity (Upmanyu & Singh, 2020).
The University of South Florida free association norms database started in 1973 and has over 750 000 responses from 6000 people to around 5000 stimulus words (76% are nouns, 13% adjectives and 7% verbs) (Nelson et al., 2004).
In this blog post, we selected 100 random words (Table 1) from The University of South Florida database, referred to as the Original Dataset hereinafter. We examine how language models, such as OpenAI GPT-3.5 and GPT-4, respond to these stimulus words and then compare their responses with those generated by humans and other models in this research.
The models are prompted through the OpenAI api Python distribution, the answers are compiled to datasets and then analysed using Python in Jupyter notebook with the Pandas library. The analyses look at the distribution of answers in general and for specific cues. A quantitative analysis is performed for synonyms and part-of-speech types based on NLTK wordnet.
We have instructed the models with 3 different prompts — a non-revealing general prompt, a prompt revealing the purpose of the test, and a prompt to spark creative answers. The prompts were the following:
- Prompt (Prompt 0) to set role: “You are given a prompt and a list of words. For each word, replace the string ‘<word>’ with a word from the list and give back a one word answer, return answers for all words.” and prompt: “Generate a word related to <word>.”
- Prompt (Prompt 1) to set role: “This is a free association test. You are given a prompt and a list of cue words. For each word, replace the string ‘<word>’ with a word from the list and give back a one word response, return answers for all words.” and prompt: “Generate a response word related to <word>.”
- Prompt (Prompt 2) to set role: “You are given a prompt and a list of words. For each word, replace the string ‘<word>’ with a word from the list and give back a one word answer, return answers for all words.” and prompt: “Provide a word that evokes emotions or thoughts related to the word <word>.”
We ran the list of the selected 100 words 100–200 times (GPT-3.5 200x and GPT-4 100x). Original data sample sizes vary between 128 and 180 (avg. 148), depending on a word. Sample sizes are similar, however GPT-4 provided some faulty responses, which were not included in the final data. The results in comparison to the original dataset are in the next sections of this post. All of the data and used code can be found in our GitHub repository.
First we are looking at the answers given by GPT-3.5.
The Original Dataset has a total of 1486 different responses to 100 stimulus words, the most responses were to words IMPORTANT (25), FACTORY (24) and ASSUME/BOND (23). Interestingly GPT-3.5 had a maximum of 649 different responses to Prompt 1, which is more than 55% less than in the Original Dataset. Words that have the most diverse responses in the Original Dataset do not match with the TOP10 different responses from the models. As can be seen from Figure 1, all Prompts share 80% of the TOP10 words. As each iteration of the model is the same, it is expected that answers are similar. However what is interesting, is the difference with human responses.
We have included a line graph (Figure 2) that shows how many different answers there are to 100 words in 3 prompts and the original dataset. As the red line shows the original dataset answers it is clearly seen that humans have more diverse answers to stimulus words.
We will now look at the GPT-4.0 responses, to see if there is a similar pattern between prompts and the human responses before we draw any conclusions.
From the results (Figure 3 and Figure 4) of GPT-4, we can clearly see that the number of different responses has gone up to 1087 different responses with Prompt 2. This is an improvement from version 3.5, however it is still lower than the human responses. As with the bold words, there is now some overlap of cue words that have the most different answers in the Original Dataset and GPT-4.
From the results (Figure 5) it can be seen that humans associate word BOND the most with the 007 movie character (pop cultural reference) James Bond as language models with the actual word synonyms like connection, unity, relation and friendships. Another interesting finding is that the word BOND can also be seen as both money and stock, which were not present in the language model responses. It seems that language models are not good at associating the words across different fields of life.
Let’s investigate this hypothesis further with other 2 words that are overlapping.
ACTIVITY (Figure 6) is a very general word, however while machine responses belong to the sport-game-movement field, humans tend to also associate the word outside of that scope with responses like busy, sex, school, and tired — words that refer to feeling something.
We have one more word to examine.
Get Liina Vesilind’s stories in your inbox
Join Medium for free to get updates from this writer.
Friendship (Figure 7) has interesting responses as humans mostly relate the word to abstract relations of feelings like love, forever and trust. However the language models use mostly synonyms like companionship, companion, and also bond.
When we looked at the cue “Bond”, the response “Friendship” was present in Prompt 1, Prompt 0 however generated only one response “Friendship”. This leads to a conclusion that some words may be associated only one way.
As we now have compared the ones that are overlapping, let’s look at what the language models answered to the most diverse word — IMPORTANT.
These results also support the previous hypothesis that models are not very good at associating words outside the synonym sphere, especially association with feelings and abstract meanings.
As prompt 2: “Provide a word that evokes emotions or thoughts related to the word <word>.” asks to return words related to emotions, it could cue for answers more similar to human responses. The most diverse responses of prompt 2 were to words FREQUENT (Figure 9) and SOUTHERN (Figure 10). Let’s compare each one with human answers. We have also added Thesaurus synonyms as a comparison for Frequent (Synonyms and Antonyms of Words | Thesaurus.com, 2020). For prompt 2, some of the answers are words that often appear together with the cue “frequent”, like visit, travel, and visitor.
From the human answers to the cue “southern”, it is clearly seen that since the participants are from the United States, many of the answers are related to geographical parts of states. Also, the language model was able to produce cultural references as answers, like “southern hospitality” and relate the word to concepts like traditionalism, nostalgia, warmth, and “dixies”.
Large language models tend to associate words with quantitatively less responses than humans (Figure 11). Clearly answers from different iterations of single language models, trained on a wealthy dataset of largely human created data, aren’t as distributed as responses based on a wide range of human experience.
Yet the qualitative value of these answers also seems subpar compared to humans. A rule-of-thumb can be seen — man answers from top-of-mind, machine answers from top-of-thesaurus. This rule seems to be broken when the models are prompted to answer more creatively and are directed towards an answering strategy relying on emotions and abstract concepts. When we checked all cue-response pairs for whether they were synonyms, it turned out that humans responded with synonyms 85.6% of the time! A one sample T-Test with the null-hypothesis that the mean of the synonym-percentage of LLM cue-response pairs is the same as the expected mean of the human dataset (α = 0.05) showed that only GPT3.5 provided a similar percentage of different synonym answers on two prompts (Table 2). Yet the only model that provided non-synonym answers statistically more frequently than humans was GPT4 with the creative prompt. So machines might make more inventive free associations than humans, but only if they’re told to be creative and people are not.
Conclusion
Looking at the results given to Prompts and comparing them to the Original Dataset, we had a Hypothesis that Machine models tend to answer synonyms to stimulus words, as humans tend to have more diverse answers that also include references to feelings (sex, love, crucial, valid, ect.) and culture (James Bond). This hypothesis stays true unless the machine is told to also include those to the answer scope. The results can be summed up by a simple sentence: man answers from top-of-mind, machine answers from top-of-thesaurus.
References
Kent-Rosanoff Free Association Test. Oxford Reference. Retrieved 19 Nov. 2023, from https://www.oxfordreference.com/view/10.1093/oi/authority.20110803100034390.
KENT, G. H., & ROSANOFF, A. J. A study of association in insanity. Amer. J. Insanity, 1910, 67, 37–96, 317–390.
Russell, W. A. (1970). THE COMPLETE GERMAN LANGUAGE NORMS FOR RESPONSES TO 100 WORDS FROM THE KENT-ROSANOFF WORD ASSOCIATION TEST. The work reported here was carried out during the writer’s tenure as a Senior Research Scholar at the University of Wuerzburg under a United States Government Grant (Fulbright Act). Appreciation must be expressed for the cooperation given by Dr. W. Arnold, head of the Psychological Institute at Wuerzburg. The assistance of Oskar R. Meseck at all stages of data collection is also . . .. In Elsevier eBooks (pp. 53–94). https://doi.org/10.1016/b978-0-12-563050-4.50006-6
Upmanyu, V. V., & Singh, S. (2020). WORD ASSOCIATION TEST RESPONSES: AN INDEX OF PATHOLOGICAL CHARACTERISTICS OR CREATIVITY. ResearchGate. https://www.researchgate.net/publication/338396324_WORD_ASSOCIATION_TEST_RESPONSES_AN_INDEX_OF_PATHOLOGICAL_CHARACTERISTICS_OR_CREATIVITY
Word Associations Network. (n.d.). https://wordassociations.net/en/
USF Free Association Norms: Introduction. (n.d.). http://w3.usf.edu/FreeAssociation/Intro.html
Aristotle, & Ross, G. R. (1906). Aristotle De sensu and De memoria; text and translation, with introduction and commentary. Cambridge The University press.
Kent, G., & Rosanoff, A. (1910). A Study of Association in Insanity. American Journal of Insanity, 67, 317–390.
Marcus, G. F., Fernandes, K. J., & Johnson, S. P. (2012). The role of association in early word-learning. Frontiers in Psychology. doi:https://doi.org/10.3389/fpsyg.2012.00283
Nelson, D. L., & Brooks, D. H. (1974). Relative effectiveness of rhymes and synonyms as retrieval cues. Journal of Experimental Psychology, 503–507. doi:https://doi.org/10.1037/h0035896
Nelson, D.L., McEvoy, C.L. & Schreiber, T.A. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers 36, 402–407 (2004). https://doi.org/10.3758/BF03195588
Rabeyron, T., & Massicotte, C. (2020). Entropy, Free Energy, and Symbolization: Free Association at the Intersection of Psychoanalysis and Neuroscience. Frontiers in Psychology. doi:https://doi.org/10.3389/fpsyg.2020.00366
Synonyms and Antonyms of Words | Thesaurus.com. (2020, September 17). Thesaurus.com. https://www.thesaurus.com/browse/frequent
https://github.com/rehe-papp/TLMproject2023
https://github.com/openai/openai-python
