Member-only story
Why King − Man + Woman = Queen: What a Word Embedding Actually Is
Word2vec, the 2013 method everyone called a neural network, was factorizing a word-pair table linguists had used since 1990.
In 2013 a group of researchers at Google described a piece of arithmetic that has since opened almost every introduction to machine learning. Take the list of numbers their program, word2vec, had assigned to the word king, subtract the list for man, add the list for woman, and ask which word’s list lies closest to the result. The program answers queen. The word vectors Google released that year had been trained on roughly a hundred billion words of news text, with no dictionary, no grammar and no information about monarchy or sex, only a record of which words turn up near which others.
The demonstration raises two questions. How can subtracting one word from another mean anything at all? And did the arithmetic really say queen? Popular accounts tend to answer only the first, and loosely. The first has a precise answer, and it runs through a table of word statistics that linguists were computing more than a decade before the first neural language models. The second has an answer too, though it flatters the demonstration less.
