word2vec
In word2vec, captures meaning of a word and establishes a relationship between words (if any) by “Feature Representation”.
Represents words as “Embeddings”
word2vec does feature classification by taking features automatically on its own, all this is done internally.
Any 2 words with almost similar vector values indicates that those two words have a relationship between them.
Vector size of word is equal to no. of features compared
Cosine similarity is the cos theta(angle) between vectors of two words
Distance = 1 - cos theta…if the distance is very less (or) negligable then the two words are considered to be similar.
Example
Supppose there are four features Royalty, Human, Food, Masculanity
For the words → King, Queen, Mango
Then, (assume) the embeddings are, it will be done internally
‘+’ indicates that the word strongly aligns with the feature
‘ — ’ indicates the word lies on the opposite side of the feature axis
Let Queen’s Masculinity be 0.6 for similarity, coz King and Queen should be similar
Calculation of cosine similarity
For King and Queen
A.B = 0.9 * 0.9 + 0.9 * 0.9 + -0.8 * -0.8 + 0.8 * 0.6
= 0.81 + 0.81 + 0.64+0.48
= 2.74
|A| = Root of (0.9)² + (0.9)² + (-0.8)² + (0.8)²
= Root of 0.81+0.81+0.64+0.64
= 1.702
|B| = Root of (0.9)² + (0.9)² + (-0.8)² + (0.6)²
= 1.618
2.74 / 1.702 * 1.618
= 0.996
Get Anonymous’s stories in your inbox
Join Medium for free to get updates from this writer.
Distance = 1 — Cosine similarity
1–0.996 = 0.004
Distance is very less, therefore both words are similar
2 Approaches
Continuous Bag of Words (CBOW)
Main purpose of CBOW is to predict “target word” based on surrounding words.
Its main aim is to learn embeddings
We’ve a concept of “Window” size.
Ex : Sentence → I ate a mango yesterday
If we given window = 2,
target word = mango
Then, it checks 2 words to the left of mango and 2 words to the right of mango
So, words are
ate, a (left)
yesterday(right) → there’s only one word to the right, that’s not an issue
When we give those words (context words), it gives an output mango (which is a target word)
By doing this repeatedly for million no. of sentences, it learns embeddings better.
That is establishing relationship between words like
mango = apple, king = queen etc…
Its called “Bag of Words” coz context words are treated as collection (bag) and ignores their order.
Order : [ate,a,yesterday] or [yesterday,ate,a] are considered same by CBoW
Skipgram
Opposite to CBoW, in skipgram, if we give target word, context words are given as output
CBOW is generally faster and works well for frequent words.
Skip-Gram is slower but performs better for rare words.
