Word Embeddings

2 min readMay 6, 2020

Introduction and Implementation

Definition

Word embeddings are the vector representations of words. It represents words or phrases in vector space with several dimensions. These vectors are simply real valued numerics which captures the meaning of the word. The words with similar meaning have a similar representation in word embeddings.

Techniques

Count Vector : It is a frequency based method where the frequency or the count of the word in every sentences assembled together represents the word’s vector.

TF-IDF Vector : It is a frequency based method where the tf-idf value of the word in every sentences assembled together represents the word’s vector.

Word2Vec : It is a word embedding technique that is used to convert the words in the dataset to vectors. Each unique word in the text is assigned to a vector and these vectors vary in dimensions depending on the length of the word. It has two different architecture :

CBOW (Continuous bag of words): It is a prediction based method where it learns by predicting the current word based on its context.
Skip Gram : It is a prediction based model where it learns by predicting the surrounding words given a current word.

Implementation

Requirement
Python, NLTK, Scikit-learn, Numpy, Gensim

Code

Get Ashin Shakya’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Count Vector

from sklearn.feature_extraction.text import CountVectorizer
from nltk.tokenize import sent_tokenize
import numpy

text = 'Your text goes here.'# Tokenize sentences
sentences = sent_tokenize(text.strip())

vectorizer = CountVectorizer()
# Generate sentence vectors
sentence_vectors = vectorizer.fit_transform(sentences)# To generate word embeddings
word_embeddings = []
for i, word in enumerate(vectorizer.get_feature_names()):
    word_vectors = []
    for j, sentence in enumerate(sentences):
        # Get the count of word from each sentence
        word_vectors.append(sentence_vectors[j,i])
    word_embeddings.append(word_vectors)

word_embeddings = numpy.array(word_embeddings)print(word_embeddings)

TF-IDF Vector

from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.tokenize import sent_tokenize
import numpy

text = 'Your text goes here.'# Tokenize sentences
sentences = sent_tokenize(text.strip())

vectorizer = TfidfVectorizer(norm=False, smooth_idf=False)
# Generate sentence vectors
sentence_vectors = vectorizer.fit_transform(sentences)# To generate word embeddings
word_embeddings = []
for i, word in enumerate(vectorizer.get_feature_names()):
    word_vectors = []
    for j, sentence in enumerate(sentences):
        # Get the count of word from each sentence
        word_vectors.append(sentence_vectors[j,i])
    word_embeddings.append(word_vectors)

word_embeddings = numpy.array(word_embeddings)print(word_embeddings)

CBOW (Common bag of words)

from gensim.models import Word2Vec
from nltk.tokenize import sent_tokenize, word_tokenizetext = 'Your text goes here.'sentences = sent_tokenize(text)
words = [word_tokenize(sent) for sent in sentences]# To save model
model = Word2Vec(words,min_count=1)
model.save("word2vec_CBOW.model")# To load model
model = Word2Vec.load("word2vec_CBOW.model")print(model.wv.get_vector('word'))

Skip Gram

from gensim.models import Word2Vec
from nltk.tokenize import sent_tokenize, word_tokenizetext = 'Your text goes here.'sentences = sent_tokenize(text)
words = [word_tokenize(sent) for sent in sentences]# To save model
model = Word2Vec(words,min_count=1,sg=1)
model.save("word2vec_CBOW.model")# To load model
model = Word2Vec.load("word2vec_CBOW.model")print(model.wv.get_vector('word'))

Full Code Implementation

Get the full codes — Count Vector, TF IDF, CBOW and Skip Gram.