Sitemap
AI/HUB

AI/HUB

Durham College - ORSIE

Press enter or click to view image in full size
Press enter or click to view image in full size

With the pros and cons of tokenization covered in the previous lesson, it is a good time to review word embeddings.

Since tokens allow us to simplify the collection of words we have gathered from our document or a pre-trained dictionary; We can now map the tokens to a three-dimensional (3D) space, which we will call the ‘embedding space’. You can think of the embedding space as a matrix of vectors, but in python terms, it is considered a list of lists.

When embedding our tokens into the embedding space, they are given weights relative to each other. In general terms, the weights are often determined by each tokens’ occurrence, frequency, and proximity to other tokens. However, there are various methods to achieve different results depending on what your needs are.

Once we have embedded our tokens, we can use our new embedding space to perform different types of matrix multiplication in order to identify different types of patterns and relationships between our tokens. We can even extract our embedding space and save it for deployment in a system or service. There are many ways to achieve this but two of the most common frameworks are word2vec and GloVe. Both have been implemented by a large number of libraries but TensorFlow, Keras, and Gensim have some of the most reliable versions that are quite simple to implement.

GloVe vs Word2Vec

“GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.”

— NLP Stanford, GloVe

“ Word2vec is an algorithm invented at Google for training word embeddings. Word2vec relies on the distributional hypothesis to map semantically similar words to geometrically close embedding vectors.”

— Google, Machine Learning Crash Course

As you can see above, both algorithms utilize the vector space but are optimized using slightly different techniques.

Let’s take a look at some Word2Vec code!

First, we need to import the libraries we will be using:

# imports
import gzip
import gensim
from gensim.utils import tokenize
import logging
# set up logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

Next, we will import the dataset and check to make sure it is intact.

At this point, we need to pass our data into a list so that Word2Vec will accept it when we create our model. We will do some simple pre-processing as well to ensure that our data does not contain foreign characters or HTML code.

Now that our data has been cleaned (enough for this demo).

Let’s pass a new Word2Vec object our list of lists and start training our model! It may take a minute or two to train depending on your PC hardware.

Press enter or click to view image in full size

You may have noticed that we passed several parameters to the model when we instantiated it above. Let’s review them quickly so you can understand their purpose.

Get AI/HUB’s stories in your inbox

Join Medium for free to get updates from this writer.

size

  • The size of the dense vector which will represent each token (word).
  • Smaller datasets will typically use smaller values than larger datasets, however, it is worth experimenting with larger datasets to see what works best.
  • In this demo, 125–150 will be an effective choice.

window

  • The maximum distance between the target word and its neighboring word.
  • If the neighbor’s position is greater than the maximum window width, some neighbors are not considered as being related to the target.
  • The smaller the window, the more related terms will be.

min_count

  • The minimum frequency count of words.
  • If a word occurs fewer times than this value it will not be stored.
  • Words that are rarely used can make contextual derivations harder later.

workers

  • The number of workers (threads) to run in the background while processing.

Now that you have an idea of what the model is looking for based on our parameters. Let’s start visualizing our results!

First, we need to import the required libraries. Bokeh, for example, will allow you to better visualize how the ‘window’ parameter comes into play.

Press enter or click to view image in full size

We also need to create a function for easily printing our plot.

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a (prize-winning) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. — Laurens van der Maaten

Press enter or click to view image in full size

With those both in place, let’s see what the output is!

Press enter or click to view image in full size

Notice it looks like someone threw a bucket of words at a wall. However, if we zoom in, the window starts to present some interesting relationships that were not present before.

Press enter or click to view image in full size

Additionally, since our model trained successfully. We can save it for use in other places. Like text prediction algorithms, or sentiment analysis.

model.wv.save_word2vec_format('w2v-test', binary=False)

It is important to note that this data is actually in a 3D space and is simply being flattened above. The full Jupyter Notebook for this lesson can be found here: https://github.com/connlaoi/word2vec-with-visualizations

It includes 3D visualizations as well that can be easily deployed on existing Word2Vec models.

Written by Connlaoi Smith under the supervision of Professor Amit Maraj.

Press enter or click to view image in full size

--

--