AI Research at Holler Technologies

With the rise of social media, pictographs, commonly referred to as “emoji” have become one of the world’s fastest-growing forms of communication [Parkin, 2016]. First introduced three decades ago, emoji become mainstream after their standardization by The Unicode Consortium in 2009, which enabled emoji adoption on mobile platforms. The rapid growth of emoji began in 2011 when the Apple iPhone added an emoji keyboard to iOS, and again in 2013 when the Android mobile platform started to support emoji on their mobile devices [Dimson, 2015]. Since then, emojis have become an important part of online communication. Today, they are being used billions of times across all major social media platforms to express sentiment, emotion, and sarcasm among other non-verbal cues. Thus, understanding what is expressed through emoji in text messages is extremely important to the modern-day Natural Language Processing (NLP) and Natural Language Understanding (NLU) applications.

As analysis and modeling of written text by natural language processing techniques have enabled important advances such as machine translation [Weaver, 1955], word sense disambiguation [Navigli, 2009], semantic similarity [Gomma and Fahmy, 2013], and search [Guha et al., 2003], the transfer of such methods (or development of new methods) over emoji is only beginning to be explored [Wijeratne et al., 2017a]. Only recently have there been efforts to mimic standard natural language processing techniques used for semantic similarity, sense disambiguation, and search into the realm of emoji. Such emoji understanding tasks are essential to automatically process, derive meaning, and interpret text fused with emoji. In this article, we will look at four emoji understanding tasks, namely, calculating emoji similarity, emoji sense disambiguation, emoji prediction, and identifying the changes in emoji meaning over time. We will go through the past research that has been conducted on these tasks and discuss what is more to be done there.

Calculating Emoji Similarity

Calculating similarity in a vocabulary of words is one of the basic problems found in language understanding research [Gomma and Fahmy, 2013]. Similarly, calculating emoji similarity is one of the basic research problems found in emoji understanding research [Wijeratne, 2018]. The notion of the similarity of two emojis is very broad. One can imagine a similarity measure based on the pixel similarity of emoji pictographs, yet this may not be useful since the pictorial representation of an emoji varies by mobile and computer platform [Miller et al., 2016]. Two similar-looking pictographs may also correspond to emoji with radically different senses (e.g., twelve-thirty 🕧 and six o’clock 🕕, raised hand ✋ and raised back of hand 🤚, etc.). Similarly, one could define the semantic similarity of emoji such that the measure reflects the likeness of their meaning, interpretation, or intended use. We prefer the semantic similarity measures based on the meanings of the emoji. Below, we will briefly go over the past research conducted on this space.

Barbieri et al. were the first ones to study the emoji similarity problem [Barbieri et al, 2016]. They collected a sample of 10 million tweets originating from the USA and trained an emoji embedding model using tweets as the input. Then, using 50 manually-generated emoji pairs annotated by humans for emoji similarity and relatedness, they evaluated how well the learned emoji embeddings align with the human annotations. They reported that the learned emoji embeddings align more closely with the relatedness judgment scores of human annotators than the similarity judgment scores. Eisner et al. used a word embedding model learned over the Google News corpus, applied it to emoji names and keywords extracted from the Unicode Consortium website, and learned an emoji embedding model which they called emoji2vec [Eisner et al., 2016]. Using t-SNE for data visualization, Eisner et al. showed that the high dimensional emoji embeddings learned by emoji2vec could group emoji into clusters based on their similarity. They also showed that their emoji embedding model could outperform Barbieri et al.’s model in a sentiment analysis task. Pohl et al. studied the emoji similarity problem using two methods; one based on the emoji keywords extracted from the Unicode Consortium website and the other based on emoji embeddings learned from a Twitter message corpus [Pohl et al., 2017]. They used the Jaccard Coefficient on the emoji keywords extracted from the Unicode Consortium to find the similarity of two emojis. They evaluated their approach using 90 manually-generated emoji pairs. Wijeratne et al. studied semantic similarity of emoji through embedding models [Wijeratne et al., 2017a] that are learned over machine-readable emoji meanings extracted from the EmojiNet [Wijeratne et al., 2017b] knowledge base. Using emoji descriptions, emoji sense labels, and emoji sense definitions, and with different training corpora obtained from Twitter and Google News, they developed and tested multiple embedding models to measure emoji similarity. To evaluate their work, they created a new dataset called EmoSim508, which assigned human-annotated semantic similarity scores to a set of 508 carefully selected emoji pairs. After validation with EmoSim508, they further evaluated their emoji embedding models using a sentiment analysis benchmark and showed that their models outperform Barbieri et al.’s and Eisner et al.’s models, achieving the state-of-the-art results in emoji similarity.

Emoji Sense Disambiguation

When emoji were first introduced, they were defined with no rigid semantics attached [Unicode FAQ], which allowed people to develop their own use and interpretation. Without rigid semantics attached to them, emoji symbols can take on different meanings based on the context of a message. Thus, for natural language processing and understanding tasks, machines must learn to disambiguate the meaning or the ‘sense’ of an emoji in text messages. For example, consider the three emojis 😂, 🔫, and 💰 and their use in multiple tweets shown in Figure 1. Depending on the context, we see that each of these emojis can take on wildly different meanings. People use the 😂 emoji to mean laughter, happiness, and humor; the 🔫 emoji to discuss killings, shootings, or anger; and the 💰 emoji to express that something is expensive, working hard to earn money, or simply to refer to money. Knowing the meaning of an emoji can significantly enhance applications that study, analyze, and summarize electronic communications. For example, rather than stripping away emoji in a preprocessing step, the sentiment analysis application reported in [Novak et al., 2015] uses emoji to improve its sentiment score. Similarly, emoji can be used to detect sarcasm in text messages [Felbo et al., 2017]. Thus, the task of emoji sense disambiguation, which is the ability to identify the meaning of an emoji in the context of a message in a computational manner [Wijeratne et al., 2016], plays an important role in natural language processing and understanding applications.

Emoji sense disambiguation hasn’t received much attention from the research community as it’s considered one of the difficult problems to solve [Miller et al, 2017] due to many reasons. For example, [Miller et al, 2017] discussed how difficult it is to come to a common consensus on emoji meanings among emoji users on social media. Since the same emoji could appear differently on different mobile and web platforms, when coupled with the issues related to coming to a common consensus with emoji meanings, it is difficult to collect a properly annotated training dataset for supervised learning of emoji meanings. Such issues can be avoided if we had access to a list of possible emoji meanings for each emoji (e.g., an emoji meaning dictionary or a knowledge-base), thus, we can use those meanings as a starting point to tackle the emoji sense disambiguation problem. Based on this idea, an attempt has been made to solve the emoji sense disambiguation problem in a knowledge-based setting using EmojiNet [Wijeratne et al., 2017a]. EmojiNet [Wijeratne et al., 2016, Wijeratne et al., 2017a] is an emoji meaning dictionary consisting of: (i) 12,904 sense labels (emoji meanings) over 2,389 emoji, which were extracted from the web and linked to machine-readable sense definitions seen in BabelNet; (ii) context words associated with each emoji sense, which are inferred through word embedding models trained over Google News corpus and a Twitter message corpus for each emoji sense definition; and (iii) recognizing discrepancies in the presentation of emoji on different platforms, specification of the most likely platform-based emoji sense for a selected set of emoji [Download EmojiNet]. To discuss how EmojiNet can be used to solve the emoji sense disambiguation problem, they illustrate disambiguation of the sense of the 🙏 emoji, which is one of the most misunderstood emoji on social media, as it is used in two example tweets.

Tweet 1 : Pray for my family 🙏 God gained an angel today.

Tweet 2 : Hard to win, but we did it man 🙏 Lets celebrate!

EmojiNet lists high five(noun) and pray(verb) as valid senses for the 🙏 emoji. For high five(noun), EmojiNet lists three definitions and for pray(verb), it lists two definitions. Wijeratne et al. take all the words that appear in their corresponding definitions as possible context words that can appear when the corresponding emoji (or sense) is being used in a sentence (tweets 1 and 2 in this case). For each sense, they extract the following sets of words from EmojiNet:

pray(verb) : {worship, thanksgiving, saint, pray, higher, god, confession}

high five(noun) : {palm, high, hand, slide, celebrate, raise, person, five}

To calculate the sense of the emoji in each tweet, they calculate the overlap between the words which appear in the tweet with words appearing with each emoji sense listed above. This method is called the Simplified LESK Algorithm [Vasilescu et al., 2004]. The sense with the highest word overlap is assigned to the emoji at the end of a successful run of the algorithm. We can see that 🙏 emoji in Tweet 1 will be assigned pray(verb) based on the overlap of words {god, pray} with words retrieved from the sense definition of pray(verb) and the same 🙏 emoji in Twee 2 will be assigned high five(noun) based on the overlap of word {celebrate} with words retrieved from the sense definition of high five(noun). The above example only shows the minimal set of words that one could extract from EmojiNet. Since EmojiNet senses are linked with their corresponding BabelNet senses using BabelNet sense IDs, one could easily utilize other resources available in BabelNet such as related WordNet senses, VerbNet senses, Wikipedia pages, etc. to collect an improved set of context words for emoji sense disambiguation tasks. Wijeratne et al. further discussed how this idea can be extended with word embeddings learned from Twitter and Google News text corpora and reported their findings in [Wijeratne et al., 2017a]. We encourage the readers to go through Section 6.2 of [Wijeratne et al., 2017a] for more information on their latest studies.

Emoji Prediction and Semantic Change of Emoji Meanings

Word prediction is a popular, widespread functionality found in several computer and mobile interfaces; however, emoji prediction software was not well developed until recently. Barbieri et al. carried out one of the first studies to address the problem of predicting an emoji given a text input [Barbieri et al., 2017]. Undertaken in a dataset of tweets each containing only one emoji, the research intended to guess the exact emoji used in the tweet given the text. They developed a computational model based on Long Short-term Memory Networks that relied on two different embedding models for representing the input sequence: words and character embedding, the latter to better model noisy social media text. Overall, their neural model outperformed competitive baselines based on word-level information. It is noticeable that the good behavior of word-level information, since word-emoji associations are common in the dataset they used (e.g., word “love” and ❤️), however, considering word sequences offers competitive advantages. They extended their work to study the use of visual information (i.e., images) in combination with text to predict emojis on the Instagram social network [Barbieri et al, 2018a]. The interest in the emoji prediction task is probably evident by the large number of research teams that took part in the first Multilingual Emoji Prediction challenge organized at the SemEval 2018 evaluation forum, where 49 and 22 teams took part in the English and Spanish emoji prediction tasks, respectively [Barbieri et al, 2018b].

Out of the problems that we’ve looked at so far, research on emoji meaning change over time has received very little attention. There’s only one recent study by Robertson et al. on Quantifying Change in Emoji Meaning from 2012–2018 [Robertson et al., 2021] that looks at this issue. They identified five patterns in emoji semantic development and reported that the less abstract an emoji is, the more likely it is to undergo a semantic change. They also provided a web interface to experiment with the semantic meaning changes of any emoji that is found in their dataset [Emoji Semantic Change Over Time]. We encourage the reader to explore [Emoji Semantic Change Over Time] web application to learn more about the changes in emoji meanings over time.

Future Research Directions

We will now look at the possible future research directions along the lines of the four emoji understanding problems we discussed in this article. Out of the four problems, calculating emoji similarity and emoji prediction has received the most attention from the research community. The emoji similarity problem has seen the most promising solutions out of the four problems, however, one could experiment with the recent large language models like BERT and XLNet to learn better emoji embedding representations to improve the performance of the downstream tasks such as sentiment analysis and emoji-based search. The emoji prediction problem has also received a lot of attention from the research community recently, however, due to the challenging nature of the problem, the results reported in the laters papers have great room for improvement. For example, solutions that look at the positional embeddings of emoji along with word and character embeddings could be one promising direction to study as research reports that the position of an emoji in a sentence affects the way it is used and interpreted. Emoji sense disambiguation could also be improved by looking at ways to incorporate platform-specific emoji datasets and meanings into the sense disambiguation process. For example, the recent emoji sense disambiguation results reported by Wijeratne et al. on Twitter data can be further improved by utilizing the source attribute returned by Twitter API with each tweet at the data collection time. The source attribute reports the platform in which the tweet was originated, allowing a computer system to utilize that information in the sense disambiguation process. Similarly, recent language models such as BERT and XLNet could also help to learn better emoji embedding models, which could improve the sense disambiguation process. We hope this article motivates the reader to take on the emoji understanding problems discussed above and extend the state-of-the-art.

Emoji Understanding — An Overview was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

What do we do?

At Holler, we aim at enriching your Conversations. We do this by enhancing the text our users write, with Stickers and Gifs. To do this the right way, we need to suggest content in appropriate situations, with the intent that it is shared by the users during their chat. This leads us to an array of algorithms that are a part of our in-house Recommendation Decision System.

These algorithms can be divided into two major blocks, a Generalization block and a Personalization block, While Generalization aims at suggesting a Stream of content appropriate for the situation, it doesn’t account for the user preferences, Personalization, on the other hand, aims at ranking the general stream and curating it for the individual user.

We aim at achieving the said personalization On-device, this helps us with getting results live and offline. There are a few blocks that need to be understood to dive deep into the On-device models.

Next, we talk about the various Sources of Sticker Content, the Streams of content, 2 on-device Storage data structures, and Interleaving(the first block of algorithm that resides on the device and aims at personalization.)

Sources

Branded Content

This is content created for specific campaigns for partnering brands. Although this content is directly tied to a revenue stream we at Holler do not bombard the users with so-called “ads”. Instead, the branded content is interleaved with Organic content to avoid brand saturation for the user.

Organic Content — Historical

This is organic content that the system has published before. We have a lot of user interaction data on this content and hence Server Side Machine Learning models can be applied to the same and they serve as data for the Generalization Block.

Organic Content — Newly On-boarded

The creative team always comes up with new content on a daily basis. The aim is to get this content to the users as fast as possible and to garner as much reaction(shares) as possible. This helps in making an informed decision of what the users want and how receptive they are to the new content.

Streams

We define a list of content as a stream, there are 2 kinds of streams with respect to each device

Incoming Stream

The Stream of content that is suggested to the user after a situation is triggered is called an incoming Stream, a very basic version of a stream that stores only the source of data looks something like this. Where A == Branded, B == Organic

A B A B A B A B A B

After the user has interacted with the Incoming Stream a corresponding Stream that goes out of the device is called the Outgoing Stream. Along with the Source of content, this Stream also stores the info regarding user interaction, namely view or share, and looks something like this

Av Bv Av Bv Av Bv Av Bs Av Bv

Here the s associated with B suggests that organic content was shared during this interaction.

Storage

The next aim is to temporarily store user interactions on the device that could be then used to make decisions instantaneously, without leaving the device. This means that this data storage must be memory-efficient{should not strain the memory resources on the device}, time-efficient{should be easily available to be utilized for instant recommendations} and should adapt to a cold-start problem.

Some streams worth storing could be

1. Source of Content using min-sketch counters and bit vectors
2. The content itself, using bloom filters
3. The message, using Locality Sensitive hashing

Let's draw our focus on the first point and shed more light on the 2 data structures namely, Bit-vector and Min-Sketch Counter

Bit vector

We want to store the information, like the one we saw above, i.e. what source of content did the user share during their last interaction.

A bit vector is an array data structure that saves bits in a compact fashion, It is effective in determining the absence or presence of a particular entity. What we want is to toggle the values of the vector between 0 and 1 based on whether a user has shared content from a particular source during their last interaction.

Min-Sketch Counter

What we want is to get the number of times a user shares the content from a particular source, we can use a hash table that is a simple counter and get away with it when we have only 3 sources of content for a short stream, but as this number increases so will the space complexity of our hash table.

Keeping this in mind we implement a Count Min-sketch, which is a probabilistic data structure that stores a frequency table of events in a stream of data.

There are 3 steps involved in its creation and use,

1. Initialization: a simple matrix (d X w) is initialized to 0, where w is the width, which can correspond to the number of content sources while d is the depth and is equal to the number of hash functions.
2. Increment: As the stream begins to come in, the element goes thru h number of hash functions, and the corresponding locations are incremented by one. This goes on till the end of the stream.
3. Retrieval: now we retrieve the counts of each source which can be used to inform our model and give a snapshot of the same to the algorithm that uses it.

Interleaving

To interleave is to present results from various sources in a fashion such that each source is given an equal opportunity to perform in front of the users. For instance, Sticker content at holler comes from three major sources, branded(A), organic historical(B) and organic newly-onboarded(C), a very basic interleaving would present the results in an equal fashion like

A B C A B C A B C

In order to improve upon this basic approach, there are 2 models which we try to implement around the concept of interleaving.

Static Interleaving

This approach interleaves a ranked list of Historical content with fresh, newly onboarded content. It is static because the interleaving takes place in a fixed ratio of 2:1 and is given to all users. This ensures a chance for the newly on-boarded content to show up to the users instead of being lost at the end of the list.

The benefits of this approach are sure shot but limited, It works well since the count of newly onboard content is usually low compared to Historical content.

For 2 newly on-boarded content static interleaving produces the stream below.

B B C B B C B B B

Once we back the Interleaving with storage and help it learn we get a smart version known as Dynamic Interleaving.

Dynamic Interleaving

We take the concept of Interleaving a step further by making it work on-device, making it personalized for the device user, and helping it learn.

Now, the task of Dynamic interleaving is two-fold,

First, we want to shuffle the order of the sources keeping the ratios intact, this is based on the very last response of the user. The logic behind this is to make the user feel a sense of personalization in the fact that the responses are immediately adapted to user preferences.

For instance, if we start with the first stream a particular user sees

A B A B A B A B A B

And if the user shares B . The outgoing stream looks like

Av Bv Av Bv Av Bv Av Bs Av Bv

The shuffling component use this info and for the next incoming stream we would switch and it will look like

B A B A B A B A B A B A

To favor what the user prefers, albeit temporarily.

And secondly, we want to change the ratio of content of the sources based on the count of the Short Term responses of the users.

For the incoming stream

A B A B A B A B A B A B

If A is preferred on a consistent basis, for instance, the last 10 times have seen a 9:1 or 8:2 in favor of A, then A is preferred more than B, and thus the next time the suggestion becomes

A A B A A B A A B A A B

and so on with the source flipping also into account.

Dynamic Interleaving for Sources of Content.

Let us dive into the 4 step process

1. STM (short term memory) storage using a Bit vector
2. Shuffling component
3. SCM (short count memory) using MinCountSketch
4. Preference Weightage

Example : For three sources A, B, C

{This is a snapshot of results stored in the 2 data structures and not the data structures themselves}

STM == A:0,B:0,C:0

SCM == A:0,B:0,C:0

IN_stream_1 (this is what the user is given when they ask for a recommendation.)

A,B,C,A,B,C,A,B,C,A,B,C

OUT_stream_1. This is what the STREAMS component on_device receives after the user has interacted with IN_stream_1.

Av, Bv, Cv, Av, Bs, Cv, Av, Bv, Cv, Av, Bv, Cv

STM == A:0,B:1,C:0

SCM == A:0,B:1,C:0

IN_stream_2:

A,B,C,A,B,C,A,B,C,A,B,C

Checks STM and SCM

1. The shuffling component activates and reshuffles the stream::

B, A, C, B, A, C, B, A, C, B, A, C. The STM is then washed out.

1. Preference weightage remains inactive since it waits for a certain threshold to be touched before making a decision.
2. Assuming the threshold for SCM is 5, once a single source hits 5(say A) the ratio of that source is doubled

For, STM == A:1,B:0,C:0

SCM == A:5,B:1,C:1 the stream would look like

A, A, B, C, A, A, B, C, B, C, B, C

Wash-Out, Both STM and SCM memory run the risk of being erased or wiped out owing to some reason or the other, while STM is agnostic to this behavior since only the last thing is what matters, the SCM can use the Call-back where at the end of the day each device reports the sum values and thus can be refreshed and resent to the device in the event of an SCM wash out.

Thus with the help of shuffling component and preference weightage, a unique combination of order and ratio is achieved for each device. This personalization is achieved by all data that resides on the device, thus creating self-sufficient on-device systems.

Towards On-device Offline Training Part 1 was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Slot filling is the subsequent task to intent detection and is very critical in Natural Language Understanding. If you are unaware of intent detection, then I would suggest reading my previous article, Intent Detection using Sequence Models for an overview. Intent detection aims to recognize the intention of the user query whereas, in slot filling, we identify the slots in the user query. We can think of slots as the parameters of a user query e.g. “I want to watch The Matrix”. Here, the intent is to watch a movie, and “The Matrix” is the name of the movie that we would like to identify. Slot filling is a key component in task-oriented dialog systems and can be used to better understand search queries.

In this article, we first go through the slot filling problem setup and the dataset used. Next, we use sequence models to solve the problem. We develop our solution in Python using pandas, Tensorflow Keras, and scikit-learn libraries.

Problem

Slot filling is traditionally treated as a sequence labeling problem. Given an utterance, the task is to assign a label to each token in the utterance. The tokens are labeled using an IOB (Inside, Outside, and Beginning) encoding scheme. For example “The Matrix” will be labeled as B-movie_name I-movie_name.

Dataset

To solve this problem, we need a dataset containing utterances that are labeled with slots. For this article, we use the Snips dataset which is a widely used dataset for intent detection and slot filling benchmarking. We take a subset of 7874 utterances containing 35 slot types across 4 intent types (AddToPlayist, BookRestaurant, PlayMusic, and SearchScreeningEvent). The data is stored in plain text files. Let’s go ahead and load the dataset (Figure 1).

Data Preparation

After we load our dataset, we split it into train and test sets. We split the dataset into 80% train and 20% test split using the scikit-learn library. The RANDOM_STATE variable is used to make sure we can replicate the experiments.

(6299,) (1575,) (6299,) (1575,)

After splitting the dataset, we need to make sure it is in the format that can be accepted by our model. First, we initialize the tokenizers using the Tensorflow Keras library. We then use the tokenizers on our train and test datasets to get sequences. We need to pad the sequence to make sure that the length is the same across the samples and datasets. We limit the size of the vocabulary to NUM_WORDS. This allows us to handle out of vocabulary words seen during the test phase using the OOV_TOKEN. We also use one-hot vectors for the slot labels.

(6299, 35, 58) (1575, 35, 58)
(1575, 35, 1) (1575, 35, 1)

Training the Model

Now that the data is prepared, we are ready for the training phase. In this article, we train a bidirectional long short-term memory (BiLSTM) model by using the Sequential class from Tensorflow Keras and adding layers to it. We start with the Embedding layer to represent each word with a vector of length EMBEDDING_DIM. Then, we add a BiLSTM layer by utilizing the Bidirectional wrapper class for the LSTM layer with dimension NUM_UNITS and relu activation. To get the output, we need to use the Dense layer with the TimeDistributed layer as we want the model to produce slot labels at each timestep i.e. for each word. The Dense layer output size is the number of slot labels and has a softmax activation. Next, we train the BiLSTM model and use categorical cross-entropy loss and adam optimizer for training. We also use 10% of the data as the validation set.

We notice the loss on the training set and validation set decreasing as the training progresses. We stop after 7 epochs here but you can experiment with a greater number of epochs too.

Plot Learning Curves

After model training, we plot the learning curves by plotting the loss function for the training and validation sets.

As shown in Figure 2, the loss function on the train and validation set keeps decreasing and starts to become stable with a small gap in-between. Therefore, we can say this is a good model for the data.

Evaluation

Now, let’s evaluate the performance of the model on the test set.

50/50 [==============================] - 0s 5ms/step - loss: 0.1515 - precision: 0.9787 - recall: 0.9531 - accuracy: 0.9624

As we can notice, the accuracy of the model is 96.24% on the test data but don’t let it mislead you. This metric is calculated on all the tokens and most of them will be the Outside (O) tokens. What we are really interested in here would be the performance on identifying the slots. An alternate way is to compute Precision, Recall, and F1-score over the spans of labels. However, it is out of scope for this article.

Prediction using the model

Let’s run our model on an example.

I want to watch The Matrix
['O', 'O', 'O', 'O', 'B-movie_name', 'I-movie_name']

The model correctly predicts the movie name in the input sentence as “The Matrix”.

Summary

In this article, we discussed the problem of slot filling, a key component for Natural Language Understanding. We prepared the data and implemented a BiLSTM model for labeling slots in a sentence. We briefly discussed the model performance and evaluation. We also demonstrated how the trained model can be used to predict slot labels for an input sentence.

A Jupyter Notebook containing the code can be found here.

Resources

If you’re interested in learning more, here are some resources:

• https://machinelearningmastery.com/gentle-introduction-long-short-term-memory-networks-experts/
• https://machinelearningmastery.com/reshape-input-data-long-short-term-memory-networks-keras/
• https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/

References

• Coucke, A., Saade, A., Ball, A., Bluche, T., Caulier, A., Leroy, D., Doumouro, C., Gisselbrecht, T., Caltagirone, F., Lavril, T. and Primet, M., 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
• https://machinelearningmastery.com/develop-bidirectional-lstm-sequence-classification-python-keras/

Thanks for reading! If you have questions or if you would like us to write on anything, please drop a comment or reach out on my website.

Slot Filling using Sequence Models was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

INTRODUCTION

Text classification is a machine learning technique that assigns a set of predefined categories to open-ended text. Text classification is one of the fundamental tasks in natural language processing. Popular examples of text classification include spam classification, sentiment classification, news categorization, chat conversations can be classified into categories, etc.

In this article, we focus on the problem of category classification using word embeddings. The main goal of this article is to explain how Neural Networks work internally. We develop our solution in Python using pandas, TensorFlow Keras, and scikit-learn libraries.

DATASET

We have developed a dataset using publicly available datasets for different categories like Food, Music, Games, Watch Next, Politics, etc.

NEURAL NETWORKS

The basic idea behind a neural network is to simulate lots of densely interconnected brain cells inside a computer so you can get it to learn things, recognize patterns, and make decisions in a human-like way. Neural networks are the usual representation we make of the brain: neurons interconnected to other neurons which form a network. The operation of a neural network is straightforward: one enter variables as inputs (for example an image if the neural network is supposed to tell what is on an image), and after some calculations, the output is returned (following the first example, giving an image of a cat should return the word “cat”).

What does a neural network consist of?

A typical neural network has anything from a few dozen to hundreds, thousands or even millions of artificial neurons arranged in a series of layers, each of which connects to the layers on either side. The first layer is the layer in which inputs are entered, which are designed to receive various forms of information from the outside world that the network will attempt to learn about, recognize, or otherwise process. Other neurons sit on the opposite side of the network and signal how it responds to the information it’s learned; this layer is known as the output layer. In between the input and output layers, are one or more layers of hidden units, which, together, form the majority of the artificial brain. These layers are called hidden layers. Most neural networks are fully connected, which means each hidden unit and each output unit are connected to every unit in the layers on either side. The connections between one unit and another are represented by a number called weight, which can be either positive or negative. The higher the weight, the more influence one unit has on another.

Although a simple neural network for simple problem solving could consist of just four layers, as illustrated here, it could also consist of many different layers between the input and the output. A richer structure like this is called a deep neural network (DNN), and it’s typically used for tackling much more complex problems. In theory, a DNN can map any kind of input to any kind of output, but the drawback is that it needs considerably more training: it needs to “see” millions or billions of examples compared to perhaps the hundreds or thousands that a simpler network might need.

What does a neuron do

First, the neuron adds up the value of every neuron from the previous layer it is connected to. In the above figure, there are 3 inputs (x1, x2, x3) coming to the neuron, so 3 neurons of the previous column are connected to our neuron.

This value is multiplied, before being added, by another variable called “weight” (w1, w2, w3) which determines the connection between the two neurons. Each connection of neurons has its own weight, and those are the only values that will be modified during the learning process.

Moreover, a bias value may be added to the total value calculated. It is not a value coming from a specific neuron and is chosen before the learning phase, but can be useful for the network.

After all those summations, the neuron finally applies a function called “activation function” to the obtained value. Sigmoid, Tanh, RELU, Leaky RELU, Softmax are some examples of activation functions. Now, the neuron is ready to send its new value to other neurons.

How does a neural network learn things?

Information flows through a neural network in two ways. When it’s learning (being trained) or operating normally (after being trained), patterns of information are fed into the network via the input units, which trigger the layers of hidden units, and these in turn arrive at the output units. This common design is called a feedforward network. Not all units “fire” all the time. Each unit receives inputs from the units to its left, and the inputs are multiplied by the weights of the connections they travel along. Every unit adds up all the inputs it receives in this way and (in the simplest type of network) if the sum is more than a certain threshold value, the unit “fires” and triggers the units it’s connected to.

For a neural network to learn, there has to be an element of feedback involved. Neural networks learn things typically by a feedback process called backpropagation. This involves comparing the output a network produces with the output it was meant to produce, and using the difference between them to modify the weights of the connections between the units in the network, working from the output units through the hidden units to the input units — going backward, in other words. In time, backpropagation causes the network to learn, reducing the difference between actual and intended output to the point where the two exactly coincide, so the network figures things out exactly as it should.

CODE

# Import all required libraries
import pandas as pd
import numpy as np
from string import punctuation
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import *
from keras.layers.convolutional import *
from keras.preprocessing import text, sequence
from keras import *
from keras.utils import to_categorical
import gensim
from gensim.models import Word2Vec
from sklearn import *
from sklearn.feature_extraction.text import *
from sklearn.metrics import *
import nltk
nltk.download(‘punkt’)

Here we will be using Google’s Word2Vec for training the neural networks. So we are going to initialize the Word2Vec model and write helper functions that will help us in extracting embedding from the text. Here we are using 300 dimension vectors which will be the input shape to the model.

input_dim = 300

wv = gensim.models.KeyedVectors.load_word2vec_format(“path_to_file/GoogleNews-vectors-negative300.bin.gz”, binary=True)
wv.init_sims(replace=True)

def tokenize_text(text):
    tokens = nltk.word_tokenize(text, language=’english’)
    return tokens

def word_averaging(words):
    mean = []
    for word in words:
        if isinstance(word, np.ndarray):
            mean.append(word)
        elif word in wv.vocab:
            mean.append(wv.syn0norm[wv.vocab[word].index])
    if not mean:
        return np.zeros(wv.vector_size,)
    mean = np.array(mean).mean(axis=0)
    return mean

We need to load the data and do training and testing data splits. The variable out_dim will be equal to the number of class labels we will be having in our data.

out_dim = #some value based on the use case

# LOAD INPUT DATA WITH TWO COLUMNS — ‘TEXT’ AND ‘CATEGORY'
path_to_input_data = ""
data = pd.read_csv(path_to_input_data)

target_names = list(set(data.category.tolist()))

data_train, data_test = model_selection.train_test_split(data, test_size=0.1, random_state=42, stratify = data.category)

We will be using LabelEncoder to encode the target variable and create mappings for each class label in our data.

encoder = preprocessing.LabelEncoder()
y_data = encoder.fit_transform(data_train.category)
y_test_data = encoder_nc.fit_transform(data_test.category)

mapping = dict(zip(encoder.classes_, encoder.transform(encode.classes_)))
print(mapping)

Below is the code for extracting features, training a model, testing it on test data, and getting the performance metrics like Precision, Recall, and F-score.

# generate 2d classification dataset
X = data_train.text
y = y_data

train_tokenized = X.apply(lambda r: tokenize_text(r)).values
X_train_word_average = word_averaging_list(wv, train_tokenized)

test_tokenized = data_test.text.apply(lambda r: tokenize_text(r)).values
X_test_word_average_nc = word_averaging_list(wv,test_tokenized_nc)
train_labels = to_categorical(y)

# Training a sequential model
model = Sequential()
model.add(Dense(300, input_dim=input_dim, activation=’relu’))
model.add(Dense(100, activation=’relu’))
model.add(Dense(out_dim, activation=’softmax’))
model.compile(loss=’binary_crossentropy’, optimizer=’adam’)
model.fit(X_train_word_average, train_labels, epochs=10, verbose=0)

score = model.predict_classes(X_test_word_average)
print(‘Test accuracy:’, score)
print(‘accuracy %s’ % accuracy_score(score, y_test_data))

print(classification_report(y_test_data, score, target_names = target_names))

CONCLUSION

In this article, I went through a detailed explanation of how neural networks work under the hood using mathematical techniques. Knowing the nuts and bolts will fortify your neural networks knowledge and make you feel comfortable taking on more complex models.

Text Classification using Neural Networks was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

AI Research at Holler Technologies

Who has not heard about blockchain these days? We hear about blockchain, coins, tokens, cryptocurrencies, NFTs, etc. almost everyday. We know FinTech and VCs pour a lot of money in this technology for the past few years. But what does that mean? What is it? Where do we start to understand this technology and also what can we do with it? Should a company integrate these into their portfolio of tech solutions? Is it another buzzword or a hype or something really crucial we need to have as a tech company? Where should we even start to navigate these waters, in a technological point of view?

This document is the first of a series on Blockchain Technology to make sense of this technology as a tech business. We will define the major concepts from a business point of view, and answer questions such as does my business need blockchain? How do we implement them? Are there development environments for blockchain? Should we have our own coin, or token, or network? What is a blockchain anyway? What’s a token? Is there any machine learning or AI in blockchain technology? Should there be? There are many questions regarding this developing technology, still in its infancy.

Let’s start from the basics, what is blockchain technology. We will build from there. First we need a good understanding of what that technology is and what it can do, before we can leverage it and use it accordingly. This brief is the journey in learning that technology.

It’s always good to have an example, right? So, we will take the example of Holler Studios, one of our divisions here at Holler Technologies. Holler Studios constantly create original content, in the form of original stickers, characters, GIFs, and such, that we distribute to our partner messaging platforms. These content are suggested by our AI algorithms to relevant conversations so participants tell their stories better using these original content. For example, when you use Venmo and see all these amazing relevant stickers, that’s us. A question we have asked ourselves a while back at Holler was how to protect these content assets as they are in the public domain, even though they are own by Holler? These characters are unique, and Holler should own them. Of course, every company will have different assets to protect, such as supply chains, patents, processes, products, etc. but also lines of codes, ideas, machine learning models, and such.

What is a Blockchain Network?

A blockchain network refers to a distributed, decentralized peer-to-peer network with no central control that represents a shared append-only ledger facilitating the process of recording immutable transactions of valuable assets in a business network. It also allows to track such assets. Anything of defined value or purpose can be an asset, and thus anything can be tracked and traded on a blockchain network. Lots of concepts to define here:

• Ledger
• Transaction
• Asset
• Decentralized
• Shared
• Immutable
• Record
• Network
• Blockchain

We define all these next to give a clear picture.

Distributed Append-Only Ledger

The ledger is a dynamic and distributed set of immutable records, or transactions regarding the assets being managed in the network. It is instantaneously updated across the blockchain network, and since a transaction is recorded exactly once, there is no risk of duplication or errors, or stealing the asset. The ledger is secure in the sense that it is viewed by all, and transaction cannot be modified once recorded.

All network members have a copy of the ledger and then have a full view of the transactions and the assets. They do not rely on a single vulnerable access point to know about an asset and its transactions.

The network is synced automatically through a protocol referred to as consensus. Every time a transaction is recorded, it is automatically propagated through the network for transparency.

Transactions are Immutable Records

Immutable means that the transactions in the shared distributed ledger cannot change over time, and the technology does not enable them to change, or be deleted. It is an important characteristic because it provides transparency and a single shared view of the ledger to the network members, enabling trust, confidence, and protection of the traded/tracked assets. Since a record in the ledger is immutable, no network member can change or tamper with it once it is recorded in the shared ledger.

Transactions can also be associated with a set of rules, called smart contracts. A smart contract is stored in the blockchain network and executed automatically. It precisely defines a change of asset ownership based on certain conditions, or an upgrade in the value of the asset based on particular conditions, or a time limit in the asset life and/or value, and such. For example, we could have smart rules depreciating the asset value automatically over time.

Assets are either Tangible or Intangible

Assets can be either tangible or intangible. Tangible means that the asset is a physical object that can be touched. Examples of tangible assets are: a jacket , a painting, a house, a phone, etc.

Intangible then means the asset has no physical presence and thus cannot be grasped or touched. These types of assets can be ideas, intellectual property, or any other digital or virtual content such as a sticker, a song, or a proprietary machine learning model or notebook or computer program.

How Blockchain Works — An Example

Let’s assume the example relevant to Holler, Holler Studio has created something unique and proprietary to Holler, say an intangible asset in the form of a digital sticker or a digital character, for which Holler fully owns the rights and content, at least at first. Now, we want to record in the some Blockchain Network relevant to stickers, the transaction that Holler has created and owns the particular sticker Asset X. That transaction is recorded as a block of data that enable the existence of the new Asset X in the Blockchain Network. The data block, that has been defined for such cases, records information about the who (unique identification for ownership), the what (unique asset identifier in the network), the when it was created for the first time and attribution to Holler’s full ownership, as well as other information such as the asset initial value, condition, and other attributes relevant to the type of asset that can protect ownership, uniqueness of the asset, and value of the asset to the creator, its first owner. At that time, we also define the smart rules associated to the particular asset.

Now, when the asset is sold, or transformed, or value is increased or decreased due to market or other conditions, or anything else, other blocks of data will be created. Say, for example that the asset is being sold to a particualr person, thus, a record of new ownership is created. These blocks are connected to the blocks already created for the asset, and thus altogether form a chain of data as the asset properties and existence evolve over time. Such asset block chain records and validates the exact time and sequence of transactions that pertain to the asset. The blocks are securely linked to each other, thus protecting the asset and its transactions, as blocks are immutable, thus cannot be altered, deleted, or even new blocks inserted between two existing blocks. This is called a blockchain, and every asset managed has a blockchain. The group of all assets and their associated blockchains are implemented as a blockchain network.

It is important to reiterate that the asset blockchain is irreversible. Each block validates the previous block and thus the entire blockchain is protected. As we have seen, it is impossible to change an existing record, but also to add or fake additional records in the existing chain.

Furthermore, the ledger is distributed among the network members for transparency and trust. These characteristics of a blockchain to be tamper-evident, more secure, and transparent protect against malicious intent, such as transferring illegal ownership of an asset, or who was the initial creator, thus enabling fair monetization to Holler content creations from start, protecting their assets from being stolen. Each asset is fully traceable from its creation, protecting Holler IPs. At Holler, assets to protect are not only Holler Studio content, but also Technology proprietary solutions, such as snippets of code, processes, and machine learning models.

Types of Blockchain Networks

There are 4 types of blockchain networks. Which one to implement depends on the business goals.

1. A Public Blockchain Network allows anyone to join and participate, for example Bitcoin. These networks are less secure and private, as every transaction is visible to the network members. Also, due to the large size, these networks require a lot of energy to operate, as every transaction is sent across the network for visibility. The strength of a blockchain network becomes its drawbacks for these types of networks. A public network is the most visible and therefore the most transparent to the population at large, gaining wider general trust in its transactions. However, it is costly to update all nodes to maintain such transparency, and it is more vulnerable security wise, as transaction creation is more difficult to verify.
2. At the other extreme, a (Fully) Private Blockchain Network is typically run by an organization that fully governs the network, such as who can be a member and participate in creating records, but also in its visibility and transparency as who can track and view an asset blockchain. Also it may seem restrictive, it also boots trust and confidence between the participants of the network. It also ensure greater security of the transaction chains. The business who owns the network has full control on the policies and rules of the network, can revert transactions, and such. The participants are known, limited, and as such, the network collisions and attacks, thus risk are limited. It is also cheaper to operate as the number of nodes can be order of magnitude less than a public network; transactions are cheaper and faster to propagate through the network. Finally, since read permissions are restricted, these networks provide a greater level of privacy.
3. Intuitively a private blockchain network is also a Permissioned (or Semi-Private) Blockchain Network, as specific restrictions are put in place to decide and control who can participate in the network. Public blockchain networks can also be restrictive and permissioned without being private. Participants will need to receive an invitation to participate in the network. Private and Semi-Private blockchain networks closely resemble how a company runs a website. Cost is being reduced compared to Public networks.
4. A Consortium Blockchain Network shares the responsibility and cost of blockchains across several defined organizations, allowing only their members to create and/or view blockchain transactions and assets journeys. The main advantage by limiting memberships is more security but also lower cost to operate, as there are less nodes in the network to update for each transaction being created, and shared cost among the consortium. They extend private networks to a larger, but still controlled audience, while limiting the cost to operate.

Conclusions

In this first article in the Blockchain Technology series, we have looked at basic definitions of a blockchain, the assets, transactions, and types of networks. We have looked at an example here at Holler with the Holler Studios content. Next, we will go deeper into tokens and coins. Stay tuned.

Blockchain Technology Explained — Part 1 was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we live in a society where the Internet and online communication have become a major part of our everyday life. Especially with the recent COVID-19 pandemic, the number of hours we spend in front of a screen has increased dramatically. Early surveys reported nearly 70% increase in the Internet use during the pandemic, out of which, half of the time was spent on social media [Beech, 2020].

People rely on social media to share and receive information including the latest news, communicate with their networks including friends and family, learn new things, develop their interests, and be entertained among others. In 2019, it was reported that more than 70% of Americans use social media sites everyday [Pew Research, 2019] while YouTube and Facebook being the most popular ones among them [Pew Research, 2021]. Majority of the users access these sites on a daily basis, however, the sentiment associated with social media use still tend to be negative [Pew Research, 2020]. Research suggests that sharing of misinformation and online hate and harassment in social media are the two main reasons why people have a negative sentiment towards social media websites. Recent studies show that roughly four-in-ten Americans have experienced online harassment on these sites [Bertazzo, 2021] and 79% of the social media users believe that the social media companies are doing a fair or poor job at addressing online harassment on their platforms [Pew Research, 2021]. In this article, we will look at the challenges in offensive language identification, which is the language often used in online harassment.

Offensive Language and Offensive Language Identification

Offensive language or profanity is the use of certain words and phrases that are considered by some to be rude, impolite, offensive, obscene, or insulting [Wang et al., 2014]. People also refer to offensive language use as swearing, cursing, or using bad language. The use of offensive language occurs more often than we think in our day-to-day conversations. [Mehl and Pennebaker, 2003] reported that 0.5% to 0.7% of all words we speak are curse words, given that 1% of all the words we speak are first-person plural pronouns (e.g., we, us, our). In another study, [Jay, 2009] found 70 curse words in an 11,609-word tape-recorded conversations of elementary school students and college students. Use of offensive language is common across social media platforms too. By analyzing 51 million tweets posted by 14 million Twitter users, [Wang et al., 2014] reported that curse words occurred at a rate of 1.15% on Twitter, and 7.73% of all tweets in their dataset contained curse words. Similar studies done by analyzing the language used in public and private Facebook posts report that the most common offensive words are used millions of times in Facebook posts, everyday [WeRSM, 2013].

Offensive language identification in social media is the process of identifying and detecting the user generated offensive content or comments that are hateful in nature that target an individual or a group [Zampieri et al., 2019]. Recent research studies the problem of offensive language identification via a three way schema that looks at (i). Offensive Language Detection where the focus is on detecting whether a social media post contains any form of offensive content (aggressive, sexist, racist, etc.) or not, (ii). Categorization of Offensive Language which focuses on detecting whether an offensive message contains a targeted insult (towards an individual or a group) or not, and (iii). Offensive Language Target Identification where the focus is on detecting the targeted individual, group, or entity (a business organization, an event etc.) present in an offensive message [Zampieri et al., 2019a]. Online offensive messages could be a result of many social media activities that link to online harassment, cyber bullying, expressing hate, aggression, racism, sexism and research has been conducted covering many such online activities where the identification of offensive language or words lies at the heart of solving them [Hate Speech Data]. However, the problems such as detecting the relationship between the sender and the receiver of an offensive message exchange and identifying offensive messages that are associated with non-textual elements present in social media messages such as emoji or stickers are not extensively studied [Kirk et al., 2021]. We will go over some of those issues later in this article.

Types of Offensive Messages

In this section, we will briefly go over some of the common offensive message types we find on social media platforms and in major datasets available for researchers who work on offensive message identification and filtering. The following is a subset of the offensive message types identified by Founta et al., in their work [Founta et al., 2018].

1. Offensive Language — Profanity, strongly impolite, rude, or vulgar language expressed with fighting or hurtful words in order to insult a targeted individual or group.
2. Abusive Language — Any strongly impolite, rude, or hurtful language using profanity, that can show a debasement of someone or something, or show intense emotion.
3. Hate Speech — Language used to express hatred towards a targeted individual or group, or is intended to be derogatory, to humiliate, or to insult the members of the group, on the basis of attributes such as race, religion, ethnic origin, sexual orientation, disability, or gender.
4. Aggressive Speech — Overt, angry and often violent social interaction delivered via electronic means, with the intention of inflicting damage or other unpleasantness upon another individual or group of people, who perceive such acts as derogatory, harmful, or unwanted.

As you may see, the above common offensive message type definitions overlap in terms of the language/offensive words used, which makes it difficult to generate computer models that clearly distinguish such messages. In the next section, we will look at such challenges.

What Makes it Difficult to Build Computer Models to Identify Offensive Messages?

Offensive message identification and filtering is a difficult task due to many reasons. In this section, we briefly go over the challenges in offensive message identification that are related to training data collection, overlapping definitions in offensive message types, issues arise due to the relationship between the sender and the receiver of an offensive message exchange, and offensive messages that are associated with non-textual elements such as emoji and stickers which could change the semantics expressed in the message text making it offensive.

1. Training Data Collection — To train any supervised training model, we need access to labeled training data. In the context of offensive message filtering, we have to have offensive messages posted on social media websites in our labeled training dataset. Most of the social media websites use human moderators to weed out the offensive messages posted on their platforms. Similarly, the majority of the social media sites let the users of their sites to report any offensive content to the moderators, so that the moderators can remove them from their websites. Even though this seems an inefficient way of removing the offensive messages, eventually they will be removed from the social media websites. Therefore, searching for social media websites for offensive messages can be harder than one would think. Moreover, many social media websites don’t let the raw message text to be shared with others for research purposes, but the message IDs. More often than not, retrieving labeled datasets used by other researchers using the message IDs results in missing data due to the fact that the removal of offensive messages by site moderators by the time of message retrieval. Thus, collecting a labeled offensive message training dataset could be a major challenge faced by researchers. For a list of already published offensive message datasets that are commonly used in research studies, please visit [Hate Speech Data].
2. Class Imbalance and the Language Used in Training Data — Even though 0.5% to 0.7% of all words we speak are curse words [Mehl and Pennebaker, 2003], non-cursing words still largely outnumber curse words in labeled datasets. For example, if we used the dataset collected by [Wang et al., 2014] in their experiments to learn a binary classifier that labels whether a message is offensive or not, our offensive dataset would still consist of 7.73% of all tweets in the dataset. This means, a little more than 92% of our training data would still be non-offensive text, which leaves us with an imbalanced dataset. This problem becomes even worse if we try to introduce fine-grained offensive message types such as Racist, Sexist, Aggressive messages etc. into our classifier instead of an umbrella class to represent all offensive messages. For example, if we wanted to develop a multi-class classifier that labels a message as Racist, Sexist, Aggressive, or Non-offensive using [Wang et al., 2014]’s dataset, the 7.73% of offensive tweets will be further divided into Racist, Sexist, Aggressive classes, making the dataset further imbalanced. Similarly, the same set of curse words are used across the above fine-grained offensive message types, making it challenging to develop multi-class classifiers that rely only on word-level linguistic features. Such simple multi-class models could be helpful when one wants to deploy their offensive classifier on resource-constrained devices such as mobile phones where the same classification techniques and linguistic features one could use on a server-side environment might not be available on client-side (e.g., The TensorFlow-Lite machine learning model deployment platform for resource-constrained devices doesn’t support all TensorFlow functionality [see here]).
3. Sender-receiver Relationship of an Offensive Message Exchange — Not all messages that contain offensive words are hateful [Jay, 1992]. For example, close friends often use curse words in their social media messages targeted at each other even though they don’t wish to harass their close friends. A majority of offensive message identification models that are based on word-level features rely on curse word lists [Xu, 2010] or they want the classifier model to automatically learn the relationships among curse words and the other words appearing with curse words to learn features to classify offensive messages. Therefore, not knowing the relationship between a sender and a receiver in an offensive message exchange could result in classifying messages that are not meant to harass the receiver as offensive. This problem is very difficult to address, especially using social media datasets as learning the relationship between two random individuals that exchange offensive messages on a social media website is a very difficult problem. Due to this reason, there has not been much research conducted in this area and it continues to exist as an open challenge for offensive message identification.
4. Issues with the Presence of Visual Communication Elements in Textual Data — Visual communication is a powerful tool for expressing non-verbal cues such as gestures and emotions which are otherwise difficult to express by only using words in online messaging [Holler, 2021]. Ranging from emoticons [Wikipedia] to Facebook’s latest addition to the spectrum of the visual elements, Soundmoji [Majcher, 2021], these pictorial characters give great expressive power to the users of online social media websites and messaging platforms. The presence of visual elements in social media messages has increased dramatically in recent years and become an important part of text classification systems [Emoji Workshop, 2022]. Thus, it is important to consider visual communication elements when building offensive message identification systems that can process the latest messages posted on social media websites. Past research has shown that visual communication elements such as emoji can take different meanings based on the context of their use [Wijeratne, 2017]. Oftentimes, certain emoji can be associated with offensive meanings as well, making them a tool for social network users to express hate towards others [Kirk et al., 2021]. A recent example for using emoji to express hate would be the racist social media comments received by Bukayo Saka, Jadon Sancho, and Marcus Rashford following England’s defeat in the Euro 2020 soccer final [Jamieson, 2020]. The angry social media users across all major social media platforms used emoji such as 🐒, 🐵, 🍌, and 🍉 to throw racial insults towards the players of color who play for the English Football team. Handling nuances such as emoji meanings in offensive message identification is still at its infancy. A recent work by [Kirk et al., 2021] was the first one to investigate this issue. They created a labeled training dataset consisting of offensive messages with emoji and showed that the current offensive message identification systems can be improved by using emoji presented in the context of a social media message. Even though their initial results are promising, there’s much more work needed to be done in this area. For example, there are other non-verbal communicative elements such as stickers and GIFs that could cause similar issues. For example, an image search tool developed by Google incorrectly labeled photos of black people as gorillas [BBC, 2015]. Similar references could be used as in the previous case with emoji but using GIFs and stickers to throw racist messages at people of color. Thus, more research needs to be done in this area to learn the relationships between the targets involved in the offensive message exchange and the visual elements used.

Conclusion

In this article, we looked at why offensive message identification is a difficult problem to solve using computer models. We discussed the negative sentiment associated with social media in general due to the presence of hate in social media websites, we defined what is an offensive message and what is meant by offensive message identification by explaining a three way schema proposed in recent research, we looked at different types of offensive messages found on social media sites, and discussed why it is difficult to build computer models that correctly distinguish them. We also shed light on research areas related to offensive message filtering that have not been extensively studied. We hope this article motivates the reader to take on the less-worked on research areas related to offensive message identification and filtering.

Why Offensive Message Identification is Hard? was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hackathons are events conducted by gathering people from different teams to collaborate, brainstorm, develop and deliver a tangible product within a short period of time.

Here are a few reasons why we think hackathons are important to the tech organization (startups especially).

Innovation and creativity:

Hackathons are proven to help in accelerating innovation by stockpiling digital ideas and solutions demonstrating the possibilities of open innovation. They encourage company-wide innovation and creativity by allowing participants to run wild with their ideas. They’re a perfect environment for experimentation with low failure costs. Even in failure cases, the lessons learned will be invaluable. Stepping away from regular sprints and project deliverables, hackathons help nurture creativity by allowing us to come up with innovative ideas which can potentially create our next big successful product.

Rapid Prototyping and MVPs:

Hackathons are the ideal place where cutting-edge technology meets real-world applications with a richer exchange of ideas. The business product is different from the business prototype. A prototype needs to demonstrate the ability to conceptualize an idea into a substantial output. The important thing is understanding of the idea being possible, then the prototype can be tested to gather feedback to refine and rebuild. At a Hackathon, we need not perfect a product, instead, we must rapidly construct a prototype of our idea. The fast-paced nature of a hackathon is perfect for testing ideas and failing fast. Hackathons allow groups to communicate the value of their hack to an audience during the final demo.

Cross-Functioning and Learning:

With deadlines and deliverables, we get minimal chances to work cross-functionally and learn about other parts of the organization in our day-to-day work. Hackathons promote cross-functional learning and development by letting participants choose to work across teams to solve a bigger problem. Hackathons encourage participants to experiment with new cutting-edge technologies and methodologies. Their open environment is ideal for developers to make meaningful connections. With limited time, people quickly learn how to ask for guidance from others. The Hackathon environment allows participants to coordinate easily and learn from each other, targeting one common goal to be achieved. Exposure to several participants from different backgrounds and varying skill sets facilitates acquiring expert knowledge in a more casual environment.

Diversity and Collaboration:

Hackathons aren’t all about being innovative. They’re also about sharing knowledge and encouraging each other. That’s why it’s important to encourage diverse participation. Hackathons promote the collaboration of tech and business experts in solving real-world problems. Having an open, collaborative format, hackathons open doors to professionals from diverse backgrounds to solve challenges together. The best result out of this can be achieved by making it a well-rounded approach for innovation.

Bridge tech and talent gaps:

Hackathons are an effective platform to bring industry challenges and business challenges to light and discuss their solutions. In a world of fast phased digital transformations, businesses need to engage with the latest technology, else risk getting left behind. Hackathons shed light on any talent gap or technology gap that companies might possess. Post hackathon feedback and analysis can help an organization to identify and take action on the areas that need any attention.

Tips: For running a successful hackathon

1. Set a theme or a clear goal

2. Plan ahead on logistics and dates

3. Motivate and engage people

4. Be communicative and responsive

5. Streamline team formation and management

6. Defining and following up on checkpoints

7. Present demos and share the success amongst the organization

8. Have a post Hackathon Plan (Feedback and potential expansion on the hackathon outcome product)

Hackathons-A Powerful Stimulus tool for Innovation and Growth was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the downstream purposes of data wrangling, as mentioned above, is data analysis.

Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. In today’s business world, data analysis (descriptive analytics, exploratory data analysis, predictive analytics) plays a role in making decisions more scientifically and helping businesses operate more effectively.

In this article, we focus on processing big data. We first will take a look at what Pandas offers in data processing, limitations of Pandas and then will dive into Apache Spark for processing big data.

Pandas is one of the most famous data science tools and it’s definitely a game-changer for wrangling and analyzing data. Pandas is a fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation tool.

Let us now look at a few examples of pandas.

Reading the data from a CSV file

We can either read from a CSV file or create a Dataframe directly. For simplicity, we will create a Dataframe manually and work with a small dataset in this article (but, including the code to read from a CSV file too).

import pandas as pd
data = pd.read_csv(path_to_file)
data.head()

# Creating a DataFrame manually
d = {'id': [1, 1, 1, 1, 2, 2, 2, 2, 2 ,2], 'words': ['hello', 'how are you', 'pizza', 'thank you', 'lol', "what's up", 'beer', 'i love you', 'good morning', 'rent'], 'value': [100, 300, 50, 10, 100, 344, 588, 22, 211, 323]}
data = pd.DataFrame(data=d)

Grouping by a certain column to generate counts and aggregations

data_grouped = data.groupby('id').count().reset_index()

Applying Lambda functions on the data frame

Applying a small lambda function that returns the number of words in a string by splitting it on whitespace.

def number_of_words(text):
    return len(text.split(" "))

data[‘total_words’] = data['words'].apply(lambda x : number_of_words(x))
data.head()

We apply all these operators/functions as part of our day-to-day analysis of the data. Let’s say we have a dataset of 100K records. We will not have any issues running all the above code on it. Now, let’s say we have a dataset of 1 million records. We might still not have any issues but it might take some time to finish the task. Let’s say we now have a dataset of 1 billion records. We would get memory issues considering the amount of memory Pandas needs since it has limitations on the size of the data it processes.

WE NEED TO VERTICALLY SCALE THE RESOURCES IN ORDER TO PERFORM THE TASK WHICH IS NOT ADVISABLE AND EFFICIENT.

Here comes Apache Spark!!

Apache Spark is a distributed data processing platform specialized for big data applications. It has become the de-facto standard in processing big data. By its distributed and in-memory working principle, it is supposed to perform fast by default.

Let’s take a look at the features of Spark.

• Swift Processing: Using Apache Spark, we achieve a high data processing speed of about 100x faster in memory and 10x faster on the disk. This is made possible by reducing the number of read-write to disk.
• Dynamic in Nature
• In-Memory Computation in Spark: With in-memory processing, we can increase the processing speed. Here the data is being cached so we need not fetch data from the disk every time thus the time is saved. Spark has a DAG execution engine that facilitates in-memory computation and acyclic data flow resulting in high speed.
• Reusability: We can reuse the Spark code for batch-processing, join streams against historical data or run ad-hoc queries on stream state.
• Lazy Evaluation in Apache Spark
• Fault Tolerance in Spark: Apache Spark provides fault tolerance through Spark abstraction-RDD. Spark RDDs are designed to handle the failure of any worker node in the cluster. Thus, it ensures that the loss of data reduces to zero.

Execution Model

In contrast to Pandas, Spark uses a lazy execution model. So when you apply some transformation to a DataFrame, the data is not processed immediately. Transformations are lazy in nature and execution will not start until an action is triggered. Let us take a look at how Spark stages tasks internally:

1. A logical execution plan is created from the transformations which essentially is a graph (a DAG) where nodes represent operations (like reading data or applying a transformation).
2. Now an analyzed execution plan is derived where all column names and external references to data sources are checked and resolved.
3. Spark iteratively applies the available optimization strategies to transforms the analyzed execution plan into an optimized execution plan. For example, filter operations are pushed as near as possible to the data sources in order to reduce the number of records as early as possible.
4. Finally, the result of the previous step is transformed into a physical execution plan where transformations are pipelined together into so-called stages.
5. The physical execution plan then is split up along the data into so-called tasks, which then can be executed in parallel and distributed by the machines in the cluster.

It is important to understand that by lazily following execution plans, the total working data set does not ever completely materialize into RAM at any point in time. Instead, all the data (including input data and intermediate results) is split up into small chunks, which are processed independently and even results are eventually stored within small chunks and those never need to fit into RAM at once.

Spark Architecture Overview

The Apache Spark framework uses a master-slave architecture that consists of a driver, which runs as a master node, and many executors that run across as worker nodes in the cluster. Spark can also be used for batch processing and real-time processing.

The driver is responsible for analyzing, distributing, monitoring, scheduling, and maintaining all the necessary information during the lifetime of the spark process. Executors are only responsible for executing the part of the code assigned to them by the driver and reporting the status back to the driver. Each spark process would have a separate driver and exclusive executors.

Deep Dive into Spark Architecture

The driver program, in the master node, drives our application. The code you are writing behaves as a driver program or if you are using the interactive shell, the shell acts as the driver program.

Inside the driver program, the first thing you do is, create a SparkContext. SparkContext is the entry point of Spark functionality. The most important step of any Spark driver application is to generate SparkContext.It is similar to your database connection. Any command you execute in your database goes through the database connection. Likewise, anything you do on Spark goes through Spark context.

Now, this Spark context works with the cluster manager to manage various jobs. The driver program & Spark context takes care of the job execution within the cluster. A job is split into multiple tasks which are distributed over the worker node.

Worker nodes are the slave nodes whose job is to basically execute the tasks. These tasks are then executed on the partitioned data in the worker node and hence return back the result to the Spark Context.

​​To combine it all together, in short, SparkContext takes the job, breaks the job into tasks, and distributes them to the worker nodes. These tasks work on the partitioned RDD/DataFrame, perform operations, collect the results and return to the main Spark Context.

Now that we have learned about Spark and its working, let us take a look at some examples of data aggregations, manipulations in PySpark.

Reading a CSV file into a Spark data frame

We can create a Spark Dataframe by reading from a CSV file or manually too. Let’s convert the Pandas Dataframe we created above into a Spark Dataframe.

# importing required libraries
import pyspark.sql.functions as pf
from pyspark.sql.types import *
from pyspark.sql import Window
from pyspark.sql.functions import pandas_udf, PandasUDFType

data_spark_csv = spark.read.csv(path_to_file, header=True)
display(data_spark_csv)

# Converting Pandas Dataframe to a Spark Dataframe
data_spark = spark.createDataFrame(data)

Anytime a DataFrame is created in Spark context, it will be distributed across various nodes and can be cached there. And any operations/transformations we do on the DataFrame’s will also be distributed across the worker nodes.

Grouping by to perform aggregations

data_grouped = data_spark.groupby("id").count()
display(data_grouped)

data_group2 = data_spark.groupby('id').agg(pf.sum('count'))
display(data_group2)

data_group3 = data_spark.groupby('id').agg(pf.collect_set('words').alias('words_list'))
display(data_group3)

User-Defined Functions

Spark also provides features for User-Defined Functions that can be used in data manipulation and data analysis. We can write UDF's to work on each row and generate a single value for each of them.

def length_of_words(term):
    return len(term.split(" "))

length_of_word_udf = udf(lambda x : length_of_words(x), IntegerType())
data_spark = data_spark.withColumn('length_of_word', length_of_word_udf(data_spark.column1))

Window Functions

Spark also provides features for Window Functions which can operate on a group of rows (like frame, partition) and return a single value for every input row.

w = Window.partitionBy('id').orderBy('value')
data_spark_words_list = dd.withColumn('words_list', pf.collect_list('words').over(w)).withColumn('value_list', pf.collect_list('value').over(w))
display(data_spark_words_list)

Grouped Map Pandas UDFs

Grouped map Pandas UDFs are designed for split-apply-combine patterns in data analysis, and they operate on all the data for some groups, e.g., “for each ID, apply this operation”. Grouped map Pandas UDFs first splits a Spark DataFrame into groups based on the conditions specified in the Groupby operator, applies a user-defined function (pandas.DataFrame -> pandas.DataFrame) to each group, combines, and returns the results as a new Spark DataFrame. Pandas UDFs optimized and faster for grouped operations.

schema = StructType([
    StructField("id", IntegerType()),
    StructField("words", StringType()),
    StructField("value", IntegerType()),
    StructField("total_words", IntegerType()),
    StructField("mean_subtracted", IntegerType())

])

@pandas_udf(schema, PandasUDFType.GROUPED_MAP)
def subtract_mean(pdf):
    return pdf.assign(mean_subtracted=pdf.value - pdf.value.mean())
data_pandas_udf = data_spark.groupby('id').apply(subtract_mean)
display(data_pandas_udf)

The best part here is that the data is distributed to all the worker nodes and aggregations are done on big data and results can be achieved efficiently using Spark.

P.S: The goal of the article is just to show the functionality of Spark and what it offers (with some examples). The data used in this article is a very small one. So if you want to test them out, try them on large-scale datasets.

Conclusion

Spark is inherently multi-threaded and can make use of all cores of your machine. By breaking down the total amount of work into individual tasks, which then can be processed independently (as soon as each task’s input data is available) in parallel, Spark makes very efficient use of the available cluster resources.

Spark also scales very well with huge amounts of data. It does not only scale via multiple machines in a cluster, but the ability to spill intermediate results to disk is at the core of the design of Spark. Therefore Spark is almost never limited by the total amount of main memory, but only by the total amount of available disk space.

The virtually unlimited scalability both in terms of data and processing power makes Spark a distributed and parallel relational execution engine for processing big data.

Processing the BIGG data was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Matthew Binshtok, Software Engineer, Holler

At the beginning of July, I celebrated two years working at Holler and I used the occasion as a time for self-reflection. Eventually, this led me to consider the technical knowledge and skills that I wish my classes introduced while I was completing coursework at my university. There are bootcamps that cover some of these skills, and during my junior year I participated in one such program. But in my opinion, practical skills like learning to use Github are just as fundamental to a career in computer science or software engineering as knowing how to find the time complexity of an algorithm or understanding how Turing machines work. My hope is that any educators that come across this article try to integrate at least one of these topics into their curriculum, that students find this to be a useful resource to help them prepare for internships and entry-level roles, and that developers keep these topics in mind when mentoring young software engineers.

Here are the six software engineering skills I wish I learned in my computer science courses:

1. Code Reviews

In my opinion, this one is a no brainer. For all the emphasis universities place on collaborative work as a part of the curriculum, professors do not explain how to effectively give feedback to your peers, or how to productively receive feedback about your code and learn for the future. Code reviews can help students see new solutions and better understand ideas that transcend any specific project. A benefit of becoming accustomed to the code review process is learning to consider how others will view your code. For junior engineers looking to impress, feedback about their code can often be misconstrued as criticism, when in fact it’s just feedback to help them become better engineers. Because it may be difficult to integrate code reviews into a course, a terrific alternative would be exercises to walk through code, explain concepts encoded to peers, as well as explain any assumptions made. In the wise words of Uncle Bob, the only valid way to measure clean code is WTFs/minute during a code review…

2. Debugging

print("wtfffff", badlyChosenVariableName)

Any student who has slammed their head on a keyboard at 11:59PM trying to find the mistake in their code will be familiar with the line above. It’s an easy thing to copy and paste throughout your code to assess how a program is behaving and whether or not it aligns with your expectations of how it should run. Those familiar with using debuggers know that they do a lot of this work for you (and more!), but learning to use a debugger can be intimidating. It often takes some system setup unless you’re using a fancy IDE (most students aren’t), and isn’t something that students may see as worthwhile to pick up on their own. Using a debugger effectively is an important skill that universities should teach formally. Even simple exercises like directing students to fix broken code using a debugger will enlighten them to the benefits of using one to step through code and making it a part of their development routine.

By the way, professional IDEs like JetBrains, and other tools are available for free for students! Check out Github’s compilation of offers for students here: https://education.github.com/pack

3. Testing

Spending an eternity debugging a large program is avoidable if your code is both well-organized and well-tested. These two concepts can be taught hand-in-hand as good coding practices that help students understand the benefits of investing time in considering the use cases prior to implementing a program. Even within implementation, teaching students to write smaller, independently testable functions starts building good habits. An important side effect of understanding good practice here is learning to think about special circumstances and edge cases as you’re writing code instead of waiting until you’ve completed the implementation. Otherwise, once contributing code in a professional environment, students will discover the cruel truth in Murphy’s law that anything that can go wrong, will go wrong.

4. Change Management and Deployment

For many students, one of the biggest hurdles in starting as a contributor at their first internship or full-time job is learning the basics of Git, Github, version control, and more generally the process by which their code goes from running on their machine to running in production environments. Introducing these concepts in college courses teaches students a few things:

• To keep their code in a working state more frequently
• To view their code as the beginning of a larger pipeline
• To determine how to change their code in ways that can be rolled back later
• To build confidence in functionality before it ends up in production

Furthermore, there are important practical topics such as CI/CD (continuous integration and continuous deployment) that are very difficult to understand without the basics of change management.

5. The Cloud and Cloud Service Providers

If we’re being honest, there could be an entire course for this topic, but the reality is that most students graduate university without knowledge of the cloud or its capabilities. Among curious and excited CS students, many have an itch to run their programs and applications in a way that other people can access them. Introducing the main services that cloud providers offer and their frequent use cases can catalyze students to explore the possibilities that exist for deploying their projects. The cloud is rapidly evolving, and thus the available technologies are evolving, but that should not deter instructors from providing students with a glimpse of this important space. Here are a few questions that can serve as the foundation for a simple introduction to the cloud:

• What is “the cloud”?
• What are some major public cloud providers?
• How can I allow real users to access my applications?
• What are the most common use cases for the cloud?

6. Breaking Projects Down

Last spring, Holler brought on two interns from my alma mater, Hunter College, as part of the CUNY 2X Tech Talent Pipeline. CUNY 2X Tech is a city-led initiative to double the number of CUNY graduates with CS degrees, and assist them in finding employment in the NYC Tech scene.

I sat down with the engineering team to brainstorm attainable goals for the interns as part of the overarching theme of preparing them for their future full-time roles, whether it be with us or another organization. Among the goals we drew up for them was one I felt very strongly about: learning to break down large, abstract requests into implementable pieces of work.

In university coursework, the instructions will say that a submission for a programming assignment has to do a number of highly specific things. It is usually pretty easy to translate those requirements into code. Things are a lot messier in the software industry — the problem statement is not so well-defined and all the requirements are not fleshed out. Aspiring software engineers would benefit from practicing things like:

• Clarifying the problem space and requirements
• Thinking about distinct implementable components
• Defining dependencies on other developers or systems
• Logically sequencing the work that has to be completed

This is an important step for aspiring software engineers to move from a university setting to a professional setting. The way one completes assignments for a course is inherently different to the way one contributes to continuously evolving projects.

I spent a lot of time with the interns working through the practices listed above. In the last week of their internship, they received tremendous acclaim from the rest of the company after presenting their projects. I believe that a large part of their success was due to their dedication to understanding and using these organizational principles.

Honorable Mentions

Before wrapping up, I want to give a special shoutout to two people who helped me brainstorm the skills for this article! Thank you to Maria Mahin of Disney Streaming Services, a fellow 2019 Hunter alum, and Kevin Liang of Hubspot, Northeastern University ’19, for their aid in drafting the list for this article.

Below is a list of additional topics/skills for which even a brief introduction to these concepts can go a long way in shaping how entry-level engineers think about software development.

1. Containers

What are they and how are they useful? Understanding common technologies for containers and cluster management/deployment would also be beneficial. This is the way of the tech world.

2. Technical Documentation and Design Documentation

In university, the goal is to pass your coursework and show that you know the concepts. In the professional world, technical and design documentation are a pre-requisite for building anything, even if the design is on the back of a napkin.

3. API Development

What is an API? What are some different types? What are best practices to consider when writing one? The RESTful API standard…

4. Web Development

How does the web work? Basic JavaScript, HTML, CSS. What are some common UI component frameworks? What is asynchronous programming?

5. Observability

What do you need to include in production code to make it easier to support that code. Until you have to fix something in the middle of the night, you don’t really have an appreciation for what it takes to make code easier to support.

6. Databases

Best use cases, SQL vs NoSQL, schema design, indexing, performance etc.

7. Best Practices for Writing Clean Code

Emphasizing things like modularity, naming conventions, reusability, separation of concerns, etc.

8. Contributing to an Existing Codebase

Being able to take code that someone else has written, logically trace through its execution, and build from it. Also, best practices to make your code easily extensible.

Six Software Skills I Wish I Learned in College was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Text Classification in Natural Language Processing (NLP) is the task of assigning a class from a set of predefined classes to a given piece of text (document/paragraph/sentence). Popular examples of text classification include spam classification (classifying an email into spam or non-spam), sentiment classification (classifying polarity of text into positive, negative, or neutral), intent detection (identify the intention of the user like play music, book restaurant, book flight, etc.), and news categorization (classifying news articles into categories such as business, sports, politics, science, tech, etc.). If the number of classes is two, the task is called binary text classification and if the number of classes is more than two, then the task is called multi-class text classification.

In this article, we focus on the problem of intent detection. We first introduce the task of intent detection and the dataset used. Next, we introduce the motivation behind using sequence models and show how they can be used to solve the task of intent detection. We develop our solution in Python using pandas, TensorFlow Keras, and scikit-learn libraries. We won’t delve into the technical explanations of machine learning concepts in the interest of time, however, we will learn how to use them along with a lot of code.

Intent Detection

Intent detection aims to recognize the intention of the user query i.e. an action that the user wants to perform e.g. “Play music on YouTube Music”, where the intent is to “play music”. The user query is a single sentence and can originate from a written or spoken utterance. Intent detection is a very crucial task in Natural Language Understanding and is usually modeled as a classification problem. Given an input sentence, the objective is to predict an intent for the sentence from a set of predefined intents.

Dataset

To solve this problem, we need a dataset containing utterances that are labeled with intent. For this article, we use the Snips dataset which is a widely used dataset for intent detection benchmarking. It consists of 14484 utterances across 7 intent types. The dataset is stored in a JSON file and we start by loading it into a pandas DataFrame and plotting the class distribution.

Data Preparation

As shown in Figure 1, the class distribution is balanced. If you’re working on a dataset with severe class imbalance then it might be helpful to balance the dataset. There are various techniques to tackle the class imbalance problem which are out of scope for this article. Now, we split our dataset into train and test sets to train a machine learning model and evaluate its performance. We split the dataset into 80% train and 20% test using the scikit-learn library. To make sure that we can replicate the results in the future, we also set the RANDOM_STATE variable.

(11587,) (2897,) (11587,) (2897,)

Sequence Models

Sequence models deal with supervised learning tasks where either model input or model output is a sequence. Sequences can be text, audio, video, temporal data, or any other sequential data. Examples of sequence modeling include sentiment classification, music generation, and machine translation. Recent advances in deep learning particularly in the area of sequence models have revolutionized the world of NLP, thereby establishing themselves as a dominant paradigm for training models for language understanding tasks. Sequence models like recurrent neural networks (RNNs) and transformers have consistently achieved state-of-the-art performance on a number of benchmark NLP tasks. In this article, we use an architecture related to RNNs, long short-term memory (LSTM) to solve the task of intent detection.

First, we need to prepare our input text for use in training. We tokenize the text and convert it into a sequence of integers by using the Tokenizer from Tensorflow Keras. Then, we pad the sequences to be of the same length for modeling as required by Keras.

(11587, 35) (2897, 35)

Next, we prepare one-hot vectors for labels by using the LabelEncoder and to_categorical function.

(11587, 7) (2897, 7)

Training a Sequence Model

Let’s define & train our model now.

LSTM

We define a Tensorflow Keras Sequential model and add layers to it. The first layer is the Embedding layer that is used to represent each word with a vector of fixed length 16. The next layer is the LSTM layer with 16 units with a relu activation. Next, we have a Dense layer with 7 units with a softmax activation for classification. Then, we compile the model with adam optimizer, categorical cross-entropy loss, and evaluate performance metrics like precision, recall, and accuracy. Finally, we fit our model on the training dataset with a batch size of 32 for 7 epochs. We also use 10% validation data for the evaluation of performance metrics at the end of each epoch. The fraction of the training data to be used for validation is specified using VAL_SPLIT.

As the training progresses, we can notice the precision, recall, and accuracy on the training set increasing. The same effect can be noticed in the accuracy on the validation data.

Plot Learning Curves

Once we fit the model, we look at the learning curves by plotting the loss function for the training data and the validation data.

In Figure 2, we notice the loss on training and validation data to be decreasing continuously and getting more stable with a small gap in-between, therefore, we can conclude that our model is not overfitting or underfitting and is a good fit for the data.

Evaluation

Performance on Test Data

Now, we evaluate the performance of our model on the test dataset.

91/91 [==============================] - 0s 3ms/step - loss: 0.1192 - precision: 0.9884 - recall: 0.9724 - accuracy: 0.9821

The model achieves a near state-of-the-art accuracy of 98.21% on the test set.

Classification Metrics

We also compute the performance metrics for each class to get a better estimate of model performance.

As shown in Figure 3, the model is performing well for all classes of intent.

Inference using a Sequence Model

Finally, we can use our trained model to perform inference.

play_music

The model correctly predicts the intent “play_music” for the user query “Play music on YouTube Music”.

Summary

In this article, we learned to use sequence models for intent detection. We introduced the task of text classification with the example of intent detection. We also introduced sequence models and the motivation behind using them for sequence classification tasks. We demonstrated how sequence models can be used to solve the task of intent detection by implementing an LSTM model using TensorFlow Keras in python.

A Jupyter Notebook containing the code can be found here.

Resources

If you’re interested in learning more, here are some resources:

• https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9
• https://machinelearningmastery.com/gentle-introduction-long-short-term-memory-networks-experts/
• https://colah.github.io/posts/2015-08-Understanding-LSTMs/

References

• Coucke, A., Saade, A., Ball, A., Bluche, T., Caulier, A., Leroy, D., Doumouro, C., Gisselbrecht, T., Caltagirone, F., Lavril, T. and Primet, M., 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
• https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/

Thanks for reading! If you have questions or if you would like to see us write on anything, please drop a comment or reach out on my website.

About Holler

Holler is here to make your texts, posts, payments, and DMs more expressive. How? By suggesting the most relevant content — animated Stickers and GIFs– right when you need it the most in chat.

Intent Detection using Sequence Models was originally published in Holler Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.