A key component of any NLP project is the ability to rapidly test and iterate using techniques. Keras offers a very quick way to prototype state-of-the-art deep learning models, and is therefore an important tool we use in our work.

In a previous post, we demonstrated how to integrate ELMo embeddings as a custom Keras layer to simplify model prototyping using Tensorflow hub. BERT, a language model introduced by Google, uses transformers and pre-training to achieve state-of-the-art on many language tasks. It has recently been added to Tensorflow hub, which simplifies integration in Keras models.

Let’s get building! First, we load the same IMDB data we used previously:

Next, we tokenize the data using the tf-hub model, which simplifies preprocessing:

We next build a custom layer using Keras, integrating BERT from tf-hub. The model is very large (110,302,011 parameters!!!) so we fine tune a subset of layers.

Now, we can easily build and train our model using the BERT layer:

Pretty easy! See the full notebook on Github and build cool stuff!

Want to work on challenging NLP, Machine Learning, and AI in a variety of industries with a team of top data scientists in Chicago? We’re hiring talented data scientists and engineers!

Learn more at strong.io and apply at careers.strong.io

BERT in Keras with Tensorflow hub was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

ELMo embeddings, developed at Allen NLP, are one of many great pre-trained models available on Tensorflow Hub. ELMo embeddings are learned from the internal state of a bidirectional LSTM and represent contextual features of the input text. It’s been shown to outperform GloVe and Word2Vec embeddings on a wide variety of NLP tasks.

Here’s a bit of code our team at Strong Analytics recently used to prototype an NLP model using state of the art ELMo embeddings in Keras.

First we load some data:

# Load all files from a directory in a DataFrame.
def load_directory_data(directory):
  data = {}
  data["sentence"] = []
  data["sentiment"] = []
  for file_path in os.listdir(directory):
    with tf.gfile.GFile(os.path.join(directory, file_path), "r") as f:
      data["sentence"].append(f.read())
      data["sentiment"].append(re.match("\d+_(\d+)\.txt", file_path).group(1))
  return pd.DataFrame.from_dict(data)

# Merge positive and negative examples, add a polarity column and shuffle.
def load_dataset(directory):
  pos_df = load_directory_data(os.path.join(directory, "pos"))
  neg_df = load_directory_data(os.path.join(directory, "neg"))
  pos_df["polarity"] = 1
  neg_df["polarity"] = 0
  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)

# Download and process the dataset files.
def download_and_load_datasets(force_download=False):
  dataset = tf.keras.utils.get_file(
      fname="aclImdb.tar.gz", 
      origin="http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz", 
      extract=True)

  train_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                       "aclImdb", "train"))
  test_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                      "aclImdb", "test"))

  return train_df, test_df

# Reduce logging output.
tf.logging.set_verbosity(tf.logging.ERROR)

train_df, test_df = download_and_load_datasets()
train_df.head()

We next process our data. Note that to use strings as input to a Keras model, we create a numpy array of objects. I’ve limited this data to the first 150 words for memory (ELMo embeddings are computationally intensive, so use a GPU!).

# Create datasets (Only take up to 150 words)
train_text = train_df['sentence'].tolist()
train_text = [' '.join(t.split()[0:150]) for t in train_text]
train_text = np.array(train_text, dtype=object)[:, np.newaxis]
train_label = train_df['polarity'].tolist()

test_text = test_df['sentence'].tolist()
test_text = [' '.join(t.split()[0:150]) for t in test_text]
test_text = np.array(test_text, dtype=object)[:, np.newaxis]
test_label = test_df['polarity'].tolist()

To instantiate the Elmo Embeddings in Keras, we first have to create a custom layer to ensure that the embedding weights are trainable:

class ElmoEmbeddingLayer(Layer):
    def __init__(self, **kwargs):
        self.dimensions = 1024
        self.trainable = True
        super(ElmoEmbeddingLayer, self).__init__(**kwargs)

def build(self, input_shape):
        self.elmo = hub.Module('https://tfhub.dev/google/elmo/2', trainable=self.trainable, name="{}_module".format(self.name))

self.trainable_weights += K.tf.trainable_variables(scope="^{}_module/.*".format(self.name))
        super(ElmoEmbeddingLayer, self).build(input_shape)

def call(self, x, mask=None):
        result = self.elmo(K.squeeze(K.cast(x, tf.string), axis=1),
                      as_dict=True,
                      signature='default',
                      )['default']
        return result

def compute_mask(self, inputs, mask=None):
        return K.not_equal(inputs, '--PAD--')

def compute_output_shape(self, input_shape):
        return (input_shape[0], self.dimensions)

Now we can build and train our model with the ElmoEmbeddingLayer:

input_text = layers.Input(shape=(1,), dtype=tf.string)
embedding = ElmoEmbeddingLayer()(input_text)
dense = layers.Dense(256, activation='relu')(embedding)
pred = layers.Dense(1, activation='sigmoid')(dense)

model = Model(inputs=[input_text], outputs=pred)

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

model.fit(train_text, 
          train_label,
          validation_data=(test_text, test_label),
          epochs=5,
          batch_size=32)

And that’s it! There are lots of great models on tensorflow hub, make sure to experiment with them all!

See the IPython notebook here: https://github.com/strongio/keras-elmo/blob/master/Elmo%20Keras.ipynb’

Want to work on challenging NLP, Machine Learning, and AI in a variety of industries with a team of top data scientists in Chicago? We’re hiring talented data scientists and engineers!

Learn more at strong.io and apply at careers.strong.io

Elmo Embeddings in Keras with TensorFlow hub was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

As described by Koeneker and Hallock [2], the standard approach in OLS regression is to estimate the conditional mean of the data by drawing a line which minimizes distance between the line and all of the points on the graph. In contrast, the goal in quantile regression is to estimate the conditional median, or any other quantile. To do so, we instead weight the distances between points on the graph and our regression line based on the selected quantile. For example, if we selected the 90th quantile to estimate, we’d fit a regression line so that 90% of the data points are below the line and 10% are above.

Tensorflow code

A great recent post on medium by Sachin Abeywardana demonstrated how to perform deep quantile regression using Keras. However, a limitation of this implementation is that it requires separate fitting of models per quantile. Using Tensorflow, however, we can instead fit an arbitrary number of quantiles simultaneously.

The notebook for this implementation can be found on the Strong Analytics github.

We first instantiate a simple regression model, and add outputs for each of the quantiles we’d like to estimate

# Define the quantiles we’d like to estimate
quantiles = [.1, .5, .9]

# Add placeholders for inputs to the model
x = tf.placeholder(tf.float32, shape=(None, in_shape))
y = tf.placeholder(tf.float32, shape=(None, out_shape))

# Create model using tf.layers API
layer0 = tf.layers.dense(x, units=32, activation=tf.nn.relu)
layer1 = tf.layers.dense(layer0, units=32, activation=tf.nn.relu)

# Now, create an output for each quantile
outputs = []
for i, quantile in enumerate(quantiles):
    output = tf.layers.dense(layer1, 1, name=”{}_q{}”.format(i, int(quantile*100)))
    outputs.append(output)

Now, we create the losses for each quantile

losses = []
for i, quantile in enumerate(quantiles):
   error = tf.subtract(y, output[i])
   loss = tf.reduce_mean(tf.maximum(quantile*error, (quantile-1)*error), axis=-1)
   losses.append(loss)

Finally, we combine the losses and instantiate our optimizer

combined_loss = tf.reduce_mean(tf.add_n(losses))
train_step = tf.train.AdamOptimizer().minimize(combined_loss)

And that’s basically it! Now we can fit our model and simultaneously estimate all of the specified quantiles:

for epoch in range(epochs):
   # Split data to batches
   for idx in range(0, data.shape[0], batch_size):
     batch_data = data[idx : min(idx + batch_size, data.shape[0]),:]
     batch_labels = labels[idx : min(idx + batch_size, labels.shape[0]),:]

    feed_dict = {x: batch_data,
                 y: batch_labels}

    _, c_loss = sess.run([train_step, combined_loss], feed_dict)

Fitting this data on a dataset of motorcycle crashes (from here) gives us the following output:

Refs

1. Regression Quantiles. Roger Koenker; Gilbert Bassett, Jr. Econometrica, Vol. 46, №1. (Jan., 1978), pp. 33–50.

2. Quantile Regression. Roger Koenker; Kevin F. Hallock. Journal of Economic Perspectives · vol. 15, no. 4, Fall 2001. (pp. 143–156).

Want to work on challenging Machine Learning and AI in a variety of industries with a team of top data scientists in Chicago? We’re hiring talented data scientists and engineers!

Learn more at strong.io and apply at careers.strong.io

Deep Quantile Regression in Tensorflow was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

When we started Strong Analytics, we did so because we loved statistics, machine learning, and building software. What we didn’t know at the time, however, was how much being a data science consultant differed from being a typical data scientist, and how much fun being a data science consultant would be.

A consultant, you say? Isn’t that the realm of overpriced suits, ‘catalyzing human capital growth through cross-platform action items’, and 600 hour workweeks?

That couldn’t be farther from the truth! With almost 2 years under our belt, I thought I’d outline a few things I love most about this job.

We solve incredibly challenging problems

I, like most of us who find our way to data science, am naturally drawn to the challenge of solving difficult problems. As consultants, companies reach out to us with the most difficult problems they have — ones that can’t be solved internally. It’s our job to help them figure out how to tackle these problems and, in most cases, build the solutions for them.

Every day brings new challenges, from building deep learning models for tracking and detection on embedded devices, mobile applications that detect and respond automatically to critical user behaviors, and platforms that ingest and process hundreds of millions of datapoints per day. It’s always challenging, but never boring!

We get to work in a variety of industries

Being a data science consultant gives us the opportunity to work with clients across many industries — including automotives, gaming, pharmaceuticals, retail, biometrics, engineering, publishing, manufacturing, and IoT. Two weeks ago, I had a lab coat and hard hat on at food manufacturing plant; the week after, I was learning about retina scans in biosecurity.

It’s always exciting to learn about a new industry and see both how they currently use data science and how they can better use data science to improve their business.

We use the ‘full-stack’ of data science

Being data science consultants, we have to be familiar with a wide variety of tools and technologies to be self-sufficient. It’s not enough to know how to analyze data using R or Python — we need to be fluent in various SQL dialects, machine learning models, optimization techniques, distributed computation frameworks, cloud services, probabilistic programming languages, APIs and web applications, UX design, data visualization, and scalable software application deployment. PHEW!

I definitely didn’t know all of these when I first started (and I’m still learning a few!), which leads me to the next thing I love most about data science consulting…

Every day is an opportunity to learn something new

Being a data science consultant means that our clients rely on us to be on the bleeding edge. Whether it’s a new tool, technique, or technology, every project that we do is unique, and therefore presents an opportunity to learn something new and stay on the forefront of data science.

We all grow together

Without a doubt, the absolute best aspect of data science consulting is being surrounded by a team of awesome, passionate, talented data scientists who share the same goals as I do. Everyone on our team brings together diverse experiences and expertise both from their previous careers and from different projects they work on. Having a strong learning community gives us the amazing opportunity to grow together. From Friday lab meetings to impromptu discussions, it’s hard to convey how much fun it is to be a part of a great data science consulting team.

There’s still lots more learning and growing to do, and I’m thrilled to have the opportunity to do so at my dream job.

Does this sound like your dream job too? Want to work on challenging Machine Learning and AI in a variety of industries with a team of top data scientists in Chicago? We’re hiring talented data scientists and engineers!

Learn more at strong.io and apply at careers.strong.io