One of the biggest challenges facing people new to building recommender systems is the lack of understanding around what these systems look like in the real world. The majority of the online content around recommender systems focuses on models and is often limited to a simple example of collaborative filtering. For new practitioners, there is an enormous gap between examples of simple models and a production system that serves recommendations.

In this blog post we’ll share a pattern that we feel covers the majority of recommender systems deployed today with examples from companies like Meta, Netflix, and Pinterest. This pattern is central to how we think about building end-to-end recsys within the NVIDIA Merlin team and we’re excited to share it with the broader community and help build an understanding and consensus of what recommender systems (not just models) look like in production. If you’re interested in this content in a different format we’ve also got a talk from KDD’s Industrial Recommender Systems workshop on the same topic.

Beyond Recommender Models

The role of a recommender model, whether it’s a simple collaborative filtering example or a deep learning model like DLRM is ranking, or more accurately Scoring, the interest that a user may have with a set of items. But these scores by themselves aren’t enough to serve recommendations to that user in most real world contexts. There are several reasons for this which we’ll dig into below before we explore the solutions and how they shape the system that we end up with.

More items, more problems

The first issue that we run into at scale is the number of items being recommended. Item catalogues can grow to millions, hundreds of millions, even billions in extreme cases. Scoring every item for every user just isn’t feasible in those (or really most) cases. Scoring is computationally expensive. In practice, you start by quickly selecting a relevant subset of those items, say a thousand or ten thousand that you’ll score.

Enter 2-stage recommender systems. Before we score the items we need to select a reasonably relevant set that contains the items that the user will eventually engage with. This stage is commonly referred to as the candidate Retrieval stage, but we’ve also heard it referred to as candidate generation. Retrieval models take many forms including matrix factorization, two-tower, linear models, approximate nearest neighbor, and graph traversal, and are generally (much!) more computationally efficient than the scoring model. YouTube has an excellent paper from 2016 that is one of the first public references of this architecture but the method was almost certainly in use at many other places and is commonly applied in industry. Eugene Yan has a great blog post on this topic and his 2-stage images were the inspiration for our 4-stage recommender diagrams we’ll introduce below. It’s worth noting that it’s also commonplace to use multiple candidate sources in the same recommender system to ensure a diverse set of candidate items are being presented to the user, but we’ll save that topic for another blog.

Anything but that!

While it might seem like we’re able to serve relevant recommendations at scale with these two stages, there are other constraints that a recommender system needs to support. In most contexts, there are items that you don’t want to show to the user. When the item is out of stock, when it’s not age appropriate, when the user has already consumed the content, or when licencing rights don’t allow you to show it in this user’s country.

Rather than relying on the scoring or retrieval models to infer this business logic and to recommend items appropriately, it’s necessary to add a Filtering stage to your recommender system. Filtering is most frequently done following the Retrieval stage, can be integrated with it (one of the more complex problems you run into with filtering is ensuring that there are enough candidates after retrieval) or can even follow scoring in some situations. Filtering allows you to apply business logic rules that would otherwise be impossible (or at least very hard) to enforce by the model. In some cases these are simple exclusion queries, but they can be more complex as is the case for Bloom filters which can be used to remove items that have already been interacted with by the users.

Order up!

The three stages introduced so far: Retrieval, Filtering, and Scoring provide us with a list of relevant recommendations at scale and their corresponding scores. These scores represent the scoring model’s guess at how interested the user will be in each item. Recommendations are generally presented to the user as a list though, and that presents an interesting conundrum; the best list is unlikely to fully align with the individual item scores. Instead we may want to provide a diverse set of items to the user from amongst candidates, or even show them items outside their normal pool of recommended candidates to explore spaces they haven’t seen, helping to prevent filter bubbles.

In some literature and examples our third stage of the recommender system would be referred to as ranking, but the final rank (or position) at which a recommendation is shown to the user is rarely aligned directly with the output of the model. By providing an explicit Ordering stage we’re able to align the output of the model with other needs or constraints of the business.

4-Stage Recommender Systems

These four stages of Retrieval, Filtering, Scoring, and Ordering make up a design pattern which covers nearly every recommender system that we’ve encountered or built. The diagram below shows these stages and presents an example of how each stage could be built. It’s significantly more complex than a basic recommender model, especially as we begin to think about deploying to production but we think it accurately represents how most production recommender systems today are built.

Examples in the Wild

Now that we have a pattern for describing recommender systems let’s see how it stands up. First, exploring examples from common recsys tasks we see that at least at a high level the four stages cover these use cases and demonstrate a consistent pattern.

Taking this a step further, we can look at examples from real-world recommender systems in order to see if we can recognize the four stages from our pattern.

First up, Meta’s Instagram has a great article about the query language they’ve developed to make recommender systems easier to develop: Powered by AI: Instagram’s Explore recommender system (IGQL query language). As we can see from the example they provide, this query language exactly maps to the four stages of our pattern:

Pinterest has a number of papers (Related Pins at Pinterest:The Evolution of a Real-World Recommender System, Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time, and Applying deep learning to Related Pins) the first of which contains this figure which outlines their system architecture’s evolution over time. Again we see this same pattern with the subtle difference that retrieval and filtering have been grouped into a single stage in their diagram.

Finally, Instacart shared in 2016 this architecture for providing recommendations that follows the four stages directly. First candidates are retrieved and the previously bought items are filtered out. The top candidates are then scored, and the final results are reordered in order to improve the diversity of results in the final set presented to the user.

Complex Systems

In our 4-stage diagram we’ve tried to articulate the components required to train, deploy and support inference time querying of all of the stages. This system is much more complex than a single model, and it’s no wonder that those who looked for information about recommender systems online and found only collaborative filtering models are overwhelmed when they actually try to build them.

In our next post we’ll look at some of the issues that this complexity introduces in more detail, and talk about some of the solutions we’re proposing in Merlin, our recommender system framework, but for now we’ll leave you with this challenge: Take a look at recommender systems that you use and see if you recognize the four stages, and if you find any that don’t please let us know! We’re constantly iterating and refining our thoughts and our libraries to ensure that we’re providing the best solution we can for the RecSys space and your input is appreciated.

As a final note for those of you who made it this far… If you’re passionate about building open source software that helps solve these solutions please reach out to us as well. Come chat with us at our online RecSys Summit! We have a number of open roles that make the building and deployment of recommender systems easier. We’d love to hear from you.

Recommender Systems, Not Just Recommender Models was originally published in NVIDIA Merlin on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building recommender systems (RecSys) at scale is a non-trivial process. The data is huge, training takes a long time, and getting models into production takes thought and care. Tensorflow provides an ecosystem for the latter, with TF.Data and Feature Columns for transforming data, a standardized data format in TF Records, and a dedicated inference server in TF Serving. This has made Tensorflow the framework of choice for many companies when designing and deploying recommender systems, and for good reason.

However, when it comes to ETL and Training, there are gaps in GPU kernel coverage and implementation details that make the training of recommender systems much slower than they could be. Unlike the computer vision and NLP domains where GPUs have been applied effectively, recommenders aren’t able to effectively leverage the parallel compute and high memory bandwidth that GPUs have to offer. In our previous blog post we dove into some of the conceptual aspects of that problem; at its heart, dataloaders aren’t able to keep the GPU utilized. To help solve this we leveraged the same tools used to develop NVTabular, Dask-cuDF and RAPIDs and built accelerated data loaders, along with a few other layers to support better acceleration of recommenders on GPU.

While NVTabular supports multiple frameworks (TensorFlow, PyTorch and Fast.AI), our focus in this blog is on its integration with the TensorFlow ecosystem. There are some specifics to Tensorflow that require additional thought to fully leverage and benefit from these accelerations. We’ll walk through some experiments for speed, convergence and GPU utilization and finally show how to use the NVTabular dataloader to speed up your own recommender system workflows in Tensorflow.

Kicking the Tires

In our previous post we argued that item by item dataloading and even the windowed dataloading that Tensorflow uses was an issue. A quick experiment shows that the throughput of Tensorflow training is approximately the same as that of the dataloader. If we can improve dataloading throughput, we’re likely to get an improvement in total training time. How much faster can we make things on the dataloading side though? Using our dataloader which loads large chunks of data into a buffer on the GPU and shuffles it there we’re able to show a 24x speedup over the native Tensorflow windowed dataloader, going from 177K samples/s to a whopping 4.2 Million. Unfortunately that’s just the dataloader speedup; training the model also takes time, so the final performance we see when training isn’t as significant, but we’ve eliminated the bottlenecks caused by inefficient dataloading.

Let’s see what this means for training; In Figure 1 below we share the results of our benchmark which compares the NVTabular dataloader to the TF native one in two configurations, full and mixed precision. From the graph we see a 8.1x speedup at full precision and a 9.3x speedup when using mixed precision. Mileage may vary depending on datasets and model configuration, but if you’re training with tabular data this should be a substantial speedup and we encourage you to try it out for yourself on your data.

In each experiment we use the same hardware, a DGX-A100 with 40GB GPU memory, with data stored on local NVME. Criteo Click Ads Prediction is used as a dataset with 150M examples for the first day used for training and 150M examples of the 3rd day used for validation. The neural network architecture uses Embedding Layers for the categorical features and has 4 hidden layers with each 1024 neurons in the MLP tower. You can find a link to our experiment notebook here.

We also see a big increase in GPU utilization, with average utilization jumping from 45% to 80% for full precision and from 20% to 60% for mixed. The NVTabular dataloader is feeding data to the GPU at a much faster rate and dataloading is no longer the bottleneck in training which allows the GPU to shine!

Too good to be true?

The NVTabular dataloader is able to speed up the training pipeline by 9x with mixed precision but does it have an impact on model quality? Data is grabbed in multiple large contiguous chunks and (optionally) shuffled on GPU. We’re not doing the same shuffling mechanism as Tensorflow’s windowed dataloader and we also don’t provide a full shuffle of the data. Aside here; we’ll have a blog post out sometime soon on shuffling and data ordering but in the meantime let’s continue for brevity’s sake. As data scientists and ML Engineers, we train deep learning models to achieve a high accuracy and data ordering can affect the model’s accuracy through the randomness in batch creation. We need to validate that training a deep learning model with the NVTabular data loader results in the same convergence behavior. Thankfully in our testing that has been the case; we see nearly identical convergence when using either dataloader.

Phenomenal cosmic power! (itty bitty code change)

When we built the dataloader we had the existing ecosystem and ease of integration in mind. KerasSequenceLoader is modelled after tf.data.experimental.make_batched_features_dataset in our API. It’s initialized with a data schema which defines the columns that are categorical and numerical, as well as the label so that the corresponding tensor types can be concatenated together when passing data to the framework.

https://gist.github.com/bschifferer/78e2ca51d004ead898ea9d3d96879c6a

Our Tensorflow solution also includes a number of model layers critical to achieving optimal performance which will be covered in a future post. The most important of these is a custom embedding layer that is designed to more efficiently handle single-hot embedding lookups. We’ve also developed a utility designed to allow you to convert Tensorflow Feature Column definitions into NVTabular for doing transformations to the data more efficiently. A more advanced example of Tensorflow feature engineering and training is available in our Movielens dataset example.

Try It Out for Yourself

Our benchmarks show a speed-up of 8–9x with consistent convergence behavior and seamless integration into a TensorFlow pipeline. Check out our notebooks with TensorFlow data loader for Rossmann, MovieLens, and our detailed notebook on Tensorflow training as additional resources. We’ve worked hard to try to make it easy for you to try this out on your own data. The experiments from this benchmark can also be found here. You may also be interested in accelerating your full pipeline with NVIDIA Merlin. For additional information, see our other Merlin blog posts and HugeCTR training deep learning model.

NVTabular is an open source project so please check out the repo if you have any questions. We’d love to hear what you’re working on and any gaps you see. You can reach us through our github with any issue or feature request or by leaving a comment here. And finally, if you’re as passionate about recommender systems as we are please check out this open role for the team. We’re growing fast and would love to work with you to help make RecSys fast and easy to use on the GPU.

This work would not have been possible without the awesome engineering efforts of Alec Gunny who wrote much of the code required for the dataloader and Tensorflow layers.

Training Deep Learning Based Recommender Systems 9x Faster with TensorFlow was originally published in NVIDIA Merlin on Medium, where people are continuing the conversation by highlighting and responding to this story.

Compute, Memory, and I/O (oh my!)

Part of the reason that GPUs have been so successful at Vision and NLP tasks is that models in those domains are large and complex. In vision, compute makes up the entire workload, and the parallelism of the GPU shines. For NLP the case is slightly more complex; BERT-base has 12-layers, with a 768-hidden width and 12 attention heads for a total of 110M parameters. It’s vocabulary size is 30K x 1024 meaning that the embeddings which store representation learned by the model make up ~30% of its total size. This is a significant proportion, but compute is still the dominant factor in throughput.

Compare that with recent session based recommender system architectures such as BST which follow the same general transformer architecture but are configured very differently. BST has a single transformer layer followed by a 1024–512–256 width MLP and 8 attention heads. Its equivalent ‘vocabulary’ on the other hand is a whopping 300M users and 12M items in the example Taobao dataset which are embedded with a width of 64. So the amount of compute is several orders of magnitude lower than we see in NLP and the embeddings which are IO bound, not compute bound, make up well over 95% of the model and are hard to fit on a single GPU (stay tuned for a future article on that). What that means is that in order to keep the GPU running efficiently we need to make sure other aspects of the workflow are well tuned..

A digression into dataloaders

Most people take this aspect of training for granted. Point the framework’s dataloader at the directory or files that you want the model to train on and you’re good to go. In situations where the model is dominated by compute this approach is often okay. An asynchronous dataloader only ever has to feed data faster than the forward and backward pass of the model. As long as it’s able to get the next batch ready before the GPU is done processing the current batch it has done its job.

For NLP or Vision architectures compute is significant relative to time taken to get the data to the model. You’re also usually working with small batches of large examples and the strategies that work there aren’t optimal for the kind of data that recommender systems use. Properly tuning the dataloader and I/O is important, and I highly recommend you check out the work being done by the DALI team at NVIDIA if your workload includes images or audio, but you can’t apply the same principles to recommendation.

For starters, not only is the compute smaller, but at the example level and even at the batch level recommender system data is usually quite small. Most dataloaders work by aggregating randomly selected examples into batches and then passing that information to the GPU. In future blog posts we’ll deep dive into the specifics of the different dataloaders available in deep learning frameworks but for this blog we’ll focus on the common dataloader case where batches are aggregated from random examples. Even if we try to solve the problem by piling on more workers to create batches, we’re still hammering on memory (or more likely disk) in an access pattern that is horribly inefficient. Pulling data example by example just doesn’t make sense for tabular data.

NVTabular dataloaders to the rescue!

When we ran into this issue a little over a year ago we started to look at ways to get data to the GPU more efficiently using RAPIDS in conjunction with PyTorch. Since then we’ve been iterating on the concept of dataloading for tabular data and have a solution. We now support fast tabular dataloading in Tensorflow 2.3+, PyTorch 1.2+, and Fast.ai v2 using APIs that are derived from each library. Using Dask-cuDF we were able to share the same back end for all frameworks and use dlpack to transfer data between our dataloader and the batch in tensor format. Our next blog post dropping in a week or so will share the details of our Tensorflow dataloader and we’ll deep dive into the others in the coming weeks.

I won’t overload you with details here, but at a high level the NVTabular dataloader transfers large chunks of data into GPU memory, creating a buffer of multiple batches on the GPU. These batches are grabbed as contiguous memory blocks from the buffer resulting in a much greater efficiency. The number of chunks grabbed and the size of the buffer are completely configurable. We also asynchronously update the buffer so that the GPU always has data available. It’s worth mentioning that the dataset can be much bigger than the total GPU or even CPU memory which we know is important in the recommender system space. An NVME or equivalent fast storage is recommended though for best performance.

Overall results are impressive. In our initial benchmarking of our dataloader in Tensorflow we saw GPU utilization jump by 2x from 40% to 85% with a corresponding 2x speedup in total training time. What’s even more amazing is that when we apply techniques meant to make compute more efficient like Automatic Mixed Precision (AMP) the gains compound to a 4x total speedup, which we don’t see at all when we only apply AMP. Increasing training speed on GPU by 4x over the standard TF windowed dataloader makes training 22x faster on GPU than CPU, a huge difference when comparing total cost to train in a cloud environment. We’ll have more details about each of the frameworks in our dataloader deep dives.

So we’re done?

Not even close. A 22x speedup over CPU is a great start, but there’s always more to be done and a lot of details yet to come. Storage, I/O, and memory bandwidth play a big role in performance. Getting embeddings to fit on the GPU is another interesting challenge that the Merlin HugeCTR team has tackled. There’s also the recently announced A100 80GB cards which will revolutionize the size of embeddings that we can fit, and with 2 TB/s memory bandwidth training times get even faster.

We’ll be sharing further improvements and ideas within the recsys space along with our research and general information related to the training and productionizing of recommender systems. Stay tuned to this blog for more recsys knowledge.

In the meantime go check out our examples in the NVTabular Repo to see how you can use NVTabular to speed up both the training your recommender models and the data preparation phase. We’d love to hear what you’re working on and any gaps you see. You can reach us through our github with any issue or feature request or by leaving a comment here. And finally, if you’re as passionate about recommender systems as we are please check out this open role for the team. We’re growing fast and would love to work with you to help make RecSys fast and easy to use on the GPU.

Why isn’t your recommender system training faster on GPU? (And what can you do about it?) was originally published in NVIDIA Merlin on Medium, where people are continuing the conversation by highlighting and responding to this story.

This June the RAPIDS Deep Learning team took part in the RecSys 2019 Challenge, where we placed 15th out of 1534 teams despite joining the competition in its final weeks. The competition centered around recommending hotel listings to users of the website Trivago, who was the host of the competition. Given a list of impressions, the task was to rank the impressions such that the clicked item appeared as close to the top as possible, with the final score determined by Mean Reciprocal Rank (MRR) on a holdout test set. Here’s an example of what the data looked like from the competition website:

Our methodology relied on feature engineering, a stacked ensemble of models, and the fastai library’s tabular deep learning model, which was the main model architecture we utilized for deep learning. The focus of this blog post, and in the more technical paper we’re presenting at the RecSys Competition Workshop, are:

1. the improvements in model training time that we were able to accomplish by accelerating preprocessing with cuDF (RAPIDS open-source dataframe library),
2. using our custom PyTorch batch dataloader instead of the standard item by item dataloader,
3. scaling batch size to extremes, and
4. the improvements to the kernel used to compute the embedding gradients in PyTorch.

In our final solution we sped up training of the fastai tabular model by a factor of 15.6x, from a workflow of 891.8s (14m52s) to 57.2s on the 40M row RecSys dataset. We also accelerated our feature generation by a factor of 9.7x, reducing the time taken to generate the features used in our model from 51 minutes to 5. With training time so drastically reduced, the iteration involved in generating new features and training new models is much more fluid. This allows for the rapid prototyping of deep learning based recommender systems in hours as opposed to days.

We’re very excited to apply the acceleration of RAPIDS to the fastai library. We’ve long been fans of Jeremy, Rachel, Sylvain and their vision of a more accessible deep learning education (for anyone interested in the topic). The process is underway to get these changes integrated into the fastai library v2 directly making it even easier to accelerate your tabular deep learning workflow with RAPIDS. For now you can use the basic_data methodology and try out in our example repo. We’d love to hear from anyone who tries out these methods on their own data. Now, let’s dive in!

Feature Engineering and Preprocessing

RAPIDS is becoming known for accelerating the feature engineering phase of tabular data workflows through previous posts by our teammate Jiwei Liu, who we were lucky enough to work with on the competition. Unsurprisingly RAPIDS shines in this case as well, accelerating the feature creation phases of the project from over 50 minutes to just over 5. This speedup of 9.7x is typical of the type of dataframe acceleration RAPIDS provides through the use of cuDF instead of Pandas. The feature engineering code can be found in its own directory here.

Getting data ready for the model also involves dataframe operations that can be accelerated on the GPU. Categorical variables must be encoded into a numerical representation for embedding lookups while continuous variables null values are mapped to the median, and a binary is_null variable is created indicating rows where the value has been replaced. New to RAPIDS 0.8 is the functionality to encode categorical variables using nvstring functionality. We were excited to try this out in the context of the fastai workflow.

Fastai performs preprocessing steps during the creation of the databunch, which aggregates training, validation, and optionally test dataloaders into a single object. The original workflow utilizes Pandas before eventually converting to Numpy, stacking the categorical and continuous variables separately and converting to a longtensor and a floattensor respectively. The creation of the RecSys databunch using the original workflow took 397s (6m37s).

The cuDF implementation takes advantage of the byte transfer capabilities of its parquet reader and avoids any unnecessary conversions, copying the parquet files created during feature generation directly into GPU memory. A future blog post will go into detail on how the dataloader works. A csv version is also available but is not quite as performant. After preprocessing has taken place, the cuDF dataframes are memory mapped directly into tensors using a dlpack based zero copy transfer of data. These tensors are optionally transferred to the CPU to be used in the PyTorch dataset. The transfer is unnecessary if your GPU has enough memory to hold both the dataset and the model.

The unoptimized workflow took 401.7 seconds to preprocess the feature engineered data and get it ready for model training. The optimized cuDF workflow shortens this to 47.41 seconds when using CPU memory to store the tensors, or 41.8 seconds if the tensor can stay on the GPU. This speedup of 9.6x, similar to what was achieved in feature generation.

Dataloaders

Dataloaders are the mechanism by which data is passed in batches to the model. In the vanilla PyTorch dataloader this takes the form of an iterator that randomly selects indices from the dataset, grabs the data, collates the results into a batch, and then passes that batch to the GPU. This process of grabbing from the dataset item by item is a relatively inexpensive one, but it’s not free, and in order to keep the GPU utilized you generally need to assign multiple workers to create batches in parallel.

The Dataloader is a function that returns an iterator over the dataset and creates batches. This functionality has been implemented in the form of a batch dataloader that loads the entirety of the batch from CPU memory into a tensor in a single memory access. BatchDataloader and BatchDataset replace the vanilla Dataloader and Dataset functions, but are otherwise using the same API. In order to maintain the randomization required when training DNN models effectively, we implemented a mechanism to shuffle the data before training and at the beginning of each epoch.

Loading data in this way has significant benefits. First, no multiprocessing is required, which reduces bus errors and significantly speeds up workflows in environments like Windows where multiprocessing is slower due to the way new processes are spawned. Second, performance of the batch dataloader is better than that of the single-item, multi-worker dataloader on tabular datasets.

Figure 2 below shows the before and after profiles of the training loop for an extreme batch size of 204,800. Initial testing was done at a more “modest” batch size of 4096, however the profiles are similar in nature. As in Figure 2, in the batch size 4096 unoptimized profile the dataloader is a largest component of the profile, taking 260 seconds or 54.6% of the total training runtime. The batch dataloader improves this to 2.45 seconds; only 2.08% of the total training time. This is a relative speedup of over 100x for data loading and a speed up of total training by more than 2x. This improvement of two orders of magnitude partly comes from the usage of the PyTorch dataloader in fastai, which does tensor conversions during __get_item__ rather than during the initialization phase. More typically we see a 5–50% improvement over the multi-worker dataloader for similar sized batches across different workloads, depending on the batch and tensor size.

PyTorch Kernel Improvements

With the batch dataloader in place training is no longer dominated by dataloading and we’re now more easily able to explore the performance of the CUDA kernels within the execution of the model by scaling the batch size to better utilize the GPU. Analyzing the model using the nvprof tool in Figure 3 we see that 69.6% of the work on the GPU is happening within a single kernel (EmbGPK) which is responsible for the calculation of embedding gradients.

The calculation of EmbGPK is basically a segmented sum over the rows of a matrix to compute the embedding gradients. In the original implementation indices access was not distributed evenly and there were not enough threads, resulting in underutilization of the processors on the GPU shown in Figure 4 (top). Instead of doing a flat sum where a single variable is used to accumulate the weights, we modified the implementation of the kernel to compute the sum of the weights in two-steps:

• Each GPU warp sums ‘NROWS_PER_THREAD’ number of row given by ‘indices’
• Sum each partial-sum from 1) and scatter into ‘grad_weight’

Not only does this increase the utilization, it also improves numerical stability. Figure 4 (bottom) shows the utilization of the GPU under the new kernel where we see that all GPU Streaming Multiprocessors are utilized fully. The optimizations have been implemented and merged into PyTorch and any method using embeddings at scale should see similar improvement in performance. When we examine the model performance again in NVProf we see a significantly improved graph. The calculation of embedding gradients has gone from ~70% of the workload to 2.3% and the GPU computation is dominated by matrix multiply kernels which are already heavily optimized. The overall speedup due to this method is ~6.5x.

Extreme Batch Sizes

One of the further advantages of the batch dataloader introduced above is the ability to scale to extreme batch sizes. On the Tesla V100 GPUs that we used for evaluation the maximum batch size we could scale our baseline model to was 819,600. Scaling to this degree sped up training to 15s / epoch, however the model took two epochs to converge. After further evaluation we reduced our batch size to 204,800 which we were able to converge in a single epoch in 15.4 seconds. This is a 7.5x speedup over the 4096 batch size GPU Memory variant, while the speedup for the unoptimized fastai workflow with the larger batch size was only 4.7%, in part due to increased dataloader costs.

The scaling of batch size to this degree was aided by using layer-wise adaptive large batch optimization, more commonly known as the LAMB optimizer, which scales the learning rate for each layer of the network. We validated model performance by evaluating mean and stdev of AUC and MRR across five runs of each batch size. The 4096 batch size version had an AUC of 0.8827 +/- 0.0049 and an MRR of 0.6143 +/- 0.005 while the 204,800 batch version had an AUC of 0.8816 +/- 0.0012 and an MRR of 0.6136 +/- 0.0008 for our baseline model, a relative difference of ~0.1% for both metrics.

Conclusions and Future Work

In this blog post we have shared several novel optimizations that can be used when training deep learning models in PyTorch using the fastai library. Using the RAPIDS.AI cuDF library we improve the preprocessing steps that prepare data for the model by a factor of 9x, performing categorical encoding, normalization and null value filling on the GPU. We introduce a novel batch dataloader which loads an entire batch from memory in a single read accelerating the PyTorch dataloader by a factor of over 100x, and training time by 2x. Upon further analysis of the GPU computations we also update the kernel responsible for calculating the embedding gradient in PyTorch reducing it from 70% of the workload to 2.3% and improving the training time by 1.9x. Then, taking advantage of the LAMB optimizer we are able to scale our batch size to the limits of memory, achieving a further 2.12x speedup in training time. A further 15% improvement in performance was achieved by keeping the entire dataset in GPU memory during training.

These optimizations collectively improve the end to end training time by a factor of 15.6x from our initial model on this problem. Figure 5 highlights the improvements to each stage of the recommendation pipeline. By reducing the training time from 15 minutes to 57.2s we enable a much richer exploration of feature space in hours instead of days, and significantly reduce the cost of training deep learning based recommenders. The iterative nature of feature engineering and model training mean that these speedups compound; long feature creation and training times result in a lot of cognitive downtime and a disrupted work process. By shortening feature creation time and training time so dramatically we hope to provide a more immediate feedback loop where Data Scientists and ML Engineers can stay in the flow of their work.

We’re working on implementing variations of this method that work in different situations. We currently have an in GPU memory version that was described in detail here, and we are close to completion of a version that operates on a subset of the data in GPU memory for mid sized datasets or for GPUs with lower memory. We’re also developing a larger than CPU memory preprocessor and dataloader that will allow for efficient pre processing and data loading when the data is too large for even the CPU.

In addition to these improvements we plan to explore FP16 mixed-precision training using Apex and multi-gpu training via Hogwild, Horovod and BytePS, further improving model performance at the cost of additional hardware. Further, these performance improvements aren’t limited to tabular data or recommendation and should be applicable to Natural Language Processing, Time Series, Tabular Data, and other problems where dataloading is a bottleneck relative to compute. We hope to demonstrate their effectiveness in other domains and provide easy to use examples for the community.

Finally, we’re very excited to see the GPU preprocessing integrated into the fastai library v2. Integration into fastai and other deep learning frameworks is extremely important to us. We want to make it as simple as possible to make use of these methods and within that context fastai is a great fit.

The work shared here is the joint effort of our two amazing interns Sara Rabhi and Wenbo Sun, Kaggle Grandmaster Jiwei Liu, Julio Perez, Mads B. Kristensen, Rick Zamora and myself.

The source code of the solution is available in our repo and we encourage you to try these methods out your own workflows. Although the full 15x speedup requires all of them, they can be used individually to provide significant speedup to workflows where that isn’t possible. As with all RAPIDS related projects we’d love to hear from you regarding your experiences with RAPIDS and we welcome contributions from the community.

Accelerating Deep Learning Recommender Systems by 15x using Rapids, FastAI and Pytorch was originally published in RAPIDS AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using the GPU for ETL and preprocessing of deep learning workflows

In this post we take a look at how to use cuDF, the RAPIDS dataframe library, to do some of the preprocessing steps required to get the mortgage data in a format that PyTorch can process so that we can explore the performance of deep learning on tabular data and compare it to the xgboost method.

While deep learning (DL) has been successfully adopted in many domains (Vision, NLP, Recommendation, etc.), there are still areas where the use of DL is not mainstream. Tabular data problems, which consists of columns of categorical and continuous variables, commonly make use of techniques like xgboost, catboost, gbm, or even linear models. The RAPIDS cuML library is focused on building out those capabilities on GPUs, and in the first notebook released by the RAPIDS team we used a GPU-accelerated version of xgboost to train a model as quickly as possible at scale on the Fannie Mae mortgage dataset.

Deep learning is, however, making inroads into tabular data problems. Recent Kaggle competition winners of the Santander, Porto Seguro, and Taxi Trajectory competitions used deep learning as a significant part of their solution, and the Rossman store sales (#3 solution) and Petfinder competitions (#6 and #9 solution) both had interesting architectures and methodologies that used DNNs to achieve top ten rankings. The Petfinder solutions are of particular interest because they combine tabular, image and text data in a single model. Many companies have come forward stating that they use deep learning in these or similar contexts, including Google, Pinterest, Twitter, Instacart, and Youtube. Deep learning library fast.ai has even added support for tabular data directly into the library in the form of categorical embeddings and a range of preprocessing options required for tabular data.

Please check out the accompanying notebook in the RAPIDS.AI repo for the implementation details. The notebook and experiments were developed and run by Christopher Green and we’re grateful for his work.

Establishing a baseline

As our starting point we’ll take the original mortgage notebook from the RAPIDS repo. Feature engineering is identical in order to prevent performance improvements unrelated to deep learning.

Here, when we say performance, we’re talking about how well the algorithm is able to classify loans, which we’ll measure as the Area Under the Precision Recall Curve, or PR-AUC for short. We chose PR-AUC over cross entropy, accuracy and ROC-AUC because we think it provides a better representation of the performance of the algorithm. Lowest cross entropy loss often doesn’t correspond to the most accurate model when examining the model’s ability to predict whether a loan will go delinquent. On the other hand, using accuracy as the only metric hides precision/recall tradeoffs necessary when evaluating a model in the real world. Finally, ROC-AUC is less able to distinguish between model performance when there is a class imbalance, which we see in the mortgage dataset.

We use a time-based split of the data, creating validation and test sets such that they occur after the training data in order to prevent data leakage. Since we’re predicting 12-month loan delinquency we need to ensure that the validation and test data start at least 12 months after the latest date in the training dataset. Similarly, we establish a test set from a later time window than the validation set to ensure that we don’t overfit the hyperparameters.

Because of the size of the dataset (1.75 billion rows!) and the nature of xgboost on GPU which requires the entire training dataset to be in memory, we are only able to train on 17% with a single DGX-1 server, which has eight Tesla-V100 32GB GPUs. A single V100 can handle ~2% of the dataset. If you were to try to run this on commodity hardware like a GTX 1080-ti you’d be limited to ~0.7% of the dataset. By taking advantage of the GPU’s massive parallelism along with recent optimizations, we’re able to train the multi-gpu xgboost model in an astonishing 60 seconds. The full run time including ETL and preprocessing is 178 seconds. Scaling resources to a six-node cluster allows training across the entire dataset with similar timing, but that isn’t the focus of this post.

After running a hyperparameter search we achieved a best PR-AUC of 0.814104 on the validation set, which corresponds to 0.8103 on the test set. This is the benchmark we try to reach or beat using a deep learning model.

Deep learning specific data transforms

There are many ways of encoding tabular data into a format that a deep neural network (DNN) can understand. One common approach is to one-hot encode categorical variables, and to normalize continuous variables before feeding them directly into the first feed-forward layer. However, in our experience this is not very effective. In our own previous testing across multiple datasets, DNNs that use one-hot encoding weren’t competitive with gradient boosting methods.

More recently, categorical embeddings — sometimes referred to as entity embeddings — have become popular for dealing with categorical variables. Categorical embeddings assign a learnable feature vector, or embedding, to each category, generally with a size correlated with the number of unique values in that category. Each unique category value has its own associated vector, allowing for a rich representation of the data where the values are represented as multiple floating-point values, rather than a single binary. The categorical embedding outputs and normalized continuous variables are then concatenated together as the input to the model.

Bag-of-features type approaches have also been used, where all of the features are embedded in the same size embedding and the input to the model is composed of the sum of its feature embeddings. This technique is modelled after bag-of-words techniques from natural language processing, the most famous of which is word2vec.

This first RAPIDS+Deep Learning notebook uses the bag-of-features approach, mapping all categorical and continuous variables into a single latent space, and uses an EmbeddingBag layer to take the average of all of the feature embeddings. These aggregated features are then fed into a sequence of four linear layers of width 600. Future posts will explore a wider range of options, and introduce some new ways of processing tabular data and new architectures for tabular data, an area of research that we are actively exploring.

Continuous Input Variables

To map continuous variables to an embedding space they must be discretized. We’ve added functionality to the cuDF Series to map a continuous input into buckets based on quantiles which split a distribution into equal parts. In this context, we’re converting continuous variables to categoricals in order to take advantage of the power of embeddings. For our quantization in this notebook we use a quantile of 20 regions, each with 5% of the input distribution. A quantile dividing into four regions is shown below for a normal distribution to demonstrate how the split is applied when the data isn’t uniform.

Sharing an Embedding Space

Because we’re mapping all of our categorical and continuous variables into the same embedding space when using the bag of features model, we also need to make sure that the same value in different categories is handled correctly and isn’t mapped to the same id. To ensure this happens we hash the name of the input column with the value to create a unique id representing the value in that specific column. These are then mapped to an embedding bag so that all embeddings for an entry can be looked up and combined in a single pass. Hashing of a series was also added to cuDF to make this step easy.

Implementation Details

Our final model is an EmbeddingBag layer followed by several feedforward layers as seen below. We perform a small hyperparameter search exploring learning rate [log_uniform(1e-4..1e-2)], weight decay [uniform(1e-6..1e-4)], batch size [512, 1024, 4096, 8192], embedding size [8, 16, 24, 32, 64], and activation type [relu, selu]. 200 different combinations were randomly sampled and evaluated using the validation set.

Results and Key Takeaways

The best performing DNN model on the validation set (PR-AUC of 0.8146) is able to achieve a PR-AUC of 0.8105 on our test set, slightly better than that of the xgboost model. The final hyperparameters are as follows: [lr = 2.005e-3, wd = 8e-6, batch size = 8192, embedding size = 32]

Key Takeaways:

• Even with a simple architecture we’re able to achieve similar performance to xgboost with deep learning.
• The deep learning model processes 208K examples/s during training, taking 6–8 hours including preprocessing and ETL on a single V100. That’s much slower than the 4.95M examples/s xgboost model that ran in 60 seconds on a DGX-1 server; but it is roughly comparable considering it equates to 620K examples/s per GPU. In future posts we’ll explore mixed precision, multi-gpu and other techniques to speed up the training of our DNN model. Our CPU benchmark processes only 2100 examples/s on a 40 core machine, which clearly demonstrates why we’re doing deep learning on GPUs. The CPU system would take over 12 days to complete a single run!
• While slower, using data loaders, we are able to train across the entire dataset using a single GPU, which is an advantage in many applications. Currently this process involves batchwise transfer of data between the CPU and GPU. In future posts we’ll be exploring in-GPU-memory data loaders to dramatically speed up training.

This first foray into deep learning for RAPIDS.AI is a significant step, demonstrating that we can achieve similar performance to xgboost with DNNS with a reasonably simple model and motivating further exploration.

Next Steps

In future posts we’ll undertake the following explorations.

• Simplify preprocessing of tabular data on GPUs and explore more direct ETL pipelines for getting data directly into a format that PyTorch and other deep learning frameworks can work with.
• Introduce some of the trappings of modern deep learning like dropout, batchnorm, learning rate schedules, concatenated embeddings, etc., and explore different architectures for tabular deep learning.
• Evaluate the performance of deep learning for tabular data on a wider range of datasets.
• Explore multi-GPU and multi-node scaling of DNN models.

In the meantime, please take a deeper look at the notebook, adapt it to your favourite dataset, and let us know your experience.

Get in touch and shape the future of RAPIDS

RAPIDS relies on your feedback to make it a great platform for data science on GPUs. We’re excited to blur the lines between data analytics and deep learning. RAPIDS is committed to providing a first-class user experience, so thank you for being a part of this community.

You can install the latest release with Conda or use the container found on NVIDIA GPU Cloud or Docker Hub. Let us know what you think via GitHub stars and issues.

Using RAPIDS with Pytorch was originally published in RAPIDS AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Like Sylvain Gugger, another alum of the course who is currently working as a Research Scientist for fastai and whose blog post inspired this one, fastai has changed my life and I’m deeply grateful to Jeremy and Rachel for their vision of a more accessible deep learning education for all. In the spirit of his post I wanted to share my journey and the things that have helped me along the way in the hopes they will aid and inspire others who are starting down this path.

The Journey

Unlike many fastai students I come from a machine learning background. One more focused on human computer interaction, and pre deep learning, so the work I did over the course of my thesis could be implemented by a fastai student in hours, but the foundation of math/programming and the ability to read papers definitely helped. Still, I rarely refer back to that time and I agree completely with Jeremy’s assessment that a PhD isn’t required to do deep learning work.

Following University I worked for the online dating site plenty of fish, writing matching algorithms, fraud detection systems, and eventually leading the research team. I left pof the year following it’s sale to the Match group, and found myself at a crossroads, looking into data engineering options and not really interested in returning to data science.

The Fastai course v1 came out that same spring, and as I worked my way through it and got to know people on the forum I got more and more hooked. Top down, applied development is a learning approach that works well for me, and I quickly fell in love with Jeremy’s teaching style. When I found out there was a part 2 I applied immediately and literally jumped for joy when I found out I’d gotten in.

With my background in recommender systems, I was very interested in that component of the course, and began researching into that area more and more. In the process I landed a role at Realtor.com working on recommendation and search ranking projects, and it was there that I really began to see the potential of what I was being taught. Deep learning for recommender systems became my passion, and I dove in head first, reading every paper on the topic, and consuming everything I could find on Youtube. I’ll talk more about the habits I’ve formed over the last two years that led to the research role in the following section, but this time was very formative for me in terms of developing the skillset and knowledge base that led to my current role at NVidia.

Along the way I’ve taken every single offering of fastai. I’m currently on my third iteration, and every class I learn something new. If you haven’t watched the old courses I highly recommend going back, and the same thing goes for the machine learning course, which taught me what I know about random forest models. The quality of education provided by fastai is so high I structured my team around it at Realtor, using classes as jumping off points for projects, which proved extremely successful.

The steps along the way

While my journey is unique to me, I want to offer some help to those who want to achieve the same dream of becoming a deep learning researcher or a better practitioner.

Tips for how to effectively complete fastai abound on the forum, and I highly recommend you look them up and follow them if you aren’t already. Going beyond the class takes additional effort, but in my mind it’s when the magic really starts to happen. A number of habits have really helped me become a better researcher and a better developer. They’re simple, but effective. It’s easy to breeze through them like you can shift-tab through the notebooks, but if you can implement them I guarantee they’ll have an impact.

1. Get comfortable with papers. Check Arxiv weekly, read a paper every night. I used to use arxiv-sanity but sadly it doesn’t seem to be maintained anymore. I’m still looking for a solution so if you have something let me know. I would recommend picking a specific research area where you can read most of the papers coming out. Find a topic that interests you like recommendation or style transfer and keep up on that small field. Getting in the habit of regularly reading papers is now my super power. It’s one you could have too if you form the habit.
2. Convert the fastai videos to audio only to listen to as a ‘podcast’ on the go. For me finding time to watch videos twice or three times isn’t feasible, particularly now that I have two young boys, but the density of the material requires it. Watch it once, and listen again and again on your commute, walk or wherever. I’ve listened dozens of times and I still pick up new concepts and ideas.
3. Implement something that’s different than the notebooks provide, end to end. This was the key for me to unlock the potential of fastai. I recreated the denoising auto encoder from the solution to the porto seguro safe driver competition. That turned out to be the starting point for a whole range of research into deep learning on tabular data and led to my current role. Pick something different from what most people are implementing, and implement it well.
4. Stay active on the forums and participate in the community. It’s one of the most overlooked aspects of the course in my mind, offering a whole bunch of people just as excited about deep learning as you are. For me it’s been a big catalyst of ideas, knowledge, and interest.
5. Look for other courses to expand your horizons. I highly recommend Stanford’s NLP course CS224n taught by Chris Manning, CS231n taught by Andrej Karpathy, Gene Kogan’s course on deep learning applied to art The Neural Aesthetic, and PyData which regularly features great deep learning talks.

Fastai has given me so much, and continues to do so, and I’m excited to be moving into a position where I can finally give something back. If you’d told me at the beginning of my journey when I saw the first video on youtube that a free online MOOC would lead me to a research role at one of the top tech companies in the world I’m not sure I would have believed you, but here I am. I have no doubt that it’s almost entirely due to fast.ai.

In order to make sure the dependencies are stated clearly I’ve included my package versions below:

• Keras 1.2.2
• Tensorflow 1.3
• Tensorflow-serving-api 1.0.0
• Python 2.7 (Tensorflow-serving only supports 2.7 currently)

Below is a link to the stripped down version of the final code snippet that worked for me after several iterations:

Challenges included:

• Format of the signatures — Documentation on the tensorflow side wasn’t clear and no examples were provided. Confusing things further several tutorial suggested just passing the tensors. For Keras running on a tensorflow backend the signature inputs for prediction look like: prediction_signature = tf.saved_model.signature_def_utils.predict_signature_def({“image”: Resnet50model.input}, {“prediction”:Resnet50model.output})
• Variable initialization — init_ops sets up the model on the tensorflow_serving side, and both global and model parameters need to be initialized here or tf-serving will complain about uninitialized variables at runtime.
• Keras has a variable called learning phase that also needs to be separately initialized. To further confuse matters the initialization has to happen before the model is instantiated. (K._LEARNING_PHASE = tf.constant(0)
  K.set_learning_phase(0) )
• sess.run(init_op) can be called only once. I was setting up the model in a jupyter notebook and on at least one occasion called this twice, which caused errors in tensorflow-serving.
• Keras associates itself with the tensorflow session that it’s running on implicitly and most operations are transferred. For some reason in the above code (possibly due to the use of pop?) the final dense layer added to the model, the one that does the transfer learning, was being randomly initialized and wasn’t being loaded when all the other weights were loaded. This lead to an incredibly challenging situation where the model would predict correctly in python but would be incorrect and would usually consistently predict one of the classes once the model was loaded into tensorflow serving. In order to fix this I had to explicitly associate the Keras backend with the active session using the command: K.set_session(sess)

Hopefully this will save you some time getting your models online in a tensorflow-serving system. It was a challenge to figure out all of these disjointed steps, and while I’m sure this won’t work one or two versions down the road it works now.