A Year in Review for Design at ReviewTrackers

ReviewTrackers grew significantly this year, and as a result, the product and the design team grew with it. This indicated that we as a company were succeeding, but with this success we, as designers, noticed that this growth facilitated inconsistency across our designs. The product had grown, making it harder to manage, and the design team had grown, resulting in more personalities and perspectives.

To combat this looming inconsistency, the team implemented several efforts to ensure consistency was maintained now and as we continue to scale. In this article, we look to explore three specific ways our design team adjusted to growth.

Implementing Design Principles

As our design team grew, so did the differences in how we designed and thought. These very differences in our design processes are critical — nay I say essential — to creating a great product. However, we began to realize that our team lacked direction and continuity in how we designed.

What we needed was a tool that would provide guidance when making design decisions; a tool that reduces individual bias and realigns our mission as ReviewTrackers designers.

Ultimately, we decided to create a set of guiding design principles. These principles were intended to hold us accountable for delivering solutions for our users.

These design principles were implemented in July 2022. You can read more about their creation and implementation below.

Getting Principled at ReviewTrackers

As we reflect on this exercise, it becomes obvious to us that creating a set of user-focused design principles is something all scaling design teams should explore. These principles have allowed our team to focus on what’s important in our design approach.

Design System Simplification

In July 2021, the team designed and built ReviewTracker’s design system, which unified the look and feel of our product. After one year of implementation, we realized that there was an abundance of type styles and spacing options to choose from.

The problem was that we had too many similar options. It was nearly impossible to justify the usage of one over another. Why should someone choose medium font weight over regular? Why use 16px spacing instead of 24? Ultimately, the differences between these options were minimal, and we as designers were wasting time, and creating divergences trying to decide between them.

The design system was intended to bring consistency to our product. Yet, with so many choices and a growing team, it was becoming harder to achieve consistent designs. We needed to take a step back, look at our product as a whole, and prioritize consistency over visual perfection.

When we sat down to examine these two core elements of our design system we uncovered that we had 45 different type styles and 10 different spacing options.

After some examination, we cut the type styles down from 45 to 10, leaving 3 for headers, 4 for body, 1 for buttons, 1 for links, and 1 for pre-titles. These remaining 10 still allowed us to utilize type as a design element aiding visual design and hierarchy, but eliminated splitting hairs between similar options.

Turning our attention to spacing, we had previously used a linear spacing scale that increased by 8.

This gave us a total of 10 options for our spacing. We determined that by using a geometric spacing scale, we would not only reduce the number of options but also increase the contrast between choices.

Reducing the number of choices in these two instances allowed us to ensure that there are fewer choices made based on personal bias and less time spent making choices surrounding type and spacing. The resulting consistency manifests itself to the user as reduced noise, resulting in a more focused user experience.

Redesigning our Navigation

Originally, ReviewTrackers utilized a horizontal-style navigation menu. However, as the product grew and features were added, the menu became overly complicated. This complexity began to impede how visitors used and understood the application.

It became clear that we needed to rethink our navigation menu to improve our users’ experience. So in September 2022, the team undertook an effort to redesign ReviewTracker’s navigational menu. Essentially, we reorganized the application’s pages and features to allow for a more intuitive user experience.

Efforts like these can sometimes feel like they pause progress. However, as a product grows, design teams should consider how their application’s information architecture (IA) supports this product growth.

It’s a basic concept that new features add complexity to an application’s IA, and yet, it can be difficult for teams to recognize the need to mitigate this complexity, especially when it means pausing work on new features and functionality. It is our team’s ability to “pause progress” and prioritize an intuitive IA that will ultimately lead to greater feature adoption and a better user experience.

Parting thoughts

As we conclude this reflection, we hope you’ve enjoyed learning a bit about our team and how we design.

We’re always interested in growing our team and hearing from talented designers. If you have any questions or would like to get in touch, please reach out to product@reviewtrackers.com.

Keeping Design Consistent was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Co-written by John Orzechowski, Renee Patel, Jake Carter, and Cici Xie

As a design team, we’re always looking for ways to improve our processes and product. Some examples of this are Establishing a Design System and Getting Principled. It’s no surprise that our ambition outgrew our current strategy: horizontal top-bar navigation.

How do we channel our ambition for continuous improvement? By completing a redesign of our product’s navigation.

Why Redesign

Our data shows that ReviewTrackers is used on larger viewports. More specifically, 85% of users use desktop and 15% of users use mobile. Users usually have ample horizontal space. But, our product matured to a point where that horizontal space was no longer enough.

As a workaround, we had to wrap the labels of some menu items. It felt clunky, outdated, and suppressive to innovation. It became clear that we needed to rethink our strategy for the sake of our company’s continued growth, and to future proof the navigation for our long-term plans. We couldn’t settle for a band-aid solution, we needed to approach this problem as a clean slate.

With guidance from our VP of Product, we aimed for an opt-out rate of less than 25%. If we stayed under that mark, then the redesign is a success.

What steps did we take to make this redesign happen and what was our release strategy?

Redesign Process

Our redesign process took place in sequential steps.

1. Research

Comparative analysis
First, we conducted a comparative analysis. This allowed us to see what other companies were doing well and what we’d like to do differently.

Research proposal
Next, we created a research proposal. This provided an overview on our research approach, which led us into the card sort phase.

Card sorts
We conducted two types of card sorts: open and hybrid.

The open card sort had 45 participants and about 14% margin of error. The hybrid card sort had 23 participants and about 20% margin of error. The card sorts provided us with behavioral insight on groupings and categories.

To determine the final structure, we referenced user input, ‘do’ vs. ‘set up’, and used design judgment.

2. Prototype and Test

Design
After conducting research, we worked on designing the new navigation. This took many iterations, which led to our final design.

Build prototype
We took our final design and built a prototype for usability testing.

Usability test
We tested the prototype with 12 users and used a spreadsheet for thematic analysis.

Collaborate with Engineering team
We added a page in Figma to communicate the final design with the engineering team.

Release and iterate with Customer Success (CS) team
Once developed, we released the navigation to CS and iterated based on their feedback.

Quality Assurance (QA) testing
Before entering a beta stage, QA Engineers tested the new navigation.

3. Release and Monitor

Beta — Release and Iterate
On June 3, 2022, we released the new navigation to 209 accounts. Within those accounts, about 700 users viewed the new navigation. We had an opt-out rate of 4.6%. From our experience with the beta, we made many improvements.

General Availability (GA) — Release and Iterate
On July 6, 2022, we released the new navigation to all remaining accounts (807)*. At the time of writing, we have an opt-out rate of 4.8%; well exceeding the goal of less than 25%.

The numbers don’t lie. And, neither does the feedback. Here’s a quote from an enterprise customer:

I do love the changes you guys made [to the navigation] in the platform. It looks so much better. It’s so much more revamped. It’s clean, I like it. I think it’s sick, in the best way possible.

Given that we achieved our success metric, what’s next for the new navigation?

Looking Forward

We’re an ambitious, growing design team in a small company that’s always challenging the status quo. One of our company values is asking “what if?”. This navigation redesign unified the design team. It coincided with each designer’s onboarding. We are grateful to have had the opportunity to bond as a team, provide value to the business, and help users decrease their time on task.

Yet, design is never done. As such, we have several next steps we’d like to complete.

Next steps:

• Make the new navigation the default experience for all users (i.e., remove the ability to opt out)
• Release the new navigation to White Label and Multi-Tenant Accounts
• Establish guidelines for whether a new feature goes in the Main section or Settings section of the new navigation
• Add global collapse and expand functionality to the new navigation
• Move review-related actions closer in context to the Reviews page

Parting Thought

It has taken a lot of work to get to this point. We’re excited about the new changes we’ve made and the future problems we’ll get to solve as our company grows. Thanks to ReviewTrackers users for providing valuable feedback. They helped make this redesign possible.

We’re always interested in growing our team and hearing from talented designers. If you have any questions or would like to get in touch, please reach out to product@reviewtrackers.com

Rethinking Navigation at ReviewTrackers was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by John Orzechowski

Products are a snapshot of a collection of decisions made over time. As products evolve, unguided decision making can turn them into a mess. There might be differing amounts of user guidance, components, and themes. As designers, we have the power to both create and stop said mess.

At ReviewTrackers, we grow constantly, collaborate consistently, and ship quickly. This is what I define as our competitive edge. This fast-paced nature allows us to learn and do a lot, but continued success is dependent on having guiding principles.

How do we avoid unguided decision making? The answer is design principles.

Why we need principles

Since June 2021, the RT design team has quintupled in size (1 → 5 designers)!

Our design team growth happened in progressive stages over the course of one year. Cici was brought on board, then Jake, then Renee, and then Jordan. Each of these individuals bring unique perspectives and skills to the team. While we had an onboarding process, we were lacking guidance on how we view design at RT. This resulted in each person having their own interpretation of successful output.

From 2019 until now (July 2022), we have created a component library (recently upgraded to v2), written component documentation, defined a type scale, spacing system, and tied that all together to establish a design system at ReviewTrackers. Despite all this, we were still missing ‘something’. That ‘something’ is principles. A tool to help guide decision making around and application of the design system in our daily work.

To set each designer up for success, we needed to agree on our core beliefs. These beliefs serve as a guide (but do not dictate) for how we think about design at RT. They hold us accountable and help us deliver on our promise to users.

We use these principles to evaluate historical, assess current, and plan future decisions.

What if I told you that you could create principles through an easy-to-follow process?

The process for creating principles

We mixed a bottom-up and top-down approach to inform these principles.

1. Analyzed user interview notes, strategy documents, mission/vision statements
2. Created a single source-of-truth document with draft principles
3. Added many comments in said document
4. Held discussion in comments and debated in meetings
5. Ranked, merged, and consolidated principles
6. Repeated steps 3 and 4 until agreement was reached

As a result, we landed on four design principles.

The principles

We value being easily understood, so each principle is in plain English with an average Flesch Reading Ease score of 77.

That way, they are inviting to all audiences; both inside and outside of RT.

Design inclusively
Meet each person where they are and give them an equal chance in managing their customers’ experience. Design for one and extend to all; make sure things are simple by default and complex by opt-in.

Nudge towards success
Let people feel one step ahead. Favor guidance over choice and be ok with deciding for them.

Promote collaboration
More often than not, there are many people in a company using RT to achieve a shared goal. Make them feel as part of a team in the product as they are outside of it.

Embrace less is more
Remove when you are able to, replace if you can, and add if you must. When replacing or adding, be minimal.

Now that we had our principles created, how did we ensure they lived beyond a Google doc?

Applying the principles

At RT, we use Figma. We turned the principles into a component placed within our design system. This is to remind ourselves of them at every stage of our work. We add them to design files we are working on using the keyboard shortcut “Shift + I”.

After the principles were established, we held a one-month trial period to apply them in our weekly Design critiques. This allowed us to develop the muscle memory of using them in conversation.

One of our designers, Cici Xie, was working on our product’s Local Search feature. It provides users with insight on how their business ranks on Google for different search terms. Through consistent reference of principles, she guided the design to a final iteration.

This is one example of the team using principles to prevent unguided decision making. Resulting in more productive design critiques, collaboration on a vision, and higher-quality product.

Parting thought

Design is never done and neither are principles. These will guide our thinking six months, one year, etc. from now. Yet, we recognize they will evolve as new user problems emerge and Design grows at RT. The important thing is that we now have a framework in place and can iterate as needed.

We’re always interested in growing our team and hearing from talented designers. If you have any questions or would like to get in touch, please reach out to product@reviewtrackers.com

Artificial intelligence (AI) principles

In H2 2023 (July 2023 — December 2023), the Design team had an AI-focused objective:

Enable the team to design effective interactions for AI to provide a highly-valuable product to our users.

Knowing the importance of design principles, we followed a similar approach to create ones that were AI focused.

Spell it out
Be clear about what AI is and how it is used in the product. Allow for the correction of AI-generated content.

Design for all skill levels
Design so that all people benefit from AI. Allow people to choose the type of relationship they want to have with AI.

Support the V formation
AI must help a person solve existing problems, not create new ones. AI is meant to streamline their ability to do this.

Mindfulness matters
Be mindful about where and when AI makes a difference. Both to a person’s experience and the product’s value.

An example of these AI principles in action is our product’s AI Responding feature. It gives users a review response with the click of a button.

Getting Principled at ReviewTrackers was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Co-written by John Orzechowski and Jennifer Zettinger

This article talks about the importance of establishing a design system at ReviewTrackers. First, we’ll explain why a design system is important and how the need was identified. Then, we’ll cover the process the design team went through to get the first version up and running. We’ll finish up with some next steps, as our goal is to always be improving.

Why is a design system important?

Overview

In a company with several departments, designers stationed on different feature teams, and an embodied company value to “set each other up for success”, there needs to be a shared resource that the organization can reference for how ReviewTrackers approaches design. This role is assumed by a design system.

A design system’s purpose is to cross-functionally unify design language, thought, and application within the product. It focuses on providing guidance and answering the following: how does ReviewTrackers talk about, think about, and apply design?

This is a question the design team inherently embodies and understands, day-to-day, but the system acts as a referenceable, supporting structure for colleagues to learn. Not only does this provide a resource to internal teams, but this question and its answers directly impact the experience users have with the product.

A successful design system requires several parts.

Component library: The entire collection and documentation of UI components used in a product.

Component documentation: The entire collection of “what, why, when, where, and how?” for foundations and components used in a product. This includes content guidelines.

Color palette: The entire collection and documentation of visual identity used in a product.

Type scale: The entire collection and documentation of typography used in a product.

Spacing: The entire collection of rules for spacing used in a product.

Defining success

ReviewTrackers believes in measuring success and using data to inform stories. Therefore, success with a design system at ReviewTrackers is broken down into two parts, qualitative and quantitative. We’ve currently achieved qualitative success but will need more time to measure quantitative success.

Qualitative:

• Provide every Product and Engineering employee with access to the design system
• Learn how to use GitHub for component documentation contribution

Quantitative:

• Reduce design-QA time necessary to bring a feature to market
• Reduce design deliverable process time
• Increase efficiency between design and engineering
• Increase ‘System Usability Scale’ (before versus after design system)
• Increase ‘Net Promoter Score’ (before versus after design system)

What process did we take to create a design system?

Establishing a foundation

Building a design system takes time. It was important to clearly define our starting point. To do this, an in-depth UX and UI audit was performed on every page of the ReviewTrackers platform to identify patterns and inconsistencies with existing components.

Next, a library of reusable components was created in Figma for the design team to utilize. This allowed for consistency in design deliverables and avoided additional design debt while the creation of the design system took place.

Getting people excited

At ReviewTrackers, we have monthly Product and Engineering lunch and learns, which allows employees to share role knowledge with their colleagues.

Given that the audience for a design system is multi-disciplinary, it’s important to take this opportunity to spread design-thought leadership.

Since the design team’s initial focus was on the component library, this is what we presented. We shared a brief overview of what a design system is, what a component library is, and our process for integrating a component library into the larger design system. We had a total of 20 attendees!

Setting ourselves up for success

Once we had our component library established and shared cross-functionally with colleagues, it was time to jump into the design system. The design team had a brainstorming session to identify a list of components to include in the first iteration. The components were added as cards in a shared design team Trello board.

Next, to help inspire and influence, we explored other design systems to learn about the approach, format, and structure others took in an effort to determine what would work best for us. From this exercise, we learned that design systems typically share the following characteristics:

• Brand outline
• Foundation/Principles
• Components
• Content/Writing
• Search capability
• Content all follows a similar do/don’t format
• Green and red accents to help visually denote the do/don’t concept

As a result, we applied the UX heuristic Jakob’s Law and ensured our component documentation followed similar principles.

We created standard templates for how we would construct our component documentation to ensure each piece followed the same approach. From here, we divided and conquered.

1. A designer would choose a component. Using the template, they’d write documentation for that component.
2. In an effort to always be improving, we’d note areas of improvement along the way and document design debt.
3. Once the documentation was finished, it would be added to a shared team Google doc. Another designer would review for content and a second designer would review for the overall format, tone, and voice.

It took about 6 months to complete this process for 19 total components.

Putting the design system into motion

The design system is meant to be a shared source of truth for designers, product managers, software engineers, and QA engineers to reference in their day-to-day work. With documentation completed, the next step was to determine where this shared source of truth would live.

After brainstorming as a team, we determined Storybook was the best location for the design system, an internal resource engineering uses for components. The design team shared this idea with the engineering team and they were immediately on board.

We worked with a lead front-end engineer to establish a process for designers to contribute design system documentation to Storybook. This resulted in designers getting access to Github, learning how to create a markdown file, and submit a pull request. We created a peer review system and had proper checks in place.

Integrating the design system into standard processes

The design team shared what we accomplished at “Team Demo Days”, a dedicated time for the product and engineering org to come together and share what they’ve been working on. With design system documentation available for anyone at the company to use, the next step was to integrate it into standard processes. With consistent knowledge sharing, regularly referencing it as a source of truth, and bringing increased visibility to the guidelines and best practices that were established in the design system, the hope is that it will become an integral point of reference in the team’s daily workflow.

Looking forward

Design is never done, and therefore, neither is a design system. What does the design team have in store for the second iteration?

Immediate next steps:

Design <> Engineering

Our engineering colleagues have been bought into the design system since day one. They’re invaluable resources when wanting to learn how design can continue to foster strong design <> engineering relationships.

Through bi-weekly meetings with engineering, the design team learns to understand engineering pain points in the front-end development process, translate those to problem statements, and determine how to make updates to our system.

Type scale

In the current iteration of the design system (component library), design worked to replicate the v1 type scale front-end engineers use in the product (size, weight, etc.)

However, we learned that there’s an opportunity to create a new type scale with engineering. This is because we uncovered v1 is not a well-defined scale, legacy, and front-end engineering also has frustrations with it.

We’re currently working on implementing our v2 type scale within the product.

Spacing system

We uncovered that a large pain point is the lack of defined, shared spacing rules between design and engineering. Design is leading the charge on what would afford the best experience for users, where engineering is sharing perspective on how they apply this concept within code.

We’re currently working on implementing our linear spacing system within the product and applying a mutually understood nomenclature to spacers.

From this work, a future iteration could be providing different layouts within the product (i.e., similar to how Gmail offers Compact, Comfy, Cozy).

Sharing component documentation

Now that component documentation is live (i.e., beyond a private Google doc) and exists within Storybook, it’s time to share the resource with the broader organization. This is important for several reasons:

1. Promotes design within the organization
2. Allows all departments to speak the same design language
3. Encourages growth, feedback, and insight to inform existing gaps or improvement areas with our system

Additionally, the reason why the design team chose Storybook was that it already serves as a cross-functional tool, so we’re seamlessly integrating it into that workflow.

Future next steps:

Having code associated with each component

An ideal state would have code related to each component in the design system, so engineering can “grab and go” with components. This would also offer a path for a consistent design system <> code parity.

Making the design system public-facing

To expand the reach of ReviewTrackers design, we have aspirations to eventually have this resource in a public-facing environment.

Injecting product principles into the design system

Design and product are working together to clearly define product principles, which will help inform future iterations of the design system to align with said principles.

Rather than the design system being seen as a separate entity, it needs to be seen as an extension of product principles. This will strengthen the system.

Before you go

Getting to this point has taken a lot of work, but we’re incredibly excited to build on the foundation we’ve created and further empower users, colleagues, and ourselves!

We’re always interested in growing our team and hearing from talented designers. If you have any questions or would like to get in touch, please reach out to product@reviewtrackers.com

Establishing a Design System at ReviewTrackers was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 1: Defining the Task

What is Sentiment Analysis?

Sentiment Analysis is a task in natural language processing (NLP) in which a piece of text is assigned a sentiment prediction. At its simplest, this could mean classifying an entire sentence as positive or negative. Let x be the input the model receives (text that has been segmented into individual tokens) and y be the desired output (our sentiment label):

x = ['I', 'loved', 'the', 'food', '!']
y = 'positive'

However, natural language is wonderfully diverse and messy, so for many applications a more fine-grained approach is more informative. Consider the example below:

x = ['I', 'loved', 'how', 'flavorful', 'the', 'burger', 
     'was', ',', 'but', 'wish', 'the', 'customer', 
     'service', 'was', 'faster', '.']
y = 'mixed'

This text chunk contains multiple different sentiments within the same sentence. Here, a one-sentiment-fits-all model is very limited. One compromise is to predict the frustratingly uninformative label “mixed” sentiment. Ideally, we would want our model to predict that within that sentence, “burger” has positive sentiment and “customer service” has negative sentiment.

What is ABSA?

Aspect-Based Sentiment Analysis (ABSA) is one way to address the problem above. In this version of the task, sentiment predictions are made for every “aspect” in the text, as shown below:

x =(['burger'],
    ['I', 'loved', 'how', 'flavorful', 'the', 'burger', 
     'was', ',', 'but', 'wish', 'the', 'customer', 
     'service', 'was', 'faster', '.'])
y = 'positive'

x =(['customer', 'service'],
    ['I', 'loved', 'how', 'flavorful', 'the', 'burger', 
     'was', ',', 'but', 'wish', 'the', 'customer', 
     'service', 'was', 'faster', '.'])
y = 'negative'

Note that in practice, this is often part of a larger pipeline in which a step called “aspect extraction” is applied before ABSA to determine which words in the text should be treated as aspects. Aspect extraction might be conceptualized as a sequence labeling task. Here, each token is predicted to be ‘B’ for the beginning token of an aspect, ‘I’ for any intermediate token in a multi-token aspect or ‘O’ for other:

x = ['I', 'loved', 'how', 'flavorful', 'the', 'burger', 
     'was', ',', 'but', 'wish', 'the', 'customer', 
     'service', 'was', 'faster', '.']
y = ['O', 'O', 'O', 'O', 'O', 'B', 'O' ,'O', 'O', 'O', 
     'O', 'B', 'I', 'O', 'O', 'O']

Part 2: Defining the Model

BERT

Assuming we have our text and its aspects extracted, how do we train the model for ABSA? Here, we will start by using BERT, a versatile, pre-trained language model.

A language model (LM) assigns a probability to a combination of tokens. For example, reasonable LM should give a higher likelihood to [‘turn’, ‘right’] than [‘turn’, ‘write’]. This deceptively simple framework is what allows voice recognition software to use context to distinguish similar sounding words. LMs also allow you to make predictions: out of all words in the model vocabulary, which is the most likely to come next in the sequence [‘take’, ‘it’, ‘or’, ‘leave’]? A good LM would probably select ‘it’ as the next token.

If we were starting from scratch with randomly initialized parameters, we would in effect have to teach the LM the basics of the language just to get started. Luckily, BERT is already pre-trained on massive amounts of data (all of English Wikipedia, plus a large corpus of books).

For many machine learning tasks, labeled training data is needed. Training using manually labeled data is referred to as supervised learning. However, data annotation is time-consuming and unrealistic for the scale of data that BERT is trained on. To get around this, BERT is trained using a clever approach called self-supervised learning. One self-supervised training task is called masked language modeling. In this task, a certain token is masked out and the model tries to predict what this missing word is. Let’s say that the sentence “The roast beef was tasty.” occurs in the dataset. From that, a training pair can be automatically generated without any manual annotations:

x = ['The', 'roast', '[MASK]', 'was' 'tasty', '.']
y = ['beef']

You can play around with the BERT masked language model here:

Post-Training

This step is optional. If you skip directly to the fine-tuning step, you can still have a high-functioning model. However, post-training has been shown to help. Xu et al. (2019) demonstrate improved performance of BERT on ABSA for online reviews by post-training it on data from that domain. Post-training simply means initializing the model with BERT’s pre-trained weights and continuing the same self-supervised training regime on a more specific dataset. For example, if your downstream task covers healthcare and you have access to a large amount of unannotated healthcare data, this may be a good step to take.

Here is an interactive example of Xu et al.’s model post-trained on online reviews:

Fine-Tuning

So far, we have only used BERT as a language model, but ultimately we need it to predict sentiment polarity, not word likelihood. Adapting the architecture for sequence classification and running fine-tuning allows us to do this. First, we add a classification layer on top of the language model. Mathematically, this functions as follows:

P = softmax(CWᵀ),

where C ∊ ℝᴴ and W ∊ ℝᴷˣᴴ

Let’s unpack that. C is the output of the final hidden layer of the LM from the special classification token (more on that later). The notation above indicates that C is composed of real numbers ℝ and its shape is that of H, i.e. the size of the previous hidden layer. In the case of the base model of BERT, H=768. W refers to the weights of the new classification layer, which will be updated during fine-tuning. It is a tensor of dimensionality K by H, where K is the number of classes we are predicting across. Since we are using a three-way distinction of positive/neutral/negative, K=3.

Taking the dot product of C and the transposition of W results in an array of size K. These are sometimes referred to as “logits”. These values need to be normalized to have a probability distribution that sums to 1. The softmax function does this for us. To get the predicted label, just return the label corresponding to the highest probability:

y_hat = labels[argmax(P)]

🤗 Hugging Face’s transformers library in PyTorch makes it easy to import BERT with this classification layer already on top:

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=len(labels))

This framework also makes it quick and easy to try out different architectures and flavors or BERT. Think RoBERTa might boost performance?

from transformers import RobertaForSequenceClassification

model = RobertaForSequenceClassification.from_pretrained("roberta-base", num_labels=len(labels))

Want to use the version of BERT that is already post-trained on online reviews?

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("activebus/BERT_Review", num_labels=len(labels))

Want a more light-weight model that can run inference more quickly?

from transformers import DistilBertForSequenceClassification

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=len(labels))

etc. How convenient!

Our model is now structured to assign a probability distribution over our label set. Because of the nature of the task, annotated training data is required, making this a supervised learning problem. There are plenty of guides with sample code available to show to to fine-tune BERT for sentiment analysis. Again, using PyTorch and Hugging Face make this a relatively easy process.

Structuring the Input

As for how to include the aspect in the input, several leading models use concatenation, such as prefixing the aspect before the whole text with a separation marker in between. For example, you could structure the input as follows:

x = ['[CLS]', 'customer', 'service', '[SEP]', 'I', 'loved', 
     'how', 'flavorful', 'the', 'burger', 'was', ',', 'but', 
     'wish', 'the', 'customer', 'service', 'was', 'faster', 
     '.', '[SEP]']

Here, ‘[CLS]’ and ‘[SEP]’ are special tokens that BERT uses to denote sentence classification and sentence separators, respectively. The classification token is the token whose weights are fed into the classification layer of the network. The separator token is often used for completely different NLP tasks where the input consists of two distinct parts (next sentence prediction, question answering, etc.), but BERT is adaptable enough to learn from the fine-tuning data that this structure is used here to indicate the aspect to focus on for sentiment prediction.

With this (and some training data and processing power), you should have all the parts necessary to make a strong ABSA model. Here is an overview of the training stages. As a reminder, pre-training has already been done for us, and post-training is optional:

If you want to compare your model to others, SemEval 2014 Task 4 Subtask 2 is a common evaluation benchmark. As of early 2021, leading papers with more complex architectures (Rietzler et al., Tian et al., etc.) have pushed the bar up to just a few percentage points shy of 90% accuracy on that dataset. Best of luck building your own model and pushing the bar even higher.

To learn more about some of the other NLP tasks that ReviewTrackers handles, check out our white paper.

Building a Cutting-Edge Model for Aspect-Based Sentiment Analysis was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we run our tests in CI

In our test environment, we stand up ephemeral copies of our development stack for both our integration and QA testing. The methods behind this are subject enough for their own dedicated blog post, but essentially engineers pass in their branch name, and we deploy a namespaced version of our full stack built around that branch into our Kubernetes cluster. QA tests receive ingress objects in order for the developer to connect via an internal URL, i.e. app.qa-1234.internaldomain.com, whereas integration tests run within the namespace, and connect directly to service endpoints using the cluster FQDN, i.e. app.qa-1234.cluster.svc.local (so that kube-proxy handles the networking, saving on networking costs).

For load testing, we do very much the same as we do for integration tests - that is to say, deploy a stack based around the requested branch sans ingress. Since we don’t yet use our Kubernetes infrastructure for our higher environments, there’s no need to add that layer of obscurity to test results. While that ephemeral environment is being stood up, we also build out a very simply Docker container based off the official Python Docker container for our Locust tests. Locust provides documentation for running the tests from a Docker container, so I won’t get too much into that.

I will say though, in our CI testing we did not find it necessary to run a distributed version of Locust. While we did develop the capability to run Locust in a distributed fashion for these tests, we found that a single Docker container on a Jenkins drone with 2CPU/8Gi RAM was more than enough to run our test suite while simulating thousands of users.

Once the stack has been stood up and the test container built, we trigger the tests, passing in user-defined parameters like user count, runtime, and hatch rate as environment variables. Although we’re running the tests in a Kubernetes cluster, we elected to invoke the tests by utilizing Docker in Docker as described by fellow Trakkie David Kirk in his previous post here. Aside from simplicity and consistency with our other test methods, we did find running the tests in DinD to be much quicker than running it via a Kubernetes job.

Retaining time-series data from test runs

Now that developers are able to run their tests with a simple Jenkins job, what should we do with that information? A CSV dump of Locust output might be helpful to some, but the vast majority of developers that merely want to validate the performance implications of their changes. So that really doesn’t really net us much gain for our general audience. We realized we needed a way to compare the performance of one run against some sort of baseline. Rather than try to transform the CSV data into a more persistent datastore, we looked to the opensource community.

Luckily, it seems we were not alone in our endeavor: the locust-plugins project contained exactly what we needed with their listeners plugin. Once you add the plugins package to your project, sending data to a database is as simple as configuring default postgres environment variables in your tasks.py [e.g. PGHOST, PGPORT, PGDATABASE, PGPASSWORD] and adding a listener to your tasks.

I won’t get too much into the Postgres setup, since there’s no shortage of guides to getting it set up in any sort of environment you can imagine. I will say though, rather than using TimescaleDB as indicated in the plugin repo, we simply used the official Postgresql helm chart, mounting the timescale schemas as an initialization script to bootstrap the DB. This was simply due to our familiarity with setup of the official Postgres chart; TimescaleDB does appear to offer their own charts, so feel free to check those out!

Visualizing the data

Now that we have our run data retained in a logical format, let’s visualize it! And what better tool for that than the ever-ubiquitous Grafana? Again, I won’t get too much into deploying Grafana within your environment, since there’s a multitude of resources out there. However, I will talk about one very neat feature that we leveraged for our setup. We are big into configuration as code here at ReviewTrackers, and if anyone has ever spent hours setting up a dashboard before, you know you do not want to lose all that hard work.

The official Grafana helm chart offers up instructions for how to import dashboards, but I still required a fair bit of tweaking in order to get it working properly. The main gotcha we found was that if you want to configure a dashboard, you must also configure a matching dashboardProvider. As illustrated below, in order for our load-test dashboardConfigMap to be properly loaded, you need to first configure a load-test provider, and your options.path for said provider must match the name as well.

dashboardProviders:

  dashboardproviders.yaml:

    apiVersion: 1

    providers:

    - name: 'load-test'

      orgId: 1

      folder: 'Load Test'

      type: file

      disableDeletion: true

      editable: true

      options:

        path: /var/lib/grafana/dashboards/load-test

dashboardsConfigMaps:

  load-test: grafana-dashboard-locust

Our dashboard is based largely on the excellent one provided by cyberw on GrafanaLabs. As you might be able to tell above, we track our dashboard.json via our own source control, and import it as a configmap into our cluster. If you’re a savant, you can technically edit your dashboard directly in this json, but we found it much easier to edit dashboards directly in Grafana, export them as json, and use that to update source control.

Now that the chart was ready (and we configured our Postgres data source), we deployed it, and after navigating to the load-test dashboard in Grafana, things were starting to come together:

If you’ve already sent some run data to your Postgres database, simply find the time range for your tests, and you should see your data! If not, kick one off and watch the data pour in! Once you have data, you can filter by endpoint by clicking it in the dropdown at the top, or clicking the name within an individual graph.

Providing meaningful run data

Now we have our beautiful dashboard for people to view locust data, but they still can’t really extrapolate much from it. Say for example there were multiple runs at the same time, or and extended period of time where no tests were run: it’s going to make it hard to isolate your data, or compare it to previous runs if there’s a large time break. In order to work around this, we need to:

1. Identify each run as an individual test
2. Provide a way to easily compare your test to previous runs

Luckily, Locust provides a very easy way to achieve the first goal - within the listener plugin, you can pass the testplan variable to uniquely identify each individual test. We simply set this to an environment variable based on the Jenkins build ID, and voila, the dropdown in the dashboard allows you to choose individual runs:

Now that we were able to identify each run, we wanted a way to compare against some sort of baseline. Rather than use an aggregate of individuals’ runs (which may be subject to a lot of outliers due to failing tests), we decided to aggregate the data from our runs against staging. Since staging is our closest environment to production - and we regularly run the same tests against staging to test capacity - we figured the best way to achieve this was to place the averages from staging next to the run’s results directly on the dashboard.

Now as great as Grafana is, can be kind of a pain. If we wanted to do something like join two queries, the result set wasn’t always as pretty as we’d like it to be. Luckily, just before we started our visualization endeavor, they released Grafana 7, which includes a neat little feature known as data transformations. Essentially what these do is allow you to chain all sorts of result filters together, such as, let’s say, joining two result sets, and re-organizing the columns in a more readable way ;)

We added the following query to the table:

SELECT Req1.name, Req1.request_type as "request type", 
  AVG(Req1.response_time) as "Average - STAGING",
  percentile_cont(0.95) within group (order by Req1.response_time)   as "95 perc - STAGING", 
  percentile_cont(0.99) within group (order by Req1.response_time) as "99 perc - STAGING", 
  max(Req1.response_time) as "max - STAGING"
FROM request Req1
WHERE Req1.testplan ='staging'
GROUP BY  Req1.request_type, Req1.name

And then simply went over to the ‘Transform’ tab, joined on the proper column (name of test), and sorted the columns how we wanted them:

An just like that, each column has it’s respective staging average right next to it! There’s more cool stuff you can do like changing colors over certain result thresholds. Definitely play around with transformations, they can make your dashboards even more powerful!

Putting it all together

By establishing an accessible method for developers to run load tests (ChatOps ftw), we are able to provide our engineers with a quick, reliable way to test the performance impact of their changes. With pretty basic deployments of Grafana and Postgres, as well as adding the Locust plugins package, we can retain our run data in a much more usable manner. Finally, by leveraging the new Grafana data transformations functionality, we are able to provide a simple dashboard to our engineers so that they can quickly determine the impact of their changes.

Visualizing Load Test Data with Locust and Grafana was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some organizations start Load Testing with a specific goal, like this:

During Black Friday, we need to support 100k users checking out of our site

At ReviewTrackers our specific goal around Load Testing was to provide evidence that we could support 3x peak user traffic. However, we also had some less quantifiable engineering goals, the kind of goals that make you feel warm and fuzzy inside. The ones that you imagine are used at the most cutting-edge techie companies. These goals were to have:

• A baseline performance for our APIs, to be able to determine if they had become slower as we did development. We’d have a check in CI that would fail if any API regressed.
• An easy way to elicit bottlenecks or outliers with our APIs at high load, or when dealing with lots of data. The tests would be able to provide consistent proof of this. Then, we could use the tests to benchmark any improvements we made to those kinds of requests.

There were some other requirements of our tests:

• Tests need to be able to run in CI, and also, against deployed environments. This is important because of the differences in hardware between environments.
• Tests need to grab data from their environment to construct the urls to hit. Many of our REST APIs require specific information to work, like sessions, review_ids, location_ids, etc.
• Tests need to be easy to use and contribute to. We want the engineers to contribute to them as they see fit and drive the adoption effort.

We searched the internet for Load Testing tools, and settled on locust.io, a great, open source load testing tool that reminded us of Flask, a python micro framework. Both libraries are small, unopinionated, and easy to use.

However, we needed to write code on top of locust to fit it into our ecosystem. I’ll mention some of the key things we did, that weren’t so intuitive:

#1 Use the same session, user, and account across tests

Our api requires a session. Each login to our website generates one of these. If you login on a different computer, you get a different session, even if you are the same user.

In the real world, 3x peak user traffic is made up of various accounts, users, and sessions. However, for our purposes, it was easiest to use the same session, user, and account for each spawned locust User. There are a couple of reasons why we decided to do it this way and not simulate the real world:

• If we pointed to existing accounts/users, there’s a possibility we’d unintentionally invalidate their sessions.
• We don’t want to create tons of accounts/users specifically for this testing. Ideally, we’d just target an existing large account.
• The differences in data between accounts/users could add variance to our results. We’d prefer to just point to one big account, to lessen the variability.
• We have a session limit per user, and we didn’t want to keep track of this number throughout the tests.

Obviously, every piece of software is different, so our approach might not work for your software, if you have rate-limiting or other limitations. But this paradigm works for us. And because we are targeting one account/user, we can ensure that the account has all their features turned on, and has the ideal data to test against.

#2 Important setup code lives outside of the locust hooks

As discussed in #1, we need a single ReviewTrackers session and user to be passed to each spawned locust User. While we’re at it, the test should pass each locust User location_ids, review_ids, and other account specific information we need to cobble together urls.

However, we need the setup code to run only once. We want to create only one session used by all spawned locust Users. As well, the setup code goes out to the databases in question, and grabs the ids, we don’t need to do that repeatedly.

All I could find in locust’s documentation was the on_start hook that runs when spawning a new locust User. That happens too often. So a good place to put this setup code was in the body of the TaskSet. By doing it this way, setup is run once, and we can pass information to each locust User through the on_start hook using class variables:

#3 Override the name parameter so requests with dynamic data are grouped together

As you can see from #2, we have dynamic urls based on data from the environment. We also add a nonce to every request, to ensure our requests aren’t cached by our service providers. This means every single url is unique.

By default locust groups requests by url, so without customization, our locust report was hard to use. However, we found this part of locust’s documentation talking about grouping requests with the name parameter. Once we implemented this, our reports were much easier to read.

#4 Make your locust tests configurable and easy to use

We wanted anyone to be able to jump in and easily run locust tests, without needing to know the details of how they worked. To do that, we did a couple of things:

• We used virtualenv so anyone could download the packages they needed for locust, and not interfere with the python on their machines
• We created an invoke task that runs the tests with good default options

We provided some environment variable arguments that users could pass in to customize their experience as well. Users could pass in:

• ENV: Environment to point to
• ACCOUNT_ID: Account ID to use
• HEADLESS: If true, tests are immediately kicked off headlessly (like they are in CI). If false, you must visit locust GUI (at http://localhost:8089), provide RUNTIME/NUM_USERS/HATCH_RATE, and kick off the tests in GUI
• RUNTIME, NUM_USERS, HATCH_RATE: defaults are RUNTIME=5m, NUM_USERS=200, HATCH_RATE=5. These arguments are necessary if HEADLESS=true. Here is the guide for running locust tests without the web ui
• FILTERS: Comma-delimited list of regex expressions. If passed in, only urls that match the expressions will be run. Useful for testing adding new urls

Here’s what our invoke script looked like:

Outstanding questions

I want to end this article with some outstanding questions I had about load testing. I wasn’t able to get any answers from load testing articles or books. I hope people reading this article can offer up answers to these questions!

• Should you ping endpoints with the same arguments or different arguments?
• Can you locust tests POSTs, or should you only test GETs?
• Does load testing measure website experience, or just lower level performance? Is there a way to get load testing to capture async requests on a page?
• Is it good design to POST a resource then GET it right away, replicating a real workflow? (Haven’t seen ANYBODY do this kind of test in example code)
• Do you make ASSERTIONS on status codes, and fail load test runs based on that?
• Do you make ASSERTIONS on the SPEED of endpoints? Or is load testing always for manual review?

Thanks for reading this article, and I hope these locust tips are helpful!

Our Load Testing Journey with Locust was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Each deployment of our product is running across 30ish docker containers, so having to build it all from scratch takes a long time —unacceptably long for a stage in a core development process. We need some solution that allows us to reuse the docker cache for every build — regardless of where it is taking place in the cluster. Additionally, we needed that solution to be straight forward to deploy and run in Kubernetes.

We’ll briefly cover some of the earlier attempts at solving this issue, then go over the solution that we landed on and are using in production right now!

Early failures — Kaniko and DinD Sidecar

We initially wanted to go with Kaniko, a tool created by Google that’s intended for building docker images in a Kubernetes cluster. While this sounded perfect for doing the job, we quickly ran into issues. First, each build is run in a separate container, which sounds like a good idea, but that also means that each individual build has a non-negligible startup time — especially if it needs to spin up a new VM to deploy to! Beyond that, we ran into consistent issues with getting a build cache to work — I won’t go into detail, but the documentation for how to use Kaniko with Skaffold is almost non-existent, so that plan had to be shelved.

We decided to bake our own solution, and, after a screening of Inception, we came across the idea of using a Docker-in-Docker container to do our builds. Outside of lame DevOps jokes, it can be hard to find a good use for Docker-in-Docker, but this scenario actually ends up being pretty appropriate! This solution allows us to treat our image building service like any other Kubernetes deployment, while also isolating those builds away from the rest of the cluster.

Our first attempt at this involved augmenting our Jenkins agent pods with DinD sidecars. While this did technically work, we ran into very significant issues regarding, again, the build cache. Initially, we were hoping to just be able to download base images to our builders’ docker caches, but that took way too long. Additional experiments with prepopulating a cache & copying it to other containers also ended up either not working, or being too slow. We felt like we were really onto something, but the final pieces weren’t really clicking in place.

The Final Pieces Clicking In Place

After a few more struggles, it became clear that we needed to rethink our strategy. We were already consuming the docker daemon endpoint from another container, so what if we stop restricting it to being just a sidecar, and instead expose it as an endpoint that can be hit from anywhere in the cluster?

We decided to spin up a DinD pod on a relatively beefy machine (c5.2xlarge), and dedicate that whole machine to docker builds. Whenever a Jenkins job calls for building an image, it runs on a “docker-builder” agent, whose DOCKER_HOST targets the DinD service. If a job calls for running a container, it instead runs on a “docker-runner” which has its own small DinD sidecar!

This allows us to share the docker cache across every build in our cluster, which has led to way more consistency in our builds, and greatly simplified the process of building a container. Our worst-case build time happens far less often. While there are some negatives, such as not benefitting from autoscaling, the positives in terms of simplicity to use and maintain make us believe that this is a solid solution that we plan on sticking with for a long time.

Our DinD Service is configured with the following env vars to enable this behavior:

spec:
  containers:
  - env:
    - name: DOCKER_TLS_CERTDIR
      value: [PATH TO TLS CERT DIR]

And our image builders contain the following env var (the DinD service is the jenkins namespace)

spec:
  containers:
  - env:
    - name: DOCKER_HOST
      value: tcp://dind.jenkins:2375

And that’s about it! We’ve been using this solution in our CI cluster for a few months now, and the only problem we’ve seen so far is disk space usage, which can be managed with just a few docker image cleanup commands. Overall, we’re very happy with how well this has performed, and hopefully it’ll treat you just as well!

Kubernetes In-Cluster Docker Image Builds was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I noticed that our traffic for this service had steadily increased over the last few months, which was now approaching about 2.5M monthly views. Going from ~160k weekly views in January to about ~575k weekly views in May or a 250% increase in traffic (go us!).

Investigating the problem

So my hypothesis was that one of our slower endpoints was eating up to much HTTP traffic, which was then slowing down all endpoints at peak usage time. I went into New Relic (the tool we use for instrumentation) to investigate. Sure enough, it looks like one of our endpoints was accounting for the bulk of the HTTP traffic

The analytics endpoint is responsible for passing analytic events like “views” to our analytics provider is taking about 75ms on average per request. For comparison, most of our widget endpoints take about 10ms on average. Additionally, it takes up about 67% of our HTTP traffic for the service, which is really high. Since this endpoint is invoked every time a widget is loaded on a page, this endpoint being slow can start to affect other endpoints with an increase in traffic to Amplify due to only so much concurrent HTTP capacity.

So to ensure we’re able to better scale to this increase in traffic, we need to address the slowness of this endpoint, which would then ensure that we can support more concurrent requests.

Short Term

Since we use Heroku to host our applications, scale up the number of web dynos (servers shared within their load balancer for the app) for the service, which will increase the number of concurrent requests it can support for the time being. This will compensate for the slowness in this one endpoint.

Long Term

Identify the bottlenecks in the endpoint and refactor it.

Digging into the endpoint

New Relic provides a way to see the breakdown of the endpoint’s time to see where the bottleneck could be. When looking at the breakdown table, it looks like most of the time is coming from the PUT to RabbitMQ. Basically we use RabbitMQ internally as a message broker for Amplify. A request is made to POST the analytics payload to our API, which then queues a message to the relevant RabbitMQ queue on our exchange. Then an analytics consumer takes the payload and interacts with the analytics system to pass along the information.

The goal of this architecture is to prevent situations where the analytics system slows down our API. As long as we can ensure that our interaction with our RabbitMQ server is fast, this request should be pretty quick. Typically publishing a message to RabbitMQ should be a really fast operation, so it seemed odd that it was accounting for most of the time.

After looking at the code, I came to the conclusion that the reason it was taking so long was because we were creating a new connection and channel to RabbitMQ every request and then waiting to close the connection. The Publisher service not pictured below creates a connection and channel as part of being instantiated.

Refactoring the endpoint

Creating the connection/channel and closing it every request seemed pretty wasteful to me, since we were incurring the overhead every request. One of the complexities however is that we use Puma as web server, which gives us additional concurrency for our API using threading. Since threads are being used, we need to ensure it’s thread-safe in that the connection/channel can be shared amongst many threads spawned by a Puma process. Additionally, I noticed that overall we have pretty low memory usage for our Web dynos. This made me think we could also leverage a feature of Puma called clustered mode. Clustered mode allows Puma to fork the main process into multiple child processes called “workers”, which then each child process spawns threads (in this case we were spawning 16 threads). If we have two workers that each have 16 threads, that means that we can support 32 concurrent requests on a single Web dyno.

So to ensure that the operations are thread-safe, I navigated to Bunny’s documentation (the Ruby gem we used to interact with RabbitMQ). It highlighted that we should create a global connection pool for each forked process. Thankfully, Puma makes this pretty easy with on_worker_boot and on_worker_shutdown blocks that it provides as part of its configuration.

Then with this new globally available $rabbitmq_channel, I can update my publisher to accept it as an argument instead of creating its own connection/channel and having to close it

With the new Puma configuration, I just need to set MAX_CONCURRENCY to 2 as an environment variable, so that I can fork my process into 2 instead of 1. This will eat up more memory, but we have a ton to spare since not much is used in production.

To understand if my changes made an impact with the endpoint, I re-used my benchmark script using Vegeta to see its performance at 500 requests/s for 10 seconds against my service locally.

Before

-- Request Info --

Num Requests: 5000
Request Duration: 9.998184723s

-- Metrics --

Mean: 313.888607ms
50th percentile: 0s
95th percentile: 99.389346ms
99th percentile: 11.099165562s

After

-- Request Info --

Num Requests: 5000
Request Duration: 9.998289959s

-- Metrics --

Mean: 45.002096ms
50th percentile: 2.728643ms
95th percentile: 88.35957ms
99th percentile: 1.167418792s

As you can see, the average response time went down considerably with these changes from 313ms to 45ms, an 86% improvement.

Measuring the impact in Production

So to understand if my changes had a positive effect on production as they did on my local, I looked at New Relic again after it was deployed for a few hours.

As you can see our average response time for the endpoint went down from 75.3ms to 12.5ms in production. Also, the endpoint now only accounts for 37.9% of our HTTP traffic. With the two Web dynos running and the forked processes, we are now able to support 64 concurrent requests, which means that we can handle 4 times the amount of traffic than where we were when we started. Hopefully, this will give us more runway for the next few months.

Key Takeaways

Instrumentation

Setting up instrumentation is super important when deploying a new service. As things like this come up, it’s very convenient to be able to understand what endpoints are slow and what parts of the request make up the bulk of that time. This makes fixing issues so much faster since most of the exploratory work is done for you in Production.

Proactively fix performance issues

As your API gets more usage, it’s important to keep an eye out for bottlenecks and prioritize fixes. This will ensure that slow endpoints don’t eat up all of the request capacity, which will then affect the performance of your API as a whole.

Benchmark performance issues locally

When testing a performance fix, it’s important to have realistic benchmarks for before/after performance so that your change has a high probability of actually working in production. I typically download a copy of the production database locally when testing, so that I can work with realistic benchmarks. For this specific fix, it wasn’t anything database related, but for others it may be. Using tools like Vegeta also help to give you a clear picture on the effects your change has made to the performance of the API at production scale.

If you have any questions about our approach or just want to talk shop, you can reach me on github at http://github.com/plattyp and email andrew@reviewtrackers.com

Refactoring slow code using instrumentation was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR GraphQL is amazing and can be incrementally integrated into a REST app but be wary of the client side cache.

The story of GraphQL at ReviewTrackers goes all the way back to our very first Hackathon. What started out as an experimental “hack” project has matured into a production service that now serves our most important page in our application — the reviews page. While plenty of articles and tutorials exist on integrating GraphQL into greenfield applications, there is a lack of resources on integrating GraphQL into existing applications. I’d like to share our journey of integrating GraphQL into our app, both the ups and and downs, so hopefully others can take from our experience when considering to adopt GraphQL. I compare our experience to swimming — incrementally dipping our feet into the pool until we jumped into the deep end…

Background: Why GraphQL?

At ReviewTrackers, we maintain two applications — a React web application and a React Native mobile application. Both applications interface with our backend through a REST API. As our app grew and our data models became more complex, our client codebases also grew in complexity as it had to fetch and update all this complex data. A lot of our data is also relational — e.g. a review belongs to a location which in turn belongs to an account. This inevitably lead to over fetching / under fetching issues on our front end which we initially solved with sql views and dedicated REST endpoints for complex, relational data. While this was performant and worked, it was a clunky, time consuming dev experience and didn’t scale well since we’d have to deploy database migrations each time we needed new data and would also tightly couple our client applications to our database model. Not to mention that we’d have to make changes to both our web app and mobile app code bases if there were any breaking changes in the REST API. In short, we found that modern front end applications require a lot of flexibility in how they declare their data requirements — the exact pain point that GraphQL was designed to solve. After a successful hackathon of proving GraphQL’s viability, we started to slowly integrate GraphQL into our application.

Incremental adoption

At ReviewTrackers, we practice continuous deployment and push to production multiple times a day. This means that any change to our system should be incrementally deployed and not block our deployment pipeline — each deployment should be de-risked and other devs should be able to deploy. This means that a complete rewrite from REST to GraphQL was out of the question. Rewrites in my opinion are also rarely the right solution. So we strategically wanted to integrate GraphQL into our app through a path of least resistance — slowly dipping into the pool. We did this by keeping our interface to our backend consistent by wrapping our REST api in a GraphQL layer rather than interfacing with our database directly. In terms of integrating GraphQL into our client applications, we identified data fetching as the natural first step.

Data fetching: getting our feet wet

Most of our existing components fetched data and subscribed to them through redux like so:

const LocationName = ({location, loading, error, fetchLocation, locationID}) => {  
  useEffect(() => fetchLocation(locationID), [])
  if (loading) return <p>Loading...</p>
  if (error) return <p>An error occurred</p>

  return <h1>{location.name}</h1>
}

connect(
  state => ({ 
    location: state.resources.location, 
    error: state.resources.errors.location, 
    loading: state.resources.loading.location  
  }),
  dispatch => ({ 
    fetchLocation: (id) => dispatch(fetchLocationAction(id)) 
  })
)(LocationName)

This seems like a really nice component since everything is encapsulated into this one component but this type of component can introduce over fetching problems. For example, if we were rendering the component in two parts of the screen, we would be redundantly fetching the same location over the network. To mitigate, we could lift this fetching up to a higher common ancestor. Over time, this strategy made our root container components unruly with multiple resources being fetched, various loading and error states managed in redux, etc. Not to mention that our original component would become disjointed and dependent on another component.

In short, our components no longer became encapsulated.

Client side GraphQL libraries like Apollo solve this through a cache layer to prevent over fetching. So in the same scenario where our component is rendered in two parts of the screen, there will only be one trip over the network to retrieve the location which will then be cached and reused when another request for the location is fired. This allows us to build the encapsulated version of our component like so:

const GET_LOCATION_NAME = gql`
  query GetLocationName($locationID: ID!) {
    location(id: $locationID) { 
      name 
    }
  }
`

const LocationName = ({locationID}) => {  
  const {data, error, loading} = useQuery(
    GET_LOCATION_NAME, { variables: { locationID } }
  )
  if (loading) return <p>Loading...</p>
  if (error) return <p>An error occurred</p>

return <h1>{location.name}</h1>
}

Personally, I love this style of building data rich components. The loading and error states along with the data fetched are all encapsulated in a react hook. As long as we kept our data shape the same, we could migrate our redux / REST based components to GraphQL — amazing!

While we were stoked, the client side cache actually ended up biting us in more nuanced scenarios…

The reviews page: going swimming

We continued getting our feet wet with GraphQL by migrating less risky components to GraphQL. Everything was going smoothly when a bigger use case for GraphQL arose when we decided to refactor our reviews page to be more performant.

As a review tracking application, the reviews page is crucial to our users. As our company grew, so did our data which lead to performance issues for accounts with large review volumes. We tried tuning it with Redis in the past but eventually realized we would need to implement a new cursor based API to reap the biggest performance boost.

Cursor based pagination is the recommended way to paginate in GraphQL and we knew we’d have to implement this new API in both our web app and mobile app so we took this as an opportunity to implement this in GraphQL. It was the perfect fit!

After a few weeks of hard work, we rolled out the first version of the new reviews page (web only) behind a feature flag to a small subset of our customers. The results were tremendous with our largest accounts loading ~1900% faster.

It wasn’t without some hiccups though…

The client side cache: jumping into the deep end

Remember that client side cache? It was amazing and worked so well that some users complained the page was unresponsive when changing filters — a result of cached reviews causing the page to load too fast. We were stoked about how well this worked until we noticed some weird things…

When users would update a review status in the review detail page:

They noticed it would not reflect on the reviews page without a page refresh:

It turns out that our GraphQL cache is not updated when the POST request to update the review status is fired, causing this stale cache issue.

Sad 😞

To fix this, we updated the review status dropdown to also be driven by GraphQL so the client cache would be updated. This worked and we were able to still reap the benefits of the client side cache 😎.

Until we realized the same issue occurred when adding tags to reviews…

And when responding to reviews…

And basically any REST mutation not going through our GraphQL layer…

Sad 😞

Apollo provides some ways to manipulate the cache directly, which we initially tried but it turned out too cumbersome and imperative that we backtracked.

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

We ultimately ended up backtracking on the use of the cache entirely by configuring our Apollo client to use the network-only fetch policy which bypasses the client side cache and fires a network request per query. While we would take a slight performance hit, this tradeoff was worth it to avoid the headaches of cache eviction. We also were only firing the query in a single root component — i.e. multiple components weren’t firing the same query like our LocationName component above. This simple change allowed us to keep our reviews page in sync with any REST mutations, fixing all the stale cache issues above.

Configuring the GraphQL client: floatation device

We found that configuring and tuning the GraphQL client when integrating GraphQL into a REST application is key — our floatation device 😉.

We configured our Apollo client to default to a cache-and-network fetch policy which returns data from the cache if present but fires a network request in the background to validate the data and update the cache if needed (by default, the Apollo client will skip network requests entirely if data is in the cache).

This fetch policy optimizes for users getting a quick response while also trying to keep cached data consistent with your server data at the cost of extra network requests.

You can configure your ApolloClient to achieve this like so:

const client = new ApolloClient({
  link,
  cache,
  defaultOptions: {
    watchQuery: {
      fetchPolicy: 'cache-and-network',
      errorPolicy: 'all'
    },
  query: {
    fetchPolicy: 'cache-and-network',
    errorPolicy: 'all'
  },
  mutate: {
    errorPolicy: 'all'
  }
}})

Now all our queries will fire network request to revalidate the cache to avoid any issues with mutations occurring outside our GraphQL client. Furthermore, each GraphQL query can be individually configured to use a different network policy if needed — e.g. our reviews page still uses the network-only network policy to make sure we never show stale review data.

Our newly tuned GraphQL configuration allowed us to successfully integrated GraphQL on both web and mobile clients 🎊. Overall it has been a huge success and produced amazing results barring some hiccups. I have no doubt GraphQL will continue to improve as the community and tooling around it improves.

Key takeaways

• Incrementally adopt GraphQL starting with data fetching
• Have good automated tests to make sure nothing breaks along the way
• Rollout GraphQL migrations incrementally to users
• Configure your GraphQL client to skip the cache or revalidate the cache to avoid stale cache issues**

** More relevant when integrating into an existing REST application

Questions or comments? Find me on Github or email!

• Github
• kanehara@umich.edu

Interested in solving challenging problems like this on a well funded, fun team? Come work for us!

Our journey of integrating GraphQL into our REST app — learning to swim was originally published in ReviewTrackers Engineering, Product and Design Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.