In this quick summary we present a finding around the central Google Core Web Vital Largest Contentful Paint (LCP) that is expected to change in a drastic way with the recent release of Chrome 88. Overall, we found that the LCP is likely to drop by as much as 20%. Given Google’s upcoming ranking changes for May 2021, the LCP is more important than ever.

Background: What are the Core Web Vitals?

The Core Web Vitals are an initiative by Google, to provide better metrics to evaluate page performance and user experience. While previously it had already been possible to somehow capture the “felt page speed” of websites using metrics such as the Speed Index (the average time to visibility of page elements) pioneered by WebPagetest or the First Meaningful Paint (the time of the biggest visual change in page rendering), these methods had one severe problem: they rely on a synthetic testing environment and video analysis. So, they cannot be measured through browser APIs and with actual “field data”. Previous metrics that were available from the browser and navigation timing APIs like the time-to-first-byte and the first paint on the other hand do not fully reflect the real experience of speed.

Google introduced three core web vitals: the Largest Contentful Paint (LCP), the First Input Delay (FID) and the Cumulative Layout Shift (CLS). For each of these metrics, Google also proposes a “traffic light” essentially putting page views into three buckets good, ok, bad based on thresholds.

This is what the Core Web Vitals capture:

• LCP: measures how fast the page loads. For this, it captures the moment during the rendering process where the largest element containing content (e.g. text of images) is rendering on the user’s screen.
• FID: measures how quickly the page responds to user interaction. It is evaluated as the time between the first interaction (e.g. click or screen tap) until the event is processed in a JavaScript handler.
• CLS: measures how stable the page is rendering. The calculation is a bit complicated but evolves around how much content is jumping/moving by what distance during the rendering process. There are ongoing discussions on how to optimize the calculation to best reflect user-felt “annoyingness”.

Naturally, many consider the LCP as the central metric as it reflects the overall loading time of a page. Currently, the Largest Contentful Paint API is available in Chromium-based browsers, so covering around 70% of end users. In a second, we will explain why many websites might have scored badly in the LCP without actually doing anything wrong or being slow.

Measuring the Web Vitals

Today the main sources for the Web Vitals are Lighthouse respectively Page Speed Insights for synthetic (“bot”) performance measurements as well as the Chrome User Experience Report (CRUX). CRUX is a highly valuable public data set that contains anonymous real-user performance data from Chrome browsers for Millions of top domains. It is not only a great tool for performance analysis but also the basis for Google incorporating end-user speed as a ranking factor into its search.

For example, one interesting thing you can do is to rank the top 10 global e-commerce companies by their LCP:

Google Page Experience Update is around the corner

The web vitals were presented in May 2020. By the end of last year, Google announced that starting from May 2021 the Web Vitals will become part of the Google Search ranking. Since its Mobile Speed Update from 2018, Google has been considering speed in the rank and — unlike in the old days — speed is no longer measured by the Google bot but instead based on CRUX. Besides the organic ranking, speed also plays a role for SEA/ad placement.

While the quantitative impact of speed on SEO is of course a well-kept secret, there are some studies. For example, Pinterest has seen an additional 15% organic search engine traffic after improving page speed by 40%.

If even before the web vitals update speed has already played such a crucial role, it is very likely to become a lot more important. Many of our e-commerce customers at Baqend for example have recently invested into getting as fast a possible in the LCP before the May update comes.

The finding: for JavaScript-heavy websites the LCP was not accurate

Since a while we noticed, that the LCP seemed to be unreasonably high on some sites that relied heavily on JavaScript for client-side rendering. We then noted that a change to the LCP was launching for Chrome 88 that potentially changes a lot.

Just saw the LCP changes in Chrome 88 (https://t.co/Fmuoj6ui8S) fixing a huge problem we see where LCP is wrongly increased when images are placed in containers by SSR rehydration or slider code.Suspect this will almost halve the LCPs reported to CRUX for a lot of top sites.

Essentially, what we observed was this: when the element that makes up the LCP is changed by JavaScript in the DOM, the LCP is reset even if the page is visually not changing at all. For example, as on this test page, if we change the hero image after 3 seconds, the LCP will be reset and reported to be more than 3 seconds, even though nothing has changed!

Why is this important?

The problem is that many websites rely on client-side JavaScript to modify elements that are defining the LCP. For example, if you change a product image slider in a shop during rendering, you can display the image as early as you want, the LCP will still be bad.

The issue is even more systematic for websites relying on modern frontend frameworks such as React, Angular, Svelte, and Vue: as soon as server-side rendering is used (usually for performance reasons!) the client-side rehydration is changing the DOM and resetting the LCP. Rehydration is simply the process of “booting up JavaScript views on the client such that they reuse the server-rendered HTML’s DOM tree and data” (see this article for more details).

Bottom line: if your JavaScript is modifying your frontend you might have been looking at very skewed LCP numbers.

Give me data: what has changed?

Since Chrome 88 has been out for a few weeks now, we actually have the data to underpin our hypotheses. At Baqend, we are building Speed Kit, a SaaS plugin solution to speed up websites based on JavaScript and Service Workers. Our approach relies on measuring the achieved performance uplift through real-user monitoring and A/B testing. Therefore, we have lots of data, since Speed Kit comes with a performance monitoring tool that measures speed metrics for every page view and in particular captures the Core Web Vitals.

Here are the changes we observed between January 20 and February 4, when comparing Chrome browser with version <88 to the current version 88. The data set consists of 122 Million page views from Chrome browsers across roughly 45 e-commerce customer websites. Around 70% of the traffic comes from Europe the rest is spread across all continents. Overall, 51.1% of traffic came from Chrome browsers with a version <88, 48.9% were coming from Chrome 88.

As expected, the impact varies heavily across customers — this seems logical since not all of them a modifying LCP-relevant elements with JavaScript. The following scatter plot shows the impact across all analyzed customers.

Key Findings:

• Across the median, 75th, 90th, and 95th percentile, the LCP has consistently become ~20% faster in Chrome 88.
• For the 99th percentile (so the slowest 1% of page views) the change is a whopping 43.75%.
• Not all websites are alike, however, with very few exceptions the new LCP calculation has led to an improvement. There seems to be no correlation between the traffic/size of the site and the LCP change.

Conclusions

From this preliminary analysis, we expect drastic changes in the LCP. Whether this warrants to adapt the — already quite generous — threshold of up to 2.5 seconds considered as a “fast LCP” remains to be seen. For now, the good news is, that JavaScript-heavy sites need not longer fear a negative skew for the LCP. We will get back with an update once the CRUX data (updated once per month) allow a first confirmation of our findings in the “official” data sources.

The Core Web Vital “Largest Contentful Paint” just changed dramatically was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

With overloaded networks and frustrated users, it becomes vital for online shops to ensure fast and stable page loads.

Due to the corona crisis, e-commerce is facing massive challenges. People are staying at home. The increasing traffic threatens to overwhelm the networks — Netflix and others are already throttling, while the loading times of many online shops are increasing. At the same time, anxiety and tension are rising. Therefore, not only the overall consumption drops. According to a study by Google, customers perceive all page loads as being dramatically slower.

In the current situation there are two types of online shops. The first type are shops fighting for survival because customers stay away or buy significantly less. The others are downright overwhelmed by customers, so that loading times are brought to their knees by overloaded backends and networks. For the former, fast loading times are crucial in order to convert customers despite their anxiety through a frictionless user experience. For the latter, stable loading times are a basic prerequisite, for customers to be able to buy in the first place and not move away to faster competitors.

Will the network infrastructure and shop systems handle the load?

In addition to the concerns of many online shops, there is currently the nagging question whether the network operators will be able to cope with the increasing traffic. After all, Netflix, YouTube, Facebook, Instagram and Amazon are already reducing the quality of their media to reduce the growing load. At the same time, Internet usage in large parts of the world has increased by more than 50% since the outbreak of the COVID-19 pandemic.

As an example, we analyzed data from the largest German online shops to answer whether load times have increased since the outbreak of the COVID-19 crisis. As the above chart shows, the answer is clearly “yes”. Comparing January to March, 85% of the shops have seen a steep performance decline, in some cases to a dramatic extent. It is also striking that the effect is not only affecting the smaller or larger shops, but is spread across the entire e-commerce industry. The data comes from Google’s so-called “Chrome User Experience Report” and contains usage data from a large number of real users. Another analysis by Fastly found country-specific traffic changes between +32% and +109% associated with dropping download speeds up to 35%.

Even though the major network operators are assuring that sufficient capacity and bandwidth are available, it is clear that 85% of all shops need to take action to compensate for the slump in loading times in March. According to current figures from the German E-Commerce Association (BEVH), March sales in online retail have slumped by almost 20%. With a few exceptions (e.g. food, drugstore articles and medicine), all sectors are affected — just as with the drop in loading times.

How customer behavior is linked to loading times during the crisis

Many people know it from their own experience in home office: If “the Internet” does not work in the usual quality, patience can snap quickly. And that it is what millions of customers are facing at the moment. But page speed is by no means just a problem for those who have to share their 10 Mbit/s home network connection with partners and children. Often without realizing it themselves, customers react to the smallest differences in performance with significantly different behavior — and always to the disadvantage of the shop operator.

Under stress, 44% of users perceive websites as slower.

Performance study by Google and Awwwards

The following chart provides a summary of the most important studies and findings that online retailers should know in the current situation:

The studies are only a small selection, but one thing is obvious: Slow loading sites lose users, while fast sites increase sales. How strong this correlation is, seems astonishing. If for example the top 10 shops in Germany were to load one second faster, this would correspond to 187 million Euros of additional revenue every month (according to EHI sales figures for 2019 and the Amazon statistics mentioned above). Interestingly, the additional revenue does not have to be “bought” through marketing channels, but is generated solely by customers who are more willing to buy. In addition, the better performance has a long-term effect on organic search engine traffic, as Google has started incorporating the end-user’s performance into the SEO rank in the past 2 years.

What retailers and manufacturers have to do for performance now

Performance is of course only one of many factors for a good conversion rate. But unlike the consumer climate, speed is a factor that every shop operator has completely in his own hands. In general, there are three areas that need to be optimized:

1. Frontend: The rendering of a website runs in a series of steps, some of which can be executed in parallel and some of which build on each other. The Critical Rendering Path describes the part of this process that has to run sequentially, i.e. step by step. For optimal loading times, this critical path should be kept as short as possible (few JavaScript and CSS resources) and executed in an optimal order (avoid blocking).
2. Network: It is crucial for network performance that as few different domains as possible are integrated and that the correct protocols are used. This includes both basics (e.g. HTTP/2 instead of HTTP/1.1) and very complex details (e.g. OCSP stapling in the TLS protocol). The browser cache must also be used by appropriate HTTP headers to avoid unnecessary requests.
3. Backend: The most complex area is the backend, since many different system are involved — from the shop system to the CMS to the ERP, PIM and database system. Therefore, there are hardly any generic solutions in this area.

Classic performance optimization according to the above criteria requires a lot of time and resources. However, there is an elegant shortcut that we have developed at Baqend.

Ensure fast page loads with Speed Kit

The Hamburg-based research spin-off Baqend enables market-leading loading times with just a few simple steps. The performance plugin Speed Kit was developed specifically for the acceleration of e-commerce websites. Speed Kit is based on innovative caching algorithms and is designed for websites of all sizes.

The installation of Speed Kit usually takes only a few hours. Speed Kit is integrated into the shop as a JavaScript code snippet and then runs in the user’s browser. Most shops benefit particularly from the fast delivery of HTML responses that often save a second of “dead time” and more. With these time savings in the Time to First Byte (TTFB), many shops are able to load as fast as the global top players. Images are automatically optimized and all data is stored in the browser cache and a distributed caching infrastructure for maximum performance.

As an example of how Speed Kit helps, check out this case study for the world’s largest online sports retailers Decathlon:

The Baqend team has done an amazing job with Speed Kit and not only accelerated our site by 2.5x, but also been a reliable partner during the result analysis. There is no doubt about the great ROI. They have become a strategic part in our online growth plans.

Florian Bischoff, Director E-Commerce, Decathlon Germany

Baqend provides Speed Kit as an immediate measure for online shops affected by COVID-19. Get in touch here to learn more and let us provide you with an offer and a performance assessment.

E-Commerce Loading Times in the COVID-19 Crisis was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last summer, the Google Speed Update made it official: If you don’t invest into mobile speed now, you will pay for it later — because poor mobile performance drags down your search rank. But how do you measure mobile web performance? What can you do to improve it? And when do you know you are making a dent? We will cover the gist of current research on mobile performance to find out what makes up a great mobile experience — and what you have to do to get there. First, we focus on user expectations to answer the question when they are most likely to engage or buy. We then assume the business perspective and explore the correlation between technical performance measures (TTFB, first paint, etc.) and pivotal business KPIs (e.g. conversion rates, bounce rates). Having thus established the business impact of web performance, we turn to the challenge of measuring it correctly. To this end, we highlight the differences between technical and user-centric metrics and then compare traditional synthetic performance tests, tracking, and log analysis with more advanced approaches like CRUX data analysis and real-user monitoring. As the main takeaway, we conclude with an overview of the most effective performance best-practices to increase web performance and drive up user engagement.

Check out speedstudy.info to subscribe for new research on web performance — or if you want to join our study and boost your business!

Video: Mobile Site Speed and the Impact on E-Commerce was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Speed Kit provides an uplift of 3.9x in page load times and more than 50% in conversions for appelrath.com.

Problem

With AppelrathCüpper, a success story that started 130 years ago, continues online. Through Speed Kit, AppelrathCüpper was able to bring its performance and user experience to a level that used to be exclusive to the large-scale market leaders. With the new level of page load times below one second, appelrath.com is the in the top tier of the best-performing online shops worldwide.

• Challenge: delivering the best online user experience & performance for a retail brand that has built over 100 years of trust and recognition.
• Approach: adding Speed Kit as a JavaScript tool to the existing OXID-based online shop for a fast and snappy experience across the site.
• Result I: 3.9x faster page loads
• Result II: 55% more conversions
• Result III: 16% longer session length

“Speed Kit does not only give us a 3.9x page speed boost you can feel, but a 55% conversion uplift we can measure.”

Jerrit Kube, Head of Digital & Marketing, AppelrathCüpper

About AppelrathCüpper — From Stationary to E-Commerce

Founded in the 19th century, AppelrathCüpper is one of Germany’s oldest fashion commerce labels and known for high-quality women clothing. Appelrath.com was launched in 2015 in order to start the transition from purely stationary commerce to a 24/7 online business. Being in direct competition with big players like Zalando and Amazon, AppelrathCüpper’s goal is nothing less than transforming their well-trusted fashion retail brand into a leader in fashion e-commerce, too. To address this challenge on a technical level, Speed Kit comes into play.

Technical Setup

Appelrath.com is running on OXID for its shop backend. Since AppelrathCüpper has the same high standards for their online business as for their products, they work with Dept (formerly Superreal) for building and operating their online shop. Dept is an international web development agency with more than 1000 experts in all fields related to online commerce, including marketing, design, and technical execution. With their expertise in e-commerce and as a Baqend partner, performance is something they take very seriously.

To bring user experience to the best possible state on a technical performance level, Speed Kit was chosen for appelrath.com as the solution for tackling page load time. Speed Kit is a web performance insurance: added as a JavaScript snippet, it typically accelerates websites by more than 50%. With its automatic optimizations, Speed Kit improves the user experience, increases the conversion rate and SEO ranking, offloads the server, and dramatically reduces marketing costs through lower bounce rates.

As a first step, Dept analyzed by how much Speed Kit would accelerate appelrath.com. To this end, they used the Page Speed Analyzer, a simple layer on top of the well-known WebPagetest. The Page Speed Analyzer accurately predicts the uplift achievable through Speed Kit by measuring and comparing (1) baseline performance of the website under test against (2) performance of the same site when loaded through Speed Kit. A few simple tests quickly showed that Speed Kit could provide a tremendous uplift across landing pages, product lists, and product detail pages.

Approach

To obtain quick but sustainable results in performance, Dept decided to integrate Speed Kit into Appelrath.com. The process of accelerating the website is very simple:

1. Speed Kit leverages the novel Service Worker browser technology. Speed Kit’s sw.js file is simply uploaded to the backend (in this case: OXID).
2. Speed Kit’s whitelist & blacklist control which requests are accelerated and which ones are left untouched. On appelrath.com, Speed Kit delivers both first-party content (e.g. the HTML website itself) as well as slow third-parties (e.g. the Google tag manager).
3. Speed Kit keeps the site up-to-date automatically by checking for content changes frequently. In Speed Kit’s dashboard, the corresponding rules can be added and configured.
4. To speed up personalized content (e.g. the shopping cart and recommendations), a data attribute is added to elements in the page that might change based on users.
5. To activate Speed Kit, a JavaScript snippet is finally included into the site by adding it to the HTML template (a tag manager also works).

With those 5 steps done, Speed Kit can be tested in a pre-production environment. The whole process took less than two developer days for AppelrathCüpper, Dept, and the Speed Kit team combined.

“AppelrathCüpper shows how smooth Speed Kit integrations typically are: We used the basic setup for Appelrath.com, without any bells and whistles, and achieved incredible performance improvements out-of-the-box. Since the teams at AppelrathCüpper and Dept have been super-helpful and quick with their feedback, configuring Speed Kit only took us a few hours. All in all, we went from the initial tests to production in less than three weeks.”

Erik Witt, Head of Speed Kit Development

Test Methodology

To quantify the performance uplift for real user of appelrath.com, a split test was conducted. Speed Kit natively supports A/B testing by capturing anonymized performance data from users browsing the site.

• Group A (50% of users) used Speed Kit
• Group B (the other 50%) loaded the regular website without Speed Kit activating itself

The raw performance data and the group information were sent to the scalable analytic systems of Baqend in realtime so that performance and conversions could be tracked. Then we all had to wait: usually, it takes a few days to obtain statistically significant results, since a sufficient amount of traffic and users have to be observed. But the empirical results quickly spoke for themselves…

Results

“I have helped companies to understand and measure the immense ROI of web performance for over 10 years. However, the challenge for them has always been the implementation of web performance optimization. Today, I’m awestruck by how easy it is to achieve these huge ROIs using Speed Kit. Appelrath.com is the perfect example for why I joined Baqend: we disrupt the web performance market by delivering such unbelievable results and not just talking about it.”

Kristian Sköld, Chief Strategy Officer, Baqend

As always for the initial analysis, we looked at the raw technical performance metrics for page load times (median values):

• Server Response Time (time-to-first-byte from the backend):
  1200% Faster (3.25 s vs. 0.25 s)
• First paint (this is when the browser renders for the first time):
  387% Faster (3.65 s vs. 0.75 s)
• Time-to-interactive (this is when links can be clicked)
  305% Faster (3.85 s vs. 0.95 s)

The perceived loading time of a website is captured well by the first paint which showed a 3.9x improvement for the group of users with Speed Kit.

It is common knowledge that better performance leads to improved sales. In a famous A/B test, Amazon found that 100 ms of load time improvement translated to a 1% conversion rate improvement on their site. So the interesting question is: what is the actual conversion uplift on appelrath.com for real users?

This is exactly what Speed Kit tracks: the conversions and engagement of users surfing the site with Speed Kit.

• Conversion Rate (share of user sessions during which something was bought):
  55% Higher
• Engagement/Session Length (number of pages visited per user session):
  16% Higher

To validate the results, we turned to one of the largest independent parties: Google.

Ever since the “mobile speed update” announced for its ranking algorithm last summer, Google has declared war on slow websites — by ranking them down in search results. To make this downranking transparent, Google collects performance data from all Chrome browsers that captures the user-experienced page Speed. This data is made publicly available in the Chrome User Experience Report (CRUX). While Google only tracks this data at the level of complete months and domains, it is a great tool to observe long-term performance trends.

So while it cannot replace a proper A/B test, we analyzed the CRUX data in order to answer the question: how does a month with Speed Kit compare to a month when Speed Kit was not active, yet?

The histogram below illustrates the share of users experiencing a given page load time in a month with Speed Kit (blue) against a month without Speed Kit (orange): Since lower is better for page loads, it is best to have a huge spike on the left and a flat line to the right. As can be seen in the plot, most users with Speed Kit (blue) saw page load times below 1 s compared to 2–4 s for users without Speed Kit (orange).

The CRUX data is particularly interesting, as Google has explicitly admitted to use this data for its ranking algorithm. This means that the obtained performance improvement also has a direct SEO impact.

Outlook

“Performance is one of the most important topics for any e-commerce website right now. AppelrathCüpper showed how performance can be turned into a true differentiator in a market with an increasingly demanding user base. We are looking forward to a long and fruitful partnership — load times are already below one second on appelrath.com now, but we are only done when they have become imperceptible.”

— Felix Gessert, CEO, Baqend

The Team Behind Speed Kit

Baqend develops Speed Kit as a performance add-on that uses innovative caching algorithms to minimize loading times of e-commerce websites. This helps shops and brands to turn page speed into their competitive advantage. Baqend’s technology is already leveraged by more than 5000 websites such as Baur.de in the Otto Group. With a team of 15 people, Baqend’s goal is to deliver performance that feels instant for each of their customers’ websites.

There is also a German version of this article.

AppelrathCüpper was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Results

Tech companies in the travel industry seem to take the issue seriously. The site of market share leader TripAdvisor loads within 0.7 seconds. Only Airbnb is faster, loading in under 0.5 seconds, thus coming close to the famous blink of an eye. This is particularly noteworthy, as the private rental site is over seven times faster than its rivals in the traditional hotel industry. Marriott International and Hilton Hotels Online are coming in last in the ranking and are the only two sites scoring over 3 seconds on the speed index with an average load time of 3.5 and 4.4 seconds, respectively.

Conclusions

Most of the travel sites we tested are quite fast and already performing at a high level. Especially the page load time of the frontrunners Airbnb and TripAdvisor are barely noticeable to users. This puts slower sites even more under pressure. Marriott and Hilton Hotels need to step up to avoid losing ground against the new generation of travel booking sites. With load times of over three seconds, they are losing a lot of customers before the site has even finished loading — potentially to faster competitors such as Airbnb.

Methodology

Baqend tested the 10 travel destination and booking websites with the most market share according to Statista in November 2016. The test was conducted using the Baqend Page Speed Analyzer using the well-known WebPageTest suite. All sites were accessed from Washington DC. Baqend tested the performance metrics speed index and time to first meaningful paint. The provided data is averaged over 5 tests for each site.

About

Baqend develops a cloud service that uses innovative caching algorithms to minimize loading times of websites. Publishers, shops, startups, and SMEs all face the challenge of competing with giants such as Amazon, Facebook, and Google. Baqend provides them with a technical platform to not only catch up with the market leaders but to turn page speed into their competitive advantage. Baqend’s Speed Kit is integrated into a website as a simple script to accelerate it by 50–300%. With Baqend Platform, new websites and mobile apps can be developed directly against a distributed cloud backend to achieve optimal performance right from the start. The mission of the research spin-off is to enable a web without loading times. Baqend is based in Hamburg, Germany.

New-Generation Travel Websites Load 7 Times Faster Than Traditional Hotel Sites was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR

Static sites provide an unmatched level of scalability and performance by completely avoiding an active backend. Modern static site generators such as Jekyll, Hugo and Hexo make it very easy to build static sites from template languages and largely technology-agnostic. For content-driven use cases like blogs and publishing sites, content management systems for editing static sites have evolved. And for the use cases that require integration of 3rd party services, data storage, and user management, Backend- and Function-as-a-Service APIs provide an option for implementing server-side logic with little code. This survey gives an overview of the static site technology stack that incorporates static site generation, content management and opt-in APIs for dynamic content. We will review the different approaches and walk through a real-world example of using the stack.

Introduction

Frontend, network, and backend performance used to be tough. But with the right technologies (e.g., CDNs, build tools, and HTTP/2) things are getting much easier (see our survey). However, most performance problems stem from single-node systems such as Wordpress or Magento. Both are based on server-side rendering and data storage in a single MySQL instance and are therefore notoriously hard to scale.

It’s not unusual for a Wordpress site to have a time-to-first-byte (TTFB) of 1–3 seconds. At the same time, 49% of users expect websites to load in 2 seconds or less, according to a survey by Akamai. And if that time is already spent for server-side rendering, there is little chance of hitting that performance goal. What makes it worse: Google uses the TTFB as a metric for their search ranking.

The good news is: you can go from your existing Wordpress, Drupal, etc. to a fully static site that is still CMS-backed, as we’ll survey in this article.

Going Static

Static sites have a significant advantage: They do not require server-side rendering or database queries, can be deployed virtually anywhere and are great for caching.

Static site generators are great tools for building static sites that use rich templating in order to keep code clean and avoid duplication. They run on your local machine and compile a static site from a bunch of content files stored in your file system. The most popular ones are:

• Jekyll (Ruby-based, 30132 Github Stars) is the most popular static site generator. It has a huge community, many plugins and great support for blogs.
• Hugo (Go-based, 18057 Github Stars) excels in building speed: complete projects can be generated in milliseconds. It uses a single binary, making it easy to install for blogs and documentation.
• Hexo (Node.js-based, 16875 Github Stars) is getting more and more popular as it fits nicely into the Node ecosystem and is easy to integrate with Octopress and Jekyll plugins.
• Gatsby (Node.js-based, 10041 Github Stars) builds single-page application blogs based on React.js with reusable components.
• Metalsmith (Node.js-based, 5948 Github Stars) is very low-level and highly extensible, so with the right tweaking it can build almost any kind of website.
• GitBook (JavaScript-based, 15605 Github Stars) publishes ebooks from markdown upon pushes to Github.
• MkDocs (Python-based, 4135 Github Stars) is a static site generator specifically made for building technical documentations from markdown.
• Octopress (Ruby-based, 1576 Github Stars) is a Jekyll fork particularly geared towards hackers.

In our example, we will use Jekyll, which is the system that also powers Github Pages. In fact, it was created by Github co-founder Tom Preston-Werner back in 2008 as an engine for building static blogs. It is based on the Ruby toolchain and uses the Liquid templating language.

Building a Jekyll-based Content Site

What we’d like to build as an example is a simple content site where we maintain a repository of bachelor and master thesis topics for computer science students:

Themen

So here is how you build a Jekyll site:

1. Do you have a static website? Then you already have a valid Jekyll project.
2. Install Jekyll. I’m running Windows, so what I did was: a) install the package manager Chocolatey, b) install Ruby: choco install ruby -y c) install Jekyll: gem install jekyll
3. As I found Jekyll’s default theme a little boring, I opted for the nice-looking theme called Minimal Mistakes. Jekyll has a large ecosystem of designs and boilerplate projects.

To use and extend the example setup of this article, check out the Github project:

git clone git@github.com:Baqend/thesis.git

Project Structure

Here is what the Jekyll project structure looks like. The most important files and folders are:

• The _config.yml contains global configurations settings in the YAML format, e.g. the site’s title
• _site contains the rendered static site
• _posts stores all the the blog-type content files and _pages the content for individual pages
• assets holds your images, videos, etc.
• In _data, YAML data is shared across views, e.g. author information
• _layouts, _includes, _sass contain the design and templates

The great thing about Jekyll is that it not only builds your static site, but also comes with a local, live-reloading server.

Running Jekyll

Let’s run the Jekyll live server:

bundle exec jekyll serve

The command will start a server on port 4000 that serves your current version.

Adding Content Management

The reason why many people use Wordpress, Drupal, Joomla etc. is that these frameworks are great for Content Management. Even unskilled users can easily create articles and landing pages using Wordpress. The good news is: There are Content Management Systems (CMS) for static sites, too. Let’s add Jekyll Admin to the project.

To add a Jekyll plugin, the Gemfile is extended, which is Ruby’s equivalents to Node’s package.json: gem 'jekyll-admin', group::jekyll_plugins

Append /admin to the URL and you have your CMS:

Jekyll is based on Markdown. When you create a post in the CMS, it will simply create a <post>.md file in your _posts folder.

Let’s walk through how Jekyll works, indepedent from how content is managed. The upper portion in a content file is called front matter. It contains (YAML) metadata used in the template, for example, tags of the post. The body will be rendered from markdown.

We specified to use the layout single which will use the Liquid templating engine to render to HTML:

This uses the<post>.md files’s front matter and body to output the static HTML page. For example, to add the post’s title, render markdown, and strip HTML and line breaks, we can use a Liquid variable and pipe it through helpers:

{{ page.title | markdownify | strip_html | strip_newlines }}

Adding media can be done by using the Static Files section in Jekyll Admin or by copying files into your asset folder. You can then reference them from you markdown to include them.

Liquid (developed by Shopify) does not allow any code in the template, so if more advanced logic is required, you need to create helper classes.

Jekyll Admin is also not the only CMS for Jekyll. For example, Forestry and Cloudcannon are SaaS solutions for doing static content management on top of Jekyll.

Building the Jekyll Site

Time to output a rendered site:

bundle exec jekyll build

This will put a deployable version into the _site folder. It uses many subfolders with containing an index.html to allow pretty navigation and permalinks. The _site folder can be deployed to almost any platform, for example, an Nginx Webserver or Amazon S3.

Deploying and Hosting the Static Site

Now, we simply have to upload the Jekyll project. Baqend has CDN-backed file hosting with automatic cache invalidation so let’s upload it there. All we need is the Baqend CLI, which can be installed from npm (alternatively you could also drag & drop the files in the dashboard).

npm install -g baqend #install CLI

baqend register #create a new app backend

bundle exec jekyll build #build the Jekyll project

baqend deploy -f _site thesis #deploy the _site folder to Baqend

baqend open thesis #open the deployed site in the browser

Adding APIs to Jekyll: Real-Time User Comments

The limitation of any static site is, of course, user-generated content (e.g., comments) and protected business logic (e.g., a shop checkout). This is where APIs of third-party services come into play, for exmaple, Disqus for comments, or Facebook for social data. If more backend logic is required, serverless architectures are the paradigm of choice: The frontend orchestrates different services (e.g., Paypal, Disqus, AWS Lambda) and contains most of the business logic.

What we want to have for our example, is a customizable comment section, where users can post comments and see any new comments from others users in real-time:

Storing Data from Jekyll

Combining Jekyll with JavaScript APIs is very powerful. You can use Jekyll to generate page information into your scripts and handle input data dynamically:

Therefore, let’s look at how we can use a Backend-as-a-Service to store and query data from Jekyll.

Backend-as-a-Service systems are designed to make serverless development easy, by having server-side code execution combined with standard APIs for things such as:

• Registering and logging in users
• Running scalable, server-side business logic (Function-as-a-Service)
• Giving access to a database-(as-a-service)
• Authorizing users and protecting content
• Sending push notifications and performing synchronization
• Hosting and delivering files

We are building Baqend, which is Backend-as-a-Service geared towards web performance. Baqend is a spin-off from database research and heavily based on caching algorithms.

Let’s add a commenting system to our static site using a Backend-as-a-Service. In order to display and post comments we have to do three things:

1. Set up a data model of how to store comments in our backend.
2. Save a comment, when the comment form is submitted.
3. Sync the comments of a post across browsers in real-time.

Setting up the Data Model

Configuring the data model is extremely simple. Just head of to the dashboard of your Baqend App:

Now, expand the Data Section and hit + to create a new model. Let’s give it just a few String properties to store the poster, the (markdown) message, and the page (slug). A created_at and updated_at property are maintained automatically.

That’s it. We can now save comments. To try it, let’s open the data tab and add a few comments:

Saving Comments from the Static Site

Now let’s head back to our Jekyll project and add the code to post a comment in the custom.html file for custom comment handling.

First, let’s include the Baqend SDK and library for rendering Markdown and connect to our app. This will transparently download the schema and caching information.

Now let’s add the (plain old jQuery) code to take input from the comment form and save it as a new object:

Here we use the Liquid templating syntax with {{ page.slug }} to generate the current page id into the script on each post. The save() call simply inserts the new object over the REST API.

Querying and Syncing Comments

Now that we have the data in the database, we need to query it. There are two ways to do this:

• A standard, pull-based query fetches data from the database and allows us to render it in the client (with jQuery, React, Angular, etc.).
• With a real-time query, we establish a websocket connection to the server, perform a full-fledged MongoDB query and receive any updates to the result set in real-time. We call this a self-maintaing query.

The great thing is: Both look exactly the same in the code and are very easy to handle.

So as soon as we are connected (DB.ready callback), we use the query builder to retrieve comments for the current page, order them by creation date and call our UI-handler each time the set of comments is changed (e.g. a comment is inserted, removed, updated etc.).

And that’s it: Now we have our realtime-capable comment system that we can customize and extend however we like, while still having the abilities and simplcity of Jekyll for creating and content and generating the site.

Let’s deploy the commenting logic:

bundle exec jekyll build; baqend deploy -f _site

So there it is: our static website, with a local CMS, and APIs for on-demand dynamic functionalities. If you look at the developer console you can see comment updates coming in via websockets and static data being loaded via HTTP/2:

You can also configure your custom domain in the dashboard and let Baqend provision the SSL certificates for HTTP/2. By default you have an <app>.app.baqend.com domain that is already SSL-protected.

Securing Access

There is one last thing: we do not want to allow everyone to delete or update other people’s comments. So let’s secure the app by adding a schema ACL to only allow inserts and queries. You can do this in the ACL tab:

The last question is: How can we scale this project across many developers and many content editors?

Collaborating on a Static Site Project

Since a static website has all its context in the file system, sharing it is very easy. Content editors need to have Jekyll installed to use the Jekyll Admin CMS UI and they also need some way of sharing content files. The two wide-spread mechanisms to share the static project are:

• Share the project folder via Dropbox, Google Drive, etc. This has the advantage of being very easy for non-technical users. However, there is no proper versioning, rollbacks, commits, etc.
• Using Git with Github, Gitlab, etc. For obvious reasons, this is usually the preferred choice for developers and tech-savvy users.

The Baqend/API part is also easy to share, as you can simply add developers to the app. Each of the collaborating developers can edit the data model, view data, deploy the site, etc. Throug build tools (e.g., Gulp, Grunt, Webpack, npm scripts), the building, staging and deploying of the site can be made very accessible.

Migrating to Jekyll from Another Project or CMS

If you already have a static site project, you can simply start using the jekyll commad line for static site generation. You can simply embed Liquid templates in your HTML e.g.,{{page.title}}.

If you are coming from another CMS, Jekyll has importers that generate markdown files in your _posts folders. This works for Behance, Blogger, CSV, Drupal 6, Drupal 7, EasyBlog, Enki, Ghost, Google Reader, Joomla, Joomla 3, Jrnl, Marley, Mephisto, Movable Type, Posterous, RSS, S9Y, S9Y Database, Textpattern, Third-party, Tumblr, Typo, WordPress, WordPress.com.

For example, say you already have a Wordpress blog. Simply run this locally, to pull content data from Wordpress’s MySQL:

ruby -rubygems -e 'require "jekyll-import"; JekyllImport::Importers::WordPress.run({})'

That will import posts and pages. You can then use them in any of the numerous Jekyll themes or build a new frontend.

Resources on Static Site Generators

To learn more about static site generators, have a look at these:

• A detailed blog post: Static Site Generators Reviewed: Jekyll, Middleman, Roots, Hugo — Smashing Magazine
• A free O’Reilly book on Static Site Generators and a longer version as a regular O’Reilly book released in March 2017
• Another long-form post on running static site generators at scale
• This nice talk “Static Sites go all Hollywood” by Phil Hawksworth on Youtube

Summary

The message of this post is: Static websites are much more flexible than their name indicates and inherently scalable. They can be combined with content management systems for creating posts and pages and with cloud APIs for handling non-static concerns like comments, payment, and user login.

The stack we proposed in the example project uses three central technologies:

• Jekyll as a versatile static site generator with rich templating.
• Jekyll Admin as a dead-simple local CMS.
• Baqend for storing and querying data and hosting the site.

To use and see the stack in action, simply clone the example project from this Github repo.

Don’t want to miss our next post on building static websites? Get it conveniently delivered to your inbox by joining our newsletter.

Building Static Sites in 2017: Cloud-Hosted, CMS-Backed, and API-Driven was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR

At Baqend, we are building a Backend-as-a-Service (BaaS) service that is geared towards scalability and web performance. Coming from research, we try to be open about the architecture of our platform and therefore would love to share how everything works at a technical level. This post is a combined writeup of different talks we gave.

Challenges of Modern Applications

Engaging web and mobile applications have to fulfill several requirements:

Each one individually may appear easy, but achieving them in combination turns out to be quite difficult in practice.

“Over 30 percent of web development teams deliver projects late or over-budget.” (Survey by New Bamboo)

The Classic 3-Tier Architecture

Today, the most common way of implementing web applications is a 3-tier architecture with a client, a server and a database tier. This design poses some difficulties regarding the requirements:

• A typical website consists of more than 100 resources fetched from the server, which makes latency critical for performance (see HTTP archive).
• With server-rendering, the client has to wait until the server has assembled the data through database queries and rendered it in a language such as Java, PHP or Python.
• The database must be highly available, which is extremely difficult if you also like consistency, according to the CAP theorem.
• The server needs to be scalable, which is difficult when shared state such as user sessions are involved.
• In many cases, business logic is duplicated between the client and server, as well as communication overhead between frontend and backend teams, making projects more difficult and delaying the time-to-market.

2-Tier Architectures to the Rescue

An alternative to the 3-tier architecture is to leave out the server tier and talk directly to a cloud service, e.g. a Database-as-a-Service (DBaaS). This has two major advantages:

• You can render the site progressively in the client to make the UX interactive and enganging.
• Much of the business logic can happen in client-side JavaScript, i.e. you get less hassle between backend and frontend team.

While there are good frameworks and technologies for building these application frontends (e.g. Angular and React) and storing the data (e.g. DynamoDB, or MongoDB hosters), some problems remain:

• You still have to do many latency-sensitive requests, somestimes even more than before, since data comes in small, separate JSON requests.
• All the heavy lifting involved with scalability and high availability is now shifted to the Database-as-a-Service.
• Cloud databases are usually not equipped to handle User Management, Access Control, Server-side Business Logic.

The Serverless Paradigm

The two common forms of serverless architectures are Function-as-a-Service (e.g. AWS Lambda, Google Cloud Functions) and Backend-as-a-Service (e.g. Baqend, Firebase).

The great thing about Function-as-a-Service (FaaS) is that the business logic hosted there, e.g. in the form of a Node.js script, is rapidly scalable and easy to operate. However, FaaS only covers stateless code execution which means that you have to orchestrate different cloud services to handle data storage, user management, push notifications, etc. Thus, the client has to talk to many different APIs. Or the FaaS has to wrap and expose various services, which quickly becomes infeasible to manage.

This is where Backend-as-a-Service (BaaS) enters the stage: the idea is to combine the easiness of Function-as-a-Service with all the APIs and capabilities a typical website or mobile app requires.

How a BaaS Works

Any BaaS will typically be composed of three layers.

The API is what makes the BaaS useful, as it allows to store and query data, run server-side code, authenticate users, and more. This is usually implemented as a multi-tenant cloud service on an Infrastructure-as-a-Service (IaaS) provider such as AWS.

A BaaS also takes care of hosting and delivery. In case of web applications, this means that it will store your HTML files and other assets and deliver them to users.

As a developer, build your website or mobile app using either a REST API or an SDK.

Let’s look at how a BaaS-backed website works.

1. First, the browser will load the HTML and the assets, including your JavaScript application logic written in the framework-of-the-month (Angular, React, Vue, Aurelia, …).
2. Next, your JavaScript will make calls using the BaaS SDK to load the data and render the site, e.g. by fetching the latest posts.
3. The browser will resolve any further references like images that were inserted into the DOM.

How is BaaS different from PaaS and IaaS?

BaaS basically pulls up the abstraction level, so as a developer you do not need to manage a full-blown application and database stack. PaaS gives you full control over the server, e.g. your Rails or Django application. In an IaaS, you can basically run any kind of application but it leaves you with the full complexity of manging and scaling it.

The high abstraction layer of BaaS often comes at a cost: performance, expressiveness and scalability are usually quite limited. But this does not have to be the case. Let’s look at how to actually build a BaaS.

Performance

I bet everyone of us has stared at blank loading screens and cursed whoever is responsible for that.

49% of users expect websites to load in 2 seconds or less, according to a survey by Akamai. These expectations are not matched in practice: the median top 500 e-commerce website has a page load time of 9.3 seconds (see source).

My three favourite findings relating web performance to user behaviour are these (there are many more):

• Amazon found that 100 ms of additional loading time decrease sales revenue by 1%. With Amazon’s current revenue, the impact is over 1 billion USD per year.
• Yahoo! saw 9% of visitors dropping out, when the site was 400ms slower.
• When comparing user reactions to showing 30 search results instead of 10 , Google measured a 20% drop in traffic. The decrease in engagement was caused by 500ms of additional latency for the search query.

What makes websites slow?

Let’s say a user from the US visits our BaaS-backed application hosted in Europe. There are two performance bottlenecks:

1. The server has to do database queries, rendering, etc. which causes processing overhead.
2. We need to transfer more than 100 resources over a high-latency network connection, delaying the moment where the browser has enough data to show something meaningful.

Interestingly, bandwidth is usally not the bottleneck. If you observe page load time under increasing bandwidth, you’ll notice that it doesn’t get any faster above 5 Mbps. However, if you are able to decrease access latency, you will see a proportional decrease in page load time.

Or put differently:

2× Bandwidth = Same Load Time
½ Latency ≈ ½ Load Time

State of the Art: How to improve web performance?

Let’s look at one example of how web performance can be improved: Google‘s Accelerated Mobile Pages (AMP).

In a nutshell, AMP prescribes the following:

• Stripped down HTML + AMP tags (e.g. amp-img) rendered asynchronously by AMP JS runtime
• CSS must be inlined and below 50 KB
• No custom JS (except in iframes)
• Only static sizes for DOM elements → no repaints
• Limited to mobile pages and forces a Google bar
• Cached in Google CDN, as long as it is crawled the next time
  → only suited for static media, e.g. news

So the basic idea of AMP is to enforce performance best pratices and use CDN caching. However, there is no reason to limit these techniques to proprietary Google technology with uncertain motiviations.

Tackling latency

So we know that we have to reduce latency in order to be fast. What is the oldest technique to achieve that? Caching!

This is the main idea behind the Baqend BaaS: reducing latency through web caching. And since interesting applications do not only use static data, we have to make dynamic data cacheable.

If you can cache dynamic data, you get two things:

• Low Latency, since every request can be answered by a nearby web cache (e.g. your browser cache or a CDN edge node)
• Less Processing overhead in the backend, as only the cache misses will hit the server

How to cache dynamic content?

The idea is quite simple. We have to keep track of every response and invalidate it upon changes in both expiration-based caches (e.g. the browser cache) and invalidation-based caches (e.g. CDNs like Akamai and Fastly).

As expiration-based caches simply use the time-to-live provided in the server’s Cache-Control HTTP header, we have to use a trick: we let the client know, when something needs to be revalidated. As the full list of stale URLs can get quite, long we compress it using Bloom filters.

The concrete algorithms to handle this are not some secret sauce, it is all published research. Here is how it works.

Caching everything, not just assets

The tricky thing when using web caches is that you must specify a time-to-live (TTL) when you first deliver the data from the server. After that you do not have any chance to kick the data out. It will be served by the browser cache up to the moment the TTL expires. For static assets, it is not such a complex thing, since they usually only change when you deploy a new version of your web application. Therefore, you can use tools like gulp-rev-all and grunt-filerev to hash the assets. By renaming the assets at deployment time, you ensure that all users will see the latest version of your page while using caches at their best.

But wait! What do you do with all the data which is loaded and changed by your application at runtime? Changing user profiles, updating a post or adding a new comment are seemingly impossible to combine with the browsers cache, since you cannot estimate when such updates will happen in the future. Therefore, caching is usually just disabled or very low TTLs are used.

The Bloom filter trick

We need to check the staleness of any data before we actually fetch them. At the begin of each user session, the connectcall therefore fetches a Bloom filter, which is a highly compressed representation of all recently changed resources. Before making a request, the SDK first checks the Bloom filter to know if it contains an entry for the resource we fetch.

An entry in the Bloom filter indicates that the content was changed in the near past and that the content may be stale. In such cases the SDK bypasses the browser cache and fetches the content from the nearest CDN edge server. In all other cases the content is served directly from the browsers cache. Using the browser cache saves network traffic, bandwidth and is rocket-fast.

In addition, we ensure that the CDN always contains the most recent data, by instantly purging data when it becomes stale.

How useful and fast is this in practice?

Here is a small example of the effects using a simple news application as an open-source benchmark. You can also run it in your browser.

This means: caching dynamic content leads to a performance advantage of roughly 15x. If you want to learn how this work’s for real applications, check out our findings from a high-traffic e-commerce scenario:

Building a Shop with Sub-Second Page Loads: Lessons Learned

Optimizing the Network

We’ve seen that caching optimizes latency. But there are more networking topics to take care of.

Baqend uses a CDN (Fastly) to improve several networking aspects:

Arguably the most important optimization is HTTP/2. For a typical use case and in particular combined with caching, it can improve performance drastically:

HTTP/2 does three important things better than HTTP/1.1:

1. Multiplexing requests over one single TCP connection (no head-of-line blocking)
2. Server Push allows you to send data before the user requests it
3. Header Compression saves bandwidth, e.g. for repeatedly sent cookies

BaaS Performance: Summary

So in a nutshell, Baqend does two things to tackle the web performance problem:

• Latency is reduced by caching dynamic content (e.g. queries).
• Using HTTP/2 and CDN delivery, data is transferred faster.

Scaling a Backend-as-a-Service

How can we scale the backend? There are a few performance best practices that any scalable backend should address:

This is how Baqend Cloud implements these:

• Data is stored in a set of different scalable NoSQL databases (MongoDB, Redis, ElasticSearch), depending on what data store suits the application best.
• Java-based Baqend servers expose the BaaS REST API and can be scaled independently. Each tenant has its own server and a Node.js instance for running server-side business logic. This pair of servers is hosted on Docker and is scaled using Docker Swarm by adding more containers per tenant on demand.
• All types of web cache (reverse-proxy caches, CDNs, ISP caches, forward proxies and browser caches) are allowed to cache the responses.

Scalability & Performance Wrap-up

All in all, there are some things that Baqend can optimize and there things that you as a developer need to take care of:

How do I use a BaaS from a development perspective?

To add Baqend to an application, you just include the SDK and connect it to your app. Then you can start loading and storing data:

The three major interfaces you will be using with any BaaS is usally the Service Dashboard, the CLI and the SDK.

The Baqend SDK is compatible with all major frameworks and also works for hybrid apps and in Node.js. Besides wrapping the REST API, the SDK does a few useful things:

• Abstracts from caching logic, and has an intelligent object-identity
• Completely ES6- and TypeScript-compatible (support for Maps, Sets, Arrays, models can be ES6-classes)
• Promise-based
• Powerful query builder for doing search & retrieval of data
• Automatic change- and dependency-Tracking, so objects are only sent to the server when they actually change

Data Modeling

Baqend combines schemaful and schemaless data modelling: you can use and check data types wherever you need strict validation and fallback to plain JSON, where the structure is not known in advance.

• Types: Boolean, Integer, String, DateTime, Time, Date, GeoPoint, List, Map, Set, JSON-Object, JSON-Array
• Files, references, embedded types & inheritance

Documents can be nested or have references to other objects. Each field can be indexed and efficiently queried, even nested documents and schemaless JSON.

For example, when building a todo list, you can choose between embedding and referencing depending on whether query or update performance is more important and what the access patterns look like.

The CRUD API makes it very easy to interact with objects and files.

Every request issued from the SDK is mapped to the REST API and automatically uses the client-side Bloom filter-logic and the server-side cache invalidations:

The REST API is specified using Swagger, which also makes it browsable and allows to generate simple SDKs for other languages.

The server exposing the REST API is built in Java and Jetty. It is highly tuned towards throughput and latency efficiency: a single server can handle roughly 15K requests per second on commodity cloud hardware.

User Management

There are simple APIs to handle users of your application. Users can be registered using classic email validation or through OAuth providers like Facebook or Google.

User can have roles. Acess rights can be set on user and role level. The User-schema is extensible and sessions for returning users are handled automatically.

How are ACL-protected ressources handled internally?

When an end user performs a requests, a token is attached to it. This token contains the roles of the user and a signature. First, the schema-level ACLs are checked (e.g., checking that anonymous read access is allowed). If they pass, the next step is to validate the object-level ACLs. This is efficiently handled by constructing a query predicate on indexed ACL-attributes that is pushed down to the database for every operation.

One of the coolest things about this ACL-scheme, is that the ACLs are already checked in the CDN if the data is cached, allowing high cache hit rates, even on private data.

Backend Code

Most applications have some business logic that cannot be done in the client, e.g. the checkout in an e-commerce shop. Baqend allows you to execute server-side code using the same SDK as in the client.

The code can be called as a microservice similar to a Function-as-a-Service providers or be triggered based on operations such as update, insert, delete, etc.:

Going Real-Time

While triggers are useful, they are also pretty limited in their expressiveness and not the right tool for complex reactive applications. This is why we built InvaliDB, a streaming system for real-time queries on MongoDB. For registered real-time queries, result updates are pushed to clients and caches alike.

As illustrated below, incoming database updates are matched in an Apache Storm cluster against queries.

The unique feature of this architecture is that it scales with both queries and updates — unlike RethinkDB, Meteor and Parse that all have centralized bottlenecks that break with increasing throughput.

The coolest part is, how easy it is to use this feature: it is used exactly like regular queries. You simply subscribe to any MongoDB query (including complex predicates, sorting, limit, etc.) and the SDK maintains the result in realtime (< 30ms end-to-end latency):

Accelerating Legacy Sites: Going Beyond BaaS

A limitation of any Backend-as-a-Service is that you have to built your app or website around its APIs in order to use it. Moving an existing application to a BaaS is a lot of work.

Fortunately, there’s a funky new browser technology to cicrumvent this problem: Service Workers. What they do is:

• They proxy any HTTP request and allow you to change the browser’s default behaviour
• They provide an Offline Cache for „progessive web apps“ — if the client is disconnected, you can choose to serve a cached copy for an offline mode

How can we use this to make legacy sites easily use a BaaS?

The idea is simple:

1. You include our Service Worker on your site, quite similar to aGoogle Analytics snippet
2. The Service Worker gets a list of patterns of what to intercept and redirect to our CDN
3. Baqend pulls the data from the original site and stores its metadata and content
4. The legacy system (e.g. Wordpress) simply triggers a refresh when something changes → Baqend iterates over all cached files and revalidates them against the origin. If something changed, it is updated in the CDN and the Bloom filter.
5. Caching in the Service Worker uses the Bloom filter logic to determine when to bypass the cache

We will release this feature soon. On sites we have tested so far, performance can often be tripled without changing the current backend architecture.

Summary: Lessons Learned

Our main learnings from building Baqend are: database systems and distributed systems are hard. Some of these complexities should be exposed to developers (e.g. indexing) and some should be hidden (e.g. scalable stream processing).

All in all, of the three categories frontend, network and backend, a BaaS can optimize many things of the network and backend that are really hard work on traditional platforms.

Curious? Try it.

Learn how to develop with a Backend-as-a-Serivce, it is a fun experience:

1. Create a Baqend app (or do: npm install -g baqend followed by baqend register)
2. Upload your assets (HTML, images, etc.) to Baqend. Learn how to deploy a simple web app in the quickstart.
3. Use the Baqend SDK to easily store & query data, register users, etc. Learn how to include the SDK.
4. Start building your website or app. Check out our starter kits and the guide.

Don’t want to miss our next post on Backend-as-a-Service topics? Get it conveniently delivered to your inbox by joining our newsletter.

Lessons Learned Building a Backend-as-a-Service: A Technical Deep Dive was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

So here are some facts and trivia that are not so well-known or published that I collected by talking to Parse engineers that now work at Facebook. As I am unsure about whether they were allowed to share this information, I will not mention them by name.

Users and Traction

• 1 million apps were deployed to Parse.
• The largest Parse app had 40M users.

• Parse was one of the world’s largest MongoDB user
• Clash of Kings used Parse for push notifications and made up roughly half of all pushes that went through Parse. They never moved any other parts to Parse, due to scalability concerns.
• Original reason for Facebook to acquire Parse was to push their mobile SDKs and to create synergies with mobile ads. Parse was often sold as a package deal with Facebook advertising.
• Static pricing model measured in guaranteed requests per second did not work well.
• Business problem: people tended to remain in the free-tier.
• Technical problem I: complicated rate limiting. If limit exceeded by a factor of 60 for a minute, requests were dropped. Limits were tracked using a shared Memcache instance. Consequence: when developers experienced rate limits in the API, they added retries. The retries incurred enormous load in the Parse backend.
• Technical problem II: the real problem and bottleneck was not the API servers but almost always the shared MongoDB database cluster.

Parse Server

• Server was Rails at first (with 24 threads max. concurrency) with very little throughput per server (~15–30 requests per second)
• The server was later rewritten in Go. The open-source Parse server is written Node.js and lacks many functionalities of the original Parse server in Go.
• Backend was completely on Amazon Web Services
• It was planned to migrate Parse to Facebook’s infrastructure (e.g. Haystack, Tao, F4, Extended Apache Giraph, Gorilla) but the project was abandoned
• Roughly 8 developers working on SDKs, 8 on the server, 8 DevOps + a few more engineers

Database

• >40 MongoDB Replica Sets with 3 nodes each

• Storage Engine: RocksDB (i.e. MongoRocks), an append-only engine based on log-structured merge trees (similar to e.g. Cassandra, HBase, CouchDB, LevelDB, WiredTiger, TokuDB). Reason: there is better handling of many collections — in contract to WiredTiger that uses one file for each collection. Compression was better by a factor of 2–3 in terms of space. Writes and replication also were more efficient in terms of latency/lag. The move to MongoRocks from MMap was done by adding a replica with MongoRocks that was later promoted as the new master.
• Used only instance storage with SSDs, no EBS.
• No sharding: each tenant was mapped statically to exactly one replica set using MongoDB’s primary database logic.
• The Mongo Write Concern was 1 (!), i.e. writes were confirmed before they were replicated. Some people complained about lost data and stale reads
• Slave reads were allowed for performance reasons
• Partial updates were problematic as small updates to large docs got “write amplification” when being written to oplog
• Frequent (daily) master reelections on AWS EC2. Rollback files were discarded and let to data loss
• Developed a special “flashback” tool that recorded workloads that could later be rerun for internal load and functional testing
• JS ran in forked V8 engine to enforce 15 second execution limit for user-provided code
• No sharding automation: manual, error-prone process for largest customers
• Indexing not exposed: automatic rule-based generation from slow query logs. Did not work well for larger apps.
• Slow queries killed by cron job that polled Mongos currentOp and maintained a limit per (API-key, query template) combination
• Backups: if important customers lost data due to human error, Facebook engineers would manually recover it from periodic backups
• The object-level ACL system was highly inefficient. Numerours indexes were required that could sometimes surpass the actual data size by a factor of 3–4.
• As there was no mechanism for concurrency control (except for minimal support for things like counters), applications were often inconsistent

What Parse should have done differently

Parse did a lot of things right. The documentation was great, the mobile SDKs were solid and the web UIs well-designed. However, they had an unspoken value system of not trusting their users to deal with complex database and architectural problems.

Coming from a database background, our idea is that developers should know about details such as schemas and indexes (the Parse engineers strongly agreed in hindsight). Also, we think that backend services are not limited to mobile apps but very useful for the web.

I think that providers should be open about their infrastructure and trade-offs, which Parse only was after it had already failed.

If this idea sounds interesting to you, have a look at Baqend. It is a high-performance BaaS that focuses on web performance through transparent caching and scalability through auto-sharding and polyglot persistence.

Don’t want to miss our next post on Backend-as-a-Service topics? Get it conveniently delivered to your inbox by joining our newsletter.

The AWS and MongoDB Infrastructure of Parse: Lessons Learned was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Together with our colleagues at the University of Hamburg, we — that is Felix Gessert, Wolfram Wingerath, Steffen Friedrich and Norbert Ritter — presented an overview over the NoSQL landscape at SummerSOC’16 last month. Here is the written gist. We give our best to convey the condensed NoSQL knowledge we gathered building Baqend.

TL;DR

Today, data is generated and consumed at unprecedented scale. This has lead to novel approaches for scalable data management subsumed under the term “NoSQL” database systems to handle the ever-increasing data volume and request loads. However, the heterogeneity and diversity of the numerous existing systems impede the well-informed selection of a data store appropriate for a given application context. Therefore, this article gives a top-down overview of the field: Instead of contrasting the implementation specifics of individual representatives, we propose a comparative classification model that relates functional and non-functional requirements to techniques and algorithms employed in NoSQL databases. This NoSQL Toolbox allows us to derive a simple decision tree to help practitioners and researchers filter potential system candidates based on central application requirements.

1. Introduction

Traditional relational database management systems (RDBMSs) provide powerful mechanisms to store and query structured data under strong consistency and transaction guarantees and have reached an unmatched level of reliability, stability and support through decades of development. In recent years, however, the amount of useful data in some application areas has become so vast that it cannot be stored or processed by traditional database solutions. User-generated content in social networks or data retrieved from large sensor networks are only two examples of this phenomenon commonly referred to as Big Data. A class of novel data storage systems able to cope with Big Data are subsumed under the term NoSQL databases, many of which offer horizontal scalability and higher availability than relational databases by sacrificing querying capabilities and consistency guarantees. These trade-offs are pivotal for service-oriented computing and as-a-service models, since any stateful service can only be as scalable and fault-tolerant as its underlying data store.

There are dozens of NoSQL database systems and it is hard to keep track of where they excel, where they fail or even where they differ, as implementation details change quickly and feature sets evolve over time. In this article, we therefore aim to provide an overview of the NoSQL landscape by discussing employed concepts rather than system specificities and explore the requirements typically posed to NoSQL database systems, the techniques used to fulfil these requirements and the trade-offs that have to be made in the process. Our focus lies on key-value, document and wide-column stores, since these NoSQL categories cover the most relevant techniques and design decisions in the space of scalable data management.

In Section 2, we describe the most common high-level approaches towards categorizing NoSQL database systems either by their data model into key-value stores, document stores and wide-column stores or by the safety-liveness trade-offs in their design (CAP and PACELC). We then survey commonly used techniques in more detail and discuss our model of how requirements and techniques are related in Section 3 , before we give a broad overview of prominent database systems by applying our model to them in Section 4 . A simple and abstract decision model for restricting the choice of appropriate NoSQL systems based on application requirements concludes the paper in Section 5.

2. High-Level System Classification

In order to abstract from implementation details of individual NoSQL systems, high-level classification criteria can be used to group similar data stores into categories. In this section, we introduce the two most prominent approaches: data models and CAP theorem classes.

2.1 Different Data Models

The most commonly employed distinction between NoSQL databases is the way they store and allow access to data. Each system covered in this paper can be categorised as either key-value store, document store or wide-column store.

2.1.1 Key-Value Stores. A key-value store consists of a set of key-value pairs with unique keys. Due to this simple structure, it only supports get and put operations. As the nature of the stored value is transparent to the database, pure key-value stores do not support operations beyond simple CRUD (Create, Read, Update, Delete). Key-value stores are therefore often referred to as schemaless: Any assumptions about the structure of stored data are implicitly encoded in the application logic (schema-on-read) and not explicitly defined through a data definition language (schema-on-write).

The obvious advantages of this data model lie in its simplicity. The very simple abstraction makes it easy to partition and query the data, so that the database system can achieve low latency as well as high throughput. However, if an application demands more complex operations, e.g. range queries, this data model is not powerful enough. Figure 1 illustrates how user account data and settings might be stored in a key-value store. Since queries more complex than simple lookups are not supported, data has to be analyzed inefficiently in application code to extract information like whether cookies are supported or not (cookies: false).

2.1.2 Document Stores. A document store is a key-value store that restricts values to semi-structured formats such as JSON documents. This restriction in comparison to key-value stores brings great flexibility in accessing the data. It is not only possible to fetch an entire document by its ID, but also to retrieve only parts of a document, e.g. the age of a customer, and to execute queries like aggregation, query-by-example or even full-text search.

3.1.3 Wide-Column Stores Wide-column stores inherit their name from the image that is often used to explain the underlying data model: a relational table with many sparse columns. Technically, however, a wide-column store is closer to a distributed multi-level sorted map: The first-level keys identify rows which themselves consist of key-value pairs. The first-level keys are called row keys, the second-level keys are called column keys. This storage scheme makes tables with arbitrarily many columns feasible, because there is no column key without a corresponding value. Hence, null values can be stored without any space overhead. The set of all columns is partitioned into so-called column families to colocate columns on disk that are usually accessed together. On disk, wide-column stores do not colocate all data from each row, but instead values of the same column family and from the same row. Hence, an entity (a row) cannot be retrieved by one single lookup as in a document store, but has to be joined together from the columns of all column families. However, this storage layout usually enables highly efficient data compression and makes retrieving only a portion of an entity very efficient. The data are stored in lexicographic order of their keys, so that data that are accessed together are physically co-located, given a careful key design. As all rows are distributed into contiguous ranges (so-called tablets) among different tablet servers, row scans only involve few servers and thus are very efficient.

Bigtable, which pioneered the wide-column model, was specifically developed to store a large collection of webpages as illustrated in Figure 3. Every row in the webpages table corresponds to a single webpage. The row key is a concatenation of the URL components in reversed order and every column key is composed of the column family name and a column qualifier, separated by a colon. There are two column families: the “contents” column family with only one column holding the actual webpage and the “anchor” column family holding links to each webpage, each in a separate column. Every cell in the table (i.e. every value accessible by the combination of row and column key) can be versioned by timestamps or version numbers. It is important to note that much of the information of an entity lies in the keys and not only in the values .

2.2 Consistency-Availability Trade-Offs: CAP and PACELC

Another defining property of a database apart from how the data are stored and how they can be accessed is the level of consistency that is provided. Some databases are built to guarantee strong consistency and serializability (ACID), while others favour availability (BASE). This trade-off is inherent to every distributed database system and the huge number of different NoSQL systems shows that there is a wide spectrum between the two paradigms. In the following, we explain the two theorems CAP and PACELC according to which database systems can be categorised by their respective positions in this spectrum.

CAP. Like the famous FLP Theorem, the CAP Theorem, presented by Eric Brewer at PODC 2000 and later proven by Gilbert and Lynch, is one of the truly influential impossibility results in the field of distributed computing, because it places an ultimate upper bound on what can possibly be accomplished by a distributed system. It states that a sequentially consistent read/write register that eventually responds to every request cannot be realised in an asynchronous system that is prone to network partitions. In other words, it can guarantee at most two of the following three properties at the same time:

• Consistency (C): Reads and writes are always executed atomically and are strictly consistent (linearizable). Put differently, all clients have the same view on the data at all times.
• Availability (A): Every non-failing node in the system can always accept read and write requests by clients and will eventually return with a meaningful response, i.e. not with an error message.
• Partition-tolerance (P): The system upholds the previously displayed consistency guarantees and availability in the presence of message loss between the nodes or partial system failure.

Brewer argues that a system can be both available and consistent in normal operation, but in the presence of a system partition, this is not possible: If the system continues to work in spite of the partition, there is some non-failing node that has lost contact to the other nodes and thus has to decide to either continue processing client requests to preserve availability (AP, eventual consistent systems) or to reject client requests in order to uphold consistency guarantees (CP). The first option violates consistency, because it might lead to stale reads and conflicting writes, while the second option obviously sacrifices availability. There are also systems that usually are available and consistent, but fail completely when there is a partition (CA), for example single-node systems. It has been shown that the CAP-theorem holds for any consistency property that is at least as strong as causal consistency, which also includes any recency bounds on the permissible staleness of data (Δ-atomicity). Serializability as the correctness criterion of transactional isolation does not require strong consistency. However, similar to consistency, serializability can also not be achieved under network partitions.

The classification of NoSQL systems as either AP, CP or CA vaguely reflects the individual systems’ capabilities and hence is widely accepted as a means for high-level comparisons. However, it is important to note that the CAP Theorem actually does not state anything on normal operation; it merely tells us whether a system favors availability or consistency in the face of a network partition. In contrast to the FLP-Theorem, the CAP theorem assumes a failure model that allows arbitrary messages to be dropped, reordered or delayed indefinitely. Under the weaker assumption of reliable communication channels (i.e. messages always arrive but asynchronously and possibly reordered) a CAP-system is in fact possible using the Attiya, Bar-Noy, Dolev algorithm, as long as a majority of nodes are up. (Therefore, consensus as used for coordination in many NoSQL systems either natively (e.g. in Megastore) or through coordination services like Chubby and Zookeeper is even harder to achieve with high availability than strong consistency, see FLP Theorem.)

PACELC. This lack of the CAP Theorem is addressed in an article by Daniel Abadi in which he points out that the CAP Theorem fails to capture the trade-off between latency and consistency during normal operation, even though it has proven to be much more influential on the design of distributed systems than the availability-consistency trade-off in failure scenarios. He formulates PACELC which unifies both trade-offs and thus portrays the design space of distributed systems more accurately. From PACELC, we learn that in case of a Partition, there is an Availability-Consistency trade-off; Else, i.e. in normal operation, there is a Latency-Consistency trade-off.

This classification basically offers two possible choices for the partition scenario (A/C) and also two for normal operation (L/C) and thus appears more fine-grained than the CAP classification. However, many systems cannot be assigned exclusively to one single PACELC class and one of the four PACELC classes, namely PC/EL, can hardly be assigned to any system.

3. Techniques

Every significantly successful database is designed for a particular class of applications, or to achieve a specific combination of desirable system properties. The simple reason why there are so many different database systems is that it is not possible for any system to achieve all desirable properties at once. Traditional SQL databases such as PostgreSQL have been built to provide the full functional package: a very flexible data model, sophisticated querying capabilities including joins, global integrity constraints and transactional guarantees. On the other end of the design spectrum, there are key-value stores like Dynamo that scale with data and request volume and offer high read and write throughput as well as low latency, but barely any functionality apart from simple lookups.

In this section, we highlight the design space of distributed database systems, concentrating on sharding, replication, storage management and query processing. We survey the available techniques and discuss how they are related to different functional and non-functional properties (goals) of data management systems. In order to illustrate what techniques are suitable to achieve which system properties, we provide the NoSQL Toolbox (Figure 4) where each technique is connected to the functional and non-functional properties it enables (positive edges only).

3.1 Sharding

Several distributed relational database systems such as Oracle RAC or IBM DB2 pureScale rely on a shared-disk architecture where all database nodes access the same central data repository (e.g. a NAS or SAN). Thus, these systems provide consistent data at all times, but are also inherently difficult to scale. In contrast, the (NoSQL) database systems focused in this paper are built upon a shared-nothing architecture, meaning each system consists of many servers with private memory and private disks that are connected through a network. Thus, high scalability in throughput and data volume is achieved by sharding (partitioning) data across different nodes (shards) in the system. There are three basic distribution techniques: range-sharding, hash-sharding and entity-group sharding. To make efficient scans possible, the data can be partitioned into ordered and contiguous value ranges by range-sharding. However, this approach requires some coordination through a master that manages assignments. To ensure elasticity, the system has to be able to detect and resolve hotspots automatically by further splitting an overburdened shard.

Range sharding is supported by wide-column stores like BigTable, HBase or Hypertable and document stores, e.g. MongoDB, RethinkDB, Espresso and DocumentDB. Another way to partition data over several machines is hash-sharding where every data item is assigned to a shard server according to some hash value built from the primary key. This approach does not require a coordinator and also guarantees the data to be evenly distributed across the shards, as long as the used hash function produces an even distribution. The obvious disadvantage, though, is that it only allows lookups and makes scans unfeasible. Hash sharding is used in key-value stores and is also available in some wide-coloumn stores like Cassandra or Azure Tables.

The shard server that is responsible for a record can be determined as serverid = hash(id)%servers, for example. However, this hashing scheme requires all records to be reassigned every time a new server joins or leaves, because it changes with the number of shard servers (servers). Consequently, it infeasible to use in elastic systems like Dynamo, Riak or Cassandra, which allow additional resources to be added on-demand and again be removed when dispensable. For increased flexibility, elastic systems typically use consistent hashing where records are not directly assigned to servers, but instead to logical partitions which are then distributed across all shard servers. Thus, only a fraction of the data have to be reassigned upon changes in the system topology. For example, an elastic system can be downsized by offloading all logical partitions residing on a particular server to other servers and then shutting down the now idle machine. For details on how consistent hashing is used in NoSQL systems, see the Dynamo paper.

Entity-group sharding is a data partitioning scheme with the goal of enabling single-partition transactions on co-located data. The partitions are called entity-groups and either explicitly declared by the application (e.g. in G-Store andMegaStore) or derived from transactions’ access patterns (e.g. in Relational Cloud and Cloud SQL Server). If a transaction accesses data that spans more than one group, data ownership can be transferred between entity-groups or the transaction manager has to fallback to more expensive multi-node transaction protocols.

3.2 Replication

In terms of CAP, conventional RDBMSs are often CA systems run in single-server mode: The entire system becomes unavailable on machine failure. And so system operators secure data integrity and availability through expensive, but reliable high-end hardware. In contrast, NoSQL systems like Dynamo, BigTable or Cassandra are designed for data and request volumes that cannot possibly be handled by one single machine, and therefore they run on clusters consisting of thousands of servers. (Low-end hardware is used, because it is substantially more cost-efficient than high-end hardware.) Since failures are inevitable and will occur frequently in any large-scale distributed system, the software has to cope with them on a daily basis . In 2009, Google fellow Jeff Dean stated that a typical new cluster at Google encounters thousands of hard drive failures, 1,000 single-machine failures, 20 rack failures and several network partitions due to expected and unexpected circumstances in its first year alone. Many more recent cases of network partitions and outages in large cloud data centers have been reported . Replication allows the system to maintain availability and durability in the face of such errors. But storing the same records on different machines (replica servers) in the cluster introduces the problem of synchronization between them and thus a trade-off between consistency on the one hand and latency and availability on the other.

Gray et al. propose a two-tier classification of different replication strategies according to when updates are propagated to replicas and where updates are accepted. There are two possible choices on tier one (“when”): Eager(synchronous) replication propagates incoming changes synchronously to all replicas before a commit can be returned to the client, whereas lazy (asynchronous) replication applies changes only at the receiving replica and passes them on asynchronously. The great advantage of eager replication is consistency among replicas, but it comes at the cost of higher write latency due to the need to wait for other replicas and impaired availability. Lazy replication is faster, because it allows replicas to diverge; as a consequence, stale data might be served. On the second tier (“where”), again, two different approaches are possible: Either a master-slave (primary copy) scheme is pursued where changes can only be accepted by one replica (the master) or, in a update anywhere (multi-master) approach, every replica can accept writes. In master-slave protocols, concurrency control is not more complex than in a distributed system without replicas, but the entire replica set becomes unavailable, as soon as the master fails. Multi-master protocols require complex mechanisms for prevention or detection and reconciliation of conflicting changes. Techniques typically used for these purposes are versioning, vector clocks, gossiping and read repair (e.g. in Dynamo) and convergent or commutative datatypes (e.g. in Riak).

Basically, all four combinations of the two-tier classification are possible. Distributed relational systems usually perform eager master-slave replication to maintain strong consistency. Eager update anywhere replication as for example featured in Google’s Megastore suffers from a heavy communication overhead generated by synchronisation and can cause distributed deadlocks which are expensive to detect. NoSQL database systems typically rely on lazyreplication, either in combination with the master-slave (CP systems, e.g. HBase and MongoDB) or the update anywhere approach (AP systems, e.g. Dynamo and Cassandra). Many NoSQL systems leave the choice between latency and consistency to the client, i.e. for every request, the client decides whether to wait for a response from any replica to achieve minimal latency or for a certainly consistent response (by a majority of the replicas or the master) to prevent stale data.

An aspect of replication that is not covered by the two-tier scheme is the distance between replicas. The obvious advantage of placing replicas near one another is low latency, but close proximity of replicas might also reduce the positive effects on availability; for example, if two replicas of the the same data item are placed in the same rack, the data item is not available on rack failure in spite of replication. But more than the possibility of mere temporary unavailability, placing replicas nearby also bears the peril of losing all copies at once in a disaster scenario. An alternative technique for latency reduction is used in Orestes, where data is cached close to applications using web caching infrastructure and cache coherence protocols.

Geo-replication can protect the system against complete data loss and improve read latency for distributed access from clients. Eager geo-replication, as implemented in Megastore, Spanner, MDCC and Mencius achieve strong consistency at the cost of higher write latencies (typically 100ms to 600ms). With lazy geo-replication as in Dynamo, PNUTS, Walter, COPS, Cassandra and BigTable recent changes may be lost, but the system performs better and remains available during partitions. Charron-Bost et al. (Chapter 12) and Öszu and Valduriez (Chapter 13) provide a comprehensive discussion of database replication.

3.3 Storage Management

For best performance, database systems need to be optimized for the storage media they employ to serve and persist data. These are typically main memory (RAM), solid-state drives (SSDs) and spinning disk drives (HDDs) that can be used in any combination. Unlike RDBMSs in enterprise setups, distributed NoSQL databases avoid specialized shared-disk architectures in favor of shared-nothing clusters based on commodity servers (employing commodity storage media). Storage devices are typically visualized as a “storage pyramid” (see Figure 5 or Hellerstein et al.). There is also a set of transparent caches (e.g. L1-L3 CPU caches and disk buffers, not shown in the Figure), that are only implicitly leveraged through well-engineered database algorithms that promote data locality. The very different cost and performance characteristics of RAM, SSD and HDD storage and the different strategies to leverage their strengths (storage management) are one reason for the diversity of NoSQL databases. Storage management has a spatial dimension (where to store data) and a temporal dimension (when to store data). Update-in-place and append-only-IO are two complementary spatial techniques of organizing data; in-memory prescribes RAM as the location of data, whereas logging is a temporal technique that decouples main memory and persistent storage and thus provides control over when data is actually persisted.

In their seminal paper “the end of an architectural era”, Stonebraker et al. have found that in typical RDBMSs, only 6.8% of the execution time is spent on “useful work”, while the rest is spent on:

• buffer management (34.6%), i.e. caching to mitigate slower disk access
• latching (14.2%), to protect shared data structures from race conditions caused by multi-threading
• locking (16.3%), to guarantee logical isolation of transactions
• logging (1.9%), to ensure durability in the face of failures
• hand-coded optimizations (16.2%)

This motivates that large performance improvements can be expected if RAM is used as primary storage (in-memory databases). The downside are high storage costs and lack of durability — a small power outage can destroy the database state. This can be solved in two ways: The state can be replicated over n in-memory server nodes protecting against n-1 single-node failures (e.g. HStore, VoltDB) or by logging to durable storage (e.g. Redis or SAP Hana). Through logging, a random write access pattern can be transformed to a sequential one comprised of received operations and their associated properties (e.g. redo information). In most NoSQL systems, the commit rule for logging is respected, which demands every write operation that is confirmed as successful to be logged and the log to be flushed to persistent storage. In order to avoid the rotational latency of HDDs incurred by logging each operation individually, log flushes can be batched together (group commit) which slightly increases the latency of individual writes, but drastically improves throughput.

SSDs and more generally all storage devices based on NAND flash memory differ substantially from HDDs in various aspects: “(1) asymmetric speed of read and write operations, (2) no in-place overwrite — the whole block must be erased before overwriting any page in that block, and (3) limited program/erase cycles” (Min et al., 2012). Thus, a database system’s storage management must not treat SSDs and HDDs as slightly slower, persistent RAM, since random writes to an SSD are roughly an order of magnitude slower than sequential writes. Random reads, on the other hand, can be performed without any performance penalties. There are some database systems (e.g. Oracle Exadata, Aerospike) that are explicitly engineered for these performance characteristics of SSDs. In HDDs, both random reads and writes are 10–100 times slower than sequential access. Logging hence suits the strengths of SSDs and HDDs which both offer a significantly higher throughput for sequential writes.

For in-memory databases, an update-in-place access pattern is ideal: It simplifies the implementation and random writes to RAM are essentially equally fast as sequential ones, with small differences being hidden by pipelining and the CPU-cache hierarchy. However, RDBMSs and many NoSQL systems (e.g. MongoDB) employ an update-in-place update pattern for persistent storage, too. To mitigate the slow random access to persistent storage, main memory is usually used as a cache and complemented by logging to guarantee durability. In RDBMSs, this is achieved through a complex buffer pool which not only employs cache-replace algorithms appropriate for typical SQL-based access patterns, but also ensures ACID semantics. NoSQL databases have simpler buffer pools that profit from simpler queries and the lack of ACID transactions. The alternative to the buffer pool model is to leave caching to the OS through virtual memory (e.g. employed in MongoDB’s MMAP storage engine). This simplifies the database architecture, but has the downside of giving less control over which data items or pages reside in memory and when they get evicted. Also read-ahead (speculative reads) and write-behind (write buffering) transparently performed with OS buffering lack sophistication as they are based on file system logics instead of database queries.

Append-only storage (also referred to as log-structuring) tries to maximize throughput by writing sequentially. Although log-structured file systems have a long research history, append-only I/O has only recently been popularized for databases by BigTable’s use of Log-Structured Merge (LSM) trees consisting of an in-memory cache, a persistent log and immutable, periodically written storage files. LSM trees and variants like Sorted Array Merge Trees (SAMT) and Cache-Oblivious Look-ahead Arrays (COLA) have been applied in many NoSQL systems (Cassandra, CouchDB, LevelDB, Bitcask, RethinkDB, WiredTiger, RocksDB, InfluxDB, TokuDB). Designing a database to achieve maximum write performance by always writing to a log is rather simple, the difficulty lies in providing fast random and sequential reads. This requires an appropriate index structure that is either permanently updated as a copy-on-write (COW) data structure (e.g. CouchDB’s COW B-trees) or only periodically persisted as an immutable data structure (e.g. in BigTable-style systems). An issue of all log-structured storage approaches is costly garbage collection (compaction) to reclaim space of updated or deleted items.

In virtualized environments like Infrastructure-as-a-Service clouds many of the discussed characteristics of the underlying storage layer are hidden.

3.4 Query Processing

The querying capabilities of a NoSQL database mainly follow from its distribution model, consistency guarantees and data model. Primary key lookup, i.e. retrieving data items by a unique ID, is supported by every NoSQL system, since it is compatible to range- as well as hash-partitioning. Filter queries return all items (or projections) that meet a predicate specified over the properties of data items from a single table. In their simplest form, they can be performed as filtered full-table scans. For hash-partitioned databases this implies a scatter-gather pattern where each partition performs the predicated scan and results are merged. For range-partitioned systems, any conditions on the range attribute can be exploited to select partitions.

To circumvent the inefficiencies of O(n) scans, secondary indexes can be employed. These can either be local secondary indexes that are managed in each partition or global secondary indexes that index data over all partitions. As the global index itself has to be distributed over partitions, consistent secondary index maintenance would necessitate slow and potentially unavailable commit protocols. Therefore in practice, most systems only offer eventual consistency for these indexes (e.g. Megastore, Google AppEngine Datastore, DynamoDB) or do not support them at all (e.g. HBase, Azure Tables). When executing global queries over local secondary indexes the query can only be targeted to a subset of partitions if the query predicate and the partitioning rules intersect. Otherwise, results have to be assembled through scatter-gather. For example, a user table with range-partitioning over an age field can service queries that have an equality condition on age from one partition whereas queries over names need to be evaluated at each partition. A special case of global secondary indexing is full-text search, where selected fields or complete data items are fed into either a database-internal inverted index (e.g. MongoDB) or to an external search platform such as ElasticSearch or Solr (Riak Search, DataStax Cassandra).

Query planning is the task of optimizing a query plan to minimize execution costs. For aggregations and joins, query planning is essential as these queries are very inefficient and hard to implement in application code. The wealth of literature and results on relational query processing is largely disregarded in current NoSQL systems for two reasons. First, the key-value and wide-column model are centered around CRUD and scan operations on primary keys which leave little room for query optimization. Second, most work on distributed query processing focuses on OLAP (online analytical processing) workloads that favor throughput over latency whereas single-node query optimization is not easily applicable for partitioned and replicated databases. However, it remains an open research challenge to generalize the large body of applicable query optimization techniques especially in the context of document databases. (Currently only RethinkDB can perform general Θ-joins. MongoDB’s aggregation framework has support for left-outer equi-joins in its aggregation framework and CouchDB allows joins for pre-declared map-reduce views.)

In-database analytics can be performed either natively (e.g. in MongoDB, Riak, CouchDB) or through external analytics platforms such as Hadoop, Spark and Flink (e.g. in Cassandra and HBase). The prevalent native batch analytics abstraction exposed by NoSQL systems is MapReduce. (An alternative to MapReduce are generalized data processing pipelines, where the database tries to optimize the flow of data and locality of computation based on a more declarative query language, e.g. MongoDB’s aggregation framework.) Due to I/O, communication overhead and limited execution plan optimization, these batch- and micro-batch-oriented approaches have high response times.Materialized views are an alternative with lower query response times. They are declared at design time and continuously updated on change operations (e.g. in CouchDB and Cassandra). However, similar to global secondary indexing, view consistency is usually relaxed in favor of fast, highly-available writes, when the system is distributed. As only few database systems come with built-in support for ingesting and querying unbounded streams of data,near-real-time analytics pipelines commonly implement either the Lambda Architecture or the Kappa Architecture: The former complements a batch processing framework like Hadoop MapReduce with a stream processor such as Storm (see for example Summingbird) and the latter exclusively relies on stream processing and forgoes batch processing altogether.

4. System Case Studies

In this section, we provide a qualitative comparison of some of the most prominent key-value, document and wide-column stores. We present the results in strongly condensed comparisons and refer to the documentations of the individual systems for in-detail information. The proposed NoSQL Toolbox (see Figure 4) is a means of abstraction that can be used to classify database systems along three dimensions: functional requirements, non-functional requirements and the techniques used to implement them. We argue that this classification characterizes many database systems well and thus can be used to meaningfully contrast different database systems: Table 1 shows a direct comparison of MongoDB, Redis, HBase, Riak, Cassandra and MySQL in their respective default configurations. A more verbose comparison of central system properties is presented in the large comparison Table 2 at the end of this article.

The methodology used to identify the specific system properties consists of an in-depth analysis of publicly available documentation and literature on the systems. Furthermore, some properties had to be evaluated by researching the open-source code bases, personal communication with the developers as well as a meta-analysis of reports and benchmarks by practitioners.

For detailed descriptions see the slides from our ICDE 2016 Tutorial, which goes over many details of the different NoSQL systems:

The comparison elucidates how SQL and NoSQL databases are designed to fulfill very different needs: RDBMSs provide an unmatched level of functionality whereas NoSQL databases excel on the non-functional side through scalability, availability, low latency and/or high throughput. However, there are also large differences among the NoSQL databases. Riak and Cassandra, for example, can be configured to fulfill many non-functional requirements, but are only eventually consistent and do not feature many functional capabilities apart from data analytics and, in case of Cassandra, conditional updates. MongoDB and HBase, on the other hand, offer stronger consistency and more sophisticated functional capabilities such as scan queries and — only MongoDB: — filter queries, but do not maintain read and write availability during partitions and tend to display higher read latencies. Redis, as the only non-partitioned system in this comparison apart from MySQL, shows a special set of trade-offs centered around the ability to maintain extremely high throughput at low-latency using in-memory data structures and asynchronous master-slave replication.

5. Conclusions

Choosing a database system always means to choose one set of desirable properties over another. To break down the complexity of this choice, we present a binary decision tree in Figure 6 that maps trade-off decisions to example applications and potentially suitable database systems. The leaf nodes cover applications ranging from simple caching (left) to Big Data analytics (right). Naturally, this view on the problem space is not complete, but it vaguely points towards a solution for a particular data management problem.

The first split in the tree is along the access pattern of applications: They either rely on fast lookups only (left half) or require more complex querying capabilities (right half). The fast lookup applications can be distinguished further by the data volume they process: If the main memory of one single machine can hold all the data, a single-node system like Redis or Memcache probably is the best choice, depending on whether functionality (Redis) or simplicity (Memcache) is favored. If the data volume is or might grow beyond RAM capacity or is even unbounded, a multi-node system that scales horizontally might be more appropriate. The most important decision in this case is whether to favor availability (AP) or consistency (CP) as described by the CAP theorem. Systems like Cassandra and Riak can deliver an always-on experience, while systems like HBase, MongoDB and DynamoDB deliver strong consistency.

The right half of the tree covers applications requiring more complex queries than simple lookups. Here, too, we first distinguish the systems by the data volume they have to handle according to whether single-node systems are feasible (HDD-size) or distribution is required (unbounded volume). For common OLTP (online transaction processing) workloads on moderately large data volumes, traditional RDBMSs or graph databases like Neo4J are optimal, because they offer ACID semantics. If, however, availability is of the essence, distributed systems like MongoDB, CouchDB or DocumentDB are preferrable.

If the data volume exceeds the limits of a single machine, the choice of the right system depends on the prevalent query pattern: When complex queries have to be optimised for latency, as for example in social networking applications, MongoDB is very attractive, because it facilitates expressive ad-hoc queries. HBase and Cassandra are also useful in such a scenario, but excel at throughput-optimised Big Data analytics, when combined with Hadoop.

In summary, we are convinced that the proposed top-down model is an effective decision support to filter the vast amount of NoSQL database systems based on central requirements. The NoSQL Toolbox furthermore provides a mapping from functional and non-functional requirements to common implementation techniques to categorize the constantly evolving NoSQL space.

Don’t want to miss our next post on NoSQL topics? Get it conveniently delivered to your inbox by joining our newsletter.

If you want to cite this article, please use this DBLP reference:

@article{DBLP:journals/ife/GessertWFR17,
  author    = {Felix Gessert and
               Wolfram Wingerath and
               Steffen Friedrich and
               Norbert Ritter},
  title     = {NoSQL database systems: a survey and decision guidance},
  journal   = {Computer Science - R{\&}D},
  volume    = {32},
  number    = {3-4},
  pages     = {353--365},
  year      = {2017},
  doi       = {10.1007/s00450-016-0334-3}
}

NoSQL Databases: a Survey and Decision Guidance was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Angular is one of the most talked-about JavaScript frameworks. In this transcript of a talk held at the HH.JS meetup in Hamburg we give our best to explain the major concepts of Angular using a real application you can play with: SPQR. It’s a collaborative platform for an audience to ask a speaker questions during a talk, and upvote the best of them. Also, it’s synchronized across users in real-time.

The code is hosted on Plunkr, an online code editor similar to CodePen with support for Angular, TypeScript and folder structures.

Angular Basics

Let’s start with what Angular is:

• A framework to build client-side applications
• Code can be written in Typescript, ES6, Dart or JavaScript (without transpiling)
• Has an expressive template language
• Powerful data binding
• Available as release 2.0.0 since 14/09/2016 and release 4.0.0 since 24/3/2017

Okay, so another client-side framework for single page applications, but why?

• Fixes many performance pitfalls of Angular 1
• Modern design using ES6 features, e.g.: Classes, Template strings
• Simplified API and clearer concepts
• Uses Typescript as the primary language for: Interfaces, Type declarations, Decorators and Annotations

Project Setup

For the SPQR app, we use a very simple project structure.

• We use System.js to load ES6 based modules directly in browser
• Webpack is used to build a distribution
• Typescript since it is the recommended language
• Hosted on Baqend (http://baqend.com/)

The building blocks of an Angular app

Components are the central abstraction of Angular. They are comprised of the template as well as controller methods and properties.

Of course, the heart of every MVC framework is its templating, i.e. describing how data is rendered. Angular supports three types of data binding:

The control flow of when and how often data is output is controlled via structural directives.

Angular.js 1 had this notion of magically binding data from form elements into your JavaScript model and back. Angular 2+ handles forms a little differently: you can trigger events from the form, accept data from the model, or both.

Structuring application logic

The main part of the business logic usually resides in services. They encapsulate functionality that is used across many components.

To access simple formatting and transformation routines from the templates, pipes are used, e.g. to format a date.

Routers are used to control the flow through the application with meaningful URLs and a history.

Routing is either triggered programmatically or by using routerLink instead of hrefs.

Components can subscribe to route changes to handle transitions more efficiently than redrawing the whole app.

Extending the behaviour of HTML

Directives are the most scary concepts of Angular.js 1. In Agular 2+ they got way simpler. Directives are a clean and easy way to extend HTML with new elements or attributes.

Model data can be the input of a directive.

And the directive can also output data, e.g. as events.

Two-way binding with directives is just a combination of an input and an output. Note, that the output property must have the same name as the input property with an appended change to work properly.

See these concepts in action!

Now that you’ve seen all building blocks of Angular (Components, Pipes, Routers, Services, Directives), have a look at the SPQR app in Plunkr or the hosted version on Baqend. Get the sources from GitHub.

On the left you have all the project files, i.e. the components, the HTML etc. On the right you can directly play with the code and see the results.

If you’re starting from scratch, consider using our Angular2 starter kit. It’s hooked up to Baqend, a serverless backend designed to make the development of single page apps as easy and fast as possible.

Angular2 and Baqend Starter

Angular 2 & 4 by Example was originally published in Speed Kit Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.