<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Algolia on Medium]]></title>
        <description><![CDATA[Stories by Algolia on Medium]]></description>
        <link>https://medium.com/@algolia?source=rss-a531947eec97------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pJhXtfGc4jgAqsj9pUxcnQ.png</url>
            <title>Stories by Algolia on Medium</title>
            <link>https://medium.com/@algolia?source=rss-a531947eec97------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:56:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@algolia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[No games, please — how to reliably rank ratings]]></title>
            <link>https://medium.com/@algolia/no-games-please-how-to-reliably-rank-ratings-37e9a4a3882d?source=rss-a531947eec97------2</link>
            <guid isPermaLink="false">https://medium.com/p/37e9a4a3882d</guid>
            <category><![CDATA[review]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[rankings]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Algolia]]></dc:creator>
            <pubDate>Tue, 20 Jul 2021 18:58:05 GMT</pubDate>
            <atom:updated>2022-01-20T08:57:57.032Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qPgU0dWOkhg8qjlKBYO3NQ.jpeg" /></figure><h3><strong>No games, please — how to reliably rank ratings</strong></h3><h4><strong>Ratings on </strong>Hacker News, IMDB, Amazon: <strong>Bayes vs vs NLP</strong></h4><p>It’s really hard to rank <em>ratings</em> reliably.</p><p>Same with <em>surveys. </em>The most reliable survey would come from perfect representation and honesty — say a small town whose residents perfectly mirror the demographics of the larger population and <em>every</em> resident takes the survey and answers honestly.</p><h3>Unreliable surveys based on unreliable volunteers</h3><p>The problem gets worse when you realize that many surveys rely on volunteers. Volunteer-based surveys contain unavoidable bias because volunteers rarely, if ever, represent the larger population.</p><p>For example, surveys about sexuality rely mostly on people comfortable speaking about sex and who tend to have a more <a href="https://pubmed.ncbi.nlm.nih.gov/7661653/">positive attitude</a> toward sexuality and carry less sexual guilt. A surprising example of this is psychologist Shere Hite’s survey published in the 1970s. She relied entirely on volunteer input. Her survey concluded that <a href="http://halweb.uc3m.es/esp/Personal/personas/jmmarin/esp/MetQ/Talk1.pdf">70% of women married five years or more have sex outside marriage</a>. This figure is <a href="https://ifstudies.org/blog/who-cheats-more-the-demographics-of-cheating-in-america">more than 50 percentage points higher</a> compared with all other surveys, all non-volunteer based, on the same subject. Whose results are closest to the truth?</p><p><strong>Hard to tell whether bias or dishonesty — intentional or not — creates more unreliability..</strong></p><p>Let’s be clear about the problem: with Hites’s survey, it was not the low number of survey participants. The problem was that<em> only volunteers answered the questions,</em> and volunteers cannot be relied upon to represent the larger population.</p><p>True representation must come from somewhere else.</p><h3>Let’s get back to ratings</h3><p>Online shopping ratings have a <a href="https://hbr.org/2018/03/online-reviews-are-biased-heres-how-to-fix-them">problem</a>. Consider the popular user ratings of products sold on <a href="https://www.feedbackwhiz.com/blog/how-does-amazon-calculate-product-ratings/">Amazon.com</a>. It’s mostly the happiest and angriest customers who rate a product or the buying experience, not the large majority of buyers, who do not express their opinions on the site.</p><p>Another example, movies. <a href="https://www.theringer.com/tv/2019/6/12/18661850/imdb-rating-system-problems-chernobyl">IMDB’s ratings</a> are non-representative of all movie goers because they do not reflect the quality of <em>unpopular </em>films. While this flaw could be consistent with IMDB’s goal, to measure movie popularity, not necessarily quality, it’s nonetheless not giving a complete picture of IMDB’s database. Too bad for indie and documentary film lovers, who are low in number and tend not to vote.</p><p>Maybe ratings are only <a href="https://hbr.org/2018/03/online-reviews-are-biased-heres-how-to-fix-them">part of the story</a>. What else is there?</p><h3>Opinions : let people on the street speak</h3><p>This brings us to an alternativeway to rank ratings: conversational assessments.</p><p>Popular online forums survive thanks to user comments. The best articles usually generate the highest number of comments , with the greatest variety of opinions. If you calculate an average rating of an article, deriving the average from from the <em>content of the comments</em>, you’ll get a more accurate assessment of the truthfulness, and value, of the article.</p><p>The key is to give <em>every distinct opinion</em> a weight in the ranking.</p><p>This works only if you can trust the comments are genuine. <a href="https://www.analyticsvidhya.com/blog/2019/12/detect-fight-neural-fake-news-nlp/">Natural language processing (NLP)</a>, anyone?</p><h3>Diogenes and NLP, or how to detect and reject the troll</h3><p>What about people gaming the system? Lurking behind website ratings is the up-to-now unresolvable problem that they can be<a href="https://www.abc.net.au/news/2018-08-17/read-online-reviews-heres-all-the-ways-theyre-manipulated/10117974"> seriously gamed</a>. Phony reviewers and online trolls are wolves, and some are quite good at dressing up as sheep.</p><p>I’m hopeful that online trolls have not completely destroyed our faith in online news and social media. Don’t forget, trolls have been around since<a href="https://brainfodder.org/diogenese-of-sinope/"> Diogenes</a>, and probably the beginning of time. And it’s thanks to them that we don’t pursue misguided ideas. Trolls destroy, only for us to create something that’s even better able to withstand their aggressiveness.</p><p>If there’s going to be any chance to objectively weigh opinions — to have a free and honest debate in our society — we need to be able to discern truth so that we can rely on what people say.</p><p>I’ll be honest: I’m putting all my faith in machine learning and common sense (and goodwill) to be able to clean up our ratings systems.</p><h3>First let’s get the numbers right</h3><p>Anyway, I want to compare two calculations for ratings: a <strong>Bayesian</strong> average system vs. a system that takes into account the <strong>age of an item </strong>and the <strong>number of comments.</strong> And then I want to add a factor that assesses the quality of the reviewer and their content using <strong>NLP</strong> techniques. NLP is a game changer because it can detect and remove false or mocked-up information.</p><h4>Bayesian ( IMDB)</h4><p>The goal with Bayesian ratings is to distinguish between items that have 10 ratings and those that have 1,000. In the image below, there are two average calculations for movie ratings. The one on the left is a simple arithmetic average that multiplies the number of reviews by the number of individual ratings. The right column uses a more advanced formula called a<a href="https://fulmicoton.com/posts/bayesian_rating/"> Bayesian average</a> (or simply Bayes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*3wVMrjmicEf1ytUP" /></figure><p>As you can see, while both columns leverage the number of reviews, the left side ranking is misleading. The items on the right, ranked using the Bayes calculation, reflect a better balance of ratings and numbers of reviews. Items B and C’s rankings won’t change because Bayes only impacts the ratings of items that have a low number of reviews. In this example, Item A, with only 10 reviews, will change: the Bayes formula lowers its average to 4.3.</p><h4>How to calculate the Bayesian average</h4><p>The Bayesian average uses two constants to offset the arithmetic average of an individual product:</p><ul><li>The arithmetic average rating of all products, m</li><li>A confidence number, C</li></ul><p>The calculation for m is a straightforward<a href="https://en.wikipedia.org/wiki/Arithmetic_mean"> arithmetic mean</a> for all products: the sum of all ratings divided by the number of ratings.</p><p>Calculating C requires a bit more math. The formula calculates C based on the distribution of the rating counts for each product, where C is equal to the 25th percentile (the lower<a href="https://en.wikipedia.org/wiki/Quartile"> quartile</a>). For example, suppose a store sells 100 products. To compute C, you sort the products by their number of ratings. Some may have 10 ratings and others 100 or 1,000 ratings. Once you’ve sorted them, you find the product at the 25% position on the sorted list and see how many ratings it has. This is the lower quartile for C. For simplicity, this guide sets C=100.</p><p>So if you calculate the overall average rating (m) of the store’s catalog to be 3.5, the Bayesian average uses both values (m=3.5 and C=100) to adjust the arithmetic average. It uses this formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*zJWN01gK_UEppdIw" /></figure><p>Here’s the same formula with the example numbers plugged in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*p_O205jWQCj8qv9F" /></figure><p>This confidence number derives from the catalog’s distribution of ratings counts and the average rating of all products. By factoring in ratings counts and averages from the whole catalog, the Bayesian average has the following effect on an item’s individual average rating:</p><ul><li>For an item with fewer than average ratings, the Bayesian average lowers its artificially high rating by weighing it down (slightly) to the lower catalog average</li><li>For an item with many ratings (more than the threshold), the Bayesian average doesn’t change the rating average</li></ul><h4>Adding time as a factor: Hacker News</h4><p>If you were to sort articles on the Hacker News site by just the number of comments they garner, some of them would stay at the top of the list forever. Because “forever” is a time factor, we can change this behavior by adding time to the equation.</p><p>This very good <a href="https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d">article</a> reveals how Hacker News ranks its articles. While it also uses other factors (which it keeps a secret in order to thwart ratings “hackers”), here is its ranking formula:</p><p><strong>def</strong> calculate_score(votes, item_hour_age, gravity=1.8):</p><p><strong>return</strong> (votes — 1) / pow((item_hour_age+2), gravity)</p><p>Time thus impacts ratings: The relative value of ratings decreases as time passes, meaning the older the article, the lower the rating. You can go further and lessen or strengthen the impact of time on a rating by adding a constant weight, gravity, to time. A rating for an older item decreases much faster if its gravity is increased.</p><h3>Let’s give honest reviews a chance</h3><p>OK, I admit that I don’t have a solution to this problem. But many data scientists are working on it, for a variety of reasons. They all have the same goal: discern truth from what people write on the Internet. There are many obstacles. Some people are more or less unqualified to write reviews, others lie, others provoke, others advance thousands of false information or misleading opinions. The game is afoot, and the game theorists are in the stands taking notes.</p><p>Here’s just a drop in the bucket of the massive research done on this topic, with attempts to solve this problem. This kind of research gives me hope.</p><p>To begin with, <a href="https://www.algolia.com/blog/engineering/algolia-v-elasticsearch-relevance/">ranking relies on a search engine</a>. All search engines rank once they’ve <a href="https://www.algolia.com/blog/engineering/inside-the-algolia-enginepart-4-textual-relevance/">found</a> the results. For example, Algolia contains a <a href="https://www.algolia.com/doc/guides/managing-results/relevance-overview/in-depth/ranking-criteria/">ranking formula</a> that can find the best matches for search terms, and also apply the best numbers (using formulas like Bayes and Hacker News above) to rank results, using <a href="https://www.algolia.com/doc/guides/managing-results/must-do/custom-ranking/">custom ranking</a>.</p><p>Next, let’s look at what researchers are saying about the best way to rank ratings:</p><ul><li><a href="https://pdfs.semanticscholar.org/c039/9025bf726d864116cd0bf88ff4a89d48ae00.pdf">Opinion mining using machine learning and NLP</a></li><li><a href="https://www.atlantis-press.com/journals/nlpr/125941255/view">Detecting misinformation</a></li><li><a href="https://crimesciencejournal.biomedcentral.com/articles/10.1186/s40163-018-0094-4">Automatically identifying the function and intent of posts in underground forums</a></li></ul><p>It’s also good to ensure diversity of opinions:</p><ul><li><a href="https://arxiv.org/abs/2002.12457">Ranking diversification in forum discussions</a></li></ul><h3>Conclusion — Why care about ratings?</h3><p>Everything from buying products to reading news to making life’s decisions is framed by how others rate and think about such items. Word of mouth is now driven by software and big data analytics.</p><p>Working for a search company, I can describe our rating system that makes sense of today’s information revolution. Our engine does this by providing an <em>agnostic</em> ranking mechanism that we call Custom Ranking: you plug in the ratings and we’ll use those ratings to rank results.</p><p>Thus, the work to get reliable ratings lies outside the search engine— it requires clear thinking about the ratings you choose to plug into the search engine.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37e9a4a3882d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Algolia Reduces Latency For 21B Searches Per Month]]></title>
            <link>https://medium.com/algolia-stories/how-algolia-reduces-latency-for-21b-searches-per-month-3959dc926f0?source=rss-a531947eec97------2</link>
            <guid isPermaLink="false">https://medium.com/p/3959dc926f0</guid>
            <category><![CDATA[stackshare]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[algolia]]></category>
            <dc:creator><![CDATA[Algolia]]></dc:creator>
            <pubDate>Tue, 11 Apr 2017 17:34:48 GMT</pubDate>
            <atom:updated>2017-04-11T17:34:48.169Z</atom:updated>
            <content:encoded><![CDATA[<h4>Originally posted on <a href="https://stackshare.io/algolia/how-algolia-reduces-latency-for-21b-searches-per-month">StackShare</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nMgtWx9-RNDDHjr1BZH7bQ.jpeg" /></figure><p><em>By </em><a href="https://stackshare.io/dzello"><strong><em>Josh Dzielak</em></strong></a><em>, Developer Advocate at </em><a href="https://stackshare.io/algolia"><strong><em>Algolia</em></strong></a></p><p><a href="https://www.algolia.com/"><strong>Algolia</strong></a> helps developers build search. At the core of Algolia is a built-from-scratch search engine exposed via a JSON API. In February 2017, we processed 21 billion queries and 27 billion indexing operations for 8,000+ live integrations. Some more numbers:</p><ul><li>Query volume: 1B/day peak, 750M/day average (13K/s during peak hours)</li><li>Indexing operations: 10B/day peak, 1B/day average (spikes can be over 1M/s)</li><li>Number of API servers: 800+</li><li>Total memory in production: 64TB</li><li>Total I/O per day: 3.9PB</li><li>Total SSD storage capacity: 566TB</li></ul><p>We’ve written about our <a href="https://stackshare.io/posts/how-algolia-built-their-realtime-search-as-a-service-product"><strong>stack before</strong></a> and are big fans of StackShare and the community here. In this post we‘ll look at how our stack is designed from the ground up to reduce latency and the tools we use to monitor latency in production.</p><p>I’m Josh and I’m a Developer Advocate at Algolia, formerly the VP Engineering at <a href="https://stackshare.io/keen-io"><strong>Keen IO</strong></a>. Being a developer advocate is pretty cool. I get to code, write and speak. I also get to converse daily with developers using Algolia.</p><p>Frequently, I get asked what Algolia’s <a href="https://stackshare.io/algolia/algolia-api"><strong>API tech stack</strong></a> looks like. Many people are surprised when I tell them:</p><ol><li><strong>The Algolia search engine is written in </strong><a href="https://stackshare.io/cplusplus"><strong>C++</strong></a><strong> and runs inside of </strong><a href="https://stackshare.io/nginx"><strong>nginx</strong></a><strong>.</strong> All searches start and finish inside of our nginx module.</li><li><strong>API clients connect directly to the nginx host where the search happens.</strong> There are no load balancers or network hops.</li><li><strong>Algolia runs on hand-picked bare metal.</strong> We use high-frequency CPUs like the 3.9Ghz Intel Xeon E5–1650v4 and load machines with 256GB of RAM.</li><li><strong>Algolia uses a hybrid-tenancy model.</strong> Some clusters are shared between customers and some are dedicated, so we can use hardware efficiently while providing full isolation to customers who need it.</li><li><strong>Algolia doesn’t use AWS or any cloud-based hosting for the API.</strong> We have our own servers spanning 47 datacenters in 15 global regions.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gAW4UbLkNgSnH_9u.png" /></figure><h4>Why this infrastructure?</h4><p>The primary design goal for our stack is to <strong>aggressively reduce latency</strong>. For the kinds of searches that Algolia powers — suited to demanding consumers who are used to Google, Amazon and Facebook — latency is a UX killer. Search-as-you-type experiences, which have become the norm since Google announced instant search in 2011, have demanding requirements. Any more than 100ms from end-to-end can be perceived as sluggish, glitchy and distracting. But at 50ms or less the experience feels magical. We prefer magic.</p><h3>Monitoring</h3><p>Our monitoring stack helps us keep an eye on latency across all of our clusters. We use <a href="https://stackshare.io/wavefront"><strong>Wavefront</strong></a> to collect metrics from every machine. We like Wavefront because it’s simple to integrate (we have it plugged in to <a href="https://stackshare.io/statsd"><strong>StatsD</strong></a> and <a href="https://stackshare.io/collectd"><strong>collectd</strong></a>), provides good dashboards, and has integrated alerting.</p><p>We use <a href="https://stackshare.io/pagerduty"><strong>PagerDuty</strong></a> to fire alerts for abnormalities like CPU depletion, resource exhaustion and long-running indexing jobs. For non-urgent alerts, like single process crashes, we dump and collect the core for further investigation. If the same non-urgent alert repeats more than a set number of times, we do trigger a PagerDuty alert. We keep only the last 5 core dumps to avoid filling up the disk.</p><p>When a query takes more than 1 second we send an alert into <a href="https://stackshare.io/slack"><strong>Slack</strong></a>. From there, someone on our Core Engineering Squad will investigate. On a typical day, we might see as few as 1 or even 0 of these, so Slack has been a good fit.</p><h4>Probes</h4><p>We have probes in 45 locations around the world to measure the latency and the availability of our production clusters. We host the probes with 12 different providers, not necessarily the same as where our API servers are. The results from these probes are publicly visible at <a href="https://status.algolia.com/"><strong>status.algolia.com</strong></a>. We use a custom internal API to aggregate the large amount of data that probes fetch from each cluster and turn it into a single value per region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*7-BGruZfNfBUQ2Oo.png" /></figure><h4>Downed Machines</h4><p>Downed machines are detected within 30 seconds by a custom Ruby application. Once a machine is detected to be down, we push a DNS change to take it out of the cluster. The upper bound of propagation for that change is 2 minutes (DNS TTL). During this time, API clients implement their internal retry strategy to connect to healthy machines in the cluster, so there is no customer impact.</p><h3>Debugging Slow Queries</h3><p>When a query takes abnormally long — more than 1 second — we dump everything about it to a file. We keep everything we need to rerun it including the application ID, index name and all query parameters. High-level profiling information is also stored — with it, we can figure out where time is spent in the heaviest 10% of query processing. A syscall called <a href="http://man7.org/linux/man-pages/man2/getrusage.2.html"><strong>getrusage</strong></a> analyzes resource utilization of the calling process and its children.</p><p>For the kernel, we record the number of major page faults (ru_majflt), number of block inputs, number of context switches, elapsed wall clock time (using gettimeofday, so that we don’t skip counting time on a blocking I/O like a major page fault since we’re using memory mapped files) and a variety of other statistics that help us determine the root cause.</p><p>With data in hand, the investigation proceeds in this order:</p><ol><li>The hardware</li><li>The software</li><li>Operating system and production environment</li></ol><p><strong>Hardware</strong></p><p>The easiest problem to detect is a hardware issue. We see burned SSDs, broken memory modules and overheated CPUs. We automate the reporting of the most common failures like SSDs by alerting on S.M.A.R.T. data. For infrequent errors, we might need to run a suite of specific tools to narrow down the root cause, like <a href="https://github.com/raas/mbw"><strong>mbw</strong></a> for uncovering memory bandwidth issues. And of course, there is always syslog which logs most hardware failures.</p><p>Individual machine failures will not have a customer impact because each cluster has 3 machines. Where it’s possible in a given geographical region, each machine is located in a different datacenter and attached to a different network provider. This provides further insulation from network or datacenter loss.</p><p><strong>Software</strong></p><p>We have some close-to-zero cost profiling information obtained from the getrusage syscall. Sometimes that’s enough to diagnose an issue with the engine code. If not, we need to look to profiling. We can’t run a profiler in production for performance reasons, but we can do this after the fact.</p><p>An external binary is attached to a profiler, containing exactly the same code as the module running inside of nginx. The profiler uses information obtained by <a href="https://github.com/gperftools/gperftools"><strong>google-perftools</strong></a>, a very accurate stack-sampling profiler, to simulate the exact conditions of the production machine.</p><p><strong>OS / Environment</strong></p><p>If we can rule out hardware and software failure, the problem might have been with the operating environment at that point in time. That means analyzing system-wide data in the hope of discovering an anomaly.</p><p>Once we discovered that defragmentation of huge pages in the kernel could block our process for several hundred milliseconds. This defragmentation isn’t necessary because we keep large memory pools like nginx. Now we make sure it doesn’t happen, to the benefit of more consistent latency for all of our customers.</p><h3>Deployment</h3><p>Every Algolia application runs on a cluster of 3 machines for redundancy and increased throughput. Each indexing operation is replicated across the machines using a durable queue.</p><p>Clusters can be mirrored to other global regions across Algolia’s Distributed Search Network (DSN). Global coverage is critical for delivering low latency to users coming from different continents. You can think of DSN like a CDN without caching — every query is running against a live, up-to-date copy of the index.</p><h4>Early Detection</h4><p>When we release a new version of the code that powers the API, we do it in an incremental, cluster-aware way so we can rollback immediately if something goes wrong.</p><p>Automated by a set of custom deployment scripts, the order of the rolling deploy looks like this:</p><ul><li>Testing machines</li><li>Staging machines</li><li>⅓ of production machines</li><li>Another ⅓ of production machines</li><li>The final ⅓ of production machines</li></ul><p>First, we test the new code with unit tests and functional tests on a host that with an exact production configuration. During the API deployment process we use a custom set of scripts to run the tests, but in other areas of our stack we’re using <a href="https://stackshare.io/travis-ci"><strong>Travis CI</strong></a>.</p><p>One thing we guard against is a network issue that produces a split-brain partition during a rolling deployment. Our deployment strategy considers every new version as unstable until it has consensus from every server, and it will continue to retry the deploy until the network partition heals.</p><p>Before deployment begins, another process has encrypted our binaries and uploaded them to an <a href="https://stackshare.io/amazon-s3"><strong>S3</strong></a><strong> </strong>bucket. The S3 bucket sits behind <a href="https://stackshare.io/cloudflare"><strong>CloudFlare</strong></a> to make downloading the binaries fast from anywhere.</p><p>We use a custom shell script to do deployments. The script launches the new binaries and then checks to make sure that the new process is running. If it’s not, the script assumes that something has gone wrong and automatically rolls back to the previous version. Even if the previous version also can’t come up, we still won’t have a customer impact while we troubleshoot because the other machines in the cluster can still service requests.</p><h3>Scaling</h3><p>For a search engine, there are two basic dimensions of scaling:</p><ul><li>Search capacity — how many searches can be performed?</li><li>Storage capacity — how many records can the index hold?</li></ul><p>To increase your search capacity with Algolia, you can replicate your data to additional clusters using the point-and-click DSN feature. Once a new DSN cluster is provisioned and brought up-to-date with data, it will automatically begin to process queries.</p><p>Scaling storage capacity is a bit more complicated.</p><h4>Multiple Clusters</h4><p>Today, Algolia customers who cannot fit on one cluster need to provision a separate cluster and create logic at the application layer to balance between then. This is often needed by SaaS companies who have customers growing at different rates, and sometimes one customer can be 10x or 100x compared to the others, so you need to move that customer to somewhere they can fit.</p><p>Soon we’ll be releasing a feature that takes this complexity behind the API. Algolia will automatically balance data a customer’s available clusters based on a few key pieces of information. The way it works is similar to sharding but without the limitation of shards being pinned to a specific node. Shards can be moved between clusters dynamically. This avoids a very serious problem encountered by many search engines — if the original shard key guess was wrong, the entire cluster will have to be rebuilt down the road.</p><h3>Collaboration</h3><p>Our humans and our bots congregate on <a href="https://stackshare.io/slack"><strong>Slack</strong></a>. Last year we had some growing pains, but now we have a prefix-based naming convention that works pretty well. Our channels are named #team-engineering, #help-engineering, #notif-github, etc.. The #team- channels are for members of a team, #help- channels are for getting help from a team, and #notif- channels are for collecting automatic notifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vgM1a7stxSYoR8MI.png" /></figure><p>It would be hard to count the number of <a href="https://stackshare.io/zoom"><strong>Zoom</strong></a> meetings we have on a given day. Our two main offices are in Paris and San Francisco, making 7am-10am PST the busiest time of day for video calls. We now have dedicated “Zoom Rooms” with iPads, high-resolution cameras and big TVs that make the experience really smooth. With new offices in New York and Atlanta, Zoom will become an even more important part of our <a href="https://stackshare.io/algolia/algolia-collaboration"><strong>collaboration stack</strong></a> which also includes <a href="https://stackshare.io/github"><strong>Github</strong></a>, <a href="https://stackshare.io/trello"><strong>Trello</strong></a> and <a href="https://stackshare.io/asana"><strong>Asana</strong></a>.</p><h3>Team</h3><p>When you’re an API, performance and scalability are customer-facing features. The work that our engineers do directly affects the 15,000+ developers that rely on our API. Being developers ourselves, we’re very passionate about <a href="https://github.com/algolia"><strong>open source</strong></a> and staying active with our <a href="https://discourse.algolia.com/"><strong>community</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gtCEp9N0-nBOXFUL.jpeg" /></figure><p><strong>We’re hiring!</strong> Come help us make building search a rewarding experience. Algolia teammates come from a diverse range of backgrounds and <a href="https://medium.com/@kasiamichalak/thanks-for-your-offer-but-i-dont-want-to-relocate-9fb8d04ace05#.nuy2xnr5w"><strong>15 different countries</strong></a>. Our values are Care, Humility, Trust, Candor and Grit. Employees are encouraged to travel to different offices — Paris, San Francisco, or now Atlanta — at least once a year, to build strong personal connections inside of the company.</p><p>See our <a href="https://stackshare.io/algolia/stacks#jobs"><strong>open positions</strong></a> on StackShare.</p><p>Questions about our stack? We love to talk tech. Comment below or ask us on our <a href="http://discourse.algolia.com/"><strong>Discourse forum</strong></a>.</p><p><em>Thanks to Julien Lemoine, Adam Surak, Rémy-Christophe Schermesser, Jason Harris and Raphael Terrier for their much-appreciated help on this post.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.stackshare.io%2Fstacks%2Fembed%2F96abeaee54c64756581339860d85a9&amp;url=http%3A%2F%2Fstackshare.io%2Falgolia%2Fhow-algolia-reduces-latency-for-21b-searches-per-month&amp;image=https%3A%2F%2Fimg.stackshare.io%2Fstack%2F54181%2Fs3SgdSkAMtcIYEOK.png&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=stackshare" width="700" height="977" frameborder="0" scrolling="no"><a href="https://medium.com/media/693413c32f13cebe01e07ab2a456d606/href">https://medium.com/media/693413c32f13cebe01e07ab2a456d606/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3959dc926f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/algolia-stories/how-algolia-reduces-latency-for-21b-searches-per-month-3959dc926f0">How Algolia Reduces Latency For 21B Searches Per Month</a> was originally published in <a href="https://medium.com/algolia-stories">Algolia Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Developer Advocate’s Guide to Github Universe 2016]]></title>
            <link>https://medium.com/algolia-stories/the-developer-advocates-guide-to-github-universe-2016-202d1957af22?source=rss-a531947eec97------2</link>
            <guid isPermaLink="false">https://medium.com/p/202d1957af22</guid>
            <category><![CDATA[events]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[developer-relations]]></category>
            <dc:creator><![CDATA[Algolia]]></dc:creator>
            <pubDate>Tue, 13 Sep 2016 19:25:59 GMT</pubDate>
            <atom:updated>2016-10-28T22:54:40.352Z</atom:updated>
            <content:encoded><![CDATA[<p>The second annual <a href="http://githubuniverse.com/">Github Universe</a> kicks off this Wednesday, September 14th in San Francisco. Universe is ‘the event for people building the future of software’ and will bring together <strong>50+ speakers</strong> and <strong>1,400 attendees</strong> from all over the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mn0lysP2rXvrAQZs7enB6w.png" /></figure><p>The attendees are intentionally a mix of developers, technical leads and business leaders, which makes it a great place to be a developer advocate. Why? Because the people who make the decisions that impact developers’ lives will be right here. Developer advocacy isn’t just about getting people to use your product, it’s about <strong>speaking up for all developers</strong>. CEOs, CTOs, and founders are coming to Universe because they want to hear this message—they want to know how to create the environments that developers thrive in. It’s our job to tell them.</p><blockquote>Handy links to <a href="https://www.universe.com/events/576d43033d50870025d93493">tickets</a>, <a href="https://www.google.com/maps/place/Pier+70+Partners/@37.7586145,-122.3865731,17z/data=!4m5!3m4!1s0x808f7fbc070091e7:0x7cd7d30c77b66b98!8m2!3d37.7586145!4d-122.3843844?hl=en&amp;shorturl=1">map</a>, <a href="http://githubuniverse.com/program/">schedule</a>, <a href="http://githubuniverse.com/speakers/">speakers</a>, <a href="http://githubuniverse.com/program/sessions/">sessions</a>, <a href="http://githubuniverse.com/bigbang/">afterparty</a>, <a href="http://githubuniverse.com/faq/">FAQ</a> and <a href="https://twitter.com/githubuniverse">@GithubUniverse</a>.</blockquote><p>There are three must-haves for developers: <strong>trust</strong>, <strong>transparency</strong>, and <strong>community</strong>. Each value is essential to getting a quality job done day in and day out. This year’s Github Universe program is <strong>packed with talks</strong> related to these values and I’ve done my best to highlight the ones that I’m most excited about. The guide is in chronological order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZbY5J--M_Und9A89AZvYQ.jpeg" /><figcaption>Octo-art from Universe 2015</figcaption></figure><h3>Tuesday Evening: Registration Social</h3><p>Avoid the lines by picking up your badge early and enjoy a specialty cocktail hour hosted by <a href="https://www.heroku.com/">Heroku</a>.</p><p><strong><em>5:30pm-8:00pm<br></em></strong><a href="http://dogpatchstudios.com/"><em>Dogpatch Studios</em></a><em><br>991 Tennessee St</em></p><h3>Wednesday: Opening Keynote</h3><p>The conference opens with a keynote by Chris Wanstrath a.k.a. <a href="https://github.com/defunkt">defunkt</a>, the CEO &amp; Co-founder of Github. <a href="https://www.youtube.com/watch?v=6-FqUO_RzUo">Last year’s keynote</a> announced the launch of the <a href="https://github.com/integrations">Integrations Directory</a> and <a href="https://github.com/blog/2069-git-large-file-storage-v1-0">Git LFS</a> and it looks like we’ll get <a href="https://github.com/universe-2016">a few more announcements</a> this year.</p><blockquote>Pier 70 will have free valet BIKE parking during the event. You can also get there on public transportation via the <a href="https://www.sfmta.com/getting-around/transit/routes-stops/k-t-kt-k-inglesidet-third-street">T Muni line</a>.</blockquote><h3>Wednesday Afternoon: Three Tracks</h3><p>Here are my dev advocate picks for Wednesday afternoon.</p><p><strong>11:40am:</strong> I’m going to give you two choices.</p><ul><li>For something <strong>Developer Experience</strong>-flavored, head to <a href="http://githubuniverse.com/program/sessions/#github-flow">The Github Flow</a> by <a href="https://github.com/beardofedu">Matt Desmond</a> and <a href="https://github.com/hollenberry">Eric Hollenberry</a>.</li><li>For something <strong>Developer Community</strong>-flavored<strong>, </strong>go to <a href="http://githubuniverse.com/program/sessions/#developing-os"><em>Developing Open Source in Service to National Security</em></a><em>, </em>a talk by <a href="http://people.llnl.gov/lee1001">Ian Lee</a> about the use of open source software at the Bay Area’s own <a href="https://www.llnl.gov/">Lawrence Livermore National Laboratory</a>.</li></ul><p><strong>1:40pm</strong>: After lunch it’s <a href="http://githubuniverse.com/program/sessions/#pains-gains">Pains &amp; Gains: Five Short Stories</a>, a session about the trials and non-technical tribulations of open source project development. You’ll hear 5 different stories from 5 different people:</p><ul><li><a href="https://twitter.com/kytrinyx">Katrina Owen</a>, Open Source Advocate, GitHub</li><li><a href="https://lynnlangit.com/">Lynn Langit</a>, Cloud &amp; Data Architect Director, Teaching Kids Programming</li><li><a href="https://twitter.com/gr2m">Gregor Martynus</a>, Frontend &amp; Backend Developer, Hoodie</li><li><a href="https://twitter.com/mogox">Enrique Carlos Mogollan</a>, Software Engineer, Salesforce Desk</li><li><a href="https://twitter.com/jhchen">Jason Chen,</a> Author &amp; Maintainer, Quill</li><li><a href="https://github.com/gamefiend">Quinn Murphy</a>, Senior Systems Engineer, Netsuite</li></ul><p><strong>2:40pm:</strong> <a href="http://githubuniverse.com/program/sessions/#innersource">InnerSource: Reaping the Benefits of Open Source, Behind Your Firewall</a> is a session about applying open source principles inside of corporate engineering organizations, which is going to <strong>make developers’ lives better</strong>. Advocates, round up!</p><p>In the same time slot over on the <em>Flight</em> track, <a href="http://githubuniverse.com/program/sessions/#lending-privilege">Lending Privilege</a> by <a href="http://www.anjuansimmons.com/">Anjuan Simmons</a> is about using your privilege to effect positive change for marginalized groups. To me this is a super important topic, for advocates and everybody in tech, and so you might see me over here.</p><p><strong>4:00pm:</strong> After the break, head over to the <em>Orbit</em> track for <a href="http://githubuniverse.com/program/sessions/#make-collab-safer">Making online collaboration safer at GitHub</a> by <a href="https://twitter.com/tsunamino?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor">Danielle Leong</a>.</p><blockquote>“The internet is not a safe place for many marginalized people, and efforts by social media companies to address the problem of online harassment have been largely ineffective.”</blockquote><p>Learn about how Github is addressing safety and inclusivity and meet the Github Community and Safety team.</p><h3>Wednesday Night: The Big Bang Afterparty</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*abx1QMB5OiwC5Wg2iF23Mw.png" /><figcaption>Even if you’re not attending Universe, you can still <a href="https://github.com/blog/2223-github-universe-big-bang-benefit-concert">get Big Bang tickets here</a></figcaption></figure><blockquote>You can buy a ticket to the afterparty even if you’re not attending the conference.</blockquote><p>The <a href="http://githubuniverse.com/bigbang/">Big Bang</a> afterparty starts at 8pm and features <a href="http://www.thinkcommon.com/">Common</a> and <a href="http://lionbabe.com/">Lion Babe</a> at <a href="http://sfmasonic.com/">The Masonic</a>. All proceeds benefit <a href="http://www.blackgirlscode.com/">Black Girls Code</a>, and even if you are not going to the conference you can still <a href="http://concerts.livenation.com/event/1C00510AC18D3389">buy a ticket</a>. This is a great setup for what is sure to be a great event.</p><h3>Josh Dzielak on Twitter</h3><p>Awesome things about the @githubuniverse Big Bang event * All proceeds go to @BlackGirlsCode * Open to the wider community * Common!</p><h3>Thursday Morning Keynote</h3><p>Grab your cup of coffee before <strong>10:00am </strong>so you don’t miss the keynote by <a href="https://medium.com/u/b3349e3e2f14">Nicole Sanchez</a>, Github’s VP of Social Impact, including special guests. Beforehand, be sure to check out a few of Nicole’s <a href="https://medium.com/@nmsanchez/diversity-in-tech-faq-v0-1-276e5acd25be#.syldrfyx8">awesome</a> <a href="https://medium.com/@nmsanchez/diversity-in-tech-and-what-we-ve-already-lost-987aeb58eb41#.uj9sx9wgt">Medium</a> <a href="https://medium.com/@nmsanchez/which-women-in-tech-371e721e71c4#.h8xasidaw">posts</a> about diversity in tech.</p><h3>Thursday Afternoon: More Choices</h3><p>After lunch, it’s back to choosing between three tracks. Some really tough choices towards the end of the day!</p><p><strong>11:40am: </strong>What could be a better talk than one that combines <strong>code</strong> and <strong>values</strong>? Very much looking forward to <a href="http://githubuniverse.com/program/sessions/#instagram">Codifying Company Values</a> by <a href="https://www.linkedin.com/in/hui-ding-76825911">Hui Ding</a> of Instagram.</p><p><strong>1:40pm:</strong> I love this title, <a href="http://githubuniverse.com/program/sessions/#tribal-knowledge">Transforming Tribal Knowledge into Learning Trees</a>, by <a href="https://www.linkedin.com/in/jimdkohl">Jim Kohl</a> of the <a href="http://www.greatamericaninsurancegroup.com/Pages/default.aspx">Great American Insurance Company</a>. Software development organizations rely <strong>heavily</strong> on tribal and guild knowledge—how can we make sure that knowledge doesn’t get lost?</p><p><strong>2:40pm:</strong> The advocate pick here is <a href="http://githubuniverse.com/program/sessions/#billions">Billions and billions of commits: Learning from open source data</a>. To advocate for developers, you have to know what developers are doing, and the Github dataset contains a wealth of information on where the community is headed. Participating in this session are:</p><ul><li><a href="https://twitter.com/arfon">Arfon Smith</a>, Program Manager for Open Source Data, GitHub</li><li><a href="https://twitter.com/abernathyca">Christine Abernathy</a>, Open Source Developer Advocate, Facebook</li><li><a href="https://twitter.com/dabbish">Laura Dabbish</a>, Professor, Human-Computer Interaction Institute, Carnegie Mellon University</li><li><a href="https://twitter.com/felipehoffa">Felipe Hoffa</a>, Developer Advocate, Big Data, Google</li><li><a href="https://twitter.com/jeffmcaffer">Jeff McAffer</a>, Director of the Microsoft Open Source Programs Office, Microsoft</li></ul><p>But then you also have <a href="http://githubuniverse.com/program/sessions/#crossing-canyon">Crossing the Canyon of Cognizance: A Shared Adventure</a> by <a href="https://twitter.com/pwnela">Pamela Vickers</a>, about learning and mentorship. See how to save new engineers from the “what-do-I-even-Google” abyss.</p><p><strong>4:00pm</strong>: <a href="https://twitter.com/roidrage">Mathias Meyer</a> of <a href="https://travis-ci.org/">Travis CI</a> is a fantastic presenter and I predict that <a href="http://githubuniverse.com/program/sessions/#distributed-inclusive">Building and Scaling a Distributed and Inclusive Team</a> will be amazing.</p><p>Competing for your attention is <a href="http://githubuniverse.com/program/sessions/#crossing-the-language">Crossing the Language Divide in Open Source Projects</a> by <a href="https://twitter.com/chimeracoder">Aditya Mukerjee</a> at Stripe. This talk is about language barriers in the OSS world, which something that developer advocates definitely need to be aware of. You’ll have to decide this one for yourself :)</p><blockquote>The Github Universe site has detailed <a href="http://githubuniverse.com/experience/">experience guides</a> for the venue, transportation, and the city of San Francisco.</blockquote><h3>Thursday 5pm: 99% Invisible</h3><p>Don’t leave too early, <strong>99% Invisible with Roman Mars </strong>comes to you at <strong>5pm</strong>. <a href="http://99percentinvisible.org/">99% Invisible</a> is one of my favorite podcasts (for a taste check out this 5-minute <a href="https://www.producthunt.com/podcasts/99-invisible-15-sounds-of-the-artificial-world">episode about</a> sounds of the artificial world). I can’t wait to see what they’ve got in store.</p><h3>It’s A Small Universe</h3><p>Reach out via <a href="https://twitter.com/dzello">twitter</a> or <a href="/service/mailto:josh@algolia.com">email</a> if you’d like to connect at Github Universe. I love talking about code, values, community, and of course search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*RumrCVem8ifDc7ISL0-mdQ.png" /><figcaption>Find me and I will give you a colorful Algolia community sticker!</figcaption></figure><p>If you maintain any open source or API documentation I’d like to tell you about Algolia <a href="https://community.algolia.com/docsearch/">DocSearch</a> :) If you’re currently building search for your website, app or database than we should probably talk about that too ;)</p><p>See you out there! 🚀🚀🚀</p><p><em>— </em><a href="https://twitter.com/dzello"><em>Josh Dzielak</em></a><em>, Developer Advocate, </em><a href="https://algolia.com/"><em>Algolia</em></a></p><p>P.S. I.C.Y.M.I. —&gt;</p><h3>Mathias Meyer on Twitter</h3><p>Rumor has it that for the first time in our existence (almost 5 years!), there&#39;ll be @travisci t-shirts available, only at @githubuniverse!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=202d1957af22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/algolia-stories/the-developer-advocates-guide-to-github-universe-2016-202d1957af22">The Developer Advocate’s Guide to Github Universe 2016</a> was originally published in <a href="https://medium.com/algolia-stories">Algolia Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Algolia’s Fury Road to a Worldwide API]]></title>
            <link>https://medium.com/algolia-stories/algolia-s-fury-road-to-a-worldwide-api-c1536c46f3a5?source=rss-a531947eec97------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1536c46f3a5</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Algolia]]></dc:creator>
            <pubDate>Fri, 14 Aug 2015 19:03:58 GMT</pubDate>
            <atom:updated>2015-08-17T10:58:15.021Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zyd0BTL2hI9D7m-a9znqcg.png" /></figure><p>by <a href="https://twitter.com/jlemoine_algo">Julien Lemoine</a>, CTO <a href="https://twitter.com/algolia">@algolia</a></p><blockquote>The most frequent questions we answer for developers and devops are about <a href="http://highscalability.com/blog/2015/3/9/the-architecture-of-algolias-distributed-search-network.html">our architecture</a> and how we achieve such high availability.</blockquote><p>Some of them are very skeptical about high availability with bare metal servers, while others are skeptical about how we distribute data worldwide.</p><p>However, the question I prefer is “How is it possible for a startup to build an infrastructure like this”. It is true that our current architecture is impressive for a young company:</p><ul><li>Our high-end dedicated machines are hosted in <strong>13 worldwide regions with 25 data-centers</strong></li><li>our master-master setup <strong>replicates our search engine on at least 3 different machines</strong></li><li>we process over<strong> 6 billion queries per month</strong></li><li>we receive and handle over <strong>20 billion write operations</strong> per month</li></ul><p>Just like Rome wasn’t build in a day, our infrastructure wasn’t as well. This series of posts will explore the 15 instrumental steps we took when building our infrastructure. I will even discuss our outages and bugs in order to you to understand how we used them to improve our architecture.</p><h3>The Cloud versus Bare metal debate</h3><p>Before diving into the details of our architectural journey, I would like to address a choice that had a big consequences on the rest of the infrastructure. We needed to decide if we should use a cloud-based infrastructure or bare metal machines. A hot topic that is regularly debated in technical discussions.</p><p><strong>Cloud infrastructure</strong> is a great solution for most use cases, especially in the early stages.They have been instrumental in improving the high-availability of many services.</p><p>Solutions having databases on multiple availability zones (AZ) or several instances running on different AZ while storing all their state in the multiple AZ database are perfect example of this. This is a standard setup used by many engineers and easily deployed in a few minutes.</p><p><strong>Bare metal infrastructures</strong> require you to understand and design the small details in order to build the high availability yourself. This is a do-it-yourself approach that only makes sense for a small set of use cases.</p><p>We often encounter deployments using bare metal machines in a single datacenter. This does not make sense as it is less fault tolerant than a quick deployment on a cloud provider, the datacenter is a single point of failure (SPoF).</p><p>Bare metal hardware remains an interesting option for businesses linked to hardware, which happens to be our case. <strong>By choosing a bare metal infrastructure, we were able to purchase higher performance hardware than that offered by cloud providers</strong>. On top of the performance gain the cost was significantly cheaper. We chose this option early on because we were fully aware we would need to build the high-availability ourself!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*EuqysdKBtPJ29eYh." /></figure><h2>.1</h2><h4>March 2013</h4><h3>High availability was designed, not implemented!</h3><blockquote>At this time we had our first private beta running of our search as a service API. At this point in time, we could only measure our performance. We had not yet developed the high availability part of the product.</blockquote><p>We were highly confident our market would be worldwide so we launched a single machine in two different locations, Canada/East and Europe/West, with the following specifications:</p><ul><li>32G of memory</li><li>Xeon E3–1245 v2 (4 cores, 8 threads, 3.4Ghz-3.8Ghz)</li><li>2x Intel SSD 320 series of 120GB in Raid-0</li></ul><p>Each machine was hosting a different set of users depending on their location. During our private beta, the focus was 100% on performance which is why the clock speed was a major factor in our decisions (for the same generation CPU, clock speed is directly related to speed of a search query in search engines). Since the beginning, we have done indexing on a separate process with a nice level of five. All search queries were processed directly inside nginx that we let with a nice level of zero (the lower the nice level of a process is, the more CPU time it gets). This setup allowed us to handle traffic spike efficiently by giving the search the highest allocation CPU priority. This worked very well compared to approaches used by other engines.</p><p>We were very surprised to have one of our first beta testers replace their previous solution with ours in production because they were so happy with the performance and relevance. As you might imagine, we were very stressed out about this. Since high availability was not implemented, we were worried about potential downtime affecting them and explained the product was not ready production! The customer told us the risk versus rewards was acceptable to them because they could rollback to their previous provider if needed. On a side note, this story helped us secure our first round of funding before the launch of the product. It ended up being our first proof of market fit. Better yet, we can call it our problem-solution fit! We can’t thank that customer enough :)</p><h2>.2</h2><h4>June 2013</h4><h3>Implementation of high availability in our architecture</h3><blockquote>After three months of development and a huge amount of testing (the monkey testing approach was really fun!), we introduced the high availability support in our beta.</blockquote><p>You can read more about it in our <a href="http://highscalability.com/blog/2015/3/9/the-architecture-of-algolias-distributed-search-network.html">architecture post</a>.</p><p>The idea was to have a cluster of three identical machines instead of one where each machine is a perfect replica with all of the data and able to acts as a master. This means that each one is capable of accepting write operations from API users. Each write operation triggers a consensus to make sure all machines have all the jobs and apply them in the same order.</p><p>We used the preliminary results from the first beta to design our new hardware setup. We discovered the following:</p><ul><li>32G of memory was not enough, indexing was using up to 10G when receiving big indexing jobs from several users, which only let 22G to cache disk IO</li><li>Disk space was too low for high availability since machines needed to keep several jobs on disk in order to handle a node failure</li><li>Having more memory required us to move to Xeon E5 series (E3 can only address 32G of memory). As the clock speed was important, we decided to go for the Xeon E5 1600 series that offered a very good clock speed and is able to address a lot more memory that the Xeon E3.</li></ul><p>With these discoveries, our setup evolved into three machines with the following specs:</p><ul><li>64G of memory</li><li>Xeon E5–1650 (6 cores, 12 threads, 3.2Ghz to 3.8Ghz)</li><li>2x Intel SSD 320 series of 300GB in Raid-0</li></ul><blockquote>At this point we were able to tolerate hardware failures! However, we were still far from what cloud providers offer with multiple availability zones. All our machines were in the same datacenter with a single provider and without any knowledge of the infrastructure.</blockquote><p>At the same time, we investigated whether or not the load balancing and detection failure between machines should be handled with hardware or software. We tested several approaches and found all hardware load balancers would make the use of several providers close to impossible. We ended up implementing a basic retry strategy in our API clients. Each API client was developed to be able to access three different machines. Three different DNS records represented each user: <a href="http://useridid-1.algolia.io/">USERIDID-1.algolia.io</a>, <a href="http://userid-2.algolia.io/">USERID-2.algolia.io</a> and <a href="http://userid-3.algolia.io/">USERID-3.algolia.io</a>. Our first implementation was to randomly select one of the records and then retry with a different one in case of failure.</p><h2>.3</h2><h4>August 2013</h4><h3>Official launch of the service</h3><blockquote>During the summer, we increased the number of API clients to 10 (JS, Ruby, Python, PHP, Objective-C, Java, C#, Node.js…). We decided to avoid using auto code generation and developed the API clients manually.</blockquote><p>While it was more work, we needed to make sure the networking code was good for things like HTTPS keep alive, using TLS correctly, having our retry strategy implemented correctly with the correct timeout, etc.</p><p>We officially launched the service at the end of August 2013 with our two locations (Europe/West and Canada/East). Each location contained a cluster of three identical hosts with the following specs:</p><ul><li>128G RAM</li><li>E5–2687W (8 cores, 16 threads, 3.1Ghz to 3.8Ghz)</li><li>2x Intel S3500 series of 300GB in Raid-0</li></ul><blockquote>Compared to the previous configuration, the main things we changed were to increase the memory size and use a better SSD. Those two changes were done based on the observation that the SSD was the bottleneck during indexing and the memory was not enough to cache all the users’ data in memory. For the CPU upgrade, it was more a question of oversizing to ensure we would have enough resources.</blockquote><p>At this point, the next big item for us to focus on was implementing availability zones for our deployments. We needed to run our three machines on different network equipment and power units. Hopefully our provider was transparent about their infrastructure and where our machines were allocated. It wasn’t perfect, but we were able to implement a solution similar to those of the different cloud providers. We suspect the cloud providers do something similar to what we implemented, but have not found any detailed documentation on this topic!</p><h2>.4</h2><h4>January 2014</h4><h3>Deployment Is A Big Risk For High Availability</h3><blockquote>At this time, one of our major concerns was making sure our development was agile while not having it at the cost of stability.</blockquote><p>For this reason, we developed a test suite of 6,000+ unit tests and 200+ non-regression tests to ensure code changes did not introduce new bugs. Unfortunately, it wasn’t enough. A new feature that passed all our tests introduced a bug in production that caused <a href="https://blog.algolia.com/postmortem-todays-8min-indexing-downtime/">eight minutes of indexing downtime</a>. Thankfully, because our architecture was designed to keep search queries separate from indexing, no search queries were impacted.</p><p>This issue helped us identify several problems in our high availability setup.</p><p>Rollbacks need to be fast so we developed the ability to perform a rollback with a single command line.</p><p>Deployment scripts need to perform sanity checks and automatically rollback if there are errors so we added it to our deployment procedure.</p><p>We shouldn’t deploy to all of our production clusters just because the tests passed.After this issue, we put in place a sequence of deployment for clusters. We start with our test clusters, followed by the community clusters (free customers), and finally the paying user clusters.</p><p>Today, our deployment is a bit different when there is a new feature to test. We select a set of clusters to test and we deploy using the following steps:</p><ol><li>Deploy to the first machine in all clusters of the selected set.</li><li>Monitor for 24 hours then deploy to the second machine in all clusters of the selected set.</li><li>Monitor for 24 hours then deploy to the third machine in all clusters of the selected set.</li><li>After a few days, we deploy to all our production clusters.</li></ol><p>Using this approach, we were able to detect bugs within a few hours that were nearly impossible to find with unit tests. This detection was possible because of the billions of queries we have per month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*RRoRUJCyMK6gFxFUWOmNZg.jpeg" /></figure><h2>.5</h2><h4>March 2014</h4><h3>Manage High Load Of Write Operations</h3><blockquote>While our clusters in Canada/East and Europe/West were handling our growth very well, we moved to solving a new problem: latency.</blockquote><p>The latency of our clusters from Asia was too high for acceptable performance. We decided to test the market first by deploying machines in AWS. We didn’t like having to make this choice since the performance of search queries was 15% lower than our E5–2687W CPU even when using the best CPU offered by AWS (E5–2680 at that time)! We went this route in order to reduce the time to launch the experiment but made sure to not introduce any dependencies on AWS. This allowed us to easily migrate to another provider if the tests were successful. On paper, it all looked good. We ran into one issue when we discovered their virtualization did not support AVX instructions, which was an important feature for us. We had to generate binaries without the AVX instruction set (which reduced again a bit the performance of search queries).</p><p>While we were enjoying our recent launch in Singapore we started to receive some complaints from European customers about an increased latency in their search queries. We quickly identified it was correlated with a big spike of indexing operations, which was weird since we were confident our architectural split between indexing and search CPU usage was working correctly. After investigating, we found that the consensus algorithm for distributed coherency across the cluster that handled write operations had a potential bottleneck. When the bottleneck occurred, it would block the HTTP server threads. This would then lead to search queries needing to wait for a thread. We had accidentally created this issue by design!</p><p>We fixed this issue by implementing a queue before our consensus. It allowed us to receive a spike of write operations, batch them, and then send them to the consensus. The consensus was no longer in the sequence of the HTTP server so write operations could not freeze the HTTP server thread anymore. During normal operation, the queue is not doing anything except passing the job to the consensus. In cases where we have spikes, it allowed us to buffer and batch before calling the consensus. After we made this change, we have not had any clusters freeze.</p><h2>.6</h2><h4>April 2014</h4><h3>Network High Availability Is Close To Impossible With One Data Center</h3><blockquote>In the beginning of April 2014, we started receiving complaints from our US-East customers that used a cluster in Canada/East.</blockquote><p>Our users in US-West were not impacted. We immediately suspected a network issue between Canada and US-East and discovered our provider was reporting the same issue. A car accident had broken a pipe containing 120 fibers between Montreal and New York. All the traffic was being routed on a different path through Chicago. Unfortunately the bandwidth was not good enough to avoid packet lost. This outage was not only impacting our provider but all traffic between Canada and US-East. This is the type of accident that can break the Internet that we often forget about, but it can still happen. There was nothing we could do that day to help our customers except reach out to each one and notify them of the current situation.</p><p>It was on this day we kicked off a big infrastructure discussion. We needed to improve our high availability by relying on more providers, datacenters, and network providers. More importantly, we needed to have our infrastructure in the US. By serving the US from Canada, we were able to cut our costs but our high availability suffered. We needed to have our infrastructure truly distributed. We couldn’t continue having several availability zones on a single provider!</p><h2>.7</h2><h4>July 2014</h4><h3>First Deployment On Two Data Centers</h3><blockquote>We had discovered our deployment in a single datacenter was not enough.</blockquote><p>We decided to start with one of our biggest customers by deploying a machine in two different datacenters (with more than 100 kilometers between them). These two datacenters were using the same provider (same Autonomous System) and already offered a level of high availability a bit above cloud provider multiple availability zones. This was due to the two datacenter locations being completely different in regards to network link and power units.</p><p>Based on our past experience, we took this time to design the next version of our hardware with next generation of CPUs and SSDs:</p><ul><li>Xeon E5–1650v2 (6 cores, 12 threads, 3.5Ghz-3.9Ghz)</li><li>128G of RAM</li><li>Intel SSD S3500 (2x480G) or S3700 (2x400G). We use the intel S3700 for machines with high amounts of writes per day as they are more durable</li></ul><p>The changes we made were mostly around the CPU. We were never using 100% of the CPU on our E5–2687W. The E5–1650v2 was the next generation CPU and provided higher clock speeds. The result was that our service was able to gain close to 15% in speed compared to the previous CPU. <strong>Milliseconds matter!</strong></p><h2>.8</h2><h4>September 2014</h4><h3>Presence In The US!</h3><blockquote>After spending a few months in discussions with several providers, we launched the service in US-East (Virginia) and US-West (California) in September 2014 with a new provider.</blockquote><p>This first step was a great way to improve our search experience in the US thanks to better latency and bandwidth. The improvement to high availability was not big but it did help us be more resilient to the past problems we had between Canada and US-East. We used the same approach as we did with previous locations: different availability zone within one provider (different network equipment and power unit).</p><h2>.9</h2><h4>October 2014</h4><h3>Automation Via Chef</h3><blockquote>With a significant increase in the number of machines we needed to manage, we migrated our management to Chef.</blockquote><p>Previously, we managed our machines using shell scripts but fixing security issues like Heartbleed (OpenSSL vulnerability) was time consuming compared to automation with Chef.</p><p>The automation provided by Chef was very useful when configuring hundreds of machine, but it had it drawbacks. A few months after migrating to Chef, a typo in a cookbook caused some production servers to crash. Luckily, we were able to discover the problem early on. It didn’t impact much of the production thanks to the non-aggressive nature of the CRON job that launched the Chef client. We had enough time to react and fix the issue.</p><p>We got really lucky with this issue because the error could have broken production. In order to avoid this type of problem from happening again, we decided to split our production cookbooks into two versions:</p><ul><li>The first version, the stable one, was deployed to the 1st and 2nd machine of all clusters</li><li>The second version, the production one, was deployed on the 3rd machine of all clusters</li></ul><p>When any modification is made to our cookbook, we first apply the modification to the production version. After a few days of testing, we apply the modification to the stable cookbook version. High availability needs to be applied at all levels!</p><h2>.10</h2><h4>November 2014</h4><h3>DNS Is A SPoF In The Architecture</h3><blockquote>Over time, we started to receive more and more feedback from users stating our service was intermittently slow, especially in Asia.</blockquote><p>After investigating, we found our usage of the .io TLD was the cause of this issue. It turned out the anycast network of the .io TLD has fewer locations than the main TLDs (.net, .com, and .org) and the DNS servers are overloaded. Users would sometimes encounter timeouts during DNS resolution.</p><p>We discussed this issue with CDN providers and they all told us the best practice was to use the .net TLD. We needed to migrate from .io to .net! While preparing for the release of our Distributed Search Network, we decided to move to another DNS provider. This new provider offered the linked domains feature, which meant they would handle the synchronization between <a href="http://algolia.io/">algolia.io</a> and <a href="http://algolia.net/">algolia.net</a> for us so we could maintain backwards compatibility easily.</p><p>We performed extensive testing from different locations for this migration and felt comfortable everything was working well. Unfortunately, after we completed the migration we discovered several problems that impacted some of our users. You can read a detailed report in <a href="https://blog.algolia.com/black-thursday-dns-issue/">this blog post</a>. DNS is far more complex than we realized. There are many different behaviors of DNS providers and our tests were not covering all of them (e.g., resolution in IPv6 first, different behaviors on TTL, etc.)</p><p>This issue also changed our minds about identifying SPoF. Having only one DNS provider is a SPoF and the migration of this piece was highly critical since there was no fallback. We started working on a plan to remove any SPoF in our architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*QesVomHiBwtpUwKNyFxIPA.jpeg" /></figure><h2>.11</h2><h4>February 2015</h4><h3>Launch Of Our Synchronized Worldwide Infrastructure</h3><blockquote>During this month, we accomplished the vision we had been working towards since April 2014, “worldwide expansion to better serve our users”.</blockquote><p>This network was composed of 12 different locations: US-East (Virginia), US-West (California), Australia, Brazil, Canada, France, Germany, Hong Kong, India, Japan, Russia and Singapore. On top of all that, we also launched our “Distributed Search” feature. With this feature, in a few clicks, you are able to set the locations in our network where your data should be automatically duplicated. You get to use the same API as before and queries are automatically routed from the end-user’s browser or mobile application to the closest location.</p><p>This was a huge step in reducing latency for end users and improving our high availability via worldwide distribution of searches. Serving international users from one location leads to a very different quality of service because of distance and saturation of Internet links. We now provide our users a way to solve that! On top of reducing latency, this improved the high availability of our search infrastructure because it was no longer dependent on a single location. It’s worldwide!</p><p>Some people compare our DSN (Distributed Search Network) to a CDN (Content Delivery Network), but it is very different. We don’t store a cache of your frequent queries on each edge. We store a complete replica of all your data. We can respond to any query from the edge location itself, not just the most popular. This means when you select three POPs (US-East, Germany, Singapore), users in Europe will go to the Germany location, users in Asia will go to Singapore, and users in America will go to US-East. All POPs will respond to queries without having to communicate to another edge. This makes a huge difference in terms of user experience and high availability.</p><p>In order to support this change, we updated our retry logic in our API clients to target the<a href="http://appid-dsn.algolia.net/">APPID-dsn.algolia.net</a> hostname first, which is routed to the closest location using DNS based on geoip. If the closest host is down, the DNS record is updated to remove the host in less than one minute in order to return the next closest host. This is why we use a low TTL of 1 minute on each record. In case of failure, if the host is down and DNS has not been updated, we redirect traffic to the master region via a retry on <a href="http://appid-1.algolia.net/">APPID-1.algolia.net</a>,<a href="http://appid-2.algolia.net/">APPID-2.algolia.net</a> and <a href="http://appid-3.algolia.net/">APPID-3.algolia.net</a>in our official API clients. This approach is the best balance we have found between high performance and high availability, we only have a degradation of performance during one minute in case of failure, but the API remains up &amp; running!</p><h2>.12</h2><h4>March 2015</h4><h3>Better High Availability Per Location</h3><blockquote>The Distributed Search Network option was a game changer for high availability but it’s only for search and international users.</blockquote><p>In order to improve the high availability of the main region, we spread our US clusters across two completely independent providers:</p><ul><li>Two different datacenters in close locations (Ex: COPT-6 in Manassas &amp; Equinix DC-5 in Ashburn : 24 miles between them, 1ms of latency)</li><li>Three different machines — just like before (two in the first datacenter in different availability zones and one on the second datacenter)</li><li>Two different Autonomous Systems (so two totally different network providers)</li></ul><p>These changes improved our ability to react to certain problems we faced like the saturation of a peering link between on provider and AWS. Having different providers gives us the option to reroute traffic to other providers. This is a big step in terms of high availability improvements in one location.</p><h2>.13</h2><h4>April 2015</h4><h3>The Random File Corruption Headache</h3><blockquote>April 2015 was a black month for our production team. We started observing random file corruptions on our production machines due to a bug in the TRIM implementation of some of our SSDs.</blockquote><p>You can read the problem in detail in <a href="https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/">this blog post</a>. It took us approximately one month to track the issue and properly identify it. During this time, we suspected everything, starting with our own software! This was also a big test for our architecture. Having random file corruptions on disk is not something easily handled. Nor is it easy to inform our users that they needed to reindex their data because our disks were corrupted. Luckily, we never lost any data of our customers.</p><p>There are two important factors in our architecture that allowed us to face this situation without losing any data:</p><ul><li>We store three replicates of the data. The probability of a random corruption of the same data on the three machines is almost zero (and it didn’t happen).</li><li>More importantly, we did not replicate the result of indexing. Instead, we duplicated the operation received from the user and applied it on each machine. This design makes the probability of having several machines having the same corruption very low since one affected machine cannot “contaminate” the others.</li></ul><p>We never considered this scenario when we designed our architecture! Even though we tried to think about all types of network and hardware failure, we never imagined the consequences of a kernel bug and even worse a firmware bug. The fact that our design was to have independent machines is the reason we were able to minimize the impact of the problem. I highly recommend this kind of independence to any system that needs high availability.</p><h2>.14</h2><h4>May 2015</h4><h3>Introducing Several DNS Providers</h3><blockquote>We decided to use NSONE as a DNS provider because of their great DNS API.</blockquote><p>We were able to configure how we wanted to route queries for each of our users via their API.</p><p>We also really liked how they supported edns-client-subnets because it was key to having better accuracy for geo-routing. These features all made NSONE a great provider, but we couldn’t put ourselves at risk by having only one provider in our architecture.</p><p>The challenge was to introduce a second DNS provider without losing all the powerful features of NSONE. This meant having two DNS providers on the same domain was not an option.</p><p>We decided to use our retry strategy in our API clients to introduce the second DNS provider. All API clients first try to contact <a href="http://appid-dsn.algolia.net/">APPID-dsn.algolia.net</a> and if there are any problems, they would retry on a different domain, TLD, and provider. We decided to use AWS Route 53 as our second provider. If there were any problems, the API client would retry randomly on <a href="http://appid-1.algolianet.com/">APPID-1.algolianet.com</a>, <a href="http://appid-2.algolianet.com/">APPID-2.algolianet.com</a> and <a href="http://appid-3.algolianet.com/">APPID-3.algolianet.com</a> that targets the 3 machines of the main cluster. This approach allowed us to keep all interesting geo-routing features of NSONE on algolia.net domain while introducing a second provider for a better high availability on algolianet.com domain.</p><h2>.15</h2><h4>July 2015</h4><h3>Three Completely Independent Providers Per Cluster</h3><blockquote>You can now consider that with the infrastructure we developed, we are now fully resilient to any problem… but the reality is different.</blockquote><p>Even with services hosted by a Cloud provider with multiple Availability Zones you can have outages. We see two big reasons:</p><ul><li>Link/Router starts producing packet loss. We have seen this multiple times between US-East and US-West, including a border ISP router of a big cloud provider while they were not even aware of it.</li><li>Route leaks. This particularly impacts the big players that a large portion of the Internet relies on.</li></ul><p>Our improved setup in US now allows us to build clusters spanning multiple datacenters, multiple Autonomous systems and multiple upstream providers. That said, in order to accept indexing operations, we need to have a majority of machines up, which means two machines out of three. When using two providers, If one of our providers goes down, we could potentially lose the indexing service although search would still be available. That’s why we decided to go even further with the hosting of clusters across three completely independent providers (different datacenters in locations close to each other that rely on different upstream providers and Autonomous systems). This setup allowed us to provide high availability of search and indexing via an ultra redundant infrastructure. And we provide all of that along with the same easy to use API!</p><h3>Building A Highly Available Architecture Takes Time</h3><p>I hope the details of our journey were inspiring and provided useful details about how we started and where we are today. If you are a startup, don’t be worried if you don’t have the perfect infrastructure in the beginning. This is to be expected! But you should think about your architecture and how to scale it early on. I would even recommend doing it before beta!</p><p>Having the <strong>architecture designed early was our secret weapon</strong>. It allowed us to focus on execution as soon as our market fit was there. Even today, we have some features on scalability/availability that were designed very early on that are still not yet implemented. But they will be coming in the near future for sure :)</p><p><em>This story was originally published on </em><a href="http://highscalability.com/blog/2015/7/13/algolias-fury-road-to-a-worldwide-api.html"><em>highscabability.com</em></a><em> under three separate articles.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1536c46f3a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/algolia-stories/algolia-s-fury-road-to-a-worldwide-api-c1536c46f3a5">Algolia’s Fury Road to a Worldwide API</a> was originally published in <a href="https://medium.com/algolia-stories">Algolia Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>