<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Damian Schenkelman on Medium]]></title>
        <description><![CDATA[Stories by Damian Schenkelman on Medium]]></description>
        <link>https://medium.com/@dschenkelman?source=rss-17ad8dff4aaa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*YW2j1niU0-vNwRyn.jpeg</url>
            <title>Stories by Damian Schenkelman on Medium</title>
            <link>https://medium.com/@dschenkelman?source=rss-17ad8dff4aaa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:06:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dschenkelman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Parallelizing and auto-scaling bcrypt]]></title>
            <link>https://medium.com/auth0-engineering/bcrypt-as-a-service-9e71707bda47?source=rss-17ad8dff4aaa------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e71707bda47</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Damian Schenkelman]]></dc:creator>
            <pubDate>Tue, 20 Sep 2016 16:39:50 GMT</pubDate>
            <atom:updated>2016-09-23T14:05:30.144Z</atom:updated>
            <content:encoded><![CDATA[<p>How we greatly scaled our ability to perform login transactions</p><h4>tl;dr</h4><p>Keeping passwords safe involves using hash functions that are CPU-intensive and <a href="http://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt">intentionally slow</a>. This presents an interesting scaling challenge when having to handle thousands of login tx/sec.</p><p>Moving to an automatically scalable, distributed worker pool model allowed us to avoid bottlenecks, save money and not have to worry about variable load.</p><h3>In the beginning</h3><p>One of the features we have had for a <a href="https://auth0.com/blog/2013/06/04/introducing-db-connections/">really long time</a> is what we call <a href="https://auth0.com/docs/connections/database">Database Connections</a>. This means that instead of using Google, Facebook or an Active Directory instance as an identity provider, we store the emails/usernames and password hashes for users and they log in using those credentials.</p><p>This is one of our most used features. About 70% of users come from Database Connections. Making sure involved components are constantly optimized is always top of mind.</p><p>As our customer base started growing, performance was one of the things we had to pay more attention to. The more customers you have, the more varied their use cases and the higher load you get. Making sure we could handle that load became one of our top priorities.</p><p>In the extreme, bad performance can be interpreted as downtime.</p><h3>Performance tests</h3><p>We usually run many instances of each of our services in our cloud deployments. For the performance tests, we had decided to run with a single instance of each service to find the most elemental bottlenecks first.</p><p>When performing a test like this, an expected distribution for the response times usually:</p><ul><li>Has a minimum greater than 0. No request takes 0 ms</li><li>Has relatively similar amounts for the lower set of response times</li><li>Has a constantly decreasing tail; some requests just take a bit longer.</li></ul><p>A good example could look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PE2jlXncztnJQ0sr0IF0gA.png" /><figcaption>Example response time distribution</figcaption></figure><p>We set up a separate environment for the tests, created a couple of JMeter scripts to simulate users logging in and got this as the response time distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*dQASI2y9tJz3Pe-IH1FryA.png" /><figcaption>Real response time distribution</figcaption></figure><p>As you can see, the shapes of the curves are quite different. In reality, the majority of the requests were taking a very long time to complete. This pointed to a clear bottleneck.</p><h3>Enter bcrypt</h3><p>Conceptually, our setup looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmeJjbSs3FDrLTNGysh8Dw.png" /></figure><ol><li>A client performs a request to authenticate against our Auth API.</li><li>Auth0 verifies the user&#39;s credentials (active authentication)</li><li>Eventually the client gets a token as a response.</li></ol><p><strong>Note: </strong>The database and caches are queried while the above is taking place.</p><p>The first thing you hope for in situations like these was that we were just missing an index and adding it would make everything better, but it just wasn&#39;t the case.</p><p>Our next hypothesis was that a (or set of) CPU-intensive task was blocking the event loop and causing requests to be queued. Instead of going through our entire code base, we used <a href="http://www.brendangregg.com/flamegraphs.html">flame graphs</a> to try to find the cause of the issue. Fortunately, we got something that looked like the one in the following figure when profiling Auth0:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWD3v88Lpc3YE1cYivoJIA.png" /><figcaption>Example flame graph</figcaption></figure><p>Most of our time was being spent on bcrypt. Bcrypt is the algorithm we use for hashing passwords. It is both <a href="http://crypto.stackexchange.com/questions/400/why-cant-one-implement-bcrypt-in-cuda">memory- and CPU-intensive</a>, intentionally slow, and the <a href="http://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt">number of iterations it performs can be configured</a> to adjust for faster cores in the future.</p><p>All of that is great from a security perspective, but a scaling and performance nightmare.</p><h3>The alternatives</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Oplo6Arkh40dI3jopV9KQ.jpeg" /><figcaption><a href="https://en.wikipedia.org/wiki/King_(playing_card)#/media/File:King_playing_cards.jpg">“King playing cards</a>” by <a href="https://en.wikipedia.org/wiki/User:Enochlau">Enoch Lau</a> is licensed under <a href="http://creativecommons.org/licenses/by-sa/3.0/">CC BY-SA 3.0</a></figcaption></figure><ol><li><strong>Faster hash</strong>: This is a simple alternative (for new users) but was out of the question for us. We like the security traits of bcrypt and were not willing to give them up.</li><li><strong>Caching</strong>: This is does not fix the issue. We are trying to scale real logins, people don&#39;t login more than once in a while and there&#39;s no temporal locality. Caching would not save us this time.</li><li><strong>Scaling up</strong>: This was definitely the way to go for a quick win. You get more cores, have more processes running and you are good. The problem is that running in a cloud environment, you are going to hit a ceiling at some point.</li><li><strong>Scaling out</strong>: That&#39;s what you want in general, but what VMs do you use? If you have single-core VMs, you are not wasting any $, but every time one authentication request comes in, all other requests are stalled; that&#39;s not good :( If you get VMs with more cores and use <a href="https://github.com/ncb000gt/node.bcrypt.js#why-is-async-mode-recommended-over-sync-mode"><em>async mode</em></a><em> </em>(which is basically <a href="https://github.com/ncb000gt/node.bcrypt.js/blob/master/src/bcrypt_node.cc#L249">using</a> <a href="https://github.com/nodejs/nan/blob/master/doc/asyncworker.md">libuv&#39;s thread pool</a>), you will have idle cores during periods when authentication requests are not coming in. This looks like a step in the right direction, but more atomicity seems to be better.</li></ol><h3>Introducing BaaS</h3><p>We liked a lot of the benefits that solution #4 brought to the table, but wanted more control over the bcrypt operations. That&#39;s how we came up with <a href="https://github.com/auth0/node-baas">BaaS</a>.</p><p>We created a TCP service that uses <a href="https://developers.google.com/protocol-buffers/">Protocol Buffer</a> to handle two types of messages:</p><ul><li>Comparing a password to a hash</li><li>Hashing a password</li></ul><p>Each BaaS instance is like a worker in a distributed thread pool (kind of &quot;distributing&quot; <em>async mode</em>). The trivially parallelizable nature of the aforementioned operations is an ideal fit for this model.</p><p>Auth0 nodes connect to a load balancer (actually 2 for HA purposes) and they distribute the load between the different BaaS instances.</p><p><strong><em>Added 2016–09–21</em></strong></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p><strong>Note: </strong>the traffic between the Auth0 node and the BaaS nodes is encrypted. using TLS.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A09rkbRvs7co9DIq9E6uwA.png" /><figcaption>High level BaaS setup</figcaption></figure><p>Since the characterizing metrics of the BaaS service are very specific (e.g., requests/sec, CPU usage), we can do more granular capacity planning and implement automatic scaling to handle variable levels of load.</p><h4>Failing gracefully</h4><p>Whenever a dependency with an external service is introduced, the possibility of failure has to be handled gracefully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*5BQIHpACY18xJqKNcbIrvg.gif" /><figcaption>An excellent example of failing gracefully</figcaption></figure><p>There are different nonexclusive options, but their end goal is to prevent degradation of the users&#39; experience:</p><ol><li>Retrying the operation against the same or different destination</li><li>Falling back to an alternative implementation</li><li>Returning an error</li></ol><p>In our case we went with option #2. If a failure occurs when trying to call BaaS, we perform the hash calculation locally. This is not ideal, but gives time to our team to figure out what is going on and get a fix in place.</p><h3>Nowadays</h3><p>We are really happy with the outcome; BaaS has been running smoothly in production for some time now.</p><p>Since this is a critical component in our infrastructure, we frequently look for improvement opportunities. For example, we want to give <a href="https://avro.apache.org/">Avro</a> a try, as we have found some perf improvements when doing some tests internally.</p><p>The explanation of BaaS provided in this post is a simplified version. You can keep up with the latest changes in the <a href="https://github.com/auth0/node-baas">github repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e71707bda47" width="1" height="1" alt=""><hr><p><a href="https://medium.com/auth0-engineering/bcrypt-as-a-service-9e71707bda47">Parallelizing and auto-scaling bcrypt</a> was originally published in <a href="https://medium.com/auth0-engineering">Auth0 Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>