<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vedran Vučetić on Medium]]></title>
        <description><![CDATA[Stories by Vedran Vučetić on Medium]]></description>
        <link>https://medium.com/@vedranvucetic?source=rss-81c7381e004a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YaQ6f3FVYTueehXLm8ut2Q@2x.jpeg</url>
            <title>Stories by Vedran Vučetić on Medium</title>
            <link>https://medium.com/@vedranvucetic?source=rss-81c7381e004a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:35:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vedranvucetic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Set up global rate limiting with AWS WAF in 5 minutes]]></title>
            <link>https://faun.pub/set-up-global-rate-limiting-with-aws-waf-in-5-minutes-bd43a9309683?source=rss-81c7381e004a------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd43a9309683</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Vedran Vučetić]]></dc:creator>
            <pubDate>Sun, 31 May 2020 21:50:45 GMT</pubDate>
            <atom:updated>2020-06-01T18:06:19.793Z</atom:updated>
            <content:encoded><![CDATA[<p>Probably every service/website needs request rate limiting, sooner or later. Especially if generating content (making responses to those requests) requires compute time (not served from cache easily). Setting up a request rate limiting is not hard with HAProxy or NGINX or any other proxy/load balancer. But, when it comes to <strong>global</strong> or <strong>distributed rate limiting</strong>, this task becomes a bit more challenging. Global rate limiting is when all load balancer or proxy nodes, which are performing rate limiting, are in sync about the global state of the rates and consistently apply to limit based on that.</p><p>In the most common setup, both NGINX and HAProxy keep internal statistics and metrics used by rate-limiting algorithms in a process’ memory. This means that as soon as there is more than 1 instance of proxy/LB accepting requests, rate limiting will not behave like expected since each instance has its counters. Given the fact that, for reliability reasons, you should have at least 2 instances of everything, this already becomes a challenge. I’ll shortly describe global rate limiting with HAProxy, NGINX, and Envoy for completeness of this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*klT1YtqFwYkpuZXjYSYfxg.png" /><figcaption>Rate limiting with inconsistent state across all rate-limiting instances</figcaption></figure><p><strong>HAProxy, </strong>like all proxies/load balancers listed here, has great support for rate limiting, but I’m gonna only focus on global rate limiting. To overcome the challenge of global rate limiting, HAProxy has peer communication for exchanging stick table values. Stick table is key-value storage used for storing different metrics and is a source for rate limiting. HAProxy can exchange keys stored in stick tables with many other peers. But, if other peers contain the same key in their stick table, it will get replaced on sync. To have a consistent rate limiting, we would need something which HAProxy calls <strong>Stick Table Aggregator.</strong> Stick Table Aggregator does exactly what’s needed. It aggregates metrics from all peers including own and applies rate limiting based on that. This feature is available only in HAProxy Enterprise edition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*hww6bDcSl1AZV2Md05yPBw.png" /><figcaption>Rate limiting with eventual consistency and aggregator between peers</figcaption></figure><p><strong>NGINX</strong> supports global rate-limiting in NGINX Pro version as well using similar peer/mesh communication which they call <a href="https://www.nginx.com/blog/nginx-plus-r16-released/#r16-cluster-rate-limiting">zone sync</a>. Alternatively, there are open source implementations of global rate-limiting using Lua scripting backed by Redis server — <a href="https://leandromoreira.com/2019/01/25/how-to-build-a-distributed-throttling-system-with-nginx-lua-redis/">this</a> or <a href="https://github.com/brandur/redis-cell">this</a> or <a href="https://github.com/timebug/lua-resty-redis-ratelimit">this</a>. This can potentially look cheaper, but it’s good to think in the long run of maintaining as well as reliability and ask questions like what would happen with service if Redis is not available, how would we scale that solution or what are the performance limits of such setup?</p><p><strong>Envoy</strong> proxy has a service interface for rate limiting. Lyft made the <a href="https://github.com/envoyproxy/ratelimit">service</a> that implements that interface. This service is also backed by Redis to keep stats synced across all instances of envoy.</p><h3>Setting up the global rate limiting with AWS Web Application Firewall (WAF)</h3><p>If you use AWS Application Load Balancer (ALB) you have everything required to start and you can have it set up in 5 minutes.</p><p>Steps for creating rate-limiting with WAF:</p><ul><li>Create Web ACL in WAF</li><li>Create rate-limiting rule</li><li>Attach ACL to the existing load balancer (ALB)</li><li>(Optionally) Set alerts on a number of blocked requests</li></ul><p>This CloudFormation snippet creates web ACL with rate limit rule which will start blocking client that has more than 1000 requests in the 5 minutes from the single IP. <strong>5 minutes period</strong> is currently a fixed period and <strong>can not be changed</strong>. <strong>IP as aggregation key</strong> for rate limiting is also the <strong>only option for that property</strong> at the time of writing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3651d0b983d47d0a830970bc9bc53c8/href">https://medium.com/media/a3651d0b983d47d0a830970bc9bc53c8/href</a></iframe><p>Note that each rule can publish CloudWatch metrics which makes alerting on throttling very easy.</p><p>Before releasing in production, you can deploy your rules and for rule action use Count instead of Block. The count will increment CloudWatch metric like it was blocked but the request will go to the next rule in ACL without being blocked or allowed.</p><p>Rules can have conditions. Let’s say we want to apply different rate-limiting rules for different routes. For example, route foo can tolerate many requests but bar the route should be more aggressively throttled.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8ff9699273d5de59f436209233d1726/href">https://medium.com/media/e8ff9699273d5de59f436209233d1726/href</a></iframe><p>This ACL ensures that route bar can get no more than 100 requests in 5 minutes from single IP, while the route foo 500 requests in 5 minutes. If the client exceeds those thresholds, WAF will return 403 until the number of requests drops below-given thresholds.</p><p>Now, let’s imagine we want to rate limit bar with 100 requests, foo with 500 and everything else with 300. In this case, it’s important to understand the lifecycle of ACL or how ACL processes rules. ACL evaluates all rules until one of the rules ends with terminating action - Block or Allow. Let’s unwind that example.</p><p>Request with route /foo/test comes to load balancer. ACL runs rules for that request. Rule 0: doesn’t match because of the condition. Rule 1 matches condition, the request is counted but the rate is still below Limit so WAF continues running the next rule. In our example, here would come rate-limit-other with Limit 300. Since that last rule wouldn’t have a condition, every request that didn’t end with Block in previous rules would be counted in the final rule with Limit 300. This is not what we want so for this to work, we need a terminating action after each rate-limiting rule if we don’t want to be processed by other rules. The example would go something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70cded77e0e2232f2543777457770174/href">https://medium.com/media/70cded77e0e2232f2543777457770174/href</a></iframe><p>In pseudo-language this can be summarised like this:<br>1. block and stop processing all requests if the route starts with bar and over the limit of 100, otherwise, continue processing<br>2. if request route starts with bar then allow and stop processing<br>3. block and stop processing all requests if the route starts with foo and over the limit of 500, otherwise, continue processing<br>4. if request route starts with foo then allow and stop processing<br>5. block and stop processing if over limit of 300<br>6. allow (default action of ACL)</p><h3>Associating to ALB</h3><p>As a final step, ACL needs to be associated with ALB. This is the simple step that can be done through UI (like all this), however, here’s the CloudFormation step:</p><pre>WAFACLAssociation:<br>  Type: AWS::WAFv2::WebACLAssociation<br>  Properties:<br>   ResourceArn: !Ref &#39;LoadBalancerArn&#39;<br>   WebACLArn: !GetAtt &#39;RateLimitACL.Arn&#39;</pre><p>One ACL can be associated with many ALBs.</p><h3>Pros and cons</h3><p>Pros:</p><ul><li>By default global rate limiting. Your AWS ALB is always running at least 2 instances of load balancer so this rate limiting is most probably “eventually consistent”. This type of consistency is the most common for this problem and it’s often “good enough”. If you need transaction-like accuracy, this is probably not a good solution for you.</li><li>No additional service to manage</li><li>Easy to deploy — WAF can be deployed within minutes. If you’re under DoS and want to quickly set up rate limiting, this can be deployed in almost no time.</li><li>Can be applied to API Gateway, ALB or CloudFront</li><li>Runs at ALB level (prevents stressing your infrastructure when defending high throughput that needs to be rate-limited).</li><li>With WAF is easy to add exceptions or white lists which won’t be rate limited. It’s just another rule which will allow request and stop processing if request satisfies some conditions before rate limiting is applied.</li><li>WAF is not just for rate limiting, we can add rules to detect SQL injection or other types of attack and block those requests. We can add geo-blocking or just simple IP blacklists filled manually or from other systems (Lambda analyzing request logs for example).</li></ul><p>Cons:</p><ul><li>Not flexible enough for just any case. At the time of writing, it only supports rate limiting per IP for a fixed time window of 5 minutes</li><li>The minimum threshold for rate limiting is 100</li></ul><p>If you don’t need high flexibility on rate-limiting aggregation key (IP in this key) or time window, this can be great protection from unwanted or aggressive clients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬<strong>.</strong></p><p><strong>To join our community Slack team chat </strong>🗣️ <strong>read our weekly Faun topics </strong>🗞️,<strong> and connect with the community </strong>📣<strong> click here⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd43a9309683" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/set-up-global-rate-limiting-with-aws-waf-in-5-minutes-bd43a9309683">Set up global rate limiting with AWS WAF in 5 minutes</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High priority request queue with HAProxy]]></title>
            <link>https://medium.com/swlh/high-priority-request-queue-with-haproxy-9efd639a8992?source=rss-81c7381e004a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9efd639a8992</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vedran Vučetić]]></dc:creator>
            <pubDate>Sun, 24 May 2020 19:09:39 GMT</pubDate>
            <atom:updated>2020-06-01T13:53:22.189Z</atom:updated>
            <content:encoded><![CDATA[<p>Load balancers help disperse traffic across a fleet of servers using different algorithms. Often we use load balancers to protect our servers from being overwhelmed by limiting the number of concurrent requests to each server. Load balancers are built to manage a huge number of connections while backend servers might not be that capable which can lead to degradation and even downtime. When there are more requests than the server can take, requests queue starts to form up. In case of a surge, some requests can wait in the queue longer than usual and even timeout with 503 while waiting to be processed. But, what if we want to <strong>prioritize some requests</strong> so that they don’t wait in the same queue with all other requests?</p><h3>The use case for request prioritization</h3><p>In our case, we have a CPU-bound service. Unlike I/O-bound services, like APIs reading from some external DB, CPU-bound services are almost entirely limited to CPU capacity of the machine which hosts service. Service like that can serve in parallel as many requests as there are CPU cores. Technically, it can process more than that in parallel, but requests will take longer since there is just no available CPU time. If the server hosting this kind of service has 2 CPU cores, we can limit to 2 parallel requests in the load balancer. This will form the queue as soon as more than 2 requests are being processed at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*zHpos_m8kQW0lNu4imCefw.png" /><figcaption>Current setup with ALB and HAProxy</figcaption></figure><p>Next, we have AWS ALB accepting all requests from the outside. Unfortunately, ALB doesn’t support request queueing so it just routes all requests to servers (targets). At each server we have HAProxy accepting connections, balancing requests across workers in the same server (exact count of workers depends on the number of cores) and making queue if there are more requests than we can serve in parallel. In case of request surge, request queue forms up at HAProxy level. In that same queue there are health check requests from ALB, waiting to be processed (<em>ALB periodically sends health check requests to ensure targets are healthy</em>). If service doesn’t respond to those health checks on time, after several tries, the instance will be marked as unhealthy. An unhealthy instance will be selected for recreation by Autoscaling Group (ASG). One instance less in rotation means more load during surge for other instances. Following this principle, in a short time, all instances of such service can be kicked out of rotation, scheduled for recreation which will left region with 0 instances serving traffic (until ASG provisions new instances).</p><h3>Request serving priority comes to help</h3><p>We solved this challenge by adding higher priority for the processing of health checks. This means that <strong>regular traffic can wait in the queue while we process all health checks</strong> as soon as they arrive. That makes sure that instance responds to health checks even during request surge by going around of all other requests. Instead of kicking instance out of rotation, by not responding to health checks, the request spike will just cause some requests to timeout (in the worst-case scenario).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*obQfiggNbUjAkYDeWzLACg.png" /><figcaption>HAProxy with 2 incoming queues</figcaption></figure><p>For this purpose, we use 2 HAProxy backends, regular backend and one for the health checks. In the frontend section we select backend for each request like in this example:</p><pre>use_backend backend_1 if req_regular<br>use_backend backend_1_health_check if req_health_check</pre><p>We defined two backends like this (some settings are omitted for brevity), both routing to the same set of workers within 1 instance:</p><pre>backend backend_1<br> timeout queue 5s<br> default-server weight 1 maxconn 2</pre><pre>backend_1_health_check<br> timeout queue 20s<br> default-server weight 1 maxconn 10</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*8uRX77i3V60-EXDZyIL_lw.png" /><figcaption>Final configuration with 3 queues and request flow</figcaption></figure><p>Instead of increasing priority for health checks (because that’s not possible as such), we defined 2 queues. One for all traffic and one for health checks traffic. HAProxy will let requests from this queue downstream respecting <em>maxconn</em> to service which has another queue. This ensures that, in the worst-case scenario, we’ll always have a health check request 3rd in line to be processed (check chart above).</p><h3>Other use cases</h3><p>Although not many services out there are entirely CPU-bound with the low number of workers so that health checks need to be prioritized, this technique can be used for other use cases. By controlling <em>maxconn</em> in the backend section you can control the ratio of specific requests and implement “soft” throttling where some type of requests are determined to wait longer compared to others.</p><p>Some extra notes:</p><ul><li>low <em>maxconn </em>is <strong>not good for I/O-bound services</strong>. It’s always good to understand what kind of workload is being balanced and configure accordingly</li><li><strong>queue has some low overhead</strong> in terms of latency. In our case, we measured ≤ 1ms overhead which was completely acceptable</li><li><strong>nginx supports similar queueing mechanism</strong> but only in Plus version (<a href="https://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue">https://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue</a>)</li><li>prioritization/QoS can be also implemented by having <strong>separate backends for high and low priority requests</strong>. For example, a higher number of faster servers serving high priority requests and a smaller number of slower servers serving low priority requests. In that case, HAProxy should be placed in front of the fleet and not in an instance as in the example above.</li><li>keeping the queue at backend (HAProxy backend, just before services), helps with better utilization of workers in the node.</li><li>Another advantage of this setup, not covered here, is the <strong>ability to retry requests on other workers</strong> in case of failure of some sort (<a href="https://www.haproxy.com/blog/haproxy-layer-7-retries-and-chaos-engineering/">https://www.haproxy.com/blog/haproxy-layer-7-retries-and-chaos-engineering/</a>).</li><li>Some other examples of priority queues — <a href="https://www.haproxy.com/blog/application-delivery-controller-and-ecommerce-websites/">https://www.haproxy.com/blog/application-delivery-controller-and-ecommerce-websites/</a></li></ul><h3>Update: Alternative method for request prioritization</h3><p>An alternative method, and probably preferred, to prioritize requests is to use a <strong><em>set-priority-class</em></strong> directive which is available from version 1.9 (Thanks to Daniel for pointing it out).</p><p>Usage:</p><pre><a href="https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.2-http-request%20set-priority-class"><strong>http-request set-priority-class</strong></a> &lt;expr&gt; [ { if | unless } &lt;condition&gt; ]</pre><blockquote>This is used to set the queue priority class of the current request.<br>The value must be a sample expression which converts to an integer in the<br>range -2047..2047. Results outside this range will be truncated.<br>The priority class determines the order in which queued requests are<br>processed. Lower values have higher priority.</blockquote><p>Using this directive, the solution to prioritizing health checks would like something like this:</p><pre>http-request set-priority-class int(1) if req_health_check<br>http-request set-priority-class int(10) if req_regular</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9efd639a8992" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/high-priority-request-queue-with-haproxy-9efd639a8992">High priority request queue with HAProxy</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Autoscaling CPU-bound API service by request queueing time]]></title>
            <link>https://medium.com/@vedranvucetic/autoscaling-cpu-bound-api-service-by-request-queueing-time-4a35e31f692b?source=rss-81c7381e004a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a35e31f692b</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vedran Vučetić]]></dc:creator>
            <pubDate>Wed, 11 Sep 2019 07:45:33 GMT</pubDate>
            <atom:updated>2020-05-31T22:17:54.132Z</atom:updated>
            <content:encoded><![CDATA[<p>We are building a product search/autocomplete/autosuggest engine. This article is about making auto-scalable infrastructure for it. I’m gonna try to stay away from implementation and underlying technology and just focus on the concept.</p><p>We have very strong requirements on latency since it’s proven that search/autocomplete/autosuggest speed directly impacts conversion rate. In order to achieve that, we’re building our own in-memory structures and AI models and exposing that as a web API. Almost all requests for this service are <strong>CPU-bound</strong>, all needed for processing request is already in <strong>RAM (cache) memory</strong>. The rest is a job for CPU to run algorithms to serve the best results possible. You can already see that this setup allows us to scale out, basically, indefinitely (limited to cloud infrastructure bellow), without hitting some bottleneck like one by some database.</p><p>Being CPU-bound means there is no sense to run our service multi-threaded. It also means we have to correlate the number of workers (processes serving requests and doing all the work) per box to the number of physical cores. Why? Well, since there is very little IO wait during the request (only in some special cases), CPU is fully utilized while serving one request. Adding more workers than CPU cores would require more CPU context switching without any gain to performance. We could even get penalized for that.</p><h3>Autoscaling by latency</h3><p>Our initial setup was to use latency to scale out by. It made a lot of sense. We care about p95, p99 latencies and first stab was to scale by these metrics. If we saw that our p95 latency went over <em>x,</em> we were adding more instances. This served us well for a long period of time but at one moment we on-boarded customer whose dataset was a couple of orders of magnitude bigger than all other customers’ datasets. Naturally, bigger datasets mean we need more time to serve results. The customer was still very happy. We’ve been doing much better than other competitors they tried. However, our autoscaling was unhappy. Since requests for that customer were taking longer than what was our goal for all other customers, autoscaling was trying to compensate by adding more instances until going to the max and staying there. Of course, adding more instances didn’t help. <strong>Requests were taking the same amount of time on barely utilized servers because one request was limited to one core</strong>. To overcome that, we could adjust scaling policy numbers, but with every new customer that could impact overall latency statistics and we would need to adjust policy. So, we decided to rethink auto-scaling metrics and policies.</p><h3>Autoscaling by CPU utilization</h3><p>Scaling out by <strong>CPU utilization</strong> makes a lot of sense for CPU-bound service. However, we haven’t decided to go that route. The main reason is this: CPU utilization didn’t say much about how we’re actually doing. Do we need to scale out at 30% CPU, 50% or 80% to get the performance that we want? We’d need to empirically find some numbers for CPU usage that give us the desired output and then hope that our CPU “utilization profile” wouldn’t change over time, messing up autoscaling again.</p><h3>Autoscaling by queueing time</h3><p>We had to answer the following question: what’s the best way to know that we need to add more of these instances? And the answer is: if the queue of requests is starting to build up, we should add more instances. If the queue is low, scale out wouldn’t help anyway. As a result, we introduced a new metric and based our autoscaling on that — <strong>queueing time</strong>. Queueing time is duration spent on waiting for available workers or <strong>time since a request was accepted until just before processing by a worker</strong>. We had to pull off some tricks to make it, but the main idea is very simple. If requests are waiting to be processed longer than what we’re ready to sacrifice, we have to add more instances to load balancing. By adding more instances, we’re directly addressing the problem of increased queueing time, unlike with increased latency which can be caused by latencies impacted by bigger datasets that can not be compensated with more instances by itself. The assumption is workers are doing their best with processing each request and we’re only scaling out if average queueing time is increased.</p><figure><img alt="Queueing time observed my monitoring system" src="https://cdn-images-1.medium.com/max/1024/1*FAgCt1P_4NoFIstpeRh6NQ.png" /><figcaption>Queueing time observed by monitoring system</figcaption></figure><p>Queueing time showed to be extremely useful for more than just autoscaling. Since queueing time in our case is &lt;10ms, this metric is very sensitive. For example, increased memory usage is always manifested in increased queueing time because of how we work with memory structures. Increased RSS memory pushes out memory structures from cache memory causing more page faults which are then observed in increased queueing time. We can spot increased memory in queueing time before memory alert warns us. Of course, we watch for the request rate to make sure it’s not due to traffic. Also, the increased CPU usage of sidecar service (not one we’re monitoring queueing time for) manifests in increased queueing time because of the “noisy neighbor” effect. Today, this is one of the most important metrics when profiling a version just before deploy as well as comparing canary instances. If our stability test notices increased queueing time, the release would be automatically stopped and we can be confident that there is a problem somewhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_grVY2YqmvbsyADHS_1aUA.png" /><figcaption>An increased queueing time which was resolved at the end by adding more instances</figcaption></figure><h3>Is this generally applicable for autoscaling?</h3><p>No. It’s not. It would be good to monitor anyway, but it’s definitely not for just any service type. This is probably applicable only to CPU-bound services. For services that are not entirely CPU-bound and have event loops like Node.js, this makes no sense. Worker, in our case, is working on just one request at the time because of what I described earlier. Event-loop based services would probably accept many more requests and switch contexts whenever blocked by IO (reading files, requesting external services, etc). In this setup, queueing time would be very low so scaling by queueing time wouldn’t be the right choice. Understanding service architecture is required before deciding how to set up scaling policies. It’s important to understand at which part of the request pipeline queue starts to make up and scale out just that.</p><p>Although I described implementing queueing time as a scaling metric for API service, we, as well, introduced that metric for <strong>queue-based background workers</strong> which do some heavy lifting jobs. Instead of scaling out workers by a number of tasks in queue, we’re now scaling out by time spent waiting in the queue because that directly reflects some SLOs we want to meet. At the same time, we monitor the total time needed to be processed, which includes queueing time, just to be sure how the overall picture looks. This gives us the best information if we need to scale out or not, regardless of tasks’ individual duration time and their count in a queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7yIvFtHVRCC7qpEZ7gztQ.png" /><figcaption>Total task duration next to task queueing time</figcaption></figure><p>This is a very simple concept, but I wanted to share our experience with it as a potential inspiration for how to solve auto-scaling problem with CPU-bound services. In our case, queueing time showed to be the best metric to scale out by, because it’s directly tight to our latency SLO. Let me know your thoughts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a35e31f692b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run only last triggered job for branch in descriptive Jenkins pipeline]]></title>
            <link>https://medium.com/@vedranvucetic/run-only-last-triggered-job-for-branch-in-descriptive-jenkins-pipeline-a07bb84ae075?source=rss-81c7381e004a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a07bb84ae075</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[jenkins-pipeline]]></category>
            <dc:creator><![CDATA[Vedran Vučetić]]></dc:creator>
            <pubDate>Thu, 14 Mar 2019 20:06:01 GMT</pubDate>
            <atom:updated>2020-05-31T22:11:51.029Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*wkr7Ss3DUUl3KrnCNTTitw.jpeg" /></figure><p>I had been searching for a solution to stop existing jobs for pull requests (PR) in the <strong>descriptive pipeline</strong> (<strong>Jenkins Pipeline Multibranch</strong>) on a push of new commits (a newly triggered job for the branch) and barely found one so I decided to put it in form of post to help someone, but also for my own future reference.</p><p>You can skip to the final solution immediately if you want, but I’m gonna make a long story because I learned more along the way than just copying solution from StackOverflow and want to share findings with readers.</p><h3>Current setup</h3><p>At <a href="https://constructor.io/">constructor.io</a>, we have Jenkins which runs tests (integration tests, unit tests) for each PR in BlueOcean. Tests are executed in agents. Agents are spun up when needed. Each agent has 3 executors to ensure tests are not competing for CPU too much (number of executors is balanced by an agent’s instance size, CPU cores to be precise). When PRs are waiting too long in the queue, Jenkins starts a new agent instance. PRs are executed only on agents, not on the master to ensure master has executors for other jobs, like deployments, rollbacks, recurring jobs, etc. When the agent is idle for some time, Jenkins terminates it.</p><h3>Problem identification</h3><p>Some tests can take up to 15 minutes to finish. Engineers sometimes make a git push, then in the middle of tests execution, notice smaller fix and make another push. In regular setup, this will make Jenkins run two jobs in parallel, one for older and one for the newer push. Ideally, when there is a new commit, we’re no longer interested in the old one. We just care about test results for the last commit of the PR. So we want to somehow stop or limit the number of executions per PR.</p><h3>Locks, milestones</h3><p>I first looked if this is possible out of the box, but found that it’s not. It is possible with <a href="https://github.com/jenkinsci/ghprb-plugin">ghprb-plugin</a> but I’m not using it with pipelines. Some answers suggested using locks and milestones. So here is what is possible with locks and milestones.</p><p><strong>Locks</strong> allow you to <strong>throttle the number of executions</strong> for some resource making sure only the x number of jobs are running. Everything else needs to wait in the queue. So we can use a lock and ensure only 1 build is running per PR, right? Yes, but! That means that we would be only throttling concurrent execution and all other commits for that PR will be waiting in the queue.</p><pre>lock(inversePrecedence: true, resource: &quot;${currentBuild.fullProjectName}&quot;, variable: &#39;lock_resource&#39;)</pre><p>Here comes the <strong>milestone</strong> to help. <strong>Milestone is like a gate that doesn’t let build go trough if newer build already passed it</strong>. Imagine you have a pipeline which deploys to environment after running test. Build #2 can finish before build #1 (for various reasons but let’s say build #2 was running on an agent that has faster CPUs or was less loaded). If #2 was deployed to the environment, we don’t want the older build to replace it. Milestone drops all older jobs than one that last passed it.</p><pre>milestone ordinal: Integer.parseInt(env.BUILD_ID)</pre><p>Note that lock has inversePrecedence which tells lock to let jobs in inverse order, not FIFO but LIFO. Combining that lock with a milestone like this:</p><pre>milestone ordinal: Integer.parseInt(env.BUILD_ID)<br>lock(inversePrecedence: true, resource: &quot;${currentBuild.fullProjectName}&quot;, variable: &#39;lock_resource&#39;)<br>milestone ordinal: Integer.parseInt(env.BUILD_ID)</pre><p>will make sure that only 1 “push” is running per PR and that all older pushes get dropped.</p><p>Imagine Jenkins running build #1. Engineers make 3 additional pushes to brach, issuing like that build #2, #3, #4, all while build #1 was still running. Jenkins will block all new jobs until #1 is done. Then it will let #4 running. #2 and #3 will still wait in the queue. Once #4 is done, #2 and #3 will be unlocked and dropped because they’re older than #4.</p><p>Why this didn’t work for us? For a very simple reason. While this solution is not too bad, the biggest problem is jobs are actually running and taking up the executor slots. They’re blocked by lock and not performing work, but still, they’re running from Jenkins’ perspective and taking up the executor. The agent can’t take additional work until this dropped which requires other jobs to finish. So for that reason, I gave up this solution.</p><h3>Final solution</h3><p>I found <a href="https://stackoverflow.com/a/48956042/818899">this</a> reply on StackOverflow. Credits to the author.</p><pre>/* This method should be added to your Jenkinsfile and called at the very beginning of the build*/<br>@NonCPS<br>def cancelPreviousBuilds() {<br>    def jobName = env.JOB_NAME<br>    def buildNumber = env.BUILD_NUMBER.toInteger()<br>    /* Get job name */<br>    def currentJob = Jenkins.instance.getItemByFullName(jobName)<br><br>    /* Iterating over the builds for specific job */<br>    for (def build : currentJob.builds) {<br>        /* If there is a build that is currently running and it&#39;s not current build */<br>        if (build.isBuilding() &amp;&amp; build.number.toInteger() != buildNumber) {<br>            /* Than stopping it */<br>            build.doStop()<br>        }<br>    }<br>}</pre><p>This can be placed at the beginning of the Jenkinsfile and called from the first step in the pipeline. I slightly modified this solution to include the reason for the abortion of the job. The final solution looks something along these lines</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fdfee67073e94e7411532ca3a3aa45b/href">https://medium.com/media/1fdfee67073e94e7411532ca3a3aa45b/href</a></iframe><p>Note that init phase, which stops other jobs, is running in master while test in agent. The reason for that is the following. Imagine you have agent running with 3 executors and all are taken. Someone pushes new commits for one of the PRs running. This job will be waiting in the queue, unable to stop a previously running job because there are no executors. To avoid that, we run this step on master making sure that stopping won’t wait for executor slot on agent.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a07bb84ae075" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>