<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Or Kaplan on Medium]]></title>
        <description><![CDATA[Stories by Or Kaplan on Medium]]></description>
        <link>https://medium.com/@kaplanor?source=rss-a044864415ef------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hLuNpe_leFE-86YcFAzsLg.jpeg</url>
            <title>Stories by Or Kaplan on Medium</title>
            <link>https://medium.com/@kaplanor?source=rss-a044864415ef------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:36:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaplanor/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to expose the right Prometheus metrics for your custom exporter implementation (Open Source)]]></title>
            <link>https://medium.com/rewire-to/how-to-expose-the-right-prometheus-metrics-for-your-custom-exporter-implementation-open-source-162815adc1b0?source=rss-a044864415ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/162815adc1b0</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[cloudflare-workers]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Or Kaplan]]></dc:creator>
            <pubDate>Tue, 27 Dec 2022 08:36:15 GMT</pubDate>
            <atom:updated>2022-12-27T08:36:15.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wWX78Iv5mMfrj1mVonNM_w.jpeg" /></figure><p><em>If you want to build a Prometheus Exporter on your own, this is the guide for you. From selecting the metrics to calculating and exporting stats using the right method, all while building an exporter for Cloudflare Workers in order to make sure your services run as expected.</em></p><p><em>The previous </em><a href="https://medium.com/rewire-to/sometimes-building-your-own-prometheus-exporter-for-monitoring-cloudflare-workers-is-the-best-4690531401d"><em>blog post</em></a><em> in this series explored our reasoning for implementing a Prometheus exporter for Cloudflare Workers. Now, we would like to focus on the metrics themselves. When monitoring extracting the right metrics can make or break your capability to monitor your services. Thus, in this post we will focus on selecting, building and serving the right metrics, so you’ll be able to build relevant dashboards and alerts to keep your business running (while giving a concrete example of our full monitoring suite).</em></p><h3>Building the Exporter</h3><p>First and foremost, it’s important to state that not everyone has to build their own Prometheus Exporter. But, if you decide to do so, there are some steps to consider. In this post, you’ll see how we, at Rewire, built our own Prometheus exporter which allows us to monitor the health of our functions running on top of Cloudflare Workers.</p><p>At Rewire, we chose to build our own exporter in Go, due to the small binary size and the prevalence of the ecosystem. But first of all, we faced a few challenges:</p><ol><li>Scrape the metrics from Cloudflare Analytics — Write a code that queries the GraphQL analytics endpoint exposed by Cloudflare</li><li>Building the metrics — Converting the aggregative metrics of Cloudflare into raw Prometheus metrics, and selecting the right labels</li><li>Putting it all together — deploying the exporter and making sure that all relevant metrics are being collected properly</li></ol><p>We then took a hard look at these challenges and translated them into action items. Check them out, they might be the steps you need to take in order to build your own exporter.</p><h3>Query the GraphQL endpoint</h3><p>This task was really straight forward, we had to make sure we have the right permissions and queries:</p><ol><li>Permissions — in order to query the analytics service we had to generate a bearer token with account analytics permissions</li><li>Queries — The maximal lookback window is 1 week, letting us pull metrics in a time window with worker and status labels.</li></ol><h3>Step #1 Getting the right metrics from Cloudflare Analytics</h3><p>In order to build our own exporter, we had to write a back to back service, which gets a request for the current metrics from the Prometheus scraper. The service had to scrape the metrics from the Cloudflare analytics API, convert it to Prometheus format, and submit the response back to Prometheus.</p><p>This task was really straightforward, as Cloudflare exposes the data using a <a href="https://developers.cloudflare.com/analytics/graphql-api">GraphQL API</a>. At this point, all we really had to do was make sure we have the right permissions and queries:</p><ol><li>Permissions — in order to query the analytics service, we had to generate a bearer token with <strong>account analytics</strong> permissions</li><li>Queries — as described in the previous post, using GraphQL we were able to scrape the request rate, error rate, and CPU usage of a specific worker instance. The maximal lookback window is one week, allowing us to pull metrics in a time window with worker and status labels</li></ol><h3>Step #2 Building the metrics</h3><p>Prometheus stores a set of timeserieses. Each time series consists of a name, a list of labels, time, and the sample value, which defines 3 types of metrics:</p><ol><li>Gauge — measured value, such as temperature</li><li>Counter — a monotonic increasing metric, such as count of visits on a website</li><li>Histogram and summary — used to calculate quantiles such as the latency of the 0.5 quantile</li></ol><p>At this point, we had to decide which metrics we want to read from the Analytics API, how to convert them into the relevant Prometheus metrics type, and how do we wisely choose the relevant labels for future query purposes.</p><h3>We managed to expose two types of metrics</h3><h4>The simpler metrics - CPU</h4><p>As a CPU metric is just a gauge metric, all we have to do is select the time period and pull the metric using a GraphQL query:</p><pre>query ($account: String!, $start: String!, $end: String!) {<br>	viewer {<br>		accounts(filter: {accountTag: $account}) {<br>			workersInvocationsAdaptive(limit: 100, filter: {datetime_geq: $start, datetime_leq: $end}) {<br>				quantiles {<br>					cpuTimeP25<br>					cpuTimeP50<br>					cpuTimeP75<br>					cpuTimeP90<br>					cpuTimeP99<br>					cpuTimeP999<br>				}<br>				dimensions {<br>					scriptName<br>					status<br>				}<br>			}<br>		}<br>	}<br>}</pre><p>Implementing it, we took a 1-minute timeframe. After getting the results, we converted the results into a gauge metric cloudflare_worker_cpu_time_percentile exposing the following labels:</p><ol><li>worker - the worker’s name</li><li>status - one of the aforementioned invocation statuses</li><li>percentile - the percentile of the measurement - 25, 50, 75, 90, 99, 99.9</li></ol><h4>The sophisticated metrics - Request Rate</h4><p>To measure requests and errors, we had to set a counter metric. In Prometheus, a counter is a monotonically increasing metric that represents a measurement of event count.</p><p>The counter definition made exporting this metric more complex. We then weren&#39;t able to use a constant 1-minute timeframe as it converts the metric from counter to rate. Meaning that each minute has a different number of events breaking the monotonic assumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*XjlwwPcEFVOaXJyK" /><figcaption>Sampling the metric value on fixed size window</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*hK2K2TcA1iGaFYcu" /><figcaption>Generating rate metric per minute</figcaption></figure><p>In order to make sure that the request rate holds the assumption of a counter metric, we had to set a fixed start time (anchor). We set the start time to the beginning of the week (e.g. Monday at midnight) and query the metrics in a time window from last Monday till the sample time (which is now).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*DlVxLiAmnBdLb2M5" /><figcaption>Sampling the metric value on a fixed start time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HlJ3eirgpzp4nI4r" /><figcaption>Generating counter metric with a resolution of the sampling frequency</figcaption></figure><p>The relevant query to pull requests, errors and subrequests since last Sunday:</p><pre>query ($account: String!, $start: String!, $end: String!) {<br>	viewer {<br>		accounts(filter: {accountTag: $account}) {<br>			workersInvocationsAdaptive(limit: 100, filter: {datetime_geq: $start, datetime_leq: $end}) {<br>				sum {<br>					subrequests<br>					requests<br>					errors<br>				}<br>				dimensions {<br>					scriptName<br>					status<br>				}<br>			}<br>		}<br>	}<br>}</pre><p>The relevant counter metrics are cloudflare_worker_requests_received_total, cloudflare_worker_errors_total, cloudflare_worker_subrequests_total</p><p>With the selected labels:</p><ol><li>worker - the worker’s name</li><li>status - one of the aforementioned invocation statuses</li></ol><h3>Step #3 Putting it all together</h3><p>Once the implementation of the exporter was completed, we had to install it on our Kubernetes. To do so, we first deployed the exporter to our cluster using the relevant configuration. Then, as we use the Prometheus operator, we had to install the relevant ServiceMonitor and rules, so Prometheus will be able to discover the exporter and scrape the metrics.</p><h3>Deploying the service</h3><p>In order to deploy the service, we had to create a Kubernetes deployment that runs the docker image of the exporter and passes the following environment variables:</p><ol><li>CLOUDFLARE_ANALYTICS_TOKEN - a bearer token with Account Analytics permissions</li><li>CLOUDFLARE_ACCOUNT_ID - our Cloudflare account ID, can be found on the main Cloudflare account dashboard</li></ol><h3>Service monitor</h3><p>If you use the Prometheus operator, you are probably familiar with ServiceMonitor, a CRD that defines a target that should be scraped by Prometheus. In our case, we defined the following YAML:</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: cloudflare-worker-exporter<br>  namespace: monitoring<br>spec:<br>  endpoints:<br>  - interval: 60s<br>    port: metrics<br>    scheme: http<br>  namespaceSelector:<br>      any: true<br>  selector:<br>    matchLabels:<br>      app: cloudflare-worker-exporter</pre><h3>Prometheus rules (alerts)</h3><p>A monitoring system is worth nothing without setting alters, which help detect failure and error spikes automatically. In order to set the required alerts we defined the following rules:</p><ol><li>High exception rate</li><li>CPU time exceeded 49ms on the 90th percentile (to avoid Cloudflare from killing the request handler)</li><li>Worker sessions that are completed with an exceededResources status, which means that we exceeded the allowed CPU</li></ol><pre>groups:<br>- name: cloudflare-worker<br>  rules:<br>  - alert: CloudflareWorkerHighCPU90P<br>    annotations:<br>      message: &quot;High cpu for {{ $labels.worker }} worker - {{ $value }} ms &gt; 49ms&quot;<br>    expr: |<br>      (sum by(worker) (cloudflare_worker_cpu_time_percentile{percentile=&quot;90&quot;}) / 1000) &gt; 49<br>    for: 5m<br>    labels:<br>      component: cloudflare-worker<br>      severity: critical<br>  - alert: CloudflareWorkerHighExceptionRate<br>    annotations:<br>      message: &quot;High exception {{ $labels.worker }} worker - {{ $value }} ms &gt; 10%&quot;<br>    expr: |<br>        ((sum by(worker) (rate(cloudflare_worker_requests_received_total{status=&quot;scriptThrewException&quot;}[5m])))<br>        / <br>        (sum by(worker) (rate(cloudflare_worker_requests_received_total[5m])))) * 100<br>        &gt; 10<br>    for: 5m<br>    labels:<br>      component: cloudflare-worker<br>      severity: critical<br>  - alert: CloudflareWorkerResourceExceeded<br>    annotations:<br>      message: &quot;Resource Exceeded for{{ $labels.worker }} worker - rate {{ $value }}&quot;<br>    expr: |<br>        (sum by(worker) (rate(cloudflare_worker_errors_total{status=&quot;exceededResources&quot;}[5m]))) &gt; 0.0001<br>    for: 7m<br>    labels:<br>      component: cloudflare-worker<br>      severity: critical</pre><h3>Grafana Dashboard</h3><p>When building a simple Grafana dashboard, you need to visualize the metrics alert in order to get information during incidents (e.g., error rate and CPU usage).</p><p>Example of dashboard panels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EXX0Rgn3zFTQpvCW" /><figcaption>Exception rate out of all responses</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vKrDifFEHf0Oj5TH" /><figcaption>CPU time in milliseconds of a specific worker (90th percentile)</figcaption></figure><h3>Where can I find it?</h3><p>So glad you’ve asked! We completed the implementation and published the exporter as an open source on <a href="https://github.com/rewireltd1/cloudflare_worker_exporter">Github</a> (we’d be thrilled to get feedback on our code and approach.) The repo README includes usage instructions and a link to a built docker image.</p><h3>The essence of this project</h3><p>When monitoring modern systems, you should always remember that not all infra is controlled by you. In the case of Rewire, we worked with a new infra that was not fully adopted in the industry at the time. The meaning was that we had to work fast in order to build a Prometheus Exporter quickly so that we would be able to meet our monitoring standards. As you can see, we are now mostly able to monitor the rate and errors while the latency receives less attention. This is fine since we can measure latency from our client.</p><p>Our work on a complete monitoring suite is not over yet. We are constantly looking to find more ways to scrape more information from the Cloudflare Workers environment, which will not affect our code performance.</p><p>As you can see, developing your own Prometheus exporter for your production environment is sometimes the right move. This is mostly helpful when you want to maintain the visibility of your production environment and keep providing highly available services to your customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=162815adc1b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rewire-to/how-to-expose-the-right-prometheus-metrics-for-your-custom-exporter-implementation-open-source-162815adc1b0">How to expose the right Prometheus metrics for your custom exporter implementation (Open Source)</a> was originally published in <a href="https://medium.com/rewire-to">Remitly Israel (formerly Rewire)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sometimes, building your own Prometheus Exporter for monitoring Cloudflare Workers is the best…]]></title>
            <link>https://medium.com/rewire-to/sometimes-building-your-own-prometheus-exporter-for-monitoring-cloudflare-workers-is-the-best-4690531401d?source=rss-a044864415ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/4690531401d</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[cloudflare-workers]]></category>
            <dc:creator><![CDATA[Or Kaplan]]></dc:creator>
            <pubDate>Tue, 31 May 2022 08:26:43 GMT</pubDate>
            <atom:updated>2022-05-31T08:26:43.447Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sometimes, building your own Prometheus Exporter for monitoring Cloudflare Workers is the best solution</h3><p><em>Monitoring serverless environments such as Cloudflare Workers can be a challenging task, especially when you want to get immediate alerts in case of an emergency. At </em><a href="https://www.rewire.com/?code=MEDIUM_R&amp;D_BLOG"><em>Rewire</em></a><em>, we took a detour and decided to build our own monitoring system for our serverless functions that run on top of Cloudflare Workers. If you’re struggling with monitoring Cloudflare Workers, you should take a look at the path we took and the challenges we faced on our way to building a full monitoring system. You might find some of our tips and tricks helpful.</em></p><figure><img alt="The outages fighters" src="https://cdn-images-1.medium.com/max/1024/1*uNCLi5I_-NXVkkmBeI9sEA.jpeg" /><figcaption>The outage fighters in action</figcaption></figure><h3>Why monitoring is important</h3><p>When we started using Cloudflare Workers, we began with a small proof of concept project — an obvious best practice when introducing any new technology into your organization. Suddenly, we found ourselves running a dozen serverless functions in production and more and more developers started to modify their code. Soon after that, we experienced our first two customer-impacting outages:</p><ol><li><strong>The application would not start</strong>: A change was made on a worker that was used to load feature flags on application startup. As a result of the change, the worker would throw an exception about 50% of the time. In these cases, the client application failed to get the base feature configuration and would not be able to load successfully.</li><li><strong>The application was not translated</strong>: We had a worker that read data from a backend service, converted it, and served it to our app. As the data started to grow, our worker passed the CPU time limit of 50 ms and requests started to fail, breaking the app translations among other things.</li></ol><p>Following these incidents, we came to the conclusion that we have to improve our infrastructure and better monitor trends over time. That’s when we started to look for Cloudflare Workers monitoring options. Preferably using our stack, which consists of Prometheus and Grafana for monitoring and alerting.</p><h3>Monitoring strategy</h3><p>Putting our SRE (Site Reliability Engineer) hat on, there are a few strategies we can consider for monitoring our services. There are many ways to look into monitoring strategies, but we had to choose one path that suits our needs. One of my favorite <a href="https://www.youtube.com/watch?v=9dRSYjBPaZM">talks</a> by Tom Wilkie lists the following monitoring methods:</p><ol><li>USE — Focuses on monitoring your infrastructure: take a resource (e.g. CPU) and measure it, then monitor utilization, saturation, and errors</li><li>RED — Focuses on monitoring your service — measure request rate, latency, and errors</li><li>The 4 golden signals by Google — similar to RED but adds saturation in order to plan resources</li></ol><p>Since we run a serverless environment, we do not have immediate control over our infrastructure resources and thus do not need to worry about the metrics that USE focuses on. RED is a great fit for our approach, allowing us to monitor relevant metrics that Cloudflare exposes and capture them using the Prometheus exporter we decided to build.</p><h3>Monitoring “how-to”</h3><p>We set out to observe the Cloudflare Workers in order to find out which metrics we can extract. One way to do this is to report the metrics from the request handler (the worker process itself). Our team had concerns over performance issues with this approach, so we decided to look into <a href="https://developers.cloudflare.com/workers/learning/metrics-and-analytics">Cloudflare’s analytics API</a> instead. With this built-in framework, our processes are not impacted but we are limited to the metrics that are exposed by the API.</p><p>As luck (or good analytics API specifications) would have it, we quickly found out that Cloudflare exposes some useful metrics that can be sufficiently used to monitor our service:</p><ol><li>Request rate</li><li>Error Rate</li><li>Exit status of a worker (success, script threw an exception, client disconnected, exceeded resources, and internal error)</li><li>CPU time</li></ol><p>Take a look and you’ll see that those metrics perfectly fit the RED method!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*trtEGiiuVJ0KO70j" /><figcaption>Request metrics from the Cloudflare dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BKTmXnp7XHPVYKWj" /><figcaption>CPU Time metrics from the Cloudflare dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2G7B435QqJm5eJjH" /><figcaption>Invocation status metrics from the Cloudflare dashboard</figcaption></figure><h3>The benefits we found in using Prometheus</h3><p>While we aim to set trends in the fintech world, we don’t want to reinvent the wheel when it comes to standardized tech stacks. Still, since we run our infrastructure on top of Kubernetes, we also use the favorite Kubernetes monitoring suite, Prometheus. In addition to Kubernetes, we were also using Prometheus to monitor non-native applications such as RabbitMQ and our Postgres DB. Naturally, it is better to keep things unified, but it was initially unclear how to use Prometheus to monitor Cloudflare Workers.</p><h3>Going custom</h3><p>When native-Prometheus support* (*services that offer native support for Prometheus expose a <em>/metrics</em> HTTP endpoint to fetch data in a specific format) is not available, an adapter can be used to expose the relevant information in the expected format — a <strong>Prometheus Exporter</strong>. In our case, we needed a Cloudflare analytics API Prometheus Exporter.</p><p>Following best practices, we began by searching for a community-supported open-source solution. Such a solution did not exist, yet). Eureka! We now know that we have to develop <a href="https://github.com/rewireltd1/cloudflare_worker_exporter">our own exporter</a>.</p><p>This task turned out to be simpler than expected. All we had to do was write an HTTP service which:</p><ol><li>Gets a request for metrics</li><li>Queries the metrics from the monitored resource (Cloudflare Workers, using the analytics API)</li><li>Converts the metrics into the Prometheus format</li><li>Sends the converted metrics back as a response to Prometheus</li></ol><p>Finally, we were ready to start developing our exporter.</p><p>Stay tuned for our next post in which we will walk you through the journey of building the Cloudflare Workers Exporter with real-world examples. In the end, you will see how we are now able to monitor our Cloudflare Workers and prevent future outages.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4690531401d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rewire-to/sometimes-building-your-own-prometheus-exporter-for-monitoring-cloudflare-workers-is-the-best-4690531401d">Sometimes, building your own Prometheus Exporter for monitoring Cloudflare Workers is the best…</a> was originally published in <a href="https://medium.com/rewire-to">Remitly Israel (formerly Rewire)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Run e2e Tests for Every Commit Using testim.io, GitLab CI, and Cloudflare]]></title>
            <link>https://medium.com/rewire-to/how-to-run-e2e-tests-for-every-commit-using-testim-io-gitlab-ci-and-cloudflare-bca77c96b3ce?source=rss-a044864415ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/bca77c96b3ce</guid>
            <category><![CDATA[e2e-testing]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Or Kaplan]]></dc:creator>
            <pubDate>Wed, 29 Sep 2021 08:27:51 GMT</pubDate>
            <atom:updated>2021-09-29T08:27:51.271Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Running e2e tests on every commit while maintaining stability is not an easy task. In this post, you’ll see how our team at </em><a href="https://www.rewire.com/?code=MEDIUM_R&amp;D_BLOG"><em>Rewire</em></a><em> uses testim.io in order to check every commit on our web app in order to maintain high product quality and limit serious bugs on production.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3wnYBfl4SpSdr_yKsn-0XA.jpeg" /></figure><h3>Looking for a fresh start</h3><p>Running e2e tests before merging our code to the main branch is not a new concept. At Rewire, our flow works like this:</p><ol><li>The developer opens a PR from the feature branch in order to integrate a new piece of code into the main branch</li><li>At least one other team member must perform a code review and approve it</li><li>Run a set of automated tests including integration and e2e tests</li><li>Once all steps are completed, the developer can merge the code (using marge bot)</li></ol><p>Before choosing to work with <a href="http://testim.io/">testim.io</a>, we used protractor in order to run e2e tests, which exposed the following challenges:</p><ol><li>A developer must run and maintain the tests, which can take a lot of time and effort</li><li>The tests failed too many times, some of them sporadically, which made it hard to follow</li><li>The test ran on a Kubernetes pod, which required us to set special shared memory settings, which affected the stability and the performance of the test</li><li>The test code was not clean, so it was really difficult to maintain</li></ol><h3>Why did we choose testim.io?</h3><p>Frustrated with our previous e2e automated test flows, we looked for a better solution. We really liked testim.io from day one as its intuitive platform could free our developers from writing complex e2e tests and allow them to focus on building a better product and improving quality in other places. That way, any team member could generate a test that simulates the most important flow of the user.</p><p>Some of our favorite features include:</p><ol><li>Record a test on a live environment and replay it when needed</li><li>AI-based element detectors — the test didn’t change if we changed the location or the color of the button, which was really important to us as we were in the middle of <a href="https://medium.com/rewire-to/rewires-makeover-what-does-it-all-mean-8b0dc82a16a3">rebranding our app</a></li><li>Easy to run from our CI (<a href="https://medium.com/rewire-to/automating-your-staging-environment-generation-for-web-development-4d634713df24">as you already know, we use GitLab CI</a>)</li><li>Thanks to the great and intuitive UI that provides console logs, network logs, and visual assistant, failure points are really easy to detect</li></ol><p>Still, there are some conflicts. Testim.io is running on its own selenium grid, on its cloud instances, which is a bit challenging for us as it requires us to expose our temporary environment externally in a secure manner.</p><h3>Challenges in setting up the e2e environment</h3><p>In order to keep our e2e test stable, we wanted to build an environment that wouldn’t be affected by other tests or human errors. Therefore, we decide to create a fresh environment for every new build. To do so, we leveraged the GitLab CI Kubernetes executor infrastructure in order to simulate the main components of our live environment in the following order:</p><ol><li>Fresh database instance</li><li>Redis instance</li><li>Our API services</li></ol><p>All of these services were loaded as part of the executor pod context and were not accessible from the internet, which proved to be a great challenge for us as testim.io requires a public endpoint for the tests.</p><p>In addition, we had to use external services that had no sandbox environment. So, we had to mock their response as part of the user flow.</p><h3>Eureka — Connecting the dots</h3><p>To make our environment accessible, we had to expose 3 main services:</p><ol><li>API service — naturally requires access to our temporary database and provides a consistent state to our web app</li><li>Web service — serves our single page application to the browser</li><li>Mocking service (aka e2e service) — exposes mocks in order to support external service provider behavior, such as payments, storage, etc.</li></ol><h3>Exposing our e2e architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r39OPWvcDorzPlam8VgLmQ.png" /><figcaption>Our e2e environment high-level diagram</figcaption></figure><p>In the diagram above, you can find our set of solutions for the aforementioned challenges. Each service was exposed differently:</p><ol><li>Using <strong>Cloudflare Argo tunnels,</strong> we were able to expose our API and Mocking services to the internet using a temporary DNS name. In order to do so, we built a special docker image that includes the Argo tunnel binary and is able to fetch the required certificates from our HashiCorp Vault instance. This image was the base image for the executor who ran the test.</li><li>We were able to expose the most up-to-date version of our web app to the internet, which was set to access a specific API endpoint (as demonstrated in our article <a href="https://medium.com/rewire-to/automating-your-staging-environment-generation-for-web-development-4d634713df24">about building staging environments</a> using <strong>Cloudflare Workers)</strong>. In that case, we had to provide testim.io with the relevant web app URL and API version to test against.</li></ol><p>Now, all that’s left to do is run testim.io’s CLI with the relevant variables in order to access the web app and relevant services that are now exposed externally.</p><h3>Sounds easy, right?</h3><p>Well, in real life nothing comes easily. As we quickly learned, there were many issues to tackle in order to make the test run perfectly such as:</p><ol><li><strong>Orchestrating the services</strong>. We had to make sure our satellite services would start only after the main executor service was loaded. Since those services needed some build artifact that the main container in the pod copied in order to run as part of its bootstrap process.</li><li><strong>Argo tunnel bootstrap time is not predictable</strong>, so we had to test the tunnel connectivity before running the testim.io CLI.</li><li><strong>CORS issue</strong> — unfortunately, this bug was not easy to find. When running the test, testim.io sends a different origin header than the base URL host, therefore, we had to support custom origin headers when replying with our CORS headers.</li><li><strong>Exposing the logs on failure</strong> since the test did not always pass, we had to provide the logs of the temporary services to the developers. In this case, we didn’t connect the services to our central logging system, so for each build, we exposed artifacts that contained the logs.</li><li><strong>Logs were not enough</strong>. The developers had to see the content of the database that stores the app state. Therefore, we added a dump of the database upon failure and a restored script.</li><li><strong>DNS collision</strong> — as multiple tests might run in parallel, and in order to avoid collisions, we need to randomize the Argo tunnel name as two Argo tunnels cannot run with the same name. Having two Argo tunnels with the same name is problematic as the test might consume the wrong API.</li><li><strong>DNS pollution</strong> — each test creates a few DNS records so we had to clean up those records in order to avoid having junk on our DNS provider.</li></ol><p>After overcoming all of those challenges, we got to a pretty stable and quick e2e test that covers the main user flow on every pull request pipeline.</p><h3>Finally, we were happy with the results</h3><p>Combining the set of tools we selected, we were able to build an effective e2e test that validates our latest version while keeping our developers happy by letting them deliver quickly.</p><ul><li>Testim.io enables us to build stable tests that are easy to construct, run, maintain and debug</li><li>GitLab CI provides the infrastructure to build and run complex testing architecture</li><li>Cloudflare workers make it easy to expose the web app and reduce the load from the build servers</li><li>Finally, Argo tunnel makes it easy to expose our internal services to the testim.io platform</li></ul><p>That’s it. Stay tuned to the following post in this series: we will talk about how we monitor our Cloudflare workers and what custom infrastructure was built for that purpose.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bca77c96b3ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rewire-to/how-to-run-e2e-tests-for-every-commit-using-testim-io-gitlab-ci-and-cloudflare-bca77c96b3ce">How to Run e2e Tests for Every Commit Using testim.io, GitLab CI, and Cloudflare</a> was originally published in <a href="https://medium.com/rewire-to">Remitly Israel (formerly Rewire)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating your staging environment generation for web development]]></title>
            <link>https://medium.com/rewire-to/automating-your-staging-environment-generation-for-web-development-4d634713df24?source=rss-a044864415ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d634713df24</guid>
            <category><![CDATA[rewire]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[single-page-applications]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Or Kaplan]]></dc:creator>
            <pubDate>Mon, 25 Jan 2021 13:56:56 GMT</pubDate>
            <atom:updated>2021-01-26T08:30:48.446Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating Your Staging Environment Generation for Web Development</h3><p><em>If you’re looking to provide your developers with an instant and publicly available staging environment relevant for every app-related change — you’ve reached the right place. Leveraging our CI system and Cloudflare Workers, we at </em><a href="https://www.rewire.com/?code=MEDIUM_R&amp;D_BLOG"><em>Rewire</em></a><em>, were able to deploy a fully working environment by simply typing a Git commit command.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HqsWoq0TJd46bbrR3zbfEg.jpeg" /></figure><p>Our previous post, about <a href="https://medium.com/rewire-to/serving-our-single-page-application-using-cloudflare-workers-5a936419a076">serving a single page application using Cloudflare Workers</a>, demonstrated how to deliver a single page application from a Google Storage bucket. Once the implementation process was completed, we came to the conclusion that this technique can probably be used to solve other challenges such as testing features during the development process.</p><h3>Speeding Development Cycles</h3><p>As our main focus at Rewire is on a high delivery pace, deploying multiple app releases a day, it’s important that we enable our developers to share their deliverables before they reach the pre-production environment, in order to review and improve quality. This is especially true in these two scenarios:</p><h4><strong>1. Product Sign-Off</strong></h4><p>Our engineering process includes final sign-off by a product manager and a UX designer. In order to finish the process, we should have a running environment with the new code that simulates the production environment.</p><p>At first, we had 5 static staging environments to which every developer on our team could deploy the final feature. However, the situation of having a limited number of staging environments created a lot of tension within the team as we had to continuously check who is using each environment. In addition, we wanted to find a way to showcase a demo of partial features while still in development, in order to get feedback from other team members.</p><h4><strong>2. End to End and Regression Testing</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ifG-UcypjlAMKRio" /><figcaption>The testing pyramid</figcaption></figure><p>Before merging to the master branch (which is our pre-prod env), our CI pipeline needs to successfully complete running a set of automated tests in order to ensure we don’t introduce any regression, and that everything works as expected.</p><p>On the famous testing pyramid concept, we can find end-to-end (E2E) testing: this testing methodology runs the application and the API all while validating visual and functional elements on basic user journey flows.</p><p>In order to develop E2E tests, we’ve decided to use a product called <a href="https://www.testim.io/">testim.io</a>, an AI-based E2E framework that runs a selenium grid on an external location and accesses the web app as a regular user. Therefore, we had to provide <a href="http://testim.io/">testim.io</a> access to our web app. Potentially, we could have solved this issue by using a local Nginx server and exposing it using an HTTP/2 tunnel. However, the solution you’re about to see seemed much more elegant.</p><h3>Back to the Drawing Board</h3><p>The production solution we created, which was demonstrated in the previous <a href="https://medium.com/rewire-to/serving-our-single-page-application-using-cloudflare-workers-5a936419a076">post</a> in this series, includes a docker image with all the static files relevant to Rewire’s web app. Then, we copied those files to the production bucket using our GitLab CI pipeline. Every time a developer opens a new pull request, we build a similar docker in order to run automated UI tests on the application. This means that we have all the relevant files on hand.</p><p>To build a staging environment, we first had to figure out how to serve the application to our users (this time, I’m referring to our developers). We could try to use the worker as described before, but the worker was set to serve a specific environment during the build time. So, we figured that our GitLab CI pipeline should be able to load the files to a dedicated Google Storage folder for the pull request. We used the pull request built-in environment variables to generate a unique folder name to which we upload relevant files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_UVEHk13fFe9nio8" /><figcaption>Our CI Pipeline Illustration</figcaption></figure><p>At this point, we had to decide how to serve the files. While we could generate a new subdomain DNS record for every instance of the staging environment, we actually wanted to avoid the housekeeping process of the DNS records that were not in use. Hence, we decided to pass the folder name on a query string. That way, we were able to maintain one subdomain for all staging environments. This worker acts as a router with the following process:</p><ul><li>On deployment, an index file is loaded to the worker key-value store which is relevant to the app version (the pull request number)</li></ul><p><em>Example</em>: load the index file for PR 35740 from <em>gs://rewire-pr-assets/</em><strong><em>35740</em></strong><em>/assets/index.html</em></p><ul><li>A user enters the router worker with the relevant version query parameter</li></ul><p><em>Example:</em> <a href="https://preview.rewire.internal/?**version=35740***"><em>https://preview.rewire.internal/?</em><strong><em>version=35740</em></strong></a><em>,</em> where 35740 is the pull request number</p><ul><li>The worker stores the version in a cookie (restricted to on the router subdomain) and serves the index page from the relevant folder</li></ul><p><em>Example:</em> <em>Set-Cookie: rewire-preview-version: 35740</em></p><ul><li>Each time a user requests a resource, the browser automatically sends the resource version using the cookie</li><li>The worker knows which resource to fetch by the version’s ID, attaches the version ID from the cookie to the bucket’s base path and as the base folder with the resource name</li></ul><p><em>Example:</em> to fetch the files bundle.js when the version is 35740 the worker will load the resource from <em>gs://rewire-pr-assets/</em><strong><em>35740</em></strong><em>/assets/bundle.js</em></p><ul><li>When a user wants to use another version, the developer sends a new request with a new query param</li><li>In case there is neither query parameter, nor cookie, the worker falls back to our pre-prod environment</li></ul><p>Using this process, a developer who wants to work in a staging environment should just open a pull request and wait for the build to finish. That way, he or she can get a dedicated environment within a few minutes. Subsequently, each push to this branch will update the staging environment and invalidate the cache instead of creating a new environment.</p><h3>Bonus — Select the API server</h3><p>Now that the web application staging environment is up and running, we understand that it might not be enough. Some big features will require us to change both the API and the web application. One way to tackle this is delivering the API prior to the web application. However, this method is not possible with every feature. In addition, in order to complete the End to End tests, we would like to run the API of the current code change. To achieve this goal, we would need to build a staging environment for the API (stay tuned for our next post) and direct the application to access it.</p><p>Up until today, we used to set the API address during the bundling time. Therefore, we had to find a way to manipulate the application so that it would connect to another API server. Again, we had two potential paths to take:</p><ol><li>Manipulate the JS code on the worker, which is a bit complex as the code is minified.</li><li>Use the same technique as before: Send the API version as a query param and set it as a cookie, which is accessible from javascript. Afterward, our client reads the cookie and selects the correct server to access. In case there is no cookie, it falls back to the bundled version.</li></ol><p>We selected the second approach and enabled it only on staging builds.</p><p>For example: to use a specific API staging environment, just pass it as a query param —</p><p><a href="https://preview.rewire.internal/?version=35740&amp;**api_version=apistaging**"><em>https://preview.rewire.internal/?version=35740&amp;</em><strong><em>api_version=apistaging</em></strong></a></p><h3>Keep It Secured</h3><p>The staging environment might contain secret features that we do not want to leak. Therefore, we had to find a way to hide it and make the staging environment private. As part of its zero-trust package, Cloudflare provides an Access proxy that enables access control to the application. Using this feature, we were only able to provide access to the team using our authentication service. Cloudflare Access is basically an application access gateway that we can attach to a specific route, and Cloudflare makes sure only authorized users are able to get it. The administration is very simple and supports roles for access control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*uRTxYfJ5S2U4VFlV" /><figcaption>Securing our Staging environment using Cloudflare Access</figcaption></figure><h3>Cost Awareness</h3><p>One important tip I acquired at a previous workplace, is to only store the necessary data.</p><p>As a startup, we have to optimize our costs in order to minimize our burn rate because every pull request loads our bundle to the object storage, which inflates our storage costs. Hence, we enabled retention rules to the staging bucket, which deletes all the files created more than X days ago. That way, we save only what we need all while saving a few bucks.</p><p>In case of a long time pending pull request, the developer may either rebase or rebuild, and the staging environment becomes live again in a matter of minutes.</p><h3>Wait, There’s More</h3><p>As you see, using Cloudflare workers helped us automate the generation of staging environments and implement scenarios like automated E2E tests, which can be expensive and tedious.</p><p>All we had to do is identify the right storage pattern and find a way to signal the resource version. One thing that made the development very quick was hooking to the current CI process.</p><p>Want to know how we built our end to end test framework using Cloudflare workers, GitLab CI, and<a href="https://testim.io/"> testim.io</a>? Stay tuned for our next post in the series. We’ll tell you all about the cool tricks that make our job much easier and can make yours easier as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d634713df24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rewire-to/automating-your-staging-environment-generation-for-web-development-4d634713df24">Automating your staging environment generation for web development</a> was originally published in <a href="https://medium.com/rewire-to">Remitly Israel (formerly Rewire)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serving our single page application using Cloudflare Workers]]></title>
            <link>https://medium.com/rewire-to/serving-our-single-page-application-using-cloudflare-workers-5a936419a076?source=rss-a044864415ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a936419a076</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[cloudflare-workers]]></category>
            <category><![CDATA[single-page-applications]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[rewire]]></category>
            <dc:creator><![CDATA[Or Kaplan]]></dc:creator>
            <pubDate>Sun, 06 Dec 2020 18:34:42 GMT</pubDate>
            <atom:updated>2020-12-06T18:53:57.000Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Here’s a new and efficient way to utilize Cloudflare Workers to serve your web application. Read more to see how Rewire managed to optimize its web application serving in no time</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5vxDKICvWnvQMfNxpn8Ijw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@frenchriera?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Anthony Riera</a> on <a href="https://unsplash.com/s/photos/code-developer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>For a long time now, our focus at <a href="https://www.rewire.com/?code=MEDIUM_R&amp;D_BLOG">Rewire</a> has been on optimizing the build, deployment, and delivery of our single page web application. Our top priorities are to (1) ease the development process and (2) to serve our customers with maximal performance.</p><p>In this blog post, we will demonstrate how we utilize Cloudflare Workers to serve our web application.</p><h3>History</h3><h4>The motivation</h4><p>As a young startup, Rewire’s R&amp;D team had to move fast and create an MVP as quickly as possible. It started with one service that served both the API and the web application, which is a set of static files served as a Single Page Application (SPA). When I joined rewire about two years ago, the R&amp;D team was focused on breaking the monolith into API and SPA services. The SPA was built into a docker image and served via an Nginx server that runs on top of a Kubernetes cluster. While this structure did simplify our development process, it was not enough — we wanted to be resilient to outages in our Kubernetes cluster. We were sure that we can do better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*VwZj1hjDIG_qmdslWPaxSw.png" /><figcaption>Diagram of the previous CI process serving using Nginx</figcaption></figure><h4>Moving to serverless</h4><p>Going back to the drawing board, we had a thought: Why not go serverless? We were already utilizing Cloudflare Workers for a few small-scale projects and it seemed like a great match. Cloudflare Workers supports a set of really cool features like zero cold start, out of the box CDN, caching, a great development experience, and fast deployments, which were very important for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*0T8sSHxOO-VkFZBOw8L09Q.png" /><figcaption>Diagram of the new solution using Cloudflare Workers</figcaption></figure><h3>Our design</h3><p>The design is pretty simple: Use the same process that builds the web app and pack it into a docker image. But instead of publishing the docker image into the container registry, copy the files to a Google Storage bucket that will be accessed by the Cloudflare worker. Then we intercept the requests, fetch the static files from the bucket, and serve them to the client. Sounds simple, right?</p><p>The implementation we suggest is pretty common: We host our code on a GitLab instance. Each time a developer pushes a commit to the relevant branch, we run the following pipeline using GitLab CI:</p><ol><li>Fetch the code</li><li>Build — fetch all relevant npm packages and run Webpack to build the web application</li><li>Pack — store the files in a docker image</li><li>Publish the docker image to the registry</li><li>Deploy the image to our Kubernetes cluster using Argo rollout</li></ol><p>In the new design, we replaced two major steps (deploy and publish) with one new step that copies the static files directly from the docker image and loads them into the relevant bucket. That’s it, now the Cloudflare worker can serve the files from the bucket.</p><h3>Serving the index</h3><p>The entry point to the application is an index HTML file. For every new release, this file loads a different bundle of javascript files, CSS files, images, and more.</p><p>Our application doesn’t directly request the index.html file. Instead, we have to serve it from the root /. When we first see a request to the root path, we fetch the index file from the bucket and cache it in the worker key-value store to boost performance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b9719c3d207e9139beef1a5a9d0bcf9/href">https://medium.com/media/2b9719c3d207e9139beef1a5a9d0bcf9/href</a></iframe><p>We make sure to clean the storage and fetch a new index file with each fresh deployment, in order to populate the index file when serving the next request.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0e52d7805c09754221f2efbee6b39cf/href">https://medium.com/media/e0e52d7805c09754221f2efbee6b39cf/href</a></iframe><p>This technique has amazingly cut the serving time of the index file by more than 80% — from 250ms to sub 50ms! But, it came with a price, which I’ll describe later on.</p><h3>Utilizing worker cache</h3><p>As you know, low latency is not the key feature of object store. In order to ensure our app’s fast loading times, we have decided to utilize the built-in cache of the Cloudflare worker. To do so, we had to push the resource to the cache for every request.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/584ed2fcdb66f4cfda815295d86bb588/href">https://medium.com/media/584ed2fcdb66f4cfda815295d86bb588/href</a></iframe><h3>Deployment Gotcha</h3><p>While testing the project in production, we encountered a deployment issue. Each index file refers to a list of bundles that are replaced on each deployment. The bundle file name contains a hash that is updated on each deployment. Hence, if the client fetches the index, and then the deployment replaces the folder, the worker will return 404 for the resources. To overcome this issue, we used the following technique:</p><ol><li>Utilize cache to return the files</li><li>In case the cache is disabled or invalidated, we have to find another solution. So, we are now saving two live versions: The first one is the current, live version, and the second one is what we previously had for backup. During deployment, we copy the live version to the backup and replace it. In case we can’t find a resource in the live version, we search it on the backup, where it should exist</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*QU5NZN-BBFuSdXdxT7s_AQ.png" /><figcaption>Blue-Green Deployment using Cloudflare Workers</figcaption></figure><h3>Summary</h3><p>Using Cloudflare workers, we were able to build a high availability solution in record time. It required us to play with HTTP headers, cache directly from the code all while providing great flexibility to build the serving mechanism as we wished. Running on the edge serves in over 200 cities across the world, making it much more performant than our previous Kubernetes based solution. In other words, we basically got CDN for free using workers.</p><p>Being a customer-centric company, we are now able to provide our customers with a much better experience and a more stable app.</p><p>The next post in the series will demonstrate how this technique can be used to build an infinite number of staging environments for developers to sign-off their features with product managers, and how it can be used for quick end-to-end testing using the <a href="http://testim.io/">testim.io</a> framework.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a936419a076" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rewire-to/serving-our-single-page-application-using-cloudflare-workers-5a936419a076">Serving our single page application using Cloudflare Workers</a> was originally published in <a href="https://medium.com/rewire-to">Remitly Israel (formerly Rewire)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>