<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by REACH NOW on Medium]]></title>
        <description><![CDATA[Stories by REACH NOW on Medium]]></description>
        <link>https://medium.com/@reachnow?source=rss-28e5e8e51ea0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*_wa35esq1qvzmFdtbgqf2Q.jpeg</url>
            <title>Stories by REACH NOW on Medium</title>
            <link>https://medium.com/@reachnow?source=rss-28e5e8e51ea0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:55:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@reachnow/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Address quality in OpenStreetMap: a comparison]]></title>
            <link>https://medium.com/reachnow-tech/address-quality-in-openstreetmap-a-comparison-47fa13b730fc?source=rss-28e5e8e51ea0------2</link>
            <guid isPermaLink="false">https://medium.com/p/47fa13b730fc</guid>
            <category><![CDATA[address]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[quality]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <dc:creator><![CDATA[REACH NOW]]></dc:creator>
            <pubDate>Thu, 08 Oct 2020 07:58:11 GMT</pubDate>
            <atom:updated>2020-10-08T07:58:11.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYtAuqOa9pi73kq4wDmLaA.jpeg" /></figure><p>The Place Search Team at moovel takes care of the place and commuter search experience in our products. Our goal is to improve the users’ overall search experience. Part of that task is having a usable and coherent place search feature with which users are able to find places they are looking for with minimal effort.</p><p>While developing and maintaining a place search engine is one part, which any ambitious team could aim at, having full control over data is a rather impossible challenge. We distinguish between three different data sources internally: Public transport stops, Points of Interest (POI) and Address data.</p><p>In the context of mobility and multimodal transit apps, data requirements are manifold. We cannot accept any (minor) differences in public transit stop information: stop names in places search aim to be as similar as possible to stop names in reality (or as users perceive them). Furthermore, locations must be exact such that we can calculate the possibilities and estimated time of arrival.</p><p>Requirements to POIs and addresses are — compared to public transit stops — less strict. Slightly different names can be accepted as long as users can easily match their expectations to our data. Minor location differences (e.g. up to 10m) or the lack of a specific address on a street where we have all others can also be accepted.</p><p>For example, for <em>Glockengießerwall in Hamburg </em>in our proprietary, comprehensive &amp; costly<em> data source</em> we have numbers <em>5</em> and <em>5a</em>, while <a href="https://www.openstreetmap.org/">OpenStreetMap</a> (OSM) will show just <em>Glockengießerwall 5</em>. And for the case of addresses it is acceptable for us, but it could be easily improved.</p><p>We have already been using OSM and <a href="https://openaddresses.io/">OpenAddresses</a> for quite some time for other parts of our service and supported regions of operating. However OpenAddresses’ <a href="https://results.openaddresses.io/coverage/world/">world-wide coverage</a> is not that good, especially in Germany, which is the focus of this research. So we decided to give open data from OSM a chance to compete on the German address search field, assuming it would not be as good as our reference address data set.</p><h3>What is the quality of an open data set then?</h3><p>In our address search we leverage a comprehensive and commercial address data set, which we can use as a reference and try to compare to OSM. We want to seek answers for two questions:</p><ul><li>How good is OSM in terms of address quality?</li><li>Does it meet our requirements?</li></ul><p>To do so, we decided to take some steps which will be described in the upcoming sections.</p><h3>Define the way a comparison can be made</h3><p>We selected some common areas for both sources, e.g. zip-code, and compared street and address data from the same area in different parts of the country. The map below shows the selected regions for a representative comparison.</p><figure><img alt="selected regions for a representative comparison" src="https://cdn-images-1.medium.com/max/1010/1*kJjPp6s6QXC9PzkhQq4srQ.png" /></figure><h3>Retrieving data</h3><p>OSM<strong> </strong>data was downloaded from <a href="https://www.geofabrik.de/data/download.html">Geofabrik</a> as a <strong>.pbf</strong> file and extracted using the <a href="https://github.com/pelias/pbf2json">pelias-pbf2json</a> tool. Below we can see an example of how we can retrieve addresses from a <strong>.pbf</strong> file.</p><pre>export ZIP_CODE=10407</pre><pre>./pelias-pbf2json.darwin-x64 \<br>  -leveldb=leveldb_folder \<br>  -tags=&quot;addr:housenumber+addr:street+addr:postcode~$ZIP_CODE&quot; \<br>  -waynodes berlin-latest.osm.pbf \<br>  | jq -s -r &#39;.[] | [<br>   .tags.&quot;addr:street&quot;,<br>  .tags.&quot;addr:housenumber&quot;,<br>  .tags.&quot;addr:postcode&quot;,<br>  .tags.&quot;addr:suburb&quot;,<br>  .tags.&quot;addr:city&quot;,<br>  .type,<br>  .lat,<br>  .lon,<br>  .id] | <a href="http://twitter.com/csv">@csv</a>&#39; &gt; berlin-$ZIP_CODE.csv</pre><p>In the example above we use <a href="https://stedolan.github.io/jq/">jq</a>, a JSON processing tool. We filtered the result output by fields which are of interest in terms of address comparison. As a result, we’ve got a <strong>.csv</strong> file for addresses within the given zip-code (10407 in this case), containing all interesting properties such as street, house number, coordinates, city and suburb. A fragment of the result <strong>.csv</strong> can be found in the following table.</p><figure><img alt="addresses within a given zip-code" src="https://cdn-images-1.medium.com/max/1024/1*EAyjUR-puw1xlpX7ufXYeg.png" /></figure><p>Our proprietary data source already has data in the CSV format.</p><p>OSM is a widely known open-data project with a free-access community, which inevitably leads to ambiguity and data quality problems, like mistakes, typos, different address naming schemes, several kinds of duplicates, among others. In order to neutralize these errors or inaccuracies we took the following steps.</p><h3>Expand house numbers for OSM addresses</h3><p>All future manipulations were made in <a href="https://jupyter.org/">Jupyter Lab</a>. We imported <strong>.csv</strong> files into a Pandas <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html">DataFrame</a> (DF). <a href="https://pandas.pydata.org/pandas-docs/stable/index.html#module-pandas">Pandas</a> is an open source library for data analysis using the Python programming language. All future manipulations were made using these tools.</p><h3>Node extraction</h3><p>Nodes, ways and relations are <a href="https://wiki.openstreetmap.org/wiki/Elements">basic components</a> of OSM’s conceptual data model. Though all of them might have address components, we are interested in nodes only. As a next step, we exclude ways and coordinates from the DF, since they do not play a part in comparing components.</p><p>In the following picture, we can see the fragment of a retrieved DF for one zip-code.</p><figure><img alt="fragment of a retrieved DF" src="https://cdn-images-1.medium.com/max/1024/1*5s3vmt0OBdnFOgzsc_C3ag.png" /></figure><p>The first problem we encounter is how house numbers are stored, like <em>Am Friedrichshain 29–32</em> for example. In our proprietary data source, that would usually mean having a few rows for <em>Am Friedrichshain 29, Am Friedrichshain 30</em>, etc. Other options include:</p><ul><li>StreetName 3a-c</li><li>StreetName 3,4</li><li>StreetName 3;4</li><li>StreetName 3/4</li></ul><p>We want to be able to unfold the examples above into different addresses. We can do so by searching for special characters in these strings, as shown in this <a href="https://gist.github.com/schorba/e49fa52f8143fa794fa503dc885baa84">small script</a>. The data frame will look different after unfolding these house numbers.</p><figure><img alt="data frame looks different after unfolding house numbers" src="https://cdn-images-1.medium.com/max/1024/1*pH820ta2appHgoax-zA9Nw.png" /></figure><h3>Normalization</h3><p>Similarly to house numbers, german street names are written in different ways. For example, we found a street which is named <em>Berliner Straße</em> in one source and <em>Berliner Str.</em> in the other. In this case, we can assume it is actually the same street and no version is better than the other. Let’s normalize streets spelling of names:</p><pre>def normalize_osm_data(df):<br>  for i in range(0, len(df)):<br>    if &#39;St. &#39; in df.loc[i, &#39;street&#39;]:<br>      df.loc[i, &#39;street&#39;] = df.loc[i, &#39;street&#39;].replace(&#39;St. &#39;, &#39;Straße&#39;)<br>    if df.loc[i, &#39;street&#39;].endswith(&#39;Str.&#39;):<br>      df.loc[i, &#39;street&#39;] = df.loc[i, &#39;street&#39;].replace(&#39;Str.&#39;, &#39;Straße&#39;) <br>    if df.loc[i, &#39;street&#39;].startswith(&#39;St.&#39;):<br>      df.loc[i, &#39;street&#39;] = df.loc[i, &#39;street&#39;].replace(&#39;St.&#39;, &#39;Straße&#39;)     <br>    return df</pre><h3>Comparison</h3><p>For each area analyzed we consider the following criteria:</p><ul><li><strong>Numbers of unique addresses for each source and area</strong> — We expect different results for both sources and this needs to be checked</li><li><strong>Unique street names</strong> — Either data source might miss not only certain addresses but also a whole street</li><li><strong>Differences on address data</strong> — It’s a combination of the previous two metrics, which should give us some numbers to make a conclusion on potentially switching data sources.</li></ul><p>Our assessment then consists of three steps:</p><ul><li>Checking unique addresses in both data sets</li><li>Checking numbers of unique street names</li><li>Overlapping information in both sources</li></ul><p>Unique street names can be found through:</p><pre>def get_unique_streets(df):<br>  df = df.sort_values(by =&#39;street&#39; , ascending=True).reset_index()<br>  df = df[[&#39;street&#39;]]<br>  df = df[df.duplicated() == False]<br>  return df</pre><p>Unique addresses and saving results into files for future analysis can be found through:</p><pre>def unique_adress_names(df1, df2):<br>  merge = df1.merge(df2, how=&#39;outer&#39;, indicator=&#39;source&#39;)<br>  result_l = merge[merge.source.eq(&#39;left_only&#39;)].drop(&#39;source&#39;, axis=1)<br>  result_r = merge[merge.source.eq(&#39;right_only&#39;)].drop(&#39;source&#39;, axis=1)<br>  result_b = merge[merge.source.eq(&#39;both&#39;)].drop(&#39;source&#39;, axis=1)<br>  result_l.to_csv(&#39;export_csv/STREET-NAMES_only.csv&#39;, index=False)<br>  result_r.to_csv(&#39;export_csv/STREET-NAMES_only_OSM.csv&#39;, index=False)</pre><p>The previous examples show that some data processing is required before we can compare the two sources: i.e. normalization, data expansion and data deduplication.</p><p>At the same time, open data has advantages over our proprietary data source. For instance, OSM could be edited and corrected right away, while our proprietary data source is only updated in regular intervals and we have practically no influence on it.</p><p>Having results for each zip-code we can simply calculate the average for all considered zones, make some corresponding charts and analyse them.</p><h3>Results</h3><p>In the graph below we can see the absolute number of unique street names in the involved regions and number of addresses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tL2sg9eyA5QfVaWrSPUzQ.png" /></figure><p>Zones 1 to 9 are inner-city areas of big cities like Berlin, Hamburg, Stuttgart, Cologne or Dresden, whereas zones 10 and 11 are parts of smaller towns in the south of Germany. This discrepancy can maybe be explained by the fact that OSM coverage is generally better in urban areas, since it is a community-driven source where content may vary with population and activity.</p><p>Similar to the numbers for addresses, we can see from the graph below that both sources follow a common pattern and show quite similar numbers in comparison to each other, despite some differences between zones. Data in urban areas is possibly adopted faster than in rural areas, which causes differences in street numbers. The numbers for Germany are actually higher than we might think: In 2017, around 11,000 streets were edited — either newly built, partitioned or renamed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tpyct8vFrm4FATeBGr6wsw.png" /></figure><p>Let’s now take a deeper look at both sources. We are now comparing each individual address and see if we find an exact match in the other data set. For the sake of simplicity, we aggregate the numbers over the selected zones and calculate their averages. From the next chart, we can see that 10% of all addresses included in our proprietary data source could be found only here, while the remaining 90% are available also in OpenStreetMap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ErpjAAmoqg7dG3D_kmx3Ew.png" /></figure><p>Aside from completely missed addresses for any reason, quite a large number of misreadings in OpenStreetMap are cases like <em>Waldowstraße</em> in Berlin. In our proprietary data set, <em>Waldowstraße</em> has the numbers <em>6A/6B/6C</em>, whereas in OSM just the number <em>6</em>. Meanwhile, it’s actually just the same building! Another example: small (sometimes “private”) streets and either part of a park area, like<strong> </strong><em>Burg 2, in Esslingen</em>.</p><p>However, OpenStreetMap also contains addresses that can only be found there, with numbers looking similar to our proprietary data source, with a rate of <em>7%</em>. The following chart shows the average address-found ratio for all involved areas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRbiH0LWDuEDEZDTyitWMA.png" /></figure><p>However, <em>87%</em> of all available addresses from a shared set could be found in both sources. While a unique part of addresses from our proprietary data source is <em>9%</em>,<strong> </strong>and for OpenStreetMap <em>4%</em>, according to the upcoming diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ow7tPyUovvGRtV_-u8J-hQ.png" /></figure><h3>Conclusion</h3><p>Regions involved in this analysis cover main areas of interest to us, big and smaller cities. Based on results and the possible reasoning on differences, we conclude that OSM is a valid and budding alternative to our proprietary address data source in many contexts and for many applications.</p><p>Using OSM as an address data source will of course require some normalization, deduplication and other kinds of processing work, but if we are looking for an <em>Open Data</em> alternative for an address data source — OpenStreetMap might be one of the options that is definitely worth considering.</p><p>written by <a href="https://www.linkedin.com/in/sergey-chorba-27130a1b">Sergey Chorba</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47fa13b730fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/reachnow-tech/address-quality-in-openstreetmap-a-comparison-47fa13b730fc">Address quality in OpenStreetMap: a comparison</a> was originally published in <a href="https://medium.com/reachnow-tech">reachnow-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Marketing Tech Stack]]></title>
            <link>https://medium.com/reachnow-tech/performance-marketing-tech-stack-d2a3bf061dc5?source=rss-28e5e8e51ea0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2a3bf061dc5</guid>
            <category><![CDATA[performance-marketing]]></category>
            <category><![CDATA[attribution]]></category>
            <category><![CDATA[tech-stack]]></category>
            <category><![CDATA[roa]]></category>
            <category><![CDATA[ama]]></category>
            <dc:creator><![CDATA[REACH NOW]]></dc:creator>
            <pubDate>Thu, 27 Feb 2020 12:26:02 GMT</pubDate>
            <atom:updated>2020-08-04T12:35:47.604Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83X-jGgnQwFGyqHjYq2ymg.png" /></figure><h3>Here’s what we’ve learned from our tech stack and how it can help your MaaS app gain traction</h3><p>Imagine you have a great mobility app, but only a fraction of your target audience is aware that it actually exists. Digital marketing can help to spread the word to your target audience and generate active users for long-term, profitable customer relationships.</p><p>To reach your audience, you’ll need to promote the app on channels that they use. And to find these, you’ll need the right tools, or, “tech stack.”</p><p>Having a solid tech stack makes it easier to measure the success of your campaigns and to optimize the campaigns in regards to key performance indicators. For example, with the right tools you can quickly see your ROAS (Return on Advertising Spend). This useful metric is calculated by dividing the generated revenue of acquired users how much it cost to acquire them.</p><p>Due to the <strong>sheer amount of potential channels</strong>, measuring your campaign’s performance and steering the campaigns can quickly become overwhelming. And so, choosing the right tools for your business is essential to running efficient marketing campaigns.</p><p>So where do you start? And how do you structure your stack? Here we’ll walk you through a case study with one of our clients to show you exactly how we used the following tools and the effect they had on the success of our campaign:</p><ul><li>Digital marketing channels</li><li>Attribution tools</li><li>Engagement tools</li><li>Analytical tools</li></ul><h4>So which mobility app are we talking about?</h4><p><a href="https://www.moovel.com/en/featured-apps/ssb-bestpreis"><strong>SSB BestPreis</strong></a> is a multi-modal app in Stuttgart, enhancing regular public transport search, book, and pay functions with other modes of mobility like carsharing or bikesharing. The app’s biggest is in its pricing mechanism that automatically calculates the best ticket option for public transport. All you have to do is to book single tickets and as soon as the cumulated price of the tickets reaches the price of a daily ticket, you only get charged for the day ticket. With multiple daily tickets you would eventually reach a weekly ticket, and so on. It’s basically like the Oyster Card in London, but digital via app.</p><h4>Defining the target audience of BestPreis</h4><p>SSB BestPreis targets people who occasionally use public transport in Stuttgart but don’t own a monthly ticket. This audience is thus more likely to ride irregularly and prefers to have the freedom to decide how they spend their money and get around.</p><h3>Digital marketing channels</h3><h4><strong>How to reach potential customers and convert them into paying users</strong></h4><p>SSB first approached us several months ago with the job to set up a digital campaign to generate new users from the defined target group. With a runtime of nine months, they wanted to prominently advertise the BestPreis app on all relevant digital channels in the metropolitan area of Stuttgart.</p><p>Their most important goal was to drive installations which would then convert into long-term paying users. In order to get maximum coverage in the Stuttgart area, we used Facebook, Instagram, Google App Campaigns, and Apple Search Ads. Those tools ensure the highest reach, since they have the biggest user base in Germany.</p><p>We structured the campaign based on these requirements, while tailoring and managing it based on the relevant channels, expenses, and target values. In order to track and evaluate the campaign, we set up a dashboard displaying all significant KPIs including impressions, clicks, installs, ad spendings, and Cost-per-Install (CPI).</p><p>On top of that, we tracked subsequent in-app events like registrations, adding of a payment profile and transactions/bookings to keep a holistic overview of paid installs vs. the actual usage of the app.</p><p>For analysis and optimization, it’s important to build so-called cohorts in order to see how specific users interact with the app after they have installed. Imagine that you start testing another audience on Facebook in the course of the campaign. Here, you need to split these users from the ones that you have acquired beforehand, since they may interact differently with your app than the audience before (the cohort here would be all paid installs on Facebook that happened in the timeframe of the test). If you were to analyze those users together, you would get the wrong picture of how the new audience performed.</p><h4><strong>Coming up with the right creatives is key</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kk9zMLrdgvWOaGTgX_istg.png" /></figure><p>When crafting creatives for Facebook or Instagram campaigns, you won’t be able to get around running a competitor analysis to determine your unique selling proposition.</p><p>We combined our chosen USP with a design concept based on SSB’s brand identity. In the end, we handed over 20 variants to our design team who then helped create over 200 ad variants, mixes of different attributes like headers, call to actions, and images.</p><p>Our competitor analysis was essential to help the BestPreis stand out from other available public transport apps in Stuttgart. With a total of four public transport apps in the city, competition is fierce. So we had to help it stand out by showing what makes BestPreis unique, using snappy claims like “hop in, pay later,” or “not for free, but always the best price.”</p><h3>Tools</h3><p>Here is a quick overview of tools you’ll need to properly track the right metrics for your campaign. All tools have been selected based on the GDPR policy and are fully GDPR compliant. Thus, all data is being stored in the EU and can be deleted at any time. Furthermore, there is a marketing opt-in/ opt-out toggle for users and all tools are taken into account in our privacy policy.</p><h4><strong>Attribution tools</strong></h4><p><a href="https://www.adjust.com/"><strong>Adjust</strong></a><strong> </strong>is a tool that helps you learn more about user behavior in your app (it allows mobile app tracking via SDK integration). With adjust you can see how users find the app and how they interact with it. This was especially valuable for this campaign, because we could track which users were led to the app organically or coming from one of our ads. We then could see who converted into a paying user (by setting up a payment profile at moovel).</p><h4><strong>Engagement tools</strong></h4><p>In order to interact with users beyond setup, we use tools like <a href="https://www.localytics.com/"><strong>Localytics</strong></a> and <a href="https://www.talon.one/"><strong>Talon.one</strong></a>. Localytics is used to create in-app messages, overlays, and push messages to directly contact users. When a user has the app installed on their phone, gives consent for marketing content, and has push notifications activated, you can send out reminders or small advertisement texts to engage them, or, even better, give them voucher codes as incentive to keep using the app.</p><p>Speaking of vouchers, <strong>Talon.one</strong> is an essential tool that we use to create them. Often times, a campaign is only successful when you incentivize the user to download the app. With the Talon.one integration, you can do exactly that. After getting the voucher code, the user simply has to install the app, set up a moovel account, and paste in the code. Since users can use these vouchers to book free rides, they will automatically be incentivized to start using the app. In this regard, vouchers are not only an incentive to install the app, but also a powerful tool in order to make new users try out your product and become regular users when they have a good experience. And in the end, that’s the ultimate goal for every business.</p><h4><strong>Analytical tools</strong></h4><p><a href="https://funnel.io/"><strong>Funnel.io</strong></a>, helps you combine different data sources (e.g. Adjust, Facebook, Google) so that you can have one single dashboard that shows you all figures and KPIs (a single source of truth). It serves as an interface in order to later on visualize all channels in a tool like Google Data Studio, for instance. Before using Funnel.io, we had to manually check the data of each source individually to then collect it in a spreadsheet. Funnel.io removes this work and connects with every channel so you can quickly see how effectively your campaigns are running.</p><p>We also use Google Data Studio to create dashboards visualizing the performance of a campaign. On the one hand, this makes it easy for our Performance Marketing to track performance quickly and optimize it when needed. On the other hand, this dashboard can also be shared with the customer. Any tool you use should include all important metrics like impressions, clicks, installs, Cost per Install, Costs per Click, registrations, payment profiles, bookings, total revenue, ROAS etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*C3ymVNpnssvagptEh7t77A.png" /></figure><h3>Results of the campaign</h3><p>Now that we’ve given you an overview about the setup setup up for SSB BestPreis’s performance marketing campaign — including — the tech stack involved, it’s time to have a closer look at the campaign performance.</p><p>As mentioned above, this campaign ran for nine months. So naturally, we analyzed this time frame in order to determine the success of the marketing campaign.</p><p>It’s important to know that the longer the time period, the higher the ROAS. Why? Because ROAS is determined by dividing the revenue by the advertising costs and if a marketing campaign ends. This means that after ad spend stops, so do the costs. If the acquired users still keep on spending money in the app, the ROAS goes up. That said, typically the revenue generated from a cohort (e.g. all paid installs in November) decreases month by month, unless you take some re-engagement actions.</p><p>As you can see, we achieved an overall ROAS of 234 percent over nine months and generated €2.34 revenue for every euro that was spent for advertising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*L8gIsw1Zyheu6MNL6ZREjg.png" /></figure><p>Keeping this in mind, ROAS should always be considered over a longer period and be observed even after the paid campaign has ended (hint: make sure to learn more about the concept of customer lifetime value). To get a clearer picture of our performance for the BestPreis App, we calculated our ROAS up until January 2020, giving the campaign a lifetime ROAS of 384 percent. In other words after the paid campaign has ended, the acquired users kept on making purchases in the app and lifted the ROAS over the course of the next 5 months by 150 percent.</p><p>As you can see, having the right tech stack is indispensable to any modern digital marketing efforts. It helps you make the right data-driven decisions on where to spend and on how to get the most out of your media budget. Moreover, a stack will give you the tools to better analyze the performance of the entire campaign. That’s essential to communicating your success within your company and for showing it to your customers. Without the right tools, you risk wasting money and not knowing how and where to optimize your campaigns. That can keep you from achieving better results and, in the end, delighting your clients.</p><p>By Florian Teufel, <a href="https://medium.com/u/bc750798b015">Anton Stopka</a>, <a href="https://medium.com/u/302504fd2267">Daniel Hurrle</a>, <a href="https://medium.com/u/c9b7fcb79ec3">Hendrik Gossmann</a> and <a href="https://medium.com/u/bc002e155eff">John Wegner</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2a3bf061dc5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/reachnow-tech/performance-marketing-tech-stack-d2a3bf061dc5">Performance Marketing Tech Stack</a> was originally published in <a href="https://medium.com/reachnow-tech">reachnow-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tweaking the tech org]]></title>
            <link>https://medium.com/reachnow-tech/tweaking-the-tech-org-43392a37537a?source=rss-28e5e8e51ea0------2</link>
            <guid isPermaLink="false">https://medium.com/p/43392a37537a</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[offsite]]></category>
            <category><![CDATA[culture]]></category>
            <dc:creator><![CDATA[REACH NOW]]></dc:creator>
            <pubDate>Fri, 08 Nov 2019 13:15:12 GMT</pubDate>
            <atom:updated>2019-11-08T13:15:12.371Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BIFUuF6FIxpTArloEmDTCA.png" /><figcaption>We are tweaking the tech org</figcaption></figure><p>At moovel, we work in distributed and autonomous teams, with Hamburg and Berlin as our main locations in Germany, alongside a few others. Consequently, Software Engineers are spread throughout those locations. Strong and effective collaboration between our locations is our focus, therefore bringing our Software Engineers together into one place is a regular, common practice.</p><p>Usually we plan those offsites every 4–6 months in varying locations. Having all colleagues in one place and to be able to chat and discuss in person is an important aspect, especially if your main communication happens primarily digitally. The discussions and decisions resulting from an offsite ideally provide important inspiration for our teams in the following months in one way or another.</p><h3>Topic Gathering</h3><p>Having topics to talk about when we gather all of our colleagues in one place is of imperial importance. In order to create an agenda, we decided to gather topics from all of the participants beforehand to get a grasp of what people would like to talk about. These topics had to be related to the Tech organization itself.</p><p>The main objective was to try to understand what kinds of processes and/or practices were executed well or not so well, general company achievements and cultural aspects engineers would like to point out or speak about.</p><p>The process was as follows:</p><ul><li>A spreadsheet was created and shared throughout the Tech organization so people could input their topic ideas</li><li>We conducted a session with all engineers to make sure all gathered topics are well understood. In this group session, we created smaller groups of around 4 people that could discuss and specify topics in a more detailed fashion</li><li>After all topics were gathered, the topics were curated and grouped together to create a list of topics</li></ul><p>The curated list of topics was then sent to the entire Tech organization before the offsite, so everyone could get acquainted with it. On the day of the offsite, the list was presented in more detail, so participants could vote on the topics that would interest them the most.</p><h3>Day Breakout</h3><p>After all topics were collected and curated, an agenda was set up to drive the day and the offsite’s activities. The general outline looked like:</p><ul><li>Morning sessions</li><li>Introduction and Tech direction</li><li>Presentation of curated list of topics</li><li>Afternoon sessions</li><li>Topic breakout sessions in smaller groups to produce actionable results</li><li>Conclusion</li><li>Evening socializing activities</li></ul><p>As outlined above, we used our morning to try to propagate the general direction of our Tech organization and how we want to work from that point on, with the addition of presenting the curated list of topics previously gathered. Each topic had a responsible person to ensure that action points would follow each one of the eventual breakout sessions. As soon as the topics were presented, each participant could vote on their preferred topics using the <a href="https://en.wikipedia.org/wiki/Dot-voting">Dot Voting Technique</a>. For each one of the most voted topics, breakout sessions were allocated in the afternoon.</p><p>Each breakout session was moderated by its responsible person who made sure the session was documented and contained actionable conclusions. After the meeting was over, all attendants engaged in networking and socializing activities with their colleagues.</p><h3>Tweaking Topics</h3><p>The topics came from various backgrounds: backend engineers, frontend engineers, data scientists, business informatics, and engineering leads. Some of the topics that were discussed included backend driven mobile apps, E2E responsibilities, engineering culture, improving and consolidating operations, logging and monitoring, to name a few.</p><h3>Data Strategy</h3><p>At moovel (becomes REACH NOW), we gather a lot of data on a daily basis — for instance, user flows or various behaviour habits. Multiple data stakeholders decided to sit together and discuss the status quo and see if we can optimize it to provide the user, partners, and even employees with an overall better experience, while at the same time protecting the privacy of the end-user. The Data Working Group was born as a result and we are constantly investigating how to improve on this front.</p><h3>Conclusion</h3><p>Tweaking the Tech organization was both a great social event and a trigger for several follow up action items that are currently ongoing. Once those action items have been executed we are open to take on new topics that our engineering group comes up with, probably with yet another tweak.</p><p>Meanwhile, we are curious to know how <strong>you follow up with topics that your engineering group raises?</strong></p><p>Written by: <a href="https://twitter.com/NelspikeTX">Nelson Oliveira</a>, <a href="http://linkedin.com/in/philipfieber">Philip Fieber</a>, <a href="https://twitter.com/kamasheto">Mahmoud Sakr</a>, <a href="https://www.lotharschulz.info/">Lothar Schulz</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43392a37537a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/reachnow-tech/tweaking-the-tech-org-43392a37537a">Tweaking the tech org</a> was originally published in <a href="https://medium.com/reachnow-tech">reachnow-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TeamCity Upgrade]]></title>
            <link>https://medium.com/reachnow-tech/teamcity-upgrade-e0eb88388f01?source=rss-28e5e8e51ea0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0eb88388f01</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[teamcity]]></category>
            <dc:creator><![CDATA[REACH NOW]]></dc:creator>
            <pubDate>Wed, 02 Oct 2019 10:43:42 GMT</pubDate>
            <atom:updated>2019-10-02T12:49:32.709Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6XgDDNPjq6aFUAkj" /></figure><h3>Introduction</h3><p>Upgrading internal services can be as challenging as that directly used by customers. Many engineers depend on tools such as CI being highly available, making service degradation costly, especially when the company offers flexible working hours (as REACH NOW does) and/or spans time zones. This blog post follows of our DevOps engineer <a href="https://twitter.com/btisdall">Ben Tisdall</a>’s daily routine on upgrading our TeamCity CI running on AWS, back in Summer 2019. We hope this post will be of help to others performing similar migrations.</p><h3>Background</h3><p>We’ve been using TeamCity as our CI system since an evaluation of various options in mid 2016. While we were broadly happy with our TeamCity setup there were several things that needed to change:</p><ul><li>We’d hadn’t taken advantage of major version upgrades and were now two off the pace. Not only were we missing out on new features but our version was no longer officially covered by our support contract!</li><li>Many of the system components were not created using <a href="https://www.thoughtworks.com/insights/blog/infrastructure-code-reason-smile">infrastructure code</a>.</li><li>Since TeamCity’s introduction we had moved to a multi-account AWS model and this system was designated to be relocated.</li></ul><h3>Diary</h3><h4>Day 1</h4><p>Today, I started by looking at the infrastructure code previously written by a colleague for this project. It was sound, but I decided to move the Elastic File System (EFS) code out into a separate template — this will promote faster testing cycles, cleaner and more comprehensive templates and, in unhappy cases, reduce blast radius.</p><h4>Day 2</h4><p>We now have an Amazon Machine Image(AMI) with latest Amazon Linux 2 and the latest TeamCity build. I’ve also automated the database user setup by via a VPC-bound Lambda that that’s invoked by our CloudFormation wrapper script, accepting the endpoint and other pertinent details via stack exports together with secrets from parameter store and using this information to create the empty teamcity database, the “teamcity” user and password.</p><p>Today I also created a new GitHub repo for the infrastructure code. Large, long-running feature branches are a bad smell and since we want to keep the old infrastructure running in parallel it makes sense to fork the code. Large teams probably don’t have the luxury of doing this!</p><h4>Day 3</h4><p>Today worked on improving the code, with a fair bit of refactoring and adding some deployment wrapper scripts. I also started building a TeamCity v10.0.3 AMI for use in the new infrastructure context (the current live version not being suitable) so that we can easily test the upgrade and rollback process. I also started to think about the migration process itself.</p><h4>Day 4</h4><p>I’m thinking out loud here about the decision to go with VM-based deployment as opposed to running on ECS, which is our standard way of running workloads. Some background: I took this project over from a colleague that left the company. The engineer previously working on the project had favoured ECS but our TeamCity deployment uses EFS as persistent storage and it had been concluded that this was not a good fit for use with ECS. My colleague had hoped to switch to EBS but had decided this wasn’t feasible, perhaps because he wasn’t aware of the ECS <a href="https://github.com/rexray/rexray">Rexray</a> integration or because had examined this rejected it. In any case, our current setup is reliable and performant and since we’re not only upgrading two TeamCity major versions but also moving everything to a different AWS account in the same operation not trying to change other things at the same time seems like a good idea.</p><p>Would we do it differently if we were starting now? Maybe, but we’d probably want to keep it on separate infrastructure anyway, so there wouldn’t be any savings on a dedicated ECS cluster vs TeamCity on a VM. At any rate we have 3 years’ experience running TeamCity on a VM — once the migration is done then changing the platform while keeping the TeamCity version the same would be a far easier operation than what we need to do here.</p><h4>Day 5, 6</h4><p>I spent these days familiarising myself with basic PostgreSQL administration.</p><h4>Day 7</h4><p>Today I’ve started to develop the migration process. The first phase involves spinning up a TeamCity 10.x environment, creating a configuration and then doing whatever’s necessary to migrate that to 2018.x. As I go along there will no doubt be some repetitive work that should be automated — as with any automation although it imposes some upfront costs it will repay as we iterate the migration steps.</p><p>Upon initialisation of a new database via the UI, TeamCity helpfully prints out some optimisation tips; these will be incorporated into the final RDS parameter group for the DB instance.</p><h4>Day 8</h4><p>Today I started performing tests on database backup/restore and looking into filesystem backup/restore. Along the way, I created some infrastructure code to spin up a migration instance in the same VPC as the TeamCity server for faster connectivity during backup/restore operations.</p><p>At some point I noticed:</p><pre>$&gt; df -t nfs4<br>Filesystem 1K-blocks Used Available Use% Mounted on<br><a href="http://oneu-west-1c.fs-34926efd.efs.eu-west-1.amazonaws.com/">eu-west-1c.fs-34926efd.efs.eu-west-1.amazonaws.com:/</a> 9007199254739968 5394827264 9007193859912704 1% /mnt/teamcity</pre><p>Oops, we have over 5TB of data on EFS! This makes backing up and restoring quite a mountain to climb (it’s also very expensive compared to S3), let’s see what we might be able to prune…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bi0iEj8RJ1FB0zNk" /></figure><p><strong>Learning:</strong> the S3 artifact plugin we were using creates a <em>duplicate</em> store of artifacts in the specified bucket. This plugin will be disabled in the new environment and we’ll switch to the bundled S3 artifacts plugin which is able to replace the default artifact store within the TeamCity data directory.</p><h4>Day 9</h4><p>Yesterday, thanks to a colleague from an Android app squad, we got to the bottom of where most of the disk usage was occurring — it turns out that at some point Android builds were profiled and every build was saving a 1.3G .hprof file as an artifact! Profiling has been turned off now and some cleanup rules put in place. A find for hprof files is now running, the output of which will then be used to clean them up. We should give more thought to cleanup policies in future. Today I’ve been writing database backup and restore scripts.</p><p>Just for fun, here’s the upgrade screen we’ll face come the hour:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/0*Jes-l2AeqN4wB3dX" /></figure><h4>Day 10</h4><p>Today I’m going to test the process of backing up the TeamCity EFS data.</p><h3>File Backup/Restore</h3><p>The most time consuming part of the operation was the data backup and restore. Since the system must be quiesced while the backup runs it also means downtime. A typical solution to minimise downtime in these kind of situations is to use a file system synchronization tool such as rsync to do an initial synchronisation with the process(es) accessing the source filesystem still running, then at the appointed time to stop the processes and re-run the synchronisation using the tool’s option to delete files on the destination that no longer exist on the source.</p><p>The first approach to backup/restore was to use awscli’s aws s3 sync command to sync the filesystem up to a bucket and then in a separate pass sync sync down to the target filesystem:</p><p>Here’s the output from the first attempt with the production filesystem:</p><pre>time sudo -u teamcity DELETE=1 ./sync-data-to-s3<br>real 655m 16.028s user 304m 32.466s sys 165m 40.343s</pre><p>So, around 11 hours just to do the initial upwards sync. How about refreshing the backup?</p><pre>real 387m 11.638s user 16m 59.872s sys 4m 7.619s</pre><p>So, it took over 4 hours to refresh, which can’t be started for the last time until TeamCity is shut down. Then we need to consider the restore situation — though I haven’t tested this part, it’s likely to take a comparable amount of time. In view of this I’ve decided to investigate other means of synchronisation. One piece of good news is that our syncing operations haven’t eaten into the EFS burst balance — here’s an interesting chart from CloudWatch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FcDlP-f5ZJdKgZrB" /></figure><p>The big red arrow indicates the start of a TeamCity cleanup operation that removed a lot of data related to Android projects. Since (in default mode) EFS accrues a burst balance related to the metered size of the filesystem, this is why the reduction in burst credits is observed. The yellow arrow is coincident with the start of the first sync to s3 while the green one lines up with the refresh runs some hours later. The good thing is that we never come within 50% of the IOLimit, the non-bursted performance of EFS in non-provisioned mode. Were we to go over the IOLimit and start eating into the burst balance that might impact normal operation, but that said, we never seem to need it. Even with normal usage we see a not dissimilar pattern to above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CBmT0mBLmZ6ck4OI" /></figure><h3>Other backup/restore options</h3><p>I looked into other solutions, none of which turned out to be fit for purpose:</p><h3>AWS DataSync</h3><p>This is a (mostly) managed service aimed at easing the process of copying on-premise data to S3 or EFS. It is strictly on-premise-to-cloud and can’t do EFS → EFS.</p><h3>AWS Backup</h3><p>This is AWS’ managed “Cloud Native” backup service. Restoring a backup creates a new resource with the data at the specified point in time, but it doesn’t work cross account.</p><h3>EFS-EFS Copy</h3><p>EFS-EFS Copy is an AWS “solution” — something you can build yourself by combining other AWS services using provided CloudFormation and other code. It does work cross account but is unnecessarily complex for our use-case where we don’t care about scheduled backups (strip away all the orchestration and it’s rsync running on an EC2 instance, we already have these pieces available in code).</p><p>Using plain rsync to sync 2 EFS proved pretty slow — around 12 hours to copy a quarter of the data (400G). Although this might be an acceptable transfer rate in some contexts, we want to proceed on the basis that:</p><ul><li>The process is repeatable within time spans of days rather than weeks.</li><li>Refreshes are fast.</li></ul><p>We therefore looked at what AWS use in their EFS-EFS solution and saw they make use the the fpsync utility — this leverages the fpart command to partition the source file set and spawns an rsync for each. We’re testing this now (as I investigated fpsync a bit more I came across another interesting <a href="https://github.com/aws-samples/amazon-efs-tutorial/blob/master/performance/README%2020180219.md">AWS resource</a>).</p><h3>File Backup/Restore conclusions</h3><p>Having chatted with AWS support today they confirmed that the best approach would be mounting the source and target EFS on an EC2 instance and use rsync over a VPC interconnection. We’ll now proceed to set up VPC peering between the VPCs and begin testing. Unfortunately fpart with 8 processes did not produce an appreciable speed up over a single rsync process, the problem here seems to be that we’re dealing with a very large number of files and the NFS overhead is the most significant factor. The next time round it would be interesting to use fpart’s option to execute each sharded process on a separate host as this might be a way to make use of EFS’ performance capabilities but our deadline was approaching and stakeholders we content to tolerate some overnight into morning downtime. Another problem is that for the final filesystem sync with deletion of target objects rsync --delete cannot be used by fpart since each rsync worker will have a different idea of what can be deleted on the target filesystem. AWS’s EFS copy solution makes use of fpart to sync new files from the source followed by a final, single process sync that includes the --delete --existing --ignore-existing flags. In our case it was as fast to simply use a single rsync --delete in place of these two steps.</p><p>In the end we were certain a “refresh” would take over six hours so at a high level our plan became:</p><ol><li>19:00: Stop Teamcity and initiate final sync. Get a night’s rest.</li><li>06:00: check the sync completed and continue.</li></ol><p>This plan required that we worked with stakeholders to ensure there were no overnight CI jobs that would create customer impact if they didn’t run.</p><h3>Database Backup/Restore</h3><p>For this we kept things simple and used shell scripts based around pg_dump and pg_restore that pushed and pulled the backup to/from S3 respectively. The other obvious option would have been to create a manual RDS snapshot, share it with the target account and use it with theDBSnapshotIdentifier parameter in our CloudFormation template. The main problem with the second approach is that we would have had to defer the creation of the new RDS instance until the step after capturing the backup in the migration operation, whereas with a pg_dump backup we could create the instance ahead of time. Both the database dump and restore only took ~2 minutes to complete so added no significant time to the migration,</p><h3>The Migration Operation</h3><p>The cornerstones of a successful migration are in my opinion:</p><h3>Have a detailed plan</h3><p>You need a plan. It should include:</p><ul><li>Guidelines for how the plan should be executed in general terms, eg the role of each participant and guidelines for handling the unexpected (eg no recovery steps to be taken unilaterally).</li><li>The roles should include a coordination role — one person whose job is to ensure that all participants are in sync and move the operation forward.</li><li>The expected duration of each step so that you can monitor progress as you go.</li><li>Success and failure criteria.</li><li>Rollback steps and the cutoff time at which they must be executed.</li></ul><h3>Publicise your plan</h3><p>We inform stakeholders over Slack (eg our #developers channel) about operations that will create downtime but we also find it really effective to create a calendar event and invite our @devs group — this gives us a way of seeing who’s paying attention via acceptance of the “meeting” and provides the invitees with a convenient reminder.</p><h3>Automate things (but know when to stop)</h3><p>Operations of this type generally involve running commands from various machines within different networks/VPCs/Cloud provider accounts. Given the cycle time required to test the overall orchestration it probably doesn’t make sense to automate <em>everything</em> unless you intend to repeat it many times but you <em>definitely</em> want to script the <em>steps</em>. Scripts should of course be treated like any other code, ie be subject to your team’s usual development processes. Scripts should also take as close to zero arguments and options as possible — spare a thought for the humans who will probably be working for a long stretch at an abnormal hour on this. We used machines in different accounts/VPCs for different steps which helped to reduce blast radius — for example our file synchronisation script ran from within one AWS account and the startup script detected this and mounted the source EFS read-only. Our scripts mostly inferred any options from the AWS account context (aws sts get-caller identity) and some had multiple safeing mechanisms (eg don’t attempt to restore a database backup to the live database even if the networking would not permit it). Machines for the purpose of running scripts were spawned via auto scaling groups and synced the scripts from s3 on startup (these having been synced from SCM during CI). The VM startup scripts set a different command prompt colour for each account, the live (source) account/VPC machines using bold red.</p><h3>Be conservative about the operation duration</h3><p>Things have a habit of taking a bit longer than expected and in an operation with many changes snags are inevitable. Provide a conservative estimate of the time you’ll hand the resource back to users and hope to exceed expectations. We’d promised our developers they’d have TeamCity back at midday and were able to hand it over at 11:00 despite a few snags along the way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0eb88388f01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/reachnow-tech/teamcity-upgrade-e0eb88388f01">TeamCity Upgrade</a> was originally published in <a href="https://medium.com/reachnow-tech">reachnow-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[(Con)currently Unavailable — Troubleshooting the Node.js Event Loop]]></title>
            <link>https://medium.com/reachnow-tech/con-currently-unavailable-troubleshooting-the-node-js-event-loop-cb488b59bbd2?source=rss-28e5e8e51ea0------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb488b59bbd2</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[flamegraphs]]></category>
            <category><![CDATA[event-loop]]></category>
            <dc:creator><![CDATA[REACH NOW]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 09:43:25 GMT</pubDate>
            <atom:updated>2019-09-20T11:50:37.602Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5NrOxJO_VgkmBGGkK7XqkQ.jpeg" /></figure><h3>(Con)currently Unavailable — Troubleshooting the Node.js Event Loop</h3><p>The following article is a resume of a specific incident in which 5xx HTTP responses popped up every now and then without our microservice being actually down. We assembled everything we learned from pinpointing, reproducing and eventually mitigating it into a comprehensive story.</p><p>We assume the reader has some familiarity with Node.js, common web technologies and specific Unix tools. However, despite this being our current setup, the observations from the article are also be applicable to alternatives (e.g. AWS ECS/Kubernetes).</p><h3>503 Whodunit</h3><p>At ReachNow some of our teams adopted Node.js as a technology to build APIs. Performance is often cited as a reason for this preference. Due to its asynchronous-first design, it promises to handle a lot of concurrent requests effortlessly, at the expense of programming ergonomics. With this in mind, we were startled as we observed rising response times and eventually even received 503 status codes from our service’s load balancer, while the load increased only moderately.</p><p>Those 503 status codes were concerning, so we started investigating them. Our services are deployed in Docker containers, orchestrated by AWS’ ECS offering. Looking at the ECS service’s event log we spotted the following succession of events:</p><pre>aws ecs describe-services --cluster our-cluster --services our-service | jq &#39;.services | first | .events&#39;<br>[<br>  {<br>    &quot;message&quot;: &quot;(service our-service) has reached a steady state.&quot;,<br>    &quot;id&quot;: &quot;5c81330e-2b86-4a84-a2b3-d714694734d0&quot;,<br>    &quot;createdAt&quot;: 1566206511.979<br>  },<br>  {<br>    &quot;message&quot;: &quot;(service our-service) registered 1 targets in (target-group arn:aws:elasticloadbalancing:eu-west-1:XXX:targetgroup/our-target-group/456)&quot;,<br>    &quot;id&quot;: &quot;70a3310a-3a69-4dab-98fe-9b492d3158c4&quot;,<br>    &quot;createdAt&quot;: 1566206490.103<br>  },<br>  {<br>    &quot;message&quot;: &quot;(service our-service) has started 1 tasks: (task def).&quot;,<br>    &quot;id&quot;: &quot;039e4069-6b7e-4675-abfc-43c1fdd260d2&quot;,<br>    &quot;createdAt&quot;: 1566206479.307<br>  },<br>  {<br>    &quot;message&quot;: &quot;(service our-service) has stopped 1 running tasks: (task abc).&quot;,<br>    &quot;id&quot;: &quot;cdbdc3b3-3bd3-45b8-9c80-124bce260e79&quot;,<br>    &quot;createdAt&quot;: 1566206467.992<br>  },<br>  {<br>    &quot;message&quot;: &quot;(service our-service) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:123:targetgroup/our-target-group/456)&quot;,<br>    &quot;id&quot;: &quot;981f9a50-1386-4d45-bd06-a95d0bf4adab&quot;,<br>    &quot;createdAt&quot;: 1566206467.974<br>  },<br>  {<br>    &quot;message&quot;: &quot;(service our-service) (instance i-abc) (port 33117) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:123:targetgroup/our-target-group/456) due to (reason Request timed out)&quot;,<br>    &quot;id&quot;: &quot;d87a10c7-601e-4402-8fc1-58d4ceccf361&quot;,<br>    &quot;createdAt&quot;: 1566206467.94<br>  },<br>  ...<br>]</pre><h3>Detour I: ECS and Health Checks</h3><p>ECS, like other container orchestration solutions maintains the <em>desired state</em> as declarative API objects (ECS Services) and constantly reconciles the physical reality (ECS Task) with those. An ECS Cluster is a logical group of compute resources (EC2 or Fargate) which register themselves in the ECS API as available hosts for container workloads. ECS assigns a scheduled container more or less arbitrarily to one of the registered hosts.</p><p>In the event of failure, e.g. when a hosts network connectivity is interrupted, ECS should act swiftly, i.e. spawn a new container on a healthy instance and redirect all traffic to it. A common method to report the healthiness of an ECS task is a /health endpoint which is called periodically by a load balancer and expects to receive a 200 HTTP status code as a response in a timely fashion.</p><p>In express.js the implementation of a simple health endpoint could look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c411c998370d8265d74427d0d7dd6519/href">https://medium.com/media/c411c998370d8265d74427d0d7dd6519/href</a></iframe><h3>Service Interruption</h3><p>With this background information we can at least make sense of the event messages and explain the 503 status code we got occasionally (we have to read the events from bottom to top):</p><ul><li>A task has been reported unhealthy, there was a timeout when calling the health check</li><li>This specific task is cut off from further traffic on the load balancer</li><li>The task is killed</li><li>A new task is spawned</li><li>The new task is registered at the load balancer</li><li>The new task has reported healthiness and hence receives traffic</li></ul><p><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-503-issues">The documentation for Application Load Balancers</a>, which we use, states the following:</p><blockquote>HTTP 503: Service Unavailable</blockquote><blockquote>The target groups for the load balancer have no registered targets.</blockquote><p>In the particular environment in which we observed the problem, there was just a single replica of the services provisioned. The 503s occurred between the deregistration of the old task from the load balancer and the point when the new task was considered ready to receive traffic. This short timespan can easily be a few seconds, if the new task is spawned on a cluster node, which needs to pull the docker image first or if the container needs some time to startup.</p><p><strong>Note</strong>: In a production setup we’re likely to have more than just one replica of a service running behind a load balancer, hence the traffic would shift immediately to the remaining replicas and we would not see a 503. However, as we develop further below, this would put more stress on the remaining replicas, which will suffer from the same issue. Finally, even with a lot of replicas, we might still occasionally witness odd status codes, like 502 from the load balancer. Getting rid of those is hairy, we have to consider things like <em>tcp</em> <em>keep-alive </em>or<em> deregistration delay period</em>. As a colleague once put it, a “TCP connection is a distributed system and can only be eventually consistent”, so it’s best not to kill your containers arbitrarily.</p><p>This being explained, we can turn to the next breadcrumb. Why did the health check time out? Looking at the sample code above, few things can go wrong in scope of that /health endpoint.</p><h3>Detour II: Concurrency in Node.js</h3><p>We will only touch on this briefly, as this is explained thoroughly <a href="https://nodejs.org/uk/docs/guides/event-loop-timers-and-nexttick/">elsewhere</a>. The reason Node.js is able to yield astonishing performance <em>in some situations</em> has to do with the nature of the workload most web services have to deal with. Quite often, especially in a tiered architecture, a web service does not actually “do much”, at least in terms of CPU cycles.</p><p>In very abstract terms: we receive I/O events from our network hardware, we decide quickly what to do with it and then we hand it over to some other tier like the database or another service. From the perspective of our web service we then idle for an eternity, until we receive an answer from those systems, process it quickly and can finally return a response to the caller. Performing remote I/O is just <em>a lot</em> slower than processing data locally in RAM with our CPU, for example when parsing a raw string into a data structure (however, this paradigm has been <a href="https://penberg.org/parakernel-hotos19.pdf">diminished increasingly</a> by advances in I/O hardware, while the speeds of CPU cores have been relatively stagnant due to physical limits).</p><p>We can capitalize this waiting time by using threads, which allows us to handle concurrent access in our API similarly to the sequential mental model of code flow that’s familiar to most programmers. Threads incur some inconveniences, like subtle data races and performance overhead from context switching.</p><p>Node.js took another approach. It exposed the idle time as a concept and rooted it deeply in the programming model. Node.js forced the programmer to embrace the asynchronous nature of IO by using callbacks (or abstractions built on them). In return, the programmer is relieved of actively addressing the concurrency issues that often haunt multi-threaded code: A synchronous block of code is guaranteed to be executed without arbitrary, non-deterministic leaps to other parts of the code.</p><h3>How to fail a health check?</h3><p>When piecing together what we know from health checks and Node.js’ concurrency behaviour, we can artificially craft a situation in which a health check could fail even under modest load:</p><p>1) We write a computationally expensive block of code. The go-to example would be the calculation of prime numbers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c60a580b54e078e6c76a73609e35a716/href">https://medium.com/media/c60a580b54e078e6c76a73609e35a716/href</a></iframe><p>A call to getPrimeNumbers() is completely synchronous (even though it’s split into smaller functions), it will run from start to end without any interruption (within the process), which depending on the parameter can easily span a few seconds.</p><p>2) We implement and run a simple express service with 2 endpoints: /health and /primes, the latter invoking the getPrimeNumbers() function from above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c6a423c2bceb0e9e1937ad67ba887f0/href">https://medium.com/media/3c6a423c2bceb0e9e1937ad67ba887f0/href</a></iframe><p>3) We periodically call the /health endpoint with a timeout (2s) and bail out once the call fails (watch -e -n 5 curl -s -S -m 2 localhost:3000/health).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FsdespYGYgvmzEmkKPSwDA.png" /></figure><p>4) We call the /primes endpoint <em>once</em>, with a big-ish max parameter, depending on CPU and environment (curl &quot;localhost:3000/primes?max=200000&quot;).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4PjHMfO8w2ipbvz9BKk6hg.png" /></figure><p>We can see that one of our recurrent health checks failed. Calculating prime numbers up to 200.000 took around 6s, during which our Node.js process could not be bothered to respond to health check requests.</p><p>When we replace the -m 2 flag in the curl invocation (timeout for the whole request) with --connect-timeout 2 (timeout for establishing connections) the health check <em>does not</em> fail, so Node.js is capable of accepting the connection and putting it on the event loop. However, the response callback is executed only after the 6s prime calculation is done <em>en bloc</em>.</p><h3>Analyse and Reenact Failure</h3><p>Admittedly having an endpoint which triggers a 6s block of computation is an odd corner case. However we have to be aware that our problems will pile up with multiple requests, performed in parallel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-pvDwBs82n5-uwZOtbTioQ.png" /></figure><h3>Benchmarking</h3><p><a href="https://httpd.apache.org/docs/2.4/programs/ab.html">Apache benchmark</a> (ab) is a simple yet very helpful tool we use to benchmark our endpoints. To simulate real-world traffic, we can issue a certain number of requests in parallel and receive statistics. Also to have a more realistic setup this time, we will only calculate prime numbers up to 15.000. This takes about 50ms on our machine, which is still a lot, but can easily happen in commonplace circumstances (we’ll come up with an example later). We then issue a total of 1000 requests, 50 of those to be performed concurrently:</p><pre>ab -n 1000 -c 50 <a href="http://localhost:3000/primes?max=15000">http://localhost:3000/primes?max=15000</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mRCxVRVqC90SliBpA1V5jw.png" /></figure><p>The ab report actually tells us why. Requests usually took around 2500ms to complete, thus our health check did run into a timeout, since it expected a response within 2000ms. Considering the request pipeline sketch above, this makes sense. We could actually predict this behaviour, the math is pretty simple: 50ms * 50 = 2500ms.</p><p>We can conclude: Having a synchronous sequence in our endpoints which takes more than 50ms to finish, while serving 50 consumers at the same time, will cause problems. Every request has to pass this particular bottleneck in the single-threaded Node.js process and there will be a large queue.</p><h3>Probing for Bottlenecks</h3><p>The whole exercise we performed above just helped us understanding the problem in principle. Equipped with this, however, we’re able to find problematic patterns in our application.</p><h4>VM Metrics</h4><p>It’s a good practice to gather live metrics from a web application in production using Application Performance Monitoring (APM). Popular tools are NewRelic, DataDog or Dynatrace. Looking at NewRelic for example, their Node.js monitoring provides us with an Event loop metric called <a href="https://docs.newrelic.com/docs/agents/nodejs-agent/supported-features/nodejs-vms-statistics-page">Max CPU time per tick</a>:</p><blockquote>This chart shows the maximum time spent in a single tick per minute. This is useful for detecting an abnormally long event loop tick, which indicates long synchronous code execution.</blockquote><p>This is exactly what we’ve examined so far. And indeed, looking at the chart for this metric in your random production Node.js service might yield scary results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*tnmjSKv5DwLmFVb6e2NEYw.png" /></figure><h4>Flame Graphs</h4><p>Now we know for a fact that we have a problem. We don’t know where to look, though. Supposedly we could litter our code with a console.time/timeEnd() statements to find the problematic part, but that would be tedious and limited in granularity to our code (still, as a desperate low-fi method for narrowing down issues via <em>divide and conquer</em> it might be fine).</p><p><a href="https://twitter.com/brendangregg">Brendan Gregg</a>, who contributed several authoritative methods &amp; tools to observe the performance of our systems, came up with <a href="http://www.brendangregg.com/flamegraphs.html">Flame Graphs</a>. These graphs are a very helpful and accessible visualization method to show where CPU time is spent in a program, using <em>stack recordings</em> collected by a profiling tool. It used to be notoriously hard to produce a readable flame graph for a Node.js application (it involved special Node.js builds &amp; running Solaris on EC2). Luckily, these days things are comparatively trivial.</p><p>To emulate a somewhat realistic scenario we’ll add an endpoint to our service, which retrieves a collection of geodata (1mb) in XML. By recursively iterating through all the nodes and uppercasing the string values (upper()), we perform a somewhat expensive data transformation. Eventually we return to the user as a JSON object:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdccc7eb09b7f47d3f611623050c9585/href">https://medium.com/media/fdccc7eb09b7f47d3f611623050c9585/href</a></iframe><p>We then run the Node.js server (v8.15.0) with the profiler option enabled:</p><pre>node --prof server.js</pre><p>After startup, we issue 10 requests to collect some samples:</p><pre>ab -n 10 -c 2 <a href="http://localhost:3000/mondial">http://localhost:3000/mondial</a></pre><p>When we’re done we can stop the server process. Mapbox provides a flame graph util called <a href="https://github.com/mapbox/flamebearer">flamebearer</a> which we’ll happily use to render the samples collected from our profiler (we can download the tool or execute directly it using npx, which comes with the Node.js distribution).</p><pre>node --prof-process --preprocess -j isolate-*.log | npx flamebearer</pre><p>This should open a browser window with a flame graph (actually a variation of it, mirrored horizontally, an “icicle graph”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAmgEhi8y5QwmxRhAOgq2g.png" /></figure><p>There is an abundance of literature around the topic, so we’ll only explain briefly what we see. The call-stack samples are grouped together, the y-axis from top to bottom represents the code flow into function hierarchies, while the width depicts the relative time the CPU has spent in a specific function. Finally the color indicates how busy the CPU was while spending time in those functions. What we’re looking for are conspicuously wide, deeply red bars. If we spot those, we have found a pattern which is most likely unhealthy for a Node.js web service. We suspected our data transformation function (upper()) to be the culprit, however it’s something else that’s dominating our graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*w8TdiWRJWVV07druuA-AWA.png" /></figure><p>The conversion from XML to JSON (toJson() in the snippet above) while it appears innocuous, it is rather costly in terms of CPU processing time, adding to the response times and eventually making the service unresponsive to health checks. Now we know which code path to optimize. Maybe we can use a JSON endpoint instead, consider a more performant parsing library, or tweak the options of the one we’re using.</p><h4>Replaying Hostile Traffic</h4><p>Sometimes it might not be trivial to reproduce a state in which a Node.js service would spend a lot of time in synchronous code blocks. In those cases we might want to make our service relive the circumstances under which it struggled. It’s not always easy, as there are dynamic factors we cannot easily reproduce (time, for example: some things actually only fail after 5:00pm). However, even if the problematic code path is not obvious, it’s often triggered by a certain set of user-specified parameters. An unexpected query parameter duration=-1 might put us in an endless loop.</p><p>Today it’s pretty common to collect logs from several (interdependent) services in an Elasticsearch index (as part of an <a href="https://www.elastic.co/de/what-is/elk-stack">ELK</a> stack). So we assume we have our logs stored in such a setup and we’ll extract a 1000 URL list with a simple URI search from Elasticsearch (we’re using <a href="https://httpie.org/">httpie</a> to spare us url encoding woes) from the problematic time period. The individual search string will look different, depending on the schema of your log events. Also, in our case the URL is put into the req.url property, so we’ll extract that with jq:</p><pre>http \<br>&#39;<a href="http://elasticsearch/logs-2019.08.29/_search?q=message:%22HTTP">http://elasticsearch/logs-2019.08.29/_search?q=message:&quot;HTTP</a> \<br>GET /v3/search&quot;&amp;size=1000&#39;  | jq -r &#39;.hits.hits | \ map(.&quot;_source&quot;.req.url) | join(&quot;\n&quot;)&#39; &gt; urls.txt</pre><p>You should be able to perform a similar extraction with <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html">Cloudwatch Logs</a> or other log stores. The resulting file looks like this:</p><pre>/v3/search?origin=52.50315639999,13.4693131&amp;destination=13.454179,52.4746541<br>/v3/search?origin=6.794479099999,51.2198279&amp;destination=51.2018594,6.776456899999999<br>/v3/search?origin=51.2198279,6.794479099999999&amp;destination=51.2018594,6.776456899999&amp;mode=BIKE_SHARING&amp;provider=SOME_THING<br>...<br>/v3/search?origin=51.2198279,6.794479099999999&amp;destination=51.2018594,6.776456899999&amp;mode=CAR_SHARING&amp;provider=OTHER_THING</pre><p>Now we produced list of 1000 URLs, split by newlines. We now convert those URLs into requests and discharge them in parallel to our local service:</p><pre>cat urls.txt | xargs -I{} -n1 -P8 curl &#39;http://localhost:3000%&#39;</pre><p>In this example -n1 makes xargs run a new command for every line, -P8 specifies 8 “pipelines” to run concurrently and -I% defines the substitution character for the command. A line /v3/hello?q=world will result in an invocation of curl &#39;http://localhost:3000/v3/hello?q=world&#39;. Again, we can run a periodical health check in parallel and monitor it for timeouts.</p><p>Often our development machines are way more powerful than the (virtual) machines on which we deploy our code, production containers have CPU and memory constraints we don’t have locally. Docker provides a simple method to impose <a href="https://docs.docker.com/config/containers/resource_constraints">such limits</a> locally. e.g: docker run -c 128 -m 512m -p 3000:3000 my-service:v2 will limit the container to 10% of the available CPU shares and 512 mb of RAM.</p><p>This technique will often yield a request or a combination of requests, which originally put our service in a bad state. When debugging, being able to reproduce issues is key: it allows us to perform a targeted analysis of the problem.</p><p>Caveat: In Node.js service’s the logging infrastructure is often set up in a way which will actually <em>omit </em>problematic request in the log. For practical reasons, a log event is often only produced when there was a (success or failure) response to an API requests, so we can add <em>response time </em>or an <em>error code </em>to the event. However, when we do run into an endless loop while processing a request, our service will get killed because it cannot respond to a health check, and we’ll never log the offending request. In those cases it makes sense to look at the <em>caller’s </em>log. If we’re using a L7 Load Balancer operating on the HTTP layer, we can also retrieve meaningful URL logs from there.</p><h3>Unblocking the Event Loop</h3><p>After investigating and reproducing the scenario we can now fathom several solutions on how to mitigate the underlying problem of a blocking event loop.</p><h4>Rate Limiting / Throttling</h4><p>Based on the examination above we can deduce a ratio of replication &amp; requests per minute that we can handle without degrading performance or risking service healthiness. Independently from Node.js, if your API is exposed to external traffic directly or indirectly, it’s advisable to protect your service and underlying tiers like databases from sudden surges in traffic. Sometimes those surges are of malicious nature (DoS), very commonly the reason for sudden spikes are programming errors (think: const results = Promise.all(myLargeArray.map(callEndpoint));). Scaling patterns like auto-scaling cannot react immediately to unexpected situations like this. A good scheme to handle those cases is <em>Rate Limiting</em>, which imposes a limit on the amount of requests that can be passed on to an API during a certain time window. Sometimes it’s an overall limit or it’s applied per API key, per client IP or other factors. When the limit is reached, the caller receives a 429 HTTP Status Code, a client can then back off and retry the request after some time. Also, Rate Limited APIs often report the consumed quota via HTTP Headers, so clients can adapt and throttle their calls. Ideally, instead of being mixed with application code, this would be implemented on the infrastructure level, e.g. in an API Gateway or as a policy in a Service Mesh. However, if you don’t have a setup like this, there are also convenient middlewares for e.g. express.js, which handle Rate Limiting within the scope of your web service.</p><h4>Scaling</h4><p>The classical approach to tackle a performance related problem would be scaling. Either <em>horizontally</em> by increasing replication to have multiple instances handling your requests or <em>vertically</em> by adding more CPU to your instances which will reduce the processing time. However in the sketched scenario this might not help much. Increasing computing power will shorten the time span the event loop will be blocked, though that still might not be enough for the health checks to succeed within a reasonable amount of time. With adding additional instances you can increase the chance that at least one of those will be healthy to respond especially non-blocking requests and thus ensure the overall reliability for your service. Depending on the usage of your service this might be a solution, though the initial problem (blocking code) remains while response times will still stack up.</p><h4>Relaxing Health Checks</h4><p>Failing health checks are the primary issue in our case — so why not simply relax the health check policy to tackle this problem? AWS’ LoadBalancers HealthCheck parameters can be configured in that regards (see CloudFormation snippet below), e.g. adjust the number of consecutive health check failures until an instance is considered unhealthy. While this approach might indeed work it cosmetically hides the underlying problem whose symptoms are merely failing health checks.</p><pre>Type: AWS::ElasticLoadBalancingV2::TargetGroup<br>Properties: <br>  HealthCheckEnabled: Boolean<br>  HealthCheckPath: /health<br>  HealthCheckPort: 80<br>  HealthCheckProtocol: HTTP<br>  HealthCheckIntervalSeconds: 20<br>  HealthCheckTimeoutSeconds: 5<br>  HealthyThresholdCount: 2<br>  # ...</pre><h4>Partitioning</h4><p>While Node.js with its single-threaded event loop is the root cause of our problem, the framework offers several ways to break out of this cycle. As already mentioned the event loop consists of different so-called <em>phases</em>, each with a different characteristic and each with its own queue of callbacks to execute.</p><p>The problematic part of our example manifests itself within the poll phase, where the calculation of prime numbers will be executed synchronously and hence block the event loop from doing something else. Node.js now offers different APIs to nudge callbacks to other parts of the event loop.</p><p><strong>setImmediate()</strong></p><p>In order to make this work we need to chop the calculation of prime numbers into smaller chunks before deferring its processing via setImmediate() o the check phase. To do so we need to come up with artificial breakpoints where we can leave the running code and offer the event loop a window to process other things.</p><p>By chopping the calculation of prime numbers into smaller chunks and deferring its processing via setImmediate() to the subsequent phase of the event loop we can create a chance for parallel incoming requests (to e.g. /health) to fulfill in time (we simplified our life by using promisify() to enable the async/await style):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10b6b542e5033a82f11b8bb88473f7a2/href">https://medium.com/media/10b6b542e5033a82f11b8bb88473f7a2/href</a></iframe><p><strong>setTimeout()</strong></p><p>Comparable to the usage of setImmediate() we can alternatively use setTimeout() as well:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/938d5d871f517112ac527290a01db692/href">https://medium.com/media/938d5d871f517112ac527290a01db692/href</a></iframe><p>Calling setTimeout() with 0 ensures the callback is executed after a minimum threshold of 0 milliseconds within the queue of the timers phase. This will effectively free the poll phase from heavy processing load the same way setImmediate() does.</p><p><strong>Chunk the Workload</strong></p><p>The approach of using setImmediate() or setTimeout() however only works for chopable data structures (e.g. processing a large array) which can be cut up meaningful. For parsing large XML responses to JSON (like in the example above) it might not be trivial or even feasible to do so. In order to split up an XML string into smaller, digestible parts you need to do this at the right location of the string (to not cut XML tags), so parsing it beforehand is required.</p><p>Drawback in our example: While chunking the processing of prime numbers in lists of 10 works efficiently for small numbers, it does not for larger numbers, as the time it takes for checking prime numbers grows exponentially. Additionally we are mixing the implementation of the primes calculation with code for the event loop scheduling, which serves a different aspect.</p><h4>Node.js Worker Threads</h4><p>The worker_threads module became available with Node.js version 10.5 (keep in mind its status is still <a href="https://nodejs.org/docs/latest-v12.x/api/worker_threads.html"><em>experimental</em></a>). It enables the usage of threads which are executed in parallel. In our example the CPU intensive task of calculating prime numbers is now running in its very own thread, moved to a dedicated file worker.js:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/611b40b8293e7219d44511d0f2a78bfa/href">https://medium.com/media/611b40b8293e7219d44511d0f2a78bfa/href</a></iframe><p>The communication between the worker thread and the parent thread are implemented via <em>MessagePorts</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/760b5a75a5b4ae523af192de27c35bda/href">https://medium.com/media/760b5a75a5b4ae523af192de27c35bda/href</a></iframe><p>While waiting for the right event (message) being emitted by the worker the event loop is continuing regularly. Keep in mind that creating a lot of workers might be costly in terms of consuming too many resources.</p><p>Remember to run Node.js with the --experimental--worker flag if you are using a version prior to 11.7.0.</p><h4>Offloading</h4><p>A complete different approach without touching any Node.js specifics is to outsource long running processes to an external system. By pushing the task of parsing the XML to easily digestible JSON to yet another microservice (written in a language more suitable) and asynchronously wait for the results, you can circumvent any problems with blocking code, as the event loop can keep up with incoming requests. However this only makes sense if your use case can cope with some delay which might be added to your overall response time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e016133ac20fc1d9f85093fe84aefb1/href">https://medium.com/media/6e016133ac20fc1d9f85093fe84aefb1/href</a></iframe><h3>Conclusion</h3><p>Node.js is a great framework for implementing high-performance I/O intensive applications. However, due to its blocking execution model, it might actually introduce unexpected performance issues or even imperil the availability or your service. Proper monitoring, error analysis and debugging is crucial to identify the critical parts in this scenario. There are multiple effective ways of dealing with CPU intensive tasks in Node.js without blocking the event loop like partitioning, worker threads or offloading, each with its own drawbacks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb488b59bbd2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/reachnow-tech/con-currently-unavailable-troubleshooting-the-node-js-event-loop-cb488b59bbd2">(Con)currently Unavailable — Troubleshooting the Node.js Event Loop</a> was originally published in <a href="https://medium.com/reachnow-tech">reachnow-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>