<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thomas Weiss on Medium]]></title>
        <description><![CDATA[Stories by Thomas Weiss on Medium]]></description>
        <link>https://medium.com/@thomasweiss_io?source=rss-a22e82772bd9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Pu_hWD0uLaMuVrR-vDbvsQ.jpeg</url>
            <title>Stories by Thomas Weiss on Medium</title>
            <link>https://medium.com/@thomasweiss_io?source=rss-a22e82772bd9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:46:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thomasweiss_io/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Planet-scale event sourcing with Azure Cosmos DB]]></title>
            <link>https://medium.com/@thomasweiss_io/planet-scale-event-sourcing-with-azure-cosmos-db-48a557757c8d?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/48a557757c8d</guid>
            <category><![CDATA[cosmosdb]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[event-sourcing]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Thu, 16 Aug 2018 12:01:00 GMT</pubDate>
            <atom:updated>2018-08-16T12:01:00.511Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(this article originally appeared on </em><a href="https://thomasweiss.io/planet-scale-event-sourcing-azure-cosmos-db/"><em>thomasweiss.io</em></a><em>)</em></p><p><a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing">Event sourcing</a> is an application design principle that I find incredibly powerful and extremely well-suited to the requirements that we routinely have to fulfill when building medium- to large-scale systems.</p><p>In a nutshell, event sourcing says that updates to your application domain should not be directly applied to the domain state; instead, those updates should be materialized as events describing the intended changes and written to an event store. It should also be possible to somehow subscribe to the stream of events received and persisted by the event store in order to apply them to some external state — usually called <em>view</em> — that’s more practical to query than the event stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*K4GRE6b27vTRd6KRWKgOlQ.png" /></figure><p>This approach brings a lot of benefits. First, this event store becomes your canonical source of truth that describes the updates applied to your domain in an unbiased form. You can apply (or rather <em>project</em> in event sourcing lingo) the events to any number of different views, depending on the queries that your system has to implement: from a relational database to a search engine to a data lake. Also, the very nature of the event log gives you invaluable auditing capabilities. You can literally go back in time and incrementally replay all the changes applied to your domain, something that’s just impossible if you overwrite your domain state every time.</p><p>For a deeper dive into the motivations behind this design principle, I highly recommend to watch <a href="https://www.youtube.com/watch?v=JHGkaShoyNs">this talk by Greg Young</a>.</p><p>Event sourcing becomes even more powerful if you combine it with <a href="https://en.wikipedia.org/wiki/Domain-driven_design">Domain-Driven Design</a> where each <em>aggregate</em> manages its own local state and has its own dedicated event stream within the event store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*v4R784QBZwIQFz-onEXJFQ.png" /></figure><p>This way of modeling your domain allows for greater scalability as each aggregate can be considered as an independent and isolated (micro?) service, with all the benefits this can carry in terms of development, deployment and operations.</p><h3>The challenges of global event sourcing</h3><p>The software ecosystem around the implementation of event sourcing techniques has become pretty mature over the past 5 years or so:</p><ul><li>Specialized data stores that meet the particular requirements of an event store (like exposing events as streams that one can subscribe to) have emerged; see <a href="https://eventstore.org/">Event Store</a> or <a href="http://kafka.apache.org/">Apache Kafka</a></li><li>Framework and libraries that provide building blocks for event-sourced applications are available on most platforms; see <a href="http://docs.geteventflow.net/">EventFlow</a> on .NET or <a href="http://www.axonframework.org/">Axon Framework</a> on Java</li></ul><p>But deploying and operating an event-sourced system on a large and possibly global scale (i.e. with no assumed limit on the number of events and spread over multiple datacenters across the world) presents very specific challenges. Not only do you have to make sure that your system can sustain high throughputs of events, but you also have to make that happen reliably over different, replicated locations.</p><p>In the rest of this article, I will explore how <a href="https://azure.microsoft.com/en-us/services/cosmos-db/">Azure Cosmos DB</a> can serve as a distributed event store and how it yields unique benefits in such a use-case.</p><p>Funnily, I’ve realized that <a href="https://medium.com/@thomasweiss_io/distributed-event-log-with-couchdb-a36a63cdf337">one of the very first blog posts I’ve ever written</a> was on the very topic of multi-datacenter event sourcing! It’s rather fascinating to witness how new products and platforms enable complex scenarios to be implemented with ever-increasing levels of simplicity, performance and reliability.</p><h3>Use the change feed to subscribe to events</h3><p>As mentioned above, an event store should expose new events being logged as streams that one can subscribe to. Although it is possible to poll for these changes, it’s much more efficient to have the new events being pushed to whatever process that needs to react on them (typically, projecting the events into views).</p><p>Cosmos DB exposes a <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed">change feed</a> that streams the changes happening to each of your database’s containers; so every time an event is written, the container’s change feed pushes the new document to anyone listening. There are many ways to consume this stream:</p><ul><li>you can <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed#using-the-sdk">query it</a> using using the Cosmos DB <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-dotnet">SQL SDK</a>,</li><li>you can use the <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed#using-the-change-feed-processor-library">Change Feed Processor library</a> that makes it easy to distribute the processing of a change feed over multiple workers,</li><li>you can use “serverless” <a href="https://docs.microsoft.com/en-us/azure/azure-functions/">Azure Functions</a> as they provide a <a href="https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-cosmos-db-triggered-function">built-in Cosmos DB trigger</a> that fires a function every time there’s new data available from the change feed of the container you’ve specified (you can read more about this technique in <a href="https://thomasweiss.io/denormalizing-data-azure-functions-cosmos-db-change-feed/">my previous post</a>).</li></ul><p>The Cosmos DB change feed doesn’t only stream new events in real-time, but can also replay the sequence of logged events from either a particular point in time or the very beginning (i.e. the creation of your container). This is necessary if you’ve updated the way you are projecting your events to a view or added a new view to your system, which requires to feed all past events through your updated or new projections.</p><p>Note that it’s currently <em>not</em> possible to apply filters to the change feed, so every subscriber receives all the events being logged to the container. This means that if you’ve applied some Domain-Driven Design to your data model and have different sets of projections for each aggregate type, you would have to implement some basic event routing to dispatch the events received from the change feed to the appropriate projections.</p><h3>Limitless scaling of event storage</h3><p>In most projects, starting with an initial system design that can accommodate massive scale is arguably some kind of premature optimization. You will most probably have to cope with moderate traffic at the beginning… and no traffic at all if the product you’re working on doesn’t take off (been there, done that!). If your product does meet success, then you can incrementally update your architecture to make it more scalable, as long as your initial design provided for modularity and extensibility.</p><p>That story is a bit different with event sourcing. In an event-sourced system, that ability to incrementally improve your overall scalability is actually made possible by the event store, as it lets you replace or add new views by replaying all historical events whenever required. This means that the event store really becomes the cornerstone of your data orchestration strategy, which in turn makes it challenging to replace or upgrade without significantly impacting your operations.</p><p>That’s a use-case where Cosmos DB’s elastic scalabity really shines. By following basic — but essential — data modeling techniques, it’s entirely possible to design an event store that can accommodate literally <em>any</em> kind of scale and <em>any</em> volume of events.</p><p>Cosmos DB achieves this by partitioning the data it stores. Each document (or event in our case) is assigned to a logical partition and each logical partition is stored in a physical partition (which roughly translates to a server instance). The shuffling of logical partitions across physical partitions is transparently managed by Cosmos DB, re-balancing your data across different servers and adding more servers to your usage without any downtime. So what is left for us to take care of is the assignment of events to logical partitions, and that’s probably the most important bit to understand in order to design a scalable data model over Cosmos DB.</p><p>When creating a Cosmos DB container, we have to specify a partition key, which is a field that the database will look for in our documents to decide which logical partition it belongs to. So all documents with the same partition key value will end up in the same logical partition. Now what&#39;s also crucial to understand is that a good partitioning strategy ensures an even distribution of documents across logical partitions and an even distribution of requests across those partitions. In particular:</p><ul><li>write requests should hit a wide range of different logical partitions to avoid “hot” partitions that would become write bottlenecks,</li><li>most read requests should ideally hit one and only one logical partition, so Cosmos DB can directly target the physical partition (i.e. the server) that has all the required data without the need to aggregate results across multiple servers.</li></ul><p>In practice, finding a partitioning strategy that ticks all these boxes is not always a trivial exercise, but in the case of an event store data model it happens to be ridiculously simple! By using a concatenation of the event’s aggregate type and aggregate ID (like user-123 for example), you make sure that all the events related to a single aggregate end up in the same logical partition. As a consequence:</p><ul><li>write requests <em>will</em> hit different partitions,</li><li>the most frequent read request, which is retrieving all events for a specific aggregate in order to hydrate its state, <em>will</em> hit a single partition.</li></ul><p>As it is currently not possible to specify a partition key that’s composed of multiple fields, we have to add a composite property to our events that is the actual concatenation of aggregate type and aggregate ID, and use that field as the partition key. This technique is also described <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/partition-data#form-partition-key-by-concatenating-multiple-fields">here</a>.</p><p>Note that there is a hard limit of 10 GB of storage per logical partition, but I guess it’s pretty safe to consider that 10 GB should be more than enough to store all events related to one aggregate (otherwise the way you’ve modeled your domain into aggregates may rather be flawed).</p><h3>Guaranteed performance</h3><p>In an event-sourced system, the event store is the backbone of your data processing pipeline and as such, must ensure strong guarantees in terms of:</p><ul><li>availability; it’s hardly possible to run in any kind of degraded state if the event store is down, as you can’t accept any write request and may serve stale reads if you’ve stopped projecting events to your views,</li><li>latency; low response times mean faster writes and faster consistency of the views,</li><li>throughput; limiting the event store’s throughput directly limits the overall throughput of your application.</li></ul><p>Cosmos DB provides financially backed <a href="https://azure.microsoft.com/en-us/support/legal/sla/cosmos-db/">Service Level Agreements</a> covering each of those metrics and more. Also, it gives the unique ability to elastically scale its throughput to accommodate your application’s performance requirements; you can <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/set-throughput">adjust each of your containers’ throughput</a> from hundreds to millions of requests per second by dragging a cursor or issuing an API call.</p><h3>Going global in a click</h3><p>We usually think about scalability in terms of request throughput or data size, but applications that have a global user base also need to eventually expand the geographical footprint of their infrastructure. By multiplying their points of presence across the globe, they ensure minimal latency and optimal user experience wherever their users are.</p><p>Geo-scaling stateless compute has become pretty straightforward as you just have to deploy your code to different datacenters. It’s much more complicated when it comes to data! You obviously want your data store to be as close to the compute as possible, otherwise the benefits of geo-scaling would be greatly limited. This means that your data has to be replicated across every point of presence so each server has direct, low-latency access to the data.</p><p>One of the biggest advantages of event sourcing is that your event log is your single-source of truth. Because all the views contain data that is derived from the event log, they can be discarded and rebuilt at will without any fear of data loss. So geo-scaling an event-sourced system only requires the geo-replication of the event store, with all derived views maintained locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOD958Ny11PqVmjqCVeA7w.png" /></figure><p>The only problem is, replicating data is usually hard and replicating it globally is even harder. You have to make sure that your data remains consistent across all replicas, minimize the latency of the replication process, guarantee the overall availability of your cluster… These are responsibilities that you want to hand over to a solution that has a proven track record of robustness and reliability.</p><p>Cosmos DB takes the pain out of global replication by providing a real (and currently unmatched) turnkey experience. Replicating a container is simply achieved by selecting which regions you want your container to be replicated to and hitting “Save” from the Azure portal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*suy85NZNV4J-e2F9EAqiTg.png" /></figure><p>This can also be done programmatically as the same level of configuration is available from the <a href="https://docs.microsoft.com/en-us/rest/api/cosmos-db-resource-provider/">Cosmos DB REST Resource Provider</a>.</p><p>Cosmos DB makes the geo-replicated nature of your event store completely transparent to your application, as its multi-homing API seamlessly directs your requests to the nearest available region. This means that you can add new points of presence by simply replicating your event store to a new region and deploy your app there, without any change in your application code.</p><h3>Ensure event log consistency with unique keys</h3><p>As easy as it is to expand our global footprint, we have to keep in mind that adding more points of presence to our application also increases the level of concurrency that we have to deal with at the event log level. Read concurrency isn’t a problem really as the partitioning strategy we’ve detailed above ensures an ideal distribution of events across logical partitions that map to individual aggregates. Write concurrency, however, requires more attention.</p><p>Although it is possible to control the write access to the event store from a single aggregate in a single region (I would personally do that with an actor framework like <a href="https://dotnet.github.io/orleans/">Orleans</a> or <a href="http://getakka.net/">Akka.Net</a> to implement aggregates as single-threaded singletons), you can’t realistically practice the same control with aggregates spread over multiple regions without drastically limiting your scalability. In other words, you may have multiple instances of the same aggregate, living simultaneously in different regions and issuing conflicting events concurrently.</p><p>The trick here is to rely on the event sequence number. Each event stream (that is, related to a single aggregate) is composed of a series of events with monotonically increasing sequence numbers; every time an event is added to the stream, we take the last event’s sequence number and increment it. So each and every event can be uniquely identified by the combination of its aggregate type, aggregate ID and sequence number. All we have to do then is to declare a <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/unique-keys">unique key</a> composed of these 3 properties so any attempt to create an event that matches those properties on an existing event will fail with some explicit error. Getting this error would mean that an other instance of the aggregate has concurrently updated the event stream, so we should refresh our aggregate’s state and try again.</p><p>Note that unique keys are only scoped to logical partitions, meaning that you can have duplicate keys across different partitions. That isn’t an issue at all in our case as we have already partitioned our events by aggregate type and aggregate ID; another sign that Cosmos DB’s features and constraints fit the event store use-case really well!</p><h3>Maximum write availability</h3><p>The replication method described so far lets you add read regions to your database, effectively increasing your read availability across the globe. By default, this setup still has a single write region, which means that read requests will be routed to the nearest read regions by the multi-homing API, but every write request will hit the same, single write region. While this may be adequate for read-heavy applications, you may also want to improve the write latency of your event store; two different approaches are possible here.</p><h3>Trigger a manual failover to change the write region</h3><p>The first option is suitable for global applications that see most of their traffic during business hours, or most generally at day time. Here you want to maximize your write availability around midday in each of the regions where your application is deployed. This technique is sometimes nicknamed “follow-the-sun”, as you slide your resources across the globe over a 24 hour period.</p><p>This can be achieved by <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/regional-failover#ManualFailovers">triggering a manual failover</a> of your write region. This feature was initially developed to let businesses test their resilience to issues and outages, but it’s also a convenient way to change the location of your write region on a schedule. Note that the failover operation is executed without any downtime or data loss, so it really is a safe way to perform this operation. If you’re interested in learning more about this technique, check <a href="https://anthonychu.ca/post/change-cosmos-db-locations-schedule-azure-functions-msi/">this great blog post</a> by Cloud Developer Advocate <a href="https://developer.microsoft.com/en-us/advocates/anthony-chu">Anthony Chu</a>.</p><h3>Leverage the new multi-master capability</h3><p>If your data access patterns are not as predictable and you need to ensure the best write availability anywhere and anytime, then your application may be a good candidate for the new <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/multi-region-writers">multi-master support</a> that was announced at the Build 2018 conference. Just as its name implies, this feature lets you have more than one region accepting writes within your global deployment. Cosmos DB ensures that every event written in every write region gets replicated to all other regions.</p><p>Multi-master support is still in private preview at the time of this writing and I didn’t have the opportunity to test it yet, so I’m not able to go into further details for now. Notably, I’m not sure how the unique key constraint we described above is enforced over multiple write regions — I would assume that any violation of the constraint ends up as a conflict that <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/multi-region-writers#conflict-resolution-with-multi-master">would have to be resolved</a>. I will update this section as soon as I’ve clarified this part.</p><h3>Ready to take on the world!</h3><p>If you want to start prototyping an event store based on Cosmos DB, you can use the <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator">local emulator</a> available for Windows or <a href="https://azure.microsoft.com/en-us/try/cosmosdb/">try Cosmos DB for free for a limited time</a>.</p><p>Whether you have already implemented event sourcing solutions on Cosmos DB or any other data store, or plan to design such a solution and seek some advice, please feel free to share your feedback in a response!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48a557757c8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Denormalizing your data with Azure Functions and Cosmos DB's change feed]]></title>
            <link>https://medium.com/@thomasweiss_io/denormalizing-your-data-with-azure-functions-and-cosmos-dbs-change-feed-c4a8687cda88?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4a8687cda88</guid>
            <category><![CDATA[cosmosdb]]></category>
            <category><![CDATA[azure-functions]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Thu, 31 May 2018 08:26:08 GMT</pubDate>
            <atom:updated>2018-05-31T08:26:08.814Z</atom:updated>
            <content:encoded><![CDATA[<h3>Denormalizing your data with Azure Functions and Cosmos DB’s change feed</h3><p><em>(this article originally appeared on </em><a href="https://thomasweiss.io/denormalizing-data-azure-functions-cosmos-db-change-feed/"><em>thomasweiss.io</em></a><em>)</em></p><p>In this article, we’re going to explore denormalization, a common data processing pattern and how it can be easily implemented using Azure Functions to process the change feed of Cosmos DB containers.</p><h4>A scalable aggregation technique</h4><p>Let’s start by exploring what is denormalization and why it is required in some situations.</p><p>Data stores that achieve true horizontal scalability do so by partitioning, i.e. splitting your data over a cluster of servers. In Cosmos DB, this means that not only will the different containers of your database (i.e. collections/tables/graphs depending on the API you’re using) be spread across different machines, but even the containers themselves may be split over multiples machines (unless they are “fixed capacity” ones).</p><p>Because all your data is not on a single machine, aggregation capabilities are limited. It’s not that cross-container aggregations would not be possible, but as they would require costly synchronization between multiple servers (your containers may be spread over dozens or hundreds of machines), the performance of such queries would be greatly affected. And because Cosmos DB is all about performance, it doesn’t expose such capability.</p><p>I really consider this as the kind of constraint that forces us to adopt sane data modeling and processing practices. It actually reminds me of a recent project I’ve been working on and that was implemented on a database that has a very rich and flexible query language, allowing us to perform literally any kind of aggregation and projection across the entire dataset. In retrospect, I think that this flexibility actually harmed us more than it helped, at least in some cases. It was so easy to respond to new product requests by amending existing queries, squeezing more joins and projections that we eventually ended up with monster queries that were difficult to maintain. Stricter constraints would have required us to implement <strong>more scalable</strong> methods, like denormalization.</p><h4>Our use-case</h4><p>So let’s consider the simple scenario of a blogging platform where we would store Users and Posts. Posts would reference their authors with an authorId field. Let&#39;s also imagine that we would store these 2 types of entities in separate collections because posts would also contain views, comments and other statistics, and so require more write throughput than users.</p><p>Now we would probably serve a “feed” of the most recently published posts and this feed would aggregate the authors directly in each post, in the following form:</p><pre>{<br>  &quot;id&quot;: &quot;post-id&quot;,<br>  &quot;content&quot;: &quot;...&quot;,<br>  &quot;author&quot; : {<br>    &quot;username&quot;: &quot;author-username&quot;,<br>    &quot;pictureUrl&quot;: &quot;https://...&quot;<br>  },<br>  &quot;publicationTime&quot;: &quot;2018-05-20T12:00:00Z&quot;<br>}</pre><p>With a relational database, this would be achieved by setting the authorId of posts as a foreign key, then querying both Posts and Users tables with a JOIN to construct this feed in real-time. But that&#39;s not an option here; instead, we will <strong>denormalize</strong> our data by maintaining this feed in a separate, dedicated container that&#39;s ready to be queried. This container will effectively contain a copy of the data present in the Posts and Users containers, with the authors already aggregated in the posts.</p><p>Note that this technique is anything but new. Relational databases like SQL Server or PostgreSQL can expose <a href="https://en.wikipedia.org/wiki/Materialized_view">materialized views</a> that precompute and cache expensive queries. Setting a denormalized container with Cosmos DB takes a bit more work than setting such a denormalized view though, as we need to gather and join the data ourselves.</p><h4>Implementation</h4><p>In order to denormalize our feed, we would need to perform the following actions:</p><ul><li>whenever a post is created or updated, add its author to the post and upsert it in the Feed container</li><li>whenever a user is updated, update all posts authored by this user in the Feed container to replace their author with the updated user</li></ul><p>At this point you may think that adding these “hooks” inside your application’s code would be cumbersome and error-prone. The good news is, they can (and should!) be implemented outside of your application by leveraging Cosmos DB’s <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed">change feed</a>. Each Cosmos DB container exposes an API that streams the changes happening to that container; so every time a document is added or updated, the container’s change feed pushes the new document to anyone listening.</p><p>There are a couple of ways to listen to change feeds. The Cosmos DB SQL SDK <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed#using-the-sdk">can query it</a> and there’s also a powerful <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed#using-the-change-feed-processor-library">Change Feed Processor library</a> that makes it easy to distribute the processing of a change feed over multiple workers. My personal favorite is to use <a href="https://docs.microsoft.com/en-us/azure/azure-functions/">Azure Functions</a> as they provide a <a href="https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-cosmos-db-triggered-function">built-in Cosmos DB trigger</a>. With this trigger, your function gets called every time there’s new data available from the change feed of the container you’ve specified.</p><p>So we’re going to implement 2 Azure Functions using the Cosmos DB trigger: one that’s watching the Posts change feed and an other one watching the Users change feed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwsfFFG_bOlGXsnhF8oL8g.png" /></figure><p>Let’s start with the first one:</p><pre>[FunctionName(&quot;PostsTrigger&quot;)]<br>public static async Task Run([CosmosDBTrigger(<br>  databaseName: &quot;denorm&quot;,<br>  collectionName: &quot;posts&quot;,<br>  ConnectionStringSetting = &quot;CosmosDbConnectionString&quot;,<br>  LeaseCollectionName = &quot;leases&quot;)]IReadOnlyList&lt;Document&gt; documents)<br>{<br>  if (documents != null &amp;&amp; documents.Count &gt; 0)<br>  {<br>    foreach (var post in documents)<br>    {<br>      // fetch author<br>      var authorId = post.GetPropertyValue&lt;string&gt;(&quot;authorId&quot;);<br>      var author = await _documentClient.ReadDocumentAsync(<br>        UriFactory.CreateDocumentUri(&quot;denorm&quot;, &quot;users&quot;, authorId));</pre><pre>      // set author to post and upsert it<br>      post.SetPropertyValue(&quot;author&quot;, author.Resource);<br>      await _documentClient.UpsertDocumentAsync(<br>        UriFactory.CreateDocumentCollectionUri(&quot;denorm&quot;, &quot;feed&quot;), <br>        post);<br>    }<br>  }<br>}</pre><p>Our second function will have to update the author field of all posts authored by the user that has been updated, so we will use a <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/programming">stored procedure</a> to implement that. Here&#39;s the code:</p><pre>function updateUser(user) {<br>  var collection = getContext().getCollection();</pre><pre>  var accept = collection.queryDocuments(<br>    collection.getSelfLink(),<br>    `SELECT * FROM feed f WHERE f.authorId = &#39;${user.id}&#39;`,<br>    function (err, feed) {<br>      if (err) throw err;</pre><pre>      for (i in feed) {<br>        var post = feed[i];<br>        post.author = user;</pre><pre>        var accept2 = collection.upsertDocument(<br>          collection.getSelfLink(),<br>          post);<br>        if (!accept2)<br>          throw new Error(&#39;upsert not accepted&#39;);<br>      }<br>    });</pre><pre>  if (!accept)<br>    throw new Error(&#39;query not accepted&#39;);<br>}</pre><p>And the corresponding function that will call that stored procedure every time a user is updated:</p><pre>[FunctionName(&quot;UsersTrigger&quot;)]<br>public static async Task Run([CosmosDBTrigger(<br>  databaseName: &quot;denorm&quot;,<br>  collectionName: &quot;users&quot;,<br>  ConnectionStringSetting = &quot;CosmosDbConnectionString&quot;,<br>  LeaseCollectionName = &quot;leases&quot;)]IReadOnlyList&lt;Document&gt; documents)<br>{<br>  if (documents != null &amp;&amp; documents.Count &gt; 0)<br>  {<br>    foreach (var user in documents)<br>    {<br>      await _documentClient.ExecuteStoredProcedureAsync&lt;string&gt;(<br>        UriFactory.CreateStoredProcedureUri(<br>          &quot;denorm&quot;, &quot;feed&quot;, &quot;updateUser&quot;),<br>        user);<br>    }<br>  }<br>}</pre><h4>Denormalization on auto-pilot</h4><p>Nothing else is needed for the feed data to automatically reflect any update in posts or users. And because the Feed container is just a normal Cosmos DB container, every feed document is fully indexed by default so you can issue queries that efficiently filter by authorId or sort by publicationTime.</p><p>This simple example showed how to perform a basic aggregation between 2 containers, but more elaborate operations can be implemented in your denormalization pipeline, like adding new fields that are combinations of existing fields (for more efficient filtering or different sorting).</p><p>Azure Functions really feel like the best way to host such a pipeline because:</p><ul><li>they’re completely independent of your application code and can be considered as an standalone “extension” to your Cosmos DB database,</li><li>if you’re using Functions in a <a href="https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#consumption-plan">consumption plan</a>, you pay only when your functions are running and they will scale out automatically when required.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4a8687cda88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Text Analytics for simple Natural Language Processing]]></title>
            <link>https://medium.com/@thomasweiss_io/azure-text-analytics-for-simple-natural-language-processing-3548547cd33d?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3548547cd33d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bots]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Tue, 30 Jan 2018 02:55:30 GMT</pubDate>
            <atom:updated>2018-02-05T08:43:28.570Z</atom:updated>
            <content:encoded><![CDATA[<p>Among all the Azure Cognitive Services one can consider when implementing a scenario that requires natural language processing, <a href="https://www.luis.ai/">LUIS</a> is the obvious choice. Short for “Language Understanding Intelligent Service”, LUIS has been around for many years now and is the actual engine that makes Cortana understand you.</p><p>LUIS lets you design your domain by defining <em>intents</em> and <em>entities</em>. You can think of intents as the verbs and entities as the nouns of the sentences you expect it to understand. So when you fire up Cortana and say “Wake me up tomorrow at 6”, “set an alarm” might be the intent and “next day at 6am” the entity. Once your model is designed, you train it by providing utterances, which are examples of inputs, review how LUIS breaks them down in intents and entities, and correct this classification if it’s wrong.</p><p>LUIS is undeniably very powerful and once you get comfortable with the process of domain modeling, you can design extensive and complex models. But there’s a significant head start required when tackling a new project; you have to figure out the best way to define your domain in terms of intents and entities, and you should then dedicate enough time to train your model in order to achieve acceptable performance.</p><h3>Looking at Azure Text Analytics</h3><p><a href="https://azure.microsoft.com/en-us/services/cognitive-services/text-analytics/">Azure Text Analytics</a> is part of the same Cognitive Services family as LUIS. When I present the spectrum of services available on the Microsoft AI stack, I usually classify them along 2 dimensions: <em>customization</em> vs. <em>ease of integration</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wz4rZLVRQaKSZRwS.png" /></figure><p>In this chart, Text Analytics sits one layer below LUIS because it doesn’t require you to train it with your own data. It is already trained by Microsoft on stock data that makes it ready-to-use.</p><p>Text analytics can extract 3 kinds of information from the text you send to its API:</p><ul><li>language detection: which language is predominantly used</li><li>sentiment analysis: as a single number ranging from 0 (negative) to 1 (positive)</li><li>key phrases: words or groups of words describing the main points</li></ul><p>The main use-case for key phrase extraction is to identify general ideas from large, unstructured text. But let’s see how it behaves with commands one would typically send to LUIS to analyse their meaning (note that this test has been made in January 2018 and only reflects the service’s performance at this point in time).</p><blockquote><em>Wake me up tomorrow at 6</em></blockquote><p>No luck here, as the service didn’t identify any key phrase</p><blockquote><em>Set an alarm tomorrow at 6</em></blockquote><p>Key phrase identified: “alarm”</p><blockquote><em>Turn on the oven</em></blockquote><p>Key phrase identified: “oven”</p><blockquote><em>Where is the nearest library?</em></blockquote><p>Key phrase identified: “nearest library”</p><blockquote><em>How will the weather be this weekend?</em></blockquote><p>Key phrases identified: “weather”, “weekend”</p><blockquote><em>Call a taxi</em></blockquote><p>Key phrase identified: “taxi”</p><p>These examples obviously show that most of the time, Text Analytics successfully extract the entities, i.e. the most important “things”, but does not provide any clue about the intent of the sentences as the verbs just seem to be ignored.</p><h3>Good enough for simple scenarios or quick prototypes</h3><p>It comes to no surprise that Text Analytics is not a replacement for LUIS. But there is certainly a class of use-cases where it can be used in place of LUIS, especially if you’re dealing with situations where intents are implicitly carried by the entity: “taxi” most probably means that the user wants to call a taxi; “nearest library” means that the user is looking for such a place.</p><p>Using Text Analytics can also help to kickstart a prototype where you don’t need to exhibit comprehensive NLP, but just want to use it as a placeholder that’s readily functional. You may replace it with LUIS later down the road, once you have some time to dedicate on building and training a LUIS model.</p><h3>Combine them to get the best of both</h3><p>It’s also important to note that a LUIS model is built for a specific language. LUIS does not perform any kind of language identification and will assume the text input to be in a specific language. So for multi-language interfaces, like chatbots exposed to a broad multi-national audience for example, you need to identify the language used by the user first.</p><p>By submitting the input to Text Analytics, not only do you get language identification in return, but you can also:</p><ul><li>use key phrase extraction to potentially disambiguate LUIS’ intent/entities analysis</li><li>use sentiment analysis to adapt the response to the user</li></ul><p>Visit the <a href="https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/overview">Text Analytics documentation</a> to learn more about this service and feel free to share your feedback in the comments below!</p><p><em>Originally published at </em><a href="https://thomasweiss.io/azure-text-analytics-for-simple-natural-language-processing"><em>thomasweiss.io</em></a><em> on January 30, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3548547cd33d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Walking backwards: a simple technique to guide live issue resolution]]></title>
            <link>https://medium.com/@thomasweiss_io/walking-backwards-a-simple-technique-to-guide-live-issue-resolution-7a3b3d29322?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a3b3d29322</guid>
            <category><![CDATA[systems-engineering]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Wed, 20 Dec 2017 10:35:45 GMT</pubDate>
            <atom:updated>2017-12-20T10:35:45.057Z</atom:updated>
            <content:encoded><![CDATA[<p>Those of you who have been exposed to issues happening in production systems know that it’s a very difficult situation to deal with. There’s a lot of pressure to fix things as soon as possible, and those issues are obviously unexpected so we would rather be doing something else (like, sleeping!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1GqLf2BVzkGdPKF6TYKlw.jpeg" /></figure><h4>Keep calm and trust only your data</h4><p>There are some best practices to follow and good reflexes to have in those stressful situations. Mark Simms gave <a href="https://channel9.msdn.com/Events/Build/2017/B8082">an excellent talk</a> on that topic at Build 2017 and I can’t recommend it enough. Mark’s main tips are:</p><ul><li>resist the urge to rush, keep your head cool and “slow down to move fast”</li><li>trust data, not your intuition (that is, if you do have telemetry data to work on!)</li></ul><p>Those are excellent pieces of advice, but even when you manage to keep calm and have valuable data at hand, it can still be challenging to decide where to start and how to proceed. Data can be overwhelming and give you wrong hints about the issue. Sometimes you may even come across data that reveals a totally different issue, and it’s tempting to explore that path and lose focus on the most urgent problem.</p><p>To handle those situations efficiently, I’m applying a technique I’ve nicknamed “walking backwards”. Now I’m not claiming that this is my invention; it’s nothing but common sense but I thought I would explain its rationale and illustrate it with some example. Here it goes.</p><h4>Walking backwards along the positive path</h4><p>We usually start the exploration of an issue with “the thing that doesn’t work” and try to find data that would explain why it doesn’t work. What I do instead, starting from “that thing that doesn’t work”, is to ask myself “<strong>what would make it work?</strong>”. Or starting from “something that didn’t happen”, “<strong>what would make it happen?</strong>”.</p><p>The logic behind that is that, in the complex systems we’re dealing with, the root cause of a problem often resides many layers or levels away from the visible symptoms. How many times have you wondered “How can that be broken? There’s no reason it doesn’t work”. And that’s usually because thinking about the potential reasons that may explain the direct symptoms is a dead-end. And so we start looking at random data, hoping to make sense of the issue by accident. But following what I would call the “positive path”, that is <em>the backwards chain of events that would </em><strong><em>not</em></strong><em> have produced the issue</em> we’re investigating, sets a trail we can walk until we eventually find the root cause.</p><p>Let me illustrate this technique with some real life example. I’m currently working on the back-end systems of <a href="https://www.keakr.com/">Keakr</a>, a France-based social network app for urban music lovers. Keakr users can upload videos and share them with their friends and followers.</p><h4>In the context of a real production issue</h4><p>A couple of months ago, some users suddenly started to report that videos were loading slowly. Now before we start the exploration of that issue, let me briefly explain how videos are processed in Keakr:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*-ZTKjTHecYHs87jGI5pZNg.png" /></figure><p>Every time a video is created (1), the frontline web servers store in blob storage the raw mp4 file coming from the app (2), then dispatch a request (3) that’s asynchronously received by some background workers to transcode the video in a streaming format using Azure Media Services (4). Because transcoding is not a quick operation, and because there may be a queue of videos waiting to be transcoded, we serve the raw mp4 until the video is ready to be streamed.</p><p>Now back to our problem. Follow me as we apply the “walking backwards” technique, starting from the reported symptoms:</p><blockquote>Some videos are slow to load</blockquote><p>What makes videos <strong>fast</strong> to load?</p><ul><li>Serving them in streaming format. A quick check in the database revealed that many recent videos had not been transcoded.</li></ul><blockquote>Transcoding of video fails</blockquote><p>What makes transcoding of videos <strong>succeed</strong>?</p><ul><li>Successful operation of the background workers. Looking at our logs and metrics, the workers seemed to work fine, processing other requests without any problem.</li><li>The reception and execution of transcoding requests. As workers store incoming requests in some cold storage for auditing, we found out that no such requests had been stored recently.</li></ul><blockquote>Transcoding request don’t arrive in the workers</blockquote><p>What makes those request <strong>arrive</strong>?</p><ul><li>Successful operation of the message queue. As stated previously, other requests were processed by the workers so the message queue was running fine.</li><li>Logical deduction was that the requests were not fed into the message queue.</li></ul><blockquote>Transcoding requests are not issued</blockquote><p>What makes the requests being <strong>issued</strong>?</p><ul><li>Successful execution of the “create video” HTTP request handler on the web server. That request handler (1) stores the raw mp4 in blob storage and some meta-data in the database, (2) dispatches a push notification to the user and finally (3) sends the transcoding request on the message queue. Looking at the blob storage and database, we knew that step (1) completed, so we dig into the dispatch of push notifications… to realize that the certificate we were using to interface with APNS (Apple’s push notification system) had expired! This led to an uncaught exception, stopping the execution at that point and preventing the transcoding requests to be issued.</li></ul><h4>From the symptoms to a very unexpected root cause</h4><p>Now take a moment to consider the root cause and its visible symptoms: <em>videos were loading slowly because a push notification certificate had expired</em>. There is just no way that we would have spontaneously associated those things together.</p><p>We could have spent an awful lot of time trying to investigate the CDN or the video transcoding pipeline, sending test videos to Azure Media Services and eventually finding out that this part was working fine. It is only by applying a rather simple technique that we were able to guide our analysis down to the source of the problem in as little time as possible.</p><p>Are you following similar methods when troubleshooting production systems? Maybe variants of what I’ve described, or some totally different approach? Please share your thoughts and suggestions in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a3b3d29322" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring performance issues with Azure Application Insights]]></title>
            <link>https://medium.com/@thomasweiss_io/exploring-performance-issues-with-azure-application-insights-962386d53237?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/962386d53237</guid>
            <category><![CDATA[application-insights]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Wed, 13 Dec 2017 03:53:03 GMT</pubDate>
            <atom:updated>2017-12-13T03:53:03.306Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been using <a href="https://docs.microsoft.com/en-us/azure/application-insights/">Application Insights</a> on most of the projects I have been working on for the past couple of years. It is an impressive, comprehensive service that lets you monitor your application’s health and behavior, and also easily investigate issues by exploring the telemetry it gathers from your app. In this post, I will demonstrate how Application Insights can be used to perform the first steps one would typically follow when exploring a performance anomaly.</p><h4>Looking at a real life example</h4><p>Among the essential metrics one should always monitor in order to have a basic understanding of a web app’s activity are the <em>number of requests served</em> and the <em>average response time</em>. I like to put those two on the same chart in order to correlate them (response time typically increases with throughput, but I will leave that for a future post!).</p><p>Imagine that you are checking your dashboard one morning and see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*M3g2EnRdVT-BX-RksFzP6g.png" /></figure><p>It seems that we got a pretty nasty spike in response time (the green line), peaking at an average of almost 10 seconds around 10:38pm. That spike was probably too brief to trigger an alert, so it was most likely a transient issue. Still, it’s a good use-case to explore how we can use Application Insights to dig deeper into what happened.</p><p>As we’re dealing with a performance issue, a good place to start is the “Performance” section that is available under the “Investigate” category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*-Gq39eEDgqbpvTFmX0N8kA.png" /></figure><p>Here we see pretty much the same information as in the previous chart: <em>response times</em> and <em>request count</em>.</p><h4>Focusing on the event</h4><p>Let’s zoom in to have a better sense of the actual duration of the issue. We just have to drag and drop the blue time sliders and center the time window around 10:30pm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*S0-vL_1CPdGDlskrG_YRiA.png" /></figure><p>This shows that the issue lasted for around 5 minutes and also that the response time degradation was much worse than originally thought! That’s because we look at an <em>average </em>of the response time, which gets more accurate as we reduce the aggregation time window.</p><h4>Investigating the underlying cause</h4><p>Time to find out what caused the spike. From this same chart, we can display the average CPU load of our web app instead of the request count to see if there’s any correlation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*KJisqjifYyBQKYjXQSDNpQ.png" /></figure><p>The CPU load did increase during the event, but to a very reasonable maximum of around 10%. So that was definitely not our contention point and it probably increased as a result of the underlying root cause.</p><p>Just below this chart lies a list of “operations” — actually, all the requests issued to your web app within the time window — ordered by duration. This can help to find out if there’s any particular operation that caused the issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*Rhn0Qsecrr2-0T8zqYNFNw.png" /></figure><p>In our case, it seems that the problem was impacting all requests, pointing to a systemic issue rather than anything application-related. Also note that once again, the average latency we saw earlier was, as any average, very optimistic! Some requests actually took more than a minute to complete…</p><h4>Correlating with dependencies</h4><p>The next thing to look at is the dependencies, which are all the external services that our web app uses to serve its requests. That would typically include databases, caches, social platforms, notification dispatchers etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*5Df0Az9xOJJ0uiaZQ-LG6g.png" /></figure><p>Bingo! We got a significant spike in dependency latency at the very same time, so the overall degradation in our web app’s response time is most likely coming from a dependency. But which one? Under that chart, we got the same kind of “operations” table, listing all dependency operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*aOU3wZ2GSZODEERDyU0d7Q.png" /></figure><p>It seems that our slow dependency was the service behind the 10.1.0.6 IP address — a database in our case. But our web app interacts with half a dozen different dependencies, so we should verify that the database was the only one having problems. Time to unleash the full power of Application Insights’ Advanced Analytics!</p><h4>Moving up a gear with Advanced Analytics</h4><p>Under the hood, Application Insights is powered by a very powerful query engine code-named Kusto (Kusto is now also integrated with Azure Log Analytics, and <a href="https://docs.loganalytics.io/">here</a> is the best place to learn about the query language). Most of the charts and tables that you see on the Azure portal’s Application Insights pane are the results of Kusto queries. What’s great is that you can have direct access to those queries from the “Analytics” links available in most sections. In our example, we have a “View in Analytics” drop-down menu that points to the Analytics side of each chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*7KTLxhSDnIc2G6HWP_uIpQ.png" /><figcaption>Notice how each chart gets highlighted as you scroll through the drop-down!</figcaption></figure><p>Choosing “Trends: response time” leads us to the Advanced Analytics portal of Application Insights and shows us the exact query that got executed to create the previous chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*pfpq2nLeqHyQ3BOKR40CtA.png" /></figure><p>And just below, we see the very same chart showing the spike in dependency latency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BymkfRrUmA_NpuW0wxmvFQ.png" /></figure><p>Notice that purple dot on top of the highest spike? That’s Application Insights’ <strong>Smart Diagnostics</strong> kicking in. We just have to click on that dot to let Application Insights perform some clustering on the data and try to identify a pattern that would explain the spike. Here’s what we got in our case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*7br0b8HWpR-IBUSx62GXmw.png" /></figure><p>Without any manual guidance, the pattern it has found is a combination of dependency target (the database) and operation (creation of a cursor). And by looking at the dependencies response time with and without that pattern (which is a chart that’s automatically generated by the Smart Diagnostics), we can further confirm the diagnosis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b---IwNOavVCdB6HqMVfqw.png" /></figure><h4>From zero to full insights in minutes</h4><p>I’ll stop this already long post here; obviously the next step would be to understand what happened with the database at that particular moment but that didn’t involve Application Insights in our case.</p><p>I’ve hopefully demonstrated how easy and straightforward it is to navigate through the several charts and analytics provided by Application Insights. This complete process is usually performed in a couple of minutes for someone who’s familiar with the UI and its different sections.</p><p>It’s also important to keep in mind that all the metrics that have served this analysis are collected by default as soon as you add Application Insights to your web app (which is done with <a href="https://docs.microsoft.com/en-us/azure/application-insights/quick-monitor-portal#enable-application-insights">just a few clicks from Visual Studio</a>).</p><p>I would love to hear your feedback about this real life cast study. Did you find it useful? Any interest for more posts in the same format? What about short videos? Just let me know!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=962386d53237" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The state of monitoring in Azure]]></title>
            <link>https://faun.pub/the-state-of-monitoring-in-azure-42cc0174c785?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/42cc0174c785</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[log-analytics]]></category>
            <category><![CDATA[application-insights]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Mon, 04 Dec 2017 13:52:29 GMT</pubDate>
            <atom:updated>2019-07-16T14:29:12.108Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve recently developed a strong interest for performance monitoring. This is the opportunity for me to expand my skills on Application Insights, which I’ve been using for many years, but also complement them with an overview of all the monitoring services available on Azure.</p><p>There are currently 3 different services related to monitoring on Azure:</p><ul><li><a href="https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/">Monitor</a></li><li><a href="https://docs.microsoft.com/en-us/azure/log-analytics/">Log Analytics</a></li><li><a href="https://docs.microsoft.com/en-us/azure/application-insights/">Application Insights</a></li></ul><p>As often on Azure, these services provide similar and overlapping features and even though <a href="https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-overview">this page</a> gives helpful examples of “when to use which”, I thought I would outline here, and in my own words, a review of their features, differences and similarities.</p><h4>Infrastructure vs. application monitoring</h4><p>Infrastructure and application monitoring are very different tasks that require different metrics and different exploitation of those metrics.</p><p><strong>Infrastructure monitoring</strong> calls for both a detailed reporting of the activity of each service of your system, and a synthetic, high-level overview of the global health of that system. It should also give the ability to conveniently overlay metrics from different services to facilitate the discovery of correlations. This can be achieved with both <strong>Monitor</strong> and <strong>Log Analytics</strong>, with differences explained in the next sections hereunder.</p><p><strong>Application monitoring</strong> requires the collection of data much closer to the code and instruments your code in order to do that. It collects query performance, custom logs and thorough exception data, so you have all the information at hand to quickly identify the piece of code to investigate in case of issues. That’s what <strong>Application Insights</strong>’ SDKs and agents provide, with a basic instrumentation experience out of the box and the ability to customize it to make it more specific to your code.</p><h4>Monitor</h4><p>Monitor provides <strong>infrastructure-level monitoring</strong> by collecting metrics from most of your Azure resources with near-to-zero configuration. Monitor is actually activated by default on each Azure subscription and doesn’t have to be provisioned. Also by default, it leverages Azure Diagnostics to automatically collect telemetry from all <a href="https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-supported-metrics">compatible resources</a> that exist in your subscription.</p><p>From the collected telemetry, Monitor lets you:</p><ul><li>Visualize it on the Azure portal and build charts that you can pin on dashboards</li><li>Store it in Azure Storage if you want to access it beyond the default retention limits and/or use it with external monitoring solutions</li><li>Query it through a number of channels including a REST API, the Azure CLI and Powershell commands</li><li>Trigger alerts and auto-scale actions based on custom rules</li></ul><h4>Log Analytics</h4><p>Log Analytics, which is a kind of spin-off from Operations Management Suite, is basically similar to Monitor in that it is suited to <strong>infrastructure monitoring</strong>, but can ingest telemetry from a wider range of sources (and not only Azure resources); it can</p><ul><li>Receive the same telemetry as Monitor (through Azure Diagnostics)</li><li>Collect additional metrics and logs through connectors</li><li>Load external telemetry from Azure Storage</li></ul><p>From the collected telemetry, Log Analytics lets you:</p><ul><li>Search interactively on the Azure portal</li><li>Create custom visualizations and build elaborate dashboards</li><li>Trigger alerts based on custom rules</li><li>Use “<a href="https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-add-solutions">Solutions</a>”, which I would describe as templates that provide off-the-shelf dashboards and alerting for typical resources (like Web Apps or SQL Server instances for example)</li><li>Access a separate “advanced analytics” portal where you can execute custom queries over the telemetry data, run machine learning algorithms and generate charts that you can pin on your dashboards</li></ul><h4>Application Insights</h4><p>Application Insights is a complete <a href="https://en.wikipedia.org/wiki/Application_performance_management">application performance management</a> product that lets you <strong>monitor the performance of your application</strong> (most likely your <em>web </em>app, although it can be used to instrument and gather custom telemetry from any kind of app). It is mainly composed of:</p><ul><li>A service hosted on Azure that receives telemetry from agents</li><li>A set of agents that you can integrate in your app (either SDKs to instrument server-side code or Javascript libraries to embed in your web pages)</li><li>A comprehensive collection of interactive visualizations and analytics widgets available on the Azure portal that help you monitor the telemetry and troubleshoot issues (check <a href="https://medium.com/@th0maswe1ss/exploring-performance-issues-with-azure-application-insights-962386d53237">this post</a> to see how to explore performance issues with Application Insights!)</li><li>The very same “advanced analytics” portal as available on Log Analytics (providing complex querying, machine learning and charting)</li><li>A handy integration with Visual Studio to access and explore performance data directly from the IDE</li></ul><h4>The big picture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X66Q-8Sf3keZen89hibNUA.png" /></figure><h4>No one-size-fits-all</h4><p>None of these 3 products would allow you to achieve both infrastructure <em>and</em> application monitoring in a satisfying form and in most cases, you will need to use both Application Insights and (at least) one of Monitor or Log Analytics. One exception though would be if you have a simple Web App with minimum infrastructure around it; in that case, the performance counter collection that Application Insights does on the underlying App Service may be enough for you.</p><h4>Monitor or Log Analytics?</h4><p>As explained above, these 2 services have similar goals and features so let’s try to outline some key considerations to take into account when deciding which one to use.</p><p><strong>If all your infrastructure is within Azure</strong>, you should start with Monitor. It doesn’t require any kind of setup and enables you to quickly assemble charts and pin them to your dashboards. <strong>If you know that some of your services will be sitting outside of Azure</strong>, like VMs running on-premises or in other clouds, it may be wiser to invest in Log Analytics from the start.</p><p>I just said <em>invest</em> in Log Analytics because <strong>its learning curve is steeper</strong> than with Monitor. Except for some Azure resources, collection of metrics isn’t activated with just the click of a button and may require some Powershell commands or the configuration of agents. Similarly, construction of charts is less straightforward and requires familiarity with the query language.</p><p>Another way to differentiate these 2 services is by their query capabilities. Monitor offers limited querying (I should rather call it “filtering”) whereas Log Analytics gives you access to the full power of the <em>Kusto</em> query language through the separate, “advanced analytics” portal. In other words, <strong>Monitor is fine for out-of-the-box visualizations</strong> but won’t let you investigate the data you have collected.</p><h4>Get started with monitoring on Azure!</h4><p>A recommended path for those who are not sure where to start:</p><ul><li>Begin with Monitor and Application Insights. Both are easy to setup (you add the Application Insights agent to your ASP.NET app with just <a href="https://docs.microsoft.com/en-us/azure/application-insights/quick-monitor-portal#enable-application-insights">a couple of clicks from Visual Studio</a>) and offer rich charting capabilities</li><li><a href="https://docs.microsoft.com/en-us/azure/azure-portal/azure-portal-dashboards">Create a monitoring dashboard</a> on your Azure portal and pin some key charts on it</li><li>Take some time to learn the <a href="http://docs.loganalytics.io/index"><em>Kusto</em> query language</a> that powers the “advanced analytics” sections of both Log Analytics and Application Insights; it’s incredibly powerful to explore data and create custom visualizations — and it’s pretty fun to use!</li><li>If/when you feel that you’ve reached the limits of what Monitor can provide and need to dig deeper in your infrastructure telemetry, consider switching to Log Analytics</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*3CTGKR_YrWZNN3jM" /></figure><p><strong>Join our community Slack and read our weekly Faun topics ⬇</strong></p><p><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmedium">Join a Community of Aspiring Developers.Get must-read articles, learn new technologies for free, chat with like-minded people, get job offers, listen to podcasts and more</a></p><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42cc0174c785" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/the-state-of-monitoring-in-azure-42cc0174c785">The state of monitoring in Azure</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I learned to stop worrying and love Cosmos DB’s Request Units]]></title>
            <link>https://medium.com/@thomasweiss_io/how-i-learned-to-stop-worrying-and-love-cosmos-dbs-request-units-92c68c62c938?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/92c68c62c938</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Thu, 27 Jul 2017 08:43:26 GMT</pubDate>
            <atom:updated>2017-08-25T03:19:50.392Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is a reboot of </em><a href="https://medium.com/@th0maswe1ss/how-i-learned-to-stop-worrying-and-love-documentdbs-request-units-a05efef7df64"><em>my initial article</em></a><em> about Request Unit provisioning on DocumentDB. It has been updated to reflect recent changes like the evolution of DocumentDB into Cosmos DB and new monitoring capabilities.</em></p><p>As a database-as-a-service platform, Cosmos DB offers a rather unique advantage: <strong>predictable performance</strong>. You specify the level of throughput you expect and the database guarantees that it will meet that level by dedicating the required resources to your usage.</p><h4>Request Units are Cosmos DB’s performance currency</h4><p>You define that level of performance by provisioning, for each container of your database, an amount of <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">Request Units</a>; more precisely, you set how many Request Units you expect the container to be able to serve per second. Provisioned Request Units can start low and scale to tens of thousands or even more.</p><p>So, Request Units being the “performance currency” in Cosmos DB, you must be curious to know what they represent… and that’s where it gets tricky, to say the least. They define what I would call a “work capacity”. Each request you issue against your container — any kind of request: reads, writes, queries, executions of stored procedures etc. — has a corresponding cost that will be deducted from your RU credits. So if you provision 400 RU/second and issue a query that costs 40 RU, you will be able to issue 10 such requests per second; any request beyond that will get throttled.</p><h4>The art of evaluating your Request Unit needs</h4><p>By now you probably guess that it’s important to properly evaluate how many RU you have to provision; provision too few and some of your requests may fail, provision too many and you will pay for unused performance!</p><p>It’s actually pretty hard to perform that kind of evaluation from scratch. Microsoft provides <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units#estimating-throughput-needs">some metrics</a> about the costs of basic operations:</p><ul><li>For a 1 KB document: a read costs 1 RU, a write costs 5 RU</li><li>For a 100 KB document: a read costs 10 RU, a write costs 50 RU</li></ul><p>And there is also a pretty nice <a href="https://www.documentdb.com/capacityplanner">capacity planner</a> that can help with guessing your target throughput based on sample documents.</p><p>Unfortunately, this only covers CRUD operations and doesn’t give a single hint about how many RU your queries or stored procedure executions will cost. And that’s a big deal because from my experience, the cost of those operations:</p><ul><li>is dynamic (the cost of a query does not only depend on its complexity but also on the number of results it returns), and</li><li>can get pretty high (tens of RU for a query that filter on a single field, for example).</li></ul><p>In a nutshell, it’s pretty hard to predict how many RU your system will require. So is this thing usable at all? Fortunately, yes it is. <strong>Read on!</strong></p><h4>Evaluate the cost of typical queries with the query explorer</h4><p>Once you’ve fed your Cosmos DB container with some relevant data — that could be test data to start with — you can use the query explorer that’s available from the Azure portal. Besides helping you build and debug your SQL queries, this tool also reports the real cost of those queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*77Ce65k75pgH0TcGEs0Cyw.png" /></figure><p>This will give you a sense of the actual charge involved with typical operations that your system will have to support.</p><h4>Cosmos DB throttles your requests intelligently</h4><p>When you’re exceeding your RU quota, Cosmos DB doesn’t reject your additional requests by just screaming <strong>ERROR</strong>! Not only does it explicitly flag these throttled requests with the HTTP status code 429, but the response also provides a very useful header: <em>x-ms-retry-after-ms</em>. As its name implies, this header tells you how much time you should wait before re-trying.</p><p>Although this hint has its own limits (it may not be very reliable if multiple clients overload your RU quota at the same time), it’s still a very useful information to have in order to define the cool-off period one should wait and avoid a retry policy that would be too aggressive (making things even worse!).</p><h4>Microsoft’s official SDKs do the work for you</h4><p>The good news is, if you’re using one of Microsoft’s official Cosmos DB SDKs (currently available for <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-sdk-dotnet">.NET</a>/<a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-sdk-dotnet-core">.NET Core</a>, <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-sdk-java">Java</a>, <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-sdk-node">Node.js</a> and <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-sdk-python">Python</a>), they all implement the retry logic based on the header mentioned above — and apply it by default. And by default, they all retry the throttled requests up to 9 times, or up to 60 seconds after the request was originally issued, whichever comes first. Those parameters are configurable when you create a new DocumentClient instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*L3qZ81gEJARtcMJufHP63A.png" /></figure><h4>Request throttling can be easily monitored</h4><p>From the Azure portal, it is straightforward to monitor the number of throttled requests as well as how many RU you’ve consumed vs. how many you’ve provisioned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CYHmKyTymgHWZkEv-EVT_g.png" /><figcaption>What it looks like to exceed your provisioned throughput!</figcaption></figure><p>Even better, you can (and should!) set alerts for when the number of throttled requests exceeds a specific threshold:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*Z5n-iNeouKjmIDGM4RhAXA.png" /></figure><p>Those alerts can dispatch an email to the account administrators or call a custom HTTP webhook.</p><h4>Scale your throughput elastically and on-demand</h4><p>Monitoring the consumption of your RU and the ratio of throttled requests will probably reveal that you don’t need to keep a constant performance level throughout the day or the week; many web applications receive less traffic at night or during the week-end.</p><p>Cosmos DB’s REST API provides <a href="https://docs.microsoft.com/en-us/rest/api/documentdb/offers">endpoints</a> to programmatically update the performance level of your containers (those endpoints are also exposed through the official SDKs), making it straightforward to adjust the throughput from your code depending on the time of the day or the day of the week. The operation is performed without any downtime, and typically takes effect in less than a minute.</p><p>Cosmos DB doesn’t offer performance auto-scaling (yet — I wouldn’t be surprised if that was in the works) but it’s totally possible to implement something similar: you could gather live metrics either from your application code or using the Azure monitoring webhooks, aggregate that data in <a href="https://azure.microsoft.com/en-us/services/stream-analytics/">Azure Stream Analytics</a> and use the output data in some <a href="https://azure.microsoft.com/en-us/services/functions/">Azure Function</a> to perform real-time adjustment of your throughput provisioning. Well, I think I’ve just found a nice topic for my next blog post!</p><h4>The path to optimum Cosmos DB usage</h4><p>The workflow that I usually recommend to configure Request Units on a new project is:</p><ol><li>Perform an initial, rough evaluation using the <a href="https://www.documentdb.com/capacityplanner">capacity planner</a> and adjust your estimate with the help of the query explorer</li><li>If possible, use one of the official Cosmos DB SDKs to benefit from automatic retries when requests get throttled — if you’re working on a platform that is not supported and use Cosmos DB’s <a href="https://docs.microsoft.com/en-us/rest/api/documentdb/">REST API</a>, implement your own retry policy using the <em>x-ms-retry-after-ms</em> header</li><li>Obviously, make sure that your application code gracefully supports the case when all retries fail</li><li>Configure throttling alerts from the Azure portal — start with conservative limits like 10 throttled requests over the last 15 minutes and switch to more permissive rules once you figure out your actual consumption (remember that occasional throttles are fine, they show that you’re playing with the limits you’ve set and that’s exactly what you want to do)</li><li>Use monitoring to understand your traffic pattern, so you can consider the need to dynamically adjust your throughput provisioning over the day/week</li><li>Monitor regularly your provisioned vs. consumed RU ratio to make sure you have not over-provisioned your containers</li></ol><p>I hope that following these steps will help you to use Cosmos DB efficiently and confidently!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92c68c62c938" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dispatch Azure alerts to Slack with webtask.io]]></title>
            <link>https://medium.com/@thomasweiss_io/dispatch-azure-alerts-to-slack-with-webtask-io-ad3226fc4539?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad3226fc4539</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Sat, 08 Apr 2017 10:42:16 GMT</pubDate>
            <atom:updated>2017-04-08T10:42:16.251Z</atom:updated>
            <content:encoded><![CDATA[<p>A customer recently asked me to improve the integration between his Microsoft Azure account and Slack by adding the ability to dispatch Azure alerts directly to a specific Slack channel.</p><h4>The origin: Azure alerts</h4><p>From the Azure portal you can define, for most of Azure’s services, monitoring rules that help you keep a close eye on your application’s health. If you’re using Application Insights, you can also benefit from the <a href="https://docs.microsoft.com/en-us/azure/application-insights/app-insights-proactive-diagnostics">Smart Detection</a>’s proactive monitoring rules. By default, the alerts triggered by those rules are dispatched as emails to the Azure account’s administrators. But it’s also possible to make them call <a href="https://en.wikipedia.org/wiki/Webhook">webhooks</a> that you specify:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*0UBRmgp81C8jDCnegCEUwQ.png" /></figure><p>When triggered, the webhook receives <a href="https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/insights-webhooks-alerts#payload-schema">a JSON payload</a> that describes the details of the alert.</p><h4>The destination: Slack’s webhooks</h4><p>As described in <a href="https://api.slack.com/incoming-webhooks">the documentation of the Slack API</a>, it’s fairly easy to inject messages in a Slack channel by POST’ing, guess what… another JSON payload. Powerful formatting options let you control the design of those messages.</p><p>The only thing to do is to set up a webhook integration (following <a href="https://my.slack.com/services/new/incoming-webhook/">this link</a>), which will give you the URL where the payload shall be posted. All authentication information is embedded within that URL.</p><h4>The bridge: webtask.io</h4><p>So all I needed was the ability to forward a POST request to the URL generated by Slack and perform some basic JSON transformation in the middle. This seemed to be the perfect opportunity to play with some serverless technology!</p><p>I’ve been following the <a href="https://en.wikipedia.org/wiki/Serverless_computing">serverless</a> trend for quite some time and it seems clear by now that it’s much more than a gimmick — although the community has, once again, chosen a poor name to describe what it is (NoSQL anyone?). The ability to deploy and run <em>functions</em> without having to care about server provisioning and capacity planning, and only paying for the time/CPU/memory used by the actual execution of those functions, is really great in a number of situations. And our little webhook project is a perfect use-case.</p><p>There are already plenty of serverless solutions available: AWS pioneered the concept with <a href="https://aws.amazon.com/lambda/">Lambda</a>, Microsoft followed with <a href="https://azure.microsoft.com/en-us/services/functions/">Azure Functions</a>, IBM is investing heavily in <a href="http://openwhisk.org/">OpenWhisk</a>… But the one I really wanted to try was <a href="https://webtask.io/">webtask.io</a>, from the Auth0 guys.</p><p>webtask.io is running on Node.js and makes it super easy to write a piece of Javascript and put it behind a URL. Browsing through <a href="https://webtask.io/docs/101">the documentation</a>, I was surprised to see that it already provides a wealth of useful features: storage, secrets, CRON tasks etc. webtask.io provides a Node.js-based CLI that facilitates the setup of a task, but I took an even easiest path by using their online editor.</p><h4>Up and running in minutes</h4><p>The first step was to store the end of the Slack webhook URL in a <a href="https://webtask.io/docs/editor/secrets">secret</a> — as it contains authentication information, you don’t want to have it in clear text in your code. Anything you store as a secret can be conveniently retrieved through the context.secrets dictionary.</p><p>From there, it was pretty straightforward to construct the payload requested by Slack and post it to the Slack webhook URL:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/863ec6de7a916e74f7ae4d257241d21f/href">https://medium.com/media/863ec6de7a916e74f7ae4d257241d21f/href</a></iframe><p>Before registering this function in Azure, I quickly tested it by posting a request to this URL and passing the kind of payload that Azure would send. And sure enough, the thing instantly showed up in Slack:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*igtUEL0XULTsRwoxcW5oTg.png" /><figcaption>(this screenshot comes from a previous version of the function that dispatched different fields)</figcaption></figure><p>And so all that was left was to register the webhook’s URL in the Azure portal to get those instant alerts… hoping that they won’t show up too often!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad3226fc4539" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I learned to stop worrying and love DocumentDB’s Request Units]]></title>
            <link>https://medium.com/@thomasweiss_io/how-i-learned-to-stop-worrying-and-love-documentdbs-request-units-a05efef7df64?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a05efef7df64</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[documentdb]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Sun, 02 Apr 2017 04:08:37 GMT</pubDate>
            <atom:updated>2017-07-27T08:45:40.786Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Obsolete!</em></strong><em> I’ve updated this article to reflect the evolution of DocumentDB into Cosmos DB and the introduction of new features. Please find the new version </em><a href="https://medium.com/@th0maswe1ss/how-i-learned-to-stop-worrying-and-love-cosmos-dbs-request-units-92c68c62c938"><em>here</em></a><em>.</em></p><p>As a database-as-a-service platform, DocumentDB offers a rather unique advantage: <strong>predictable performance</strong>. You specify the level of performance you expect and the database guarantees that it will meet that level by dedicating the required resources to your usage.</p><h4>Request Units are DocumentDB’s performance currency</h4><p>You define that level of performance by provisioning, for each collection of your database, an amount of <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-request-units">Request Units</a>; more precisely, you set how many Request Units per second you expect the collection to be able to serve. The minimum is 400 RU/s but you can scale it to tens of thousands or even more.</p><p>So, Request Units being the “performance currency” in DocumentDB, you must be curious to know what they represent… and that’s where it gets tricky, to say the least. They define what I would call a “work throughput”. Each request you issue against your collection — any kind of request: reads, writes, queries, executions of stored procedures etc. — has a corresponding cost that will be deducted from your RU credits. So if you provision 400 RU/s and issue a query that costs 40 RU, you will be able to issue 10 such requests per second; any request beyond that will get throttled.</p><h4>The art of evaluating your Request Unit needs</h4><p>By now you probably guess that it’s important to properly evaluate how many RU you have to provision; provision too few and some of your requests will fail, provision too many and you will pay for unused performance!</p><p>It’s actually pretty hard to perform that kind of evaluation from scratch. Microsoft provides some metrics about the costs of basic operations:</p><ul><li>For a 1 KB document: a read costs 1 RU, a write costs 5 RU</li><li>For a 100 KB document: a read costs 10 RU, a write costs 50 RU</li></ul><p>And there is also a pretty nice <a href="https://www.documentdb.com/capacityplanner">RU calculator</a> that can help with guessing your target throughput based on sample documents.</p><p>Unfortunately, this only covers CRUD operations and doesn’t give a single hint about how many RU your queries or stored procedure executions will cost. And that’s a big deal because from my experience, the cost of those operations:</p><ul><li>is dynamic (the cost of a query does not only depend on its complexity but also on the number of results it returns), and</li><li>can get pretty high (tens of RU for a query that filter on a single field, for example).</li></ul><p>In a nutshell: under-provisioning your RU will result in request throttling and, at the same time, it’s very hard to predict how many RU your system will require. So is this thing usable at all? Fortunately, yes it is. Read on!</p><h4><strong>DocumentDB throttles your requests intelligently</strong></h4><p>When you’re exceeding your RU quota, DocumentDB doesn’t reject your additional requests by just screaming ERROR! Not only does it explicitly flag these throttled requests with the HTTP status code 429, but the response also provides a very useful header: <em>x-ms-retry-after-ms</em>. As its name implies, this header tells you how much time your should wait before re-trying.</p><p>Although this hint has its own limits (it may not be very reliable if multiple clients overload your RU quota at the same time), it’s still a very useful information to have in order to define the cool-off period one should wait and avoid a retry policy that would be too aggressive (making things even worse!).</p><h4><strong>Microsoft’s official SDKs do the work for you</strong></h4><p>The good news is, if you’re using one of Microsoft’s official DocumentDB SDKs (currently available for <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-sdk-dotnet">.NET</a>/<a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-sdk-dotnet-core">.NET Core</a>, <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-sdk-java">Java</a>, <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-sdk-node">Node.js</a> and <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-sdk-python">Python</a>), they all implement the retry logic based on the header mentioned above — and apply it by default. And by default, they all retry the throttled requests up to 9 times, or up to 60 seconds after the request was originally issued, whichever comes first. Those parameters are configurable when you create a new DocumentClient instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*f7EdNf3DyfCvwlZEmt0UgQ.png" /></figure><h4>Request throttling can be easily monitored</h4><p>From the Azure portal, it is straightforward to monitor the number of throttled requests:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*Ox5vcj-3wE27AjhXpGycOQ.png" /></figure><p>as well as how many RU you’ve consumed vs. how many you’ve provisioned:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*WkIy0xGrDIRXaHliw3FV7A.png" /></figure><p>Even better, you can (and should!) set alerts for when the number of throttled requests exceeds a specific threshold:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*QOQGTMUaaMml1GEgZwgFUA.png" /></figure><p>Those alerts can dispatch an email to the account administrators or call a custom HTTP webhook.</p><h4>The path to the optimum DocumentDB usage</h4><p>The workflow that I usually recommend to configure Request Units on a new project is:</p><ol><li>Perform an initial, rough evaluation using the <a href="https://www.documentdb.com/capacityplanner">RU calculator</a> — if you use stored procedures, inflate the result by 25% to reflect the cost impact</li><li>If possible, use one of the official DocumentDB SDKs to benefit from automatic retries when requests get throttled — if you’re working on a platform that is not supported and use DocumentDB’s REST API, implement your own retry policy using the <em>x-ms-retry-after-ms</em> header</li><li>Obviously, make sure that your application code gracefully supports the case when all retries fail</li><li>Configure throttling alerts from the Azure portal — start with conservative limits like 10 throttled requests over the last 5 minutes and switch to more permissive rules once you better understand your actual consumption</li><li>Monitor regularly your provisioned vs. consumed RU ratio to make sure you have not over-provisioned your collections</li></ol><p>Remember that occasional throttles are fine, they show that you’re playing with the limit you’ve set and that’s exactly what you want to do. I hope that following these steps will allow you to use DocumentDB with confidence!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a05efef7df64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure DocumentDB vs. MongoDB]]></title>
            <link>https://medium.com/@thomasweiss_io/azure-documentdb-vs-mongodb-6d5806c16239?source=rss-a22e82772bd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d5806c16239</guid>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[documentdb]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Thomas Weiss]]></dc:creator>
            <pubDate>Thu, 01 Sep 2016 04:29:34 GMT</pubDate>
            <atom:updated>2017-04-02T04:15:08.629Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A synthetic comparison</blockquote><p><a href="https://azure.microsoft.com/en-us/services/documentdb/">DocumentDB</a> is a NoSQL <em>database-as-a-service</em> this is part of the Microsoft Azure platform. As a document store, it falls into the same category as MongoDB, CouchDB or RethinkDB and just like those, it handles documents in the JSON format.</p><p>When considering the integration of a NoSQL document store in their systems, many companies choose MongoDB because it’s among the most popular NoSQL engines out there and it has become very reliable over the past years. I feel that DocumentDB usually doesn’t get considered when making this decision although its characteristics make it a serious contender to MongoDB, even offering stronger advantages in some situations.</p><p>Because I feel that DocumentDB doesn’t get the love it deserves, I decided to write this synthetic and unbiased comparison between DocumentDB and MongoDB, supported by pointers to their respective documentations. I hope this can serve you as a guide when trying to weight the pros and cons of each platform.</p><h3><strong>Updates to this post</strong></h3><p><strong>[November 2016]</strong> Removed all mentions of the lack of local emulator for DocumentDB as Microsoft announced the general availability of such a <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-nosql-local-emulator">local development version</a>. Note that the local emulator is only available for Windows currently (thanks <a href="https://medium.com/@nostriluu">David Mason</a> for the suggested edit!).</p><p><strong>[November 2016]</strong> Removed the mention of auto-expiring documents being a feature that is exclusive to DocumentDB, as <a href="https://medium.com/u/5c2b6a8b7a1">Bo Bendtsen</a> kindly pointed out that MongoDB has <a href="https://docs.mongodb.com/v3.2/core/index-ttl/">similar capabilities</a>.</p><p><strong>[January 2017]</strong> Added a section about DocumentDB’s out-of-the-box, built-in security as suggested by <a href="https://medium.com/@marypcbuk">Mary Branscombe</a>.</p><h3>The similarities</h3><p>Conceptually, there are some basic similarities between the two databases:</p><ul><li>Documents are stored and served in the JSON format</li><li>Documents can be retrieved using a rich query language that plays well with the JSON syntax</li></ul><h3>Features unique to MongoDB</h3><p>Let’s start by enumerating the main MongoDB features that don’t have any (reasonably matching) DocumentDB counterpart.</p><h4>Rich query capabilities with the aggregation pipeline</h4><p>MongoDB’s <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline/">aggregation pipeline</a> is a very powerful feature that lets you build a pipeline composed of data processing stages, each filtering and transforming the documents coming from a collection. The possibilities offered by this pipeline are nearly limitless and its flexibility can cater for virtually any kind of query.</p><p>In comparison, DocumentDB’s SQL-like query syntax only allows simple filtering over the documents, even lacking “basic” constructs like <em>count</em> or <em>sum</em> (although they are <a href="https://feedback.azure.com/forums/263030-documentdb/suggestions/6333963-add-support-for-aggregate-functions-like-count-su">working on it</a> and you can work around with server-side Javascript in the meantime). A handy query cheat sheet can be found <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-sql-query-cheat-sheet/">here</a>.</p><h4>Map-reduce</h4><p>Somehow similar to the aggregation pipeline, MongoDB’s <a href="https://docs.mongodb.com/manual/core/map-reduce/">map-reduce</a> feature lets a collection’s documents flow through 2 separate stages that iteratively transforms (or projects) then groups the documents. Both stages are defined in Javascript.</p><p>There isn’t any such concept in DocumentDB, although similar results can be achieved using stored procedures (see below).</p><h4>Full-text indexes</h4><p>Among the <a href="https://docs.mongodb.com/manual/indexes/">different types of indexes</a> available on MongoDB, the <em>text</em> index offers <a href="https://docs.mongodb.com/manual/text-search/">full-text search</a> capabilities.</p><p>DocumentDB doesn’t provide any full-text indexing. The recommended way to add full-text search to a DocumentDB database is to pair it with an Azure Search service; there is a good <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-search-indexer/">integration story</a> between the two.</p><h4>More platforms supported by client-side drivers</h4><p>I think it’s important to mention that MongoDB’s drivers support a very <a href="https://docs.mongodb.com/ecosystem/drivers/">large spectrum of platforms</a>, whereas DocumentDB only has SDKs for .NET, Java, Python and Node.js — but you can try your luck using any MongoDB driver with DocumentDB thanks to its <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-protocol-mongodb/">support for MongoDB’s protocol</a>.</p><h3>Features unique to DocumentDB</h3><p>Now let’s do the inverse exercise and list the DocumentDB’s features that can’t be found in MongoDB.</p><h4>Server-side Javascript</h4><p>That’s a <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-programming/">key feature of DocumentDB</a>. It has a rich server-side Javascript API, letting you create data processing functions. Those server-side functions can take 3 different forms:</p><ul><li>stored procedures that can do pretty much anything (inserting, querying, updating documents) and get called through the SDKs or the REST API</li><li>triggers (or hooks) that get executed before or after specific operations (like on a document insertion for example)</li><li>UDFs (user-defined functions) that can be called from and augment the SQL query language, somehow narrowing the gap with MongoDB’s rich query capabilities</li></ul><p>Now MongoDB can execute server-side Javascript <a href="https://docs.mongodb.com/manual/core/server-side-javascript/">as</a> <a href="https://docs.mongodb.com/manual/tutorial/store-javascript-function-on-server/">well</a>, but my understanding is that:</p><ul><li>Map-reduce and the <em>$where</em> query operator can only be used for queries, not updates</li><li>The Javascript functions that you can store in the special <em>system</em> collection are only suitable for administration or maintenance purposes.</li></ul><p>MongoDB’s documentation clearly states that there are performance limitations in executing server-side Javascript; in comparison, DocumentDB is really designed for this purpose as it pre-compiles your Javascript code, then stores and executes the resulting bytecode.</p><h4>Transactions</h4><p>Thanks to the Javascript stored procedures we’ve just mentioned, it is possible to run ACID transactions on a DocumentDB collection. The way it works is really simple: if your Javascript function completes, all write operations it has performed get committed; if the function throws any exception, all operations get rolled back.</p><p>There isn’t really any concept of transaction in MongoDB besides <a href="https://docs.mongodb.com/manual/core/write-operations-atomicity/">single-document atomicity</a>, which means that inserting or updating a document is guaranteed to be atomic, but a write operation involving multiple documents is not atomic as a whole.</p><h4>Full indexing by default</h4><p>DocumentDB takes a rather drastic approach to indexing: by default, it indexes all the fields of the documents you are storing! Many of you may see this as a waste of processing time and storage space — which honestly it is to some extent — but this gives the interesting advantage of offering excellent query performance out of the box. For those who prefer to have a better control over what gets indexed, it is always possible to define <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-indexing-policies/">custom indexing policies</a>.</p><h4>(Easy) global distribution</h4><p>Another pretty recent addition to DocumentDB’s capabilities is <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-distribute-data-globally/">global distribution</a>. Basically, this feature lets you scale your DocumentDB instance across different regions around the world and define what type of consistency you expect between the regions, from strong to eventual. It is even possible to configure an automatic and transparent failover over the different regions.</p><p>Of course, deploying a world-wide cluster of MongoDB nodes is certainly possible, but what I want to emphasize here is how easy it is to setup such a cluster. That’s obviously beyond DocumentDB’s core features and is related to its PaaS nature, but I don’t believe there is any service provider offering such geo-distributed setup for MongoDB (at that cost and ease of use).</p><h4>Security</h4><p>It’s worth mentioning that as a service, DocumentDB provides <a href="https://docs.microsoft.com/en-us/azure/documentdb/documentdb-secure-access-to-data">built-in security and access control</a> that are there by default… No password-less admin access! And beyond that, it also gives the ability to control the access to collections and documents in a fine-grained fashion by creating users and linking them to those resources through password-protected permissions.</p><h3>Pricing</h3><p>The last, but certainly not least criteria of comparison to consider is the cost. But we should be careful not to compare apples and oranges here: DocumentDB belongs to the PaaS family whereas MongoDB is a database, not a service. So let’s take <a href="https://mlab.com/">mLab</a>, a MongoDB PaaS offering, as a point of comparison.</p><p>First I should clarify how DocumentDB is billed. Each collection is billed over 2 dimensions:</p><ul><li>storage used, at 0.25 USD per GB / month</li><li>reserved Request Units per seconds, at ~6 USD per 100 RU / month</li></ul><p>The number of RU you reserve dictates the guaranteed bandwidth you will get (want to learn more about Request Units? check <a href="https://medium.com/@th0maswe1ss/how-i-learned-to-stop-worrying-and-love-documentdbs-request-units-a05efef7df64">my post</a>!). Basically, a RU represents “the processing required to read a single 1KB document with 10 properties”. That being said, it’s not easy to evaluate the actual cost of complex operations like big queries or elaborate stored procedure, although <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-request-units/">this guide</a> helps a lot. But we can do the reverse exercise of looking at how many RU we could get for the price of a mLab plan.</p><p>What I didn’t mention so far is that DocumentDB runs on local SSD, so in order to do a fair comparison, let’s take the “High Performance M3” plan from <a href="https://mlab.com/plans/pricing/">this page</a>, which at the time of this writing (September 2016) is priced at 1,390 USD monthly for 80GB of storage.</p><ul><li>Those 80GB would be billed 20 USD on DocumentDB</li><li>That leaves 1,370 USD, or more than 22,800 RU</li></ul><p>I mentioned before that it’s difficult to evaluate the “value” of a RU, but from my experience, 22,800 is <em>a lot</em>, something in the range of 200 complex queries per second. And even though it’s similarly difficult to evaluate the capabilities of that “High Performance M3” plan, I would say that we are playing on a similar scale, or at least not orders of magnitude different.</p><p>Besides, what’s nice with the elasticity of RU is that it is designed to be a unit of scale, which means that you can start with a modest amount of RU and (seamlessly) scale it out as the usage of your collections increases, while still taking advantage of local SSD performance from the beginning.</p><h4>What about the cost of vendor lock-in?</h4><p>A concern expressed by many is vendor lock-in: if I use DocumentDB, I’m not only locked with Microsoft, but also with Azure as a platform. You could even argue that the lack of such lock-in should have been listed in MongoDB’s advantages over DocumentDB. I agree. But what’s the real cost of this lock-in?</p><p>DocumentDB stores documents in the JSON format. That’s a standard format used by most NoSQL databases (hey, even <a href="https://msdn.microsoft.com/en-us/library/dn921897.aspx">SQL Server speaks JSON</a>!), so moving your documents out of DocumentDB and injecting them in some other database should not be an issue.</p><p>The main technical lock-in you have to deal with is the query interface: each database has its own way of querying documents. Most of the time, you perform those queries through some SDK or driver, so from the perspective of your application code, the lock-in or adherence to a particular database comes mainly from the interface of that SDK. But then, if your developers are doing it right, that interface should be encapsulated behind some kind of data access interface that hides the implementation details to the rest of the application.</p><p>And if your concern is that you may want to migrate to MongoDB at a later stage, remember that DocumentDB has protocol-compatibility with MongoDB, which means that you can use any MongoDB driver to access DocumentDB and perform <a href="https://azure.microsoft.com/en-us/documentation/articles/documentdb-mongodb-guidelines/">most of the CRUD operations</a>.</p><h3>How I recommend to guide your decision</h3><p>Wrapping it up, here are the first questions I think you could ask yourself when having to make a choice between those databases (in no particular order):</p><ul><li><strong>Complexity of queries</strong>: do your queries require the full power of MongoDB’s aggregation pipeline, or can you implement them with DocumentDB’s SQL and some server-side Javascript?</li><li><strong>Transactions</strong>: does your business logic require collection-wide, multi-document transactions, or is MongoDB’s single-document atomicity enough for your requirements?</li></ul><p>Based on your answers and the general direction they give, you can then refine your analysis and consider the rest of the features I mentioned (full-text search, global distribution etc.)</p><h3>Please comment!</h3><p>I tried to perform this comparison in the most honest and unbiased way, but I could be wrong on some aspects. So feel free to reach out if you feel that some features are missing, or were over- or underestimated! I intend this post to evolve over time and get complemented to become a good reference on the comparison between DocumentDB and MongoDB.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d5806c16239" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>