<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Olivier Mallassi on Medium]]></title>
        <description><![CDATA[Stories by Olivier Mallassi on Medium]]></description>
        <link>https://medium.com/@omallassi?source=rss-14b281727aa5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*THE6JKO9qBnXBzbg3C-bDA.jpeg</url>
            <title>Stories by Olivier Mallassi on Medium</title>
            <link>https://medium.com/@omallassi?source=rss-14b281727aa5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:07:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@omallassi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Amazon Linux vs. RHEL; a first understanding]]></title>
            <link>https://medium.com/@omallassi/amazon-linux-vs-rhel-a-first-understanding-91b4b036c904?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/91b4b036c904</guid>
            <category><![CDATA[aws-ec2]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Mon, 05 Aug 2024 14:26:01 GMT</pubDate>
            <atom:updated>2024-08-05T14:26:01.544Z</atom:updated>
            <content:encoded><![CDATA[<p>I have to admit choosing the right instance type, considering performance, cost, scalability and others is not something easy. The operating system is one element of your ec2 costs because :</p><ul><li><em>(1) </em>the hourly fee differs between the OS, given an instance type</li><li><em>(2)</em> because the per-second billing may not apply; it was the case until early 2024 for RHEL.</li></ul><p>The latter really depends on your workloads, their need for elasticity over short period of time... The former is interesting to understand in more details.</p><p>As always you can get the AWS ec2 prices from aws cli.Running, as an example, the following aws pricing list-price-lists --service-code AmazonEC2 --currency-code USD --effective-date &quot;2024-05-16 00:00&quot; will get you a list of prices, as json files.</p><h4>The difference — in absolute value — is fixed whatever the instance type and whatever the AWS Region</h4><p>Parsing the previous json files is not that easy, in particularly with the blur sku notion but you can visualise the following :</p><p><em>y-axis</em>: hourly fee on RHEL minus hourly fee on Amazon linux</p><p><em>x-axis</em>: instance types</p><p><strong><em>as of date</em>: 2024–05–16 00:00</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cz6lkh2whPmXjxXkxW57ug.png" /></figure><p>The difference seems fixed across the instance types and regions with two main values</p><ul><li>0.06$ for “small machines” (<em>ie.</em> the <em>small</em> or <em>large</em> ones)</li><li>0.13$ for “larger ones” (<em>ie. </em>the <em>2xlarge</em> and upper)</li></ul><h4>The OS difference is fixed given an instance family, whatever the number of vCPU</h4><p>The following zooms on g5 family but can obviously be done on any instance family. The following tables show prices differences over the AWS regions and compare 2xlarge and 24xlarge larger instance types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*dbg8Ta5GsPj-1VOra6qx2w.png" /><figcaption>g5.24xlarge across AWS regions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*sC1jUJxLDPZw9zU5mq8c0g.png" /><figcaption>g5.2xlarge across AWS regions</figcaption></figure><p>As you can see the difference is the same — in absolute value — which means the gain is likely to be more important if you can support your workloads with more smaller ec2 instances than few larger ones.</p><h4>Conclusion (Kinda…)</h4><p>As you can imagine this topic will complexifies with the diversity of instance families and types you rely on but :</p><ul><li>the potential gain due to the OS does not relate to instance family or AWS Regions</li><li>the potential gain is fixed in absolute value, making it likely more interesting if your workloads can run on a large fleet of smaller ec2 instances (<em>ie.</em> <em>2xlarge</em>) rather than few larger instances (<em>ie. 24xlarge</em>)</li><li>This type of cost analysis to be run regularly; because the instance type you use will evolve, because your workload will evolve, because some licensing terms could evolve etc…</li></ul><p>In all cases, I do hope it will help you running your own analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91b4b036c904" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand and visualise your AWS costs]]></title>
            <link>https://medium.com/@omallassi/understand-and-visualise-your-aws-costs-ec2c8172919e?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec2c8172919e</guid>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Tue, 30 Jul 2024 07:56:27 GMT</pubDate>
            <atom:updated>2024-07-30T07:56:27.686Z</atom:updated>
            <content:encoded><![CDATA[<h3>Visualise to understand your AWS costs</h3><p>At some point in time you will receive your AWS invoice and at some point in time you will want to have a better understanding of it (likely to try to reduce them).</p><p>Of course AWS provides some tools to help : Cost Explorer, the <a href="https://wellarchitectedlabs.com/cloud-intelligence-dashboards/">Cloud Intelligence Dashboards</a>…but when I have faced this question, these tools did not help me (or I was not able to use them) and I was feeling <em>the need to have more freedom in the analysis of these invoices</em>. More precisely, I was looking at a way to answer this specific question : “<em>are we using AWS resources the right way?</em>” ; which means <em>(1)</em> understand your cost items but also <em>(2)</em> try to correlate your cost with your workloads and infrastructure usages.</p><h3>Set up a reliable process to get your invoice data points</h3><p>To be honest, the title is more complex than what you really have to do. Yet, reality is that day one, you will have nothing and people will try different exports, each exports will give you differences (in cost, services used etc…) and you will feel lost.</p><p>Yet AWS provides you with :</p><ul><li><a href="https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting/faqs/">Cost &amp; Usage Report (a.k.a CUR)</a>. This is csv files that contains, for a given period, all data points related to your invoice, break down per hour if needed, per AWS services (and related operations etc…). The level of precision is tremendous and you will be able to understand when you have been billed for this ec2 instance type running Amazon Linux (or another OS), or how much GBs you transfered from your AWS S3 bucket.</li><li>a way to automate the export of these CUR and have them, stored in an AWS S3 bucket at the end of each month.</li></ul><h3>Start Visualising…</h3><p>A CUR is a CSV file that contains all you have been billed for. I have to admit the level of details in a CUR is insane (or does just reflect the complexity of AWS billing). <a href="https://docs.aws.amazon.com/cur/latest/userguide/data-dictionary.html">You can get — as example</a> — uptime of your resource, bandwidth per hours, the instance type of ec2, running OS…</p><p>From there<a href="https://jupyter.org/"> Jupyter Notebook</a> is your next best friend as it is easy to get a DataFrame from CUR file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9j4Ihxh1vo2ORm_Vzf1-7g.png" /></figure><p>Now you have a DataFrame, it becomes easy to break it down on different axis : per AWS accounts, regions, services (ie. line_item_product_code), operations (ie. line_item_operation), per time slot or day. It will help you</p><ul><li>spotting your main category of cost items: RDS vs. ec2 vs…</li><li>spotting some unexpected usage.</li><li>(assuming a breakdown per time slots) spotting some spikes in usage.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VCHr2zx8x3v9WSldi9YtLg.png" /></figure><p>This first level of visualisation is interesting as <em>it gives you the main trends</em>. It also provides you with <em>a reliable way of back-testing the architecture and potentially detect unexpected things</em> (used unexpected AWS regions, operating systems, instance types…).</p><h3>A first set of optimisations</h3><p>The optimisations you will find heavily depends on your architecture, your workloads. But at this stage, you do have full transparency on which AWS services you really use, how much you use them and how much it costs you.</p><p>In addition, having this type of visualisation (or break down) will also help you working on the tip of the iceberg. There is basically no value optimising an S3 bucket that costs you a hundred of bucks a month whereas ec2 costs you thousands.</p><p>The other interest of the CUR is that <em>it also provides you with quantities </em>: how many VPC endpoints, uptime of each, how many ec2 instances etc. Meaning you can (not necessarily easily) estimate the financial impacts of a change :</p><ul><li>reducing number of VPC endpoints will reduce the total uptime cost</li><li>moving from GP2 to GP3 could an optimisation</li><li>changing your ec2 instance type or Operating System could be an option as (1) the ec2 hourly fee depends on the OS but (2) in some cases, you may be billed for the full hour (even for an ec2 instance used for a couple of minutes). Recently <a href="https://aws.amazon.com/partners/redhat/rhel-pricing/">RHEL billing changed</a>.</li><li>deploying in another region as AWS cost varies between regions. Ok this one can be complex and open other topics but still is valid from a cost perspective.</li><li>Change the way you use RDS or AWS S3 (and archive or purge more frequently)</li><li>…</li></ul><h3>Correlate (ec2) cost and infrastructure usage</h3><p>Again your costs will likely depend on your architecture and workload but let’s assume a big part is ec2. ec2 bills you as soon as the machine is up, whatever the CPU or Memory consumption of the instance. From there (and beside resiliency needs) the goal is to keep the instance(s) busy (&gt; 60 or 70% of the CPU / memory resources).</p><p>One way to do this is to correlate your infrastructure usage and your CUR based on the instance type. Typically :</p><ul><li>get a CSV file from your favorite monitoring tool that shows, hourly, the max CPU and/or memory consumption per instance type.</li><li>join this DataFrame with the CUR (on time slot and instance type)</li></ul><p>One way to visualise the correlation is to use an heatmap where for each hourly data point (y-axis), for each instance type (x-axis) you have the max of usage of the instance type. If there is no value (ie. blank in the following picture), it means no instance for the instance-type was running (and billed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_zpn7TBxKQzqr56RlrV9ew.png" /></figure><p>Note that here the instance type is used but depending on your context, a service name, tags etc… could be more appropriate.</p><p>The interesting part is that you can start understanding how your resources are used and if they are fully used. It will likely help you to drive</p><ul><li><em>discussions around sizing</em>; are we using the right instance-type? assuming most of the time the max CPU consumption is not over 50% can we use a smaller instance and be more horizontally scalable and elastic?</li><li><em>discussions around elasticity</em>; can we be more elastic? in particular can we release “unused instances”</li></ul><h3>Correlate (ec2) cost and your workload</h3><p><em>This is particularly true for the AWS account(s) that will host your dev &amp; test environments : </em>you need to have a better understanding of how your AWS infra is used, based on your workload. In dev &amp; test you will run a lot of tests, machines can be kept up because of debugging needs, you can have randomness in the deployment pipeline or flakiness in the test itself, your testing strategy could be too heavily relying on “integration or end-to-end tests” that are executing way too frequently and, in addition failed.</p><p>To better understand this space, one option is :</p><ul><li>to tag each AWS resources with the test execution id or something equivalent that represent “an instance of the test execution”</li><li><em>export these tags as part of your CUR</em>; meaning you will be able to know exactly how much each test execution costs you</li><li>export your test execution reports so that you know how long each test execution last, its status (green/ red), why it failed</li><li>potentially rely on Cloud Trail logs (that contains these tags). In particular for ec2 it will help you understanding which test execution started which instances and for how long, by which user…</li></ul><p>From there, there are the following main things to visualise :</p><ul><li>the total duration of all test execution over a period break down per status (green vs. failed). It will help you improving your tests, shift some of them left. Nothing else than the <a href="https://martinfowler.com/articles/practical-test-pyramid.html">Test Pyramid</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFhjFG0_ATZr_1FmU2x5dg.png" /></figure><ul><li>the main reasons of failures. Again it could be assertion, applicative code, randomness in the test or in the deployment pipeline itself. It will generate a backlog of “things to improve or fix”. Sounds obvious hum?</li><li>the lifecycle of ec2 instances based on Cloud Trail. you could be surprise to see that instances that should be up for the duration of a tests (let”s say a couple of hours) are in reality up for several days. The image below shows one way of displaying this data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RMdexUv0TFB0ZdqVW4lvQ.png" /></figure><p>And at this stage, you should have some interesting statistics which does not mean you will deeply understand why you have these profiles of test executions, why machines are kept so long etc. But you are now welcomed to the world of discipline and “continuous improvements”.</p><h3>Conclusion</h3><p>It is hard to provide a unique recipe to optimise your AWS costs as there are so many different architectures and workloads but using <em>Jupyter Notebook </em>(and the whole python ecosystem) was a real game changer in the way to understand our costs and also correlate (via AWS tags) them with internal data sources (like tests executions, AWS Cloud Trail, monitoring, infrastructure consumptions etc…).</p><p>This is also — from organisation perspective — a multi-facet topic as it will likely implies architecture, quality, people in charge of AWS landing zones, maybe your alliance teams…</p><p>In all cases, this is a long and not so easy journey to the world of “continuous improvements”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec2c8172919e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS FxLustre, Athena & QuickSight to ease handling your data]]></title>
            <link>https://medium.com/@omallassi/using-aws-fxlustre-athena-quicksight-to-ease-handling-your-data-1866cbe7dbcd?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1866cbe7dbcd</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Sat, 10 Jun 2023 13:31:30 GMT</pubDate>
            <atom:updated>2023-06-10T13:31:30.442Z</atom:updated>
            <content:encoded><![CDATA[<p>As soon as you start using AWS services, there is a high chance that some of data will end up stored in AWS S3. In particular, if your “applications” are computation or simulation oriented, you may generate and store — beside the final results themselves that are usually and at most a couple of MBs — a lot of intermediary data and a lot of (technical or functional) troubleshooting related data that are useful to explain the results themselves (and this part generally represents TBs).</p><p>Later, you will enter the “<em>how do I access these data</em>” discussion. Indeed, if you store them, this is likely that at some points they are useful to support some activities; meaning you need to access these data.</p><p>In addition, it is not unlikely that you work with legacy code, that in best case assumes there is a POSIX File System (e.g. NFS), uses log4j or log4cxx or directly output things into stdout or stderr; resulting in data stored in json, xml… but for sure not in the way you would like them to be stored.</p><p>And to complete the picture, the time criticality (or maybe just common sense) forces you to “lift and shift” this part, at least for the first increment(s).</p><p>Long story short; <em>(1)</em> you need to store all these data, <em>(2) </em>you cannot really afford the cost of cleaning the code that generate them (at least for now) and <em>(3) </em>you need to provide filtered access via APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*oC5dSbo7Kgi8SFKrRQ5eEg.png" /></figure><p>The following depicts <em>one way</em> of doing it (keeping in mind only bits can be used)</p><h3>Shipping &amp; Sharing your data to AWS S3</h3><p>Clearly there are multiple ways of doing this. Here we will assume an existing compute-intensive application made of multiple processes that generates, transfers and shares TBs of data, that has been designed to work with NFS (<em>ie.</em> POSIX). One possible way is to integrate with <em>AWS FxLustre</em>. Of course you have alternatives like using EBS or EFS or even have you “own” NFS setup based on ec2 disks and it will work but;</p><ul><li><em>FxLustre</em> will help you as it can be tranparently mounted to multiple ec2 instances ; which will provide a completely transparent integration w/ your existing application. It is likely that you observe some regressions in your performance tests (likely because your accesses to local disks now induced network) but it could be mitigated w/ parallelim (multi-threading / processes of your application)</li><li><em>FxLustre</em> may the cheapest option in particular if you need “high” bandwidth (<em>e.g.</em> 500 MBps/TiB compared to the default EFS one). As said, “it may be”…but to be clearer, EFS for instance comes with default bandwidth that increase with the volume (in TiB) you consume. you may be in a situation where you need additional bandwidth for the same volume and here, <em>FxLustre</em> may be a better option.</li><li><em>FxLustre</em> can help you with recovery as your storage is decoupled from your compute and can be made HA (depending on the deployment options you choose). This is interesting as your legacy application is certainly using the storage to store state, share data etc…</li><li><em>FxLustre</em> provides an out-of-the-box bi-directional integration w/ AWS S3 that could be useful because your data retention for some of these data is longer than the computation itself, because it integrates with your disaster recovery procedures.</li><li>it may also help you with security as the integration <em>FxLustre</em> / <em>S3</em> relies on <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html">service-linked role</a>; meaning you may not have to handle it yourself based on ec2 roles or AWS STS token etc…</li></ul><p>So, by integrating <em>AWS FxLustre, </em>you have reached the following state;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*lkMoeaxiesmz1u4rAi0iqQ.png" /></figure><h3>Preparing your data for read queries</h3><p>Again, assuming a legacy application, this is very unlikely that the way you store the data is adapted to the way you need to query it:</p><ul><li>the data could be spread across multiple files that maps how the data was generated but not how the data will be used</li><li>the data you need may be lost into a myriad of log lines…</li><li>the data is not necessarily json. Indeed if you are dealing w/ legacy code, it could be csv-like format, XML, HDF5…</li><li>the data could be hierarchical, one example being as below that generically represents a graph</li></ul><pre>{&quot;version&quot;:1,<br>...<br>&quot;data&quot;:[<br>  {&quot;name&quot;:&quot;-1&quot;,<br>...<br>    &quot;children&quot;:[<br>      {&quot;name&quot;:&quot;10015&quot;, <br>       &quot;data&quot;:{&quot;value&quot;:&quot;5&quot;,&quot;additional_attributes&quot;:null},&quot;outputs&quot;:[],<br>       &quot;children&quot;:[<br>        {&quot;name&quot;:&quot;10001&quot;,<br>         &quot;data&quot;:{&quot;value&quot;:&quot;AAA&quot;,&quot;additional_attributes&quot;:null},<br>         &quot;outputs&quot;:[],<br>         &quot;children&quot;:[<br>          {&quot;name&quot;:&quot;10100&quot;, <br>            &quot;data&quot;:{&quot;value&quot;:&quot;13763&quot;,&quot;additional_attributes&quot;:null},<br>            &quot;outputs&quot;:[<br>              {&quot;name&quot;:...<br>              {&quot;name&quot;:....<br>...<br>            ],<br>            &quot;children&quot;:[]},<br>          {&quot;name&quot;:&quot;10100&quot;,<br>            &quot;data&quot;:{&quot;value&quot;:&quot;13764&quot;,&quot;additional_attributes&quot;:null},<br>            &quot;outputs&quot;:[...<br>...</pre><p>Beside doing nothing, one option here would be to rely on AWS S3 notifications and Lambda integration; meaning when logs are written into AWS S3, you can execute code to post process the data and create / store it in a format or structure that is more adapted to the way you will query it later. As an example, if you know you will need to get a set of data by ids but is currenlty lost in the middle of log files, you can post-process these log files and store multiple files, each organised by id (which comes back to a lookup into AWS S3 at the end).</p><p>Meaning you can reach the following state;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*LmfpCFF0Rlpqz9NSN0HTVg.png" /></figure><h3>Query your AWS S3 data via AWS Athena &amp; SQL like access</h3><p>At this point in time, your data is shipped to AWS S3 and the question becomes “<em>how to query and value them?</em>”</p><p>In case the previous step was not enough, you can integrate w/<em> AWS Athena</em>. I have to admit there is something magic behind <em>AWS Athena</em> (and <em>Presto</em> generally) in a sense that you can use SQL queries to transform your initial json files and create table (or views), and of course query these tables. It also considers the “typical” use-cases of hierarchical data w/ the cross join unnest ; meaning you can transform your very hierarchical data (as the json above) into a flat table. The below example is not ideal but shows the feasibility of flattening your graph and strongly types your json attributes</p><pre>with results as (<br> select<br>   json_extract(results, &#39;$.data&#39;) as results_data,<br>   results as results<br> from &quot;default&quot;.&quot;results_raw_json&quot;<br> cross join unnest(cast(json_extract(endstage.result, &#39;$.views&#39;) as array(json))) as t(results)<br>),<br>first_children as (<br> select <br>   json_extract(results_data, &#39;$.data&#39;) as first_children,<br>   first_children<br> from results<br> cross join unnest(cast(json_extract(results_data, &#39;$&#39;) as array(json))) as t(first_children)<br>),<br><br>…<br>fifth_children as <br>(<br> select <br> col_entity,<br> col_ctp, <br> col_date,<br> json_size(fourth_children, &#39;$.outputs&#39;) as data_size,<br> cast(json_extract(fourth_children, &#39;$.outputs[0].data.value&#39;) as double) as data_0,<br> cast(json_extract(fourth_children, &#39;$.outputs[1].data.value&#39;) as double) as data_1,<br> ...<br> json_extract(fourth_children, &#39;$.outputs&#39;) as fifth_children<br> from fourth_children<br> cross join unnest(cast(json_extract(fourth_children, &#39;$.outputs&#39;) as array(json))) as t(fifth_children)<br>)<br>select <br> data_size, col_entity, col_ctp, col_date, data_0,<br>   ....fifth_children<br>from fifth_children <br>order by col_ctp asc, col_date asc limit 1200</pre><p>The previous query will provide you with the following flat table that can then be queried via JDBC / ODBC…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wRmhL3fKnPfpn8ibDyLAw.png" /></figure><h3>QuickSight to go evenfurther and create interactive dashboards</h3><p>In some cases, you may need dashboards to visualize these data. Of course you can develop ad-hoc Web UIs on top of previously provided APIs but sometimes you may need something that is more “BI oriented” (w/ breakdowns, drill down, graphs etc…). Here QuickSight can help you as it integrates w/ a large variety of AWS Storage and in particular w/ <em>AWS S3</em> and <em>AWS Athena</em>.</p><p>The below are basically screenshots of screens created on top of Athena</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*We0Syr7EcXLCb-KOGWgLqQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*l0hEx-XBag9W9nlGGOdVXw.png" /></figure><h3>Which at the ends, gives us</h3><p>At this stage, your architecture looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhSPHJgCoNjwfWj9Qb8J4w.png" /></figure><p>Of course, in most of the cases you will not need all these pieces but “who knows”. Of course, it is more complex but this could provide an evolution path of your a legacy application architecture towards another architecture which leverages AWS managed services, which generates required data in the right format etc.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1866cbe7dbcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrate Contract Testing into your API Governance]]></title>
            <link>https://medium.com/@omallassi/integrate-contract-testing-into-your-api-governance-3839b9d65a77?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3839b9d65a77</guid>
            <category><![CDATA[contract-testing]]></category>
            <category><![CDATA[api-governance]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Wed, 22 Mar 2023 06:12:09 GMT</pubDate>
            <atom:updated>2023-03-22T06:12:09.076Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integrate Contract Testing into your API Design</h3><p>Along the time, you will evolve your fonctionnalities and APIs will evolve, that’s a fact. The question then becomes “<em>how do I ensure they evolve in a proper way?</em>” and “<em>how do you define proper way in your context?</em>”</p><p>One thing that is generally implicit in “<em>proper way</em>” is backward compatibility or “<em>is my API evolution backward compatible w.r.t to my consumers?</em>”. This sounds like a simple question and we all know about the good practices around backward compatibility (<em>e.g.</em> add only optional fields, do not remove fields, use default values etc.). The thing is that with the increasing number of APIs, the increasing number of consumers, the different type of audience of your APIs, these rules will no longer looks so simple to apply. As examples : you expose a field that is in fact not used and you would like to remove it (and you still can…), you “control” the consumer and producer lifecycle (in terms of deployment) and can change both at the same time, your modelling has evolve and your state machine is much more well defined and the <em>READY</em> state becomes <em>ALMOST_READY</em> and <em>MORE_THAN_READY</em>…Or you made a mistake and model an attribute as a string and not datetime but your consumer are already sending a datetime. All of these “small mistakes” happens and the reality — assuming we do not talk here about public APIs — is that it is sometimes doable or far more easier to have breaking changes than bumping the major versions and supporting the two versions while consumers are moving…</p><p>Yet, with the increasing number of endpoints — for instance 300+ endpoints — this is just impossible to know for sure if this specific endpoint can evolve in a non backward compatible way, or not and which consumers may be impacted.</p><p>There are two mechanisms that easily helps. The first one is the <a href="https://opensource.zalando.com/restful-api-guidelines/#219">x-audience</a> that helps understanding the “scope of use” of the API. The second one is <a href="https://github.com/OpenAPITools/openapi-diff">openapi-diff</a> that tells you if this is a backward compatible change or not by comparing two OpenAPI specifications. The x-audience is often “fragile” and you end up lacking of confidence because all of this evolve and you are no longer sure the x-audience is up-to-date. The openapi-diff works fine but as we mentioned earlier, sometimes we may afford a non-backward compatible changes w/o bumping the major version of the API.</p><p>A third and complementary mechanism would <strong>consist in integrating contract-testing directly in your API design phase</strong> and validate against declared pact contracts that your API evolution does not break anything or understand which and how many consumers are broken. Of course, this implies you are already relying on contract testing as part of your testing practices and have a <a href="https://docs.pact.io/pact_broker">pact-broker</a> or <a href="https://pactflow.io/">pact-flow</a> or equivalent available in your organisation.</p><p>The overall solutions looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*ocuynYAFlpguJJZfnmxXjA.png" /></figure><p>Long story short;</p><ul><li>Given an OAI specification you can start a mock implementation based on <a href="https://stoplight.io/open-source/prism">prism</a>. Prism can use the OAI examples defined in the specification as the mock implementation. This is also worth noticing that <a href="https://docs.stoplight.io/docs/spectral/4dec24461f3af-open-api-rules#oas3-valid-schema-example">spectral can help you ensuring example are inline w/ declared schemas</a>.</li><li>you can for instance annotate your OAI specification with a custom annotation (<em>e.g. </em>x-provider-name) that specifies the provider name. This will be useful to execute the provider test.</li></ul><pre>...<br>x-provider-name: &quot;My Provider Name&quot;<br><br>paths:<br>  /v1/audit/...:</pre><ul><li>you can have a generic provider test that will get the pact contracts and execute them against your provider implementation. For instance, in Javascript, something like below helps:</li></ul><pre>const { Verifier } = require(&#39;@pact-foundation/pact&#39;);<br>describe(&#39;Pact test&#39;, () =&gt; {<br>  describe(&#39;testAllMethods&#39;, () =&gt; {<br>  return new Verifier({<br>    logLevel: process.env.logLevel || &#39;debug&#39;,<br>    pactBrokerUrl: process.env.pactBroker || &#39;https://homz.pactflow.io&#39;,<br>    pactBrokerToken: &#39;lfJV8........w&#39;,<br>    //the server against which the pact contracts can be tested<br>    providerBaseUrl: process.env.providerBaseUrl || &#39;http://localhost:4010&#39;, <br>    //at runtime, will be set with the x-provider-name<br>    provider: process.env.providerName || &#39;ProviderMayNotExist&#39;,<br>    providerVersion: process.env.providerVersion || &#39;0.0.0&#39;,<br>    providerVersionTags: [&#39;api:v1&#39;],<br>    consumerVersionSelectors: [<br>      {latest: true,}, <br>      {deployedOrReleased: true,}],<br>    changeOrigin:true, <br>    enablePending: true,<br>    publishVerificationResult: false,<br>}).verifyProvider();<br>})<br>});</pre><ul><li>a Jenkins file (or anything else) to orchestrate “all this little world”. As an example;</li></ul><pre>.....//as a stage for instance<br>&quot;Contract Testing&quot;: {<br>script {<br>if (env.BRANCH_NAME != &#39;master&#39; &amp;&amp; env.CHANGE_ID == null){<br>    echo &#39;will do nothing as this looks to be a branch, need a PR to compare files&#39;<br>}<br>else {<br>  script{<br>    try{<br>      <br>      def modified_specs=//get all modified specs as an array<br>      def prism_host=&quot;0.0.0.0&quot;<br>      def prism_port=4009<br>      for (int i = 0; i &lt; modified_specs.size(); i++){<br>        //for each spec<br>        prism_port=prism_port + 1<br>        def modified_spec=modified_specs[i] <br>        spec_as_yaml = readYaml file: modified_spec<br><br>        //get the provider name from the spec<br>        def provider_name=&quot;ProviderNotExist&quot;<br>        if (spec_as_yaml.containsKey(&quot;x-provider-name&quot;)) {<br>          provider_name = spec_as_yaml.get(&quot;x-provider-name&quot;)<br>          println(&quot;Got providerName [${provider_name}] for [${modified_spec}]&quot;)<br>        }<br>        else{<br>          error(&quot;no [x-provider-name] found in [${modified_spec}]&quot;)<br>        }<br>        println(&quot;[mocked] the following spec [${modified_spec}] have been modified - contract testing will be run for provider [${provider_name}]&quot;)<br><br>        //will start a prism docker container<br>        container(&#39;prism&#39;){<br>          try {<br>            echo &quot;Is starting provider implementation for spec [${modified_spec}]&quot;<br>            sh &#39;touch prism.log&#39;<br>            sh &quot;node /usr/src/prism/packages/cli/dist/index.js mock -h ${prism_host} -p ${prism_port} ${modified_spec} &gt; prism.log 2&gt;&amp;1 &amp;&quot;<br>            sh &#39;echo $! &gt; prism.pid&#39;<br>            //TODO find a better exit condition<br>            sh &#39;(tail -f prism.log &amp;) | grep -q &quot;Prism is listening on http://&quot;&#39;<br>            sh &#39;cat prism.log&#39;<br>          } catch (err) {<br>            echo err.getMessage()<br>          }<br>        }<br>        //will execute a generic provide test (here in js)<br>        container(&#39;node19&#39;){<br>          echo &quot;Will run pact tests for provider [${provider_name}]&quot;<br>          sh &#39;node -v&#39;<br>          //assuming you run behind a firewall<br>          sh &#39;cd contract-testing-src &amp;&amp; npm install --loglevel verbose --tarball=...node-v19.4.0-headers.tar.gz --registry=http://internal-registry/nexus3/repository/npm-all/&#39;<br>          sh &quot;cd contract-testing-src &amp;&amp; providerName=\&quot;${provider_name}\&quot; providerBaseUrl=http://127.0.0.1:${prism_port} npm run test:producer&quot;<br>        }<br>        container(&#39;prism&#39;){<br>          sh &#39;cat prism.log&#39;<br>        }<br>    }<br>  }<br>  catch(ex){<br>    //TODO<br>  }<br>  finally{<br>    //TODO<br>  }<br>}<br>}</pre><p>This is not that difficult to integrate your provider tests in your API design phase and get a more concrete feedback about your API evolution and if it is breaking consumers or not (before writing a single line of code). There are for sure some limitations here; for instance all consumers tests cannot be handled (e.g. the ones that use <em>given a state</em> will not work etc…). Yet, it helps and that would be awesome to hear about equivalent approaches in other contexts. It is also worth noticing commercial options like the <a href="https://pactflow.io/integrations/swaggerhub/">SwaggerHub and PactFlow</a> integration that also aims to solve this issue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3839b9d65a77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some toughts about APIs…]]></title>
            <link>https://medium.com/@omallassi/some-toughts-about-apis-5ee87829f4f8?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ee87829f4f8</guid>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[api-design]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[api-governance]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 11:41:33 GMT</pubDate>
            <atom:updated>2023-03-13T11:41:33.569Z</atom:updated>
            <content:encoded><![CDATA[<h3>“Some Thoughts” About APIs…</h3><p>&quot;Creating APIs” is clearly not something new and a lot has already been said and written. <strong>Yet</strong> I have to admit I am always baffled by the complexity, the broadness and at the same time the level of details and precision it requires and maybe most of all by the “holistic approach” this type of design demands.</p><p>“Holistic” could sound like a “strong word” but after a couple of years being exposed or involved in these discussions, I came to the conclusion that “Creating APIs” means considering <strong>all</strong> (not one of) the following topics.</p><h3>Modeling</h3><p>“yep”, an API is one of the concretisation of the modeling you have defined for your domain;</p><ul><li>it exposes your <strong>ubiquitous langage</strong>, it exposes the relationship between your objects(composition, association, aggregates… whatever the terminology), it exposes your “domain service or events”.</li><li>it has to respect some <strong>naming conventions</strong> that can be more (how are domains named) or less (enum are uppercases etc…) tricky.</li><li>it should respect <strong>“technical-ish” good pratices</strong> (in particular if you think about RESTful APIs).</li><li>it is described in an <strong>IDL</strong> (likely OpenAPI is you are using REST, an interface sometimes), and an IDL has a grammar</li><li>it <strong>does not </strong>(or try to not)<strong> expose implementation details</strong>; the most famous one being “<em>am I exposing my sequence id as resource identifier or an UUID?</em>”</li></ul><h3>Dev Experience</h3><p>This is mostly about “<em>how you make developpers’ life as easy as possible</em>”. Likely, you will end up discussion about;</p><ul><li><strong>contract first approach</strong> in which you generate the code from the specification is — in my mind and in typical langages like java, cpp etc… — the most reliable approach. It will help you to include and evolve cross cutting concerns like authentication, authorisations …It will also enable you to ensure — by design — your implementation is inline with the designed API.</li><li><strong>testing</strong>. You will also have topics around “<em>how do I test my APIs</em>” and there you will discuss <strong>contract testing </strong>or <strong>BDD</strong> to define scenarios that illustrate typical usages of your APIs and also detect breaking changes.</li></ul><h3>Versioning</h3><p>This topic is huge and goes from product management to deployment of the APIs;</p><ul><li>what are the <strong>APIs versionning policies</strong> at corporate levels. how many major versions do we support? how many elapsed time for decommissionning? have all APIs the same versioning policies? is this depending on the <a href="https://opensource.zalando.com/restful-api-guidelines/#219">audience</a>?</li><li>are my <strong>APIs evolutions backward compatible</strong> or not? how do I ensure this (including older versions)?</li><li>is my provider implementation still <em>backward compatible</em> with consumers?</li><li>how do I <strong>model my versioning</strong>? is this part of the URL, is this an header ? how does this affect my load balancing? am I still able to route HTTP requests to different instances of the services etc…</li></ul><p>There are no right answer but we do not all have the level of excellence of AWS (or others) on this topic.</p><h3>Quota Management</h3><p>Exposing APIs, you are likely to need “<strong>to control how your customers are using these APIs</strong>” and ensure it does not blow your cost up. Thus the questions of quotas: how can my customer use the system, at which frequency, how do I bill the customer back, are my quota hard limit or soft limits and is this enforced then <em>[1]</em>…</p><p>On the technical side, you will likely rely on an API Gateway for this, at least as a first level of defense because you may have more complex dimensions than a number of HTTP calls impacting your cost. But it is worth noticing <strong>this is likely to impact your APIs modeling</strong> as<em> (1) </em>API key will be a mandatory header and <em>(2)</em> quota are usually based on URL path so distinguish two “activities” could be easier if modeled as different path than an attribute in the payload.</p><p><em>[1] Note that quota is not only a cost concern but a way to protect your platform against unexpected usage.</em></p><h3>Deployment, performance &amp; observability</h3><p>API Gateways have proven their usefullness w.r.t API lifecycle management but they are part of larger processes</p><ul><li>is this version of the API deployable? should we un-deploy this version of the API (or not)?</li><li>how do we monitor APIs usage?</li><li>are my endpoints performing well? am I respecting my SLOs?</li><li>is my payload to big so that I need to paginate? how do I handle the case where my payload is upper than the 10MB API Gateway limit?</li><li>do I add this header or not keeping in mind the total size can also be capped? do I add this claim in my Bearer or is this “too big”?</li></ul><p>In some cases, <strong>this “performance-related” issues will also impact your modeling</strong>.</p><h3>Security</h3><p>All of the <a href="https://owasp.org/www-project-api-security/">API Security aspects</a> will be handled in different phases of you CI/CD pipelines but some of them can and should be shifted left;</p><ul><li><em>BOLA</em>, <em>BFLA</em> are things you are think about while designing the API. what are the permissions, how do I segregate them etc…As a simple example, using UUID instead of monotonic integer for resource ID will not solve BOLA but is an easy solution that can help.</li><li><em>Excessive Data Exposure</em> is also something that should be done early in API design phase to desing a response, maybe changing a little bit your modeling and have two resources instead of one.</li><li><em>Mass Assignment</em> and <em>Injection</em> could be not solved but “limited” by having scheme (e.g, string regex, string max length etc…). This sounds obvious but too often not done (especially if you think about APIs that were internally used and start being exposed to larger audience)</li><li>“<em>platform protection</em>” (DoS…) that rely on quota management also…</li></ul><p>So clearly, <strong>all will not be handled at design level and it will require a lot of tests but having these items in mind while designing APIs will help you</strong> down the road.</p><h3>Governance (or Peer Reviews…)</h3><p>Governance often sounds like the forbidden word but as Google explained in <a href="https://research.google/pubs/pub45294/">API Design Reviews at Scale</a>, <strong>having review of API design clearly improves the quality </strong>and help ensuring most of the concerns have been considered, at a lower cost than doing it once tested, implemented and even deployed and used.</p><ul><li>you <em>can detect and avoid implementation leakage in your API design</em>. The most famous one being to use your DB sequence ID as your resource ID.</li><li>understand if you are <em>able to stick to the “unique System Of Record” principle</em> or if you start observing multiple write method on “something that looks like the same resources….”.</li><li>understand if the<em> underlying implementation or service seems maybe to “know” too much data</em>. For instance if you have an API and a service to handle dispute, it may be fair to have customer related information. But it may be weird to have the dispute system returning the address or phone or age or… of the customer. Again it may be correct in your modelling and these are symptoms.</li><li>measurements. This is very interesting to have metrics to better understand “<em>your API space</em>”: one many endpoints, how are they spread across domains, how many permissions, how many x-zally-ignore…These are elements to help you understanding the volumes you are manipulating because changing a naming convention on 2 endpoints is not the same than if you have 2000 endpoints across 60 teams. I have tried to <a href="https://github.com/omallassi/apis-catalog-web">POC to get these information</a> and may need to refocus on it.</li><li><em>understand your permissions</em>. Are all my endpoints authorized? This is particularly interesting if integrated w/ contract first and code generation approach because it gives you a strong process to limit the number of “unprotected” endpoints.</li></ul><h3>Search-ability (as maybe a first step in discoverability)</h3><p>Assuming your APIs will grow (internally or externally), you are likely to have the need to be able to search and discover them. A typical question would be “hum, I need to get the counterparty details sector and area, do I have a way to get them?”. So here, having a way to search across your API, to be able to understand them will help you a lot. Also depending on your architecture (if you have open / close layers etc…) having this documentation organised this way will help.</p><h3>…</h3><p>There are likely others topics I am not aware of or simply forgot here but “Creating APIs” requires to think about all of these elements and this completely agnotic from the “<em>type of API</em>” (AMPQ, gRPC, REST, native SDK etc…..) even if the underlying tools will differ, of course.</p><h3>A pipeline to “rule them all”</h3><p>I was trying to find a way to use this sentence…so forget about this part of the previous line.</p><p>Yet, an of course commercial tools exist and will help a lot, this is feasible to have a pipeline that integrates some tools to help you in this journey</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*s7xk36luJH5nQMgEMuXivA.png" /></figure><ul><li><strong>respect of Modeling</strong>. you are on your own…or maybe ChatGPT can do it for you. Joke apart, I am not aware of tools that can support this but having design sessions is key here.</li><li><strong>respect your IDL Grammar</strong>. Assuming <a href="https://www.openapis.org/">OpenAPI</a>, tools like <a href="https://stoplight.io/open-source/spectral">spectral</a> can be easily integrated (w/ the default or custom rules) and will help ensuring consistency amongst your OAI specifications.</li><li><strong>respect Guidelines &amp; naming conventions</strong>. All cannot be automatically done but tools like <a href="http://opensource.zalando.com/zally/">zally</a> (or even <a href="https://stoplight.io/open-source/spectral">spectral</a> with custom code) can help you to check some naming conventions (e.g. attributes are kebab-case or CamelCase, enums are uppercase, paths are not too long etc…)</li><li><strong>respect Security</strong>. Well, to be honest I have still not been really able to to integrate API Security tools (like <a href="https://owasp.org/www-community/Vulnerability_Scanning_Tools">ZAP</a> etc…) into pipelines. But at least, you can check the basics; ensuring permissions are defined via the OAI.security attribute (and fail if not). This is also feasible to check that string have a schema (max-length, regex..) associated.</li><li><strong>respect backward compatibiliy</strong>. here you can run some “<em>static and dynamic analysis</em>”. The static analysis could be based on <a href="https://github.com/OpenAPITools/openapi-diff">openapi-diff</a> and can be useful to compare the specification changes from your pull-request against the specification in the master and see if the changes are backward compatible. The dynamic analysis would consist in integrating contract-testing as part of the design of the API. For instance, you can annote your OAI specification with x-provider-name and you can rely on <a href="https://stoplight.io/open-source/prism">Prism</a> to generate a mocked (but generic) implementation of your provider and execute provider tests against consumer contracts declared in <a href="https://docs.pact.io/pact_broker">pact-broker</a>. This way, you are able, at design time to have a <em>first level</em> of check against declared consumers.</li><li><strong>respect versioning</strong>. additional checks can be done to ensure the right header is used for versioning, or that you bump the version attribute of your OAI spec accordingly to your changes.</li><li><strong>generate and deploy documentation</strong>. Surprisingly not necessarily the easiest part to be honest but <a href="https://redocly.com/">redoc</a> helps a lot here.</li></ul><h3>Long Story Short; a collaborative effort (and a little bit of discipline)</h3><p>It is likely that I miss something here and honestly, let me know. but I hope I have been able to properly exposed all the topics that relates to “Creating API” and how they influence each others;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zr_YqGfP6oKRbRAk3BQpEQ.png" /></figure><p>In short, “Creating API” requires a strong collaborative work, likely a lot of discussions often perceived as “lost of time”, of “theoritical” or… but in most of cases and if well facilitated, fruitful over the long run.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ee87829f4f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[About Sharing & Reuse…]]></title>
            <link>https://medium.com/@omallassi/about-sharing-reuse-3cd18974ad12?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd18974ad12</guid>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Sat, 17 Feb 2018 17:28:05 GMT</pubDate>
            <atom:updated>2018-02-17T17:28:05.842Z</atom:updated>
            <content:encoded><![CDATA[<p>A couple of years ago, I was in a discussion with one of my friends, about what at this time was called SOA and about “reuse”.</p><p>I have no idea nor why nor how I remember this part of the discussion, maybe a current context I am currently working in but from a general perspective, a system, an organisation can share the following things:</p><ul><li><strong>Nothing</strong>. Well, you are in big troubles…as it is likely that you in the middle of individuals, or many sub-systems. Yet, it exists...</li><li><strong>Culture</strong>. In “<a href="https://www.amazon.fr/Leading-Change-John-P-Kotter/dp/1422186431/ref=sr_1_1?s=english-books&amp;ie=UTF8&amp;qid=1518874848&amp;sr=1-1&amp;keywords=leading+change+kotter">Leading Change</a>”, John P. Kotter defines culture as <em>norms of behavior</em> and <em>shared values</em>. As you can imagine, all of this dictates how you interact with others, what is important, what is not, what is good, what is not etc… As stated by John P. Kotter, do not underestimate <a href="https://en.wikipedia.org/wiki/Boiling_frog">the ability of a culture to change yourself</a>, with time.</li><li><strong>Principles</strong>. <a href="https://www.principles.com/">Principles</a> are much more stringent that culture. As shown by <a href="https://www.principles.com/">Ray Dalio</a>, principles help with <em>(1) creating a culture</em> and <em>(2) making (consistent) decision</em> (within an organisation). There is a blur boundary between principles and culture but I tend to think that contrary to a culture, principles are explicit and should be written.</li><li><a href="https://en.wikipedia.org/wiki/Software_design_pattern"><strong>Design Patterns</strong></a>. A pattern is a “<em>general, reusable solution to a commonly occurring problem within a given context</em>”. it becomes then a way to describe &amp; share best practices. <br>For organisations that share previous items, then, it is not uncommon to share patterns. It is not uncommon neither to share patterns across organisations, this is quite obvious in Software Engineering or Architecture fields.</li><li><strong>Libs or code</strong>. Libs at some point <em>can be seen as an opiniated implementation of design patterns</em>. <a href="https://www.apache.org/">Apache</a> is made of hundred of examples. At this stage, people from an organisation trust people from another organisation enough so that they accept to reuse something external (and so build their code on code owned by someone else). As we know, it creates a kind of (compile-time) coupling but it clearly helps both part to be more “efficient”. The “customer” organisation is more productive, the “supplier” organisation is more focused and <a href="https://www.amazon.fr/Facts-Fallacies-Software-Engineering-Robert/dp/032111742%205">provide higher-quality code</a> (as these libs are used in many types of use-cases). In fact there is nothing really new here: we do this everyday with what we call “Open Source”. The real mystery for me is the following: “Assuming equivalent quality, why are we willing to easily use any open source technologies whereas we cannot trust code done by dev teams on the next floor?”. Do not be mistaken, I do believe in the Open Source model. But why is it so hard to <a href="https://www.oreilly.com/ideas/getting-started-with-innersource">set up the type of models inside our own organisations</a>?</li><li><strong>Services</strong>. This is the ultimate level, just look at what Cloud providers have built in 10 years to convince (if needed) yourself. The capabilities (hosted by the service) are here, somewhere, easy-to-use. <a href="http://akfpartners.com/growth-blog/when-should-you-split-services">The lifecycles are completely decoupled (not in terms of runtime but I let this to sales people…), organizations can move on their own pace, capabilities are massively reused</a>. This is also the more complex stage for the organization that provides the service.</li></ul><p>I like to keep these “stages” in mind; it helps me when moving from functional architecture to technical one, when discussing a design etc… I do not know if it can help you, if you can reuse it but I am sharing it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd18974ad12" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kill Switch Pattern: “Circuit Breaker” for AMQP consumers]]></title>
            <link>https://medium.com/@omallassi/kill-switch-pattern-circuit-breaker-for-for-amqp-consumers-9341b51b2a30?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9341b51b2a30</guid>
            <category><![CDATA[amqp]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Sun, 03 Sep 2017 09:05:55 GMT</pubDate>
            <atom:updated>2017-09-03T09:11:22.715Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="http://www.enterpriseintegrationpatterns.com/patterns/messaging/DeadLetterChannel.html">Dead Letter Queue</a> is a classical pattern for every applications that use a Messaging Infrastructure (e.g. Apache Qpid etc…). The nominal behaviour is that if a message is “rejected” (i.e. not acknowledged) by a consumer, the Messaging Infrastructure will replay this message. The number of times the message can be replayed is usually configured at the Messaging Infrastructure level; if the message replay exceeds the configured threshold, then the message can either be discarded or stored and kept in a Dead Letter Queue.</p><p>This pattern is widely used and <strong>works fine when the </strong><a href="http://www.oreilly.com/programming/free/understanding-message-brokers.csp"><strong>consumer cannot “treat” one message from time to time</strong></a> (e.g. the message cannot be deserialised). Yet, nowadays, it is likely that the consumer have <strong>more dependencies </strong>(on Downstream services) <strong>that could completely fails, resulting in rejection of all the incoming messages, and the saturation of the DLQ</strong> (with technical implications like disk space consumption and functional implications to treat all these messages…)</p><h4>Disconnect the consumer when too many messages cannot be “ack”</h4><p>The pattern can be enhanced by adding “<a href="https://martinfowler.com/bliki/CircuitBreaker.html">Circuit Breaker</a>s” that will</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*WGWuVJ43gjtIgM4wdIYrvw.png" /></figure><p>completely disconnect the consumer from the Messaging Infrastructure in case too many messages cannot be consumed (due for instance to a complete service interruption of external dependency) .</p><p>In fact, this is a kind of combination of two Circuit Breaker: (1) one that will open when external dependencies are unavailable, (2) one that will disconnect the consumer from the Messaging Infrastructure when the previous one gets opened.</p><p>The cool thing now is that as long as the consumer stays disconnected, messages are not replayed at all and thus cannot be routed to the DLQ.</p><h4>Some Tests…</h4><p>I did a small test and wrote a python consumer script that calls a downstream service which, in the context of the test, regularly fails. In short,</p><ul><li>Inject 500k messages as fast as possible to a Messaging Infrastructure that proposes a queue with a DLQ (a message can be replayed 3 times before being moved to the DLQ)</li><li>get a message from Messaging Infrastructure &amp; calls the downstream service (as fast as possible)</li><li>if the downstream service answers w/o errors (this is “randomly” handled by a background task that plays the <a href="https://medium.com/netflix-techblog/netflix-chaos-monkey-upgraded-1d679429be5d?source=linkShare-14b281727aa5-1504382820">monkey</a>…and runs every 20 to 30 sec), then the message is acknowledged.</li><li>else, reject the message. The message will thus be replayed by the Messaging Infrastructure. In this test, if the message is rejected more than 3 times, then it goes to the DLQ.</li><li>and open the circuit breakers: (1) downstream service is no longer called, (2) AMQP Receiver is closed (w/ its underlying connection).</li><li>and start some background tasks to try closing the circuit breaker (back to normal state)</li></ul><p>As you can see on this diagram, consumer throughput (orange line) goes to zero events/sec when the external dependency is unavailable: In fact, consumer is disconnected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhwfVpnmrdlHAPQK2GCGTQ.png" /></figure><p>But what about the DLQ? This diagram shows the number of messages in the DLQ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3sNqQDfr5dNLLaZIQIhOQQ.png" /></figure><p>As you can see, the DLQ contains 30 messages (not bad compared to the 500k injected messages) but this is not zero…</p><p>Actually, the explanation is that the tested implementation closes the circuit breaker of the AMQP consumer first (so reconnect it to the Messaging Infrastructure. In that case, the consumer gets messages (which can be replayed messages), try to call the downstream service that is maybe not up yet and can fail. In this case, the current message will rejected and the <em>delivery-count</em> AMQP header is increased. If you repeat this multiple times in a short time period (which is what the monkey does), then the configured limit of 3 replays is reached, and the message moved to the DLQ…</p><p>The implementation has then been changed to reconnect the consumer only when the downstream service is up again. Formulate otherwise, the implementation will close the circuit breaker of the service before the consumer one:</p><ul><li>While Circuit Breakers are opened, there is a task that regularly check the downstream service healthcheck and decide to (half-)open the circuit breakers or not.</li></ul><p>New results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sbHpXw51ZpOeudcJkcy3ig.png" /></figure><p>In that case, the results are more compelling since the DQL contains no messages.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9341b51b2a30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Events Log with Apache Geode]]></title>
            <link>https://medium.com/@omallassi/events-log-with-apache-geode-e2691b9cafb8?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2691b9cafb8</guid>
            <category><![CDATA[event-logs]]></category>
            <category><![CDATA[apache-geode]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Mon, 03 Oct 2016 19:39:00 GMT</pubDate>
            <atom:updated>2016-10-03T19:39:00.133Z</atom:updated>
            <content:encoded><![CDATA[<p>A lot has been said, <em>e.g.</em></p><ul><li><a href="http://www.confluent.io/blog/making-sense-of-stream-processing/">http://www.confluent.io/blog/making-sense-of-stream-processing/</a></li><li><a href="https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying">https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying</a></li><li>…</li></ul><p>regarding <strong>Events Log</strong>, their benefits, how it can help…This pattern is based on a couple of principles:</p><ul><li>This is an <strong>append-only storage of events</strong></li><li>All the <strong>events are immutable</strong>.</li><li>Each event has a <strong>unique and strictly monotonic offset</strong>.</li></ul><p>Regarding the technologies, <a href="http://kafka.apache.org/">Kafka</a> has been built with these principles in mind and has proven to work, at scale most of our systems will never be.</p><p>Yet, for whatever reasons (because you want to minimize technical diversity, because of latency, because of …) you cannot use <a href="http://kafka.apache.org/">Kafka</a> directly. So you have to re-implement things. Here are the main highlights to implement it on <a href="http://geode.incubator.apache.org/">Apache Geode</a>.</p><h4>The offset is hold in Apache Geode</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*4e96A9RrNoRevykItGhjhw.jpeg" /></figure><ul><li>The offset is set server-side, before inserting the events</li><li>The offset is hold in Apache Geode (mostly for recovery), thus, the dedicated region.</li><li>The offset is a long that is incremented when a new event is inserted.</li></ul><h4>Inserting an event is done via a Function</h4><p>In this case, saving an event is a little bit more complex than a simple <em>put</em>().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*RwniZ0JEu9smChe_WGHYVA.jpeg" /></figure><p>Thus, a “custom” implementation of a<em> put(</em>) method uses a<a href="http://geode.docs.pivotal.io/docs/developing/function_exec/how_function_execution_works.html"> Functio</a>n that save the event. This function is executed on the <em>“event</em>s” region and, to guarantee it is executed on a single node (the one that will store the event), the<em> withFilter(eventKey</em>) is used.</p><p>The algorithm is “simple”: (1) get the next offset value, (2) set this value on the current event, (3) put the current event into the “<em>events</em>” region (which is usually partitioned and redundant with, sometimes much more advanced configuration like disk-overflow).</p><h4>The devil is in the details …</h4><p>I have to admit that the tricky part is in the implementation of the “<em>put() Function</em>” and of the offset…</p><p>Yet, this gives you some highlights to implement an <strong>EventLog </strong>on<strong> </strong>Apache Geode: an immutable and ordered list of events…</p><p>…and the order is guarantee by the offset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2691b9cafb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalable & highly available AMQP Infrastructure w/ Apache Qpid]]></title>
            <link>https://medium.com/@omallassi/scalable-highly-available-amqp-infrastructure-w-apache-qpid-87dc222e06f8?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/87dc222e06f8</guid>
            <category><![CDATA[apache-qpid]]></category>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[amqp]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Thu, 29 Sep 2016 19:42:10 GMT</pubDate>
            <atom:updated>2016-10-14T06:52:26.707Z</atom:updated>
            <content:encoded><![CDATA[<p>AMQP is the de facto standard for <em>asynchronous integration</em>. Whereas AMQP 0.10 (and prior versions) were focused on the broker implementation (with the concepts of binding keys, exchange etc…), AMQP 1.0 is much more focus on the protocol itself and is far less prescriptive regarding the broker internals which can (or not) implement AMQP 0.10 (or prior versions).</p><p>AMQP 1.0 has generated a lot of discussion on the web, but there is something I like in this “new” version: AMQP 1.0 tends to reproduce what has been done a couple of decades ago with HTTP/1.1, create an interoperable an very descriptive protocol so that you can create large infrastructure. This is exactly what HTTP has enabled: interactions between platforms and Information System without knowing the underling infrastructure (firewall, reverse-proxies, HTTP servers, Servlet containers/nodeJS etc…)</p><p><a href="https://qpid.apache.org/">Apache Qpid</a> is one of the most complete open-source project that supports AMQP 1.0 evolutions and is, by itself, a complete ecosystem that provides:</p><ul><li>Two brokers implementations, <a href="https://qpid.apache.org/components/java-broker/index.html">one in java</a>, <a href="https://qpid.apache.org/components/cpp-broker/index.html">one in C++</a> (compliant with AMQP 0.10 and 1.0).</li><li>Client APIs for java (<a href="https://qpid.apache.org/proton/index.html">proton-j</a> or <a href="https://qpid.apache.org/components/jms/index.html">JMS</a>), C++ (<a href="https://qpid.apache.org/proton/index.html">proton-c</a>) and others (python…). proton-c has been designed to be integrable in large variety of environments and thus, has no external dependencies (e.g. boost etc…) which make it the perfect candidate to be used in legacy contexts.</li><li>A component called <a href="https://qpid.apache.org/components/dispatch-router/index.html">Dispatch Router</a>, which is, as named states it, a router. We will discuss this component in more details but it can be compared (from a high level perspective) to a HTTP reverse proxy / load balancer.</li></ul><h3>Content-based filtering &amp; QoS</h3><p>AMQP 1.0 does not specify anything regarding Content-based filtering. In fact AMQP 0.10 was much more explicit on this topic but the qpid brokers implementation, supporting AMQP 0.10 <a href="https://medium.com/@omallassi/amqp-headers-filtering-competing-consumers-decd50f3115#.i1rmw54et">helps with filterings</a>. Moreover, they also support “advanced”, but very useful features, to protect the infrastructure itself from unexpected clients behaviors. Typically, you can:</p><ul><li><em>Limit the destination sizes</em> (and thus disk space usage)</li><li><em>Control the messages retention duration</em>. Of course you can specify the message TTL via headers but you can also enforce TTL per queue and limit the history to, <em>e.g.</em>, 30 days.</li><li><em>Control the messages flow</em>. Flow control is part of the <a href="http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-complete-v1.0.pdf">specification</a> but this is also a useful mechanism to avoid a publisher to flood the infrastructure.</li></ul><h3>Infrastructure interoperability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*VB7Fowmz_m3tEWFpVjceTw.jpeg" /><figcaption>AMQP 1.0 Interoperability</figcaption></figure><p>AMQP 1.0 being more focused on the protocol itself than the broker concepts, the promise is that it is widely interoperable and that two AMQP endpoints can communicate to each other without knowing anything else from each other, exactly how we work with HTTP. <br>On this, one may notice that nothing impose you to have a broker as destination.</p><h3>Scalability &amp; distributed “destinations”</h3><p>A complex topic regarding messaging infrastructure is to make it scale beyond the capacity of a single broker (a.k.a. process). This is traditionally something we achieve by “<em>splitting by different things</em>” (the <a href="http://theartofscalability.com/">Y axis of the scale cube</a>) with some overhead regarding the administration of all those destinations and a real (but rare) limitation: the destination by itself is not distributed and is “limited” to a single process (<em>a.k.a.</em> broker)</p><p>This is one topic where the Dispatch Router will help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*rtkZ7MI-KLwzo_51vtE6Mw.jpeg" /><figcaption>Destination(s) scalability w/ Dispatch Router</figcaption></figure><p>As depicted in the picture, the destinations (colored boxes) are distributed across multiple brokers. The Dispatch Router (which is aware of these destinations), then load balanced the messages across the underlying brokers. For those who are familiar with HTTP, it can easily be compared to a Layer 7 Load balancer that load balanced HTTP requests across a pool of Tomcat. This is exactly the same architecture.</p><p>There are interesting points to note on this architecture:</p><ul><li>The AMQP destinations are <em>logically unique</em> (from a client standpoint, I use a unique <em>amqp://dispatchIP:5672/Destination#A</em> but <em>physically distributed</em> across multiple brokers.</li><li>The Dispatch Router ensures load balancing and improve availability (if a broker fails, another one will be used for incoming messages). Note that unconsumed messages from the failed broker may stay unconsumed as long as the broker has not been restarted.</li><li>Brokers can be from different (AMQP 1.0 compliant) implementations: qpid, activemq…and can, of course use their own HA mechanism (shared storage, Active/Passive etc…), depending on your TTR.</li><li>Nothing prevent you to use Content-based filtering if your broker support it, meaning: Publishers publish into a distributed exchange (e.g. topic), Messages can be filtered (using binding keys and JMSselector-like) and routed in (distributed) queues where (competing) consumers are listening.</li></ul><h3>AMQP Routing infrastructure &amp; Security</h3><p>Distributed destinations are one usage of the Dispatch Router. There are other and more advanced usages that will help you create a complete AMQP infrastructure, able to distribute and route messages across a network of brokers or AMQP endpoints, “simply” given an <em>amqp://</em> <a href="https://www.oasis-open.org/committees/download.php/50806/amqp-addressing-v1.0-wd03.pdf">address</a>.</p><p>As an example, you can create an infrastructure where <em>Destination#A</em> and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjDzwPSQQUnX6Tl5hF-a3Q.jpeg" /><figcaption>Complex AMQP Infrastructure with routing capabilities. More on this <a href="https://access.redhat.com/documentation/en/red-hat-jboss-a-mq/7.0/single/using-a-mq-interconnect/#routing">https://access.redhat.com/documentation/en/red-hat-jboss-a-mq/7.0/single/using-a-mq-interconnect/#routing</a></figcaption></figure><p><em>Destination #B</em> are distributed but where <em>Destination#C</em> is handled remotely and is transparently routed through multiple Dispatch Router. In this example, two of the Dispatch Router are authenticated via “client certificates”.</p><p>A really promising world especially if you think about this “<em>as a service</em>” where you tend to decouple the infrastructure from the usage (modulo the “<em>service contract</em>” but that’s another story).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87dc222e06f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining Stream Processing and In-Memory Data Grids for Near-Real Time Aggregation and…]]></title>
            <link>https://medium.com/@omallassi/combining-stream-processing-and-in-memory-data-grids-for-near-real-time-aggregation-and-5873bd78db05?source=rss-14b281727aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5873bd78db05</guid>
            <category><![CDATA[apache-storm]]></category>
            <category><![CDATA[apache-geode]]></category>
            <dc:creator><![CDATA[Olivier Mallassi]]></dc:creator>
            <pubDate>Thu, 31 Mar 2016 12:04:07 GMT</pubDate>
            <atom:updated>2016-03-31T12:05:45.163Z</atom:updated>
            <content:encoded><![CDATA[<h3>Combining Stream Processing and In-Memory Data Grids for Near-Real Time Aggregation and Notifications</h3><p>I hope I will have the time to go into deeper details but for now, this is all I have…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Fembed_code%2Fkey%2FJ8V4NRCJQU0xoo&amp;url=http%3A%2F%2Fwww.slideshare.net%2FPivotalOpenSourceHub%2Fgeodesummit-combining-stream-processing-and-inmemory-data-grids-for-nearreal-time-aggregation-and-notifications&amp;image=http%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2F1145omallassi-1-0-160315224808-thumbnail-4.jpg%3Fcb%3D1458082239&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=slideshare" width="600" height="500" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aa4d12f868da09500dadc2c7666b3d7/href">https://medium.com/media/1aa4d12f868da09500dadc2c7666b3d7/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5873bd78db05" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>