<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maxime Thomas on Medium]]></title>
        <description><![CDATA[Stories by Maxime Thomas on Medium]]></description>
        <link>https://medium.com/@maximethomas_46287?source=rss-b55541eeb7e1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*TpU7IBRMsGCtlnjM.</url>
            <title>Stories by Maxime Thomas on Medium</title>
            <link>https://medium.com/@maximethomas_46287?source=rss-b55541eeb7e1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 06:28:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maximethomas_46287/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to scale Data Management ?]]></title>
            <link>https://medium.com/swlh/how-to-scale-data-management-5bad3f5f26b5?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bad3f5f26b5</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Thu, 30 Nov 2017 15:41:31 GMT</pubDate>
            <atom:updated>2017-12-02T04:47:32.646Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpL4cKhOYEedFyvPRdSvPg.jpeg" /><figcaption>Collecting data is serious matter — with no data, no good decisions — Courtesy of <a href="https://quotefancy.com/quote/1163033/Philip-K-Dick-Don-t-try-to-solve-serious-matters-in-the-middle-of-the-night">Quotefancy</a></figcaption></figure><p>When starting a business adventure, the data you may collect is very important as it will lead your vision of your product and your business. Data gives you indirect feedback of how people are using your platform and how they consume it. When growing on the data team, you will get new challenges from raw data and basic analysis to building dashboards for global governance.</p><p>It’s easy to get some free analytics tool and implement it in your different clients and have basic analysis about users browsing path or amount.</p><p>You shall have difficulties to manage all that data :</p><ul><li><strong>collecting</strong> data from those free tools: it’s going to be expensive or limited;</li><li><strong>normalizing </strong>everything : you need to script transformation macros or whatever to enrich and fix your data;</li><li>dealing with <strong>several sources </strong>of data as all those free tools are specialized on a specific data collection (only mobile device, only for ads, etc.);</li><li>dealing with <strong>several destinations </strong>once you have collected data;</li><li>handling <strong>availability, coherence and freshness </strong>of the data;</li></ul><p>In this article, I describe why it’s important to <strong>focus </strong>on the correct level of data depending on needs, skills and maturity of your company. We also talk about <strong>solutions</strong> and how to implement it, it may give you a good look on what <strong>options</strong> are available at this moment to answer to all those issues.</p><h3>Big plan for Big Data</h3><p>Transcript from executive committee meeting :</p><blockquote>We have a big plan for big data, we are going to hack the market, provide best product to our users, and maximize income.</blockquote><blockquote>We are going to collect everything, from FB Connect to strong personal data and are going to exploit it in real time to get a reactive response from our platform so users can be more engaged and it will increase the transform rate to 80%.</blockquote><blockquote>Reports from the last 5 years show than we have really well performed. We can compute any metrics as we know the semantic of our data. We use these data to predict the trend for next year.</blockquote><p>This is more or less the act of faith of the digital entrepreneur / C-level on the data perspective. We know what we want, now we have to understand how to get it.</p><p>Some people may believe that the data driven way is too much when you are in a small company, it’s exactly wrong. Data is the wheel of your car, without it you can’t drive, you just go somewhere your car has decided.</p><p>In the following sections, I give keys to understand the big picture that goes with the data driven approach and all the issues you may encounter in the process of handling data in your company.</p><h4>Company vital constraints</h4><p>First of all, you have to understand the implied constraints of a data project in the company which is the first brake for data management :</p><ul><li><strong>Cost</strong> : it costs money, it’s not free : it’s not because it’s simple to implement that it is free. You have to consider time of dev / ops on implementing and designing the solution;</li><li><strong>Availability</strong> : the solution must be available and should produce data analysis in a decent time frame, else it is useless;</li><li><strong>Scalability</strong> : the solution should handle large and large amount of data;</li></ul><p>Data is not a subject that can be processed aside, it’s <em>serious</em>, <em>bulky</em> and some time <em>tricky</em>. So, it really should be discussed with all the stakeholders and started as a real project (not a feature on one part of the Android new release).</p><h4>Product and Business data driven methods</h4><p>The second step is to understand who is going to consume the data generated by the platform users. From my experience, there are two approaches :</p><ul><li><strong>Product teams</strong> : I want to know <strong>how my user is consuming</strong> his product and why they stopped at step 2 and why another one completed the full path to the action; I want to know how I can <strong>engage more</strong> my user to use my product;</li><li><strong>Business teams</strong> : I want to know how to <strong>maximize gross market value</strong> and learn which commercial actions can be used on users to <strong>transform</strong> more efficiently;</li></ul><p>Those teams are basically trying to justify their existence with those metrics. Intakes are not necessarily compatible but the collected data should take care of both aspects. One way to consolidate both approaches is to design data generation from a <em>consumption path</em> which is very similar to a classic e-commerce funnel but applied on each feature of your product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*kjmjw2a-JGrkJYHsxHGJOA.png" /><figcaption>Consumption path for a feature</figcaption></figure><p>By this way, you can segment your consumption to identify causes of success (<em>nominal path</em>) and cause of give up (<em>abandon path</em>) and finally get the core product which reached both enthusiasm from users and satisfaction from customers.</p><h4>State of the art in Big Data</h4><p>There is a lot of literature on the subject but one author, <a href="http://nathanmarz.com/">Nathan Marz</a>, has reached a precursor rank with his book <a href="https://www.amazon.com/Big-Data-Principles-practices-scalable/dp/1617290343/ref=sr_1_1?ie=UTF8&amp;qid=1511360864&amp;sr=8-1&amp;keywords=Principles+and+Best+Practices+of+Scalable+Real-time+Data+Systems">Big Data : Principles and best practices of scalable real-time data systems</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*XLH0x6L6hxiYkXr5TlChvQ.jpeg" /></figure><p>Companies have evolved and big data automation has brought some comfort in data consumption. Data freshness is split in two categories, <em>hot</em> and <em>cold</em> data. <strong>Hot</strong> data is considered as it’s just out from the oven. <strong>Cold</strong> data is on the other hand something you put on the fridge for latter consumption. This distinction is important as data collection may not differ but data aggregation, enrichment and composition should.</p><p>Nathan Marz distinguish two <em>layers</em> :</p><ul><li>the <strong>slow</strong> layer which is the classical data management from past decades (Business intelligence, batches and so on);</li><li>the <strong>speed</strong> layer which is a stream approach of data management, allowing to access directly data during its ingestion (think of the real time component in Google Analytics);</li></ul><p>It also has an impact on the <strong>data core process</strong> of the company.</p><p>The<strong> slow layer</strong> is handling core business data, related with money, users, acquisition, all metrics a decent investor should look at before investing. This is real stuff, it has to be rigorous and unalterable. It’s <em>slow</em> to implement.</p><p>The<strong> speed layer</strong>, <em>a contrario</em>, is the perfect field to either build new metrics or observe specific behavior to determine a trend or confirm an intuition.</p><p>We then consider three stages for data :</p><ul><li><strong>data lake</strong> : raw (and <em>wild</em> ) data, you can fish there and get a wild fish;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Rbi8Nza__aQyjX2H6_n18w.jpeg" /></figure><ul><li><strong>data warehouse</strong> : the data has been industrially collected and conditioned but it is still from a low level of semantic, you can take a frozen fish from the freezer;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*cJLcAHleYFxgFHxJ-rO-lQ.jpeg" /></figure><ul><li><strong>data mart</strong> : the data is branded and accompanied with its story and extra content, you take a pack of <a href="https://www.nomadfoodseurope.com/brands/iglo">Captain Iglo’s</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aK8XWTHGUffpTdp6HcTRAA.jpeg" /></figure><p>Data lake is the <strong>reference</strong> and is collecting the pure data, without alteration. Other stages are totally expandable.</p><h4>Global platforms considerations</h4><p>Finally, architects readers will agree on the fact that global platforms are bringing new ranges of data volumes and freshness that makes data solutions a bit more complex.</p><p><strong>Data isolation matters</strong></p><p>On a technical point of view, we should considerate two kinds of technical data : the data that is <strong>useful for the product</strong> and without which the product could not work, for example <em>user evaluations</em>; and the data that is built or copied for <strong>analytics and statistical</strong> purposes.</p><p>The point is to separate durably those data so intakes from product and data consumption should not interfere. For example, a guy from the data management team that asks to change the product because the shown data in the product is not the one that should be used on the data side.</p><p>It’s why the data for analytics purposed should be <strong>copied and isolated</strong>.</p><p><strong>Shifting paradigms</strong></p><p>And last but not the least, data management has evolved from a <strong>discrete </strong>approach to a <strong>continuous </strong>approach. I apologize about the inaccuracy on the semantics to my dear mathematician readers, I lack a better term for this.</p><p>Formerly, engineers took snapshots of data in the data source (SQL dumps, static files, and so on) to consolidate a state of the data at one moment. This is quite efficient if 1/ you don’t have a big volume of data, 2/ you don’t have many sources, 3/ you don’t need to do complex requests on the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*bJ8OV0R_zW-dC2m8JLPQUw.png" /><figcaption>Discrete approach on data consolidation</figcaption></figure><p>However, if you consider the life cycle of the data, you will notice that those snapshots are not really representing the life of your data but only <strong>instant pictures</strong> of a given moment of this data. Your analysis will fill the gap between two snapshots. And if your snapshots are distant, you may loose crucial information about the trends.</p><p>It’s why full event driven model appeared so your data systems could rely on a more continuous approach. Because what is the most important is not to know the state of the data (ie metric = 42) but mostly its trend (metric is growing fast +25%). The event driven model gives you the <strong>granularity</strong> and the <strong>loose couple</strong> to reach a <strong>modular refine data</strong>. By this way, it is also well designed to fit the consumption path of Business and Product teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*CBQltLKXThW8Z03c7HdOMA.png" /><figcaption>Continuous (aka more discrete with higher granularity) approach for data collection</figcaption></figure><p>The more accurate and generic you can be in the design of your events, the more information you will get for later analysis.</p><p>This approach is perfectly compliant with the consumption path described above and gives more semantic weight to what to tag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*8JO3I_HVnMxge1A7zFjbbw.png" /><figcaption>On classic tagging, events are collected at the end and not along the user journey which results to fat event</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*nnkfpHrREtLXEhTS67sgcw.png" /><figcaption>On extreme tagging, events are sent along the user journey and you should recreate the previous fat event from collected data</figcaption></figure><p>In this section, I will present some solutions which are examples of common data architectures. The aim here is to present the choice of the solution regarding the required <strong>skills</strong> on IT and Business teams or the <strong>effort</strong> to give to implement it.</p><h4>Imponderables</h4><p>We can consider some aspects that are real constraints on the data collect :</p><ul><li>tag plans and event implementation are a <strong>real cost </strong>for the company, and moreover <strong>developers hate to do that</strong>;</li><li>it’s <strong>long </strong>to test and to prove that everything is correctly set up;</li><li><strong>client technology</strong>, specially on mobile device that are not always on wifi, may <strong>struggle data collect</strong>, so consider very early the use of a specific SDK or a <a href="https://www.google.com/analytics/tag-manager/">tag manager</a>.</li></ul><h4>Do it yourself</h4><p>Creating a small tagging system is easy, you need :</p><ul><li>A client library;</li><li>A endpoint to receive data;</li><li>A good storage solution that will be queryable;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*slAMCKTbusWsGULf1ptcQA.png" /><figcaption>Tag is simple — just need an API and a DB</figcaption></figure><p>Client library will be loaded and connected to what you want to listen to, for example the user is clicking on a button. When the user is doing that, you collect information about it (which user, what button) and build an event object.</p><p>The library will send events to the endpoint which will be as simple as a REST API endpoint and will store the data in a database or a nosql system.</p><p><strong>Pros</strong></p><ul><li>Easy to implement and to maintain;</li><li>It’s not expensive;</li><li>Does not require specific skill, a web developer can do that;</li></ul><p><strong>Cons</strong></p><ul><li>Not very suitable for high availability : when too much requests arrived on the endpoint, it will take time to write everything down and the client will have to wait the answers from this service;</li><li>Your storage system will quickly grow as the amount of events will arrive, you will have to <em>forget </em>some data if you want to maintain an effective system;</li><li>If you add some sources, you will have to code specific integration in your system and deal with several formats from the different providers, it also will contribute to make your storage grows up;</li><li>It is also perfect for a small team of business and product guys, more than two people and complex requests will bring this solution down;</li></ul><p>This is perfect for a startup that wants to start to collect data but do not have resources.</p><h4>Use SaaS solutions</h4><p>This solution relies on the use of SaaS solution that helps on several ways to handle data collecting and provides :</p><ul><li>SDKs to quickly implement their solution in your product;</li><li>Platform access and hosting so you can rely on efficient infrastructure;</li><li>Out of the box integration to plug in other systems</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*uk7u7Ra2HJu6EslJ4YAvLA.png" /></figure><p>Your client use the SDK to enter data inside an already designed event. The SDK sends seamlessly events to the platform. Platform is distributing data between sources and destination.</p><p><strong>Pros</strong></p><ul><li>Handles infrastructure complexity for you;</li><li>Save you time to do yourself the integration;</li><li>Ensure you to not loose data;</li><li>Somehow gives you an unified format for your events;</li><li>Still requires you a basic engineering skill set to implement it;</li><li>Helps you to test a lot of marketing / advertisement solutions based on data without really integrating them;</li></ul><p><strong>Cons</strong></p><ul><li>Integrations on those systems are sometimes not always complete, so you have to watch if your source / destination system is fully supported or compliant;</li><li>It’s difficult to test before sending in production;</li><li>It’s expensive and price is going to scale with your users or your separated accounts (for example an account = a country);</li><li>It does not fix the problem of the data expansion;</li></ul><p>This is perfect for a company that started to have a real data management and life cycle. Not handling data will lead the company to take bad decisions and this is critical. However, it’s complex and we don’t have time for this, so we need to be quick and have data collection done for yesterday.</p><h4>Integrate a cloud platform</h4><p>The last solution is to fully integrate a cloud platform, I give here an example of what could be done on AWS as it’s what I know the most but you can definitively find equivalent solution on other platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDioFJ3FMe0ssUZ-T4xEzA.png" /></figure><p>The <a href="http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html">API Gateway</a> is configured to receive specific events from your clients. <a href="http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html">Kinesis Firehose</a> is set up to split the stream of event in two : one for cold data (the slow layer) on <a href="http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html">S3</a>, and the other one in a relational database called Redshift. The transformation logic is done in <a href="http://docs.aws.amazon.com/lambda/latest/dg/welcome.html">Lambda </a>functions.</p><p><strong>Pros</strong></p><ul><li>Soft and slow collection layers are handled by API Gateway, Firehose and lambda which are serverless;</li><li>This solution is global;</li><li>S3 is the data lake so it is storing everything meanwhile Redshift is the data warehouse and it collects only the last <em>x</em> months (for example). This is solving the data expansion issue;</li><li>Completely elastic system that provides you fault tolerance, high availability, high performance, durability;</li><li>You mostly pay as you go for the computing part and you can handle both slow and speed layers on predicted budget;</li><li>Once the data collected inside your Redshift, you can dispatch it again to other external solutions, you are sure to get the freshest data;</li></ul><p><strong>Cons</strong></p><ul><li>You need Cloud skills and a strong data management team to design and implement data analysis in the data mart;</li></ul><p>This solution is for mature companies that are already mastering their data.</p><h3>Conclusions</h3><p>Data Management can be complex to handle if you do not focus on what you actually need. The main trap here is to <em>want every data now</em>. It leads to focus on data variety not considering if the solution is designed to take the load on higher and higher collected volumes. Start small, change often.</p><p>Intakes contradictions may be another problem as you will have to deal with business and production scopes wars. Data isolation and event approach should reconcile everybody around the table.</p><p>The data ingestion with the different stages of industrial refining is another serious concern as you will spend a lot of time doing and undoing things around the data. Data warehouse is the data foundation, you should, whatever the mean, start by this as all the other stages are <em>generated</em> by the collected data.</p><p>Considering free tooling, DIY, SaaS or cloud solutions, depending on your needs and maturity, you should consider two aspects : <em>security</em> and <em>availability</em>.</p><p>As your business team is connecting a lot of data marketing and analysis tools on top of your data warehouse, a lot of data starts to get out of the scope of the company. This is a major concern as you may hear all days <a href="http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/">data leak and breaches</a>.</p><p>The other point, availability, could also be a struggle for the company. Hosting a big data solution is a real job, not something that should be done on a table corner.</p><p>Cloud providers have understood this and provide more and more PaaS solution to ease this hosting, in your own infrastructure, guaranteeing by this way an optimal security. <strong>The sooner you can go on this, the better it is</strong>.</p><p>Feel free to comment and start the discussion about data management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gfnVvkMRFtjVsWF7vkClA.png" /></figure><h4>This story is published in <a href="https://medium.com/swlh">The Startup</a>, Medium’s largest entrepreneurship publication followed by 270,416+ people.</h4><h4>Subscribe to receive <a href="http://growthsupply.com/the-startup-newsletter/">our top stories here</a>.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gfnVvkMRFtjVsWF7vkClA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bad3f5f26b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-to-scale-data-management-5bad3f5f26b5">How to scale Data Management ?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure a microservices architecture]]></title>
            <link>https://medium.com/@maximethomas_46287/secure-a-microservices-architecture-9f148c4f3b5e?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f148c4f3b5e</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Fri, 27 Oct 2017 21:46:06 GMT</pubDate>
            <atom:updated>2017-10-27T21:46:06.615Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KO-TAWqHDKlzCWmeGVUs9Q.png" /><figcaption>The Walking Dead’ secured prison — zombie free</figcaption></figure><p><em>This article follows the talk I gave during the</em><a href="https://www.meetup.com/fr-FR/Microservices-Paris/events/244050659/"><em> Paris Microservices Meetup, the 24th of October at Deezer venue</em></a><em>.</em></p><p>Working on distributed architecture for such a while, I’ve noticed that companies, architects and developers too, are a bit confused when we talk about security and how it should be applied in the enterprise, specifically when the company is a small company and when the platform is relying on a microservices architecture.</p><p>Mainly, security is considered as unnecessary and an overkill, rightly because experience showed that security systems are very intrusive in people all days life and brings lot of complexity for everybody. But if you don’t have security systems, you are vulnerable.</p><p>Let’s dig in the way to secure specific distributed architectures.</p><p>Security main approach will be explained, so it will be easier to understand how, on the specific case of distributed architectures, components can be secured.</p><h3>Security concern</h3><p>Security is always a difficult subject to approach either by the lack of experience you may have on it ; either by the fact you should know when the level of security is right for what you have to secure.</p><p>Security is major concern for individual people and more and more for <a href="http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&amp;from=FR">companies</a>. As an architect, I consider that working with technical educated people (engineers, experts) and tools (systems, frameworks, IDE) should prevent the company to have issues like this.</p><p>However, when your company grows, it’s quite difficult to avoid, a certain infatuation from different categories of people to try to hack your platform. The fabulous website <a href="http://www.informationisbeautiful.net/">Information Is Beautiful</a> updates quite often its visualization of <a href="http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/">World’s Biggest Data Breach</a>.</p><p><a href="http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/">World&#39;s Biggest Data Breaches &amp; Hacks - Information is Beautiful</a></p><p>What is remarkable in this, is that you can see that great companies loose a large amount of data due too bad or weak security systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cO4QvqzEG6B7z2cHBIV28Q.png" /><figcaption>Information Is Beautiful — World’s Biggest Data Breaches — Number of records stolen</figcaption></figure><p>They also have a more interesting visualization that shows us the sensitivity of data. Ok, you have lost data, but if it’s interesting like banking account numbers of personal health data, it’s not valuable, you cannot sell it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfDSbo5QVZpG6JQXHmOV-A.png" /><figcaption>Information Is Beautiful — World’s Biggest Data Breaches — Data Sensitivity</figcaption></figure><p>As we can see, it’s uplifting — Swedish Transport Agency, French Mutuelle Generale de la Police, Philippines’ Commission on Elections… Pew.</p><p>Security must be taken seriously and in the same way we evaluate it for a car. The day I will send my car in a store front I will be happy to have chosen the car with the airbag included.</p><p>In IT systems, everything is virtual, so it’s not really a problem when you have security issues, it’s IT stuff. Actually no. Moreover, you cannot fix it by buying a single product that will magically seals all the lost and forgotten breaches in your platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*gOI-CFAuoGw8EeAE4rzf4Q.png" /><figcaption>Bruce Schneier — Security is not a product, it’s a process</figcaption></figure><p><a href="https://www.schneier.com/">Bruce Schneier</a> is so right. Imagine you have a bike and you buy a bike lock. It’s not because you have bought the lock that your bike is secured. You have to take it when you ride, you have to take the key, you have to put the lock on when you stop somewhere… You have to follow a process. For IT systems, it is the same.</p><h4>Security is a process</h4><p>In company, security is a main concern that impact either physical security (building, belongings, people) or virtual, such as IT.</p><p>If it’s a process, it means that is not an <strong>one shot thing</strong>, you have to set it up and <strong>maintain it</strong>. Generally, companies hire a Chief Information Security Officer — CISO — to take care of that. Maintenance can be done <em>a priori </em>by setting up standards for example, and <em>a posteriori </em>by auditing code and infrastructure (physical and virtual).</p><p>As distributed architecture are mostly based on <a href="https://en.wikipedia.org/wiki/Domain-driven_design">DDD</a>, each people is owner of domains, so <strong>everybody is concerned</strong> by security. For example, if you have a microservice called <em>Message</em>, the guy in charge of this microservice should be in charge of the security of its own microservice.</p><p>Finally, the modality of security should be <strong>discussed</strong> with all the stakeholders to find the point of overhead. It’s just because when you are talking about security, expectations may vary from the CEO and the developer.</p><h3>Pillars of IT Security</h3><p>To avoid misunderstanding about IT security is about, let’s get back to the fundamentals, its pillars.</p><ul><li><strong>Confidentiality</strong> : no one can access what belongs to you</li><li><strong>Integrity</strong> : information cannot be changed without your consent</li><li><strong>Availability</strong> : you can access data when you need it</li></ul><p>There is a direct correlation between confidentiality and integrity, and obviously, a direct opposition of availability between the others pillars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*L5cfRuXwh9_0ttzS3oZcfQ.png" /><figcaption>Impact of securitisation against accessibility level</figcaption></figure><p>As we can see, the security may affect the level of accessibility of the system in bad ways. It can kills the productivity of your platform, meaning no one will be able to use it; it will be complex and will generate bugs; or worse, it will bother people.</p><p>We all have in mind security systems such as <em>credit card number </em>or <em>3DS</em> that are very painful on the user side. If you are not following the nominal way of doing things, you will <em>loose</em> accessibility to your money.</p><p>In the other hand, if you reduce your requirements on security, you will maintain a high accessibility level but you will expose your company. From my window, I guess that it is what happened to all those companies that has been hacked, they reduce security to better match budgets and people availability. They pay it now.</p><h3>Monolith vs Microservices</h3><p>We’ve seen pillars of IT security and that the point of overhead is important in the solution selection. This part will describe the main differences between monolithic architectures and microservices architectures regarding security pillars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*TWoQgb8AXAbdPi-O6x40UA.png" /></figure><p>Monolithic architectures are set in one block (sic) and data is generally stored in one storage system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*hPuxpf2zD3CpsaqbhPPZnA.png" /></figure><p>If we break down pillars, well, we can see that the <em>monolith</em> has to handle each pillars.</p><p>Now, if we are with a distributed architecture, such as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*Yu3JcdMHQxsJR5VP7Tkdfg.png" /></figure><p>Where we can considerate 3 layers : <em>clients </em>(android, ios, web, …), <em>API Gateways</em> (or Backend For Frontend, BFF), and <em>microservices </em>and their data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*u0nHY6WHVyEOmQ-f__6QXA.png" /></figure><p>There’s a<strong> responsibility segregation</strong> through the different layers. Mechanically, exposed layer and front layers are taking care of <strong>confidentiality </strong>meanwhile microservices are handling <strong>integrity</strong>. <strong>Accessibility </strong>is also shared between all the layers as if they’re not accessible, the platform is not working.</p><p>Thus, the main difference here is that we split security requirements in the platform to ease accessibility and reach the best point of overhead. The main takeaway is here :</p><blockquote>Distributed software does not allow you to create security breaches</blockquote><p>So we simplify security requirements but in the other way, we have to be at the point of overhead for each components.</p><h4>Pragmatic Secured Microservices Architecture</h4><p>I purpose this pragmatic architecture as it may reach all the intakes of small companies, have the best of security without stopping product development or platform usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*WpA8SX67i1K0umRTArKFiQ.png" /></figure><p>Let’s define that inside our platform, this is a <strong>trusted zone</strong>, where exchanges can be done easily with few restrictions. Outside, however, it’s the <strong>jungle</strong>, we don’t know what can happen and we have to do more efforts on security and particularly on exchanges between clients and gateways.</p><h4>Security References</h4><p>Before digging in the set of rules for this architecture, we should consider security references as it is easy to find reliable sources of how to implement security in your system.</p><p>I guess <a href="https://www.owasp.org/index.php/Main_Page">OWASP</a> is the more known security reference as they collect a lot of best practices and help you to prioritize rules to implement. I particularly point on the so named <a href="https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series">Cheat Sheet</a> that will give you clues on how to hack your system :</p><ul><li><a href="https://www.owasp.org/index.php/REST_Security_Cheat_Sheet">REST</a> Cheat Sheet</li><li><a href="https://www.owasp.org/index.php/Projects/OWASP_Mobile_Security_Project_-_Mobile_Cheat_Sheets_Series">Mobile</a> Cheat Sheet</li><li><a href="https://www.owasp.org/index.php/Application_Security_Architecture_Cheat_Sheet">Network</a> Cheat Sheet</li></ul><p>Another good reference system is <a href="https://www.cvedetails.com/">CVE</a> that gives you a score — <a href="https://www.first.org/cvss/">CVSS</a>-referencing the severity of the breach for your company. It may help you to find the best implementation order to maximize the impact on security.</p><h3>Rules</h3><p>Now we have the main approach of security, we can dig in detail each rules that will allow us to reach the point of overhead of our security system on our distributed platform.</p><h4>Global infrastructure</h4><p><strong>Certificate rotation</strong></p><p>Client requests a server. Server is providing official information and specifically a secured certificate proving that client asks to the good server and not a fake one. Rotating the certificate will reduce the opportunity for a hacker to access a copy of this certificate. If it is updated regularly, the probability that the hacker gets an outdated certificate is stronger.</p><p><em>Solutions</em> : <a href="https://letsencrypt.org/">Let’s encrypt</a>, <a href="https://aws.amazon.com/en/certificate-manager/">AWS ACM</a></p><p><strong>SSL/TLS Termination</strong></p><p>When a request arrives on the server, it’s encrypted through a secured protocol. API Gateway is in charge to decipher a request and transmit it to microservices on a deciphered way. We do that because it’s a pain for developers to set up SSL in their application servers and it’s a pain for ops to maintain certificate rotation on all the components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*odJnRTdbJ_XVp1-HP060pA.png" /><figcaption>Global infrastructure rules</figcaption></figure><p><strong>Isolated Subnets Layers</strong></p><p>Global network is locked, there is only one entrance. It should be the same inside. You should apply bulking, compartmentalization of your layers, to avoid the opportunity for a hacker to progress from layers to layers.</p><p>The more sensitive the component is, the deeper it should be in the infrastructure.</p><p><strong>Isolated Inbound and Outbound Rules</strong></p><p>Network rules should define exactly what can be received and what can be let out. This is the next granularity level from the previous rules. If you master that, you will then be able to define specific exceptions on it. For debug purpose, it should be handy.</p><h4>User sessions</h4><p>The most sensitive data in our platform is the user session, we should then handle it with secured rules.</p><p><strong>Outside session</strong></p><p>We encrypt an authentication token that should be ideally a totally random and reproducible token. It can be as simple as a UUID. We do that for two reasons :</p><ul><li><strong>we don’t care </strong>about maintaining a specific coherency outside our platform, client is logged in with direct user input so it gets back a token and then use it;</li><li><strong>random </strong>is the new defense against hacking, we don’t give information about how we handle user information inside our platform;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*4Te_jm2zCIQtSRlWXroY7g.png" /><figcaption>User session rules</figcaption></figure><p><strong>Inside session</strong></p><p>The user data that is useful must be in the global context, ie in each request header. The <strong>API Gateway</strong> is in charge to do the mapping between inside data and outside token. The logic of handling data must be <strong>externalized </strong>in a specific <em>account</em> microservice that will store this user data (not in the api gateway).</p><p>We do like this because :</p><ul><li><strong>user data is centralized</strong>, it can be also be optimized for performance ;</li><li><strong>other microservices </strong>receive the user data that is in context as they don’t own this data and they <strong>don’t have to request</strong> it each time;</li></ul><h4>Microservices Data Isolation</h4><p>The data belongs to the microservice so it should not let other microservices read or write inside the data system storage <strong>directly</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*Vi_orkJW2yQEJV2T3OgK9A.png" /><figcaption>Microservices Data Isolation</figcaption></figure><p>The microservices must access to it via an <strong>API call or asynchronously</strong>.</p><p>And obviously, set up <strong>different users / password</strong> to your data storage systems.</p><h4>Microservice ownership check</h4><p>This one is about applying confidentiality : when the microservice is receiving a user id, it has to <strong>verify</strong> that the data which is sent effectively belongs to the right user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*Z2vCGRozmU-U2JfFnw_Z4w.png" /><figcaption>Microservice ownership rules</figcaption></figure><p>This mechanism may be extended to <strong>roles</strong> if you have any.</p><h4>Correlation Id</h4><p>Your API Gateway / BFF must transmit a <strong>correlation ID</strong> to all the requests, which is set by the clients. UUID v4 is enough. It should be transmitted in the headers of your request as it’s a meta information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*IroJW-7ZsogzwSz3epNxFQ.png" /><figcaption>Correlation ID transmission rule</figcaption></figure><p>We do that so then we will able to track precisely the origin of a request and the consequences of a request in your platform. It gives you the knowledge of what is happening in your platform.</p><h4>Microservices — Diagnose and recover</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*ABSpGXkjM8i9uzjnCzanlw.png" /><figcaption>Logs and exceptions managers</figcaption></figure><p>Once you have a correlation id, you should also extend your system to quickly diagnose issues and recover from those.</p><ul><li><strong>Tag </strong>everything in your infrastructure, following a naming convention to identify every elements in it;</li><li><strong>Log </strong>all the request to a central log system that can be either in your infrastructure or as a third party service, omit headers and post information;</li><li>Use an<strong> exceptions</strong> platform to handle exceptions and get technical context on it, ie with what you did not log;</li></ul><p>We do that to split information for tracking (log) from technical incidents (exceptions). Logging trace is a static way to collect information meanwhile exceptions are a dynamic way to do it. Some exception managers also handle the resolution state and can tell you when you have close or open again an exception.</p><h4>API Gateway Detect traffic patterns</h4><p>Now that the platform is collecting a lot of tracking information, we are going to analyze it in order to detect traffic patterns. We are following here a statistical approach against the collected log.</p><p><strong>Automated traffic</strong></p><p>Half of the internet traffic is due to automated scripts, such as robots, spiders and crawlers, but also to marketing companies that scan your website for example in order to provide automated marketing analysis to its customers.</p><p>Legitimate robots are following the <a href="https://en.wikipedia.org/wiki/Robots_exclusion_standard"><strong>robots.txt</strong></a> and will follow your directives about what they are allowed to scan or not.</p><p>You should <strong>exclude </strong>those logs from your statistics as it is <strong>noise</strong>.</p><p><strong>Real traffic</strong></p><p>You may also want to <strong>exclude </strong>real traffic of your statistics as a user should have more or less the same profile and the same statistical behavior. For example, you can consider that a user generates between 2 or 3 hits by min on our api gateway, with a range of maximum half an hour.</p><p><strong>Accumulated traffic</strong></p><p>Last, but no the least, you should identify clients that generates a large amount of requests on different frames. I set here a common example that should match for a lot situations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*ob9u2j_DY6Z9h-vkupfYlw.png" /><figcaption>Accumulated traffic for edge traffic usage</figcaption></figure><p>I define here three categories of users that are spotted statically because the frequency of requests or the length of usage is <strong>abnormal</strong>.</p><h4>Prevent and block suspicious behaviour</h4><p>Now you have spotted people or companies, you can speculate about their intention :</p><ul><li><strong>DDoS</strong> attacks → Put your platform down</li><li>Attacks through <a href="https://www.techopedia.com/definition/30958/dynamic-application-security-testing-dast"><strong>DAST</strong></a><strong> </strong>→ Find backdoors to hack you</li><li><strong>Brute force login</strong> → Scammers</li><li><strong>Enumeration</strong> of urls → Steal data through scraping</li></ul><p>In order to block those people, you may use a Web Application Firewall (<a href="http://searchsecurity.techtarget.com/definition/Web-application-firewall-WAF">WAF</a>) that include specific rules, either on rate limiting, geographic belonging or IP origin (useful for <a href="https://www.torproject.org/">TOR </a>endpoints). This should be set at the entrance of your platform, prior to your API Gateway. My opinion is that a WAF is pretty dumb and intelligence can be difficult to give it.</p><p>You may also complete this with a third party solution giving you complementary advice, in real time, about the request you received. You should integrate this inside each of your api gateways. If interested, you should look at <a href="https://www.sqreen.io/">Sqreen</a> or <a href="https://www.distilnetworks.com/">Distill Network</a>.</p><h4>Mobile</h4><p>Finally, advice for mobile security, a bit at the edge of the subject of this article but that is worth it.</p><ul><li>Master all the <strong>interfaces</strong> of your app : in its <strong>environment</strong> : file system, identity system, http layer, … and what you <strong>expose</strong> : share / publication system;</li><li>Be careful of <strong>what is left</strong> on the device after uninstall;</li><li><strong>Certificate pinning</strong> : a user can usurp client identity through self signed certificate;</li></ul><h3>Conclusion</h3><p>We have seen in overview and in detail that security is important and should be not neglected by companies or developers. Security is a process that has to be <strong>integrated </strong>in the main <strong>development process</strong> and<strong> platform life cycle</strong>.</p><p>Microservices are leveraging complexity in code by adding complexity in interaction. My approach is to leverage extra security layers that are not needed inside the platform. Security has also to be tempered to reach <strong>the overhead point </strong>and applied <strong>only when needed</strong>.</p><p>Don’t hesitate to react and share, I will be happy to discuss alongside this subject.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f148c4f3b5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices Factory with Amazon ECS]]></title>
            <link>https://medium.com/@maximethomas_46287/microservices-factory-with-amazon-ecs-5131f56efd75?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5131f56efd75</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Tue, 17 Oct 2017 09:27:05 GMT</pubDate>
            <atom:updated>2017-10-17T09:29:26.500Z</atom:updated>
            <content:encoded><![CDATA[<p>This article highlights the main principle of producing software for distributed architecture such as microservices pattern in the AWS Cloud platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DwBXcvAFAp5uxcwLi9F9Vg.jpeg" /><figcaption>Pink Floyd’s Animals cover</figcaption></figure><h3>CI ? CD ?</h3><p>CI is <em>continuous integration</em>, also known as the process that checks you did not messed up with your code and that everything is alright. Nowadays, every continuous integration systems are based on flows with steps, generally triggered by a change of code.</p><p>The historical system is <a href="http://hudson-ci.org/">Hudson </a>that has been given by Oracle to the Eclipse foundation. Then rebooted in a more known <a href="https://jenkins.io/">Jenkins </a>systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*gCOywkxXcJHlr3TawlyAew.png" /><figcaption>Hudson interface</figcaption></figure><p>In the cloud era, lot of systems appeared to wrap this system in an online platform, in a dedicated solution such as <a href="https://circleci.com/">CircleCI</a>, a complement of the version control system such as <a href="https://about.gitlab.com/features/gitlab-ci-cd/">Gitlab CI</a> or finally part of the cloud such as <a href="https://console.bluemix.net/devops/getting-started">IBM Bluemix DevOps</a>. This is now one of the main pillar of the DevOps manifesto.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzIKSeQMoFCXQ-q1bxzsgA.png" /><figcaption>DevOps loop</figcaption></figure><p>Then, you basically program a process to test your code systematically if it changes. It is the developer <em>Casting out nines</em> way to ensure the robustness of its code, accessing then to higher levels of software quality. It also saves a lot of time to developers as tests are centralized and run automatically.</p><p>The last step for the software in its <a href="https://www.youtube.com/watch?v=-1YjmXSyHa8">Fantastic Voyage</a> is to be automatically deployed on a targeted environment, what is called <em>continuous delivery</em>. In tests we trust, so let’s install your app in the development environment. The main gain here is also automation that leads to time savings.</p><p>For those who have installed themselves a CI / CD system, it’s quite complex and harsh for what it gives you back. From my window, I spotted the following issues :</p><ul><li>install and maintain tests environments;</li><li>setup new CI/CD flows or environments;</li><li>tight coupling on technology (system that tests must know the technology used).</li></ul><p>And then containerization solutions reached a stable state.</p><h3>Docker</h3><p>It actually started before Docker with the first virtual machines (VM), from <a href="https://www.vmware.com/">VMWare</a>, promising a total isolation between the host and the VM. It also procured a snapshot system to save an already built version of your machine. The key information here is <em>snapshot</em> as it means that we can reproduce, copy and share already prepared images of a specific setup. It leveraged the loose coupling issue of this kind of systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*fKfDkwInPjW5VD4dLD-A-Q.jpeg" /><figcaption>A Windows OS running in another Windows OS</figcaption></figure><p>Alas, virtualization paradigm went wrong for VMWare products. The isolation between the host and the guest was too complete and affected performances of the whole system. Other hypervisors, such as <a href="https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine">KVM</a>, had a different approach and were isolated on a higher level <strong>*and*</strong> only on a Linux base which simplified a lot the integration layers.</p><p>During the last decade, <a href="https://en.wikipedia.org/wiki/Aufs">AUFS</a> appeared, an innovative way to handle “multi layered file system”. Eh ? Don’t worry, it’s magic. It allows you to modify a file system and save the state of this file system as an image (similar to a vm snapshot). On the design side, it may be compared to a file system versioning tool, allowing you to build incrementally a file system, starting by the OS, installing software, setting up things, doing stuff.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*GIJ4Dw7IyT077LbTCZlX1g.jpeg" /><figcaption>AUFS mechanism, courtesy of <a href="https://www.slideshare.net/febain/develop-qnap-nas-app-by-docker">Terry Chen/QNAP</a></figcaption></figure><p>In 2014, <a href="https://en.wikipedia.org/wiki/Docker_(software)">Docker</a> was launching its engine, based on AUFS, with a loose coupling between host and guest. They actually succeeded to bring VM principle to a second life, avoiding performances issues.</p><p>They even did more.</p><p>They transcended the way to develop, giving the power to the developper to quickly and easily have a production-like environment on their machine. More, they give us the way to <em>integrate</em> the environment, linking containers based on images between them. This is a real revolution. This is why this project / product totally exploded as there was an infatuation for distributed architectures.</p><p>Their technology platform is composed of several components :</p><ul><li>the <a href="https://www.docker.com/"><strong>Docker Engine</strong></a> to build containers;</li><li>the <a href="https://docs.docker.com/compose/"><strong>Docker Compose</strong></a> to setup architectures and run it;</li><li>the <a href="https://hub.docker.com/"><strong>Docker Hub</strong></a> to give already built images to the world;</li></ul><h3>Amazon ECS</h3><p>Naturally, all the big cloud actors have seen the docker revolution and quickly started to think of how to host this technology and AWS released <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html">ECS</a> platform during 2016 summer as the <a href="https://kubernetes.io/">Kubernetes</a> software was uprising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*c_X9xKSIDx656waxhuH-_Q.png" /><figcaption>ECS Architecture</figcaption></figure><p>ECS gives similar concepts to Docker :</p><ul><li><a href="https://aws.amazon.com/ecr/"><strong>ECR</strong></a><strong> :</strong> equivalent of Docker Hub to store privately your docker images;</li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html"><strong>Task definitions</strong></a> : equivalent of Docker Compose file, an entity that explain how to run the container;</li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html"><strong>Tasks</strong></a><strong> </strong>: equivalent of container that is run;</li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html"><strong>Services</strong></a><strong> </strong>: a service is the entity gathering all containers and applying resources strategies on it;</li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_clusters.html"><strong>Clusters</strong> </a>: EC2 instances that are used to run instances and software layer to orchestrate services;</li></ul><p>Deploying on ECS is basically summarized in 3 steps :</p><ul><li>Build locally your image and send it in the <em>ECR</em>;</li><li>Create a <em>task definition</em> referencing this new image, giving you a new revision of this task definition;</li><li>Updating the <em>service</em> to point on the new revision of your service;</li></ul><h3>Platform Architecture</h3><p>ECS has been designed to handle a lot of architecture types, from monolithic applications that just need cost efficient hosting to full distributed platform needing drastic elasticity.</p><h4>Orchestration</h4><p>This is made automatically by the service depending on the <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html">task definition</a> which has several important parameters and among those, I precized the following :</p><ul><li><strong>memory</strong> : amount of maximum memory to set for this container, if you need to ensure a minimum, you must reserve it;</li><li><strong>cpu</strong> : the same but for CPU.</li><li><strong>essential</strong> : if you set more than one container in the task definition, you may want to specify one container as the <em>master. </em>If this one is killed, all others will be to.</li><li><strong>environment</strong> : environment variable to set for this container, mostly integration parameters, in order to keep the <em>immuable</em> way of the docker container intact.</li></ul><h4>Load balancing</h4><p>The load balancing should be set up for publicly available containers, such as api gateways. This is possible by setting up things like this :</p><ul><li>A <a href="http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html">Target Group</a> which will dynamically update the instances of your clusters if they change;</li><li>An <a href="http://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html">Application Load Balancer</a> (ALB) which will point a certain uri path to the correct target group;</li><li>A ECS Service that will take care of linking tasks (containers instances) to instances of the cluster; and cluster instances to target group; and finally maintaining the port number and container availability on each nodes.</li></ul><p>This is a great platform, I mean, it really avoids people to have nightmares about maintaining all those parts together which can be very painful. The service is the glue between all the components and will take care to apply a resource strategy. We are talking about that below.</p><h4>Front facing</h4><p>Finally the front facing is the most powerful feature of the platform, as it’s allowing you to handle distributed software hosting in one and hand; and monolithic exposition API in the other hand.</p><p>When you type your public url, Route53 is resolving your DNS and pointing to your ALB, the ALB analyzes the path in the url and send it to one of the instances of your target group.</p><p>For example, you may have the following urls :</p><ul><li><a href="https://mysite.mybrand.com/service-a/my-function">https://mysite.mybrand.com/service-a/my-function</a></li><li><a href="https://mysite.mybrand.com/service-b/my-other-function">https://mysite.mybrand.com/service-b/my-other-function</a></li></ul><p>For both, Route53 points to the same ALB as it’s sub zone resolving.</p><p>For <em>Service A</em>, ALB is configured to point to a dedicated target group, and for <em>B</em> to another target group.</p><h3>Software flows</h3><p>So that’s nice, we can deliver pieces of a software independently but it will behave like a whole thing observed from outside.</p><p>The next point to set up is, before sending everything in your cluster, how you may organize the life cycle of your software. Generally, you will code until you have something stable in your local environment and then you will commit those code modifications on the origin / remote branch of your code versioning system.</p><p>As far as I have seen, <a href="https://git-scm.com/">Git</a> is very permissive about its usage and Git authors explain at the very beginning of their documentation that the software flow is not about tooling but <strong>development team organization</strong>.</p><p>This leads me to detailed the two software flows that are currently mostly used in companies.</p><h4>GitFlow</h4><p>This flow is totally linked to classic environments and way to develop software :</p><p><strong>Named branches</strong></p><ul><li><strong>develop </strong>is the branch of early integration, just putting things together without consideration of end to end QA</li><li><strong>master</strong> is the branch of delivery, having only few commits that are representing deliveries on the system. Tag is useful here to give technical information on when this appears.</li></ul><p><strong>Prefixed branches</strong></p><p>Those branches are just prefixed by a chosen significant word that may trigger a specific action. Those are totally droppable, ie their existence is just for WIP purpose.</p><ul><li><strong>features/*</strong> are branches that are forking <em>develop</em> and used to develop the so named feature. Example : <em>features/new_home</em></li><li><strong>releases/*</strong> are branches that are built for release candidates, processing delivery on the <em>staging </em>environment. Example : <em>releases/1.0.24</em></li><li><strong>hotfixes/*</strong> are branches that are normally forked from <em>master</em> in order to fix a direct problem on production. Example : <em>hotfixes/payment_issue</em>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*lv4ET8_gSH5FTHruIAJvqg.png" /></figure><p>I won’t explain it in details as it has already been done on this website : <a href="https://danielkummer.github.io/git-flow-cheatsheet/index.html">https://danielkummer.github.io/git-flow-cheatsheet/index.html</a></p><h4>GithubFlow</h4><p>The GitHubFlow is simpler as it starts with a simple finding : people using GitFlow <strong>mix</strong> <em>project management</em> and <em>software edition management</em>.</p><p>In other words, the moment you want to deliver is not necessarily the moment the software is stable or complete.</p><p>So GitHub purpose is to simplify GitFlow by squashing named branches and letting the developer choose their destination. This is done by tagging their commit with the name of the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Bgb7ErelV4SjtSgz_QFevQ.jpeg" /></figure><p>Everything is explained here : <a href="https://guides.github.com/introduction/flow/">https://guides.github.com/introduction/flow/</a></p><h4>What to choose ?</h4><p>GitFlow has been the first flow to appear so it is well spread in developer communities. Nevertheless, this flows suffers from some design issues that can annoy people using it :</p><ul><li>It implies to have <em>merge commits </em>each time you want to promote your code;</li><li>You are limited to destination environments if you plug CI/CD on it;</li><li>If you strictly respect this or put rules on git to ensure GitFlow is applied, your time to market will be very long again.</li></ul><p>The <em>GitHubFlow </em>coupled with <em>IaaC </em>and <em>automation </em>may lead to something more interesting : developers can trigger a brand new platform by creating a new tag. For example, I tag my commit <em>feature/payment</em> and once CI has been passed, CD uses an IaaC template to set up a droppable environment and install my commit in this new environment. So I have an ad hoc environment with only my changes so it can be reviewed by product people or we can put some performance tests over it.</p><h3>Microservices</h3><p>So far, we have : a target architecture on AWS using the container solution Docker and a software life cycle including a CI/CD solution. Some other consideration to have in mind if you plan to develop microservices architecture on this solution :</p><h4>On CI/CD</h4><p>The first part is to have an isolated workflow for each microservice. It means that people who may handle it that won’t be the OPS. This is the heart of DevOps, Dev must take a part of the work of the Ops and they have to master the whole cycle of the software. No more responsibility split between Dev and Ops.</p><p>The second part, which is inherent to the CI solution, is to be more accurate on what to test on each microservice. As the software is distributed, you will have the opportunity to split all the tests and refined factory test on each part. We have simplified this to :</p><ul><li><em>Back </em>: smoke tests on a container build;</li><li><em>Front </em>: functional tests, End to End tests.</li></ul><h4>On ECS</h4><p>ECS is a great tool if you master it. However, it implies you make some strategic decisions while designing your cluster.</p><p>The deployment strategy can be set up in the service withe the parameters <em>healthy tasks percentage </em>and <em>maximum healthy percentage</em>. If you put 50% and 200% and you have 4 containers, the delivery system will start 1 new task on the cluster, will register it and then will de-register older tasks. This is rolling updates <em>only</em>.</p><p>The cluster has two levels which allows you to have different levels of scaling. You may scale on :</p><ul><li>EC2 instances of your cluster, number of machines;</li><li>Tasks of you service, number of containers launched on your cluster.</li></ul><p>AFAIK, EC2 instance scaling parameter is a static parameter defined at the creation of the cluster. If changed in the CloudFormation template at the scaling up, a new instance will appear.</p><p>Scaling can also be a bit trickier as your EC2 instances host now a variety of services and not only one. So basic triggers such as CPU or RAM won’t work for services. You will have so specify a new metric that will represent the actual stress of your service.</p><p>You may also want to have dedicated instances for one of your service which needs lot of memory or CPU. You can do it by setting up a launch configuration that will handle the type of instance according to your service. The best is to create a new cluster with a different typology of instance instead as the usage may be different (worker vs microservice for example).</p><p>Finally, ECS gives you ways to define the placement strategy of your tasks which are :</p><ul><li><strong>binpack </strong>: choose instance depending on their CPU or RAM level;</li><li><strong>random </strong>: random placement;</li><li><strong>spread </strong>: apply uniform placement on instances based on a specific parameters.</li></ul><p>And you can chain them so you can have an elaborated placement strategy for your platform.</p><h3>Conclusion</h3><p>You can quickly set up a software life cycle using Git, CI/CD solution and a cloud platform such as AWS. On an architecture point of view, this solution is proving the following :</p><ul><li>Containerization leverages the coupling on the cloud provider.</li><li>IaaC leverages the coupling on pre defined environment.</li><li>Regular SaaS solutions for standard computing will disappear as they will be not flexible for .</li></ul><p>It still remains questions or solution to tests :</p><ul><li>Should we, like <a href="https://eng.uber.com/dockerizing-mysql/">Uber</a>, containerize our data services (MySQL/Elasticsearch, Redis) our still use managed services ?</li><li>Should we define the whole microservice stack in task definition ?</li></ul><p>Feel free to comment and react.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5131f56efd75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Complexity kills]]></title>
            <link>https://medium.com/@maximethomas_46287/complexity-kills-8a3b1702a724?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a3b1702a724</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[complexity]]></category>
            <category><![CDATA[organization]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Sat, 02 Sep 2017 17:38:11 GMT</pubDate>
            <atom:updated>2017-09-02T17:38:11.229Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLcpZ74uvexpif6twRtzrw.jpeg" /><figcaption>Frank Gehry’s Lou Ruvo Center, Las Vegas, Nevada</figcaption></figure><p><a href="https://stootie.com/">Stootie</a> is the first application for peer-to-peer services in France. A simple, fast and inexpensive way to find a helping hand to repair a leak or move a sofa. We design and produce a technology platform for the general public in the form of applications (iOS, Android and Web) and partner APIs.</p><p>In recent months we have seen a radical change in our product: we have simplified it. Product simplification is needed in the evolution of a company’s products, as announced in <a href="https://www.amazon.com/Crossing-Chasm-Marketing-Disruptive-Mainstream/dp/0060517123">Cross the Chasm </a>— Geoffrey A. Moore. We had experienced in recent years many product and marketing precepts, losing as read and in use.</p><p>To serve our ambitions, it was necessary to realize this pause in the race to feature and simplify our product. This complexity prevents the execution of ideas. But we have domesticated it and here is how.</p><h3>Complexity kills IT projects</h3><p>In a pre-digital world not so far away, companies were organized according to established physical flows, which sometimes had to be simplified because they were too complex or intertwined.</p><p>The advent of computers in the workplace, and then of computers in personal life, the famous <em>digitization</em>, led to the birth of technological platforms, which have recovered for them the complexity of these flows.</p><p>The projects in the big companies were followed with more or less success, trying to make enter into technical concepts a reality business. Nevertheless, it must be admitted, the technical teams have developed the functionalities without understanding the business flows. How many of us have had to endure badly damaged software, wobbly ergonomics, porous fundamentals? These famous 9 clicks through the tool to do an action that takes only one on an online SaaS platform?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*EmwQrKH-qeSQhIvs_s5OAQ.jpeg" /><figcaption>Carelman’s Unfoundable ojects</figcaption></figure><p>After many years of hard work and errors, people who finally understand the user have arrived in companies, the <strong>UX designers (User eXperience)</strong>.</p><p>They have in turn taken over the complexity of the business in order to build effective human-machine interfaces. They intervene on the complexity, both on the bottom and on the form by reducing the number of brakes to the use and generally require a reactivity without equal to the platform.</p><p>This makes sense, for example, as part of a form with many fields and convoluted management rules. An engineer will explain that it is impossible to manage this technically. They will both be right. An UX will say that the user is drowned in complexity.</p><p>The engineer explains in more detail that we oppose a monolithic model to a distributed model. The monolithic model maintains and holds a generally very broad global context. The distributed model, on the other hand, holds a set of pieces that, when combined, can reconstitute a global context. The elements of the distributed model are smaller, simpler, and in fact easier to develop, maintain and evolve. For once, we can guarantee that the sum of the parts is equal to the whole.</p><p>The UX does not think less of this famous global context and even involves the triggers that lead the user to perform actions. Once again, the all-in model no longer works: users disengage applications or systems that require a lot at once. For example, some applications had their acquisition collapse as soon as they asked the registration to access all the required permissions to the user.</p><p>The UX dismantles these screens, legitimizes the <strong>triggers</strong> and facilitates the life of the engineer who finds himself again in a position of force on his code.</p><h3>Complexity is needed</h3><p>In our pre-digital world, in 1984, the Bordeaux researchers developed the <a href="https://en.wikipedia.org/wiki/GRAI_method">GRAI</a> method, which makes it possible to streamline the flows of a company by projecting them along functional, hierarchical and systemic lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*sutcjqp4BgNOcrtLjPCx3w.png" /><figcaption>GRAI matrix after analysis</figcaption></figure><p>This method has brought out another rule of thumb that we pursue at <a href="https://stootie.com/">Stootie</a>:</p><blockquote>Does the complexity have to be on the platform or in the organization?</blockquote><p>Several examples concerning the responsibility of the platform:<br>- Sending notifications: <em>platform</em><br>- Litigation: <em>organization</em><br>- Moderation: <em>mixed</em> because there is a complex part that can be automated but the decision always comes back to the human being in the organization</p><p>This stems from the <em>Product over Organization</em> paradigm induced by Conway’s law. We have to understand this in the sense that the service we provide is both a technological platform and a set of human operators that manipulate it.</p><p>When I arrived at Stootie, the administration interfaces were not considered a functional element of the product and was always weakly prioritized vis-à-vis user features. Since then, we have understood that the <a href="https://www.definitions-marketing.com/definition/symetrie-des-attentions/">symmetry of attentions</a> is paramount. Our platform operators need interfaces as fluid and relevant as those presented to our users.</p><h3>Conclusion</h3><p>At Stootie, it took about 6 months to reorganize, flatten the product, test it and finish what was not finished. We ended up crossing the gulf.</p><p>If you want to embark on this simplification, some tips:<br>- Hire a UX Designer that will purify the use of the product,<br>- Review your organization to reflect your product,<br>- Do not ignore the symmetry of attention and give your team the means to succeed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a3b1702a724" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Now or as soon as possible]]></title>
            <link>https://medium.com/@maximethomas_46287/now-or-as-soon-as-possible-9d036734e105?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d036734e105</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Sat, 02 Sep 2017 17:28:23 GMT</pubDate>
            <atom:updated>2017-09-05T06:13:55.650Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://stootie.com/">Stootie</a> is the first application for peer-to-peer services in France. A simple, fast and inexpensive way to find a helping hand to repair a leak or move a sofa. We design and produce a technology platform for the general public in the form of applications (iOS, Android and Web) and partner APIs.</p><p>For more than a year, our technical team has carried out a change of architectural model, moving from a monolithic model to a more widely distributed model. Small focus on prioritizing interactions with the platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XV8nzxxQUWLMPZwG0skmdA.jpeg" /></figure><p>Sam Newman’s excellent <a href="https://www.amazon.com/Building-Microservices-Sam-Newman/dp/1491950358">book</a> on microservices discusses the form of managing technical requests: <em>synchronous</em> or <em>asynchronous</em>. And if we eliminate technical jargon: <strong>now</strong> or <strong>when I have time</strong>.</p><p>When you’re a user, you click or type and expect a system reaction, because your time is precious. These technical requests are special, they must be dealt with immediately, especially when making a payment or if you want to send a mission to the collaborative community. In the event of failure, this generates frustration and dissatisfaction among users.</p><p>There are other interactions with the platform that are generated by the same platform. Through <strong>business rules</strong>, the platform performs periodically a set of visible or invisible actions, direct or indirect. For example, you want to retrieve users who have not finalized their shopping cart, update a user’s network newsfeed, or send the summary of the week regarding user account performance. All of these actions point to users who do not specifically expect to be contacted by the platform.</p><p>Our migration to a microservice architecture forced the technical team to reconsider what should have priority over what could wait.</p><p>We have chosen a rule of thumb:</p><blockquote>Does a user expect a result from the platform?</blockquote><p>Some examples of answers to this question:<br>- I send an image on the platform to be able to save my request on the platform. <em>Right now</em><br>- I am a subscriber to a particular topic and I want to receive notifications when there are new features. <em>When I have the time</em><br>- I chat with another user and I see that he is writing to me. <em>When I have the time</em> (it’s a trap, I stay on the chat but I do not know when the person on the other side will have finished writing)</p><p>This helped us to identify the modes of communication of our microservices and to clear the ground for the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d036734e105" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production at Stootie]]></title>
            <link>https://medium.com/@maximethomas_46287/production-at-stootie-72ba4407c8e7?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ba4407c8e7</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[feature-team]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Wed, 30 Aug 2017 16:01:34 GMT</pubDate>
            <atom:updated>2017-08-30T16:01:34.223Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvvZDB7oNWxFXMd2mZ0YHA.jpeg" /></figure><p><a href="http://stootie.com/">Stootie</a> is the first application for peer-to-peer services in France. A simple, fast and cheap solution to find a hand to fix a leak or move a sofa. We design and produce a technology platform for the general public in the form of applications (iOS, Android and Web) and private and partner APIs.</p><p>When I arrived at Stootie, I conducted an audit of this platform and confronted it to the ambitions of our CEO, <a href="http://fr.linkedin.com/in/arnal/fr">Jean-Jacques Arnal</a>.</p><p>My vision is that a company spearheading the field of collaborative economics, a field in full revolution, must be innovative in its technical field but also in its organization. We need to work together as much as possible.</p><p>My first recommendation was to make a technical pivot and to migrate the architecture of the platform towards a microservices architecture. The second recommendation was to accompany this technical pivot by an organizational pivot.</p><h3>Law of Conway</h3><p>First of all, it is appropriate to speak of <a href="http://en.wikipedia.org/wiki/Melvin_Conway">Melvin Conway</a>, eminent computer scientist who established the eponymous law:</p><blockquote>Companies are organized in the same way as the software they produce</blockquote><p>In Conway’s idea, this went quite far and we could determine by reading the code of an application, if the developer was a freelance, or if he was isolated, in the middle of chapel wars, technical or product oriented, and so on.</p><p>It also meant that if the software was buggy or malfunctioning, it could be a reflection of a bad organization.</p><h3>Toward the feature teams</h3><p>We relied on the contraposition of this law in the framework of its technical pivot:</p><blockquote>In any case, we have to produce microservices, then as much organize accordingly, in micro teams</blockquote><p>This seems trivial, but the classical paradigm of the production of a digital society is broken. We moved from:</p><blockquote>Organization &gt; Perimeter</blockquote><p>To :</p><blockquote>Perimeter &gt; Organization</blockquote><p>And it seems logical, how to define how to be organized if we do not know what we have to produce?</p><p>In the traditional <a href="http://en.wikipedia.org/wiki/Project_management">delivery management</a>, the company chooses to lead its project by the dates, or by the costs. Conway’s law requires leading it by the features, and by the way to get things out when they’re ready. Some companies, such as <a href="http://gitlab.com/">Gitlab</a>, say they have frozen their delivery schedule for the year and what does not come out on schedule, will come out in the next delivery time slot.</p><p>This is quite aligned with <a href="http://en.wikipedia.org/wiki/Agile_software_development">Agile methodologies</a> and more over with the <a href="http://en.wikipedia.org/wiki/DevOps">DevOps philosophy</a>.</p><p>We are not talking about organizing an IT department or small isolated team in a company. <em>It’s a global pivot !</em></p><p>Spotify had already understood this in 2014 when publishing its videos on the culture of engineering (<a href="http://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/">1</a> and <a href="http://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/">2</a>) and proposed complementary organizational models (Squads, Tribes, Chapters, and so on.).</p><p>Their proposal, tinged with agile method, comes from what I call the “reduction of entropy”, namely the reduction of organizational complexities and constraints in favor of productivity.</p><p>Concretely, the fact of passing in feature teams had a direct impact on the teams:</p><ul><li><strong>Mixed team</strong>: the people who needs to achieve a feature are grouped together, during the whole time of the production of the feature;</li><li><strong>Reduction of the hierarchy</strong>: there is no manager in the feature teams;</li><li><strong>Flat working circuit</strong>: the delivery workflow is simple and standardized, leaded by the feature team;</li><li><strong>Team goals rather than personal goals</strong>: if the feature team succeeds then it is the company that succeeds and therefore everyone;</li><li><strong>Distinction by seniority</strong>: the evolution of the collaborators is made by the experience and the mastery of each of the competences necessary to the realization of its work.</li></ul><h3>Basic lessons</h3><p>After 4 months of experimentation, we have learned the following lessons:</p><ul><li>It is necessary to work on the <em>meaning of each element </em>of the organization, to appropriate it and to understand how it interacts with the rest of the organization;</li><li>The <em>commitment</em> of the company must be <em>total</em> in this organizational pivot: we organized a vote by show of hands to decide to commit the company in this adventure;</li><li><strong>The product must be ahead of developments</strong>;</li><li>The structuring into “<em>guilds</em>” does not only concern the technical team, but all the teams involved in the creation of features;</li><li>Emergencies and bugs have their place in this model, they must be managed;</li><li>Everything does not fit into a feature team, typically the work of some support functions is sidelined, or some subjects that are dealt with in the long run.</li></ul><p>The organization must serve primarily the product and is such code, it must be <strong>maintained</strong>.</p><p>We evaluate and test each ritual to understand what works and what could be improved or abandoned.</p><p>We note that the atmosphere of the company has changed a lot, a lot of work but also a lot of collaboration, even complicity. The first feedback from the team is unanimous, it is necessary to continue working in feature team.</p><h3>Conclusion</h3><p>At <a href="http://stootie.com/">Stootie</a>, as I write, we are a Series-A fundraising company and reaching the 40 people, +25 in 2016.</p><blockquote>Should a startup organize itself in feature teams ?</blockquote><p>The transition to the feature teams is the organizational counterpart to the transition to a microservices architecture. So, if you already planned to change your architecture, you <em>might</em> <em>strongly</em> consider organizing your company in feature teams.</p><p><strong>Stootie takes the advantage now by organizing itself as a company that efficiently scales its agility.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ba4407c8e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices at Stootie]]></title>
            <link>https://medium.com/@maximethomas_46287/microservices-at-stootie-3873a06b7ada?source=rss-b55541eeb7e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3873a06b7ada</guid>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Maxime Thomas]]></dc:creator>
            <pubDate>Wed, 30 Aug 2017 15:58:49 GMT</pubDate>
            <atom:updated>2017-08-30T15:58:49.773Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VSa3XVLdIQPYDfh54XUxcA.png" /></figure><p><a href="http://stootie.com/">Stootie</a> is the first application for peer-to-peer services in France. A simple, fast and cheap solution to find a hand to fix a leak or move a sofa. We design and produce a technology platform for the general public in the form of applications (iOS, Android and Web) and private and partner APIs.</p><p>When I arrived at Stootie, I did an audit of this platform and I confronted it to the ambitions of our CEO, <a href="http://fr.linkedin.com/in/arnal/fr">Jean-Jacques Arnal</a>.</p><p>In less than 5 years, the organic growth of the company led all the different technical contributors to setup an API Platform.</p><p>This backend reached at the beginning of 2016 more than 450 endpoints and covered the totality of functional needs of the company: the product (Mobile and Web), the exposure of data to the moderation team and a graphic user interface for statistics with numerous indicators.</p><p>It had a known limit of traffic (around 400 req/sec) and had been designed for vertical evolution, always increasing the hardware capacity on which it was installed.</p><p>We were at this moment in a dead end, sure to succeed on the business part but strained by our current system, unable to evolve.</p><p>We stepped up our level of ambitions when preparing our 4th Series-A fundraising and initiated two pivots which transformed our company, both technical and organizational. I describe here the technical one.</p><h3>Retrospective on software architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/0*qDcdIjxICrBNNIX_.png" /></figure><p>During the 90’s, developers where artisans, realizing unique piece of code, monolithic. When software have expanded, the base code became quickly jumbled, illegible, incomprehensible, and if we changed something on the left side, this might have consequences on the right side. Not really reliable and so expensive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/0*B1Ivl7VaEYZinYKk.png" /></figure><p>In the 2000s, the advent of Java and its first frameworks brought a first distribution of the code, splitting into Model View Controller (MVC), where the elements of a software were stored in different layers (lasagne code). This guaranteed a better independence of each layer and, in fact, its robustness and extensibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*aOcpZ7f1IwVAhGpo.png" /></figure><p>Unfortunately, companies have grown, globalizing this layered architecture model to sprawling levels, thus replicating spaghetti code within each layer of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/0*0pKtW6Lq9m97A_uH.png" /></figure><p>Since 2012, the <a href="http://www.martinfowler.com/articles/microservices.html">microservices</a> approach is convincing more and more companies whose agility to adapt their technical platform is critical to their business: Amazon, Google, Netflix, Spotify, etc.</p><p>Microservice vision is borrowed from the Linux philosophy: full of small tools and each tool has a single responsibility and realizes it perfectly.</p><p>This architecture involves cutting out all of our code in layers to have a set of small software dealing with one and only one feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/0*AKx0Kdb-8TNjTid4.png" /></figure><p>This is the approach we have adopted at Stootie to meet our ambitions as the world’s leading collaborative economy platform.</p><h3>Technological pivot</h3><p>We decided to rotate in January towards a <a href="http://www.martinfowler.com/articles/microservices.html">microservices</a> architecture in the following ways:</p><ul><li>The services in place can not stop evolving or functioning;</li><li>A microservice is always attached to a finite functional domain, thus reinforcing the <a href="http://en.wikipedia.org/wiki/Domain-driven_design">DDD</a>;</li><li>A microservice defines an interface contract that is stable over time;</li><li>A microservice belongs to one and only one technical owner;</li><li>A microservice is implemented on its own stack and one can only assume this stack when the appropriation of the need has been realized;</li><li>The tech owner has the freedom to choose the technologies he wants to use but must “prove” his stack to the rest of the team;</li><li>A microservice is developed only in a very short period of time, namely two weeks because in two weeks it is not possible to recreate spaghetti code in microservices;</li></ul><p>These simple and precise rules allowed us to quickly set up a team where each member became a tech owner of one or more topics.</p><p>Sam Newman, in his book “Building Micro Services”, gives many feedbacks on the establishment of microservices. It includes a number of tips that we have applied that have saved us a lot of time:</p><ul><li>When cutting the monolith, always start with the data;</li><li>Well weigh the pros and cons between the synchronous and the asynchronous (we have changed our opinion several times on some microservices);</li><li>Do not cascade synchronous microservices;</li><li>Use the CI / CD from the beginning of development and for each microservice;</li></ul><p>He also defines an architecture model called Backend For Frontend (<a href="http://samnewman.io/patterns/architectural/bff/">BFF</a>), which corresponds to a gateway API between the outside of the company and the microservice ecosystem. We put it in place easily and created a complementary principle, the BFB — Backend For Backend — which corresponds to a façade interface for external services.</p><h3>Basic lessons</h3><p>After one year of testing and discovery of microservices, we have learned several fundamental lessons:</p><ul><li>Define the functional areas to be reduced to autonomous minimum perimeters;</li><li>Everybody in the team is a Dev/Ops, let devs learn how your cloud works;</li><li>Migration means technical migration to the same identical functional scope (evolving is tempting and dangerous)</li></ul><h3>Conclusion</h3><blockquote><em>Should a startup migrate to a microservices architecture?</em></blockquote><p>You have to be willing to do it and go all the way.</p><p>Keep in mind that this is <strong>costly</strong> in time because each feature is specific and requires custom engineering. The technical team spent <em>3 month</em>s developing the first microservice in order to implement a continuous integration / continuous delivery pipe on each technology. The next microservice has been released in <em>one week</em>. Exciting.</p><p><strong>The results are there, we release a new micro services every 2 weeks and have never stopped delivering new functionalities in production.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3873a06b7ada" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>