<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Inside the Tech by SoftServe - Medium]]></title>
        <description><![CDATA[Tech сases, tips&amp;tricks, how to`s, trends etc. by SoftServe`s top notch experts - Medium]]></description>
        <link>https://medium.com/inside-the-tech-by-softserve?source=rss----2f321e7469d3---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Inside the Tech by SoftServe - Medium</title>
            <link>https://medium.com/inside-the-tech-by-softserve?source=rss----2f321e7469d3---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:59:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/inside-the-tech-by-softserve" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Modern web application’s realities: micro frontends, server-side rendering and PWA]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/modern-web-applications-realities-micro-frontends-server-side-rendering-and-pwa-6036f18d4722?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/6036f18d4722</guid>
            <category><![CDATA[frontend-architecture]]></category>
            <category><![CDATA[server-side-rendering]]></category>
            <category><![CDATA[single-page-applications]]></category>
            <category><![CDATA[micro-frontends]]></category>
            <category><![CDATA[single-spa]]></category>
            <dc:creator><![CDATA[Vladyslav Bezruchko]]></dc:creator>
            <pubDate>Mon, 20 Sep 2021 11:12:13 GMT</pubDate>
            <atom:updated>2021-09-20T12:12:07.038Z</atom:updated>
            <content:encoded><![CDATA[<h3>Modern web application’s realities: micro frontends, server-side rendering, and PWA</h3><p>Authors: <a href="https://medium.com/u/83d05e90b5fa">Vladyslav Bezruchko</a>, <a href="https://medium.com/u/d3d8f1ae4cf1">Volodymyr Fedak</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*7t7Pho8zpgGW3OxPhxeNtQ.png" /></figure><p>This article provides an overview of single-page applications (SPA) Web Applications evolution and a sample project that can be used for micro-frontends implementation (<a href="https://github.com/SoftServeInc/mfs-accelerator">https://github.com/SoftServeInc/mfs-accelerator</a> ). Mentioned above sample application can be used as a reference to validate your decisions before the development (or integration) of “Enterprise” Web Applications (a kind of application used to satisfy the needs of an organization — businesses, schools, interest-based user groups, clubs, charities, and governments. In our experience, such SPAs had a huge code base up to 300K-0.5M LOCs).</p><p><strong>Single Page Applications: good and bad parts</strong></p><p>Nowadays, single-page applications take a crucial path in web development. Let’s start our journey from the basic definition. What is a single-page application? It’s a web application built all over a single HTML page to replicate native app behavior in the web browser. Initially, the term single-page application was first used by Steve Yen in 2005[1]. At the beginning of 2010x, a set of powerful javascript frameworks for building a single page was introduced: AngularJS, BackboneJs, etc.</p><p>During the first years, developers were happy because of new and awesome experiences working on single-page applications based on mentioned frameworks. Users also enjoyed it because web applications behaved very similar to desktop: without page reloads and having a great performance (of course, only in case a single-page app was implemented correctly). Later it occurred that there were lots of cons in single-page web application applications. Let’s go through the list of problems that single-page applications had at the early beginning:</p><ol><li>The load of the first page took too long. It’s happening because firstly browser gets a very tiny HTML with one tag &lt;app/&gt; and with &lt;script&gt; tags with JS files. The browser receives JS from the server and starts building (rendering) HTML. During this time, the user sees only a white page, and that’s a problem because, according to statistics, the more user waits the more chance that he will leave and never come back to the site.</li><li>Problems with search engines (Google, Yahoo, Bing, etc.). Search engines could not render HTML and see only tiny HTML with one tag &lt;app/&gt; and scripts. That’s why search engines were not able to index single-page applications.</li><li>Performance problems: very often, because of the pure design of frameworks or the incorrect usage of the frameworks, single-page applications became terribly slow and took lots of system (user’s browser) memory. A good example might be Angular1 watcher’s memory issues (<a href="https://stackoverflow.com/questions/9682092/how-does-data-binding-work-in-angularjs">https://stackoverflow.com/questions/9682092/how-does-data-binding-work-in-angularjs</a> ).</li><li>Compatibility problems: it was complicated and error-prone to have few frameworks working together in one application. Furthermore, different versions of frameworks were not compatible: Angular1 and Angular2+. It led to a significant problem: imagine a large enterprise company developing an application from 2013 using Angular1. In 2016 it occurred that they cannot switch to Angular2 because it’s another framework and Angular1 is promised to be deprecated. It meant that companies had to invest a solid amount of money for migration to the new Angular2 which wasn’t predictable upfront.</li></ol><p><strong>Natural SPAs evolution into micro frontends</strong></p><p>Let’s firstly start this topic with the analysis of the backend microservices architectural style. Wikipedia says: “A workshop of software architects held near Venice in May 2011 used the term “microservice” to describe what the participants saw as a common architectural style that many of them had been recently exploring.” If we look at Google trends — we will see that the search term “microservices” started in 2015 became increasingly popular.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*EU4vYi7NYapi6rY2lemPWw.png" /></figure><p>Figure 1. Growing popularity of microservices over the time</p><p>Microservices architecture style focuses on solving the scaling problem of big monolithic applications (especially in clouds). The front-end developers started to observe similar problems. People started to create SPAs at the beginning of the 2010s, and in 2015th there were many huge monolithic SPAs where every change was extra complicated, and achieving reliable performance was also a big problem. A typical example is a big company that has been working on large-scale enterprise SPA for five years. The company built a huge monolith SPA based on an old framework, deeply coupled into the application’s core, so migration to the new frameworks is very painful and expensive. At the same time, every change affected the whole application. Considering the problems with performance in such cases, we can understand why the front-end world started to think about the applicability of microservices architecture principles to SPAs. Here is where micro frontends started. The micro frontend is an architectural style where independently delivered frontend applications are composed into a greater application.</p><p>Let’s look again at Google trends. We can see that the search term “micro frontends” has become increasingly popular starting from 2018.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*UWLTbH7cPXk-akg1sUyIGg.png" /></figure><p>Figure 2. Growing popularity of the micro frontends over the time</p><p>The modern process of building next-gen web solutions requires separate teams to work on independent applications (micro frontends). Every team has its own functional scope, builds and tests process, deployment pipeline, and independent release process.</p><p>The following picture below represents that idea (a separate release new functionality in Home micro frontend without affecting all other micro frontends):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*YGX4cMqqAiI8G6r_Kc6BPQ.png" /></figure><p>Figure 3. Typical Ci/CD setup for micro frontends</p><p>Consequently, this approach allows to:</p><ul><li>Have a separate delivery pipeline for the micro frontend (it is usually represented as a group of pages)</li><li>Establish more scalable organizations with decoupled, autonomous teams</li><li>Have smaller, more cohesive, and maintainable codebases</li></ul><p>Martin Fowler did a great micro frontends definition [2]</p><p>Nowadays there are lots of different frameworks and libraries that can help you to organize micro-frontends. We propose to concentrate on two of the most popular frameworks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*EGDqH0ZtP0xN8SDJrrp0rw.png" /></figure><p>Table 1. The comparison between two the most popular frameworks/libs for micro frontends</p><p>BIT is a different philosophy in building micro frontends because initially, it’s a tool for component management that focuses on using the BIT cloud for hosting those components and organizing CI/CD for them. At the same time, Single SPA is more like just a wrapper over micro frontends.</p><p>There is always a possibility to build custom shells (wrappers) that will host micro frontends and avoid using the above library/frameworks, which will help mitigate vendor lock problems. On the other hand, it will couple your delivery team with some self-made solutions that might be hard to support in the future.</p><p>In the scope of this article, a sample application is provided (<a href="https://github.com/SoftServeInc/mfs-accelerator">https://github.com/SoftServeInc/mfs-accelerator</a>) with a typical micro frontend approach (usually we see such approach utilized when several teams have been working on a particular solution). It’s not required to have such a variety of frameworks however it happens in real life when different teams work in separated subprojects. Please see the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uGN_bUjJ-ABJ7uSvCPP6hQ.png" /></figure><p>Figure 4. Sample architecture for micro frontends application</p><p>The key point here is that over time WEB-UI teams should have resilient and “framework-agnostic” Web-UI app. In our terms “framework-agnostic” means having the possibility to extend web app with a necessary framework. On the provided sample Single SPA has been used as a shell (wrapper) for micro frontends based on Angular/React/Vue. Each micro frontend is a stand-alone SPA that knows how to bootstrap, mount and unmount itself from the DOM using Single SPA tooling. That is the only thing that couples your micro frontend to a wrapper (Single SPA ).</p><p>Another feature of the provided sample is to have an easy ability to write reusable shared UI components using any JS library/framework. It might be achieved by wrapping them with web components (a suite of different technologies allowing you to create reusable custom elements — with their functionality encapsulated away from the rest of the code via browser API). [3]</p><p>For internationalization, the i18next is used which supports all popular frameworks.</p><p>It will look similar in all frameworks(angular, react, vue), please see this example:</p><pre>&lt;mfs-button ref={refreshButton}&gt;</pre><pre>  {i18nnextCommonDashboard.t(“Users.”)}</pre><pre>&lt;/mfs-button&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*L1fCwLQyGXCwLc6G3I32iw.png" /></figure><p>Figure 5. i18next ecosystem</p><p>For this article, we prepared an example that was built using the architecture approach mentioned above. SystemJS approach was used for module sharing together with Single SPA, but at the same time, you can go with Module Federation (another modern approach). In short, the Module federation is a part of webpack core which allows a JavaScript application to dynamically load code from another application — in the process, sharing dependencies. If an application consuming a federated module does not have a dependency needed by the federated code — webpack will download the missing dependency from that federated build origin. This tool should help to build micro frontend applications in a very efficient way. You only have to remember that module federation is a new technology and it became stable in Webpack 5.0.0 ( 10’th of October 2020 ). You must test it carefully in some PoC to understand if it works well with your setup. In our case, we used the standard for the Single SPA SystemJS approach, but if you tried to use Module federation with Single SPA — please let us know in the comments to this post.</p><p>To learn more about module federation please check the following articles [4, 5]</p><p>If you plan to build a micro frontend application, it makes sense to look into Monorepo, which will allow you to separate build and deployments having one codebase. There are lots of alternatives, but in our example, we used yarn workspaces for it. Yarn workspaces were selected since it’s simpler, but you can try to use a more advanced solution like LernaJS or Nx workspaces.</p><p>In PoC we did not cover cross micro frontends communication because it’s not a general use case to have such communication, and you must be very careful with it. If your micro frontends communicate too much with each other probably you need to combine them into one micro-frontend [6]</p><p><strong>Server-side rendering as an approach for improving performance and SEO</strong></p><p>According to this research [7], the slower page response time results in an increase in page abandonment.</p><p>Taking into account that the slow load of the first page is a weak side of SPA — this aspect might be crucial for many different business applications. Another weak part of SPA might be the problem with search engines: originally search engines were not able to render the page via js and in fact, they saw just a tag to load SPA (see picture below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*HRoBrTD7MRwXodf_dqlyvQ.png" /></figure><p>Figure 6. Typical response from the server the with first(root) SPA page.</p><p>In 2009 Google team made a proposal for making AJAX crawlable [8]. And in the middle of 201x Google crawler started to render SPA and index it. But there were some problems with it because the crawler had an old Chrome version (for example in 2019 it had <strong>41 </strong>Chrome Version but the actual was <strong>73</strong>). Finally in 2019 Google SEO team fixed it and now Chrome can index pages better. But still, there is a difference: sites with the classic server-side approach are now indexed faster than general SPA because for SPA Google crawler uses an additional queue (for the pages which have js and need to be rendered before indexing). You can get more details from this speech <a href="https://www.youtube.com/watch?v=vjj8B4sq0UI">Technical SEO for JavaScript developers [eng] / Martin Splitt</a></p><p>So, the businesses that needed at the same time both SPA, SEO, and quick first-page load were struggling, and server-side rendering was introduced to overcome that (you can check how it’s done in the Angular ecosystem [9] ). The idea is that the NodeJS backend can build HTML using Angular, React, etc and return to the client(browser) ready page so that the user sees complete HTML in a shorter period and the search engine can index it. Originally such solutions were implemented by enthusiastic people in separate repositories, some of them became part of the framework later (for example Angular Universal). React ecosystem has Nextjs for it, vuejs has Nuxtjs, and so on. Interesting thing is that you can run NodeJS and organize server-side rendering using non-js environments like java and c# [10]</p><p><strong>CI/CD</strong></p><p>In the context of micro frontends, it is essential to have the right automation framework for every component. At least the following properties must be configured: code builds, integrations, static/dynamic code analysis, quality gates, and performance analysis.</p><p>Among the other parts of CI, it will be good to highlight performance analysis automation. It can be organized by integrating into CI the tool (for example Lighthouse CI by Google) which will monitor different aspects of the pages. It’s possible because the tool can connect to the browser and get different information like performance, accessibility, etc.</p><p>There is one more thing that is about the agility of a web interface to respond to business demands. What if some of the new or updated pages have to be available only to a part of the customers. And later, these changes should be rolled out to the wider audience in case the feedback is positive.</p><p>Limited availability deployments with feature toggles approach will provide the possibility to gradually release new features for specific users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*K3K-SIPfN7IgFW8YLTr77g.png" /></figure><p>Figure 7. Feature toggle approach for new feature deployment in micro-fronted.</p><p><strong>Mobile-first and progressive web applications</strong></p><p>Last year, the conversion rate (the percentage of visitors to your website that complete a desired goal (a conversion) out of the total number of visitors) of mobile applications compared to desktop was growing. It means that more and more users prefer to access web applications via smartphone. That’s why we suggest paying attention to such patterns as mobile-first and PWA.</p><p>Mobile-first tells us that we have to build a web application that will be efficient on the mobile, first of all. This approach gives the following benefits:</p><ol><li>Performance (mobile devices are not so fast as desktops which force developers to optimize performance)</li><li>Better Search Engine optimization (because search engines rank higher web applications which are optimized for mobile devices)</li><li>Better UX (because it’s becoming easier for users to access the content)</li></ol><p>You can read more about mobile-first here [11]</p><p>Another important approach that will allow getting more users is Progressive Web Application(PWA). According to this article, [12] PWA Progressive Web Apps are web apps that use emerging web browser APIs and features along with traditional progressive enhancement strategy to bring a native app-like user experience to cross-platform web applications. This means that you can, for example, open your PWA website in a browser on your phone and add it as an application to your home screen which will give you the possibility to use your web application as native. The main items here are:</p><ol><li>HTTPS for the contents securing</li><li>Service workers (for offline support)</li><li>Manifest (for the controlling on app’s appearance for the user)</li><li>App Shell approach ( will give you faster page load )</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-MabvFypNfNYXtkDvSVyQ.png" /></figure><p>Figure 8. PWA explained</p><p>In provided sample application PWA support is added to the root micro frontend wrapper (Single SPA) — you can check it here. Also, in this example, use cases related to offline support were handled via service workers.</p><p>Testing Aspect</p><p>Talking about the modern approach for building SPA we cannot forget about testing. Historically front end has not been tested well but we are sure that it is the wrong way. The testing strategy plays a crucial role in the success of the solution. We suggest using a classic approach called test pyramid when the test suite consists of three layers: unit testing, integration testing, and end-to-end (e2e) testing. The core motivations are linked to Agile objectives: reducing cycle-time, meeting frequent deadlines, rapid feedback, improved and co-located collaboration. Also, the largest part of the testing will be automated that allows having quick feedback on any change in the codebase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wh_pu8N3bAfG6Af7LLCfXA.png" /></figure><p>Figure 9. Testing approach for SPAs</p><p><strong>Summary</strong></p><p>Single page Web applications had come a long way since 2005 when they were introduced. Nowadays, there are tons of JS-based frameworks that you can use to develop your SPA. If you build a simple application, you can choose the framework based on various aspects ( e.g. your developers are familiar with React or Angular ). However, suppose you are planning to build a huge application from scratch. In that case, you must be incredibly careful with your decisions not to face a case when your engineering team will be required to rewrite the majority of applications as it was with Angular1-&gt;Angular2+ migration in huge projects. The micro frontend architecture style is a helpful tool for huge applications, and it has to be as framework agnostic as possible. Additionally, use such techniques as mobile-first and PWA to improve your web application from the mobile device’s point of view.</p><p>Use provided example (<a href="https://github.com/SoftServeInc/mfs-accelerator">https://github.com/SoftServeInc/mfs-accelerator</a>) as a reference for your own decisions and make contributions if you find it useful.</p><p><strong>Resources:</strong></p><ol><li><strong>What Compelled Single Page Applications to Jump into Trends </strong><a href="https://www.linkedin.com/pulse/what-compelled-single-page-applications-jump-trends-katy-slemon/">https://www.linkedin.com/pulse/what-compelled-single-page-applications-jump-trends-katy-slemon/</a></li><li><strong>Micro Frontends </strong><a href="https://martinfowler.com/articles/micro-frontends.html">https://martinfowler.com/articles/micro-frontends.html</a></li><li><strong>Web Component </strong><a href="https://developer.mozilla.org/en-US/docs/Web/Web_Components">https://developer.mozilla.org/en-US/docs/Web/Web_Components</a></li><li><strong>Webpack 5 Module Federation: A game-changer in JavaScript architecture </strong><a href="https://medium.com/swlh/webpack-5-module-federation-a-game-changer-to-javascript-architecture-bcdd30e02669">https://medium.com/swlh/webpack-5-module-federation-a-game-changer-to-javascript-architecture-bcdd30e02669</a></li><li><strong>Module Federation </strong><a href="https://webpack.js.org/concepts/module-federation/">https://webpack.js.org/concepts/module-federation/</a></li><li><strong>Cross micro frontends communication </strong><a href="https://dev.to/luistak/cross-micro-frontends-communication-30m3">https://dev.to/luistak/cross-micro-frontends-communication-30m3</a></li><li><strong>How Fast Should A Website Load &amp; How To Speed It Up </strong><a href="https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds/">https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds/</a></li><li><strong>A proposal for making AJAX crawlable </strong><a href="https://developers.google.com/search/blog/2009/10/proposal-for-making-ajax-crawlable">https://developers.google.com/search/blog/2009/10/proposal-for-making-ajax-crawlable</a></li><li><strong>Angular Universal </strong><a href="https://blog.angular-university.io/angular-universal/">https://blog.angular-university.io/angular-universal/</a></li><li><strong>Angular Universal Angular &amp; ASP.NET Core Engine </strong><a href="https://github.com/angular/universal/blob/master/modules/aspnetcore-engine/README.md">https://github.com/angular/universal/blob/master/modules/aspnetcore-engine/README.md</a></li><li><strong>Mobile First </strong><a href="https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Responsive/Mobile_first">https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Responsive/Mobile_first</a></li><li><strong>Progressive Web Applications </strong><a href="https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps">https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6036f18d4722" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/modern-web-applications-realities-micro-frontends-server-side-rendering-and-pwa-6036f18d4722">Modern web application’s realities: micro frontends, server-side rendering and PWA</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS privilege escalation: exploring odd features of the Trust Policy]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/aws-privilege-escalation-exploring-odd-features-of-the-trust-policy-7a970a32861?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/7a970a32861</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[infosec]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[hacking]]></category>
            <dc:creator><![CDATA[Pawel Rzepa]]></dc:creator>
            <pubDate>Thu, 02 Sep 2021 07:22:33 GMT</pubDate>
            <atom:updated>2021-09-03T07:47:50.793Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html">IAM roles</a> are commonly used, for example, to grant access to AWS service, account, or federated identity. Each role has a document associated with it, which is called a Trust Policy. This document specifies who can assume the role and under what conditions it’s allowed or denied. The Trust Policy is often a subject of special interest among attackers, because it may be configured to work as a <a href="https://ebookreading.net/view/book/EB9781789136722_152.html">backdoor to compromised AWS account</a> or even an <a href="https://unit42.paloaltonetworks.com/iam-roles-compromised-workloads/">entry point for anonymous users</a>. However, in this blog post, I want to focus on some inconsistencies in the Trust Policy access model, which in consequence can be used for privilege escalation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*W67lNgPfQPLgJ6wLjQC-cQ.png" /></figure><h3>Abusing the Trust Policy</h3><p>In general, an IAM role can be assumed when the 2 following requirements are met:</p><ol><li>The target role has a trust relationship with the principal, which attempts to assume the role.</li><li>The principal attempting to assume the role has the sts:AssumeRole permission.</li></ol><p>However, there are exceptional situations when the second requirement is not essential. A Trust Policy is in fact an IAM <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_identity-vs-resource.html">resource-based policy</a>. That means that within a single AWS account you only need permission from resource-based policy OR principal policy (across AWS accounts you need both). There are certain use cases when Trust Policy can “grant” the sts:AssumeRole permission and the principal doesn’t need to have explicitly such permission and still can assume a role. That may sound confusing, so please let me demonstrate this mechanism in an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*Mr3KJvLi_T7PYapd1mO4HA.gif" /></figure><h3>Escalating privileges through a chain of trust relationships</h3><p>Let’s assume, there’s an AWS account, which has 2 groups of IAM users: the unprivileged Operators and the Admins (who can do anything they want and also have sts:AssumeRole permission to assume any role). Furthermore, this demo AWS account has the role called <em>maintenance-role,</em> which can be assumed by administrators in the whole organization. This is defined by the following Trust Policy:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>       <strong> &quot;AWS&quot;: &quot;*&quot;</strong><br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          <strong>&quot;aws:PrincipalOrgID&quot;: &quot;o-abcd12efg1&quot;</strong><br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>I’ve seen such Trust Policy several times in the wild and it’s even presented in the official AWS Security Blog (s̵e̵e̵ ̵t̵h̵e̵ ̵<em>”̵L̵i̵m̵i̵t̵i̵n̵g̵ ̵o̵r̵ ̵e̵x̵t̵e̵n̵d̵i̵n̵g̵ ̵a̵c̵c̵e̵s̵s̵ ̵t̵o̵ ̵a̵ ̵r̵o̵l̵e̵ ̵b̵a̵s̵e̵d̵ ̵o̵n̵ ̵A̵W̵S̵ ̵O̵r̵g̵a̵n̵i̵z̵a̵t̵i̵o̵n̵s̵”̵</em> section of <a href="https://web.archive.org/web/20210225192754/https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/">this article</a>; please note that I refer to the archived version of this blog post, because after publishing this article the above-mentioned section was removed; big-ups to AWS team for really quick action!). Surprisingly, the wildcard character in the <em>Principal</em> section causes the strange behavior, which is: <strong>every principal in the ‘o-abcd12efg1’ organization, and in the same AWS account as the assumed role, can assume the role WITHOUT </strong><strong>sts:AssumeRolepermission</strong>😯</p><blockquote>Please note that in the above-mentioned use case Trust Policy behaves like a resource-based policy. It “grants” sts:AssumeRole permission to the principal in the same AWS account. However, principals in other AWS accounts and under the same OU will need the sts:AssumeRole permission, because resource-based policies don’t “grant” permissions cross-account (only under the same AWS account).</blockquote><p>In other words, any unprivileged principal (even IAM users who have no permissions at all), who are in <em>o-abcd12efg1</em> organization and in the same AWS account as the assumed role, can assume the <em>maintenance-role</em>. Pretty scary, huh?</p><p>Furthermore, let’s assume there’s an additional <em>admin-role, </em>which can be assumed by the <em>maintenance-role. </em>This can be done by specifying the following Trust Policy associated with <em>admin-role:</em></p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>       <strong> &quot;AWS&quot;: &quot;arn:aws:iam::111111111111:role/maintenance-role&quot;</strong><br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>      &quot;Condition&quot;: {}<br>    }<br>  ]<br>}</pre><p>As you may read in the <em>Hacking the Cloud </em><a href="https://hackingthe.cloud/aws/general-knowledge/assume_role_logic/">article</a>, <strong>if the target role’s trust relationship is tied to a specific role, then the base role does NOT need to have </strong><strong>sts:AssumeRolepermission😯</strong></p><p>To sum up, the attack flow to escalate privileges from nothing to administrator is the following:</p><ol><li>The unprivileged IAM user assumes the <em>maintenance-role. </em>The user doesn’t require sts:AssumeRole permission, because the role’s trust relationship contains the wildcard character.</li><li>The <em>maintenance-role </em>entity assumes the <em>admin-role. </em>There’s no need to have sts:AssumeRole permission because the <em>admin-role’s </em>trust relationship refers to the <em>maintenance-role’s </em>ARN.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ms0YGlJe5r7PJ2wx2DfOvQ.png" /></figure><h3>Conclusions</h3><p>IMHO missing the requirement of having explicitly the sts:AssumeRole permission in the use cases mentioned above is a kind of inconsistency in the Trust Policy model, which may result in a serious security threat. It isn’t intuitive, so I’m not surprised that some administrators may unwittingly configure IAM roles in an insecure way. Quoting Murphy’s law:</p><blockquote>“Anything that can go wrong will go wrong”</blockquote><p>Do you know any other ways of abusing a Trust Policy, which wasn’t mentioned in this article? If so, please let me know in the comments or on my <a href="https://twitter.com/Rzepsky">Twitter</a> or <a href="https://www.linkedin.com/in/pawel-rzepa/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a970a32861" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/aws-privilege-escalation-exploring-odd-features-of-the-trust-policy-7a970a32861">AWS privilege escalation: exploring odd features of the Trust Policy</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps in Plain Words]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/mlops-in-plain-words-9b118bba3d13?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/9b118bba3d13</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[SoftServe]]></dc:creator>
            <pubDate>Thu, 10 Jun 2021 12:26:59 GMT</pubDate>
            <atom:updated>2021-06-10T12:25:20.799Z</atom:updated>
            <content:encoded><![CDATA[<p>What is MLOps? Is it here to stay or just another trend to fade away? How has it changed the way ML projects are done?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*nCMtH_aVrGYiwCMNeinE2w.png" /></figure><p>SoftServe launched MLOps practice about two years ago as a reasonable response to the very same challenges, Data Scientists have been experiencing over and over while implementing different projects.</p><p>So far so good. So, in this article Volodymyr Solskyy, leader of MLOps practice, demonstrates why this new Ops is so to the point.</p><h3>To start with, let’s take a look at the recent role of Machine Learning</h3><p>Growing complexity of modern-day tasks has been exposing limitations of traditional Software Engineering for quite a while now, forcing us to search for new approaches. Machine Learning has already proven to be one of the best candidates for the job so far, with its ability to scale to the complexity without requiring extensive domain experience from ML engineers. This leads to some people naming ML as new Software Engineering or Software 2.0.</p><p>Sounds promising, yet it’s too early to state that we’ve got a silver bullet. <strong>Why?</strong></p><p>To answer this question, we need to look holistically at the whole process of applying ML to meet particular objective or to address a problem.</p><p>There is misperception that Machine Learning is all about model training and tuning. Well, not really, in fact it’s just a small piece of a larger puzzle. To deliver value from Machine Learning <strong>multiple steps must be taken</strong>, generally grouped in two phases:</p><ol><li>Model development — starts with data ingestion, then data analysis, feature crafting and model creation.</li><li>Model serving — is needed to make your model useful. It involves ingesting new, previously unseen data that needs to be validated against your existing model pipelines.</li></ol><p>While we managed to harness the first phase, the second one remains <strong>a major challenge</strong>.</p><p>According to Gartner survey, in 2018 <strong>only 47% of all AI/ML projects made it fully to production</strong>. In 2020, this figure grew by 8%. Pretty slow progress, right? The root of the problem lies in difficulties with integrating ML model into real environment.</p><blockquote>McKinsey: The inability to integrate analytic solutions into workflows and achieve frontline adoption is the number one inhibitor to why data and analytics initiatives fail.</blockquote><p>What makes bringing AI/ML solution to production such a tall order?</p><p>Out of numerous factors we can distinguish three central ones:</p><ul><li><strong>Insufficient data quality</strong></li></ul><p>ML is a completely data centric paradigm. Getting enough data seems to be a no-brainer as we have numerous sources of information and millions of data points generated daily. The key issue is quality of this data from both business and technical perspective.</p><ul><li><strong>Rapid growth of complexity</strong></li></ul><p>As the industry matures, complexity of tasks for us to solve grows exponentially. ML by nature will scale to everything we are going to throw at it, but we are still left to manage the complexity. Therefore, we generate more and more experiments involving more and more data, which requires more computational power, GPUs, FPGs, etc. Consequently, we bury ourselves in the sheer ramification of infrastructure, code and processes required to deliver value.</p><ul><li><strong>Unclear success criteria</strong></li></ul><p>Traditional Software Engineering heavily relies on functional specifications and requirements, making evaluation of progress and results relatively straightforward task.</p><p>In ML, we operate “proxy” functions and metrics. This also gets the job done. But the way how we get there is different, quite often making it quite hard to reconcile business objectives with obtained results. Also, reaching good numbers for objective functions does not necessarily transforms into success from business standpoint (look no further than Netflix challenge, for example).</p><p><strong>How can we do better?</strong> Before addressing this question, let`s clarify what this “better” stands for.</p><h3>Measuring progress in bringing ML solutions to production</h3><p>To be able to quantify progress of any effort, we need some set of objective metrics for it, and ML in production is not an exception.</p><p>Over the years of accumulating experience, our team has arrived at 4 major categories that cover most of the process, specifically KPIs related to:</p><p><strong>ML:</strong></p><ul><li>Implementation time</li><li>Time to deploy</li><li>Model performance</li><li>Business impact</li><li>Number of model calls (i.e. relevancy or popularity)</li><li>Average call duration</li></ul><p><strong>Data dependencies:</strong></p><ul><li>Time to access</li><li>Data ingestion time</li><li>Number of data issues</li></ul><p><strong>Infrastructure:</strong></p><ul><li>Resource provision time</li><li>Auditability</li><li>Chargeback and showback reports</li></ul><p><strong>Operations:</strong></p><ul><li>Number of models in production</li><li>Number of new models deployed</li><li>Model versioning</li><li>Number of calls and errors</li><li>Hardware utilization</li></ul><p>Over the years of experiments, industry worked out what can be considered <strong>an ideal ML project lifecycle</strong>.</p><p>It starts with framing your business idea or need — doing sanity check whether that idea is feasible at all — going over the rapid cycle of modelling iterations until good model is created — proceeding to production phase (to have everything deployed) — measuring the impact to understand if the initial objectives are met. The result of such analysis is an input to any next large iteration of the project.</p><p>Here is an extensive scheme of this process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*-S-rK5_qeBlFK8cVAT9lpg.png" /></figure><p>Can we expect a typical Data Science team to deliver this cycle end-to-end? Well, experience so far has shown us that no. And here’s why.</p><h3>The brutal reality of developing ML project without MLOps</h3><p>At average, ML teams are built around the idea that desirable outcome of their work is either a model, or a nice, ad-hoc research report.</p><p>Hence, the primary focus is on stuffing those teams mostly with Data Scientists that are extremely well versed in research, mathematics, and data analysis, but usually less knowledgeable/experienced in engineering topics like scaling, automation, code optimizations, etc. In theory those parts can always be covered by Software Engineers, but <strong>this is precisely where things start to break</strong>, since both sides usually work at different velocities using different (often conflicting) approaches, different set of tools, and, in general, speak different languages. This leads to missed deadlines, poor overall quality, and teams being completely frustrated with each other. All together it causes huge losses for business.</p><p>Now, deploying ML to production is challenging stage, yet it is <strong>not the end of the journey</strong>. Congrats for making it this far and welcome to a new exciting world of operating ML-based systems in production. Your data sets will drift, and models will degrade; you will encounter biases, privacy and compliance issues, deal with latencies, throughputs, and general infrastructure instabilities, debug complex problems, and scratch your head over reproducibility issues.</p><p>This brutal reality can quickly and easily overwhelm unprepared teams, leading to countless overtimes, wasted budgets, and frustrated stakeholders effectively spelling doom for another ML initiative.</p><p>However, the journey does not necessarily need to end like this…</p><h3>Ingredients of successful ML project</h3><p>As we briefly touched upon already, successful ML project requires at least the following set of skills:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*BF87AmpIJxOtyg0A8CWLZA.png" /></figure><p>In a nutshell:</p><ul><li>Data Scientists doing all that stuff around model training.</li><li>Data engineers doing lots of heavy lifting with the data.</li><li>DevOps engineers supporting the environment.</li><li>Software Engineers implementing integration with the upstream and downstream systems.</li></ul><p>But as we have also shown in previous section, having experts to cover each of those skills wouldn’t do any good.</p><p><strong>This is where MLOps comes into play.</strong></p><h3>The essence of MLOps</h3><p>Just like others in Ops family, MLOps is not a particular profession or role. Instead, it’s a form of a contract, a set of practices aimed at enabling Data Scientists and Subject Matter Experts to deliver their work faster and in more robust and scalable way.</p><p><strong>The pillars of MLOps contract, among others, include:</strong></p><ul><li>Shared responsibilities and trust.</li><li>Self-service APIs and services.</li><li>Reusable code templates and playbooks.</li><li>Coding conventions and style guides.</li><li>Support for ML toolkit and frameworks.</li><li>Unified environments and images.</li><li>Unified data access and operations.</li><li>CI/CD and SDLC guidelines.</li></ul><p><strong>Applied properly, this contract will:</strong></p><ul><li>Establish tight collaboration between researchers and engineers.</li><li>Minimize code refactoring and rewrite between research and production.</li><li>Establish iterative, rapid, and high-quality ML model delivery lifecycle.</li><li>Ensure MLOPs practices are shared among the stakeholders.</li></ul><p><strong>What does this seamless collaboration bring? Ability to deliver ML models to clients or business users faster and more efficiently.</strong></p><h3>What changes does it introduce to our work routine?</h3><p><strong>Centralized ML data management</strong></p><p>If there’s one single improvement that should be considered for every AI project, it’s this one — data unification for model development and serving.</p><p><strong>Key idea:</strong> ML requires data that is properly consolidated (preferably into a single source), QCed, and automated.</p><p>Quite often, the project involves multiple data sources such as streams, data lakes and databases, object stores, and manual annotations. This complex landscape is the exact reason why a Data Scientist spends 80% of their time searching for relevant data and doing data munging. This is also source of too many surprises in production while the model is live and serving users.</p><p>The solution to this problem is <strong>Feature Store</strong> — a central vault for storing curated, well documented, tightly controlled features that can be used for training and serving across multiple models. Essentially, feature store is data warehouse for ML. Having one helps Data Scientists to quickly search and reuse available features, avoid duplication, ensure that features are always up to date and there’s no discrepancy between data used for training and serving.</p><p><strong>Versioning</strong></p><p>While in traditional Software Engineering the end result primarily depends on the code written by developers, the situation is much more complex for ML-based systems. Since ML is purely data-centric approach, even slightest change in incoming data or data pipeline can and will lead to a different result. Also, as many algorithms are non-deterministic by nature, the results might heavily depend on initial values, hyperparameters used, etc.</p><p>As such, to ensure reproducibility, provenance, and ease hand-offs of models, it makes sense to version control:</p><ul><li>Training data: raw data snapshots.</li><li>Environments: images, libraries, dependencies.</li><li>Data Transformations: data pipelines and processing code.</li><li>Features: feature metadata and extraction code.</li><li>Models: model code, weights, hyperparameters, binaries.</li><li>Experiments: configuring, metrics, artifacts.</li></ul><p><strong>Modularity</strong></p><p>While it’s quite tempting (and actually practiced) to put everything into a single script or a notebook, such monoliths are hard to optimize, scale, and reuse. Instead, it’s way better to structure the code as a set of abstracted out, loosely coupled components with well-defined interfaces and join them in a form of a DAG or pipeline.</p><p>Applying this approach enables us to:</p><ul><li>Reuse single components in other pipelines.</li><li>Scale each component individually if needed.</li><li>Test, debug, and troubleshoot the code faster and easier.</li></ul><p>It also allows us to utilize full power of platforms such as Kubernetes/Kubeflow and Airflow.</p><p><strong>Building CI/CD pipelines for ML</strong></p><p>No matter how many improvements introduced, we won’t be able reap the benefits if we do things manually. In fact, the level of automation for the data, models, and code pipelines determines level of maturity for ML process. Let’s take a look at how the fully automated process might look like:</p><p>The process starts from data that is stored in central location — feature store.</p><p>From here Data Scientists begin model development on the data sample. They go through the regular cycle of experimentation, data validation, feature crafting, building models, etc. Here we encountered first major change — the output of Data Scientists work is not a model anymore, but the reproducible code that is committed to a source code repository.</p><p>Once the code is there, the operations team can orchestrate the deployment into staging/production environment, where fully automated pipelines will run all the steps on entire dataset, produce a model, and store it in model registry. All parameters and details of this process are to be recorded in metadata storage for later audit (if required). From model registry the model is deployed in semi- of fully-automated fashion via A/B testing or Challenger process. Once the model is in production, it will be closely monitored by the ops team along with its data pipelines to ensure system stability and all of the important findings will be fed back to development/training/deployment loop.</p><p>As you might have noticed already, this new process is faster, less prone to human error, and ensures proper separation of responsibilities between various roles.</p><h3>What does it bring us? The most essential benefits of MLOps</h3><p>Reproducibility &amp; Auditability:</p><ul><li>All artifacts can be tagged and audited.</li><li>Pipelines are reproducible and their results are verifiable.</li><li>Code drives deployments.</li></ul><p>Validation:</p><ul><li>Minimize bias and enable provenance.</li><li>Online/offline comparisons of model quality.</li><li>SWE best practices for quality control.</li></ul><p>Automation &amp; observability:</p><ul><li>Controlled rollout capabilities.</li><li>Live comparison for model performance: real vs. expected.</li><li>Feedback loop for model improvements and drift.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b118bba3d13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/mlops-in-plain-words-9b118bba3d13">MLOps in Plain Words</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python and Stripe: Implementing an online payment system in e-shop.]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/python-and-stripe-implementing-an-online-payment-system-in-e-shop-13b617ebf243?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/13b617ebf243</guid>
            <category><![CDATA[stripe]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[payments]]></category>
            <dc:creator><![CDATA[Maciej Januszewski]]></dc:creator>
            <pubDate>Mon, 26 Apr 2021 15:58:37 GMT</pubDate>
            <atom:updated>2021-04-26T15:58:37.016Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you wondering about your own business? Already have a website? If so, you’ve come to the right place! In this article I’ll tell you how to implement Stripe — one of the most popular online payment platforms.</p><p>Pssst… <strong>#python</strong> on the board!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*yyMIPoDSsFLLuiAkoc_HFA.png" /></figure><p>At the beginning, let me introduce you one of the most popular online payment platforms — <a href="https://stripe.com/"><strong>Stripe</strong></a>. It operates in over <a href="https://stripe.com/global"><strong>44 countries</strong></a> and is constantly expanding its activities to new countries. It allows you to accept transfers, settlements, payments, and also provides a ready interface for user management. You, as the owner, do not store users card / bank account details — Stripe provides full support and relieves you of any responsibility for sensitive data.</p><figure><img alt="www.stripe.com" src="https://cdn-images-1.medium.com/max/1024/1*UakO1Sq0_QvUSD5hZ7qScA.png" /><figcaption><strong>www.stripe.com</strong></figcaption></figure><p>It is also worth saying that Stripe operates on the basis of a <strong>commission model </strong>— depending on the country, the appropriate amount is charged for a successful debiting of the card (in Poland it is <strong>1.40% + PLN 1</strong> for each transaction).</p><p>For you, as a seller, using online payments means charging commissions on transactions: transfers, card payments etc. In a word, you<strong> earn</strong>.</p><p>And most of all, Stripe provides an <strong>API</strong>, written in <strong>#python</strong>. Let’s get down to the implementation!</p><h3>Stripe implementation</h3><p>Let’s assume we keep basic information about the users in database (we’ll create tables in <a href="https://docs.djangoproject.com/en/3.2/topics/db/models/#module-django.db.models"><strong>Django</strong></a>). The user model (<strong>User</strong>), in addition to the profile (<strong>UserProfile</strong>), will also have a <strong>stripe_id</strong> field in which we will store the unique user ID generated by Stripe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87cf29f39e2417cf67519da99e60da84/href">https://medium.com/media/87cf29f39e2417cf67519da99e60da84/href</a></iframe><p>Having previously defined models, we can create test users:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd52edeb1dc3df1fc13176c950c3858a/href">https://medium.com/media/fd52edeb1dc3df1fc13176c950c3858a/href</a></iframe><p>We have all the information we need, we’ve created two users and we can start working with Stripe.</p><h3>Creating Accounts</h3><p>Everything really depends on what type of account we would like to associate our user with. If our user orders services (<strong>Employer</strong>), we create a <a href="https://stripe.com/docs/api/customers"><strong>Customer</strong></a> account:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3388cdcad506a7dc1a97ef8ca4786270/href">https://medium.com/media/3388cdcad506a7dc1a97ef8ca4786270/href</a></iframe><p>On the other hand, when our user will provide services (<strong>Employee</strong> — contractor), we create a <a href="https://stripe.com/docs/api/connected_accounts"><strong>Connected Account</strong></a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/378e7e8f391de54a3f6553008d0f7671/href">https://medium.com/media/378e7e8f391de54a3f6553008d0f7671/href</a></iframe><p>In both cases, Stripe’s API returns an <strong>unique ID</strong> (<strong><em>id</em></strong> or <strong><em>stripe_id</em></strong>), thanks to whom we can identify our users.</p><p>If we create a <strong>Customer</strong> account, the ID is preceded by the prefix “<strong>cus_xxx</strong>” and in the case of <strong>Connected Account</strong> the prefix is “<strong>acct_xxx</strong>”.</p><p>While the Customer account does not require additional steps in the context of registration, in order to have a fully verified Connected Account, you must:</p><ul><li>accept the Stripe policy:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81765203aa24d64cc1eae6b8d004539b/href">https://medium.com/media/81765203aa24d64cc1eae6b8d004539b/href</a></iframe><p>We provide here the current time (date) and the user’s IP address (ip) in the form of a timestamp.</p><ul><li>attach the documents <strong><em>confirming identity</em></strong> (passport, ID card or driving license):</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a762d81d84e048db85befe23c553b92/href">https://medium.com/media/2a762d81d84e048db85befe23c553b92/href</a></iframe><p>In this example, we submit a <strong><em>passport</em></strong> for verification — in addition to the front photo of the document, the same object must also be submitted as an additional document. Otherwise, the verification will fail…</p><p>These are all of the required steps when it comes to customer registration. It’s time to add your <strong><em>card</em></strong> to your Customer account:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6d4062578fc553b164af6534ee7ea0c/href">https://medium.com/media/b6d4062578fc553b164af6534ee7ea0c/href</a></iframe><p>First of all, we need to add a new <strong>payment source</strong> to the Customer (<strong><em>token</em></strong> which contains encoded information about the card). Then we need to set the previously created payment method as the <strong>default</strong> (<a href="https://stripe.com/docs/api/setup_intents"><strong><em>SetupIntent</em></strong></a>).</p><p>Now we can try to withdraw the funds from <strong>Customer’s</strong> account:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9a9d1705d8c6ba23f2a7c37a8012a4a/href">https://medium.com/media/a9a9d1705d8c6ba23f2a7c37a8012a4a/href</a></iframe><p>In this example, the Customer’s card is debited by the value given in the <strong><em>amount</em></strong> variable. Our commission is <strong><em>application_fee_amount</em></strong> — here we have to enter an already calculated amount of commission that should go to our account.</p><p><strong><em>transfer_data</em></strong> is responsible for the target account (where the funds for the order are to go) — in <strong><em>destination</em></strong> we enter the client’s <strong><em>Stripe ID</em></strong> (Connected Account). Then we need to confirm that the Customer card is charged (<strong><em>payment_intent_confirm</em></strong>) — as a result there will be a temporary amount blockage on the Customer’s card.</p><p>When we want to withdraw funds from the Customer card, we do:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/add0bd67568f14f2832f0845847b4f22/href">https://medium.com/media/add0bd67568f14f2832f0845847b4f22/href</a></iframe><p>After calling this method, funds should appear on the account (Connected Account).</p><p>To check its <strong>balance</strong>, enter:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff4fdd3f89ce2f88facfee0707f3d728/href">https://medium.com/media/ff4fdd3f89ce2f88facfee0707f3d728/href</a></iframe><p>In response we’ll receive information about the funds <strong>currently available </strong>on the account:</p><ul><li><strong><em>pending</em></strong> — in progress, which means that the funds are not yet available for withdrawal.</li><li><strong><em>available</em></strong> — available, which means the funds can be withdrawn now.</li></ul><p>Sometimes there may be a situation in which we will have to charge one of the users, e.g. for the lack of an order, and we have already charged the Customer’s card:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b198108b1a35fa3e0db0e3af2747d9c/href">https://medium.com/media/0b198108b1a35fa3e0db0e3af2747d9c/href</a></iframe><p>In this way, the contractor (Connected Account) can be charged with value specified in the <strong>amount</strong>. In a result we’ll reduce its current account balance.</p><p>And now we have to return the funds (Customer):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8bb4f9966b30141c51b46102daa6531/href">https://medium.com/media/a8bb4f9966b30141c51b46102daa6531/href</a></iframe><h3>Summary</h3><p>Stripe provides an API that allows us to quickly and easily build an online payment support system. Depending on our “business model”, we’re able to easily select properly implemented methods.</p><p>The API documentation is so transparent that, after proper reading, we can start writing a multi-line code that does all the work for us — of course, we need to properly test the logic we wrote to make sure that every transaction will be processed according to our expectations. This is can be achieved by test tokens, as well as test bank and card numbers, thanks to which we can perform real transactions.</p><p>After properly performed tests, we can switch from <strong>development</strong> to <strong>production</strong> mode. The only difference will be the <strong>API key</strong> that will be used for every request to Stripe.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13b617ebf243" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/python-and-stripe-implementing-an-online-payment-system-in-e-shop-13b617ebf243">Python and Stripe: Implementing an online payment system in e-shop.</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guide to Successful Preparation for a Client Interview]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/guide-to-successful-preparation-for-a-client-interview-e068a4b7a94?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/e068a4b7a94</guid>
            <category><![CDATA[client-relationship]]></category>
            <category><![CDATA[customer-interview]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[clients]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[SoftServe]]></dc:creator>
            <pubDate>Fri, 12 Feb 2021 11:30:45 GMT</pubDate>
            <atom:updated>2021-03-26T09:45:49.233Z</atom:updated>
            <content:encoded><![CDATA[<p>Is a client interview your nightmare? Read this article to know how to prepare effectively and get a client to hire you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*95qszy0gcLYZ5E5aARTo0A.png" /></figure><p>Yevhen Behen, a competence expert at the Software Development Office of SoftServe, has passed dozens of client interviews and now helps engineers to succeed in this tricky yet important step. On top of that such preparation benefits clients as well, as they can get all the needed information about a specialist and decide if it`s a perfect fit for the position.</p><p>Below he outlines the most crucial aspects.</p><h3>How a client interview is different from a job interview?</h3><p>A job interview consists of two parts:</p><ul><li>formalized technical interview — checks your professional level and has pretty much the same structure for all engineers within one profile.</li><li>Project interview — checks if you meet the requirements of the project and the particular vacancy.</li></ul><p>A client interview is similar to the second one. Though it is different because the mentality of the foreign interviewer is different and the focus will be shifted towards the needs of the client, and not an employer. I`ll dwell upon why it`s important below.</p><h3>Two steps that are often missed while preparing for a client interview</h3><ol><li><strong>Prepare CV</strong></li></ol><p>Don`t underestimate the role of CV. Refusals often happen at this stage. Reasons are trivial:</p><ul><li>Grammar mistakes — it looks as if the person is irresponsible.</li><li>Poor stylistic design — it looks as if the person did not prepare for the interview properly and does not meet the level claimed.</li></ul><p>There are two versions of this document:</p><ul><li>The advanced version containing the entire professional history (CV) — start with your latest projects and describe them in detail. Place projects that started your career, or irrelevant to the client’s technology stack projects only with a general description at the end.</li><li>Shortened version (resume, or excerpt from CV) — the last few projects, or the most relevant to the client&#39;s needs.</li></ul><p>In the general summary and skills description, focus on matters that will present you as a specialist best suited for a particular project. The rest is not of interest to this particular client.</p><p>When describing projects, check the NDA compliance with previous clients and do not disclose their confidential information.</p><p><strong>2. Rehearse self-presentation</strong></p><p>Self-presentation is an opportunity to promote yourself. Please use it. But impromptu speech (especially in English) is a tricky thing. Believe me, investing time into preparation is beyond important.</p><p>Simply writing a text is not enough. I strongly recommend having several rehearsals and deliver your speech. If you don’t have a listener to give you feedback, try to practice in front of a mirror.</p><p>3 key points to check here:</p><ul><li>Duration of the speech</li><li>Sequence of facts</li><li>Logic of presentation</li></ul><p>While saying out loud the words you will understand what parts can be confusing or not present you in the best light.</p><p>Here is an example of a proper structure:</p><ul><li>Education (if you have a relevant one) because for Americans and Europeans it is important</li><li>A general summary of your experience (how many years in the profession, what technologies you work with, what projects you have performed on, what methodologies you have worked with)</li><li>Details on the last two or three projects (project ecosystem, team structure, responsibilities within the team, requirements) to make it clear that you have a helicopter view, and not just know your technical part</li><li>The logical ending (to make it clear that you have completed your story).</li></ul><h3>How interview goes?</h3><p>I distinguish three types of interviews:</p><ul><li><strong>An interviewer is a non-technical person</strong> — they will check your soft skills. Usually, a conversation is built around common topics, whereas technology is brought up on a very high level.</li><li><strong>Classic technical interview</strong> — the interviewers often want to know about your latest projects and ask a lot of questions from the profile (what frameworks you have worked with, what approaches you have used, your code quality, etc.).</li><li><strong>In-depth technical interview</strong> — in addition to (or instead of) technical questions, you will be asked to solve tasks. This is the most stressful option since you will code in front of the interviewers. Even experienced professionals can get confused because it is hard to work when you are watched by strangers and expected to get the result as soon as possible. When we prepare a colleague for this kind of interview, we model similar sessions and go through them several times (starting with clarifying questions for the task in English, and then perform the actual coding and presentation of the result).</li></ul><p>The best preparation here is to train and perform certain technical tasks while someone is watching you.</p><p>There are also <strong>homework interviews</strong>. In this case, the client sends a link to an assignment and an interviewee has from 24 to 48 hours to do it. In this case, you need to carefully analyze and code the solution that meets the basic attributes of code quality and is optimal for performance.</p><h3>What non-technical questions you can be asked?</h3><p>There is a pretty standard set of questions that interviewers ask most frequently. Something like:</p><ul><li><strong>What are your weaknesses?</strong></li></ul><p>Of course, there is no need to list them all.</p><p>I advise you to answer in such a way that presents your negative traits as a side effect of the positive ones.</p><p>Example: “I am meticulous and very attentive to details, so my weakness is that I sometimes need more time to perform tasks”. In the end, it may be advantageous, for example for a QC engineer in the medical field.</p><ul><li><strong>Share some examples of your failures (clients from America love this question)</strong></li></ul><p>It is perfectly normal to talk about it because failures happen to everyone. But you need to make it clear that you have drawn conclusions and know how to prevent a similar situation in the future.</p><h3>What to do if you don`t know the answer?</h3><p>If you cannot answer the question, <strong>do not say it directly</strong>. Instead, show your ability to find a way out of any situation, e.g. “I haven’t used this technology yet, but I know a similar one that performs this task. I assume that it has identical functionality, so I will be able to figure it out very quickly.”</p><p>Foreign clients expect a person to praise themselves and to show themselves in the best light. They will smile and not show that they are dissatisfied with the answer, little can be read on their faces. They ask more conceptual things — how do you see business, how do you develop (because they want to see a person within a broader competence).</p><p>Mentality difference:</p><ul><li><strong>Americans</strong> tend to ask how you feel about working on the weekends. Since it is their pace of life, they often demand it from the people they work with.</li><li><strong>Europeans</strong> may have a different understanding of life-work balance, and they usually do not ask about overtime.</li><li><strong>The British</strong> clients are always very polite, but even their conventional complex language structures can confuse a person.</li></ul><p>I strongly advise you to thoroughly study the information about the client, considering that questions like “What do you know about our company and why do you want to work with us?” occur quite often. It is essential for some clients that a person chose to work with them consciously.</p><h3>Is it ok to ask an interviewer questions?</h3><p>Not only it is ok, but it’s also can be extremely productive for both sides.</p><p>It shows your interest and gives you a bonus in the eyes of the interviewer.</p><p>In most cases, it`s better to do it at the end, after all topics of interest to the interviewer were covered. But it depends on the style of the interviewer though — there are people who are looking for a dialogue during the interview. Therefore, if you feel that the person is eager to talk, you can already start asking questions.</p><p>I`d recommend avoiding the following topics:</p><ul><li>religion</li><li>politics</li><li>sport (you cannot be certain what sport a person likes and which team he supports)</li><li>operational matters that will be decided with the project manager (for example, whether you will have an on-site trip to the client country or salary issues)</li><li>about the history of your company’s cooperation with the client. It is something the interviewer may not even know or be interested in.</li><li>some basics about the client, that are easy to google. E.g. client’s history, how many years the company has been working, etc. Stay focused on the project: what are the plans for further development of the project? What is the structure of the project team? How are the processes built?</li></ul><h3>Last but not least ….. proficiency in English</h3><p>It`s a must.</p><p>If you do not have regular speaking practice, it will be difficult for you to talk for an hour. Or if you use English most often at work, it can be difficult to talk about some general topics unrelated to technology.</p><p>Therefore, it is better to start <strong>preparing in advance</strong> and at least visit the language courses or conversation clubs a few times. When I prepare a colleague for a client interview, I also check how long a person can speak English fluently.</p><p>In conclusion, I strongly recommend that <strong>you prepare for the interview and start not a day or two earlier, but even more in advance</strong> to understand your gaps, so you have time to tackle them. The preparation benefits clients as well because they can get all the needed information about a specialist and make a better hiring decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e068a4b7a94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/guide-to-successful-preparation-for-a-client-interview-e068a4b7a94">Guide to Successful Preparation for a Client Interview</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On the visualization of longitudinal data]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/on-the-visualization-of-longitudinal-data-fed0db846ca1?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/fed0db846ca1</guid>
            <category><![CDATA[longitudinal-study]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualisation]]></category>
            <dc:creator><![CDATA[Borys Kabakov]]></dc:creator>
            <pubDate>Mon, 01 Feb 2021 09:30:50 GMT</pubDate>
            <atom:updated>2021-02-02T13:45:50.769Z</atom:updated>
            <content:encoded><![CDATA[<h3>Workable method of longitudinal data visualization</h3><p>Data visualization is a tall order. Especially for eHealthcare project, that may contain too much data of different types. Well-known methods not always come in handy. Here`s a step-by-step guideline to workable alternative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*pwk0-z8UBcOoZnIy9euY0Q.png" /></figure><h3>Background</h3><p>While working on eHealthcare projects we measure many features at different times. E.g., in a clinical trial, we could have a thousand patients to check blood pressure (BP), biochemistry, pulse, etc. weekly throughout few months.</p><p>Then we need to visualize all the gathered data to gain insights or to present results. And there is only a handful of common ways to do it:</p><ul><li>Individual line trends</li><li>Boxplots</li><li>Violin plots</li><li>Sankey plots</li></ul><p>All those methods have their flaws.</p><p>Individual trends could be a mess, especially if you have a lot of patients. Boxplots could be misleading and lose to violin plots almost in everything (e.g., <a href="https://blog.bioturing.com/2018/05/16/5-reasons-you-should-use-a-violin-graph/">5 reasons you should use a violin graph</a>). Violin plots do not show transitions between the states and sankey plot could be hard to read.</p><p><strong>Why data visualization matters? Real-life case</strong></p><p>Once we were working with a Fortune 500 list company on improving the treatment protocol for a certain chronic condition. The client had all the patient data divided into two clusters (which seemed pretty natural for such case):</p><ul><li>Responding to treatment</li><li>Non-responding to treatment.</li></ul><p>However, having taken a closer look from the different timescale, we understood that they had two more groups of patients:</p><ul><li>Who responded to therapy at first but failed at the end</li><li>Those who failed at first but responded at the end.</li></ul><p>It was valuable insight with a great business value, that turned the whole project in a totally different direction.</p><p>Our challenge here was to make it clear for the client. No question that a poor presentation of the finding would lead to misunderstanding, client`s dissatisfaction, and reputational losses. Moreover, the decision makers usually don’t have the time and/or expertise to dive into the tangled DS Jupyter notebooks or series of complicated plots for answers.</p><p>That`s where a proper data visualization comes in. And it totally worked for us. Below I`ll walk you through our approach which we have called Vyshyvanka plot.</p><p><em>This is my first article on Medium which I created personally, so I was not aware how actually bad this platform fits for publishing technical articles with a lot of code (no syntax highlighting, no formulas, no spoiler blocks, no footnotes, etc.). Basically, if one wants to paste highlighted python code he has to create a gist and embed it here, which has at least two bad consequences: the author will be left with a ton of gists, readers will have long unfolded sheets of code to scroll through.</em></p><p><em>I decided to carve out all code to perceive readability. The occasional reader who wanders in here may find all code in </em><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article"><em>the accompanying repository</em></a><em> or read </em><a href="https://hackmd.io/@banderlog/By2MYxtSv"><em>the original article on hack.md</em></a><em>.</em></p><h3>Use case data generation</h3><p>Consider the case: you have a thousand patients whose BP was measured 6 times at certain time points. You decide to group your BP results in 5 groups (1...5) by range, where 1 could represent the ‘very low’, and 5 ‘very high’. And now, you want to visualize results.</p><p><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb">Here</a> I generate a completely artificial dataset for illustrative purposes, with the very strange behavior of BP levels.</p><p>It looks like this (columns are patients, rows are points in time, cell values are BP group that Patient’s BP fell into at a given time point):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/0*aFDJ_AvNaaaF0axx" /></figure><h3>Why Vyshyvanka plot?</h3><p>Let’s plot generated data using each of four common methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*2pSvpRbFgzDPLg7Q" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb</a></figcaption></figure><p>We may see that individual trends are a mess, boxplots are not very informative and a violin plot shows us that patients are heavily distributed between 2nd and 4th groups and their distribution does not change significantly over time.</p><p>But this is simply not true, we created the dataset in a way where each patient has an equal chance to appear in the 2 or the 4 group at the next step. Will Sankey plot help us?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*EHTCpGofbhOdijwr" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb</a></figcaption></figure><p>Well, a sort of (here N_ numbers stands for BP group and _M numbers stands for certain time point, e.g. 3_2 is 3rd BP group at 2nd timepoint). We are able to see some transactions, but the plot is too fiddly and obviously needs some tuning. You may find a separate chapter dedicated to Sankey plot problems on python below.</p><p>Meanwhile, compare it with a better way to visualize this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*FMdQ7Nmy7--mPm-d" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb</a></figcaption></figure><p>It shows BP group transition over time. The line width and markers opacity (black dots) correlates with the number of patients. Here we can see that a majority of patients started in the 2 and the 4 groups and these groups have a stable amount of patients during the whole period of observation. But it also shows that there were 2 types of transitions with relatively equal probabilities: to remain in the initial group or switch between 2nd and 4th, with a small chance to switch to group 1, 3, or 5. Here we do not see patients’ individual pathways, rather the transition probability trends.</p><p>And this could be a pretty valuable insight. Also, it is a convenient plot to present and explain.</p><p>We have invented this plot almost by accident under the pressure of practical needs during our work with the real customer. Our Lviv teammates have named it “Vyshyvanka plot” due to similarity of appearance with the <a href="https://en.wikipedia.org/wiki/Vyshyvanka">national Ukrainian costume</a> pattern.</p><h3>Python problems with Sankey plot</h3><p>There are not so many python plotting libs (<a href="https://mode.com/blog/python-data-visualization-libraries/">link1</a>, <a href="https://mode.com/blog/python-interactive-plot-libraries/">link2</a>) and only a few that could do Sankey plots — <a href="https://matplotlib.org/api/sankey_api.html">matplotlib</a>, <a href="https://holoviews.org/reference/elements/bokeh/Sankey.html">holoviews</a> and <a href="https://plotly.com/python/sankey-diagram/">plotly</a> (also there is a jupyter widget <a href="https://github.com/ricklupton/ipysankeywidget">ipysankeywidget</a>). But only one lib could create beautiful enough Sankey plots — plotly (through JS frontend). Let’s plot our generated dataset again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*OZDeILwO6lvYxog0" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb</a></figcaption></figure><p>To make it more intuitive we must force-order our BP groups like on Vyshyvanka plot, thus we will be able to keep track of group transitions much more easily:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*Y9tPQLXTrgUkNlY4" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb</a></figcaption></figure><p>Despite that on this example everything seems OK, plotly has a lot of problems with changing default nodes order, which makes it hard to use it on most data (see <a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">examples</a>). Consider these github issues (especially the last one):</p><ul><li><a href="https://github.com/plotly/plotly.py/issues/3004">FR: Add textposition property for Sankey plot’s node labels</a></li><li><a href="https://github.com/plotly/plotly.py/issues/1732">Sankey node x and y positions require both variables set to reflect in chart</a></li><li><a href="https://github.com/plotly/plotly.py/issues/3002">Sankey node x and y positions could not be equal to 0 or np.nan</a></li><li><a href="https://github.com/plotly/plotly.py/issues/3003">Sankey node x and y positions ignored if node is empty</a></li></ul><h3>More examples</h3><p>Here I’ll be plotting violin, vyshyvanka, and Sankey plots only because they are obviously superior to individual line trends and boxplots. Also, to make it more general, we will discontinue the BP groups example and just talk about state transitions over time.</p><p>You may find all relevant Jupyter notebooks with all code from this post <a href="https://gitlab.com/banderlog/vyshyvanka_plot_article">here</a>.</p><h4>State distribution changes with mode of 1-&gt;5 transition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*dBF2_GAusDqbSOT3" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/0*VWGI0ukAYKpEBvdq" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb</a></figcaption></figure><h4>State distribution changes, bi-modal 1-&gt;5 and 5-&gt;1 transitions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*rIEV-WphLH-gZ800" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/0*HEWlzpFCleqH5TjB" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb</a></figcaption></figure><h4>State distribution splits from initial 3 flowing to either 1 or 5 ultimately</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*Cp5Siwi9hxkgCLf2" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/vyshyvanka_examples.ipynb</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/0*bTZOq75TDF334fju" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/sankey_plot.ipynb</a></figcaption></figure><h3>Interactive Vyshyvanka plot</h3><p>In rare cases, you might want to make it interactive so that to be able to know the exact number and percent of transitioned states by clicking on a line. Also, interactive mode allows for finding optimal values for opacity and line widths. For this, we provide you with the code below. Keep in mind, that it will work in Jupyter notebook or a Jupyter-lab upon installation of <a href="https://mplcursors.readthedocs.io/en/stable/">mplcursors</a> and <a href="https://github.com/jupyter-widgets/ipywidgets/blob/master/docs/source/user_install.md">ipywidgets</a> (be careful with virtualenv), next try <a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/interactive_vyshyvanka.ipynb">that notebook</a>.</p><p>You should get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*6aCJgFfXPhWX4R3C" /><figcaption><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/interactive_vyshyvanka.ipynb">https://gitlab.com/banderlog/vyshyvanka_plot_article/-/blob/master/interactive_vyshyvanka.ipynb</a></figcaption></figure><p>We hope some of you might find the Vyshyvanka plot useful :)</p><p><em>Originally published at </em><a href="https://hackmd.io/@banderlog/By2MYxtSv">https://hackmd.io/@banderlog/By2MYxtSv</a></p><p><em>Accompanying GitLab repo: </em><a href="https://gitlab.com/banderlog/vyshyvanka_plot_article"><em>https://gitlab.com/banderlog/vyshyvanka_plot_article</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fed0db846ca1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/on-the-visualization-of-longitudinal-data-fed0db846ca1">On the visualization of longitudinal data</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to prepare for the Azure DP-100 Exam]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/how-to-prepare-for-the-azure-dp-100-exam-ea01042b3bf?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea01042b3bf</guid>
            <category><![CDATA[azure-ml-studio]]></category>
            <category><![CDATA[azure-ml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[dp-100]]></category>
            <dc:creator><![CDATA[Oleh Lokshyn]]></dc:creator>
            <pubDate>Tue, 05 Jan 2021 10:42:10 GMT</pubDate>
            <atom:updated>2021-01-05T10:42:10.305Z</atom:updated>
            <content:encoded><![CDATA[<h4>Azure DP-100: one thing that makes a huge difference</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*X5ehuD6HSAk5JlTsbt0YvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*YHa4OTAZDLgAre2z.png" /><figcaption>this badge is issued after passing the exam</figcaption></figure><h3>Introduction</h3><p>On the 23rd of December, 2020, <a href="https://www.youracclaim.com/badges/6f776153-8c9f-4a51-b505-6adb3d08e170">I have passed a Microsoft exam called <strong>DP-100: Designing and Implementing a Data Science Solution on Azure</strong></a>. I did it exactly a month later I have passed the Google Professional Machine Learning Engineer exam. <a href="https://towardsdatascience.com/google-professional-machine-learning-engineer-exam-what-to-expect-f1188e356046">You can read about the Google ML Engineer exam experience here</a>.</p><p>Naturally, I hoped that the two exams are roughly the same except for the platform-specific services, so I could save some preparation time by taking these exams in a sequence. The truth is, the two exams have different philosophies.</p><p>Of course, this is subjective, but I would claim that while Google’s ML Engineer is testing your knowledge on how to do Data Science on GCP, Microsoft’s DP-100 is testing whether you have the experience of doing Data Science on Azure. In other words, I would strongly recommend you gain the practical Azure experience before attempting DP-100, while you could pass Google’s exam just by reading and understanding all the materials (haven’t tried it though).</p><p>The DP-100 exam focuses on practice and code. You really should know how to use the Azure ML Python SDK, even remember the names and signatures of the methods. You also should remember the exact workflow of building the model in the Designer and then deploying it. Unless you have a phenomenal memory, it’s impossible to just memorize these things — they should come from experience.</p><p>I bet the last paragraphs made you thinking: “how should one prepare for this exam then?” and “will I ever be ready to take it?”. Don’t worry — I’ve got you covered. In this article, I’ve compiled a preparation plan that should make you bullet-proof on the exam.</p><p><strong>My gratitude to </strong><a href="https://www.softserveinc.com/en-us"><strong>SoftServe</strong></a> for covering the cost of the exam seat and the practice test.</p><h3>Study Guide</h3><p>As always, start from the official materials to make sure you understand the basics. General knowledge of Machine Learning, like the model training and serving workflows and understanding the models’ architectures, is a prerequisite.</p><p>Remember that apart from the knowledge of the Azure services you really want to get the practical experience of using them. Thus, for each official learning path listed below, <strong>make sure to implement the practice assignments yourself</strong>, not just running cells in the notebook.</p><p>As for practice, I would recommend solving two types of problems with Azure ML:</p><ol><li>Regression with tabular data. For example, <a href="https://www.kaggle.com/serigne/stacked-regressions-top-4-on-leaderboard">follow this excellent Kaggle notebook</a>, I will refer to it as the regression notebook further on.</li><li>Classification with image data. For example, <a href="https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457">follow this excellent Kaggle notebook</a>, I will refer to it as the classification notebook further on.</li></ol><p>Finally, the study guide:</p><ol><li>Official learning path: <a href="https://docs.microsoft.com/en-us/learn/paths/build-ai-solutions-with-azure-ml-service/">AI Solutions on Azure ML</a>. The most important official learning path for this exam. It guides you through the Azure ML SDK and covers most of the topics mentioned in the exam’s skills outline.</li><li>Official learning path: <a href="https://docs.microsoft.com/en-us/learn/paths/create-no-code-predictive-models-azure-machine-learning/">No-code predictive models with Azure ML</a>. This one essentially teaches you the UI workflow of the Designer and Automated ML. Remember the modules you use in the Designer, their inputs and outputs, how to write code in the Execute Python Script module, try out hyperparameter tuning. Tinker with the configs in the Automated ML.</li><li>Almost useless <a href="https://docs.microsoft.com/en-us/learn/paths/create-machine-learn-models/">Official learning path on ML in general</a>. You can glance through it, just don’t waste too much time. I’ve included it here because if you find this learning path useful this may indicate that you are not ready for the exam. You can find more resources on general ML in <a href="https://towardsdatascience.com/google-professional-machine-learning-engineer-exam-what-to-expect-f1188e356046">the article on the Google ML Engineer exam</a>.</li><li>Optional: <a href="https://acloudguru.com/course/dp-100-part-1-preparation">A Cloud Guru’s DP-100 preparation course, part 1</a>, and <a href="https://acloudguru.com/course/dp-100-part-2-modeling">part 2</a>. While the course is generally useful, you can get a similar experience in less time by just doing the assignments on your own.</li><li>Required Practice: re-create the regression notebook using Azure ML Python SDK. Store data in an Azure ML Tabular Dataset. Do this practice in an iterative manner, when you implement only part of the feature engineering code and run an experiment, then add more features and look at how the metrics improve over time. Deploy the resulting model as a batch inference pipeline and call it on a new set of data registered as a separate Dataset.</li><li>Required Practice: re-create the classification notebook using Azure ML Python SDK. Store data in an Azure ML File Dataset. Deploy the resulting model as a real-time inference web service protected with JWT, and call it.</li><li>Required Practice: re-create the regression notebook using Azure ML Designer. Perform hyperparameters tuning. Deploy the model as a real-time inference web service protected by key authentication.</li><li>Required Practice: feed the data from the regression notebook in the Automated ML. Exclude some of the model architectures from the search. Deploy the resulting model using Azure ML Python SDK (make sure you know how to fetch the best model).</li><li>Learn by heart (I mean it!) what compute targets are suitable for what workloads <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target">as described here</a>.</li><li>A Secret Sauce, one thing that makes a huge difference, see below.</li></ol><h3>The thing that makes a huge difference</h3><p>If I was forced to name a single most useful preparation resource, this, undoubtedly, would be <a href="https://eu3.mindhub.com/dp-100-designing-and-implementing-a-data-science-solution-on-azure-microsoft-official-practice-test/p/MU-DP-100">the Official Practice Test</a>.</p><p>Microsoft links to this test on <a href="https://docs.microsoft.com/en-us/learn/certifications/exams/dp-100">the very page of the DP-100 exam</a>, but if you, like me, prefer the obvious things repeated explicitly, here you go: “you need this practice test, my friend, I doubt you can pass the exam without it.”</p><p>So, why am I bringing it up here? Because the practice test is not free, and in my country, it actually costs more than an exam attempt. It may be tempting to buy a second exam attempt instead, but the practice test is a more wise investment.</p><p>Pros of the practice test:</p><ol><li>Understand the format of the exam. The format is quite different from Google’s exams and was confusing for me at first.</li><li>Understand the level of the complexity of the questions. The practice test is absolutely representative in this regard.</li><li>The practice test has explanations for almost every answer option, which makes it much more useful than the second test attempt. Besides, you can go through it several times.</li></ol><p>Cons of the practice test:</p><ol><li>Many questions are incorrectly stated. It’s important to know that on the actual exam almost all the questions I got were correctly stated (I’ve reported what I didn’t like, so your experience may be better). So, don’t worry that you’ll get questions with several correct answers or no correct answers at all on the actual exam (unless stated otherwise). It’s a mystery for me why the questions on the practice test are of such low quality.</li><li>Some questions have wrong explanations, so if the explanation contradicts your experience, go and check it with the documentation.</li><li>The testing software that delivers the practice test is just garbage.</li><li>Questions are outdated when compared to the documentation. This should be expected. Again, on the actual exam, the situation is much better.</li></ol><p>The bottom line is, you still need the practice test. I can only wish the quality of the practice test was any match to the high price they ask by the time you buy it.</p><p>Practice the practice test until you know and understand the answers to all the questions. This is called overfitting :)</p><h3>Links to read</h3><ol><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target">Compute targets in Azure ML</a>.</li><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-network-security-overview">Private Link in Azure ML</a>.</li><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-data#access-data-in-storage">Datastore and Dataset</a>.</li><li><a href="https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.ipynb">Authentication in Azure ML</a>.</li></ol><h3>Conclusion</h3><p>Overall, I was pleased by the depth of this exam. It is satisfying to realize that at least you think you know answers to all the questions but one. I’ve scored 945 on this exam, the passing score is 700 and I don’t know what the max score is, just hope it’s 1000 :).</p><p>The exam tests your theoretical knowledge, your practical skills with Azure ML, and, surprisingly, your attention, so read the questions carefully. Actually, read them at least 3 times in full each, the time permits.</p><p>Wish you luck on your path,</p><p>Oleh</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea01042b3bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/how-to-prepare-for-the-azure-dp-100-exam-ea01042b3bf">How to prepare for the Azure DP-100 Exam</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Tech Lead to Architect — Part 2: a key to successful Quality Attribute Workshop]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/from-tech-lead-to-architect-part-2-a-key-to-successful-quality-attribute-workshop-8c19f168118f?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/8c19f168118f</guid>
            <category><![CDATA[it-career]]></category>
            <category><![CDATA[technical-leadership]]></category>
            <category><![CDATA[tech-lead]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[it-consulting]]></category>
            <dc:creator><![CDATA[SoftServe]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 14:11:00 GMT</pubDate>
            <atom:updated>2020-12-08T14:11:00.624Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Tech Lead to Architect — Part 2: a key to successful Quality Attribute Workshop</h3><p>Tips &amp; tricks on Quality Attribute Workshop: how to conduct, what to ask your clients, and how to handle unclear answers in order to get all the needed inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*uYfxN6e0z7o33zGQphBxyw.png" /></figure><p>Dmytro Ovcharenko, Senior Solutions Architect at SoftServe DevCenter in Dnipro, continues series of blog posts about basics of IT Consulting.</p><p><a href="https://medium.com/inside-the-tech-by-softserve/from-tech-lead-to-architect-part-1-it-consulting-process-38f131661af8">In my previous article</a> I walked you through IT consulting process. Now I`ll dwell upon one specific part of it — the Quality Attribute Workshop (QAW). I`ll shed light on what pitfalls you might come across here and my experience how to handle it.</p><h3>Why you need the Quality Attribute Workshop (QAW) and what stages it consists of?</h3><p><strong>What is QAW?</strong></p><p>In plain words it`s a meeting with stakeholders on the client side, that is conducted within Discovery or Pre-Sale session to agree on expectations from a future decision.</p><p><strong>Why is it needed?</strong></p><p>To determine non-functional requirements for a system.</p><p><strong>What if you skip or fail it?</strong></p><p>Expectations will remain at the “highly-available / highly-secure / highly-performed” level, where everyone understands this “highly” in their own way. As a result, your client may still have questions for a ready-made solution, but you will have nothing to appeal to.</p><p><strong>Why is QAW so tricky?</strong></p><p>Things are not always so straightforward and there can be difficulties — from the clients’ incomprehension of what they want to get, to political disputes between stakeholders, when everyone takes their stand and cannot reach a common solution. You have to work through this to get what you want from them anyway.</p><p><strong>QAW step-by-step</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kgaSyZmm4CKMLgwktsOArw.png" /><figcaption>The scheme of how QAW is organized</figcaption></figure><p><strong>Part 1</strong> — preparation within the team</p><p>We collect material for the future brainstorm:</p><ul><li>case studies to work out non-functional requirements</li><li>templates for scripts and architecture (if it is an assessment)</li><li>concerns, constraints, etc.</li></ul><p>Now, it takes me a few hours: an hour to prepare general scenarios without measurements and about an hour to detail and refine those scenarios with the team.</p><p><strong>Part 2</strong> — work with clients</p><p>We usually go to face-to-face meetings. Now, under quarantine and closed borders, we work online, but the principle remains the same.</p><p>I strongly recommend gathering all stakeholders on the client side before the main sessions for a short session (not more than half an hour for them not to get too bored) to explain what we will do and why.</p><p>Then you can proceed to the main part of the QAW, which consists of two stages:</p><ul><li>Brainstorming to generate scenarios</li><li>Scenario prioritization session</li></ul><p>That was a bit of theory, now let`s get down to real life.</p><h3>Hands-on advise to make the most out of QAW</h3><p><strong>Build the discussion around clients’ use cases</strong></p><p>The more specific you are while explaining your ideas the better.</p><p>Too generic explanation — “We plan to make system A that will respond to system B in a certain way; the data loss will not exceed 0.001%”</p><p>Much more clear and closer to the client`s reality explanation — “The user needs to log in to the system to be able to generate a report. This critical feature is always available (in 99.95% of cases), while the system chat can be turned off for a while, which is not critical.”</p><p>This is how we move script by script: we discuss, listen to what stakeholders think, and let everyone speak without criticizing.</p><p><strong>Agree on measurements</strong></p><p>Ask about clear indicators or ranges for each scenario. Be ready, that clients may not know what they want. Yet it`s crucial to get a precise answer to secure yourself.</p><p>I suggest taking a proactive position and advise something, referring to your experience or the world’s best practices. If that doesn’t work, you can go for small provocations. For example, you discuss the latency of a SaaS healthcare solution. The client does not know what it should be. Try to offer 10 seconds — it will make them think faster and offer better options. And eventually you`ll agree on some compromise number.</p><p>Use this approach to go through all your scenarios.</p><p><strong>Remember prioritization</strong></p><p>If prioritization is not worked through, all previous steps will lose their meaning because architecture is built on the understanding of priorities. For example, if the most important quality of the system is security, then we are likely to lose performance. Such things need to be considered and clarified right away.</p><p>Stakeholders on the client side can start disputes, as everyone loves to fiercely defend their interests.</p><p>What helps me in such situations is giving each participant a certain number of points to distribute between the scenarios. This method clearly shows what prevails.</p><p><strong>Track time</strong></p><p>Determine in advance the needed time to discuss each topic, state it in the meeting agenda, and adhere to this timing to go through all the mentioned steps and find out what is truly essential.</p><p>If stakeholders have an hour for everything, we can briefly run through the main points that will help understand the complexity of future infrastructure, deployment, and the general course of the client’s thoughts. It’s important to clarify the security and performance requirements of a system. Depending on the context, there may also be other requirements, such as sustainability or usability (for web application).</p><p>Here, we clearly will have no time to go script after script, so I prepare a quality attribute script in advance, leave the question on the measurements so that we brainstorm specifically them, and not the scripts. Of course, even in this case, there may not be enough time to discuss everything. So, define the most important points for yourself, clarify them at first, and agree to finish the rest in correspondence.</p><p><strong>As a result of all the steps described above, we form a Utility Tree</strong></p><p>The name is not obvious, but it is a table where we describe all our scenarios and define for each of them the level of Importance and Complexity. Risks, assumptions, etc. can also be described there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eoIEBSSfZPBgf5gt1BubcQ.png" /></figure><p>Medium and High priorities are key for us. We will build a design and achieve measurements primarily for these scenarios. The combination of L-H indicates a problem, that we always try to investigate and take into account at the design stage.</p><p><strong>QA Scenario refinement</strong></p><p>The last step, when we refine the requirements, their content, and bring clarity. If you have more questions, it is better to ask for another session to find out everything before the work starts.</p><p>Here is an example of a detailed scenario description. I usually do a shorter version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*_DMRUZ53_yWg_fDN1Qx2EQ.png" /></figure><p>In most cases, we send this document to all stakeholders and ask for confirmation. I remember only one case when we had to continue the discussion for about two more hours struggling with prioritization.</p><h3>How to distribute team roles for QAW?</h3><p>As you can see, QAW is a rather complex process that requires a lot of consideration: from the proper organization of work with stakeholders to finding out all necessary technical details.</p><p>To make your work more convenient, I advise to clearly outline the roles.</p><p><strong>Facilitator</strong> moderates discussions:</p><ul><li>communicates with the client</li><li>holds attention of an audience</li><li>clearly follows an agenda</li><li>resolves controversies during any conversation</li><li>conducts small talks and maintain a friendly atmosphere (it`s important!)</li></ul><p>Business Analyst is an excellent fit for this task.</p><p><strong>Question-keeper:</strong></p><ul><li>asks questions on the technical part, risks, and other relevant topics</li><li>helps the facilitator to direct the conversation in the right direction.</li></ul><p>It is an expert in a particular technology.</p><p><strong>Scrubber</strong> writes down everything that is voiced during the discussions: requirements, risks, assumptions.</p><p>Based on these records, non-functional requirements will be further elaborated, so this role is very important.</p><p>Project Manager can handle this role perfectly, if a team has one. I am often a scrubber myself. In this role, I ask additional questions and watch the reaction of stakeholders, which is very useful.</p><p><strong>What recording tools to use?</strong></p><p>MS Teams + SharePoint is a nice option.</p><p>Files and tasks are stored in one place, and it is convenient to share materials and build team interaction.</p><p>Business Analysts often use XMind.</p><p>I also have a Word template that I can easily fill out.</p><p>But be careful with using a board: it ends quickly, you need to write legibly, take pictures of all records, and then transfer them to electronic form.</p><p><strong>Some useful sources:</strong><br><a href="http://resources.sei.cmu.edu/%E2%80%A6%E2%80%8Bt-view.cfm?assetid=474329">Quality Attribute Workshop Collection</a><br><a href="http://resources.sei.cmu.edu/%E2%80%A6%E2%80%8Bt/2018_010_001_513488.pdf">How to Clarify Architectural Requirements to Ensure Successful Development</a> <br><a href="http://kilthub.cmu.edu/%E2%80%A6%E2%80%8B582656/files/12069557.pdf">Guide on QAW by Carnegie Mellon Software Engineering Institute</a></p><p>Feel like this is something you`d like to try? <a href="https://hubs.ly/H0CbXcK0">Don`t hesitate</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c19f168118f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/from-tech-lead-to-architect-part-2-a-key-to-successful-quality-attribute-workshop-8c19f168118f">From Tech Lead to Architect — Part 2: a key to successful Quality Attribute Workshop</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How i migrated a legacy banking application to Amazon EKS step-by-step]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/how-i-migrated-a-legacy-banking-application-to-amazon-eks-step-by-step-d9ce568d781?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/d9ce568d781</guid>
            <category><![CDATA[amazon-eks]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-migration]]></category>
            <dc:creator><![CDATA[SoftServe]]></dc:creator>
            <pubDate>Wed, 25 Nov 2020 12:01:59 GMT</pubDate>
            <atom:updated>2020-11-27T08:26:21.853Z</atom:updated>
            <content:encoded><![CDATA[<h3>How we migrated a legacy banking application to Amazon EKS step-by-step</h3><p>Banking legacy app, hosted in on-prem datacenter, with high maintenance cost, long Time To Market, poor 20-year-old UX and no Agility. Client wants an upgraded Amazon EKS-based solution. Here`s a step-by-step guideline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*lyHvZOyyEvdIv9fQs7u42A.png" /></figure><p>I am Oleksandr Vorobiov, DevOps Engineer at SoftServe Development Center in Kharkiv. For the last three years I have mostly been working with Amazon Cloud (apps migration, optimization, and re-architecture). The project, I`m writing about, is one of the most challenging and interesting so far.</p><h3>Resolution plan</h3><p>Along with re-platforming, we took four additional approaches:</p><ul><li><strong>Infrastructure as a Code (IaaC)</strong> to easily process all the needed resources.</li><li><strong>Helmfile deployment</strong> to orchestrate deployment of the components in Kubernetes.</li><li><strong>CI/CD</strong> to automate the whole process of the migration.</li><li><strong>GitFlow</strong> to develop microservices.</li></ul><h3>Preparation</h3><p>Prior to the migration development team:</p><ol><li>Broke down legacy code to small chunks by converting the source code to Java and C++ and added additional features.</li><li>Wrapped the app into Docker containers.</li><li>Designed a brand new architecture for our app as shown in the diagram.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6DCoKmIloiZ7NBcEEQZFEQ.jpeg" /><figcaption><em>In-cloud app architecture</em></figcaption></figure><p>To AWS Cloud we deployed VPC with three subnets in it. Each subnet contains pre-deployed services:</p><ul><li>Amazon EKS.</li><li>Aurora databases, those numbers vary depending on the environment.</li><li>Kafka services, provided by Amazon.</li><li>Elasticsearch with basic components set.</li></ul><p>4. Used <em>nested stacks</em> to easily update and make changes to the product. Each nested stack is represented by a .yaml file that leverages a separate set of AWS services. This approach lets us make changes to different parts of infrastructure without any disruptions to the whole product.</p><p>5. Used <em>Cloud Formation Custom Resources</em> which essentially are Python scripts, wrapped into AWS Lambda. Custom Resources are used to make custom configuration to some services in case it is not supported by CloudFormation, such as Kafka.</p><h3>Migration overview</h3><p>The most obvious solution, when it comes to leveraging microservices in Kubernetes, would be using Helm charts. Indeed, we can use Jenkins to compile Helm charts and pack Docker images with executable code into them. Additionally, Helm charts allow customizing parameters for different environments, such as Production, Staging, or Test.</p><p><em>But…the issue is Helm charts don’t allow you to define deployment dependencies between the microservices.</em></p><p>We can use umbrella helm chart but still it doesn’t provide required level of configuration and dependency management.</p><p>And the solution was…</p><p><strong>Helmfile</strong></p><p>Think of it as of the Helm for your Helm. You practically wrap your Helm charts into Helmfile. With this, you can easily manage environments using parameters in values files and, most importantly, seamlessly define dependencies between your Helm charts.</p><p>The structure of Helmfile is as easy as pie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*I2b4Vs6bhnXeu6QNYtZZ7Q.png" /><figcaption><em>Snapshot of Helmfile</em></figcaption></figure><p>From top to bottom you:</p><ol><li>Specify your environments’ info.</li><li>Register internal Helm chart’s repository, hosted in Artifactory.</li><li>Describe releases.</li></ol><p>Overall, Helmfile is the best idea when it comes to deployment of the (micro)services on different environments using different parameters set.</p><p><strong>Helmfile features</strong></p><p>Separately, I would like to highlight the major features of Helmfile:</p><ul><li>12-factor style configurations.</li><li>Inline values.yaml.</li><li>CI/CD integration.</li><li>Environments synchronization.</li><li>Go templating.</li></ul><p><strong>CI/CD pipeline</strong></p><p>For the microservices development, we followed <strong>GitFlow</strong> approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*VjhYzy_toiPSYss5bIkigA.png" /><figcaption><em>GitFlow approach</em></figcaption></figure><p>In this case, we have at least four branches:</p><ul><li><strong>Master</strong> with release tags of the app.</li><li><strong>Release </strong>(not in the picture) for preparing releases</li><li><strong>Develop</strong> where changes to the code were being made.</li><li><strong>Feature</strong> — derivative to Develop. For each new product feature development, we were creating a separate branch.</li></ul><p>For Helmfile — so it automatically establishes deployment dependencies — we followed <strong>GitOps</strong> approach. It states that the infrastructure state represented in Infrastructure Git repo should represent actual state of running environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Ky-mPawwLgn9i0pCJoKtvw.png" /><figcaption><em>Sync workflow between cloud and Git</em></figcaption></figure><p>Briefly, the synchronization works as follows:</p><ol><li>Developer makes a change to a microservice and tests it.</li><li>A microservice undergoes testing on CI stage.</li><li>Jenkins compiles Helm chart and Docker image for this microservice, and tags a new version.</li><li>Jenkins updates the version of the corresponding microservice in a infrastructure Git repo, for a specified environment.</li><li>Automatically-triggered CD job applies the initial change in a corresponding environment.</li></ol><p>As you can see, no manual efforts are needed to make changes to environment. Also, to find out which versions of services are running in specific environment, you can simply check your Git repo.</p><p><strong>Local pipeline</strong></p><p>For microservices packaging, we used Minikube to simulate whole or at least part of real environment in Kubernetes. To orchestrate process locally we use Gradle, created a Gradle file and described necessary logic.</p><p>Gradle file runs automatically upon testing of some change. The script compiles Docker image from a Git repo, installs minikube, and starts Kubernetes cluster if missing. When a Kubernetes cluster is up and running, simply a new version of updated microservice is installed. Minikube here validates if local changes were executed successfully.</p><p>Additionally, in a repo we have Helmfile that describes only that part of the infrastructure that is needed to run tests for that specific microservice.</p><h3>Secrets management</h3><p>To build communication between the app and secrets, we used <strong>Kubernetes External Secret Controller</strong>. This solution is also installed from Helmfile.</p><p>Inside a Kubernetes cluster, Kubernetes External Secret Controller creates Custom Resource. This Custom Resource has API to fetch secrets from Secrets management (SSM component in Amazon) to the needed namespaces.</p><p>To summarize, Kubernetes External Secret Controller enables automatic movement of the secrets to the new namespaces upon their creation. As easy as that…</p><h3>Service mesh for secure communication</h3><p>Luckily, the days when <strong>Istio</strong> was a novelty have passed. This allows me to focus on the peculiarities of this solution’s adaptation to our needs. The major areas that service mesh solved in our case were:</p><ul><li>Secure communication between microservices.</li><li>gRPC-based communication between microservices.</li><li>gRPC load balancing.</li><li>Istio serves our case best due to its following features:</li><li>Provision of endpoints for user’s communication with the services via the product’s UI. Istio routs requests, made on the UI, to the app’s components.</li><li>Possibility to set up mutual TLS between the services. Traffic between the services is encrypted.</li><li>Balancing of the gRPC protocol between the services.</li></ul><h3>Logging and monitoring</h3><p>Of course, we set up logging and monitoring.</p><p><strong>Infrastructure logging</strong></p><p>To collect logs from the services (Amazon Aurora, Kafka, and others), which already are leveraged in the cloud, we employed <strong>Kibana</strong>. For convenient collection of the logs, we used <strong>AWS Lambda</strong>. Obviously, we configured it slightly to make each component get into Kibana’s needed index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nyw5Wz2-yJzwrYmt5GpAxg.png" /><figcaption><em>Infrastructure logging workflow</em></figcaption></figure><p><strong>Infrastructure monitoring</strong></p><p>All the metrics are reported to<strong> Amazon CloudWatch</strong>. Then <strong>Grafana</strong> fetches data from it and stores metrics in an appropriate format for the user’s convenient access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXyCnzT7Ce_b_bBtymcaLQ.png" /><figcaption><em>Infrastructure monitoring workflow</em></figcaption></figure><p><strong>Microservices logging</strong></p><p>To collect logs from the microservices, we deployed <strong>Fluentd</strong> to every Amazon EKS worker node. Inside every node, microservices write logs to a Fluentd daemon. To properly retranslate logs from the microservices, which are written in Java and C++, every daemon has a pre-configured parser. Eventually, Fluentd writes logs to appropriate Elasticsearch indexes that visualize data in Kibana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSphRXrYJcxeS5RgVv8J8g.png" /><figcaption><em>Microservices logging workflow</em></figcaption></figure><p><strong>Microservices monitoring</strong></p><p>We implemented microservices monitoring using Prometheus. It collects all metrics from the microservices and puts data into Grafana dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*pN8w3OFErgnrr0MQEOTTNw.png" /><figcaption><em>Microservices monitoring workflow</em></figcaption></figure><h3>Improvement areas</h3><p>Our current top-priority is setting up autoscaling, which is planned to be implemented using the two solutions:</p><ul><li><strong>Horizontal Pod Autoscaler</strong> for pods scaling in a Kubernetes cluster. Based on custom metrics of our services, it scales pods based on their load.</li><li><strong>Cluster Autoscaler</strong>, provided by Amazon, for worker nodes’ scaling.</li></ul><p>To sum up, migration to cloud is an extremely beneficial idea when it comes to effortless app updating and maintenance. Even for big enterprise, market offers abundance of solutions to make this process easy and risk-free. In our case, Helmfile was what provides flexibility to the team and automation for the product.</p><p>Feel free <a href="https://hubs.ly/H0BxlZx0">to check out open positions in our team</a> if such complex projects for leading enterprises is something you`re interested in.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9ce568d781" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/how-i-migrated-a-legacy-banking-application-to-amazon-eks-step-by-step-d9ce568d781">How i migrated a legacy banking application to Amazon EKS step-by-step</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Anthos: how it has changed DevOps]]></title>
            <link>https://medium.com/inside-the-tech-by-softserve/google-anthos-how-it-has-changed-devops-60d10968a645?source=rss----2f321e7469d3---4</link>
            <guid isPermaLink="false">https://medium.com/p/60d10968a645</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-anthos]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[devops-tool]]></category>
            <dc:creator><![CDATA[SoftServe]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 13:41:29 GMT</pubDate>
            <atom:updated>2020-11-18T14:40:26.184Z</atom:updated>
            <content:encoded><![CDATA[<p>Google Anthos is a game-changing platform for DevOps. Here`s why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*uje9nadTo5xHitisaFkH7g.png" /></figure><p>Max Kozinenko, Solutions Architect at SoftServe, embraced Google Anthos as an experiment right after it was released in spring 2019. Now, 1,5 years later, he`s Google Anthos Cluster Lead. Having implemented 5 projects together with the team of engineers from SoftServe Development Centers from Kharkiv, Kyiv and Lviv, he`s sure that this technology is a breakthrough.</p><p>Here are Max`s thoughts on that.</p><h3>Few words on what Anthos is and what pet peeves it eliminates for enterprises?</h3><p>Managing all existing infrastructure environments may be quite a nightmare for a company, especially enterprise, with no easy solution how to get from point A to point B. Often this path is long, expensive and impeding overall business growth.</p><p>Anthos is an ecosystem, that provides the solution.</p><p>It gives a full control over a company`s infrastructure. Its every object is treated as a Kubernetes object and managed using the GitOps workflow.</p><p>These are the crucial advantages, why companies will eventually go for Anthos:</p><ul><li>Allows managing different environments: AWS, GCP, on-premises,</li><li>Gives a full control over the whole infrastructure,</li><li>Enables changing environment configuration quicker and easier.</li></ul><p>But it`s not only a client who wins.</p><h3>Wins for engineers. How Anthos makes DevOps` lives easier?</h3><p>It`s a DevOps nightmare to containerize some complex setup, as it`s too time- and efforts-consuming</p><p>Here is how Migration for Anthos (M4A) can be integrated into current CI/CD pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*9eezfffPVM-0Blo-87ARng.png" /></figure><p>Much simpler, right?</p><p>The result won`t be the ideally looking containerized application like after replatforming, yet workflow will be containerized.</p><p>It comes in handy when you are expected to do a quick lift&amp;shift. This time saving allows to have ROI increased. Then you`ll able to invest time and money into optimization.</p><p>That`s the main use case where M4A is applicable.</p><p><strong>Workloads, currently supported by M4A</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*XMY-91Hg5HWTJTBnPfto-g.png" /></figure><p>As for Windows apps, keep in mind, that it produces Windows containers, that can`t be added to a Service Mesh. So, it`s very tricky to set up a Service Mesh between a standard Linux and Windows containers as you`ll spend a lot of time on designing how they should communicate via the Istio Ingress. On top of that, you won`t be able to utilize all the Service Mesh features in Windows Containers.</p><p>But for LAMP, WordPress, Java-based apps this approach gives a huge advantage.</p><p><strong>Opportunity to control all the workloads from a single place</strong></p><p>Here is how it looks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*S3_sZenuF4rqngvUF_nBXA.png" /></figure><p>This is a GKE cluster hub, that unites all your workloads and gives a single point of view on all the clusters, registered here. It`s not only about Google Cloud projects, but on-premise and AWS as well (Azure will be added in the immediate future).</p><p>Here you can:</p><ul><li>track everything (issues, warnings, alertings etc.) easily in a centralized way,</li><li>manage all the Anthos features like configuration management, Service Mesh,</li><li>use Ingress for Anthos or Multicluster services.</li></ul><p><strong>Load balancing</strong></p><p>Ingress for Anthos utilizes the Google Network backbone to do the intelligent drawing of the inbound traffic.</p><p>As you can see on the scheme below, Ingress utilizes Google Front-end. It gets a Google request from a specific area to a single virtual IP and then this request is routed to the closest Google Front-end and gets routed to the closest cluster in terms of latency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zVweFnhidmyRhj5LoA2H4Q.png" /></figure><p><strong>The most convenient integration with other cloud environments</strong></p><p>As I`ve mentioned above, Anthos is integrated with AWS. It means that we get here the GKE experience with the operability, manageability etc.</p><p>There are other similar tools like Kops, AKS or simply Kubernetes on AWS to eliminate the complexity aspects.</p><p>Here are 6 key factors why Anthos gives a better experience:</p><ul><li>Turn-key, production-grade, conformant Kubernetes with best-practice configuration,</li><li>Easy upgrade path to the latest Kubernetes releases that have been validated and supported by Google,</li><li>Get started quickly: use the GCP Cloud Console as the single pane of glass to create and view your workloads,</li><li>GitOps-style day 2 management for all your Anthos clusters with familiar tooling,</li><li>Integrated with ACM, ASM, Cloud Run, Cloud Logging &amp; Monitoring and marketplace solutions,</li><li>Available and reliable: highly available control plane and node pools with support for auto-scaling.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60d10968a645" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-the-tech-by-softserve/google-anthos-how-it-has-changed-devops-60d10968a645">Google Anthos: how it has changed DevOps</a> was originally published in <a href="https://medium.com/inside-the-tech-by-softserve">Inside the Tech by SoftServe</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>