<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anton Lytunenko on Medium]]></title>
        <description><![CDATA[Stories by Anton Lytunenko on Medium]]></description>
        <link>https://medium.com/@antonlit?source=rss-dbb15ab862b4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OaFz-u4ge8bU_TwQD2Wo-A.png</url>
            <title>Stories by Anton Lytunenko on Medium</title>
            <link>https://medium.com/@antonlit?source=rss-dbb15ab862b4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:50:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@antonlit/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From principles to tools in self-development]]></title>
            <link>https://medium.com/@antonlit/from-principles-to-tools-in-self-development-87ec0bd17c9a?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/87ec0bd17c9a</guid>
            <category><![CDATA[tools]]></category>
            <category><![CDATA[principles]]></category>
            <category><![CDATA[self-development]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Sun, 04 Feb 2024 09:01:10 GMT</pubDate>
            <atom:updated>2024-02-04T09:01:10.142Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MXHXXfrofm8xfXwJF_LeQ.png" /></figure><p>The market for self-development books and courses is blooming in the world of achievements and growth. Just google the “<a href="https://www.google.com/search?q=self-development+books+guide">self-development books guide</a>,” and you will find up to a hundred books on this topic. Add to this aggressive marketing combined with “recency bias” and your limited time to read, and your choice becomes even harder.</p><p>Which one should you pick? It depends on your goals and blind spots you would like to improve. It also depends on the way you learn. I prefer to start with principles and understand the territory&#39;s map before choosing the route and tools.</p><p>“<a href="https://www.amazon.com/Habits-Highly-Effective-People-Powerful/dp/0743269519">The 7 Habits of Highly Effective People</a>” became my map. Not only is it a map, but it is also a step-by-step guide to exploration that built my understanding of how different areas are connected. I read it much later than the other books mentioned further. It clarified previously gained knowledge and highlighted gaps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxs9e52FG9G9eS1mR-GY6w.png" /></figure><p>Another aspect of my personality is that it is easier to do something if I understand its reason and purpose. Reason and purpose are important because they give you more clarity on whether what you will do is aligned with your principles and goals. “7 Habits” emphasizes the importance of the Character Ethic based on fundamental principles of integrity, human dignity, fairness, honesty, etc. These fundamental principles, in their turn, drive personal productivity.</p><p>So, how do you act with integrity? Using a Personal Mission Statement, you must define what is important for you. “<a href="https://www.amazon.com/Start-Why-Leaders-Inspire-Everyone/dp/1591846447">Start with Why</a>” covers the topic of purpose in more detail.</p><p>When clarity on purpose and mission is there, you need to plan and execute. “7 Habits” brings the notion of the Magic Quadrant and states that the more time you spend in Quadrant II, the easier it gets in Quadrant I. Quadrant II is about planning. High-quality planning requires time to think deeply. “<a href="https://www.amazon.com/Deep-Work-Focused-Success-Distracted/dp/1455586692/">Deep Work</a>” is a great book that covers challenges related to deep thinking and concentration and suggests a set of tools to overcome those.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/0*VpcC5pvxPtaGHiwq.png" /><figcaption><a href="https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People#/media/File:7_habits_decision-making_matrix.png">https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People#/media/File:7_habits_decision-making_matrix.png</a></figcaption></figure><p>“Essentialism” briefly touches on all three habits from Privat Victory, but its core is establishing your focus, keeping it, and executing flawlessly.</p><p>Moving forward, “7 Habits” walks you from Private Victory through Independence to Public Victory. The key here is people and how you interact with them to build deep, Interdependent relationships.</p><p>“<a href="https://www.amazon.com/Nonviolent-Communication-Language-Life-Changing-Relationships/dp/189200528X">Nonviolent communication</a>” is a set of practices for deep and productive conversations. The Fifth Habit, “Seek first to understand, then to be understood,” is covered by this book in a comprehensive and easy-to-understand way.</p><p>“<a href="https://www.amazon.com/Never-Split-Difference-Negotiating-Depended/dp/0062407805">Never split the difference</a>” is about negotiation best practices. And not surprisingly, at its core, you will find “thinking win-win” and “synergizing” principles. It also touches on “Seek first to understand, then to be understood,” as without full understanding of your partner, win-win is not possible.</p><p>As with any worthy thing, neither Private nor Public Victory comes easy. It is not enough just to read books. It takes an iterative cycle of constant improvement and life-long learning. “Sharpen the saw” is about the constant balanced development of 6 other Habits and other aspects of your life. It is hard to find a universal book applicable to different professions, cultures, and life circumstances. For me, “<a href="https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555">Thinking, Fast and Slow</a>” is such a book. It sharpens your mind, provides insights into how your brain operates, and gives you more control over your decisions.</p><p>There are many books about tools and practices for building habits, enabling consistent execution, establishing deep communication, and negotiating. There are even more books specific to your hobbies and profession. To navigate through this variety, you need a map. So start with a map!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87ec0bd17c9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is ongoing transformation possible?]]></title>
            <link>https://medium.com/@antonlit/is-ongoing-transformation-possible-64cfa1374d3c?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/64cfa1374d3c</guid>
            <category><![CDATA[transformation]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Mon, 18 Sep 2023 15:00:14 GMT</pubDate>
            <atom:updated>2023-09-18T15:00:14.285Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dpetk3U8Pm13jBxfX8IGCg.png" /></figure><p>In his article “<a href="https://aws.amazon.com/blogs/enterprise-strategy/everyone-is-busy-who-has-time-to-transform/">Everyone is Busy: Who Has Time to Transform?</a>” Mark Schwartz (Enterprise Strategist at Amazon Web Services) emphasizes the importance of treating transformation not as a single monolith project but as a set of day-to-day activities to achieve high-level business goals. In this article, I want to reflect on some thoughts and ideas expressed by Mark.</p><h3>Summary of the original article</h3><p>Let me first provide a set of key points from the original article (please read it to grasp the thoughts from the origin):</p><blockquote>Fact: your employees are already busy.</blockquote><blockquote>Of course, major technical initiatives are involved (moving to the cloud, for example), but those initiatives are in the service of changing the way the company works. They are best driven by changes to everyday work; the two come together.</blockquote><blockquote>First, you must demonstrate commitment to the transformation.</blockquote><blockquote>Next, you must change incentives and role definitions. Since the goal of transformation is behavior change (new ways of working and thinking), you can best advance the transformation by incentivizing new behaviors and using them to drive requirements for technology change.</blockquote><blockquote>Employees must have a clear sense of where the transformation is headed. A vision of the transformed enterprise’s new state and how it will be different — not a detailed roadmap — is important… Remember, the goal is to have everyone working toward the transformational vision <em>in their everyday activities</em>, not as a side project.</blockquote><blockquote>Having a single-threaded leader is not at all a bad idea, but it must be combined with commitment across the organization’s leadership so that busy employees become part of the effort</blockquote><h3>Conditions of day-to-day transformation</h3><p>To make it even more concise, these are conditions for day-to-day transformations (as I understood them):</p><ol><li>Clear company vision and direction expressed as concrete business goals.</li><li>Commitment to an ongoing transformation from executives.</li><li>Employees must have enough time to drive and implement change.</li><li>IT is ready to respond to support transformation initiatives coming from employees.</li></ol><p>While these criteria seem smooth, some gaps make this set of conditions incomplete. Let&#39;s go through them one by one.</p><h4>Executives — employees gap</h4><p>Assuming a clear vision and full commitment to transformation, executives are not executing. Execution is done on the ground, which requires translating the vision into specific actions that will drive the company closer to the desired outcome.</p><p>Depending on the company&#39;s size, a gap between executives and employees might not even be an issue. Consider a small startup where founders execute themselves. This is an ideal case of vision transformed into action. But such companies usually are agile enough to react to challenges in days, if not hours. So, they are in a state of constant transformation anyway.</p><p>The other extreme are global companies with numerous middle management levels between executives and on-the-ground employees. Remember that everyone is busy delivering to the current goals, and most likely, employees will rely on the input from their direct management. Transforming a vision into tangible actions requires a chain of vision interpretations and translations running through middle management. There is a huge probability that the meaning and principles behind the vision will be lost.</p><blockquote>This gap can be addressed through the culture of transparency and feedback bypassing management levels. Such culture minimizes a vision interpretation error and increases the chances of successful transformation.</blockquote><h4>Employees — IT gap</h4><blockquote>you can best advance the transformation by incentivizing new behaviors and using them to drive requirements for technology change.</blockquote><p>I see how this can work in Digital Product companies and startups where IT is in the company&#39;s DNA and is a reliable partner to the business (or sometimes it <em>is</em> a business) in making customers happy.</p><p>But as with the previous example, there is another end of the spectrum: global companies or just “brick and mortar” types of businesses where IT is seen as a cost center. IT and employees most likely have totally different priorities: employees want to drive constant change, while IT (being a cost center) tries to minimize costs and risks.</p><blockquote>This type of risk requires a sit at the table for IT and transformation of the IT into a product delivery organization.</blockquote><p>One other aspect of Employee — IT gap in such cases is employee proficiency with technology and job safety. Continuous transformation at some point will require switching technology, which comes with the necessity to learn it. And remember, employees don’t have free time. This might result in a status quo that is hard to change, no matter the vision.</p><blockquote>One of the ways to address it is to have a culture of constant learning accompanied by a reduced employee load.</blockquote><p>Operating under 100% load will not lead you to the best performance anyway, according to the <br><a href="https://www.amazon.com/Goal-Process-Ongoing-Improvement/dp/0884271951">Eliyahu M. Goldratt and his book The Goal: A Process of Ongoing Improvement</a></p><h4>IT — Platform change gap</h4><p>The bottom-up approach of incremental improvements introduces another challenge on the IT level: should we go for substantial investment into new platforms vs. keep supporting and incrementally improving existing ones?</p><p>Again, depending on the type of company, this might be a different beast. Digital Product companies and startups most likely rely on the combination of custom build hosted in the Cloud + PaaS/SaaS services with an incline to the former. Such a setup is moderately easy to change and adapt to business needs as such changes are at the heart of the company business.</p><p>Their counterparts from the other side of the spectrum have tenth, if not hundreds, of applications in their portfolio, including custom build + off-the-shelf software hosted On-Prem + PaaS/SaaS with an incline to later. This number of components in companies of this size most likely resulted from numerous M&amp;As and never-ending reconciliation efforts. Again, IT in such companies tends to minimize risks and costs. Licenses are bought in bulk for multiple years ahead; huge ERPs are used with hundreds to thousands of integrations. Incremental ongoing transformation with such a setup is possible only to a certain extent. The choice between consolidating everything under the roof of the brand-new platform and not being able to support growing business needs is inevitable.</p><blockquote>The ability of IT to execute such transformations as a set of controllable tangible deliveries is a must.</blockquote><h4>Disruptive technologies as a driver of transformation</h4><p>In a model proposed by Mark, Employees should be a driver of the transformation, but what about breakthrough technologies like AI? Is there a place for IT to drive transformation? In my opinion, yes, under the condition that IT is not a cost center but one of the drivers of the business.</p><h3>Summary</h3><p>Mark Schwartz has written a <a href="https://aws.amazon.com/blogs/enterprise-strategy/author/schmark/">great series of articles</a> on transformation. Thank you, Mark. I’ve learned a lot!</p><p><strong>Is ongoing transformation at all possible? </strong>The answer is <strong><em>yes </em></strong>under a few conditions:</p><ol><li>Clear company vision and direction expressed as concrete business goals.</li><li>Commitment to an ongoing transformation from executives.</li><li><em>The company implements a culture of transparency and feedback, bypassing levels of middle management.</em></li><li>Employees must have enough time to drive and implement change.</li><li><em>The company implements a culture of constant learning</em></li><li>IT is ready to respond to support transformation initiatives coming from employees.</li><li><em>IT is a Product Delivery organization with a “sit a the table” and the ability to deliver and switch to new platforms when necessary.</em></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64cfa1374d3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is new in Microservices in 2023]]></title>
            <link>https://medium.com/@antonlit/what-is-new-in-microservices-in-2023-a2c913b798e?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2c913b798e</guid>
            <category><![CDATA[trends]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Fri, 21 Jul 2023 09:57:35 GMT</pubDate>
            <atom:updated>2023-07-21T09:57:35.199Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_GVbBCBnZqXRWaSr.jpg" /><figcaption>This slightly simplified version of the rotating snakes illusion uses color and brightness contrast to enhance the illusion. Credit: Akiyoshi</figcaption></figure><p>A lot is happening in software engineering, and Microservices are everywhere. But was there something new in the Microservice world recently?</p><p>The short answer is NO. But it sounds weird and counterintuitive at first glance. How can nothing new be happening for a topic with a lot of buzz?</p><h3>Industry trends</h3><p>Microservices have existed for over a decade, making them a well-established architectural style. A plethora of literature, including books, papers, and articles, is dedicated to exploring this concept. Over the years, microservices have gained recognition for their distinct advantages, drawbacks, and recommended practices, forming a widely known set of benefits, limitations, and recipes. Furthermore, many tools and libraries have been developed to support microservices implementation. Their adoption has become widespread across companies of all sizes. Given this extensive history, there are no groundbreaking developments concerning microservices themselves.</p><p>So let&#39;s look at industry trends to explore this statement further.</p><h4>Digital transformation trends</h4><p>The tech industry faced tough times with the threat of inflation and recession. Layoffs became common, leaving many employees out of work. Companies set targets to stay competitive, which often puts employees at risk. Workers feared being replaced by younger, tech-savvy employees. However, companies tried to balance their targets with employee needs by providing training programs and support. They also invested in new tools and technologies to engage with customers and build stronger relationships. Despite the challenges, companies were committed to weathering the storm and emerging stronger on the other side.</p><ul><li>As we look toward the future of work, it is clear that improving employment experiences will be a key trend. This includes offering remote, hybrid, and office work options to ensure that employees can work in a way that suits them best.</li><li>It’s also important to track employee satisfaction and invest in task automation to streamline processes and boost productivity.</li><li>Additionally, there is a growing need for more Machine Learning Projects (MLOps, ModelOps), IOT, Digital Twins, Sustainable Infra, and AIOps for monitoring complex infrastructures.</li><li>Multi-cloud and Hybrid Cloud solutions will also be important for scalability and flexibility.</li><li>Developing Transformation Leaders and Agile self-organizing standards will help to grow high-potential employees and foster self-organization.</li><li>Building trust with cybersecurity will be essential, as will Security Hyper Automation and AI in Security.</li></ul><p>With these trends in mind, companies will be better equipped to face the challenges of the future.</p><h4>State of DevOps 2022</h4><ul><li>Establishing autonomous, trusted, and supported teams can significantly improve performance in the workplace.</li><li>It’s also important to prioritize software supply chain security and implement the Supply Chain Levels for Secure Artifacts (SLSA) framework.</li><li>Teams that focus on establishing security practices are more likely to recommend their team to others, which can help reduce developer burnout.</li><li>Cloud and multi-cloud solutions are also crucial for improving organizational performance. Companies that use cloud platforms and have multi-cloud capabilities perform better overall, as they can inherit many capabilities and practices that can boost productivity and streamline processes.</li></ul><h4>Software Development Culture and Methods 2023</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1kD7a5mjrjje1HzC" /><figcaption>Software Development Culture and Methods 2023</figcaption></figure><p>I agree with the findings presented in the InfoQ infographic. The adoption of digital transformation is driving the need for refactoring legacy software and transitioning to a Microservice architecture. Cross-functional teams and a focus on Domain-Driven Design are crucial to support the implementation of Microservice architecture and ensure proper organizational setup. And all these aspects are in the Late Majority quarter.</p><h4>Summary of industry trends</h4><p>Companies aim to achieve <em>Flexibility and fast adaptation, Cost optimization and Efficiency, Tackle Complexity and ambiguity,</em> and <em>reduce Risks </em>through <em>Operational and delivery excellence</em> powered by <em>Self-organizing independent cross-functional teams</em> and Optimization of <em>SW Engineering Experience</em> through Self-service tools that help <em>Automate tedious tasks</em> and keep the focus on business complexity enabled by <em>Commoditization and unification of infrastructure</em> and <em>Commoditization and unification of the software supply chain.</em></p><h3>Technology trends</h3><h4>Cloud-native trends</h4><p>According to CNCF Annual Survey and State of Kubernetes report by Dynatrace Containers have become a new normal and Kubernetes keeps gaining momentum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*ZLhiERKkdDDpQ2aUpPI_-w.png" /><figcaption>Containers are new normal (by CNCF Annual Survey 2022)</figcaption></figure><p>In today’s landscape, the majority of organizations find themselves operating in hybrid or multi-cloud environments. Despite the availability of numerous managed alternatives, Kubernetes continues to experience significant growth in the cloud space. Many companies have adopted Kubernetes as a universal operating model, encompassing applications and supporting services such as CI/CD, security, and API management. It is a central framework for managing and orchestrating various aspects of their cloud infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tN1CQd-EbSUL_N-R" /><figcaption>Kubernetes is an operational model of the Cloud (by Dynatrace State of Kubernetes 2023 report)</figcaption></figure><h4>Emerging operations and delivery trends</h4><p><strong><em>Software Delivery Supply Chain Security</em></strong><em> </em>has already been highlighted earlier as one of the key trends in Digital Delivery. Sonatype State of the Software Supply Chain report explores it in detail and confirms that dependencies carry a security threat that can be mitigated by <em>mature software supply chain management</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NnSo54HJUmZ_GlZXVhbAjQ.png" /><figcaption>Software supply chain security is crucial for modern SW delivery organizations (by SonaType State of Supply Chain report)</figcaption></figure><p>Additionally, the <a href="https://slsa.dev/"><em>Supply Chain Levels for Secure Artifacts (SLSA) </em></a>framework gains traction and is recommended by Google as a framework to mitigate SSC security risks. It proposes the following principles:</p><ul><li>Trust platforms, verify artifacts</li><li>Trust code, not individuals</li><li>Prefer attestations over inferences</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*skFvQGSHsuoPCoXv" /><figcaption>SLSA Framework</figcaption></figure><p><a href="https://www.infoq.com/articles/observability-driven-development/"><strong><em>Observability-driven development (ODD)</em></strong></a> uses data and tools to observe a system’s state and behavior throughout development. It encourages developers to prioritize application availability and uptime by implementing detailed logging, metrics, and KPIs. ODD improves overall observability, enabling the detection of predictable and unpredictable failures. By continuously monitoring system behavior, it establishes a baseline for normal operation.</p><p><a href="https://www.brighttalk.com/webcast/17791/582541"><strong><em>AIOps</em></strong></a><strong><em> </em></strong>supports <em>ODD</em> by combining big data and machine learning to automate IT operations processes, including event correlation, anomaly detection, and causality determination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*UGPDBJc_QZDLXfOmdtJXsA.png" /><figcaption>Benefits of AIOps (State of AIOps Report 2023 by OpsRamp)</figcaption></figure><p><a href="https://platformengineering.org/talks-library/the-magic-of-platforms"><strong><em>Platforms</em></strong></a><strong><em> and </em></strong><a href="https://teamtopologies.com/key-concepts"><strong><em>Platform Teams</em></strong></a> are essential pillars of unification and commoditization within infrastructure and software supply chains. They serve two primary goals: optimizing infrastructure costs by consolidating and unifying application workloads and reducing complexity for development teams by taking care of infrastructure and deployment aspects. By offloading these responsibilities, development teams can focus more on crucial business aspects, leading to increased efficiency and productivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*l7-W6YdBjuHwTdad-yzYng.png" /><figcaption>Four fundamental topologies by Team Topologies</figcaption></figure><p>While Kubernetes adoption is growing, there is an alternative: <strong><em>Serverless</em></strong>. Companies afraid of Cloud lock-in rely on cloud-agnostic technologies like Kubernetes, while others utilize the cloud&#39;s power to the full extent. <a href="https://serverlessland.com/content/guides/refactoring-serverless/introduction"><strong><em>Refactoring to Serverless</em></strong></a> is an implementation tactic that supports strategic bets on Serverless.</p><blockquote>You can improve the design of your serverless application by replacing application code with automation code, while using the same programming language.</blockquote><h4>Bridging the Data Gap with Data Mesh</h4><blockquote>A data mesh architecture is a decentralized approach that enables domain teams to perform cross-domain data analysis on their own. At its core is the domain with its responsible team and its operational and analytical data.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6sTha7mfOL3i2rR-" /><figcaption>What is Data Mesh?</figcaption></figure><p>Here again, we have a concept of further decentralization of implementation and operational concerns based on a <strong><em>commoditized and unified platform.</em></strong></p><h4>Making it all possible with Architecture</h4><p>Let&#39;s have an industry trend summary once again.</p><blockquote>Companies aim to achieve <em>Flexibility and fast adaptation, Cost optimization and Efficiency, Tackle Complexity and ambiguity,</em> and <em>reduce Risks </em>through <em>Operational and delivery excellence</em> powered by <strong><em>Self-organizing independent cross-functional teams</em></strong> and Optimization of <em>SW Engineering Experience</em> through Self-service tools that help <em>Automate tedious tasks</em> and keep the focus on business complexity enabled by <strong><em>Commoditization and unification of infrastructure</em> and <em>Commoditization and unification of the software supply chain.</em></strong></blockquote><p>How can you keep teams <strong>independent and self-organized</strong> while making them agree on a <strong>unified</strong> stack that should be <strong>commoditized</strong> across all teams? Additionally, Updfront design is a MUST to embrace technologies like ODD and AIOps as they require well-defined NFRs to translate to SLI, SLO, and SLAs. Those SLIs should then be supported by metrics within the application code and a well-structured Logging approach.</p><p><strong><em>Architecture</em></strong> is the answer. And trends in this field follow concepts of agility, adaptation, and focus on value delivery.</p><p><a href="https://emtemp.gcom.cloud/ngw/globalassets/en/doc/documents/enterprise-architecture-function-primer-for-2023.pdf"><em>Enterprise Architecture trends</em></a> that bring associations with Ivory Tower architects do not stay aside and adapt to modern challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*JrPfi2CtXvniYbglgQzlyg.png" /><figcaption>Enterprise Architecture Trends</figcaption></figure><p>Enterprise architecture trends continue on Solution levels with <a href="https://www.youtube.com/watch?v=9Q7GANXn02k"><strong>Minimum Viable Architecture</strong></a><strong> and Domain-driven design.</strong></p><h3>So what is new in Microservices?</h3><p>The short answer is still <strong>NOTHING.</strong> With a few “marginal” exceptions and new buzzwords like AIOps and ODD, all the trending concepts have been there for years. Even industry trends do not carry anything unique and new.</p><p>Not so short answer is <strong>A LOT</strong> is happening around Microservices. What we see is that concepts of decentralized enterprise are gaining maturity. Naturally, Microservices transitioned from the innovation phase to the active adoption and commoditization phase. This shift would not be possible without corresponding infrastructure, software supply chains, and architecture shifts.</p><h3>References</h3><ol><li><a href="https://www.techtarget.com/searchcio/feature/Top-5-digital-transformation-trends-of-2021">https://www.techtarget.com/searchcio/feature/Top-5-digital-transformation-trends-of-2021</a></li><li><a href="https://business.adobe.com/blog/basics/digital-transformation-trends">https://business.adobe.com/blog/basics/digital-transformation-trends</a></li><li><a href="https://services.google.com/fh/files/misc/final_2022_state_of_devops_report.pdf">https://services.google.com/fh/files/misc/final_2022_state_of_devops_report.pdf</a></li><li><a href="https://www.infoq.com/articles/culture-trends-2023/">https://www.infoq.com/articles/culture-trends-2023/</a></li><li><a href="https://www.cncf.io/reports/cncf-annual-survey-2022/">https://www.cncf.io/reports/cncf-annual-survey-2022/</a></li><li><a href="https://www.dynatrace.com/news/blog/kubernetes-in-the-wild-2023/">https://www.dynatrace.com/news/blog/kubernetes-in-the-wild-2023/</a></li><li><a href="https://www.sonatype.com/state-of-the-software-supply-chain/introduction">https://www.sonatype.com/state-of-the-software-supply-chain/introduction</a></li><li><a href="https://slsa.dev/spec/v1.0/principles">https://slsa.dev/spec/v1.0/principles</a></li><li><a href="https://www.infoq.com/articles/observability-driven-development/">https://www.infoq.com/articles/observability-driven-development/</a></li><li><a href="https://www.brighttalk.com/webcast/17791/582541">https://www.brighttalk.com/webcast/17791/582541</a></li><li><a href="https://sciencelogic.com/blog/2023-trends-in-aiops-observability-and-itops">https://sciencelogic.com/blog/2023-trends-in-aiops-observability-and-itops</a></li><li><a href="https://platformengineering.org/talks-library/the-magic-of-platforms">https://platformengineering.org/talks-library/the-magic-of-platforms</a></li><li><a href="https://teamtopologies.com/key-concepts">https://teamtopologies.com/key-concepts</a></li><li><a href="https://serverlessland.com/content/guides/refactoring-serverless/introduction">https://serverlessland.com/content/guides/refactoring-serverless/introduction</a></li><li><a href="https://www.datamesh-architecture.com/">https://www.datamesh-architecture.com/</a></li><li><a href="https://emtemp.gcom.cloud/ngw/globalassets/en/doc/documents/enterprise-architecture-function-primer-for-2023.pdf">https://emtemp.gcom.cloud/ngw/globalassets/en/doc/documents/enterprise-architecture-function-primer-for-2023.pdf</a></li><li>Minimum viable architecture: <a href="https://www.youtube.com/watch?v=9Q7GANXn02k">https://www.youtube.com/watch?v=9Q7GANXn02k</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2c913b798e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Being effective as a part of an outsourcing team]]></title>
            <link>https://medium.com/@antonlit/being-effective-as-a-part-of-an-outsourcing-team-f8292fcbc699?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8292fcbc699</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[self-development]]></category>
            <category><![CDATA[effectiveness]]></category>
            <category><![CDATA[outsourcing]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Tue, 09 Aug 2022 14:05:47 GMT</pubDate>
            <atom:updated>2022-08-09T14:05:47.903Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kzcImvs5pqGex6QVZHF2LA.png" /><figcaption>Picture by Brooke Cagle</figcaption></figure><p>Digital transformation is a major buzzword in the last five years at least. Software development outsourcing is one of the forces that fuel digital transformations worldwide. This article is based on my own experience and focuses on the software development outsourcing team member: the reality she works in, the challenges she comes across, and ways of transforming challenging reality into enjoyable work.</p><h3>Outsourcing team members&#39; reality</h3><p>Meet Mary! She is a software engineer in an established Software Development Service Provider DevBeast Inc. Mary has experience of 5 years in the industry and about 2 years with her current employer. She has been assigned to a customer account (Pear inc.) and is now about to start working on one of the projects with her new team.</p><h4>Multiple domains of knowledge</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2hMKQq04SjcO29lfCWXBjQ.png" /><figcaption>Multiple domains of knowledge</figcaption></figure><p>Before joining her current employer, Mary has grown her expertise in multiple different ways: learning from the industry and community (books, blogs, podcasts, etc.); learning by doing actual tasks, collaborating with her teammates, and so on; learning from knowledge bases of her previous employers: processes, practices, policies.</p><p>Upon joining DevBeast, Mary had to pass a long onboarding process. She extended her knowledge with information about how DevBeast operates, what the org structure is, what the career ladder is and what is required to move up this ladder. She also gained additional knowledge about the company&#39;s recommended ways to approach certain industry-specific challenges.</p><p>By joining the Pear account Mary learns that there is a Pear-specific way of doing things. Some areas are specified explicitly like security policies and preferred tech stack, some are defined only partially, like developer onboarding, and some are not explicitly defined at all, like responsibility boundaries.</p><h4>Multiple governing structures</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QxnPQfctCLFtEjITTvBftQ.png" /><figcaption>Multiple Governing Structures in Outsourcing</figcaption></figure><p>Mary is an employee of DevBeat. This means that she is probably a part of a certain department inside DevBeast. Additionally, Mary is a part of the Account that has an account manager assigned. And finally, Mary works on a customer project assignment with management on the customer side. Each manager and the corresponding department has its goals and KPIs.</p><p>And let&#39;s not forget that as an employee seeking growth Mary has her development plan with objectives to achieve.</p><h3>Major challenges</h3><p>Although the team tasks look challenging and the technology stack is promising, the assignment experience is far from smooth.</p><h4>Industry vs Company vs Customer practices</h4><p>Mary is puzzled about what practices and approaches to apply. On the one hand, the Pear has some practices defined. On the other hand, DevBeast has its own recommended way.</p><p>The answer might seem obvious: customer practices always override the practices of the vendor. But what if these are far from optimal for this specific problem? What if the industry moved forward since these practices were established, and DevBeast, a company with wide expertise in the domain, can propose a better way? What if the customer company moved forward since the time these practices were applied? What if Mary has her expertise in a similar domain and problem area and knows how certain solution approaches work in practice?</p><blockquote>How can Mary apply her expertise in the best way to the problem at hand?</blockquote><h4>Effective communications and decision making</h4><p>Communication is the key to solving the previous challenge. Normally Mary would have to raise her questions within the team, get to a consensus and move forward.</p><p>Let&#39;s assume that the team agreed with Mary’s concerns and that current Pear practices are suboptimal and will be rather an obstacle for the team on the path to achieving its goal. What should happen next?</p><blockquote>Are there mechanisms in place for the team to provide feedback to a higher level?</blockquote><h4>Conflicting responsibilities</h4><p>In an ideal world Pear and DevBeast do a great job and set up a deal that does not impact Mary and her team members. This deal sets the stage for productive relationships between Pear and DevBeast and Mary can just apply her expertise to the full extent.</p><p>But this is not always the case. Depending on the setup and, there are “us” and “them” whether you are looking from the Pear point of view or the DevBeast point of view. Multiple governing structures do not add to the transparency. What goals should Mary focus on? She is an employee of the DevBeast company so probably DevBeast goals (maximum profit from the project) are the priority. What if this goal conflicts with Pear&#39;s project goals (high quality, fast time to market, etc.)? For example, should Mary spend time writing those integration tests if the release goal can be achieved without them, they require more time from Mary and the team than projected by the contract, but tests that will improve the quality of the delivery and are very useful for the Pear long term. Should Mary spend time coaching Pear employees?</p><h3>How to be effective and enjoy what you do</h3><p>What makes an engineer happy at work? I assume the ability to apply skills to the full extent is one of the most important aspects. This means you need to be heard, trusted, and supported by the team.</p><h4>You are a “customer employee” mindset</h4><p>This is the mindset I have by default. Why is it helpful? Well, imagine that you are a fresh employee in a Pear company. What would you do? You would do your best to understand what the company does, its org structure, and its goals. You would meet your colleagues get to know each other more and understand their drivers and motivations. This attitude helps break the ice between you and customer employees, you naturally start focusing on customer goals which naturally leads to a much better application of your skills.</p><h4>Build trust</h4><p>Trust is really the biggest blocker for any newbie. Nobody knows you, nobody understands your intents, nobody knows your skill level yet.</p><p>Adopting the “customer employee” mindset helps you with people&#39;s side of the trust, but you have to prove your skill first to overcome skill-based trust issues.</p><blockquote>So the best thing you can do is deliver not changing or questioning anything.</blockquote><h4>Develop a good understanding of the customer context</h4><p>The previous suggestion might sound counterintuitive, but questioning practices that have been before you without even trying those is even more questionable.</p><p>Delivering using established practices will help you better understand the context those practices were created in. Additionally, you will have time to gather more information.</p><p>Understanding the context you are working in, assessing a <a href="https://medium.com/@antonlit/misterios-big-picture-vision-dae2388d3922">Bigger Picture</a> is a must. As a rule of thumb, I usually try to gather context for my team level + 2 levels up, i.e. my team manager context, peers, and goals + my manager’s manager context, peers, and goals. The same rule applies to the IT architecture contexts.</p><h4>Communicate and do your best</h4><p>The actions above are a great basis that helps you communicate your ideas based on facts (context) to the people who trust you (“customer employee” mindset + ability to deliver) helping your team deliver against goals (“customer employee” mindset +context)</p><h3>Notes</h3><ol><li>The challenges described in this article cover challenges specific to an employee of the Software Development Services provider.</li><li>Culture, language, and timezone differences only increase the impact of mentioned challenges and the suggested practices&#39; importance.</li><li>Being a part of the outsourcing team is a challenge and a great source of self-development opportunities.</li><li>I will focus on the customer (Pear) perspective in the next article.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8292fcbc699" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Misterios “Big picture vision.”]]></title>
            <link>https://medium.com/@antonlit/misterios-big-picture-vision-dae2388d3922?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/dae2388d3922</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Fri, 08 Jul 2022 08:53:37 GMT</pubDate>
            <atom:updated>2023-06-02T13:04:18.069Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mysterious “Big picture vision.”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNmVMK1oEulT8IGYgFxQIA.jpeg" /><figcaption>The <a href="https://en.wikipedia.org/wiki/Mandelbrot_set">Mandelbrot set</a>: its boundary is a fractal curve with <a href="https://en.wikipedia.org/wiki/Hausdorff_dimension">Hausdorff dimension</a> 2 (Wikipedia: <a href="https://en.wikipedia.org/wiki/Fractal">https://en.wikipedia.org/wiki/Fractal</a>)</figcaption></figure><h3>You should improve your “Big picture vision” skill…</h3><p>… is trendy recommendation engineering managers give to their team members as a part of their performance reviews. At first glance, there is nothing wrong with it. And it really raises a lot of questions, as it feels like we instinctively understand what it means to improve a “Big picture vision.” But it is hard to find less actionable feedback than this one. So as a quick check, try to quickly list 3 specific action items that you can use to improve this skill. (You can add those to comments).</p><p>I gave these kinds of recommendations myself (Yes, I’m guilty, your Honor). However, those recommendations really meant things like understanding the system/component architecture and how your job fits in. Is this it? Can we replace “Big picture vision” with “System/component architecture vision” and close the case?</p><p>I believe these are different things, and the BPV term is much broader than that.</p><h3>The intent behind the BPV</h3><p>Why do engineering managers suggest improving the BPV? My bet is that they want you to improve your <strong><em>problem-solving skills</em></strong>:</p><ol><li><em>Information gathering</em></li></ol><ul><li>Technical system analysis</li><li>Process analysis</li><li>“People” skills</li><li>Analytical skills</li></ul><p><em>2. Decision support</em></p><ul><li>Candidate solution generation techniques</li><li>Solution verification techniques</li></ul><p><em>3. Solution approval</em></p><ul><li>Expectations management</li><li>Written and verbal communication skills</li><li>Presentation skills</li></ul><p>So let&#39;s take a look at a high-level problem-solving process.</p><p><strong>Inputs</strong>: Problem statement, Constraints, Stakeholders</p><p><strong>Goal</strong>: Find a solution to the Problem Statement that satisfies Constraints and can be approved by Stakeholders.</p><p>The ability to see the BPV intuitively can improve the ability to find a solution with given constraints. For example, understanding how the component I develop fits into the overall system architecture helps the developer make better decisions during the implementation phase.</p><p>But what about Stakeholders? Should I care? What Stakeholders do middle backend engineers have?</p><h3>Stakeholders of the software engineers</h3><p>Problem-solving for software engineers is usually associated with the engineer&#39;s ability to design and implement changes to the existing system. This means that Stakeholders are people who review the design and conduct a code review.</p><p>Designing and implementing changes are important but these are not the only activities that are a part of the problem-solving for software engineers. What about quality assurance, deployment, transition to support, communication with the customer, etc.?</p><p>People who do these activities are your stakeholders, and understanding System architecture will not ensure that they will approve your solution.</p><blockquote>In my opinion, understanding people and processes, and how people interact within those processes is at least as important as understanding a system architecture.</blockquote><p>At the end of the day, who cares that your component is designed according to the best practices of software engineering if it was not properly communicated to the right people and is not approved late in the release pipeline?</p><h3>Is it a Big or Bigger picture vision?</h3><p>Problem-solving skill assumes some context that includes a level of decisions that can be proposed as a part of the solution. So the Big picture is always scoped by 1 at rear cases 2 levels above the level of the operation. So the Big picture is relative to your current level. And it means that it is much better to call it the Bigger picture.</p><h3>Summary</h3><p>BPV is a bucket of skills and improving it means improving each skill included in that bucket. So whenever you give/hear the recommendation to improve the BPV please be more precise by specifying a specific skill(s) and a context (level of operation).</p><p>I recommend <a href="https://www.opengroup.org/dpbok">The Digital Practitioner Body of Knowledge™ Standard</a> as a comprehensive guide for contexts (levels of operations) and skills required for each level.</p><p><em>I accidentally found this article written a long time ago in my drafts. So publishing it as is.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae2388d3922" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Quick Way to Refine Raw Data Into Dollars]]></title>
            <link>https://medium.com/greenm/how-to-rapidly-increase-the-value-of-you-business-using-data-you-already-have-1a4534b4bc0d?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a4534b4bc0d</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[business-analysis]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Fri, 09 Aug 2019 15:29:51 GMT</pubDate>
            <atom:updated>2019-08-29T10:19:34.664Z</atom:updated>
            <content:encoded><![CDATA[<h4>One of The Most Effective Approach to Grow Business Fast Using Non-structured Massive of System Logs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2XsaeX3pM6rgWxfas5g2yA.jpeg" /></figure><blockquote>Imagine a Company where software is not their primary business, but rather a necessity to support and tie together communication devices, such as wireless voice-activated badges and smartphones.</blockquote><blockquote>Software binds these devices into a powerful network, enabling a wide range of use cases. This includes calling into a group with first-to-answer policy, or a calling to a job role instead of a specific person, such that employees currently on duty can be reached, or broadcast voice messages.</blockquote><blockquote>The Company deploys this System on the customer’s premises on the local network. The system generates a massive amount of logs that can be delivered to the Company data center for further analysis in the event of any issues.</blockquote><p>Logs are delivered as Zip packages to the Company FTP server. The ZIP package contains various log files, including log files for multiple system-related services, operational system logs, database dump, and system configurations. While having a weak structure, and no single format even for system-related components, system logs still contain useful data.</p><p>For example, they provide communication patterns between customer employees, level of system utilization, device battery levels, and how frequently they require re-charges, and much more. Additionally, the Company stores backups of logs, further enabling retrospective analysis.</p><h3>What options does the Company need to have to unleash the power of logs?</h3><h4><strong>Implement structured logging and build analysis on top</strong></h4><p>First of all, <strong>instrumenting code with structured logging</strong> is the right way to go. Having logs in a single format with at least partially common schemas such as JSON or CSV enables a wide range of software such as ELK stack and Splunk to do the first-level analysis and provide simple insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8heyhJbbe4ZLtnOG5mQROw.jpeg" /></figure><p>However, let’s take a closer look at this approach. In our case, the Company already owns and supports the existing System. There are plans to deliver new features to end-users, to support new modern device models and to extend the System with faster communication protocols.</p><p>At the same time, it would require dedicating some precious time from the Team actually developing the System to outfit the existing code with proper logging. What features should be excluded from the scope then? Should the Company extend the schedule and change the delivery date?</p><p>But, even in the case of resources for log instrumentation being found, what would the Company get? <strong>Structured logs are not the end product</strong>; tools on the market are not cheap and require a broad set of skills to make sense of logs data.</p><p>Additionally, to enable structured logging for a specific customer, the Company has to install this latest version on the customer premises, which limits the possibilities for good customer coverage. The opportunity to do retroactive analysis on backups is lost as well.</p><h4><strong>Collect existing logs and build analytics on top</strong></h4><p>What if the Company takes another approach and tries to make sense of existing logs? Well, this approach takes the pressure off the primary product team as no changes to the System are required, and a dedicated team can develop it.</p><p>No updates to the systems on customer premises are currently required as well, and we keep the opportunity to do retroactive analysis.</p><p>This option is great to keep fast time-to-market and also gracefully migrate to better logging in the primary System to support even more insights.</p><p>Let’s review what it takes to follow this path.</p><h3><strong>Log-based analytics with weakly structured logs</strong></h3><h4>Log analysis and pattern discovery</h4><p>So, we have a massive amount of ZIP packages containing various log files. How do we discover useful patterns in them? How do we identify the meaning of each pattern?</p><p>An excellent way to start with a massive task, is to take a bite out of it; in other words, we can narrow the scope. This is not a set of random log files, and we know the system they belong to, and we know it does communication.</p><p>So let us pick some useful event that is present in the log files and try to identify a potential set of patterns covering it. For example, we can take a call from one device to another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Pv9ugKdpUKTqXbAgMUxdw.jpeg" /></figure><p>Another way to narrow the scope is to choose a specific version of the system to work with and a particular component of the system. Moreover, we can start with one particular set of log packages big enough to cover a variety of cases we need, but small enough to be parsed on one machine using a console app quickly.</p><p>The system source code can be complementary to the log files. We can decide to start with identifying the code paths leading to the call event and finding lines of code with logging. Another way to identify patterns, is to use the system and monitor what log lines it generates during specific use cases.</p><p>In parallel with log discovery, the team can build a log parsing POC implementing found log patterns. At this stage, a pluggable approach can be utilized to quickly extend a POC with newly found patterns.</p><h4>Data analysis challenges</h4><p>Now that we have identified a basic set of patterns, we can start extracting events out of them, and combine these events to build facts and insights.</p><p><strong>Extract events</strong></p><p>It is quite trivial to generate events where an event is one to one with a logline. However, it might be challenging to build events based on multi-line patterns, as it is not a rare case when services work in a multi-threaded mode. For logging, this means that log lines from other threads can be mixed into the multi-line pattern you are seeking.</p><p>We can <strong>solve this challenge</strong> using the following logic:</p><ul><li>If log patterns belong to some entity with identity such as an id or a name available as a part of the logline, we can<strong> split log lines into groups</strong> by this identity and then apply event extraction logic. In the case of our communication system, a device id might be a part of the logline. Sometimes, if we are lucky, the thread id might be a part of the log as well.</li><li>If case logs can’t be split into groups related to a particular entity, it might be a good idea to <strong>build a stateful log parser</strong>. By stateful, I mean that a log parser memorizes incomplete patterns (for example where just first line of the pattern is found, but not the finishing one) and keeps seeking until the pattern is complete. This approach might be especially useful if a pattern’s meaning depends on previously registered events. In this case, not only incomplete events should be kept as a part of the state, but also the last event, or a series of latest events.</li></ul><p><strong>As a result</strong> of event extraction, we get a set of events like “call initiated,” “call started,” “call finished,” “call failed,“ and so on. And most likely, with structured logging you would quickly get this set of events, but not more. Can we build some analysis on top of this data? Yes, we can. However, it might be too low level and won’t bring useful insights all at once.</p><p><strong>Advanced analysis</strong></p><p>To build a deeper insight, we have to <strong>combine </strong>these <strong>low-level events </strong>into facts. For example, we can combine “call initiated,” “call started,” and “call finished” events into the fact “Call” with attributes like call initiated time, call picked up time, call finished time, caller device, and callee device.</p><p>This information is much more useful for the analyst as it <strong>opens up</strong> a higher <strong>space </strong>for <strong>higher-level analysis</strong>. We can identify a period with a pick load, or identify communication patterns and critical paths related to a specific business operation on the customer side.</p><p><strong>Utilities</strong></p><p>T here are other challenges not directly related to the analysis, but profoundly impacting its quality. One of those is <strong>log quality</strong>. As you remember, the Company collects logs on it’s FTP server. So there is no guarantee that the log package is not broken, or has not been sent twice.</p><p>Additionally, some of the log files inside the package might be still running log files, and they will be present in the next package as well. In order to ensure the best possible quality of logs coming for analysis, we have to build a log registry containing not only the package itself but also information about its content such as file name, file size, timestamp and so on.</p><p>The log processing component can then use this registry to update the status of processed files and exclude already processed files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zG6imbKbbkGN4oY_aDJ_RQ.jpeg" /></figure><p>As mentioned in the beginning, the Company deploys the system on customer premises, which means that at any particular moment in time, there are <strong>multiple </strong>different <strong>versions </strong>which <strong>have to be supported</strong>.</p><p>For the analytics system, this means that we have to support different versions of patterns for the same event or even the absence of specific events. This challenge can be resolved using the pluggable approach. We can implement each pattern, fact, and insight as plugins with the system versions they belong to as a part of their metadata.</p><p>Now <strong>we can identify the system version</strong> at the beginning of processing, or even as a part of the log registry and use only compatible plugins.</p><p>The Pluggable approach is also key to further system logging improvements. Now the primary team can steadily improve logging inside the system, and on the analytics system side, new plugins can be added to support the new version.</p><h3>Technical solution</h3><p>Let’s see how the solution described above can look like if hosted in the Company data center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*firw68q076jYhmWbf04kIw.png" /></figure><p>On a <strong>high-level solution</strong>, it consists of <strong>4 major steps</strong>:</p><p>1. The Package Registration process copies a ZIP package from File Server to the Application Server.</p><p>2. The Package Registration process validates the ZIP package integrity, reads its contents and registers the ZIP package, its validity status and contents info into Meta-data storage. If a package is valid, it is copied to the HDFS of the Processing cluster.</p><p>3. The Spark job on the Processing cluster consists of 2 stages:</p><p>a. ZIP package and its contents are read, and Events are extracted by the LogParser package and a corresponding set of plugins.</p><p>b. FacctProcessor transforms and combines Events into facts and insights, and stores the result in a local HDFS.</p><p>4. Facts are copied from the Processing cluster to the Data Store cluster and are registered in Hive Metastore. Next, Facts become available for querying by Cloudera Impala.</p><h3>Summary</h3><p>With the approach described above, you can get to market with a log-based BI solution as quickly as possible. This means that the product team, sales, and management can gather valuable facts and insights about the System usage patterns and potential for further upselling or about clients at risk.</p><p>The primary team keeps its focus on the System and end-user features delivery and gains time to plan and implement structured logging inside the System properly.</p><p>Would you like to discuss this case further? Let us know how <a href="https://is.gd/T6XylN">GreenM </a>can help you — our high-level dev team would be more than pleased to schedule a free online meeting to devise the best solution with you.</p><p><em>More articles:</em></p><p><a href="https://medium.com/greenm/data-lake-as-an-efficient-response-to-complexity-a2bf54eaf4d2?source=collection_home---4------0-----------------------">Data Lake as an Efficient Response to Complexity</a></p><p><a href="https://medium.com/greenm/scalable-data-pipeline-f5d3c8f7a6d9">Scalable Data Pipeline. Be Ready for Big Changes</a></p><p>For more stories follow us on <a href="https://medium.com/greenm"><strong>Medium</strong></a><strong> </strong>or visit our site👇.</p><figure><a href="https://greenm.io/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ul-21Yvs1BpyGa49.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a4534b4bc0d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/how-to-rapidly-increase-the-value-of-you-business-using-data-you-already-have-1a4534b4bc0d">A Quick Way to Refine Raw Data Into Dollars</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Lake as an Efficient Response to Complexity]]></title>
            <link>https://medium.com/greenm/data-lake-as-an-efficient-response-to-complexity-a2bf54eaf4d2?source=rss-dbb15ab862b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2bf54eaf4d2</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Anton Lytunenko]]></dc:creator>
            <pubDate>Mon, 01 Jul 2019 17:15:21 GMT</pubDate>
            <atom:updated>2019-11-07T19:37:01.940Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y0fZ_apS38jZnWZ-zd2Tqw.jpeg" /></figure><blockquote>Do you have a complex data storage setup? Are you overwhelmed with the amount of ETL components your company supports? Is your company feature-focused at the moment and have minimum time and resources to develop a platform? If so, you will find this piece very useful. We will show why data lake is a natural answer to the system complexity and is an extremely great fit for quickly growing companies.</blockquote><h3>Why is a data lake a good response to complexity?</h3><p>A traditional way to integrate multiple, diverse data sources is to build a data warehouse. <strong>A data warehouse is a great thing</strong> to have, as it serves as a single source of truth and, if it’s companywide, it integrates multiple different business domains into a single, clean, normalized datastore. Having that makes it very easy <strong>to build</strong> multiple data marts, each providing very specific business insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*721b9UBRscGlD8YTfsoYMw.png" /></figure><p>If you already have a data warehouse and are doing a great job of keeping it up to date with all the changes in your business, my sincere congratulations. You have an ideal system.</p><p>But unfortunately, <strong>the traditional scenario with the data warehouse is not all that widespread among companies</strong>. The main reasons for this are the following:</p><ul><li>Data warehouse works great when it matches up with the business domain logic and relationships inside it. As such, <strong>it either requires</strong> a lot of upfront <strong>design </strong>or a lot of <strong>continuous work </strong>to keep it up to date with all the quickly changing business demands. It first means that <strong>you can’t move your product quickly</strong> to market. Later it means that you should have a dedicated resource <strong>supporting a data warehouse</strong>, instead of building features valuable for your business.</li><li>Traditional data warehouse <strong>software </strong>like MS SQL Server or Oracle <strong>is quite expensive</strong>, both in terms of the license and in terms of hardware resources required to run it.</li></ul><p>Let’s consider the following <strong>real-world example</strong>. The Company has a great business idea and decides <strong>to build a ProductA in-house</strong>. Everything goes according to plan; customers are happy with the new product and request more features. The ProductA grows:</p><ul><li>the Company builds new components to support new features</li><li>some back-end processes are introduced with their own meta-data storage</li><li>eventually, reporting is introduced</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*RGV3rvx4PYT-DhJ-U0vxLA.png" /></figure><p>As soon as the Company is at a good spot on the market, it looks for expansion opportunities and <strong>acquires a ProductB</strong> to cover complementing business domain. And of course, ProductB has its own architecture including various data storage and reporting. So now the technical landscape looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ztWholYEU3vADYsZHa4jA.png" /></figure><p><strong>The Company does a great job</strong> refactoring common components and makes its way towards a single platform. For example, separate operational storage is created for the intersecting part of the domain and common data storage for reporting is introduced to get rid of the reporting load from operational storage. <strong>But </strong>most likely <strong>ProductA </strong>and <strong>ProductB</strong> still keep major components of their architecture.</p><p>From <strong>an engineering point of view</strong>, this situation means that there are:</p><ul><li>Multiple independent apps/modules (data stores)</li><li>Heterogeneous storage types/vendors (MS SQL, MySQL, NoSQL, etc.)</li><li>Different historical data storage policies (ProductA might keep last 6 months of data, while ProductB keeps a full history due to SLA obligations)</li><li>Each module has a separate team which owns it</li></ul><p>In the meantime, from <strong>a business point of view</strong>:</p><ul><li>ProductA and ProductB cover essential parts of the business domain</li><li>ProductA and ProductB have the same customer base (at least partially)</li><li>ProductA and ProductB have all the necessary data to provide more sophisticated and valuable insights</li></ul><p>This setup is neither good nor bad. It’s just a natural <strong>result for a quickly growing business</strong>. Let’s look at requests coming to the Engineering team supporting this architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*BK0XKGN85AbNXcnzLQ7jeA.png" /></figure><p>And the natural <strong>answer </strong>from engineering to this need <strong>is more ETLs</strong> resulting in the following <strong>downsides</strong>:</p><ul><li>The long feedback loop for customer-facing development (demo, custom reports, on-demand requests)</li><li>Consumes a lot of engineering resources</li><li>Code-base growth</li><li>Unnecessary load on data storages in production</li><li>High cost</li></ul><p>If at least partially this looks familiar to you, then <strong>data lake is what you need</strong>.</p><h3>Why is a data lake-based approach so efficient?</h3><p>Considering all the scenarios described above and all the pros and cons of the solutions a company usually comes up with, here is <strong>a set of qualities</strong> which alternative approach should have:</p><ul><li>Contain all the data</li><li>SQL-like interface</li><li>Fast enough for on-demand analysis</li><li>Cheap</li></ul><p>So wait. <strong>What is a data lake essentially? </strong>It’s scalable BLOB storage combined with a query engine usually providing SQL-like interface. Examples of such a setup can be HDFS + Hive, HDFS + Spark SQL, Amazon S3 + Amazon Athena and so on.</p><p>This type of setup allows you <strong>to dump</strong> your operational <strong>data stores</strong> and any other data you have <strong>to a single location </strong>and then run queries across different data sets using the query engine.</p><p>All your <strong>data is available</strong> on-demand <strong>with a SQL-interface</strong> to access it. The later is really important because SQL knowledge at least on a basic level is very widespread among engineers. This means operational teams and data analysts will be able to use this setup by themselves, keeping Engineering team resources for more technically challenging features.</p><p>Let’s check out <strong>how </strong>to quickly and efficiently <strong>implement a data lake</strong>.</p><p><strong>Firstly</strong>, a data lake should be <strong>filled in with data</strong>. What exactly should be used as a source depends on your case, but in general, you would use either a primary instance of the data store or some sort of copy of it:</p><ul><li>Backup restored on another server. This choice is very widespread as backups are done regularly anyway and it is just a matter of creation of an automated process to restore them on another server.</li><li>Read-only replica or shadow instance for always-on setup. This case is preferable when you need a high frequency of updates in reporting, but your solution is not ready to adopt a streaming approach.</li></ul><p><strong>The next step</strong> is to build a job that <strong>extracts data from data sources</strong> and stores it into the BLOB storage. As an example of such a job, we built a configurable generic Spark job running in Amazon EMR cluster which extracts database table by table and stores it to S3. In order to make this data discoverable, we added functionality to register extracted data inside <strong>Amazon Glue</strong> Data Catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kzA-8Kmkhv5w8Hk8evAugQ.png" /></figure><p><strong>At this stage</strong>, we have a data lake filled in and ready for processing. Having only this stage implemented, your team can either <strong>start the implementation</strong> of cross-product reports (if you have specific requirements prepared) <strong>or start a research</strong> to come up with a set of useful insights. In any case, eventually, either analyst team or engineering comes up with a set of stable queries. So why not take the <strong>next step</strong> and start running them <strong>automatically </strong>on a regular basis, making these insights refreshed with new data, say, every night?</p><p>In order to achieve this level of automation, setup for Stage 1 <strong>is missing </strong>a minimum of <strong>2 components</strong>:</p><ul><li><strong>A generic component</strong> which can run a set of SQL scripts from pre-defined storage. For example, Spark Job in EMR cluster which runs Spark SQL scripts from S3 bucket or AWS Batch job which run scripts from S3 bucket using Amazon Athena.</li><li><strong>A trigger </strong>which starts script execution. For example, CloudWatch Scheduler to trigger the job using a fixed schedule or S3 Put object event to trigger the job as soon as new files are available for processing</li></ul><p>Additionally, it is a best practice to treat your data lake-based <strong>insights </strong>as a <strong>separate project</strong> with its own source repository and CI/CD pipeline. This view will ensure that every new insight (which is essentially a SQL script) passes all the stages from Dev to Prod thereby ensuring quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mbW1B5SJdcR5xTQ9f_IxeQ.png" /></figure><h3>Summary</h3><p>Data lake tackles complexity:</p><ul><li><strong>Combines data</strong> from multiple heterogeneous data sources into a single homogenous data store. In this way, it gets rid of the details and complexity of separate products technical stacks.</li><li>Basic data lake <strong>setup is simple</strong>. It does not result in another monster to support.</li><li>Data lake SQL <strong>interface is a standard</strong>. No advanced knowledge is required to leverage data lake power.</li></ul><p>Data lake is efficient:</p><ul><li>A setup, described above, <strong>doesn’t require a lot of time and resources</strong> to implement. In our experience, we saw similar projects implemented by analysts’ teams with the support of one engineer in two to three months. This includes both stages with data lake filled in and useful insights queries run automatically on a predefined schedule.</li><li>On-going support and extensions with new insights <strong>do not require engineering involvement</strong>. New insight is a new SQL query.</li><li>Very short and efficient feedback loop for on-demand requests like sales demo, research and incident resolution. Raw data is there and <strong>available for queries at any time</strong>.</li></ul><p>It already looks great, but this isn’t a paramount yet. In the next articles for data lake series, we will show how to extend this basic setup with orchestration to enable multiple independent ETL running in parallel. <a href="https://medium.com/greenm">Stay tuned</a>.</p><p>If you want to know how data lake can be a good response to complexity for your business — <a href="https://is.gd/T6XylN">GreenM’s architects team</a> would be happy to schedule a free whiteboarding session to collaboratively think through the right solution with you.</p><p>More articles:</p><p><a href="https://medium.com/greenm/deploy-evolution-serverless-feedback-platform-a76be810f72">Deploy Evolution: AWS Serverless Feedback Platform</a></p><p><a href="https://medium.com/greenm/scalable-data-pipeline-f5d3c8f7a6d9">Scalable Data Pipeline. Be Ready for Big Changes</a></p><figure><a href="https://is.gd/T6XylN"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSRX2Q0W155kT78SGvYaQw.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2bf54eaf4d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/data-lake-as-an-efficient-response-to-complexity-a2bf54eaf4d2">Data Lake as an Efficient Response to Complexity</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>