<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Lifion Engineering - Medium]]></title>
        <description><![CDATA[All things tech at Lifion by ADP - Medium]]></description>
        <link>https://medium.com/lifion-engineering?source=rss----5986961e21c5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Lifion Engineering - Medium</title>
            <link>https://medium.com/lifion-engineering?source=rss----5986961e21c5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:18:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/lifion-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaos Is Good! — In Tech]]></title>
            <link>https://medium.com/lifion-engineering/chaos-is-good-in-tech-2c487fce102f?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/2c487fce102f</guid>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[trending]]></category>
            <category><![CDATA[lifion]]></category>
            <category><![CDATA[devops-practice]]></category>
            <dc:creator><![CDATA[Narayanan Krishnamurthy (@NarainKrishh)]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 15:11:53 GMT</pubDate>
            <atom:updated>2021-11-03T15:11:53.611Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chaos Is Good! — In Tech</h3><p>We at <a href="https://medium.com/u/3a05c9d351fc">Lifion by ADP</a>, strongly believe that <a href="https://en.wikipedia.org/wiki/DevOps">DevOps</a>, <a href="https://en.wikipedia.org/wiki/Security_engineering">Security Engineering</a> and <a href="https://en.wikipedia.org/wiki/Site_reliability_engineering">SRE</a> best practices help make our systems not just agile but robust. Further to this, we are heavy users of <a href="https://aws.amazon.com/?nc2=h_lg">AWS Cloud</a>, and in our <a href="https://kubernetes.io/">Kubernetes</a> cluster we proudly host more than 200 microservices that power our <a href="https://flowofwork.adp.com/">award-winning</a> Next Gen <em>“Human Capital Management”</em> (HCM) platform.</p><p>“Being proud” to be part of the infrastructure and reliability team responsible for such an important system is an understatement and this role comes with significant responsibilities to ensure we maintain our stakeholders’ confidence in our infrastructure and applications.</p><p>To help with this, we decided to adopt <a href="https://en.wikipedia.org/wiki/Chaos_engineering">Chaos Engineering</a> in Lifion and this is the focus of this post. We hope sharing some insights into our approach may help you in your organization too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWPTwnLBSgrKVb3uqAvdvw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mparzuchowski?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Michał Parzuchowski</a> on <a href="https://unsplash.com/s/photos/jenga?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>How can we ensure our foundation is strong? — By randomly shaking some pillars — By creating turbulence — Welcome to Chaos Engineering</blockquote><h3>The Definition</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fI0nqUHsFMRNLOUPyZW7fA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@agathadepine?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ágatha Depiné</a> on <a href="https://unsplash.com/s/photos/fire-truck?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.</blockquote><p>The easiest way to explain Chaos Engineering is to compare it with a fire drill:</p><blockquote>A<a href="https://en.wikipedia.org/wiki/Fire_drill"> <strong>fire drill</strong></a> is a method of practicing how a building would be evacuated in the event of a fire or other emergencies.</blockquote><p>Just like a fire drill ensures our evacuation procedures, machinery and tools are intact before waiting for real havoc to happen, with Chaos Engineering, we ensure our <a href="https://en.wikipedia.org/wiki/Quality_of_service">Quality of Service</a> (QoS) is intact by randomly introducing failure conditions, reviewing results, and addressing any gaps in resiliency.</p><h3>The Environment</h3><p>Before we get into the framework, tools, and “<a href="https://aws.amazon.com/blogs/architecture/build-your-own-game-day-to-support-operational-resilience/">game days</a>”, I would like to emphasize that Chaos Engineering makes most sense if we run it in the <a href="https://principlesofchaos.org/">production</a> environment. Our goal in applying Chaos Engineering was not to create downtime, but rather to ensure our reliability, scalability, and availability <a href="https://en.wikipedia.org/wiki/Service-level_agreement">SLAs</a> were intact. In other words, if there is a known instability then the first step is to fix it and we can then continue to use Chaos Engineering approaches to test from time to time and randomly to ensure confidence.</p><p>For our Chaos Engineering efforts to be successful we needed a neat framework with an easy-to-use toolset, a developer-friendly pipeline, reporting of outcomes, and last but not least a feedback loop to fix any new findings.</p><h3>The Framework</h3><p>When we started evaluating Chaos Engineering tools and frameworks a couple of years back, we had few viable options in hand that would meet our requirements.</p><p>We were looking for a toolkit that:</p><ul><li>Aligned with <a href="https://principlesofchaos.org/">Chaos Engineering principles</a></li><li>Had built-in support for <a href="https://chaostoolkit.org/drivers/aws/">AWS related fault injection</a> such as AZ failure</li><li>Supports <a href="https://chaostoolkit.org/drivers/kubernetes/">Kubernetes related fault injection</a>, such as pod random kill, and out of memory</li><li>Supports <a href="https://chaostoolkit.org/drivers/toxiproxy/">Network fault injection</a>, such as latency and HTTP 404 errors</li><li>Is extensible using <a href="https://docs.chaostoolkit.org/reference/api/experiment/#process-provider">custom shell scripts</a></li><li>Supports <a href="https://docs.chaostoolkit.org/drivers/prometheus/">monitoring and observability</a></li><li>Has <a href="https://github.com/chaostoolkit/chaostoolkit-reporting">reporting</a></li><li>Is open source</li><li>Is light-weight and declarative</li></ul><p>Some of the options we considered were <a href="https://github.com/Netflix/chaosmonkey">Netflix Chaos Monkey</a>, <a href="https://www.gremlin.com/">Gremlin</a>, and <a href="https://litmuschaos.io/">LitmusChaos</a>. However, Chaos Monkey seemed to no longer be very actively maintained, Gremlin was not open source and did not appear to have some of the functionality we needed at the time, and LitmusChaos was still very new. We ended up choosing <a href="https://chaostoolkit.org/">Chaos ToolKit</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXCFaSUgOm-A6whhs6gXpw.png" /><figcaption>Ref: <a href="https://chaostoolkit.org/">https://chaostoolkit.org/</a></figcaption></figure><p>Getting started with Chaos Toolkit was very easy — write a JSON experiment, trigger the CLI, and boom we could get our first experiment running. This was cool for trying out and getting familiar with the toolkit. But how could we get the hundreds of our developers across our organization to buy into and adopt Chaos Engineering as a practice along with this toolkit? We decided to build a framework to help streamline their work and ensure it could support our engineering teams various use cases and needs and that it would integrate well into our specific environment. The framework needed to make it easy for engineering teams to write, test, run, fail, and fix their experiments for them to start and continue using it.</p><p>We built our framework around our existing ecosystem of tools, namely <a href="https://bitbucket.org/">Bitbucket</a>, <a href="https://medium.com/u/bb7e6f26a54">Jenkins</a>, and <a href="https://jfrog.com/artifactory/">Artifactory</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*REwpgHwQ2Z6naQcSctMciA.png" /><figcaption>The Framework Repositories</figcaption></figure><p>We wrapped following in our framework:</p><ul><li><strong>chaos-runner:</strong> Base Chaos ToolKit + pre-packaged extensions — This created a single docker image where we take chaos toolkit as a base and then install required extensions to it such as AWS packages, Kubernetes extensions and more.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90a0c2819c52540a07fdc9d75543e5e9/href">https://medium.com/media/90a0c2819c52540a07fdc9d75543e5e9/href</a></iframe><ul><li><strong>chaos-extensions </strong>—<strong> </strong>Chaos Toolkit comes with prebuilt extensions, but how could we integrate our organization’s ecosystem of tooling in it? We achieved this by writing custom modules using python. Some examples included probing metrics from our monitoring platform to determine system stability as well as integrating our custom functional validation framework APIs, and more. Basically, we were able to create <a href="https://chaostoolkit.org/reference/extending/extending-with-python/">our own </a>actions, controls, and probes as we wished. It’s also worth noting that we included this in the Dockerfile shown above.</li></ul><pre>.</pre><pre>├── common<br>│   ├── formatting.py<br>│   └── utils.py<br>├── actions<br>│   ├── aws<br>│   │   └── elasticsearch.py<br>│   ├── k6<br>│   │   └── actions.py<br>│   └── lifion<br>│       └── persistence.py<br>├── controls<br>│   ├── jira<br>│   │   └── tickets.py<br>│   └── platform<br>│       ├── lifion_aws_auth.py<br>│       └── resources<br>│           └── docker-compose.yml<br>└── probes<br>    ├── aws<br>    │   ├── elasticsearch.py<br>    │   └── route_53.py<br>    ├── http<br>    │   └── make_multiple_calls.py<br>    └── platform<br>        ├── availability.py<br>        ├── error_rate_percentage.py<br>        └── responsetime_95_percentile.py</pre><ul><li><strong>chaos-experiments</strong>— In Chaos Toolkit, every fault injection is called an experiment. These experiments are written in JSON, as per the example below. An isolated repository is kept just for these so that developers need not worry about the base toolkit or custom modules and can focus on their experiments. The following example experiment is from <a href="https://github.com/chaostoolkit/chaostoolkit-samples/blob/master/service-down-not-visible-to-users/experiment.json">Chaostoolkit examples</a>:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af0ab31349c943df8f013abfe9d16454/href">https://medium.com/media/af0ab31349c943df8f013abfe9d16454/href</a></iframe><ul><li><strong>chaos-scheduler</strong> — Easy-to-use, Jenkins-based scheduling for experiments, which can take parameters such as environment variables and VPC IDs. This is just a simple Jenkinsfile for every experiment. It supports one scheduler with one or more experiments.</li></ul><p>With these components, we provided a well-defined framework for developers to easily adopt and start contributing to our experiment test suite.</p><h3>Shared Responsibility Model</h3><blockquote>In our organization we have modular teams and a microservices architecture. We wanted each team to be able to own and manage their chaos experiments independently.</blockquote><p>We defined the below process for onboarding experiments, where our team would own the framework, engineering teams were responsible for their own experiments, and then we would come together to run game days where we would run tests together in a given environment, address any issues, and iterate and improve our experiments with rapid feedback from the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*Oqd2Iq-WSOfket4z4fYYhA.png" /><figcaption>Shared Responsibility Model</figcaption></figure><h3>Summary</h3><p>Adopting Chaos Engineering has been very beneficial for us. It has helped us efficiently identify and fix potential weaknesses in critical areas of our platform and client-facing system processes, such as payroll runs and disaster recovery testing, through controlled experiments before our clients may be impacted by real failures. Summarized below are some of our key learnings and takeaways, we hope you may have found this article useful and if this is an area of interest to you let’s connect — we’d love to hear your thoughts and experiences also!</p><ul><li><strong>Choose the tool that fits your needs</strong> — Now there are many options. For example, AWS has its <a href="https://aws.amazon.com/blogs/aws/aws-fault-injection-simulator-use-controlled-experiments-to-boost-resilience/">AWS Fault Injection Simulator</a>, and there are cloud-native tools emerging such as <a href="https://litmuschaos.io/">LitmusChaos</a> and <a href="https://www.gremlin.com/">Gremlin</a>.</li><li><strong>Build a framework</strong> — Support your developers with an easy-to-use framework for writing and creating chaos :)</li><li><strong>Share the responsibility </strong>— Let developers own their tests and help them to onboard new scenarios easily. Consider using game days also.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c487fce102f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/chaos-is-good-in-tech-2c487fce102f">Chaos Is Good! — In Tech</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Key Steps to Managing a Multi-Quarter Initiative]]></title>
            <link>https://medium.com/lifion-engineering/5-key-steps-to-managing-a-multi-quarter-initiative-96387b9d31a8?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/96387b9d31a8</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[scaled-agile-framework]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[process-improvement]]></category>
            <dc:creator><![CDATA[Sanjana Raj]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 15:49:30 GMT</pubDate>
            <atom:updated>2021-10-20T15:49:30.757Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BbzYYvWAPevQnjAklt6kaA.png" /></figure><p><em>How we implemented a new ETL job queueing solution in phases</em></p><p>With <a href="https://flowofwork.adp.com/">ADP’s NextGen HCM solution</a>, <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> jobs are often used as part of business critical, synchronous, and end-user facing workflows. As we continue to build our <a href="https://en.wikipedia.org/wiki/Low-code_development_platform">low-code development platform</a> at Lifion by ADP and offer it to more clients, we recently saw a need to shift to a different queueing architecture which would allow for increased scalability of ETL functions.</p><p>After evaluating the <a href="https://aws.amazon.com/sqs/">Amazon SQS</a> architecture we had in place, we identified an opportunity to improve the processing time for ETL jobs and, more importantly, make the expected execution time more consistent and less vulnerable to large, multi-job workflows run by other clients.</p><p>So — how did we go about executing a project of this complexity and size? We followed these five steps to manage this longer-term project and lead our teams with confidence.</p><ol><li><strong>Plan to re-plan with each phase</strong></li></ol><p>This may sound counterintuitive for ETL, but we avoided turning our roadmap into a queue itself. Instead, we leveraged a valuable approach in <a href="https://en.wikipedia.org/wiki/Scaled_agile_framework">scaled agile</a>, where we committed to one phase at a time and forecasted the rest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*quhEpt0BzNjMCEVl88SeMQ.png" /><figcaption>Dividing the project into three clear phases helped us commit to a short-term roadmap while forecasting the remaining work.</figcaption></figure><p>We divided the project into three clear phases which aligned with our quarterly roadmaps:</p><ul><li>In the first quarter, we began with investigating different technologies, and the team built out four proofs of concept (POCs) before honing in on an optimal solution.</li><li>During the second quarter, we re-planned and dove deeper into the actual implementation, with a focus on deploying the solution in a lower, production-like environment.</li><li>In the final phase as part of the third quarter, we began rolling out the solution in other environments and migrating over client instances from the previous queueing approach.</li></ul><p>For similar long-term projects, we try to avoid having a long list of features that we intend to work through in a queue. Instead, our teams work alongside their respective product and engineering leads to prioritize frequently and constantly. This allows us to remain flexible and adapt quickly to changes along the way.</p><p><strong>2. Start from the end</strong></p><p>For each quarter, we identified a primary objective and worked backwards to detail out specific milestones and activities to be completed within each sprint. This was important for us to be able to make reasonable estimations, and update timeframes regularly as the project continued to evolve. Aside from that, this approach also made it easier to keep stakeholders informed and on board, and celebrate delivery of those interim milestones.</p><p><strong>3. Keep high-level and granular views at all times</strong></p><p>You might wonder how an agile methodology, based on regular, continuous delivery, can work with big-picture planning?</p><p>The way we married these two concerns was by ensuring we were keeping track of daily, weekly, monthly, and quarterly priorities to create a realistic forecast over a long period of time.</p><ul><li>We used a Gantt chart to identify progress over a long-term timeline and plan around release windows and freeze periods (time windows where no software changes are permitted to avoid breaking existing functionality).</li><li>We managed and reported on shorter-term milestones using Confluence.</li><li>Sprint work was captured in Jira, which allowed us to keep a pulse on development of specific functionality on a daily and weekly basis.</li></ul><p>These multiple perspectives ensured that project details and task estimates were in sync with our roadmap, and gave us a complete view of the project at any given point.</p><p><strong>4. Establish dedicated sessions outside of existing meeting cadences</strong></p><p>Over the course of this project, we quickly learned that strategic discussions need their own time and can’t be covered as part of a regular stand-up.</p><ul><li>Holding a separate sync session twice a week helped to ensure we allocated dedicated time to brainstorming and following up on next steps. These meetings were generally kept to fifteen minutes for quick updates on action items from the previous session, but were occasionally extended to at most an hour when longer discussions were needed to address open questions.</li><li>Aside from internal check-ins as a team, we also held monthly touchpoints with leadership to communicate our progress. While we still kept our stakeholders informed with periodic updates to a project plan, these meetings allowed us to have two-way communication with senior leadership and seek feedback on our approach, as well as guidance on working through issues as they arose.</li></ul><p><strong>5. Define a comprehensive testing plan and rollout strategy in advance</strong></p><p>When it came time to actually going through the testing and rollout processes, documenting all details in advance with input from product and engineering leads helped set us up for success. Moreover, it helped us build confidence and mitigate risks early on.</p><p>Below are some questions which helped us plan ahead:</p><p><strong><em>Testing Approach</em></strong></p><ul><li>How will we test across environments and across different types of clients?</li><li>What types of testing do we need to conduct (e.g., functional, process-based, etc.)?</li><li>How will we benchmark performance?</li><li>How are we tracking client migrations and any related issues?</li></ul><p><strong><em>Rollout Strategy</em></strong></p><ul><li>How can we de-risk live clients?</li><li>Are there any prerequisites, technical considerations, or assumptions we need to call out?</li><li>What are the success criteria that will give us confidence before moving to the next phase of rollout?</li><li>What will be the communication plan to keep stakeholders informed?</li></ul><p><strong>Conclusion</strong></p><p>Large and highly technical projects such as this one can often be overwhelming, even for the most experienced of project managers. Some of the steps outlined above, like continuous re-planning or working backwards, were new learnings that my team and I realized over the course of the project. These are just a few differences in approach that we found helpful to consider as compared with managing a shorter-term project. Ultimately, these five key steps helped us successfully manage the rollout of a new ETL queueing solution, and may help you lead similar long-term projects too:</p><ol><li><strong>Plan to re-plan</strong> (often) by using a phased approach</li><li><strong>Work backwards</strong> from your end goal to define milestones</li><li><strong>Manage multiple views</strong> of your project, ranging from high-level to more granular</li><li><strong>Hold dedicated sync meetings </strong>— a large project deserves its own time</li><li><strong>Define testing and rollout processes in advance</strong> to stay ahead</li></ol><p><em>banner image designed by </em><a href="http://www.freepik.com/"><em>freepik</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96387b9d31a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/5-key-steps-to-managing-a-multi-quarter-initiative-96387b9d31a8">5 Key Steps to Managing a Multi-Quarter Initiative</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Platform Reliability Engineering at Lifion]]></title>
            <link>https://medium.com/lifion-engineering/platform-reliability-engineering-at-lifion-6804765ca2da?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/6804765ca2da</guid>
            <category><![CDATA[reliability]]></category>
            <category><![CDATA[teams-and-teamwork]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[high-performance]]></category>
            <dc:creator><![CDATA[Gabriel Rojas]]></dc:creator>
            <pubDate>Wed, 08 Sep 2021 15:56:45 GMT</pubDate>
            <atom:updated>2021-09-08T15:56:45.352Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3YnU7iXbO-BjIVI9qhPig.png" /><figcaption>SRE @ Lifion</figcaption></figure><p>Many are familiar with Site Reliability Engineering (SRE), Google’s definition of this, and how they have implemented a set core of practices across their organization. Here at Lifion by ADP, we have tried to apply these core principles and practices for our organization to benefit our customers.</p><p>However, we realized that to build a successful SRE team, we have to meet customer and market demands in a way thats fit the culture and goals of our business. This article touches on some of the adaptations we have employed to build a high-performing SRE function for our organization. <br> <br>One common characteristic of a high-performing reliability team that we have identified over time is embracing change.<strong> Our SRE operating model drives values and improves the customer experience while bringing costs down. </strong><br> <br>There are three distinct modes within the operating model that the SRE team operates within.</p><ul><li><em>Ad-hoc</em> — I call this part of our operating model “running the business”. This is where software teams have requested us to help them unblock, advise, answer specific questions, architect, review production readiness, improve the process, perform capacity planning, test performance, assist with client requests, etc. As you can imagine, this type of work can overwhelm the team if not properly managed. Every quarter we allocate 35% for this type of work, and we always make sure that this work strategically aligns with business goals. <strong>We always ask how the work that is being requested impacts customer happiness.</strong></li><li><em>Consultative</em> — We pair with other software development teams to advise and give workshops on business <a href="https://eng.lifion.com/how-to-okr-e1afcac6111d">Objective &amp; Key Results (OKRs)</a>. This kind of engagement take up to 25% of the team capacity every quarter, and the resources can be allocated to one specific software development team or multiple ones. A good example is when we worked with numerous software development teams on the self-service adoption of application monitoring and distributed tracing for over 500 micro-services.</li><li><em>Embedding</em> — Perhaps, this is my favorite part of our operating model that our teams follow. From time to time, we need to embed reliability engineers full-time in cross-team initiatives periodically to create new capabilities or improve our customer sentiment. <br>When there is a business need for this type of engagement, we request clear goals, timelines, and a great definition of success. I have witnessed in previous companies that resources are not utilized to their full potential, waste engineers’ time, and create low morale.<br>Reliability engineers are one of the most expensive resources the company has because of their knowledge and demand of projects for this skillset. At Lifion, we engage in this embedded mode in a sprint (two weeks), a quarter(s), or even a one-year length project. <br>A good example of where we have deployed this engagement approach is when we were building a global footprint of our <a href="https://flowofwork.adp.com/">HCM metadata-driven platform</a> to meet global customer demands.</li></ul><p>Let us talk a little more about the processes to enable us to be a high-performing team. Firstly, the ecosystem of people supporting us such as our Project and Product management organization is essential to us having a positive impact. We follow standard agile practices and ceremonies like any software development team, with the slight difference that we use a combination of <a href="https://www.atlassian.com/agile/kanban/kanban-vs-scrum">Scrum and Kanban</a> to manage our complex workload week to week. We have had much success by adopting both while delivering value to Lifion’s strategic goals.</p><p>To build a high-performing SRE team(s), it is critical to ensure the right people are in place in a first-class support ecosystem to build trust and safety. There is no one-size-fits-all answer to what an operating model should look like for SRE teams. Still, by driving from the customer needs and company goals, we can design and adopt an operating model that aligns with our business and culture and strategically drives value.</p><p>If working on a team like this sounds interesting to you, please head over to see our current open positions.</p><ul><li><a href="https://tech.adp.com/job/13574521/lead-devops-sre-can-be-remote-new-york-ny/">Senior Reliability Engineer</a></li><li><a href="https://recruiting.adp.com/srccar/public/RTI.home?c=1046545&amp;d=External&amp;r=5000758416906#/">Senior Technical Product Manager</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6804765ca2da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/platform-reliability-engineering-at-lifion-6804765ca2da">Platform Reliability Engineering at Lifion</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Manager’s Diary]]></title>
            <link>https://medium.com/lifion-engineering/product-managers-diary-538f8d10fd7f?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/538f8d10fd7f</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[pda]]></category>
            <category><![CDATA[engineering-management]]></category>
            <dc:creator><![CDATA[Jude Murphy]]></dc:creator>
            <pubDate>Fri, 03 Sep 2021 16:07:59 GMT</pubDate>
            <atom:updated>2021-09-03T16:07:59.667Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*qni-E_dIoEKIasTened2Pw.jpeg" /></figure><h4>A reflection on optimizing productivity</h4><p>Over the past few years, as a Senior Technical Product Manager with <a href="https://lifion.com/">Lifion, by ADP</a>, I’ve been enamored with coming up with new ways to optimize my output, the output of our engineering teams, and to improve the quality of the products I am responsible for. I’m driven by the belief that the more one can produce, with the highest possible quality, the more value one can then deliver to customers and the more win-win opportunities they can create.</p><p>At Lifion, one of our <a href="https://lifion.com/about">credos</a> is to <em>master your craft. </em>Following this, each of us is encouraged to think of new ways to perform more effectively within our roles towards us building the best <a href="https://flowofwork.adp.com/">NextGen HCM Solution</a> possible for our clients. Provided with this freedom to experiment, the goal of this article is to document and share what’s worked for me in my role and might help others.</p><p>Let’s level-set on what a Product Manager is responsible for. For a formal definition, if one were to look up what a “Product Manager” is, they’d find “<a href="https://en.wikipedia.org/wiki/Product_management#:~:text=Product%20management%20is%20an%20organisational,stages%20of%20the%20product%20lifecycle.">an organizational function within a company dealing with new product development, business justification, planning, verification, forecasting, pricing, product launch, and marketing of a product or products at all stages of the product lifecycle.</a>” Given this definition, the Product Manager role can be very diverse in its responsibilities and it takes a lot of self-reflection to really hone the craft over time.</p><p>There are key changes to working patterns that can be incredibly useful and help a product manager work more effectively in their role. While not all may be generalizable to everyone and every situation, regardless, some that stand out to me are shared below.</p><h3>Despise Meetings, Love Efficient Processes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BoECMP_L6_Ky61zJpg9cAw.jpeg" /></figure><p>A product manager is likely one of the top three people on a team setting up the most meetings. Meetings can be expensive especially when the team is trying to hit deadlines. Keeping in mind, when scheduling a meeting, especially with engineers, it may be taking time away from the people that contribute to the team’s goals. It goes without saying, supporting the team as much as possible is something that works for everyone’s benefit.</p><p>In modern technology product development teams, there are generally two types of meetings; those that are part of regular scrum ceremonies, and ad-hoc meetings. If a team is following the <a href="https://en.wikipedia.org/wiki/Scrum_(software_development)">Scrum</a> framework, as part of ceremonies, at a minimum, there exists stand-up, backlog grooming, sprint planning, and retrospective. Of these, the product manager is responsible for running and facilitating backlog grooming. Grooming should optimally take 30 minutes to 1 hour, regardless of team size or complexity of tasks. Given this is a meeting that requires the entire engineering team, this is a top priority for the product manager to optimize. Choosing work items <em>before the meeting </em>to be groomed, and ensuring they have enough detail on them, is critical. When explaining the goal of a ticket, it should require minimal discussion to understand what’s needed, and why. Understandably, alignment, team feedback, and “vision painting” are done within grooming as well, but once the product manager and engineering team are on the same page, it should be smooth sailing. Also — whether using JIRA, Rally, Trello, etc., <em>memorizing</em> the hotkeys needed to add story points to tickets and to mark them as groomed can dramatically speed up team grooming sessions. For example, when I started using hotkeys, my grooming sessions on average went from 45 minutes to around 20 to 30 minutes.</p><p>For the scrum ceremonies that the product manager is not directly facilitating, it is possible to assist in improving the efficiency of those as well. For example, before a sprint planning session, I find it extremely valuable to ensure that everything is organized by priority and appropriately groomed within the upcoming sprint. This way ticket organization is not happening during the meeting, taking up others’ time.</p><p>As a rule of thumb, where possible, a product manager can use their time as a buffer to save the teams’ time, because expending one person’s 10 minutes is significantly better than 6 people’s collective 10 minutes. <em>Again — the team needs to hit deadlines, a product manager should strive to not be the reason or excuse for when that doesn’t happen.</em></p><p>Regardless of the type of meeting, whether it be a scrum ceremony meeting or an absolutely required ad-hoc meeting, always looking for ways to optimize is key because collective time adds up. If a meeting is scheduled to take an hour, that doesn’t mean that it must take an hour.</p><h3>Wear Out Your Customer’s Shoes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*NFQUr3a5spXFYdMWpS0iFg.jpeg" /></figure><p><a href="https://www.merriam-webster.com/dictionary/empathy">Empathy</a>: <a href="https://www.merriam-webster.com/dictionary/empathy">the action of understanding, being aware of, being sensitive to, and vicariously experiencing the feelings, thoughts, and experience of another of either the past or present without having the feelings, thoughts, and experience fully communicated in an objectively explicit manner.</a> A common buzz-phrase used amongst product managers in the industry is <a href="https://transformationtalk.substack.com/p/008-empathy-driven-program-and-project">“empathy-driven product management”</a>. While this is excellent in theory, it’s not very descriptive as to how a product manager actually acquires empathy for their stakeholders. Admittedly, this is different from person to person on how they actually empathize with those using their product. To truly get to a place that can be considered to be empathy-driven, it can help to use a consistent process that supports making better product decisions in line with this philosophy.</p><p>At the simplest level, a product manager should use their own product. What better way to empathize with users than to walk in their shoes? For example, I schedule time every week to use the products I am responsible for, document pain points, find potential bugs, review product hot paths, etc. This has helped drive more efficient discussions with designers when coming up with new features and improvements, more effective communications during grooming sessions from being able to cite anecdotal reasonings for changes or adding a new feature, and improved communications with stakeholders via being able to better relate to how they use the product(s).</p><p>Lastly, product decisions should not be made solely based on qualitative empathetic notions as one may consider it as being opposed to quantifiable data-driven decision-making. As a product manager, it is important to be both empathy-driven as well as <a href="https://amplitude.com/blog/product-strategy">data-informed</a>. Features that have been verified by both these decision-making methodologies together have tended to yield noticeably higher stakeholder satisfaction in our organization.</p><h3>Avoid All-Flash, No Substance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*4AwOutFDEpiKKfXaob-Tvw.jpeg" /></figure><p>Organizations judge product managers on many different <a href="https://uxdesign.cc/how-to-evaluate-product-managers-including-yourself-2af350674190">axes</a>, one of which is their ability to roll out high-value features. This can be viewed from the stakeholder’s perspective as enabling them to do something with the application that they could not before or as an improvement to an existing process. While this may be a common way to measure the performance of a product manager, it can inadvertently foster a mentality of incessant feature building rather than a focus on how these features work together to serve users’ needs, along with performance and other <a href="https://en.wikipedia.org/wiki/Non-functional_requirement">non-functional</a> requirements. In this situation, it is easy for product managers to push non-functional concerns to the responsibility of engineering teams and primarily focus on new functionality. However, this can lead to prioritization of work that may not be truly most valuable for the product.</p><p>When defining a new feature it can help to include expectations for performance and any other relevant non-functional requirements as part of acceptance criteria. Obviously, this should be considered case by case but if an engineer says they need more time to improve the performance of the feature and there is buffer in the plan it helps to make sure such requirements are prioritized and addressed. These considerations directly impact the overall success of the product as well. <em>Performance affects user’s perception of the product too.</em></p><h3>That’s All Folks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*WeS-s7vWT3JCQYWuLfIRkA.jpeg" /></figure><p>Product Management is a unique craft that impels you to be everlearning. It requires wearing multiple hats, balancing various stakeholders, and communicating in a way that influences and inspires others. Learning to enjoy the process, listening to others, and speaking with candor while bringing life to new and exciting ideas is truly the beauty of the craft of product management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=538f8d10fd7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/product-managers-diary-538f8d10fd7f">Product Manager’s Diary</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Online Co-op Experience at Lifion]]></title>
            <link>https://medium.com/lifion-engineering/the-online-co-op-experience-at-lifion-41241e24f9bd?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/41241e24f9bd</guid>
            <category><![CDATA[mentorship]]></category>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jinwu Liu]]></dc:creator>
            <pubDate>Fri, 20 Aug 2021 18:12:28 GMT</pubDate>
            <atom:updated>2021-08-20T20:18:45.730Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written by </em><a href="https://medium.com/u/b36ef5e72628"><em>Jinwu Liu</em></a><em> and </em><a href="https://medium.com/u/c453f3e2b8f2"><em>Armaan Samuel</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rS6MRjKgppKY0rxS" /><figcaption>Photo by <a href="https://unsplash.com/@surface?utm_source=medium&amp;utm_medium=referral">Surface</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Lifion, by ADP is a big believer in the value of <a href="https://www.usnews.com/education/best-colleges/articles/2015/03/31/understand-the-differences-between-a-co-op-internship">co-op programs</a> and how they are a win-win for co-ops and companies alike. Co-ops gain experience from putting their classroom skills to work in a real-life environment and also expanding their networks while companies benefit from co-ops adding value to time limited projects and from investing in a pipeline of diverse talent for future hiring.</p><p>Lifion has had a well-established paid co-op program for a number of years now, whereby engineering students are embedded in the platform engineering organization for around 16 weeks, working shoulder-to-shoulder with engineers and others on a wide range of interesting and important projects. Unlike some companies, Lifion sees co-ops as a long-term investment in talent for their future and it’s great to see many co-ops becoming full time employees. I’ve nearly completed my tenure as a Technical Product Owner Co-op at Lifion on the Platform Web UI team and based on a dialog with my buddy Armaan Samuel, a fellow Software Engineering Co-op on the Platform ETL team, this blog post is intended to provide an insight into what the experience was like for us, notably while working remotely due to the pandemic. We hope this may be of use to potential co-ops in future and other readers alike!</p><h3><strong>Conversation</strong></h3><p>Jinwu Liu (JL): Hey Armaan, how are you doing?</p><p>Armaan Samuel (AS): Hey Jin, I can’t believe the term is already over. 16 weeks have come and gone.</p><p>JL: Yeah, I can’t believe it. As cliché as it sounds, I feel like I’ve learned so much in such a brief time at Lifion — what are your thoughts?</p><h4><strong>Skills Learned in School vs Workplace</strong></h4><p>A: Absolutely agree! Whenever I complete a co-op term I like to think about what are all the courses I’ve taken so far in school and see how the co-op relates to what I’ve completed during the term. At school, I’ve found that most of what we learn is theoretical and our assignments test our implementation and understanding of that theory. For example, learning about Data Structures, Algorithms and Operating-Systems using C was interesting but I rarely needed to apply these skills directly on the job. However, the problem-solving and critical thinking skills I developed from these challenging courses helped me learned quickly on the job, and aided me in conceiving intelligent Typescript solutions to complete my work.</p><p>Conversely, Lifion introduced me to technologies that I had not been exposed to in my education. Working with Jenkins, Docker, Kubernetes and cloud development were new experiences that I’ll be able to apply in my future career of software engineering. Additionally, the large code base and nuanced microservice interactions pushed my debugging and testing skills to the limit! Working in the Lifion codebase was as fun, yet just as challenging, as working in our Operating Systems course codebase!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_qmMMJNmJIKkx3hgX6JmQ.jpeg" /><figcaption>Source: <a href="https://www.piqsels.com/en/public-domain-photo-jonwr">Piqsels</a></figcaption></figure><p>JL: As a Computer Science student myself, your points resonate with me from a software development angle. As for working in product management, it can be hard to find colleges that provide students with “Product Management 101” style courses as part of an undergraduate degree. I think the closest to what I’m doing can be found in courses on Human-Computer Interaction, Design, Machine Learning/Artificial Intelligence, and higher-level software development/principles courses — but because of the multi-faceted nature of the product role, it’s often necessary to take learnings from a wide variety of courses. What I’ve noticed is that nothing beats getting industry experience, especially in a product and technology focused organization such as Lifion.</p><p>For example, right off the bat my manager made it clear to me that I’d be leading a large-scale and cross-team project on an Organizational Chart UI component that would provide business value to Lifion and clients well after my term there. No course prepared me to work autonomously with so many different teams and perspectives. Understanding requirements and the “why” of such products is unique to Lifion and similar companies, something it seems school curriculum struggles to keep up with. That said, more and more seem to be adding product management courses to their offerings — pretty cool to see.</p><h4><strong>Importance of Mentorship for Co-ops</strong></h4><p>JL: Another valuable takeaway I’ve grown to appreciate through talking with others at Lifion is the significance of mentorship, especially as co-ops. A few weeks into my co-op term, my manager suggested a few Technical Product Managers, Designers, and Developers that would help me in not only understanding the work that I was doing but also in obtaining any career advice in general. Given the variety of prior experiences of people I had informal chats with, I was able to gain fresh advice from many perspectives that I may have otherwise overlooked. This is an opportunity that I hopefully will be able to carry into my future co-op experiences. And that’s aside from all the help my manager has been giving this entire term, finding a perfect balance of assigning work yet still allowing me to autonomously grow and collaborate. What are your thoughts on mentorship?</p><p>AS: I am so grateful for my manager! He has invested a lot of time in me to help me grow as a software engineer. From the start of my co-op term, he hand-picked progressively more challenging work for me to onboard onto our large codebase in a way that has helped me to be successful. Every time I have had a new ticket assigned to me that used Lifion proprietary concepts that I was not familiar with, he spent time not only explaining the concepts themselves but also how they tied into the bigger picture at Lifion. His coaching and mentorship has made me feel like my contributions really matter.</p><p>Also, whenever I am blocked, he doesn’t directly give me the answers to my problem. Since he knows the huge codebase like the back of his hand, he provides me with targeted hints to encourage me to think differently and approach my problems from a new angle. I have learned a lot from him and am a much sharper software engineer than when I started. I definitely plan on mentoring future co-ops I may work with just as he has mentored me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DxBHpSXdKq7fOZ8D" /><figcaption>Photo by <a href="https://unsplash.com/@next_academy?utm_source=medium&amp;utm_medium=referral">NEXT Academy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>No Such Thing as Over-Communicating</strong></h4><p>JL: In my time at Lifion, I’ve also learned the value of over-communicating. On top of working remotely and dealing with the challenges it brings (such as Internet connections cutting out or bad audio), a Product Manager must be able to ensure all stakeholders are on the same page in a concise yet descriptive way. I’ve noticed it to be sometimes hard — such as when you just had a big lunch and you’re 45 minutes into your hour-long afternoon meeting — to always have everyone’s attention, especially when not all co-workers are face-to-face. Thus, it is especially important to emphasize the “why’s” behind decisions and to continuously question and clarify any ambiguities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GM4qdzTWYUy_juSU" /><figcaption>Photo by <a href="https://unsplash.com/@headwayio?utm_source=medium&amp;utm_medium=referral">Headway</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>AS: I’ve definitely had to increase and adapt my communication due to working remotely this term. In previous co-op experiences that were on-site, whenever I’ve been blocked in my work, I’ve been able to walk over to a Senior Developer’s desk, show them my code, and see if they can help me solve issues. During this remote co-op, the best I can do is WebEx screen share! This term, I’ve needed to work harder to communicate the changes I made, the issues I’m facing, and what I have done to try to resolve them. One example this term has been issues I’ve had configuring node.js on my machine. It took much screen-sharing with 4 different developers to finally discover the issue was due to some nuanced version incompatibilities. If I hadn’t communicated well enough, it might’ve taken even longer!</p><p>The second aspect of remote communication that stood out to me was ensuring that my understanding of the work needing to be done matched the functional requirements defined on user stories and communicating this effectively. This meant working with our Product Manager to truly understand the specific uses cases and detailed functional requirements. She would clearly communicate the expected behavior and we would check for understanding. I would then demo and clearly communicate how the changes I implemented were working and satisfying the acceptance criteria. Any final thoughts about the co-op?</p><p><strong>Conclusion</strong></p><p>JL: Yeah! Initially, I didn’t know much about the HR software industry — now I realize that there is a whole world of innovation possibilities. I think this sentiment is captured well in Lifion’s mission of “defining the future of work.” Ultimately, my time with Lifion has helped me more deeply appreciate how vast of an influence technology is having in transforming all kinds of industries.</p><p>AS: Absolutely. Having been at Lifion for 3 months, I’ve learned that my team is tight-knit, with ownership of critical services and huge responsibility and impact to the platform and program. I’ve also really enjoyed scratching the surface of <a href="https://en.wikipedia.org/wiki/Low-code_development_platform">metadata-driven development</a> and <a href="https://en.wikipedia.org/wiki/Cloud_computing">cloud development</a>. Lifion truly gives their co-ops the feeling of responsibility in their work and has definitely been my favorite co-op term. Despite the remote nature of this co-op, I felt the work I did was more meaningful and impactful than other in-person co-ops. I’m also excited to be continuing next term so I can solidify my understanding of their software development practices and continue to make a real impact with the team.</p><p>JL: Great to hear! Anyways, it’s been a really great term with you!</p><p>AS: For sure!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41241e24f9bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/the-online-co-op-experience-at-lifion-41241e24f9bd">The Online Co-op Experience at Lifion</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NIST Cybersecurity Framework and Kubernetes]]></title>
            <link>https://medium.com/lifion-engineering/nist-cybersecurity-framework-and-kubernetes-770d3df84d6c?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/770d3df84d6c</guid>
            <category><![CDATA[lifion]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[nist-framework]]></category>
            <category><![CDATA[nist]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Gabriel Rojas]]></dc:creator>
            <pubDate>Wed, 18 Aug 2021 17:49:06 GMT</pubDate>
            <atom:updated>2021-08-18T17:49:33.868Z</atom:updated>
            <content:encoded><![CDATA[<p>Customers trust Lifion, by ADP with some of their most important data and they rely on us to preserve its security and privacy following the highest industry standards. In this article we talk about some ways we bridge the strategy and tactical implementation of NIST Cybersecurity Framework in Kubernetes.</p><p>The <a href="https://www.nist.gov/cyberframework/framework">NIST Cybersecurity Framework</a> (NCF) provides a framework for companies with guidelines and practices on managing cybersecurity risks and protecting organizations. Following this framework is required for essential services, critical infrastructure, and government agencies to adopt.</p><p>Many private organizations have adopted NCF in recent years, but many still do not have adequate cybersecurity controls. The framework is not one size fits all but is up to every organization to understand their crown jewels and manage their risks.</p><p>Gartner predicted that in 2020 that 50% of the U.S organizations will use the NIST Cybersecurity framework.</p><figure><img alt="https://www.nist.gov/industry-impacts/cybersecurity-framework" src="https://cdn-images-1.medium.com/max/1024/1*faDnWy3Iqc8YoIJrjH_M7A.png" /><figcaption><a href="https://www.nist.gov/industry-impacts/cybersecurity-framework">https://www.nist.gov/industry-impacts/cybersecurity-framework</a></figcaption></figure><p>Companies want to adopt a good security posture. Still, they often get overwhelmed with the number of controls and often do not know where to start.</p><p>This is where the Cybersecurity Framework helps companies organize a plan of action. There are five main components to follow:</p><ol><li><strong>Identify:</strong> In this step, organizations identify their more important assets or crown jewels. E.g. data lakes, databases, blob storage, proprietary code, etc.</li><li><strong>Protect: </strong>This is where organizations invest resources to protect the assets identified in the prior step. E.g. data loss prevention tools, antivirus, vulnerability management tools, code scanning tools, firewalls, etc.</li><li><strong>Detect: </strong>This component allows organizations to alert and identify a breach in the controls in place. E.g. operational dashboards, alerts, abnormally behavior detection tools, etc.</li><li><strong>Respond: </strong>Building an incident response plan when a breach occurs is critical not only for engineers but also for the C-Suite and board of directors.</li><li><strong>Recover: </strong>This component is responsible for creating a business continuity plan (BCP)</li></ol><p>This framework fits well within cloud-native organizations. <a href="http://cncf.io/">The Cloud Native Computing Foundation</a> (CNCF) defines cloud-native as:</p><blockquote>Cloud native allows IT and software to move faster. Adopting cloud native technologies and practices enables companies to create software in-house and allows business people to closely partner with IT people, keep up with competitors and deliver better services to their customers.</blockquote><p>When the framework is in place, it should not follow a linear approach as the threat landscape is always changing. Instead of a linear approach, the framework should follow a continuous improvement approach. If not, the risks of not having proper controls could be catastrophic to the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*gSH29dyZWTZnwmuyvA7kiA.png" /><figcaption><a href="https://www.nist.gov/cyberframework">https://www.nist.gov/cyberframework</a></figcaption></figure><p>Now that we understand the theory lets see how we can apply some of the Cybersecurity Framework categories to <a href="https://kubernetes.io/">Kubernetes</a>. The NCF offer over 180 controls that can be applied to organizations.</p><h3>IDENTIFY</h3><ul><li><strong>ID.AM-2 </strong><em>Software platforms and applications within the organization are inventoried.</em><br>An organization has a configuration management database (CMDB) to help keep track of all the Kubernetes clusters and applications running on top of these clusters.</li><li><strong>ID.GV-1 </strong><em>Organizational cybersecurity policy is established and communicated.<br></em>Policies include a least privilege approach for human and machine access to the data. A good policy includes a data classification to understand where and who can access personally identifiable information (PII) and (SPI).</li><li><strong>ID.RA-1</strong> <em>Asset vulnerabilities are identified and documented.<br></em>Every application deployed has vulnerabilities; having this document is critical for the business to decide if it is willing to accept the risk and plan to remediate. This includes items such as:<br>- Are <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/">Kubernetes namespaces</a> utilized for application segregation?<br>- Is <a href="https://linkerd.io/2/features/automatic-mtls/">mutual TLS</a> (mTLS) enabled for application to application communication?<br>- Are TCP/IP network rules enabled to block or allow traffic among apps? E.g., only allowing communication from app A to app B on port TCP port 443</li><li><strong>ID.SC-1</strong> <em>Cyber supply chain risk management processes are identified, established, assessed, managed, and agreed to by organizational stakeholders.<br></em>This has been an important control on Cybersecurity for quite a long time, but now it may be more important than ever after <a href="https://www.cisa.gov/news/2020/12/13/cisa-issues-emergency-directive-mitigate-compromise-solarwinds-orion-network">SolarWind&#39;s cyberattack was discovered in December 2020</a>.<br>Understanding what third-party vendors and software libraries are used in a platform ecosystem is important because this could open backdoors to the ecosystem and compromise security.</li></ul><h3>PROTECT</h3><ul><li><strong>PR.AC-1</strong> <em>Identities and credentials are issued, managed, verified, revoked, and audited for authorized devices, users, and processes.<br></em>Rotating application credentials monthly or even daily is easier than ever in Kubernetes. With Kubernetes, using <a href="https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/">service accounts</a> allows following the least privileged approach where an application only has access to the resources that it needs.</li><li><strong>PR.AT-1</strong> <em>All users are informed and trained<br></em>This could be the most important category in the NIST Cybersecurity Framework. Errors and misconfiguration remains high in the list of data security breaches. <a href="https://enterprise.verizon.com/resources/reports/2020-data-breach-investigations-report.pdf">The 2020 Verizon Data Breach and Investigation report categorizes misconfiguration and human error as more common than malware</a>.</li><li><strong>PR.DS-1</strong> <em>Data-at-rest is protected<br></em>This is one of the easiest controls to implement. Kubernetes offers encryption of data at rest by using a <a href="https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/">KMS provider</a>.</li></ul><h3>DETECT</h3><ul><li><strong>DE.AE-5</strong> <em>Incident alert thresholds are established.<br></em>Kubernetes is great at capturing logs and events from the Kubernetes platform and the applications. However, it does not offer alerting and behavior anomaly analysis out of the box. Third-party products are needed to detect anomalies in the Kubernetes ecosystem, such as <a href="https://www.paloaltonetworks.com/prisma/cloud">Prisma Cloud</a>, <a href="https://www.stackrox.com/">StackRox</a>, and <a href="https://sysdig.com/opensource/">Sysdig</a>.</li><li><strong>DE.CM-8 </strong><em>Vulnerability scans are performed.<br></em>There should be a layered approach that can be achieved with the tools mentioned above. The vulnerability scans should be performed at 5 different levels.<br>- Host level: Kubernetes infrastructure such as worker nodes, ETCD, master nodes running on top of Linux and or Windows servers should be scanned and patched.<br>- Kubernetes infrastructure: The cluster must run the latest supported version and follow <a href="https://cloud.google.com/kubernetes-engine/docs/security-bulletins">security bulletins</a> to make sure any vulnerabilities are being mitigated. <br>- Docker images: Scanning docker images should be part of an organization’s Cybersecurity Framework.<br>- Running images: scanning running images is a critical component because it is important to be aware of any new process or actor that may be changing the way an application was supposed to behave.<br>- Code scanning: Several scanning tools can help with this. Making sure there are no vulnerabilities or bad coding practices is critical. Tools such as <a href="https://www.sonarqube.org/">SonarQube</a> or <a href="https://snyk.io/">Snyk</a> can help find code vulnerabilities.</li></ul><h3>RESPOND</h3><ul><li><strong>RS.AN-1</strong> <em>Notifications from detection systems are investigated<br></em>Having systems that detect and alert are not sufficient, and based on experience, this category takes time to master. Ensuring a Security Operations Center (SOC) look into real alerts is key; the challenge is to eliminate all the noise from alerts. Automation should be a first-class citizen to ensure proper actionable alerts are getting to the SOC.</li><li><strong>RS.IM-2 </strong><em>Response strategies are updated<br></em>This category needs discipline by all teams in the organization. Good processes and frameworks are key with documentation up to date. As technology changes rapidly, a Cybersecurity strategy should be updated at the same time. It is important to avoid relying on an out-of-date response strategy that does not apply to the organization anymore. I recommend reviewing response strategies once a quarter.</li></ul><h3>RECOVERY</h3><ul><li><strong>RC.RP-1</strong> <em>Recovery plan is executed during or after a cybersecurity incident<br></em>Knowing what to do when Cybersecurity happens is important not only for engineers but also for senior leadership. Running <a href="https://en.wikipedia.org/wiki/Business_continuity_planning">tabletop exercises</a> at least once a quarter is important to build muscle memory.</li></ul><h3>CONCLUSION</h3><p>In conclusion, there are many areas in how an organization can improve Cybersecurity. It is highly beneficial to make early investments in identify, protect, and detect categories and less expensive to invest in these areas than fixing security after a breach. Without this, there is risk of not only financial but also reputational damage.</p><p>Finally, it’s critical to remember that security is everybody responsibility or as we say at Lifion, <strong>“Lifionites keep Lifion secure”</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=770d3df84d6c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/nist-cybersecurity-framework-and-kubernetes-770d3df84d6c">NIST Cybersecurity Framework and Kubernetes</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning at Lifion]]></title>
            <link>https://medium.com/lifion-engineering/machine-learning-at-lifion-57b1bc9299c4?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/57b1bc9299c4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mano Vikash]]></dc:creator>
            <pubDate>Wed, 02 Jun 2021 16:03:01 GMT</pubDate>
            <atom:updated>2021-06-02T16:03:01.232Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Bf21oI9_KN-5QDq" /><figcaption>Photo by <a href="https://unsplash.com/@franckinjapan?utm_source=medium&amp;utm_medium=referral">Franck V.</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article is about how we approach the problem of building ML models for features at Lifion, by ADP. When building ML models in practice, due to various reasons theoretical underpinnings may at times be overlooked. The discrepancy between theory and practice in machine learning is a <a href="https://syncedreview.com/2017/12/12/lecun-vs-rahimi-has-machine-learning-become-alchemy/">fiercely debated</a> topic. We will discuss how we build cutting edge ML models in practice while being conscious of the <a href="https://www.scientificamerican.com/article/just-a-theory-7-misused-science-words/">theoretical</a> aspects behind our work.</p><p>Here is the opening line of the preface of <a href="http://www.cs.cmu.edu/~tom/mlbook.html">Tom Mitchell’s book</a> on machine learning from 1997</p><blockquote>The field of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience.</blockquote><p>This statement about machine learning still makes sense to this day and resonates with us in our work on a variety of ML-powered services and features. We will start by making this statement more precise by looking at a concrete problem we worked on for a particular feature which required predicting the items in a sequence, which we shall refer to here as <em>next word prediction</em>. We wanted to use ML to help our Application Developers, who use our low-code Lifion Development Platform, to be more productive when building the Next-Gen HCM Solution that we provide to our clients. Similar to “Smart Reply” and “Autocomplete” features when writing emails and searching online, our feature here helps provide suggestions in our development tooling to help them build with greater velocity and best practices in mind.</p><p>In the quote above, the end result of constructing a computer program is an artifact which is usually called an <em>ML model </em>or for sake of brevity, <em>model. </em>The experience is data from the past and the improvement is quantified with respect to some task at hand. The model improves during the <em>training phase</em> by learning patterns from the data. We later use the model during the <em>inference phase</em> to make predictions.</p><p>In-depth ML knowledge will not be assumed for this article. We will motivate understanding of technical concepts via intuition, give informal definitions, and point the interested reader to other sources for more details. The target audience is anyone who is curious to learn more about the approach we take for building models for developing a new feature and ML practitioners who want to learn more about the theoretical aspects<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*UKIHA2AHtB9WPG-KrfwSZg.png" /><figcaption>source: <a href="https://xkcd.com/1838/">https://xkcd.com/1838/</a></figcaption></figure><h3>Problem statement</h3><p>The problem of predicting the next word in a sequence is encountered in a lot of scenarios. While typing text, our phones and laptops are able to predict what is the next word we are going to type. These features have become more common in recent times as models have come a long way in learning patterns in human language. There are many standard libraries that can be used to develop these models and these language models are usually trained on a large corpus such as Wikipedia. <a href="https://projector.tensorflow.org/">Here</a> is an example of a nice visual representation of some patterns that a language model can learn. In the Jupyter notebook below, we show how to load a previously trained model, feed in “<em>The weather is getting” </em>as input to the model and see that it predicts<em> “better”. </em>All this takes less than 10 lines of code. Cool! Example code for the training phase is more involved and can be found <a href="https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8b640d837be1de331985490730f6e19/href">https://medium.com/media/c8b640d837be1de331985490730f6e19/href</a></iframe><p>In the output of the last cell, we see each word is mapped to a unique token. The total number of tokens in the model is the size of the vocabulary.</p><p>The size of the vocabulary for the model we used in the example above is about 50,000. For the actual feature we were building, we were not working on human language. Our language consisted of metadata and the vocabulary size was much smaller. For illustrative purposes, let us think of the vocabulary size to be 50.</p><p>Putting all this together formally, our problem looked like the following-</p><p>1. During the training phase we observed a set of <em>m</em> sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*WPtxR9M_NzbW0itD8dFlnw@2x.jpeg" /></figure><p>2. Each <em>Sᵢ</em> was an ordered sequence of alphabets of length <em>f(i)</em> generated by some underlying probabilistic process from a vocabulary <em>V</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*v88swjdLNjywO5oBcKvyoQ@2x.jpeg" /></figure><p>3. We used the <em>m</em> sequences to construct the model. During the inference stage, we were given a partial sequence of alphabets and the model predicted the next alphabet like the following-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*610SqoVkabqHfGWJcasDXg@2x.jpeg" /></figure><p>where the sequence of <em>aᵢ</em>’s generated during inference phase was generated by the same underlying probabilistic process that generated the data during the training phase. This is called i.i.d. assumption (independent and identically distributed) and is necessary to make sure that the same patterns are present during the training and inference stages.</p><h3>Feature</h3><p>For our feature, the words were not typed on a keyboard. They were selected from a list of options in a web-based graphical user interface similar to how we may select items from a drop down list. The model performance was crucial to the value of the feature and there were various metrics to quantify it. One potential metric could have been whether the prediction made by the model was what the user actually ended up selecting during the inference stage. This is a measure of <em>accuracy</em>. When we did some data exploration, we noticed that some words were more frequent than others, i.e. <a href="https://link.springer.com/referenceworkentry/10.1007%2F978-0-387-30164-8_110"><em>class imbalance</em></a><em>.</em></p><p>As the most frequent classes were already easily accessible to the end user in our user interface, we felt it would not provide a lot of value to the user by predicting them. Hence, if a model keeps predicting the most frequent class all the time, it would not add much value though the accuracy of the model may be high. Let us take a closer look at the metric we chose to optimize that we felt would provide better value for our use case and how we estimate it.</p><h3>Metric</h3><p>The standard practice for estimating any metric is to split the <em>m</em> sequences during the training stage into <em>training data, validation data, </em>and <em>testing data</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEzYfDb6XySli40Fc1WfZg.png" /></figure><p>The model is trained and fine-tuned using the training and validation data respectively. We use the testing data and the trained model to construct the test set which is composed of test samples that look like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*5z-AUwGsYyD5wc_F5QL8Bw@2x.jpeg" /></figure><p>We then use the test set to estimate the metrics. Suppose we are interested in the accuracy of a model, we can easily measure this by comparing the predicted value and actual value for each test sample in the test set. For a given sample if the predicted value and actual value agree, then we say that the sample is correctly predicted and the accuracy of the test set is simply the fraction of test samples that are correctly predicted. In other words, it is the number of correctly predicted samples divided by the total number of samples. The accuracy <em>ø</em> of a model <em>M</em> on a test set is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*pk5w266UscJ8rTMhN1Vwbg@2x.jpeg" /></figure><p>Because of class imbalance in our dataset and other user interface considerations discussed earlier, <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">accuracy did not seem to be the relevant metric</a> for us to optimize. The main problem with accuracy is that it places less weightage on the classes with low frequency. On the other hand, if we focus on making the accuracy of each class to be high, it will make sure that we are not ignoring the classes with low frequency. Let us quantify this metric more carefully. For a fixed class <em>c</em>, we first take the subset of the test set whose actual value is <em>c</em>. The fraction of correctly predicted samples in this subset is the accuracy of the model on a test set for the class <em>c </em>defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*m_SDpAPLYZMnwAX6ISCpNg@2x.jpeg" /></figure><p>The average of this quantity over all the classes is a metric which gives equal weightage to each class. This gives us the metric called macro-averaged accuracy denoted by <em>µ</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*8t3PB-BeeGJw_8FYy8cdqg@2x.jpeg" /></figure><p>We can construct a validation set and train set similar to how we constructed the test set. We denote the value of the metrics in the validation set and train set as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*zOp_O9J-ntkBVltNEy-4LQ@2x.jpeg" /></figure><p>Finally, we denote the expected value of these metrics during the inference stage as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*_kWMI8iUrCseXsYLxjwxPA@2x.jpeg" /></figure><p>As mentioned earlier, we will focus on <em>µ </em>for the rest of the article. But most results can be rephrased for other metrics. Remember that <em>µ</em> can be computed exactly for a given model for a given test set and this serves as an estimate for how well the model will perform during the inference stage under the i.i.d. assumption. What happens during the inference stage can not be foreseen and we can only talk probabilistically. The expected value of the metrics in the inference stage is equal to the expected value of the metrics computed on the test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*cv0rB3v6kjjutMzLtaQcEw@2x.jpeg" /></figure><p>We next jump into the training stage, where we discuss the procedure for constructing a model by optimizing the metric on the data.</p><h3>Training stage</h3><p>There are many libraries (such as <a href="https://en.wikipedia.org/wiki/Scikit-learn">scikit-learn</a>, <a href="https://en.wikipedia.org/wiki/TensorFlow">TensorFlow</a>, and more) which give us the ability to build out models. We have to make decisions on which library or set of libraries we want to try out for solving our problem. To make a careful decision, we must first understand the problem we are trying to solve and the methods that these libraries employ under the hood.</p><p>Getting into the details of all the libraries and their underlying methods would be too much detail for this article. If you want to dive into more details on these methods, see <a href="https://www.deeplearningbook.org/contents/ml.html">here</a> [1] and <a href="https://developers.google.com/machine-learning/crash-course">here</a> [3] for a theoretical and practical treatment respectively. There are many intricacies involved but at the same time there are some general principles that are common across all these methods. They abstract the model training problem in some mathematical framework such as support vector machines, bayesian models, and so on. In the mathematical framework, the problem translates to the problem of optimizing a mathematical function (referred to as <em>loss)</em> which in turn depends on the metric we want to optimize. We then use appropriate optimization techniques such as linear programming, gradient descent, and so on to optimize the loss using the data to construct a model. For a fixed mathematical framework, the set of all potential models that it could represent is denoted by <em>∏</em> and the end result of the optimization is one of the models in this set and is denoted by <em>Mₒ</em>.</p><h3>Model selection</h3><p>The level of complexity that we expect from a given feature gives us a sense of the complexity of the mathematical framework that is necessary and that in turn affects which library we pick for building our models. There are many variables at play here and it is very difficult to foresee which model is going to work for our feature and datasets.</p><h4>Practice</h4><p>The general approach is to try out models of various levels of complexity depending on the data and evaluate them using our metric. Every time we try a model and evaluate, we gain some knowledge and use this to make a more informed choice of what we want to try next. Experimental results computing <em>µ</em> for various models on <a href="https://github.com/mano-lifion/blog-sample-data">sample data</a> is shown below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c1f633b9858362f3c0e3124094ef403/href">https://medium.com/media/7c1f633b9858362f3c0e3124094ef403/href</a></iframe><p>In practice, the model that gives the best value of <em>µ</em> on our validation set is picked and further hyperparameter optimization is carried out. This is called cross-validation. This leads us to the following <a href="https://quoteinvestigator.com/2015/08/30/practice/">folklore</a> question:</p><blockquote>That works very well in practice, but how does it work out in theory?</blockquote><h4>Theory</h4><p>On a more serious note, let us look at the theoretical justification behind what happens in practice. We trained a bunch of models using the training data, computed <em>µ</em> for these trained models on the validation data and picked out the model with the highest value. How do we relate this to the model performance during the inference stage? There are many ways to think about this relationship and one of them is using <em>generalization bounds</em>. At the end of the day, these bounds are a formulation of <a href="https://en.wikipedia.org/wiki/Occam%27s_razor">Occam’s razor</a> and they serve to make sure we do not overfit on data during the training phase. In other words, we want a model that does well for our feature during the inference phase and anything that we do during the training phase (splitting data, computing metrics, selecting models, etc.) should serve to optimize for it.</p><p>Some common types of generalization bounds are VC dimension bounds and <a href="https://en.wikipedia.org/wiki/Rademacher_complexity">Radamacher complexity bounds</a>. The Radamacher complexity bound states that with high probability over the data generation process, the following bound holds</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*XBLmYCTKvhShwW2D2V1WFw@2x.jpeg" /></figure><p>Let us go over the terms left to right-</p><ol><li>The first term is the expected value of <em>µ</em> during inference. [Inference <em>µ</em>]</li><li>The second term is the exact value of <em>µ</em> evaluated on the train set. [Training <em>µ</em>]</li><li>The third term is the Radamacher complexity. It is a measure of the representational capacity of the underlying mathematical framework<em>. </em>It is dependent on a set of all potential models that it could represent <em>∏. </em>It also depends on the data generating process and the number of samples in the train set. [complexity term]</li><li>The last term contains ∆ which quantifies the high probability with respect to which the bound holds and <em>n</em> which is the number of train samples. We usually think of <em>n</em> being large enough that we can just ignore this term completely. [what we can ignore under reasonable assumptions]</li></ol><p>Some things to note:</p><ol><li>The bound holds for any model <em>M </em>and in particular the optimized model <em>Mₒ.</em></li><li>We do not give the bound in full generality. For instance, we can replace the <em>µ</em> with other metrics, the set we evaluate on need not be the train set, etc.</li></ol><p>This bound motivates us to optimize for the entire right hand side (RHS) instead of just the training <em>µ </em>term. For some mathematical frameworks, this can be incorporated into the loss function during training. This gives us a way to build principled algorithms that generalize to new data and gives theoretical justification for the concept of regularization. Libraries such as sklearn contain many algorithms (like linear support vector classification) that do this in the loss function and hence use the train set to optimize the entire RHS. When comparing models built out by different frameworks, picking the model that gives the best value of <em>µ</em> on the validation set can also be motivated as optimizing for the entire RHS of the bound. In general, these bounds give us a way to motivate model choice decisions made in practice.</p><p>Finally, we have not leaked any information from the test set into our model selection. Hence, we can get an unbiased estimate of the performance of the model during inference using the following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*cv0rB3v6kjjutMzLtaQcEw@2x.jpeg" /></figure><p>The complexity bound in this section is paraphrasing Theorem 3.5 in Chapter 3 of <a href="https://mitpress.mit.edu/books/foundations-machine-learning">Foundations of Machine Learning</a> [2]. This bound is for binary classification but a similar bound for the multi-class regime can be found in the later chapters.</p><h3>Our experiences</h3><p>Oftentimes, when developing ML models for features in the industry, practitioners tend to take shortcuts such as splitting the data into two parts (train and test) instead of the three (train, validation and test) that we mentioned earlier. Taking such shortcuts could mean that practitioners end up using the same dataset for validation and testing. This is an obvious case of information leakage and there are methods such as nested cross-validation to mitigate the effects of such leakage.</p><p>When iterating on models over longer time periods, data is being collected and models are being updated continuously. Every time we update a model, we care about how the model is going to perform during the inference stage. All our experiments should be structured towards optimizing and estimating this performance. However, model updates affect the way users interact with the system which in turn affect the data being generated (i.e. the data distribution has evolved and it is no longer iid) and all this is happening repeatedly. In this case, how do we partition the data and compute the metrics? The solution is not as simple as making a new data split on all the available data as the data distribution may be changing over time and there are other costs involved. For instance, conducting all the previous experiments on the new data split may require a lot of effort (time, compute power, etc.). Such situations lead to complex cost-benefit tradeoffs where practitioners may be forced to take decisions which leads to more nuanced forms of information leakage. Even in these situations, empirically computing some of the terms of the bound in the previous section over time helps us to get a better sense of what is changing and hence guides us towards making improvements to the model.</p><h3>Conclusion</h3><p>Working in our fast-paced environment at Lifion requires us to be agile and developing ML models is no different. In such an environment, delivering in an iterative way and with high velocity can be challenging, at times making it necessary to make pragmatic choices. Rigorous results from the theoretical world guide us to build models in a principled way that deliver better experiences for our clients. We use our agile approach as an opportunity to continually iterate towards striking the right balance between velocity and rigor when we develop models for our features.</p><h3>Additional reading</h3><p>If you’re interested to read further into these topics, I recommended the following resources. The chapter on machine learning basics in [1] gives a concise introduction and does an excellent job of giving intuitive explanations while setting things up in theoretically solid footing. [2] gives a thorough introduction to the theory behind ML and [3] gives a quick practical guide to ML. Enjoy!</p><p>[1] <a href="https://www.deeplearningbook.org/">Deep Learning</a> by Ian Goodfellow, Yoshua Bengio and Aaron Courville</p><p>[2] <a href="https://mitpress.mit.edu/books/foundations-machine-learning-second-edition">Foundations of Machine Learning</a> by Mehryar Mohri, Afshin Rostamizadeh and Ameet Talwalkar</p><p>[3] <a href="https://developers.google.com/machine-learning/crash-course">Machine Learning Crash Course</a> by Google</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57b1bc9299c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/machine-learning-at-lifion-57b1bc9299c4">Machine learning at Lifion</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting the Dots]]></title>
            <link>https://medium.com/lifion-engineering/connecting-the-dots-e5af4ab42e92?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/e5af4ab42e92</guid>
            <category><![CDATA[technical-design]]></category>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[technical-specification]]></category>
            <category><![CDATA[developer-experience]]></category>
            <dc:creator><![CDATA[Vadim Solodukhin]]></dc:creator>
            <pubDate>Wed, 21 Apr 2021 18:25:19 GMT</pubDate>
            <atom:updated>2021-04-28T18:49:35.396Z</atom:updated>
            <content:encoded><![CDATA[<h3>Connecting the Dots: from Technical Design to Documentation</h3><p>Designing software can often involve napkin sketches, sticky notes, and whiteboarding to initially explore, visualize, and communicate an idea. So, what’s next? Creating a technical design document seems to be a no-brainer. But what really requires some brain activity is <em>how</em> to make it happen. This is a common challenge that many companies and individuals have faced and written about, here are some good examples:</p><ul><li><a href="https://stackoverflow.blog/2020/04/06/a-practical-guide-to-writing-technical-specs/">A practical guide to writing technical specs</a></li><li><a href="https://www.range.co/blog/better-tech-specs">How to write a better technical design document</a></li><li><a href="https://www.freecodecamp.org/news/how-to-write-a-good-software-design-document-66fcf019569c/">How to write a good software design doc</a>.</li></ul><p>However, we quickly realize that there is no silver bullet to effective documentation, for example how should the developer’s setup, workflow, and visualizations be married and expressed to readers without creating a confusing mess? A document might be really well structured, but readers won’t necessarily be able to grasp the entire picture.</p><p>Also, how do we deal with the fact that a design document might become obsolete after release if it’s not kept up to date? How do we ensure what may be documented or <a href="https://en.wikipedia.org/wiki/Self-documenting_code">self-documented</a> in code complements what is in the technical design document? And finally, how can it be turned into a useful reference, ideally for the long term?</p><p>In this article, I’d like to share a bit about how we have tackled these problems in our group at Lifion, by ADP.</p><h3>A developer’s experience of documentation</h3><p>As a developer, I imagine in the past while ploughing through somebody else’s design doc you may have ended up scratching your head once or twice.</p><h4>Not all documents are created equal</h4><p>When new to a team, it’s not uncommon to spend much time reading documentation, trying to relate it to and understand the relevant code base. And depending on the author and level of organizational standards, documents can vary sometimes quite substantially. So we are sometimes forced to start not with understanding the actual feature in question, but with the structure of the document itself.</p><h4>Recursive context-switching</h4><p>Gaps in the design can lead us to jump back and forth between the code and a document (and even within the document itself!) Having fun?</p><h4>You can’t step into the same river twice</h4><p>Uh oh, it looks like the code had diverged from the design document since it was written! And maybe even the person who wrote the code or document has left the company… Now is the time to hone Git skills because we will be going commit by commit to understand the design intentions and how to work with the code.</p><h4>Rube Goldberg machine?</h4><p>OK, I got the idea of the code, how do I start working with it? The next step is to build your own developer setup. What are the steps? What components do I have to set up and how?</p><p>‘I would do things differently’ — one might say after figuring everything out. But when writing a document, without noticing it can be easy for anyone to fall into the same pitfalls.</p><h3>One doc to rule them all</h3><p>Now when we have clearly identified the list of grievances, we can define some guidelines for our design documents:</p><ul><li>Use the same structure and tools for all documents</li><li>Provide a visual representation of the developer setup: all prerequisites and their settings needed to start the development</li><li>A consumer of the design document should be able to understand its logic without examining the code</li><li>More images, less code snippets</li><li>Any subsequent changes should be reflected in the document</li></ul><h4>Use the same structure and tools for all documents</h4><p>Having all documents adhere to the same structure and be developed using the same tooling gradually shortens the time it takes for a reader to understand the document’s essence. As an example, some structure we try to follow:</p><p><strong>I. Overview of the existing behavior</strong></p><p><em>Show the area(s) where your changes are to take place, completed with the current wireframes. For example:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*76POxUan23b3_jCh" /></figure><p><strong>II. Use cases</strong></p><p><em>Provide a full list of use cases, use wireframes to display the workflow from the user’s standpoint</em></p><p><strong>III. Updates to prerequisites and their settings</strong></p><p><em>Clearly depict (possibly using mockups) updates that are required for a development environment</em></p><p><strong>IV. Workflow</strong></p><p><em>The heart and most complicated part of the design document. The goal is to put together a diagram that describes the new process with the changes clearly indicated.</em></p><p><em>The diagram might include a limited number of snippets to show the key data structures, their usage, and impact on the visualization.</em></p><p><em>Aim to use the same flowchart-generation software across the board (</em><a href="https://app.diagrams.net/"><em>diagrams.net, formerly draw.io</em></a><em> is a good example.) Compile and share a vocabulary of the visual elements, for example:</em></p><ul><li><em>all affected code areas</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/0*6Llwk-1xQrIX4jP0" /></figure><ul><li><em>all changes</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/0*k_-5280DHbnPabRt" /></figure><p><em>Where appropriate, make your diagram interactive. A reader should if possible be able to collapse or expand components. This will make the diagram multi-faceted, allowing consumers to focus on specific paths and details of interest:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*Z8ceyo9Nh2Q_mm9d" /></figure><p><strong>V. Conclusion</strong></p><p><em>Use this section to list the proposed changes; when a planned feature is broken into work items with corresponding implementation tickets, the links can be pasted here, for example:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/0*Ak4jlJ-1ZBtOc_bw" /></figure><h4>Provide a visual representation of the prerequisite settings</h4><p>If the new feature requires changes in the developer’s setup (in <strong>I.</strong> above) the readers of a document will be thankful for the images on top of the written instructions.</p><h4>A consumer of the design document should be able to understand its logic without examining the code</h4><p>No more jumping between the document and the code: the workflow diagrams in the design document should be clear and concise, eliminating the need to examine the code base. Use your peers as reviewers: they will point out the gaps and the areas that are not clear enough.</p><h4>More images, less code snippets</h4><p>It’s important to assume that a document will likely be used as a reference by new team members when working with a given code base. Ideally, they should be able to follow it image by image to learn, get set up, and be productive with the code. It is up to the writer to make it as easy as possible to understand.</p><p>I typically recommend that code snippets be used only if absolutely necessary as they can be harder to understand for some, employ flowcharts instead.</p><h4>Subsequent changes should be reflected in the document</h4><p>This is probably the most important bullet point. The evolution of a design document does not end when it is approved for development:</p><ul><li>there might (and most likely will) be gaps discovered during the implementation stage</li><li>when development is complete, there might/will be updates in the same area to accommodate new features or fix bugs</li></ul><p>A simple rule can help keep a design document up to date. A developer that is responsible for a given code change <em>must</em> update the design document: no pull request should be merged otherwise.</p><p>This rule can help turn a technical design document into a reference document.</p><h3>Conclusion</h3><p>Large projects can suffer greatly from a lack or inconsistency of documentation. Impacts can include a longer learning curve for new team members, regression issues caused by outdated specifications and as a result, unnecessary increased project and ongoing costs.</p><p>Bridging the gap between technical design and well-maintained documentation is one of the ways to help bring delivery costs down while significantly improving developer productivity and experience.</p><p>What has been your experience regarding technical documentation and what approaches have you seen work well? If you’re working on or are interested in solving similar challenges then let’s connect, we’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5af4ab42e92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/connecting-the-dots-e5af4ab42e92">Connecting the Dots</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring CI/CD Pipelines]]></title>
            <link>https://medium.com/lifion-engineering/measuring-ci-cd-pipelines-226b2ddc8a0a?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/226b2ddc8a0a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Sivaraman Hari]]></dc:creator>
            <pubDate>Thu, 04 Feb 2021 15:42:26 GMT</pubDate>
            <atom:updated>2021-02-04T15:42:26.121Z</atom:updated>
            <content:encoded><![CDATA[<p>The aim of <a href="https://en.wikipedia.org/wiki/DevOps">DevOps</a> is to shorten a system’s development life cycle while delivering high quality features, fixes, and updates frequently in close alignment with business objectives. Consistent high quality can be ensured via continuous measurement. Measurement is one of the most important aspects of the development cycle as it helps developers and others understand system and environment usage and any changes or actions needed as they are continuously delivering their features. Measurement also helps our leadership understand whether teams are improving and delivering on agreed goals. In this post, we list and discuss the key measurements that we use to gauge our Continuous Integration (CI) and Continuous Deployment (CD) pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*MWeARnhmjZCWwA8eY9TJHw.jpeg" /><figcaption>Tavis Ford via Flickr</figcaption></figure><h3>Background</h3><p>Here at <a href="https://flowofwork.adp.com/">Lifion, by ADP</a> we use <a href="https://www.jenkins.io/">Jenkins</a> for both <a href="https://en.wikipedia.org/wiki/Continuous_integration">Continuous Integration</a> (CI) and <a href="https://en.wikipedia.org/wiki/Continuous_deployment">Continuous Deployment</a> (CD). Using <a href="https://www.jenkins.io/2.0/">Jenkins 2.0</a>, we have built a flexible CI/CD model and this helps developers maintain their own build process without compromising on governance and security. We have different sets of preconfigured build pipeline steps based on the programming language and artifacts produced by our Jenkins projects, which we refer to as ‘<strong>Flavours</strong>’. We provide flexibility to pipelines in a way that developers can create their own Flavours, add new steps in their pipeline, or override preconfigured steps. Even within our team, we are <a href="https://en.wikipedia.org/wiki/Eating_your_own_dog_food">dogfooding</a> our CI/CD process by following the same CI/CD release processes for Flavour scripts themselves and other CI/CD modules and components. We manage more than 10,000 code projects and so we expect at least 10,000 daily Integration and Deployment builds respectively. Our microservices must pass various quality gates and successfully deploy on our testing and staging environments before they can reach our production environment. We have a multi-master Jenkins system and Jenkins agents as EC2 instances in AWS that we have configured to elastically scale up and down. We use <a href="https://www.datadoghq.com/">Datadog</a> to capture relevant metrics and present them in easy to read, aesthetic dashboards.</p><h3>Key Measurements</h3><h4>1) Productivity</h4><p>This measurement helps us understand how efficient our CI/CD processes are. <a href="https://en.wikipedia.org/wiki/Productivity">Productivity</a> is often expressed as a ratio of aggregate output per unit of input, hence the metric that we use for measuring productivity of our processes is what we refer to as their <strong>Total Success Rate</strong>. A higher Success Rate indicates the process is succeeding more often in producing its intended output.</p><pre>Total no. of successful jobs<br>____________________________ x 100 = % Total Success Rate</pre><pre>Total no. of jobs completed</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuoBzNGMjf7ao-OVkuf5VA.png" /><figcaption>Graphing CI/CD metrics in Datadog</figcaption></figure><p>With this metric and combination of filter options in Datadog we can create a useful dashboard for our monitoring needs.</p><p>For continuous deployments we also filter the metric based on the relevant deployment environment. We compare the Success Rate across environments which helps us better understand what is happening in each including identifying any potential problems that may need to be addressed.</p><p>We can also filter this metric by other dimensions relevant to us such as source code repository, release branch, Flavours, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WY4Ii4frTUUgH8uz32va7A.png" /><figcaption>Filtering metrics by various dimensions using Datadog widgets</figcaption></figure><h4>2) Time</h4><p>Another key measurement is <strong>Time</strong>. Time is money. The less time our CI/CD processes take to complete, the less time our engineers need to wait and the more build-test-deploy feedback cycles they can execute, leading to greater developer productivity and higher quality features.</p><blockquote>“The wisest are the most annoyed at the loss of time.” ― Dante Alighieri</blockquote><p>We capture job duration metrics in Datadog, which then enable us to plot the time taken for builds. We have configured the Datadog plugin in our jobs to include useful tags such as name of the pipeline step, name of the Flavour being used, and more. This then enables us to filter job durations in multiple ways, helping us better understand which specific step(s) may delay a given pipeline process or which specific Flavour(s) may be taking longer than expected to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGCgZg7gLh-gD2EFbqLPPQ.png" /><figcaption>Time dashboard</figcaption></figure><h4>3) Frequency</h4><p>Frequency helps us get insights into the stability of our CI/CD processes. The more frequently a build may be failing, the less stable the process. The metric we use here is <strong>Run Rate</strong>. We also measure Success Run Rate and Failure Run Rate to understand the frequency of Successful and Failed jobs respectively.</p><pre>No. of completed jobs / hour = Run Rate</pre><pre>No. of successful jobs completed / hour = Success Run Rate</pre><pre>tNo. of failed jobs completed / hour = Failure Run Rate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKfHID5PCRoLTsEQRnHpOg.gif" /><figcaption>Plotting rate metrics</figcaption></figure><h4>4) Quality</h4><p>We have many different quality check components in place such as Linters (Docker Lint, ESLint, etc.), Static Code Analysis (Codenarc for Groovy, Shell check for shell scripts, etc.), plus other Code Quality and Security tools (SonarQube, Prisma). Together these automated checks help us ensure the quality of the code that we use to develop our CI/CD processes. Since these quality check components are part of the pipeline steps and we are practicing dogfooding as mentioned earlier, we can easily pull such quality metrics for our CI/CD components as well as for our platform microservices themselves.</p><pre>No. of jobs with successful quality check <br>_________________________________________ x 100 = % Quality Rate</pre><pre>      Total no. of jobs completed</pre><h3>Summary</h3><p>Capturing and surfacing these measurements has helped us provide feedback to our various engineering teams for their analysis and to help inform ongoing development and enhancements. Key challenges we faced included deciding what to measure and how to analyze the results to evaluate quality and completeness of our services appropriately and to help us eliminate waste in our work. The above measurements can be applied not only to Jenkins jobs but to any processes, providing actionable insights and helping facilitate continual improvement of development and operations in an agile way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=226b2ddc8a0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/measuring-ci-cd-pipelines-226b2ddc8a0a">Measuring CI/CD Pipelines</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An SRE approach to Security Engineering]]></title>
            <link>https://medium.com/lifion-engineering/an-sre-approach-to-security-engineering-585526975ad4?source=rss----5986961e21c5---4</link>
            <guid isPermaLink="false">https://medium.com/p/585526975ad4</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[lifion]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Gabriel Rojas]]></dc:creator>
            <pubDate>Sat, 19 Dec 2020 00:46:40 GMT</pubDate>
            <atom:updated>2020-12-19T00:47:26.220Z</atom:updated>
            <content:encoded><![CDATA[<h3>An SRE approach to CyberSecurity</h3><p>Before we get into how <a href="https://sre.google/">Site Reliability Engineering (SRE)</a> approaches benefit security, I would like to give you my view on how SRE needs to empower an organization.</p><p>Many of you have heard enough about SRE and how this approach has helped organizations worldwide adopt a mindset that fosters collaboration, communication, and improves the products&#39; reliability and scalability.</p><p><em>Having this understanding, the ultimate goal of SRE is to improve and maintain customer happiness.</em></p><p>I want to take a moment to explain my personal view of SRE. At Lifion, we drive all our product capabilities with client happiness in mind.</p><p>Understanding that customer happiness is key, let&#39;s breakdown measuring SRE contributions into three specific metrics.</p><p>1. Adoption<br>2. Utilization<br>3. Satisfaction</p><p><em>Capturing metrics around those 3 areas allows SRE teams to understand the impact of their work in the organization</em></p><p>Before we get into how Security Engineering benefits from SRE approaches, I want to discuss the Equifax breach that exposed over 140 million social security numbers in 2017. If you are not familiar with the incident, this is the summary of what happened:</p><blockquote>“On March 7, 2017, a critical vulnerability in the Apache Struts software was publicly disclosed. Equifax used Apache Struts to run certain applications on legacy operating systems. The following day, the Department of Homeland Security alerted Equifax to this critical vulnerability. Equifax’s Global Threat and Vulnerability Management (GTVM) team emailed this alert to over 400 people on March 9, instructing anyone who had Apache Struts running on their system to apply the necessary patch within 48 hours. The Equifax GTVM team also held a March 16 meeting about this vulnerability.</blockquote><blockquote>On May 13, 2017, attackers began a cyberattack on Equifax. The attack lasted for 76 days. The attackers dropped “web shells” (a web-based backdoor) to obtain remote control over Equifax’s network. They found a file containing unencrypted credentials (usernames and passwords), enabling the attackers to access sensitive data outside of the ACIS environment. The attackers were able to use these credentials to access 48 unrelated databases.”<br><a href="https://republicans-oversight.house.gov/wp-content/uploads/2018/12/Equifax-Report.pdf">Equifax Breach Congressional Report</a></blockquote><p>This specific data breach could have been prevented if machine-driven controls were in place and not relying solely on humans.</p><h4>Security and SRE</h4><p>Now, let&#39;s get into Security and SRE; we will focus on Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for cybersecurity vulnerability management. Most known vulnerabilities and patches are created on any day and any time, except for Microsoft&#39;s famous <a href="https://en.wikipedia.org/wiki/Patch_Tuesday">Patch Tuesday</a>.</p><p>Enterprises currently have the challenge of patching a vulnerability within hours of discovery. There are many reasons for this; their application lifecycles do not allow to release of changes to production in hours because proper testing is needed, and making sure application functionality is not broken is as important as security.</p><p>This is where SLOs and SLIs come into play to help manage risk and help provide a good framework for vulnerability management.</p><p>The first step to take is to define the SLO. In this case, our SLO is going to be defined as critical vulnerabilities on all production accounts.</p><p>The second step is to define our SLI for the SLO described above. In this case, we can target that zero (0) critical vulnerabilities found in the last 30 days should be true in the production environments.</p><p>The third step is to work with the error budget. In our case, the team has X number of days to fix the critical vulnerability in the production environment.</p><p>Now that we have defined our SLI and SLO for this example, it is critical to have our machines send alerts when this SLI is not met and to make sure this information is available to the teams, such as via reports.</p><p>The question then becomes, what happens when there is a breach of the defined error budget. There are a couple of actions that can be taken and the most common is to stop development work and have the teams focus on fixing the vulnerabilities. Another approach used is with security gates on the CI/CD pipeline to block any new deployments into the environment.</p><p><strong>SAMPLE REPORTS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dprbzm3mh2Nh1EyXj0yeLA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D53gRYQzd31Gd11J8nTCMg.png" /></figure><h4><strong>FRAMEWORK</strong></h4><p>We can follow the 5 step framework below when reducing the vulnerability of security risks in our organizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PnWMvohL2DyJ53Mkbf_eMQ.png" /></figure><h4>Conclusion</h4><p>Providing the organization with a framework is important to reduce risks within the organization. Simply having a tool spitting out report vulnerabilities does not really help you manage this risk.</p><p>What is key to this approach is to provide <strong>actionable data</strong> to Dev, SRE, SEC teams, and senior management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=585526975ad4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lifion-engineering/an-sre-approach-to-security-engineering-585526975ad4">An SRE approach to Security Engineering</a> was originally published in <a href="https://medium.com/lifion-engineering">Lifion Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>