<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John R. Harris on Medium]]></title>
        <description><![CDATA[Stories by John R. Harris on Medium]]></description>
        <link>https://medium.com/@john.r.harris?source=rss-2798b4f73193------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JP_5_6iL_SsIjwkHVU7Ffg.jpeg</url>
            <title>Stories by John R. Harris on Medium</title>
            <link>https://medium.com/@john.r.harris?source=rss-2798b4f73193------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:57:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@john.r.harris/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Digital Transformation and Continuous Change]]></title>
            <link>https://medium.com/span-digital-insights/digital-transformation-and-continuous-change-c56892a2a051?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/c56892a2a051</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Sun, 16 Aug 2020 19:22:29 GMT</pubDate>
            <atom:updated>2020-08-16T19:22:29.381Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Digital transformation increases business agility but the price is continuous change. It takes new strategies and tactics to be successful.</p><p>The spectacular success of Facebook, Amazon, Apple, Netflix, Google, and other digitally-enabled companies, has encouraged many businesses to seek similar agility through <em>Digital Transformation</em>. Many organizations are striving to become more agile — able to rapidly respond to market opportunities and demands — by converting existing, discrete, often manual, processes into digital, automated, continuous processes. The result is agility, but the unexpected price is having to manage continuous change. New strategies are needed for managing the uncertainty, and risk inherent in ever-changing systems, and new tools are required that provide situational awareness, and decision support to those grappling with continuous change.</p><h4>Digital Transformation Strategies</h4><p>Digital transformation strategies require the conversion of existing manual, or semi-digital, processes to fully automated services that orchestrate core business activities. Digital transformation requires businesses to become proficient at software development and digital service operation. Successful digital companies draw their success from a loose collection of business and engineering practices that enable their agility. Some of these practices, and their consequences, are described below.</p><p><strong>Agile Development Practice</strong>s have flattened large organizations thereby creating many small autonomous teams that each own a portion of the overall offering. These autonomous teams can closely monitor their narrow area of responsibility and quickly respond to sudden events or emerging trends. When an organization has more than a handful of self-organizing agile teams collaborating on a single solution, it becomes more difficult to maintain a coherent strategy. Trade shows, earnings calls, and competitors’ new offerings demand a coordinated response. But, when each team is focused on different objectives and must deal with the continuous distraction of maintenance, and emergency support, predictable product management becomes a challenging task.</p><figure><img alt="Agile Process. Typical agile process with short iterations, daily stand-ups and frequent releases" src="https://cdn-images-1.medium.com/max/600/1*iR8UXhIGE6BPW0heynB-cg.png" /><figcaption><strong>Agile Process</strong>. Typical agile process with short iterations, daily stand-ups, and frequent releases</figcaption></figure><p><strong>Cloud Deployment </strong>of solutions has allowed many companies to focus more closely on their core competencies while outsourcing less differentiated features of their offering. Instead of buying and operating hardware organizations can outsource their compute, storage and network infrastructure to IaaS providers. Instead of deploying and managing common applications, they can compose offerings from third-party PaaS managed applications like databases, message buses, and frameworks. Instead of building common services that won’t differentiate their offerings they can delegate entire business services to SaaS providers, such as billing and credit card processing. The resulting hybrid solutions deliver more, and better, features, more reliably than any one organization could develop alone. Hybrid solutions allow organizations to focus on what their customers want, not the digital table stakes needed to deliver a basic service.</p><p><strong>Microservices</strong> are independently deployable units of business capability that can be “podded” (grouped into gangs that share the work) to increase reliability and scalability. Microservices collaborate with each other to provide features and have enabled further acceleration by modularizing development, deployment, and operation of services. Microservices are usually owned by a single team and can be upgraded independently of other deployed services thus freeing the team that owns the service to move at its own pace. By breaking up monolithic solutions into many microservices large organizations have empowered small teams to take ownership and act without the usual constraints of organizational scale. However, microservices can increase complexity making it hard to diagnose problems.</p><p><strong>Containers</strong> standardize the deployment and orchestration of applications and lend themselves to automation. Container orchestration software can automate the deployment and upgrade of highly complex solutions. Instead of directly changing the deployment, system engineers now modify the specification and let the orchestration software take care of deploying the changes. This “Infrastructure as code” approach can massively increase consistency and reliability, thus enabling more frequent upgrades of partial or complete systems, without interruption to the service.</p><p><strong>Open Source</strong> adoption is intrinsic to digital transformation. Most of the tools and many of IaaS, PaaS, and SaaS offerings are based on, or enabled by, open-source projects. Open source offers many advantages, more eyes looking at the source code makes for more robust, secure solutions, with less defects. Open source projects move quickly, major versions are released regularly and patches, to fix defects, are frequent. Most significantly open source projects tend not to support many prior versions. If you are on an older version the only way to get a fix is to upgrade. There is no extended support for open-source software. Adoption of open-source software, which is often a requirement for digital transformation projects, requires you to mount a never-ending, upgrade treadmill.</p><p>By bringing together, previously separate, development and operations teams and processes, into a single, unified <strong>DevOps</strong> approach the efficiency of the overall service delivery life-cycle can be vastly improved. A DevOps mindset means system reliability engineers and software developers collaborate to ensure the delivered service is continuously monitored and, if necessary, frequently upgraded to ensure it remains, at all times, within acceptable service levels. DevOps teams seek to automate the flow of software releases through the software delivery tool chain. Continuous Delivery Pipelines sequence delivery activities ensuring they are consistently and rapidly executed, steps are never missed, and software can be released in minutes vs the days, weeks, or even months of traditional manual approaches.</p><figure><img alt="The DevOps process model showing the continuous cycle of plan, code, build, test, release, deploy, operate, monitor. The comb" src="https://cdn-images-1.medium.com/max/600/1*zjfqEtNiTl8Hvk80i3qGhA.png" /><figcaption>The <strong>DevOps Process Model</strong> showing the continuous cycle of plan, code, build, test, release, deploy, operate, monitor. The combination of development and operations teams with the practices described above have shortened this cycle from months to days.</figcaption></figure><h4>Where is it all Leading?</h4><p>When agile, DevOps teams create microservices composed of open-source software and deploy them to the cloud using containers, multiple accelerators combine. By leveraging Iaas, Paas and SaaS providers teams can focus exclusively on high-value differentiators. This can result in highly agile solutions that can rapidly adapt to market demands. The agility that allows an organization to make 50 controlled releases to production on a single day is, no doubt, a potential competitive advantage. But, such speed means the production system is continually changing. Tomorrow’s systems are guaranteed to be different from today’s. Continuous daily or weekly change of this nature accumulates rapidly and creates its own problems.</p><p>Which brings us to the elephant in the room, seldom mentioned in articles on digital transformation. Staff Turnover! The average tenure of engineering staff in Silicon Valley is less than 24 months and not much better elsewhere. This means that half of most organizations’ engineering staff compliment has less than 24 months experience at the company, and 1/3rd have less than nine months.</p><p>Loyal staff must train newcomers to extend and maintain partially understood, poorly documented systems that must be continually upgraded to keep up with the latest security patches and open source releases. In such environments, it is often easier to add a new service than try to enhance an existing poorly understood one. This can lead to a proliferation of new services that replicate, or partially replicate, an ever-accumulating collection of legacy services.</p><p>Enterprise transformation initiatives create business agility by leveraging the technologies and the practices described above to enable and empower organizations to rapidly evolve their offerings in the face of fierce competition. But, the same rapid change creates legacy technology faster than ever before! The modern enterprise typically has as many services in production as it has engineers to maintain them and both the engineers and the services they maintain are rapidly changing.</p><p>Recognizing and accepting this state of affairs as the “new normal” and then turning it to your advantage is the second step of digital transformation.</p><h4>Strategies for Managing Continuous Change</h4><p>Many industries already employ strategies for effectively functioning in environments characterized by continuous change, uncertainty, and risk. There is much to be learned from these examples; the investment strategies of financial institutions, clinical decision making in hospitals, mission command approaches employed by the armed forces, and risk management strategies of insurance companies all have lessons for companies embarking on digital transformation strategies. We have found three strategies, in particular, to be highly valuable when starting to grapple with these issues.</p><p>- Situational Awareness and Decision Support<br>- Knowledge Management<br>- Portfolio Management of Technology Stacks</p><h4>Situational Awareness for Software Engineers</h4><p>Designers, developers, and operators of solutions must understand the systems they are asked to manage. In rapidly changing environments decision-makers require good situational awareness — access to accurate, timely, structured information, that they can use to understand the situation and then draw inferences and forecast likely future states. Given good situational awareness, they can decide on an effective course of action. Instead, we often find we are overwhelmed by rapidly changing low-level disorganized data that is usually incomplete, out of date, or just wrong. Fortunately, this problem has been faced before in a wide range of environments; vehicles, financial systems, aircraft cockpits, clinical systems, and process control rooms must all provide effective situational awareness and decision support. Similar capabilities must be provided for engineers who manage and develop today’s ever-changing software systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*eMGRWqms3AgddoG9iL0FIA.png" /><figcaption><strong>Situational Awareness and Decision Support. </strong>A process model showing the three levels of situational awareness; observe, understand, and forecast that supports decision making and action, along with the individual and environmental factors that influence decisions. Based on Endsley’s model of SA</figcaption></figure><p><strong>Situational Awareness Level 1</strong> Ensure engineers can <strong>observe</strong> the elements in the situation. Engineers must have access to a snapshot of the current systems, service, and technologies, that comprise the solution, and their co-dependencies. In addition, they must understand how the teams, and individuals in their organization influence and interact with these elements. Armed with such an inventory they can begin to form opinions and hypotheses.</p><p><strong>Situational Awareness Level 2</strong> Help decision-makers <strong>understand</strong> the situation. Knowing the static components is a start but understanding how they interact and affect each other is the next stage of situational awareness. An understanding of the dynamics of the solution, over varying timescales, is required to make sound decisions about the evolution of the system over days, weeks and months.</p><p><strong>Situational Awareness Level 3</strong> Provide tools so engineers can <strong>forecast</strong> the future state of the system based on current trends and planned interventions. This capability supports the creation and evaluation of alternate strategies and allows for what-if scenario planning and financial modeling.</p><p>Once engineers have a good situational awareness based on observation, understanding, and forecasting they can make informed decisions. Decision Support systems help them formulate and select strategies for specific use cases like; onboarding new employees, designing system enhancements, or planning upgrades of aging technologies. Decision support system further help execute these complex use cases in a consistent manner and create a record of planned and completed tasks for higher-level monitoring purposes.</p><h4>Knowledge Management</h4><p>Knowledge lives in the minds of people, while information is stored in some medium. Knowledge Management is the process of transferring information from a storage medium to the mind of a person who needs it, or capturing knowledge from one person and converting it into information so that it can be shared with others. The average employment tenure of a software engineer is currently less than 24 months. New engineers must be on-boarded and up-skilled efficiently to reduce the time it takes until they can make productive contributions. At the same time, the valuable knowledge of experts must be harvested and transformed into information so that others can learn for their experience.</p><p>Knowledge management as a strategically important business process that must be managed holistically from the first day an employee joins the organization to the day they leave. Modern agile practices have allowed far too much knowledge that should be explicit — captured as information, to remain tacit — undefined, and experientially learned. As a result, when senior staff leave organizations irreplaceable knowledge leaves with them. Successful services that have endured for more than the average tenure of an employee become harder and harder to maintain as knowledge of their design, construction, and operation gradually evaporates. The band-aid solution of providing a wiki and hoping staff will ‘do the right thing’ and document their work just leads to a chaotic, repetitive mess that is not fit for purpose. Information and knowledge must be managed and curated by dedicated staff.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*1UeJj25yqtsD7n5WLXKRkw.png" /><figcaption>An I<strong>nformation Asset Life-cycle</strong> showing how knowledge is acquired, organized, shared, delivered to the point of need for use where it is converted back to knowledge again, subsequently being evaluated and revised. This life-cycle is key to good knowledge management</figcaption></figure><p>Engineering Knowledge can be divided into three main categories based on the intended audience and scope of distribution. Adapting the information asset life-cycle for each type of knowledge is an essential part of a good strategy for knowledge management.</p><p><strong>External information</strong> is targeted at third-party partners and customers. This is usually the best-managed knowledge, it is structured and formally published comprising manuals, instructional content, and training courses. This content must be kept in line with major and minor product and service releases and is usually planned along with those releases.</p><p><strong>Inter-team information</strong> is targeted at other teams within the organization — teams that consume the deliverables of the team generating the information. In flat organizations with many teams, there are many such dependencies. Overviews, instructions on getting started, key concepts, API specifications, best practices, typical usage scenarios and use cases, common problems, FAQs, and other such documentation. The benchmark for this type of content has been set high by the many successful open source projects that software developers use on a daily bases. If a team expects their colleagues to use their solution it must be documented to at least the standard of the open-source modules and libraries they already use. This area is the worst defined in most enterprise organizations today, and yet with the flattening of organizations and adoption of micro-services, this area is the fastest-growing.</p><p><strong>Intra-team knowledge</strong> is the team’s internal operational knowledge, it includes, day-to-day, plans and road-maps, lists of defects, and tasks, This knowledge is the most volatile and is often kept tacit or can be inferred from well-written code and a few design documents. There are many specialized solutions for this type of information in use today from wikis to task management systems.</p><h4>Portfolio Technology Stack</h4><p>A balanced portfolio is key to a good investment strategy, it spreads risk and allows for periodic readjustments to maintain balance. Both investment and insurance companies balance their portfolios to manage risk. A similar approach can be used for managing a tech stack in a changing environment. Like investments in a portfolio, you should re-balance your tech stack on a regular basis. Unlike investments the cost of switching technologies can vary significantly — some technologies are much harder to swap out than others. The tendency of some technologies to create lock-in <em>should</em> drive your selection strategy.</p><p>Organizations should place long term core technology bets conservatively. Before making such a choice consider the maturity of the technology, the size of the ecosystem it supports, how easy it is to hire qualified people, and the stability of the organization that stands behind the technology.</p><p>Organizations should embrace open source <em>and</em> plan to upgrade continuously. Open source is a good path to high-quality feature-rich code, but it is not free, the cost is continuous upgrades. Establish a regular cadence for managing the life-cycle of technologies, invest in keeping them up to date. If you don’t your troubles will multiply and your options for fixing them will diminish.</p><p>Organizations should provide support for teams using preferred technologies, but tolerate technical diversity. They should encourage the internal adoption of recommended technologies by providing support and training but <em>not</em> mandate the use of these technologies. Organizations should allow teams to deviate from the recommended approach, but make each team <em>accountable</em> for the cost of going it alone. This is how engineering organizations learn. Accept that some of these experiments will fail and set hard limits on how long experiments can run before showing a return. Invest in helping teams migrate off failed technologies.</p><h4>Embracing Continuous Change</h4><p>Digital Transformation is the beginning of a journey that leads to the next challenge — mastering continuous change. As companies embark on digital transformations they should look further down the road at the current challenges of companies who are already fully digital. Digital transformation is not a destination it is an enabler for business agility. The price of business agility is continuous change, but, with the correct strategy and tooling, continuous change can be harnessed and turned to a competitive advantage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c56892a2a051" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/digital-transformation-and-continuous-change-c56892a2a051">Digital Transformation and Continuous Change</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Risk Management for High-Stakes Projects]]></title>
            <link>https://medium.com/span-digital-insights/risk-management-for-high-stakes-projects-c60d61c62275?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/c60d61c62275</guid>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Fri, 27 Jul 2018 04:10:48 GMT</pubDate>
            <atom:updated>2018-08-06T17:16:42.515Z</atom:updated>
            <content:encoded><![CDATA[<p>A simplified form of risk management based on quantification and pooling of risk can be applied to software development projects. For most projects this level of risk management and tracking is unnecessary, some would even say it is an agile anti-pattern, but for high-stakes projects where failure to deliver on time, or failure to deliver a defined feature set, would be catastrophic, an approach like this is warranted. It’s up to you which is more appropriate for your project; being so agile you can dance around all risks, or buying insurance by budgeting to manage predictable risks.</p><h3>Risk Quantification and Pooling</h3><p>Successful risk management requires a consistent approach to classification and quantification of risks. While most people can identify risks, quantifying them requires some practice. The approach described here is widely used in many industries and works well. A risk is an event or occurrence that has not yet happened but has a realistic possibility of happening during the project and would have a non-trivial impact on at least one <a href="http://insights.spandigital.com/posts/setting-objectives-for-agile-projects/">project objective</a>. This definition is somewhat convoluted, but the details are important. We don’t care about risks whose probability is vanishingly small or catastrophes from which we cannot possibly recover. The first is not worth tracking and the second would lead to immediate cancelation of the project and so, is not worth managing.</p><p>Given a list of risks that are reasonably likely to happen, and not so catastrophic that they would lead to immediate cancelation of the project, we can proceed with classification. To correctly classify each risk we must define five aspects.</p><blockquote><strong><em>Context - </em></strong><em>The frame of reference for the risk, usually a phase of a project or initiative<br></em><strong><em>Hazard -</em></strong><em> The cause of an unplanned occurrence with negative consequences<br></em><strong><em>Impact -</em></strong><em> The cost of recovering from the negative consequences of an occurrence<br></em><strong><em>Probability -</em></strong><em> The likelihood of an occurrence<br></em><strong><em>Exposure -</em></strong><em> The impact, adjusted for likelihood, of an occurrence</em></blockquote><p><strong>Example risk</strong></p><blockquote><strong><em>Context -</em></strong><em> Commuting 20 miles each way, during rush-hour, freeway traffic, in a sedan vehicle<br></em><strong><em>Hazard -</em></strong><em> A fender-bender collision, being rear-ended by another vehicle (not at fault)<br></em><strong><em>Impact -</em></strong><em> Up to $3000 bodywork. No medical fees<br></em><strong><em>Probability -</em></strong><em> 1 in 100,000 (100,000 cars at rush-hour in the city with one accident a day)<br></em><strong><em>Exposure -</em></strong><em> $3000 * 1 in 100,000 = $0.03 per day = $10.95 per year</em></blockquote><p>If every commuter on the road buys collision insurance at $10.95 for the year, we can cover all likely collision risk for all commuters. Risks can also be aggregated across a portfolio of multiple different types of risk each with different probabilities. By pooling all the risks for a project in this way, we can calculate the aggregate exposure across the entire project and ensure that, while we may not anticipate the exact impact or probability of every risk, over the project lifetime we are accurate enough to manage the aggregate risk successfully.</p><p>In practice, the success of this approach comes down to our ability to identify the risks and accurately define exposure. We need to define reasonable exposure for each risk for the risk bundling approach to work. It takes practice and experience to estimate exposure successfully. For example, a typical poorly defined risk would be “The sub-contractor will be late delivering their components!” While this may be true, it is not quantified and is therefore not manageable. This risk should be redefined as follows:</p><blockquote><strong><em>Context -</em></strong><em> A three-month project with 6 staff<br></em><strong><em>Hazard -</em></strong><em> The sub-contractor will be late delivering their billing component by up to 3 calendar days Impact A 2-day delay for 4 team members ( 8 project days) Probablity 75%<br></em><strong><em>Exposure</em></strong><em> = Impact x Probablity 8 x 0.75 = 6 days LoE</em></blockquote><p>Usually, we express this as a single sentence and exclude the context as this tends not to change during the project — all risks are in the same project context. So the above risk would be expressed as:</p><blockquote><strong><em>Risk -</em></strong><em> The subcontractor will be late delivering the billing component by up to 3 days leading to a 2-day delay for 4 team members<br></em><strong><em>Impact -</em></strong><em> 8 LoE<br></em><strong><em>Probability -</em></strong><em> 75%<br></em><strong><em>Exposure -</em></strong><em> 8 x 0.75 = 6 LoE</em></blockquote><p>The difference between the poorly defined risk and the well-defined risk is in the quantification of the impact and the likelihood. But what if the subcontractor is even later? We can extend our coverage to handle further delays by adding another risk as follows:</p><blockquote><strong><em>Risk -</em></strong><em> The sub-contractor will be late delivering the billing component by between 4 and 8 days leading to an additional 4-day delay for 4 team members<br></em><strong><em>Impact -</em></strong><em> 16 LoE<br></em><strong><em>Probability -</em></strong><em> 20%<br></em><strong><em>Exposure -</em></strong><em> 3 LoE</em></blockquote><p>Together the two risks account for uncertainty in the delay and spread in probability. The impact and exposure are both defined in terms of Level-of-Effort (LoE) in days for a single team member. If you don’t like estimating in LoE, you can use story points or T-shirt sizing or whatever metric you want to quantify tasks.</p><h3>Planning to Manage Risk</h3><p>At the start of a project we identify and quantify all the risks we can and calculate the total Aggregate Risk Exposure in LoE. The total LoE for all tasks AND the aggregate risk exposure are summed and used to forecast the delivery date. By adding the risk exposure to the forecast we are effectively adding a risk buffer to the project that should cover all reasonable eventualities. If we have identified and estimated our risks correctly, the risk buffer will have been consumed by the time we finish the project — all tasks will have been completed, and all risks will have been managed.</p><p>Before starting the project, we assess the aggregate exposure as a percentage of planned task LoE using the following rule of thumb.</p><p>Low Exposure (aggregate exposure &lt;5% of total project task LoE) Moderate Exposure (aggregate exposure &gt;5% &lt;10% of total project task LoE) High Exposure (aggregate exposure &gt;10% &lt;15% of total project tasks LoE)</p><p>We aim to keep the project exposure well below 15% of the total estimated project task LoE. Experience has shown that while 15% is just about manageable, project exposures greater than this are hard to manage and generally indicate that we don’t understand the project well enough to commit to on-time or defined scope delivery.</p><p>If the risk exposure is higher than 15% of the total estimated project LoE, we continue to adjust the plan until the risk exposure is reduced to an acceptable level. Various strategies for these adjustments are described below.</p><h3>Risk Mitigation (Risk Backlog Grooming)</h3><p>The following table is used to classify each risk and assign a mitigation strategy. Risks are assigned to the table based on their probability and impact. Impact is classified “Very Low” to “Very High” based on the normalized size of the impact compared to the total project task LoE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/0*jyxHtoq2rpxGfW8M.png" /><figcaption>Risk Mitigation Strategy. All risks are assigned a mitigation strategy by assigning them to the table using their probability and impact .</figcaption></figure><h4>Mitigation Strategies</h4><blockquote><strong><em>Accept</em></strong><em> Add the exposure to the project risk backlog<br></em><strong><em>Reduce</em></strong><em> Accept the risk but develop a plan to reduce the impact as much as possible<br></em><strong><em>Transfer</em></strong><em> Try to transfer the risk to someone else or buy “insurance”<br></em><strong><em>Avoid</em></strong><em> Don’t move forward with this risk. Add it to the plan as a scheduled task (assume it will happen and plan for it) or move it until after delivery</em></blockquote><p>This strategy is applied to all risks before the plan is finalized. The project proceeds when the project risk exposure is less than 15%, and all reasonable steps have been taken to manage the risks.</p><h3>Managing the Risk Backlog</h3><p>All risks are created as tasks and added to the bottom of the project backlog with a LoE set to the estimated exposure for the risk.</p><p>As each sprint is planned,</p><ul><li>Newly identified risks should be added to the backlog</li><li>Any risk that has occurred should be moved into the sprint and should have its LoE adjusted to the full estimated impact</li><li>Any risk that is no longer possible should be closed/retired</li></ul><p>Eventually, if our risk estimation is correct, we will reach the end of the project on time with no risks left.</p><p>At the start of each sprint we should ensure the aggregate LoE for the outstanding risks is not too high. If the risk backlog exceeds 15% of the remaining task backlog then we should take steps to reduce the risk backlog using the strategy described above. In the worst case, this may lead to a change in the forecast completion date or adding staff to the project.</p><p>This approach works provided there are enough risks to average out our impact and probability estimates. As the project proceeds and the number of remaining risks reduces this method becomes erratic. Particular care should be taken at the end of a project if there are remaining risks with substantial impacts as these can prevent on-time delivery.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c60d61c62275" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/risk-management-for-high-stakes-projects-c60d61c62275">Risk Management for High-Stakes Projects</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Enduring Agile Enterprise]]></title>
            <link>https://medium.com/span-digital-insights/the-enduring-agile-enterprise-d111a41ebb74?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/d111a41ebb74</guid>
            <category><![CDATA[scaled-agile]]></category>
            <category><![CDATA[knowledge-management]]></category>
            <category><![CDATA[agile-development]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Mon, 16 Jul 2018 16:28:45 GMT</pubDate>
            <atom:updated>2018-07-16T16:28:45.762Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="http://www.spandigital.com/">SPAN</a> we work with many organizations that employ Agile at enterprise scale. Despite the many differences between these organizations, their problems are surprisingly similar. Effective software development at scale is challenging to establish and even more challenging to maintain.</p><h3>The Symptoms</h3><p>Engineers within large agile organizations are frustrated. Making decisions at a strategic, or even tactical, level is hampered by poor situational awareness. Individuals and teams find it difficult to identify the most important issues in their area of concern, let alone the solution in general. Comprehending the interactions between issues is even harder, and making projections about future states based on their best current assessment is a black art which only the most experienced, and long-serving, employees can manage effectively.</p><p>When an organization has more than a few self-organizing agile teams collaborating on a single offering, it becomes difficult to produce a coherent release. Trade shows, earnings calls, and competitors offerings demand a coordinated response, but, with each team focused on different objectives and continuously “distracted” by urgent patches, and emergency releases, predictable release scheduling becomes a near impossible task.</p><p>Onboarding new employees takes months when internal systems are an obscure alphabet soup of acronyms with no high-level map of the overall solution. When vital operational information is mixed with outdated and misleading content in unstructured wikis, new employees struggle to get traction. Favoring working code over documentation is excellent for a single, fast-moving, team working on a new solution, but when the overall system is spread across many poorly documented code repo’s that have evolved over many years, finding the information you need becomes an archaeological expedition.</p><p>I am frequently asked “Is everyone this messed up?” to which I answer “No, only the most successful companies are this messed up! It takes time and effort to get like this!” Few organizations have a solution to this <em>problem</em>. Indeed, what many perceive as a problem may be the unavoidable price of enterprise agility, steady growth, and staff turnover. Rapid adaptation to emerging needs arises from a steady churn. Evolution proceeds through a series of wasteful experiments. However, there may be a way to reduce the pain. Many of these inefficiencies have the same root cause.</p><h3>The Root Cause</h3><p>The <a href="https://hbr.org/2018/05/agile-at-scale">Agile Enterprise</a> has evolved to exploit the advantages of agile development. Large organizations have sub-divided into agile teams that each independently build and operate their portion of the solution. This approach is susceptible to problems that build slowly over time, gradually becoming chronic. Technical debt accumulates, crucial knowledge evapourates, and teams struggle to adapt their solutions to a continually coevolving environment. Developers must continually re-educate themselves and their users. All these problems take time to emerge they are a function of enterprise longevity, organizational growth, and staff turnover.</p><p>The Agile Manifesto famously stated:</p><blockquote><em>We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:</em></blockquote><blockquote>Individuals and interactions over processes and tools</blockquote><blockquote>Working software over comprehensive documentation</blockquote><blockquote>Customer collaboration over contract negotiation</blockquote><blockquote>Responding to change over following a plan</blockquote><blockquote><em>That is, while there is value in the items on the right, we value the items on the left more.</em></blockquote><p>With the experience of enterprise-scale agile, we can see that the Agile Manifesto is a small-scale solution that does not solve long-term problems. Specifically, the Agile Manifest does not solve the problem of staff turnover. Software Engineers typically stay in their jobs for 2 to 3 years before moving on. Most software organizations experience something like a +15% turnover per year. Agile projects lasting 6 to 9 months can often avoid turnover issues. Engineers consumed by a project tend not to look around for new opportunities. Together, these factors mean the agile manifesto works well for single teams on single projects, but in an organization with many teams and projects, the manifesto is itself the cause of long-term problems.</p><p>Valuing “Individuals and interactions over processes and tools” is fine if the individuals stick around. When individuals move on it is often only the processes and tools that remain.</p><p>Valuing “Working software over comprehensive documentation” is great if detailed knowledge of the software is already in the heads of the developers. However, when developers move on, the knowledge goes with them, all that remains is the code. Software inevitably breaks, leaving broken code with no documentation, which is the worst possible combination. Reverse engineering a complicated piece of broken code to fix a defect, with little understanding of what the code was originally intended to do, is fraught with peril.</p><p>Valuing “Customer collaboration over contract negotiation” works as long as the customers and the software developers are around to keep each other honest. When either party moves on it becomes hard to honor half-remembered agreements that were never written down. While negotiating contracts is seldom fruitful, some effort to document what we were trying to produce can avoid many subsequent misinterpretations.</p><p>“Responding to change over following a plan” is an act of faith that turns to hubris as the challenge scales. For a single skilled team, it can be justified. However, if ten-plus agile teams are trying to coordinate a coherent product release in time for next year’s trade show, trusting to responsiveness will inevitably lead to teams thrashing as they all try to adapt to each other’s changes. To be successful a project with so many moving parts requires a strategy for resolving conflicting demands. Agreed goals and interim objectives, clear ownership of responsibilities, internal commitments to deliverables and dates, and a robust escalation procedure to resolve the inevitable problems is one way to manage such conflict. Internal marketplaces where resources can be traded is another.</p><p>All this is not to say the Agile Manifesto is wrong. However, it is insufficient for large organizations with many teams. It is missing mechanisms to harvest, improve and transfer knowledge between employees as they join and ultimately leave the organization. Institutional memory is crucial to the success of enterprises that aim to survive for the medium to long-term.</p><h3>Example 1: 5% growth per quarter</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/0*BnqOAZkK4iiaWLRC.png" /><figcaption>Headcount by quarterly new hire cohort over 6 years at a 5% growth rate and 2.5 year average tenure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/0*ZWaeJYopOxbCWwNH.png" /><figcaption>Tenure distribution in months after 6 years at a 5% growth rate and 2.5 year average tenure</figcaption></figure><p>Starting from 100 employees and growing at a modest 5% per quarter the company will reach about 325 employees in 6 years, by that time:</p><ul><li>30% have less than 9 months experience</li><li>21% have more than 9 but less than 18 months experience</li><li>25% have more than 18 but less than 36 months experience</li><li>Less than 25% have more than 36 months experience</li></ul><h3>Example 2: 10% growth per quarter</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/0*j_8Zhp0nWe9HmssC.png" /><figcaption>Headcount by quarterly new hire cohort over 6 years at a 10% growth rate and 2.5 year average tenure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/0*2Rv7fyt32roxnYIi.png" /><figcaption>Tenure distribution in months after 6 years at a 5% growth rate and 2.5 year average tenure</figcaption></figure><p>Starting from 100 employees and growing aggressively at a rate of 10% per quarter the company will reach about 1000 employees in 6 years, by that time:</p><ul><li>39% have less than 9 months experience</li><li>24% have more than 9 but less than 18 months experience</li><li>23% have more than 18 but less than 36 months experience</li><li>Less than 14% have more than 36 month experience</li></ul><p>The more aggressively you grow the worse the knowledge management problem becomes. Without an effective plan to harvest and redistribute knowledge, aggressive growth companies can easily forget how they became successful in the first place.</p><blockquote><em>Organizations spend a lot of time and resources developing knowledge and capability. While some of it gets translated into procedures and policies, most of it resides in the heads, hands, and hearts of individual managers and functional experts. Over time, much of this institutional knowledge moves away as people take on new jobs, relocate, or retire. Knowledge also degrades when a new senior executive or CEO introduces a different agenda that doesn’t build on earlier knowledge, or contradicts what was done previously. And knowledge disappears even more rapidly when a firm reorganizes or merges with another and there is a subsequent reshuffling of the cast of characters.</em></blockquote><blockquote><a href="https://hbr.org/2013/03/how-to-preserve-institutional"><em>How to Preserve Institutional Knowledge.</em></a><em> by Ron Ashkenas</em></blockquote><h3>The Solution</h3><p>Preserving institutional knowledge is essential if organizations want to retain agility and velocity as they scale. Knowledge lives in the heads of people; it is generated from information that is itself created by processing discrete, unorganized objective facts or observations.</p><p>In Enterprises, knowledge becomes embedded in static representations like documents, and wikis. Knowledge also becomes embedded in dynamic systems, processes, and working practices. Managing the transfer of knowledge from peoples heads to these static and dynamic representations and back again is essential for knowledge to persist as people move through the organization and the organization grows.</p><p>Institutional knowledge management requires an explicit strategy that identifies a few key areas of focus. Organizations should not attempt to retain or control all knowledge; institutional learning is as much about retiring outdated information, as it is about capturing and retaining new information. Knowledge turnover is essential for organizational learning. By identifying a few key focus areas, organizations can ensure they retain knowledge in vital areas while allowing new knowledge to replace outdated thinking.</p><p>Enforcing knowledge management top down has a poor track record. Instead, it must be encouraged to emerge bottom up, as the people who directly benefit either share or consume valuable information. Organizations must incentivize these individual transactions if they wish to ensure the emergent benefits.</p><p>To retain knowledge, information must be acquired, revised, classified, utilized, and repeatedly evaluated by the people who use it. Providing recognition and rewards to staff who undertake these activities is essential. If knowledge management is not recognized as valuable, it will not happen. The problem with the Agile Manifesto is that it’s only solution for knowledge transfer is either word of mouth or code. In reality, there are other, often more effective mechanisms for capturing and transferring knowledge. Large enterprises must use all these methods if they are to learn and endure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d111a41ebb74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/the-enduring-agile-enterprise-d111a41ebb74">The Enduring Agile Enterprise</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Limits of Evolutionary Design]]></title>
            <link>https://medium.com/span-digital-insights/the-limits-of-evolutionary-design-bdeb5821336b?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdeb5821336b</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Thu, 19 Apr 2018 13:01:02 GMT</pubDate>
            <atom:updated>2018-05-08T08:23:14.368Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Evolution is wasteful — many failures are needed to explore a problem space. After multiple rounds of experimentation and recombination, effective and novel solutions may emerge, but few companies have the time and resources to indulge in such a wasteful process.</p><p>Evolution, in its classic Darwinian form, is merely an effective way to develop a <em>barely-good-enough</em> solution. Rather than a wasteful Darwinian exploration of a problem space, it is cheaper and more effective to guide the process.</p><p>A typical Agile approach may divide a six-month initiative into twenty six week-long iterations — creating multiple generations, or experiments, each building on the lessons learned from the preceding trial. Numerous opportunities for discovery and adaptation are produced in this way. But we cannot trust that providence alone will guide us efficiently to the best solution. We need to understand what evolution is good at and how best to harness it if we are to make the best use of our scarce resources.</p><h3>Avoid Bootstrapping and Aim for the Adjacent Possible from the Vantage Point Created by Previous Successes</h3><p>Life on earth first appeared soon after the planet cooled around four billion years ago. Multi-cellular animals didn’t arise until six hundred million years ago. For about 3 billion years evolution busied itself with bootstrapping the basic mechanisms of the cell. It took another two hundred million years for animals with backbones to get emerge from the oceans. All the terrestrial backboned animals that have ever existed have lived in the last 10% of life’s history on earth.</p><blockquote>Evolution takes a long time to bootstrap.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*syJYNTrwdq_o9Qtn.png" /><figcaption><strong>The Tree of Life</strong>. Notice the timescale is not linear. We have to wait 3.5 billion years until the Cambrian Explosion creates the first backboned animals and another 200 million years for backboned animals to emerge on land. From <a href="https://www.evogeneao.com/learn/tree-of-life">evogeneao</a></figcaption></figure><p>Using a purely evolutionary approach, we are in danger of wasting too much time bootstrapping features that have been developed many times before. To avoid this, we should start with a rough design comprising a mashup of a few well-proven, pre-existing core components. We should reserve evolution for what will become the unique differentiators of the new solution, not its basic features. We should focus on evolving those features we don’t understand, not those we do. By maximizing the use of open-source or commercial products for core components, we can obtain basic capabilities quickly without waste. All software is built from pre-existing software. Every solution developed today benefits from over 60 years of preceding software development. We must acknowledge this and use it to our advantage.</p><p>Knowing where to draw the line between re-using existing components and building new components requires an honest assessment of the market requirement for novelty. The requirements must be challenged to identify the real need for innovation. Many compelling solutions have been built by combining a handful of existing components in a novel way. In fact many, possibly most so-called “innovations” are actually just novel combinations of pre-existing components and ideas.</p><p>Stuart Kauffmann introduced the idea of the <em>Adjacent Possible</em> to describe the available opportunities for evolutionary innovation. Evolution innovates by advancing into new territory from the vantage point created by combining existing solutions. It is only by combining a few tested capabilities at the edge of known solution space that it is possible to reach something new. Subsequent <a href="https://www.technologyreview.com/s/603366/mathematical-model-reveals-the-patterns-of-how-innovations-arise/">mathematical analysis</a> has shown that this is indeed the way innovation proceeds. We should turn this finding into a guiding principle and continuously seek vantage-points, created by combining existing solutions, from which we can launch our efforts to reach the adjacent possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kBZF1kET_oxh666W.jpg" /><figcaption><strong>The Adjacent Possible</strong> A simplified example showing the emergence of tags in a musical classification system. The adjacent possible is what can be created based on what already exists. Psychadelic can only be created after rock, guitar, and synthesizer tags exist. From — Significance and popularity in music production. Bernardo Monechi, Pietro Gravino, Vito D. P. Servedio, Francesca Tria, Vittorio Loreto. R. Soc. open sci. 2017 4 170433; DOI: 10.1098/rsos.170433. Published 12 July 2017</figcaption></figure><h3>Budget Your Solution Exploration and Focus on Hypotheses that Offer the Greatest Benefit, for the Least Risk</h3><p>The iterative structure of agile projects provides two significant benefits. First, it allows us to manage risk by assessing our exposure and adopting mitigating strategies at the start of each iteration. Second, it provides opportunities to obtain feedback from real users, or their proxies, at the end of each iteration and use it to improve the solution going forward. While this iterative approach allows us to adjust our solution as we proceed, it also places limits on how adaptive we can be. By understanding the nature of these limits, we can maximize the available benefits.</p><p>To adapt, or evolve, we must experiment and then eliminate failed solutions so better solutions can emerge. This approach means we will not progress at a steady rate. There will be setbacks; some iterations will disprove our hypotheses. We must understand that our tolerance for failure and our desire for innovation are flip sides of the same coin. We must speculate to accumulate.</p><p>Each of our project iterations sets out to prove a hypothesis. If all hypotheses are proven correct, we have learned nothing. If all are proved wrong, we have wasted our time. There is an optimal success ratio that allows us to explore the largest number of possible solutions while simultaneously delivering the most functionality. We are seeking the best solution we can afford, given the resources available to us. We are not seeking the absolute best solution, which is always beyond our budget, given the size of the solution space.</p><p>Just as the <a href="https://en.wikipedia.org/wiki/Secretary_problem">Secretary problem</a> teaches us that we don’t need to interview all the candidates for a job to hire one of the best. We don’t have to explore every possible solution to find a “good enough” solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*_67ACX8QSiigvuB3.jpg" /><figcaption><strong>The Secretary Problem. </strong>The probability of finding a better solution, in a random series of candidate solutions decreases after 37% of the solutions have been evaluated. We, however, can do better than random by evaluating solutions in order of their potential benefit and likelihood of success.</figcaption></figure><p>We should explicitly budget the amount of solution exploration we can afford for any initiative and focus on evaluating only the hypotheses that offer the highest possible return for the least risk of failure. As a rule of thumb, we typically aim for, 60% low or no risk iterations, and 40% higher risk iterations. This mix results in 80% of iterations delivering successfully and 20% failing, or partially succeeding. Overall we waste as little as possible while still allowing ourselves to learn as we go and evolve the solution.</p><h3>Remove Path Dependencies Only When the Investment Will Break-Even in the Short-Term</h3><p>Even with the inevitable waste caused by speculation, we will incrementally accumulate benefits from prior iterations. As we do so, it will become increasingly difficult to revise or replace earlier contributions as they become embedded in the solution. Evolution can sometimes lead down a blind alley from which the only way out is to go back. If the cost of retracing our steps is high, we may be faced with a dilemma. Either accept that our future choices are constrained by the path we have already taken, or pay a high price by giving up the progress we have made so we can back up and take another more beneficial path. This situation is a classic example of negative evolutionary path dependency.</p><p>Fortunately, many software design best practices address this problem. Modularity of code, encapsulation through well defined APIs, and microservices, to name a few, all make it easier to replace or refactor portions of a solution. All agile development teams will face this dilemma at some point — it is a consequence of the evolutionary approach. In these cases, we must compare two alternative approaches: press on, or go back and evaluate whichever avenue delivers the better outcomes. Several factors must be evaluated:</p><ul><li>The cost of removing the path dependency will almost certainly increase in the future. But maybe it won’t grow by much.</li><li>It is usually worth removing a path dependency in the long run. But many things are worth doing in the long term, that does not mean we should do them now.</li><li>The path dependency may reduce the velocity of the development team and yet be unnoticeable to the user. Sacrificing some velocity in the short term may be worth it to reach our goals in the medium term.</li></ul><p>The problem we face is to find the break-even point of the investment in removing the path dependency. If the benefit will directly contribute to the success of the initiative, and generate returns quickly, and can be achieved at an acceptable cost, then it may be worth it. If the benefit is indirect, or will only create returns in the distant future, or will come at a too high price, there is probably a better alternative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cBX2olfuAZukqugT.png" /></figure><h4>Project Break Even Point</h4><p>The break-even point is reached after the cost of development and the opportunity cost have been recovered with interest. Typically the internal rate of return (IRR) for most businesses is considerably higher than typical banking interest rates. It can, therefore, take a long time to reach the break-even point.</p><p>The decision to remove a path dependency is an <em>economic</em> decision as much as an <em>engineering</em> decision. Such choices are never easy, many projects have sacrificed short-term success attempting to refactor or replace components that would return little, or no, benefit in a reasonable time-frame. We must guard our limited resources and spend them wisely. Wasting resources on poor investments, that return little benefit, must be avoided.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdeb5821336b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/the-limits-of-evolutionary-design-bdeb5821336b">The Limits of Evolutionary Design</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Rough Design is Good enough]]></title>
            <link>https://medium.com/span-digital-insights/when-rough-design-is-good-enough-a78e0fc9ee4b?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/a78e0fc9ee4b</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Thu, 12 Apr 2018 13:01:02 GMT</pubDate>
            <atom:updated>2018-05-08T08:56:32.528Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Rough Design Up Front can help create project momentum, shorten the overall development time, reduce risk, and help form a new team. However, deciding when the rough design is good enough for the work of construction to begin, is a crucial milestone in the life of the project. At SPAN we often start a project with an <em>inception workshop. </em>For smaller projects the workshop may be sufficient, but for larger projects, an iteration (or two) elaborating a rough design may be required before construction can commence in earnest.</p><p>Starting in the workshop, and continuing after, we collect and refine requirements in two distinct areas: <em>Functional Scope</em>, and <em>Business Objectives</em>.</p><h4>Functional Scope (Epics and Stories)</h4><p>Epics and stories collectively describe the functional scope of a solution - they comprise all the intended features and functions. We usually triage the scope into three distinct priorities based on business need and potential to reduce risk. Finer sequencing of the backlog can happen later — at this stage a rough prioritization is all that is needed.</p><h4>Business Objectives</h4><p>Business Objectives can be divided into two parts: <em>Solution Capabilities</em>, and I<em>nitiative Externalities</em>. As discussed before, setting objectives for agile projects is a useful practice for agile teams. With a little effort, it is possible to frame the entire initiative with meaningful and measurable targets that define success. These targets can also be used to calibrate progress.</p><h4>Solution Capabilities</h4><p>Solution Capabilities define essential required performance characteristics of the solution. For example, a new vehicle could have a target set for its fuel efficiency as “twenty one miles per gallon in normal freeway traffic”. For most solutions, it is useful to define a few target values for measurable capabilities. Selecting the right capabilities, defining meaningful target values and measuring them is often challenging. Teams that fail to do this usually fail to deliver the necessary capabilities because they have no target to aim for, and no yardstick to measure success.</p><h4>Initiative Externalities</h4><p>All solutions are built in a business or political context. This context can often be summarized as a budget and a timeline made up of externally defined milestones. Design to cost and deliver on schedule is a fact of life for many businesses. Earnings calls, trade shows, and product launch schedules mean that delivering a viable set of features by a specific date is essential for success. Rather than fight these externalities, it is often better to produce a rough design that can plausibly satisfy the most important requirements well within the deadline, and only then evolve the remaining features.</p><h4>Co-Evolving Requirements and a Rough Design</h4><p>Creating a rough design is an iterative process with both requirements and design being developed in parallel — it is a negotiation between team members and stakeholders. The design artifacts embody the agreements and decisions that the team has reached. The process of requirements refinement and (re)design continues throughout the life of the project, but knowing when to transition the primary focus from design to construction requires balancing several factors: the maturity of the team, their familiarity with the problem domain, the size and complexity of the project, and the extent to which the requirements can be pre-stated or must be discovered. These variables are usually principal factors in deciding when to begin construction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRNxmJP2NwM8a-NS0IYSZw.png" /><figcaption><strong>Architecture Evolution (example 1)</strong><br>The six stages in the evolution of a design for a sophisticated medical work-flow solution. This design had to retain legacy components, shown in yellow. Notice how the yellow components gradually migrated to lower, infrastructure, levels where they could be encapsulated and managed. This design took a team of 6 people 3 weeks to define.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1g9P4rI9K08DVuGX2tawUw.png" /><figcaption><strong>Data Model Evolution<br></strong>The evolution of a data model over several days. From a hand-drawn diagram on a whiteboard created during a workshop design session to a reasonably detailed model with working notes in Google Drive.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hHFsscIdx3W5MXlm_P3gpw.png" /><figcaption><strong>Architecture Evolution (example 2)<br></strong>The five stages in the evolution of an architecture for a custom publishing solution for specialized time-series data. Notice how the final diagram is heavily annotated with links to corresponding Jira tickets.</figcaption></figure><p>The rough design usually comprises a handful of diagrams that represent a conceptual model of the solution and collectively explain the approach we are taking and how the solution will work. We continually review the rough design against the requirements, all the while asking the following questions:</p><blockquote><strong>Does the conceptual model account for system requirements at a reasonable level of abstraction?</strong></blockquote><blockquote>- The conceptual model is sufficiently generalized when it can account for all significant epics and stories in a concise way that reduces complexity by consolidating similar features.<br>- The conceptual model is sufficiently specific when domain-specific concepts are reflected in the design of core components and domain-specific terminology (nouns and verbs) are used consistently throughout the design.</blockquote><blockquote><strong>Can team members use the design artifacts to explain the solution to stakeholders and each other convincingly?</strong></blockquote><blockquote>- How will the solution deliver the required features and functions?<br>- How will the solution meet its capability targets?<br>- How can the solution be delivered on time and on budget and thereby meet external milestones?</blockquote><h4>Starting Construction</h4><p>Construction can start when these questions can be answered satisfactorily and a backlog of stories and tasks has been prioritized covering the construction of the majority, but not all, of the design. How far we go in defining the backlog depends on the business need. If we are building to cost or building to a deadline we try to define at least 80% of the solution. If the deadline and or cost is less important than some other business outcome, then we may start with as little as 40% of the solution defined.</p><p>The design is <em>good enough</em> when:</p><ul><li>the team responsible for building the solution can confidently look each other in the eye and claim they understand what they are about to build (and have a reasonable plan for building it).</li><li>the team feels confident that they can work out, as they go along, what they do not already understand.</li></ul><p>Inevitably, during the project, we have to rethink many of our design decisions and plans. We seldom, if ever, implement a design without change, but this does not mean the initial work is wasted — far from it. We change our plan because we have learned valuable lessons, or need to adapt to changing circumstances. The initial preparation makes the inevitable adjustments and reorganization far easier, since the team has already learned to negotiate and compromise with each other, considered and evaluated many alternates, and is armed with a common vocabulary and understanding of the problem domain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a78e0fc9ee4b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/when-rough-design-is-good-enough-a78e0fc9ee4b">When Rough Design is Good enough</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Objectives for Agile Projects]]></title>
            <link>https://medium.com/span-digital-insights/setting-objectives-for-agile-projects-5fffba4cfa52?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fffba4cfa52</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[okr]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[software-design]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Sat, 07 Apr 2018 21:39:03 GMT</pubDate>
            <atom:updated>2018-04-08T02:24:31.729Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Most Agile methodologies omit a crucial aspect of successful software development — Defining measurable business objectives for the initiative. When agile projects are aligned with well-defined objectives they become a powerful business tool, without such alignment, teams can drift aimlessly from one epic to another, and often fail to deliver their intended business outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vc__7rE3cEDpsPJysdVcDw.png" /></figure><p>The Agile movement has revolutionized the development of software. It is clearly better than traditional alternatives. Organizations representing the full range of industries, business models, and sizes are using Agile. Interestingly, they aren’t all doing it the same way. There are many variants of the approach each with differing benefits. Overall, however, the approach remains robustly effective, despite significant variation in implementation.</p><p>Many of the practices and principles embodied in the Agile approach existed long before the <a href="http://agilemanifesto.org/">Agile Manifesto (2001)</a> codified them. In his 1988 book, The Principles of Software Engineering Management, <a href="https://www.gilb.com/">Tom Gilb</a> dedicated a chapter to Evolutionary Delivery in which he describes the critical concepts of an Evolutionary Delivery approach as</p><blockquote>1. Multi-objective driven<br>2. Early, frequent iteration<br>3. Complete analysis, design, build and test in each step<br>4. User orientation<br>5. Systems approach, not merely algorithm orientation<br>6. Open-ended basic systems architecture<br>7. Results orientation, not software development process orientation</blockquote><p>These concepts overlap significantly with Agile Principles. The writers of the Agile manifesto cite Gilb as an influence, and Gilb himself quotes earlier work by <a href="https://en.wikipedia.org/wiki/W._Edwards_Deming">Deming</a>. Other examples of Agile-like approaches, stretching back many years, can be found in non-software engineering history books. But Gilb’s call for an approach that is objective-driven and results-orientated seems to have been ignored in favor of a focus on features over outcomes.</p><p>The collection of practices embodied in Agile and similar approaches has been co-evolving for many years. They work well together. These practices are starting to spread outside engineering into other areas of business. But this does not mean they are complete or perfect. They are merely a step on a path towards better engineering practices. In choosing how we might improve Agile it is important to remember what makes it so effective.</p><h4>Agile is Human Scaled and Transparent</h4><p>Each person on an agile team can see how their contribution, and that of their peers, adds to the delivered solution. Agile teams are small enough that each individual can have a positive impact, and no individual can hide their weaknesses. Timescales are short enough for feedback to be meaningful and actionable. The team can individually, and collectively, learn from its mistakes, hopefully, fast enough to make a difference. People who like teamwork, and who like to learn, like Agile.</p><h4>Agile is Easy to Explain and Self-Correcting</h4><p>It is easy to communicate the basic principles and practices of the Agile approach. It is much harder to explain why these practices work. The benefits of the approach are emergent — practices interact in various ways to create benefits that are not predictable beforehand. This is why so many zealots demand complete adherence to their flavor of Agile — they can’t explain how the benefits arise from the practices because they emerge from a complex interaction of many factors. The short feedback loops inherent in the approach create many opportunities for correction and improvement. In reality, it doesn’t much matter where you start. Not only do the practices lead to emergent benefits, they also act as an attractor that pulls and pushes teams toward an ever more effective approach. The practices are valuable individually and are self-reinforcing, becoming even more valuable collectively.</p><h4>Agile Drives the Evolution of Solutions and Teams</h4><p>Agile can be employed to evolve better solutions in a rapidly changing environment. However, there is more than one reason why a team may lack a clear understanding of what it is trying to achieve.</p><p>When a novice team is uncertain of how to deliver a well-defined solution an agile approach forces the team to continually confront their own lack of experience, to fail in small recoverable steps, thus allowing them to succeed in the longer term if they can learn fast enough. The approach allows investors and other stakeholders to keep inexperienced teams on a short leash while they learn how to deliver effectively. Agile is a great way to reduce delivery risk caused by inexperience. Agile forges self-calibrating teams that know the limits of their own capabilities.</p><p>Conversely, agile forces expert teams to confront their lack of understanding of the complex problems they are trying to solve. Agile helps expert teams explore the problem space by allowing them to innovate in small chunks and incrementally evolve a solution. Evolution is a fundamentally wasteful process as Tom Gilb has said: “rapidly iterating in the wrong direction is not progress”. When guided experts, an evolutionary approach can be the best way to explore a new problem space.</p><p>In practice, most teams are experts at some things and beginners at others. This ability to help both Novice and Expert teams incrementally improve and create better solutions is the key to the success of Agile.</p><h4>Avoiding the Agile Cargo Cult</h4><p>The conditions that favor the Agile approach do not universally apply. The <a href="https://hbr.org/2016/05/embracing-agile">HBR Article Embracing Agile</a> points out it is important to “understand where agile does or does not work.” Not all initiatives require innovation. Some merely require a reliable implementation of a straightforward design.</p><blockquote>Agile is not a panacea. It is most effective and easiest to implement under conditions commonly found in software innovation: The problem to be solved is complex; solutions are initially unknown, and product requirements will most likely change; the work can be modularized; close collaboration with end users (and rapid feedback from them) is feasible; and creative teams will typically outperform command-and-control groups.<br>&gt; <a href="https://hbr.org/2016/05/embracing-agile">HBR Article Embracing Agile</a></blockquote><p>Even when Agile is applicable, there are multiple flavors, each suited for different purposes. <a href="http://www.disciplinedagiledelivery.com/">Disciplined Agile (DA)</a> is a process decision framework that provides guidance to organizations struggling with deciding which Agile Practices to adopt.</p><blockquote>In general, Disciplined Agile (DA) is a hybrid framework that builds upon the solid foundation of other methods and software process frameworks. DAD adopts practices and strategies from existing sources and provides advice for when and how to apply them together.<br>&gt; <a href="http://www.disciplinedagiledelivery.com/introduction-to-dad/">Introduction to Disciplined Agile</a></blockquote><p>With an objective expert assessment of the needs, capabilities, and constraints of a business, and the problem it is trying to solve, it is usually possible to tailor a successful Agile approach. Tailoring the approach is not that easy, it requires experience, but <a href="http://www.disciplinedagiledelivery.com/">Disciplined Agile (DA)</a> does a reasonable job of providing guidance.</p><h4>Agile Goals and Objectives</h4><p>Despite the variety of agile approaches available, few define a practice concerning business outcomes, yet outcomes matter far more than features. Agile is silent on how to define and use objectives to guide project execution. There is no principle or practice regarding this obvious need. Agile is mute on how objectives can be used to filter and interpret user feedback. Not all feedback should be acted upon, if it doesn’t align with project objectives, then maybe it should be ignored? Most concerning, Agile has nothing to say on how to address the omnipresent business objective to deliver a viable set of features, by a particular date, for a pre-approved budget. In fact, many Agile adherents would claim this is the wrong way to frame the problem. The best that most Agile methodologies can do is to offer a trite promise to deliver “value” without defining what that is or when it will be delivered.</p><p>What is needed is a lightweight objective definition and tracking practice that can be incorporated in Agile projects. Fortunately, this exists. Filipe Castro has done a nice job of explaining the <a href="http://felipecastro.com/en/okr/what-is-okr/">OKR (Objectives and Key Results</a>) approach to goal setting. He further explains how OKRs can be used for Agile Development.</p><blockquote>OKR (Objectives and Key Results) is a goal setting framework adopted by Silicon Valley companies that can complement Agile and Lean, creating an agile approach for setting goals and measuring value. Used by Google, Twitter and LinkedIn, OKR can help evolve the Agile Community and the Agile Manifesto itself from a focus on outputs (delivering features) to a focus on outcomes (delivering results).<br>&gt; <a href="https://www.infoq.com/articles/agile-goals-okr">Agile Goal Setting with OKR — Objectives and Key Results</a></blockquote><h4>Example Business Objectives</h4><p>Using OKRs to define the business outcomes for a project is a practice that most Agile teams would benefit from. Objectives are NOT the deliverables, they are the beneficial outcomes the deliverables will produce. There need only be a few — 3 to 5 at most, with 1 to 3 key results per objective. Objectives should be shared with the team and stakeholders, they should be printed out and posted on the wall. They become the team’s objectives providing guidance in times of confusion and doubt.</p><p>Some examples may help solidify the concept.</p><p>For a medical imaging solution the following three product claims guided engineering efforts:</p><blockquote>- Display the first image in under 3 seconds on any hospital computer. <br>- 99.99% availability measured monthly. Monthly fees waived if cumulative downtime exceeds ~ 5 minutes per month.<br>- All images online all the time, no tiered access for older images,</blockquote><p>These claims were public and contractual. They helped differentiate the product in the market and were treated as objectives by the engineering team. They influenced many engineering decisions on a daily basis.</p><p>A recent project to deliver an application development framework for consumer devices defined some objectives as follows:</p><blockquote>- Using the new framework we can consistently complete new Apps (development and testing) in 2 months (elapsed time) or less. (The most recent application developed without the new framework took 4 months, and 1 week of QA, Current best without the framework is 6 weeks.)<br>- An engineer who is already proficient on the new framework can transfer and ramp up on an unfamiliar framework based App in one day (8 hours)<br>- An engineer who is not proficient on the new framework can ramp-up on the framework in 1 week 40hrs or less.</blockquote><p>It would have been easy to develop a framework that had all the required functionality but failed to deliver the business requirement for speed of development and ease of adoption. These objectives helped the team focus on the real business value.</p><p>For a sports-related social network application.</p><blockquote>Increase the number of concurrent users by an order of magnitude while simultaneously reducing median user perceived latency to less than 1 second. In time for the launch of a new customer segment in 3 months.</blockquote><p>This last objective kept a small team fully occupied leading up to the launch. The team designed and built a distributed test harness to emulate high volume traffic, implemented and tuned a Redis cache with event based de-caching, modified a few other services to remove bottlenecks and were able to demonstrate the performance characteristics of the system up to the point at which it failed. They exceeded the requirements by a modest amount and were able to identify the maximum load after which performance would start to degrade. This work enabled the business to make strategic decisions about hosting and burst capacity planning.</p><h4>Using Objectives to Manage Expectations</h4><p>Once a team has taken ownership of the project it is up to them to manage changes to the objectives over the lifespan of the project. Inevitably objectives start to drift as the project proceeds. Sometimes valuable lessons are learned along the way and objectives can be reset to take advantage of our new understanding. Sometimes externalities move the goal posts, and the project must adjust to the changing business circumstances. Sometimes objectives that were initially excluded are added back into scope. Whatever the case, having defined objectives allows the team to monitor such drift, managed it, and stakeholder expectations, in a controlled way.</p><p>Without defined objectives teams can drift aimlessly at the whim of changing circumstance and fickle stakeholders, with well-defined objectives teams can ensure they stay on track to deliver the required business outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fffba4cfa52" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/setting-objectives-for-agile-projects-5fffba4cfa52">Setting Objectives for Agile Projects</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rough Design up Front]]></title>
            <link>https://medium.com/span-digital-insights/rough-design-up-front-a071836f5658?source=rss-2798b4f73193------2</link>
            <guid isPermaLink="false">https://medium.com/p/a071836f5658</guid>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[John R. Harris]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 20:22:10 GMT</pubDate>
            <atom:updated>2018-03-16T03:10:37.025Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Lean Agile adherents advocate emergent design over design up front. When employing a lean approach, features are built incrementally, one iteration at a time, and continually re-factored, based on feedback, until a design emerges. This evolutionary approach makes sense for situations where the solution is poorly understood and there are few, if any, experts to guide the team. But this is seldom the case outside startups or research departments. In most cases a rough design up front is a better way to start an initiative.</p><p>Most businesses understand, at least partially, what their customers need, how soon they need it, and what the business can afford. Most businesses have an existing customer base, product, technology stack, and workforce. In short they have a business and technology context that the new solution must complement and extend. There are few green fields left.</p><p>When armed with well defined goals and objectives, and a solid understanding of the existing business and technology context, experienced lead software engineers and product managers can develop successful strategies and designs for an initiative. Incrementally evolving a design is perverse, if a rough design, that is fit for purpose, can be rapidly defined upfront. Skilled professionals are more than capable of anticipating problems and identifying exploitable opportunities in advance.</p><p>Significant parts of a solution can, and should, be designed up front. Not every project is so complex and uncertain that it must be evolved from scratch — In fact most are not. Experienced Product Managers and Technical Leads can usually create a rough design for up to 80% of a solution in the first few iterations of a project.</p><h4>Example Design Artifacts</h4><p>Discussing the merits of Rough Design Up Front is difficult without showing some actual design artifacts. This article presents some example artifacts from projects we have completed at SPAN. These have been deliberately obfuscated and redacted, for obvious reasons, but hopefully still illustrate the type of artifact and the amount of detail necessary for a rough design.</p><h4>System Architecture Diagrams</h4><p>Based loosely on the <a href="https://c4model.com/">C4 Architecture Model</a> we typically create a combined context, container, and component diagram. We seldom include any classes on these diagrams. This is a single diagram that shows the system under construction; the external systems it interacts with, the containers it is made up of (web applications, mobile apps, desktop applications, databases, file systems, etc), and the components that comprise each container. The diagram is usually arranged into layers and functional areas. Its purpose is to define the key abstractions and concepts, major functions, dependencies, data flows, implementation technologies (where known) and potential construction sequence. Other factors can be shown on the diagram as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*e6M_sFFSBsfqQ9LF6CGORQ.png" /><figcaption><strong>Architecture Diagram</strong>. A typical C4 based architecture diagram color coded by implementation technology showing many text labels linked to key Jira tickets.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*6u2iTeM-zhsk21U17VBhFw.png" /><figcaption><strong>Architecture Diagram</strong>. A group of interoperating systems and the data items (green) they manage.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*nX9GwK5XXVPI29eBNd1RBg.png" /><figcaption><strong>Architecture Diagram</strong> A simple components diagram with two important scenarios that describe the basic operation of the system.</figcaption></figure><h4>Entity Relationship Diagrams</h4><p>The data entities within the system are represented by a logical entity relationship diagram. It is important to stress, this is a data model NOT a database design. Regardless of how the data will eventually be stored a logical ERD identifies the key entities and their relationships. This helps the team define terminology, start to group together tightly coupled entities into resources that can be managed collectively, and begin to understand the life-cycle of the resources that the system will manage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*uoOU6ON5_42KQgbmFWBbOg.png" /><figcaption><strong>Entity Relationship Diagram </strong>A typical ERD diagram with it’s ongoing Google docs conversation.</figcaption></figure><h4>State Transition Diagrams</h4><p>The behavior of key resources, entities, and objects within the system is often represented via one or more state transition diagrams. These diagrams show the allowable transitions between resource and entity states, the component or person that can initiate each transition, and the interdependencies between state transitions of different resources or entities. State diagrams help the team understand the life-cycle of resources and are often labeled with estimates of event frequency and geographic distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*ckYXAxoyTX-UUF_zTX3QzQ.png" /><figcaption><strong>State Transition Diagram</strong>. A typical state transition diagram showing the transitions for all the major objects in a systems and how these transitions affect each other.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*forF1BiZHXISZehZ1QuCQg.png" /><figcaption><strong>State Transition Diagram</strong>. Account state transitions in a subscription payment system based on credit card processing status labeled with the three outstanding issues that need to be resolved.</figcaption></figure><h4>Rough Design in Practice</h4><p>Most of the artifacts we produce are Diagrams. These are created in GSuite because it supports contemporaneous multi-user collaboration. These documents are frequently used in team meetings and are under continuous revision. Diagrams are often color coded and annotated and usually act as host to discussions about various design issues. Over time objects and labels on each diagram are converted into links to Jira tickets, Github repositories, and other important resources. These design artifacts are few in number, very high in information content, and create a common baseline from which engineers can evolve the solution. Attempting to design everything up front assumes perfect foresight on the part of the team and little value to be gained from learning as we proceed. This is obviously the epitome of hubris — we can’t anticipate everything and should be eager to learn as we go along. But equally, designing nothing, or too little, up front ignores the benefits to be gained from a rapidly produced rough design. Such a design saves time, reduces cost and risk, and leads to a better end result.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a071836f5658" width="1" height="1" alt=""><hr><p><a href="https://medium.com/span-digital-insights/rough-design-up-front-a071836f5658">Rough Design up Front</a> was originally published in <a href="https://medium.com/span-digital-insights">SPAN Digital Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>