This is the second post of a two-post series. You can find the first post here.

In my last post, I explored the phenomenon that while most activities on projects are delivered on time, most projects are delivered late. I introduced the concept of long-tails in distributions and how it’s the activities that reside in these long-tails that cause most of the delay in projects. If you haven’t checked it out yet, read it first here.

Given this, it’s critical that we learn how to model these events when undertaking schedule risk forecasting, yet the status quo in QSRA disregards these types of events leading to project managers being unprepared for events that could derail their projects.

In this post, I want to explore the way uncertainty modeling is done in QSRA, in particular looking at the impact that using one of the 3 bounded distributions described in my previous post has on the overall project forecast. As a reminder, the 3 distributions you typically see used are the following:

Note that they are bounded because of the limits that are imposed by a human when defining these distributions.

PERT vs Lognormal Distribution

What happens if instead of using a bounded distribution to model activity durations, we use an unbounded distribution? What impacts does that have on our overall project forecasts when we undertake QSRA?

A good candidate for an unbounded distribution to model activity durations is the Lognormal distribution, pictured below.

Some nice properties that make the Lognormal distribution suitable is that it is strictly positive (so no need to worry about negative durations) and it allows for non-zero probability of large delays. That means that we can begin to assign a very small probability to large delays instead of completely disregarding them.

When compared to the PERT distribution we notice that there is a large overlap around the area of “on-time” completion, as we would expect. The main difference is of course the areas outside of the bounds set in the PERT distribution.

Armed with these two distributions, we can begin to compare how uncertainty is propagated through the project as we sample 1000s of times for activities taking into account their dependencies.

Viewing a project as a graph

At its simplest, a project is a set of tasks that are joined by links (specifying dependencies between two tasks). This can be easily visualized as a network like in the image below. Each point (or node) is a task, with an associated duration, and the edges that connect them are the dependencies.

Broken down even further, project graphs are made up of two patterns:

Linear Chain: Task A can only start once all the previous tasks are completed i.e. after the sum of all preceding durations

Merge Points: Task B can only start once the last of all preceding tasks finishes i.e. after the max of all preceding durations

Therefore, if we want to understand how uncertainty is propagated for an entire project, we can begin by investigating these smaller cases and use the findings to investigate the overall project forecast. The subsequent graphs below show the resulting distribution for Task A and Task B in the examples above as we increase the number of predecessors, for both the PERT distribution and Lognormal distribution.

There is nothing very surprising or different in the linear chain case. As we add more activities to the chain, the distribution shifts to the right since we’re adding more duration. On top of that, the distribution becomes wider because each preceding activity adds some uncertainty to the overall sum. Note that in the case of the lognormal distribution, the distribution becomes wider at a faster rate because of the probability of events that lay outside of the bounds of the PERT distribution.

On the other hand, the merge points case is much more interesting and surprising. As we increase the number of predecessors for Task B, the resulting distribution in the Lognormal case shifts to the right and gets wider. This is because, as we add more predecessors, we’re increasing the chance that at least one of them is an extreme event, therefore making the distribution wider. However, the resulting distribution in the PERT case actually becomes narrower! This means that the start of Task A becomes more certain the more predecessors Task A has! Why does this happen? Similar to the Lognormal case, the more predecessors we add, the higher chance there is of an extreme event occurring. Except, in this case, there is an upper bound to the original distribution, meaning that we are increasing the chances that at least one of the tasks will equal that upper bound, resulting in a more “confident” forecast.

The ramifications of this simple phenomenon could be quite wicked because every merge point in a project schedule will actually decrease the uncertainty in our resulting forecast leading to a false confidence in the overall project forecast. When this comparison is carried out for a full project forecast, in a project with 1000s of activities, you get the distributions below. Not only are the distributions quite far apart, but also the overall uncertainty with regards to the project outcome changes significantly when using a bounded vs an unbounded distribution.

Conclusions

I started this article series by highlighting the seeming paradox that while most activities are delivered on time, most projects are delayed. We see that extreme events don’t just exist, they are prevalent across the construction industry, yet they are not given enough consideration when it comes to forecasting our project end dates. In fact, we are not just ignoring them, we are creating a false confidence in our forecasts by using bounded distributions! This false confidence can lead to misallocation of resources, reputational damage and financial repercussions to name a few.

Is using a distribution such as the Lognormal distribution the right solution? Probably not. While it may give a better picture of risk in a project, it might not be fit for modeling all kinds of activities in the world. That’s why at nPlan we’ve taken the approach of using machine learning to learn unique activity distributions for each activity based on millions of historical data points.

Making decisions when everything is affected by uncertainty was originally published in nPlan on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the first part of a two-part story. You can find part 2 here.

In early 2022, nPlan conducted a study on close to 500,000 schedules (see link to case study here) and found that around 80% of projects are delayed. While it is a high number, I’m sure it is not surprising to many people reading this given the headlines we see every year around large infrastructure projects running over time and over budget. What may in fact be a more surprising statistic exposed in the report is that less than 25% of activities in projects are delivered late. This means that while the majority of activities in projects are delivered well, we still see an extraordinary number of projects being delayed (and ⅓ are delayed by more than a year). This begs the question: “where are projects going wrong”? In two words, long tails.

What are “long tails”?

In the field of probability and statistics, the long tails of a distribution refers to the area of the distribution that has some meaningful probability and is far away from the mode / peak of the distribution. The diagram below illustrates a distribution with a long right tail (highlighted in red).

Long tails are present in the study of online platforms, finance, insurance and social networks, to name a few. A classic example of a long-tailed distribution is a Pareto distribution, named after Vilfredo Pareto who is also known for the “80–20” principle. While studying land ownership in Italy, he discovered that roughly 20% of the population owned 80% of all the land. It turns out this principle applies to a vast number different areas of nature, where 80% of outcomes are due to 20% of causes. Many times this is referred to as a “Power Law”, and when present is an indication that the phenomena we’re observing is not normally distributed.

Below is an image showing the distribution of completion ratios* of all activities in our data set. The vast majority of the data sits in the range of 0–1 which aligns with the statistic in the introduction. Perhaps more interesting is the occurrence of activities with really large completion ratios, greater than 10 or even 15 times the original planned duration! Note that this graph had to be cut off for the sake of presentability but we see instances in our dataset of activities that are even 100 times delayed. This is the long-tail of activity durations, and what I believe contributes to the delays we see in construction today.

The plot above demonstrates the existence of long-tail events in our historical project data, yet to this day the status quo in schedule risk management is to disregard tail events entirely. I want to use the rest of this post to examine what happens when we disregard tail events in schedule risk management and explain the dangers of these kinds of approaches.

*A completion ratio is the ratio of actual duration to planned duration. In other words, it’s a measure of how long an activity took as a multiple of its planned duration.

The Status Quo — Quantitative Schedule Risk Analysis (QSRA)

QSRA is the most common practice when it comes to creating a forecast of project outcomes and also for evaluating what are the top risks in the project based on an accurate schedule. It starts with the assumption that each activity carries with it a certain level of uncertainty and a level of susceptibility to discrete risks, and so by modeling that uncertainty we can create probabilistic forecasts of projects that help project managers in making better decisions about how to allocate resources. Typically, the way activity duration uncertainties are modeled is using one of the following 3 distributions:

These 3 distributions are very intuitive because they are defined by setting 2 or 3 parameters — the best case outcome (minimum), worst case outcome (maximum) and the most likely outcome (not set for uniform distribution since all are equally likely). Project professionals use their own experience, with some historical data, to set the parameters of these distributions when undertaking QSRA. After all the uncertainties are set, you can simply use monte-carlo sampling to get an overall distribution for your project end date and for any key milestones in between.

One great thing that QSRA introduced into the project delivery world is this idea of thinking about the world in a probabilistic way. Humans are riddled with biases when it comes to thinking about the future, so having a quantitative way of discovering all the ranges of possible outcomes helps us make better and more objective decisions.

However, there are limitations to how QSRA works in the industry today. Firstly, activity duration uncertainties are largely determined by humans and therefore still subject to the same biases mentioned above. Secondly, it is a lot of manual effort to agree on uncertainties for all activities in a project before running the whole analysis, meaning that QSRA tends to be undertaken on higher-level “summary” schedules. Lastly, and the limitation I want to focus on most, is that QSRA completely disregards long-tail events.

Notice how all three distributions illustrated above are bounded, one must explicitly define a minimum and a maximum. Furthermore, it is uncommon to find any maximum bounds used in QSRA that are to the scale of the tails we see in our historical dataset (i.e. to the order of 10x planned duration or more). We are therefore missing an entire range of possible outcomes for our project!

Why is this important? If we go back to the original premise that most project delay is created by activities that are heavily delayed, then how can we afford to ignore modeling these kinds of events in a process that is supposed to provide us with more clarity and better decisions?

In the next post of this series, I will dive deeper into how using different distributions in the QSRA process can lead to vastly different outcomes, and how we can start thinking about modeling extreme events.

Making decisions when everything is affected by uncertainty was originally published in nPlan on Medium, where people are continuing the conversation by highlighting and responding to this story.

Through my experience of working with companies, big and small, from 1 founder to tens of thousands of employees, what I’ve learnt is that no matter the size of your company, or the industry you’re in, data will be a key part of your business. However, not everyone has the luxury of being able to hire a Chief Data Officer or Head of Data from day 1, especially when you’re at the beginning of your journey as a company. That being said, I firmly believe that every company should be thinking about their data, and my goal is to provide some guidance for founders who are starting their data journey.

There is more data being created every day, as the world is increasingly connected and the lines between our analogue and digital lives are blurring. In addition, the field of data science and machine learning has matured significantly over the past decade, surfacing new techniques to extract value from structured and, more importantly, unstructured data (which represents the majority of data being created today).

The vast amounts of data being created is a fantastic opportunity to come up with new and interesting propositions for your business. However, when starting out, it can be daunting to figure out the first step to becoming a data-driven company.

A problem well-stated is half-solved — Charles Kettering

This is one of my favourite sayings, as it has taught me to not jump to solutions quicker than I have fully understood the problem I am trying to solve. It is extremely relevant in this context, as often I have seen startups eager to start building data science tools without having previously thought holistically about how their business is positioned to leverage their data. Investing time into developing a data strategy early will help you scope out any current limitations of your data & infrastructure as well as identify which applications of data science will deliver the most impact to your business.

What is a Data Strategy?

A data strategy is a living document that defines how the business is setting itself up, both from a technical and organisational perspective, to proactively manage and leverage data towards the long term business strategy.

Wow that’s a mouthful, let’s break it down:

• A data strategy is a living document… — A data strategy should absolutely not be a document that takes months to develop and then is put to one side gathering dust for no one to read again. By all means, the data strategy can be a document, a presentation, a Notion page, or whatever other medium you choose but above all it should be a living document — meaning that it is constantly updated and refreshed as the business evolves
• …defines how the business is setting itself up, both from a technical and organisational perspective… — A good data strategy should ideally cover both the engineering and infrastructure approach, as well as how you plan to structure the organisation and teams to make the most of your data
• …to proactively manage and leverage data… — By creating a data strategy, you make sure that you are taking proactive measures to make the business as resilient as possible, but also to give it as much of a competitive advantage as early as possible
• …towards the long term business strategy — Last, but certainly not least, a data strategy should not exist in isolation from the rest of the business. Rather, a good data strategy will have the business strategy as its north star and all of the plans should stem from this.

Now that we have a unified understanding of what I mean by a data strategy, let’s take a look at a couple of reasons why I think every early stage company needs one.

1. Avoid “data debt”

I have worked with several organisations in the past that have a very clear idea of which data science application will help move the needle in the business, only to realise that they had deficiencies in the data collection process meaning they don’t have the required data to undertake the project.

I like to call this “data debt”, and depending on the size and scale of the business, it can have drastic implications on your overall plans.

A data strategy will answer questions like:

• Which data science applications will bring me closer to achieving my business strategy?
• What data should I be collecting to undertake those projects?
• What external data can help me accelerate my progress towards my objectives? Often in early stage companies, it takes time to collect enough data internally to start building ML algorithms and one should consider turning to external data sources (check out Google Datasets) to kick start those projects. However, finding the right external data can be expensive and time-consuming, making it more imperative that you have thought about what types of data will deliver the best ROI

2. Minimise bias in your data

Bias has become a central topic of conversation in the data science and machine learning community in recent times, and for good reason too. Many more decisions these days are being made by algorithms, with people wrongly assuming that this would remove “human bias”. However, the truth is that algorithms only work because they seek to replicate patterns found in data, including bias!

For example, say you were trying to build an algorithm to determine who should or shouldn’t receive a loan. Historically, those decisions would have been made by humans who could have been prejudiced across a variety of characteristics (race, gender, religion etc.). If you train an algorithm on historical decisions made by these humans, the algorithm will only perpetuate these biases.

This topic merits an entire post for itself, but for now, make sure you are considering the following points:

• Is the data you are collecting representative of the population you are serving, or are there subsections of the population that are being excluded?
• Are there systemic errors in the data collection process that will lead to bias?
• If you plan to expand into different geographies, have you set up your systems and algorithms such that they won’t break or begin misbehaving in those new areas?

3. Ensure data security & governance

While I endorse the creative use of data to gain a competitive advantage in the market, there is no question that users’ privacy and the security of their data must be guaranteed. Consider which regulations apply to your company (e.g. GDPR, CCPA) and use the data strategy as a tool to double check that you are compliant. Furthermore, think about how to best position yourself for any information security audits or certifications that will be required by any clients if you are in a B2B business.

A few questions to ask yourself:

• How much data do you need to collect to meet your business requirements?
• What permissions are required to collect someone’s data?
• How will you ensure the data is stored securely?
• What processes do you have in place to make sure the data is kept accurate and reliable?

4. Hire the right talent

As a data scientist, I may be tempted to believe that data is the most important asset in a business, however, no one can dispute the fact that people are the lifeblood of any business. In order to hire the right people, you must first understand what kind of data and tools will form part of your strategy and what skill gaps exist within the organisation as a result.

When approaching hiring, these are the steps that help me frame the skills required:

• At the core of your data strategy is your business strategy. From this you should identify the top priority data projects for the next 6–12 months — ideally, your new hires will have had experience in these types of projects
• Not all data is created equal, data can be structured or unstructured, and within those there are several different types of data (time-series, text, geospatial, image). Each data type has its own nuances of how to best process it and what kind of techniques to use. Someone who has previously worked with similar data types will be able to hit the ground running quicker
• Finally, what tools and technologies are used in your company? While this is maybe less important than the above two points (seeing as how you can generally transfer skills learnt across tools), again hiring someone who understands the ins and outs of these will be more suitable

It is really important to strike a balance between hiring to accomplish short-term objectives, and building teams that will evolve with your business. Ensure the diversity and adaptability of your team before making any big hiring decisions.

5. Decisions, decisions, decisions

In my experience, the best teams are those capable of making quick & optimal decisions about the direction of the work. The teams that have the right data available at the right time are much more likely to succeed in quickly making decisions that are closer to the optimum.

I would argue one of the most important KPIs to track when thinking about a data strategy is “Time to Answer”, defined simply by “how long does it take to find an answer to a question about the data?”. For example, say I wanted to find out what % of people landing on my site are converting to users, how long would it take me to get an accurate answer?

A few things to consider here are:

• Does the business have the tools in place for technical and non-technical people alike to find answers in the data?
• Are there any clear bottlenecks that need to be resolved?
• Is there a single source of truth of the data, that makes it easy to find an accurate metric?

With this in mind, a good data strategy will explain both where different sources of data will be stored and how different teams in the company will be able to access them.

There are many other reasons to invest time in building out a data strategy, but I hope that these 5 have shed light into the importance and value of one that is well thought out.

In my next post, I will cover what I think are the fundamental aspects of a data strategy for early stage companies. Stay tuned!

Thanks for reading. Are you an early stage founder or working in a startup? I’d love to hear what are some of the challenges you’ve faced when thinking about implementing data science in your organisation. Please comment below or let’s connect on LinkedIn.

Gerard

Gerard is a data scientist at Founders Factory, where he advises early stage companies on how to develop good data practices and leverage data science & machine learning.

5 Reasons Why You Need A Data Strategy was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Below are some of the trends and highlights that I’ve taken away from Day One:

There are imperative questions surrounding AI progress that we need to answer

Joanna Shields, CEO of Benevolent.ai, began the day with a keynote speech at CogX’s biggest stage. Her talk was on the AI revolution, describing the AI revolution as a “quantum leap”, relative to the “warp speed” of the digital revolution. However, in her talk, she raised the critical issue of needing to think about how to ensure AI brings beneficial outcomes for all, and how we can keep power in check. In the future, when people write about these years as the Age of Artificial Intelligence, will they write about this time as an age of prosperity, or will they look back with regret, for what humans have lost? At the end of her talk, Joanna provides us with 3 questions to begin the conversation on ensuring beneficial AI:

“When it comes to abdicating responsibilities for our decisions to machines, how far should we go? What boundaries do we set? And who gets to set them?

“What happens when our relationship with technology stops being that of creator and creation? What happens when AI begins creating itself?”

“How do we make sure we get the AI we deserve? How do we get the AI that ensures we, as a society, can fix our biggest problems and provide well-being for all?”

Healthcare is incredibly ripe for disruption

Healthcare turned out to be one of the main topics of discussion in Day One. The impact stage was host to several talks about healthcare, including “Could AI cure cancer?” and “Is artificially extending life intelligent?”. These talks were presented by both executives of AI companies and medical doctors alike, but the main message was abundantly clear; there is an incredible potential to disrupt the healthcare industry using AI, all the way from self-care to drug discovery. There was even talk of a newly discovered drug that delivered all the effects of exercising without having to exercise… though in its current form it is toxic to the human body!

Trending AI Articles:

1. TensorFlow Object Detection API tutorial

2. Deep Learning Book Notes, Chapter 1

3. Deep Learning Book Notes, Chapter 2

Earlier in the day, Professor Jürgen Schmidhuber, Director and Professor at the Swiss AI lab, alluded to the fact that AI up to now has conquered the world of marketing. Applications of AI have perfected the art of understanding what ads we are likely to click on, what products we are likely to purchase, and what content we would prefer to read. Marketing and advertising, he said, sits at around 2% of the global economy and is dominated by a few big players (Facebook, Google, Amazon etc.). However, healthcare makes up 10% of the global economy, and there are no outright big players in the industry yet. It is likely this will be a fiercely competitive market to fight for that piece of the big pie.

AI teams (and tech teams in general) need to be more diverse

There is a diversity epidemic in the world of AI. Companies are struggling to form diverse teams, and to attract talent from a wide variety of backgrounds. A panel made up of some incredible women, Hanna Naima McCloskey, Josie Young, Sofia Olhede, and Carol Reiley, discussed not only the need for diversity in our data sets that we use to train our machine learning models, but also the need to have more diverse product teams such that the products we build take into consideration the perspectives of minority populations.

My favourite analogy of the day was describing the recruitment and development journey of women, and minority groups in general, as a leaky pipe.

Sure, we can and should inspire more women to take up STEM subjects and to pursue further education in these areas (equivalent to pumping more water into the leaky pipe), but if we do not fix the leak in the pipe then this will only solve half the problem.

The leaky pipe here refers to the fact that while organisations may strive for a more balanced gender ratio at entry level positions, this ratio quickly becomes male dominated again when going up the corporate ladder. This doesn’t only happen in corporations, it is also the case in academia for STEM subjects. To fix this leaky pipe, we need to make sure our organisations are inclusive, and support minorities in tech throughout their career development. We must also teach startup founders to learn to prioritise diversity early on in the founding of their venture. After all, if a company prioritises speed and profits, then it can be no surprise when their organisation is not as diverse as they would have liked, simply because diversity becomes an afterthought.

CogX Day One Highlights was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.