Does project manager matter?

12 min readJan 11, 2023

Game theoretical approach. Working draft.

Abstract

The debate continues about a project management contribution to a success of a project. Some researchers try to answer directly this question. This article proposes a non-biased, agnostic game theoretical approach to measure a project management contribution using as a reference some well known a project risk assessment model.

Introduction

Since the 1950s when PERT technique was developed by US Navy there is ongoing effort to make a project outcomes more predictable. In the 1960s a project management practice became common in many industries. In 1969 The Project Management Institute (PMI) was founded. PMI established a project risk management as a separated body of knowledge. PMI has reported better project performance when formal project risk management practices are implemented, as noted in [14]. PMI in [14] lists a number of a qualitative and quantitative risk assessment techniques.

Among a quantitative risk assessment techniques the one based a Bayesian Networks (BN) seems most suitable to deal with a project uncertainty. BNs were discovered by J. Pearl in [5]. A Bayesian Network is a directed acyclic graph that represents a factorisation of a joint probability distribution over random variables as nodes in a graph where arcs in a graph represents a causal relationships between connected nodes [7]. BN gives both qualitative (by a network structure) and quantitative (by associated to each node a conditional probability table) description of a modeled domain. Another quantitative technique is Monte Carlo simulation, but it doesn’t use relationships between risk factors.

BNs were applied in many domains, among them in a medical diagnosis, insurance, cybersecurity and machine learning. One of the first application of BN to a project risk assessment was in [6]. In XXI century a number of a project risk assessment models based on BN were developed, like in: [1], [3], [4], [12]. A risk models are either based on a collected project data and experts opinions, or in addition on known a qualitative project risk assessment frameworks like in [8], [9]. In the article the well documented BN model described in [1] will be a referencial.

In 1928 J. von Neumann discovered Game Theory, defined in [15] that was further developed in XX century. There is a long list of game theoreticians who won Nobel Prize in economic sciences, among them: K Arrow (general equilibrium), J. Nash, R. Selter, J. Harsanyi (equilibria in non-cooperative games), T. Schelling, R. Aumann (conflict and cooperation), L. Hurwicz, E. Maskin, R. Myerson (mechanism design), L. Ostrom (governance), A. Roth, L. Shapley (market design), J. Tirole (industrial organization), O. Hart, B. Holmstrom (contract theory), P. Milgrom, R. Wilson (auction theory). Nobel Prize 2012 winner L. Shapley discovered the concept known as Shapley value, that will be used for the purpose of the article.

Based on PMI findings regarding an application of a formal project risk management practice in organizations, it is assumed that a sofisticated, calibrated for a given organization, quantitative risk assessment model might be a good source of wisdom about a performance of an organization. In the article the referencial [1] model is used to assess impact of a project manager (or a project management practice) on a success of a project.

This article aims to use the [1] BN model and Shapley value to measure the impact of project managers on project success, and to provide insights for organizations looking to improve their project management practices.

Methods

The referential project risk assessment model [1] developed by E. Lauria and P. Duchessi is the Bayesian Network, presented on the drawing below made in pyAgrum [16], based on research of 312 project, consists of 14th connected nodes.

The Bayesian Network nodes are as follow (according to [1]):

Business Planning represents a quality of a business planning, an alignment of IT strategy
Maintenance represents a maturity of a system maintenance processes
Project Management represents an ability to execute a project
Vendor Support represents a degree of support
User interaction represents a quality of relationship between IT department and a business users
Organizational (Org) arrangements represents a strength of project steering commitment and a quality of allocated project team
IT Management represents an ability to align a project portfolio based a business requirements and provide needed capacity
Functional Management represents a business departments contribution to IT strategy and business requirements
Application Complexity represents overall complexity
Application Suitability represents an application ability to improve a business capability
Application Analysis represents a requirements enginering maturity
Standards Integration represents a quality of vendor evaluation and systems integration
Reengineering Intention represents a business intention to redesign business processes
Benefits represents overall benefits to be delivered by a project.

All the nodes, in order to simplify the model and limit a data needed to provide, have only discrete values: LOW and HIGH. For example a well planned project means that Project Management variable value is HIGH. Business Planning node (or variable) is a parent of 3 nodes: IT Management, Organizational Arrangements and Functional Management. Initially Business Planning variable have a value HIGH with probability 45% (and LOW with probability 55% respectively). A probability distribution of a child node (or variable) depends on its parent or parents nodes. The described model from [1] is a base to assess impact of a project manager (or a project management practice) on a success of a project.

The Shapley value is a concept in cooperative game theory [10]. The Shapley value is unique (or best of known), fair distribution of value among a contributing players. The cooperative game theory assumes that players might form a coalitions. N players might form up to 2^N coalitions (a power set of all possible coalitions over N players). Each formation of the coalition gives some outcome (or payoff) of the played game, according to some characteristic function v that allocates a real number to a given coalition. A pair (2^N, v) is a coalition game. The value of (2^N, v) coalition game is maximum of a characteristic function v, and we assume that value of biggest coalition is value of the game. A formation of the biggest coalition is a dominant strategy for all players. In order to fairly distribute the value of the game among the players based on their contributions to the different coalitions, the Shapley value is calculated for them.

Let’s set up the coalition game for a project. The players set N are all the nodes from the above Bayesian Network without Benefits node which gives a project risk prediction. Among those nodes is the Project Management node, which we focus on. The actual coalition is the set of nodes taking some discrete values, either LOW or HIGH. There are 2 special coalitions: the one in which none take part with all nodes setting LOW, and the one in which all nodes take part with all nodes setting HIGH.

The characteristic function v is defined as follow:

v: power set of N-> [0, 1],
v(X1,…, Xn) = P(Benefits = HIGH | X2, …, Xn) that is conditional probability with the target node Benefits, where n = 13
min v = P(Benefits = HIGH | X2 = LOW, …, Xn = LOW), when all players contribute LOW to the project
max v = P(Benefits = HIGH | X2 = HIGH, …, Xn = HIGH), when all players contribute HIGH to the project.

The function v is monotonical, which means that for any 2 given coalitions S, T, such that S is subset of T, v(S) < v(T). In addition, there is only one coalition possible to form at a time. Function v has some deficiences to be discussed in the next chapter.

For the characteristic function v the values of all 2¹³ = 8 192 coalitions were modeled and calculated with the Bayesian library pyAgrum [16].

The Shapley value was originally proposed in the context of cooperative game theory, but has found many applications in machine learning and data science, especially in explainable artificial intelligence. One of the most popular is Lundberg SHAP (SHapley Additive exPlanations) algorithm [17] that uses a local explanation approach, which means that it computes the contribution of each feature (a player) for a specific instance (a profile) or an observed game state and later generalizes it to get a global view.

In our algorithm, we use a similar approach to the Lundberg’s SHAP algorithm to calculate both local and global Shapley values for a Bayesian network. The reason why SHAP was not used is because of the problem size. We needed an exact Shapley values, SHAP gives approximated values for our problem. Another limitation is that SHAP does not handle Bayesian Network models so far. It might only be done indirectly, by turning Bayesian Network into another prediction model, but still without getting an exact values.

To calculate the local Shapley value for each player in a given profile (a state of a game understood as an instantiation of a some players actions from set: HIGH, LOW), we consider all possible coalitions of outstanding players and their contributions to the prediction, where some players actions are already set.

After calculating the local Shapley value for each player at the instance level, we need to aggregate these values to determine the global Shapley value for each player. One common approach to aggregating local Shapley values is to take the average across all instances. This approach assumes that the instances are representative of the overall dataset and that the local Shapley values are consistent across instances. However, it’s important to note that this approach may not always be appropriate, and in some cases, alternative aggregation methods may be more appropriate. For example, if the instances are heavily skewed or imbalanced, a weighted average may be more appropriate to ensure that instances with greater importance are given more weight in the calculation.

Get Jomsborg Lab’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The calculated Shapley values for all the players are as follow:

Business Planning —0.0069375676515828965,
IT Management — 0.008891450497514867,
Organizational arrangements —0.01683219026444306 ,
Vendor Support —0.0029096888011184473,
Application Complexity —0.002699800377324748 ,
Functional Management — 0.005208489691202088,
Maintenance — 0.023438311607621932,
User interaction — 0.007205021646990719,
Standards Integration —0.003932016402957606 ,
Application Analysis — 0.015547308262584025,
Reengineering Intention —0.07442731062190364,
Application Suitability —0.05834299088275353,
Project Management —0.02417305483712791.

Discussion

The Shapley values are a measure of the marginal contribution of each player to the prediction of the target variable. The higher the Shapley value, the more important in determining the target variable.

Based on the given Shapley values (normalized to %):

The most important players are Reengineering Intention and Application Suitability, with values 29.71% and 23.29, respectively.
Other highly important players include Organizational arrangements with value 6.72% and Application Analysis with value 6.21%.
The least important players are Vendor Support and Standards Integration, with values 1.16% and 1.57%, respectively.

Based on the value of 9.65% for Project Management, we can claim that it has a significant impact on the prediction made by the model. Specifically, it represents the marginal contribution of Project Management when all other players are taken into account, and it indicates how much the prediction changes when we include Project Management in the model.

The higher the Shapley value for Project Management, the more important it is in making the final prediction. Therefore, it suggests that the Project Management is a crucial factor in determining the outcome of the project.

Conclusion

This approach, which combines game theory concepts with a well-established quantitative project risk assessment model, is a unique and novel approach in the project management discipline. While project risk assessment models are commonly used in the field, the use of game theory and Shapley values to measure the contribution of project management to project success is relatively new.

Developed approach provides a non-biased and agnostic way to measure the effectiveness of project management practices, which can be useful for organizations looking to improve their project management processes and outcomes. Additionally, the use of Shapley values allows for a more granular understanding of the impact of project management practices on specific project outcomes, which can aid in the development of targeted interventions and improvement strategies. Overall, this approach has the potential to enhance the field of project management by providing a new perspective and methodology for measuring and improving project success.

While the proposed method provides a non-biased, game-theoretic approach to measuring project management contribution, there are limitations and potential biases that should be acknowledged.

Firstly, the selection and construction of the Bayesian Network model may introduce biases. The model is developed based on a particular set of projects, and may not fully capture the range of project characteristics and risks that are relevant to other organizations or industries. Additionally, the discrete nature of the variables used in the model may not accurately capture the complexities and nuances of real-world projects.

Secondly, the calculation of the Shapley values may also introduce biases. The Shapley values rely on the assumption that each player contributes to the prediction independently of other players. However, in real-world projects, players may interact in complex ways, and the impact of a particular player may be influenced by the presence or absence of other players in a coalition. Additionally, the size of the coalitions (the way how a action taken by players are instantiated) used in the Shapley value calculation may also affect the results, and different coalitions may lead to different Shapley value estimates.

Thirdly, there may be limitations in the interpretation of the results. While the Shapley values provide a quantitative measure of the contribution of each project risk factor and the project management practice to project success, they do not provide information about the direction or causal relationships of the factors, however it is partly resolved by using a Bayesian Network as a model. It is also important to note that the results are based on the assumptions and limitations of the Bayesian Network model, and may not be applicable to other project risk assessment models or real-world project situations.

Finally, the proposed method is limited to measuring project management contribution to project success based on the risk factors included in the Bayesian Network model. It does not account for other factors that may influence project success, such as external market conditions, competition, or organizational culture.

Overall, while the proposed method provides a novel approach to measuring project management contribution, it is important to acknowledge its limitations and potential biases, and to interpret the results with caution.

Future development

There are several directions for future development of the method:

Reviewing other a qualitative risk assessment models based on Bayesian Networks.
Adapting to different models. The current method is designed for use with Bayesian Network models, but it could potentially be adapted to work with other types of models as well. Future research could explore the feasibility of extending the method to other types of models.
Improving the computational algorithms used to calculate the Shapley values. As the size of the model increases, the computational complexity of the method can also increase. Future research could focus on developing more efficient algorithms to reduce computation time. One of them might be an application of recent advancement of a quantum computing.
Automating the risk response strategies. The Shapley values generated by the method could potentially be used to generate an automated risk response strategies.

References

[1] Eitel Lauria, Peter Duchessi, A methodology for developing Bayesian networks: An application to information technology (IT) implementation, 2006

[2] L. S. Shapley, Notes on the n-person game — the value of an n-person game, RAND Project, 1951

[3] Chin-Feng Fan, Yuan-Chang Yu, BBN-based software project risk management, 2004

[4] Yong Hu, Xiangzhou Zhang, E.W.T. Ngai, Ruichu Cai, Mei Liu, Software project risk analysis using Bayesian networks with causality constraints, 2012

[5] J. Pearl, Bayesian Networks: A Model of Self-Activated Memory for Evidential Reasoning, 1985

[6] B. Mccabe, Belief Networks in Construction Simulation, 1998

[7] Norman Fenton, Martin Neil, Risk assessment and Decision Analysis with Bayesian Networks, 2019

[8] Christopher J. Alberts, Audrey J. Dorofee, A Framework for Categorizing Key Drivers of Risk, SEI Technical Report, 2009

[9] Linda Wallace, Mark Keil, Software Project Risks and their Effect on Outcomes, 2004

[10] Robert B. Myerson, Game Theory, Analysis of Conflict, 1991

[11] Marly Monteiro de Carvalho, Leandro Alves Patah, Diógenes de Souza Bido, Project management and its effects on project success: Cross-country and cross-industry comparisons, 2015

[12] Adalberto Ramos Cassia, André Felipe Librantz, Ivanir Costa, Mauro de Mesquita Spinola, Erika Midori Kinjo, Risk Evaluation in Software Project Using Bayesian Network Modeling, 2020

[13] Carsten Kaufmann, Alexander Kock, Does project management matter? The relationship between project management effort, complexity, and profitability, 2022

[14] PMI, The Standard for Risk in Portfolios, Programs and Projects, 2019

[15] J. von Neumann, On the Theory of Games of Strategy, 1928

[16] pyAgrum https://pyagrum.readthedocs.io/en/1.6.1/ — scientific C++ and Python library dedicated to Bayesian networks (BN) and other Probabilistic Graphical Models

[17] Scott Lundberg, Su-In Lee, A Unified Approach to Interpreting Model Predictions, 2017