Breaking the Iron Law of Projects

Karlene Agard explains how reference class forecasting can be a valuable and accurate forecasting method that draws on actual outcomes to help you deliver your project to budget, on time and with the promised benefits. What’s not to like?

Project management has accomplished many fantastic feats of ingenuity, but it is not known for its ability to deliver as planned. Less than half of all projects are on budget, only eight per cent are on budget and on time, and a mere 0.5 per cent are on budget, on time and deliver the planned benefits. That’s the ‘Iron Law of Projects’: over budget, over time, under benefits, over and over again. This term was coined by Professor Bent Flyvbjerg, the first BT professor and inaugural chair of major programme management at Saïd Business School, University of Oxford, and the chair of Oxford Global Projects (OGP), based on analysis of over 10,000 projects.

Across the public, private and third sectors alike, the Iron Law results in wasted money, inefficient delivery and time pressure. While the private sector loses out on profit, the third sector is less able to help its beneficiaries.

Alexander Budzier, fellow at Saïd Business School and CEO of OGP, notes that “project management has stagnated for too long. We have to get better at forecasting and that necessitates learning from the past performance of projects: not just through anecdotes and experience, but using quantifiable, objective data.”

Some believe that projects are underestimated because of technical errors. For instance, when constructing a tunnel, the ground conditions are unknown and may be worse than accounted for in the plan. Those who believe technical error is the cause of forecasting inaccuracy would argue that it could be combatted through additional data or modelling. However, technical error is not a satisfactory explanation, because it would lead to underruns occurring roughly as frequently and with similar impacts as overruns. This is clearly not borne out by the data, as fat tails (extremely large values) abound.

Biases account for the project distributions that we see. Project resources are regularly underestimated due to optimism bias (the tendency to have a rosy perspective of the future) as well as other psychological biases, and strategic misrepresentation or political bias – the use of forecasts that are known to be flawed due to political pressure (which can be internal or external).

Reference class forecasting (RCF) bypasses the biases shared by experts and non‑experts alike. It is the most accurate forecasting method because it draws on actual outcomes and not just flawed estimates. Consequently, we are seeing a continual increase in the recognition of RCF as a crucial methodology for forecasting. RCF is now required for large rail and road projects in the UK and Denmark. Other governments have also used RCF, including those of Sweden, Switzerland, Norway, the Netherlands, South Africa and Australia. RCF has been recommended by the American Planning Association, the Royal Institution of Chartered Surveyors and the Major Projects Association. It will also feature in APM’s upcoming Project Risk Analysis and Management Guide refresh.

The benefits of RCF

Nobel Prize winner Daniel Kahneman praises RCF in his book Thinking, Fast and Slow as “the single most important piece of advice regarding how to increase accuracy in forecasting”. Having a more accurate forecast makes managing resources across programmes and portfolios far more effective and would reduce the likelihood and impact of having to pay a premium for delivering later than expected. For instance, avoiding direct costs for contracting penalties, higher overtime rates and unnecessary logistics, as well as the indirect impacts of stress and opportunity costs. There is also the added reputational benefit of being able to deliver as planned, rather than adhering to the Iron Law.

Flyvbjerg notes that, “RCF is extremely versatile and it’s the most accurate forecasting method that exists. It can be used on any quantifiable prediction problem, as long as you can get the data. You can use it to forecast tunnelling speed for track boring machines in England, solar farm energy output in Ghana or sales for a tailor in China. If you can get data, you can use RCF to create the most accurate forecast that exists, no matter what you’re forecasting.”

The three steps of RCF

Another benefit of RCF is that it is relatively simple; you can complete a forecast using the internet and a spreadsheet with three straightforward steps:

1. Identify a relevant reference class of completed projects.  Find at least 15 similar projects. The more projects you can find, the better. Make sure that you get quality data from a credible source.
2. Establish a probability distribution for the reference class.  If you were looking at cost overruns, you would line up the overruns from smallest (which could be negative, ie an underrun) to largest. To create a cumulative probability distribution function, you would correlate each overrun with a probability. The smallest overrun would be 0 per cent and the largest would be 100 per cent. Intermediate overruns are assigned a probability at equal intervals (eg if 20 projects are in a reference class, each project has a five per cent share).
3. Position your project on the probability distribution to make your forecast. It is generally safest to assume that you will perform in a similar manner to projects in the reference class, rather than that you will be better. Uplift your project estimate in line with the performance of the reference class and your risk appetite. If you are happy to have an 80 per cent likelihood of your project falling within the budget and accept a 20 per cent likelihood of going over budget, you would set the budget at P80. In the example above, that would correspond to a 55 per cent uplift.

While RCF doesn’t require coding, machine learning or an advanced degree in artificial intelligence (AI), these more complex methods can also be used to improve your project forecasts.

An early warning system in Hong Kong

Hong Kong is a rarity because its projects often come in under budget. Construction projects typically cost 15 per cent less than forecasted. Although Hong Kong outperforms most jurisdictions in cost management, the Hong Kong Development Bureau was keen to further enhance cost management by providing early warnings to assist projects in getting back on the right track.

OGP and Octant AI were brought in to address the challenge of the early detection of projects that are starting to go awry. It’s helpful to know which projects are spending more quickly than expected (which can indicate that a project’s progress is fast or it’s heading towards a cost overrun) or more slowly, so they can proactively take steps to improve it earlier on.

OGP partnered with Octant AI to implement a tool that could do just that. The early warning system analysed the projected cashflow and compared it with the emerging actual spend. The tool flagged projects that were deviating from planned performance and highlighted the extent of the deviance using a red, amber or green (RAG) rating. Amber or red ratings were given when the cashflow was off‑piste and would trigger senior managerial intervention to get the project on track.

After a year, it became clear that the monitoring system could be used for benchmarking targets. A total of 849 completed projects were analysed in depth by an unsupervised learning algorithm, which found that three factors affected performance in a statistically significant manner. These were:

1. Category: building/non‑building
2. Forecast cost for buildings: less than HKD150m or higher
3. Duration of non‑building projects: shorter than six years or longer

The AI produced typical cashflow s‑curves for the four project types. These s‑curves were then RAG rated. The AI assigned flags based on cost and schedule performance. It compared their performance to the original plan, as well as a percentage of the ultimate performance for that project. Seventy per cent of the Bureau’s project data was used in developing the AI and its performance was tested on the remaining 30 per cent of projects. The AI was far more accurate at correctly flagging projects. Up to 69 per cent of the flags were correct and the red flag assignments were 20 times more accurate than a random allocation.

The next phase was forecasting outturn project costs. This was done with a more advanced AI, a deep artificial neural network. It predicted the final outturn cost with an average error of only ±8 per cent. The Association for the Advancement of Cost Engineering suggests that international best‑in‑class cost estimates have an error between ±3 per cent and ±15 per cent. It would expect that a design needs to be completed with detailed unit cost and prices to achieve this level of accuracy. The permanent secretary for development (works) of the Government of the Hong Kong Special Administration Region “welcomes the adoption of AI to help all major project leaders across different sectors take Hong Kong’s construction industry to new heights”.

Cuong Quang, the general manager of innovation and technology at Octant AI, said “we are excited at the potential of improving forecasting generally, and RCF in particular, using AI. Our work and research shows that historical data and machine learning can be used to extend the forecasting benefits of ‘outside view’ methods like RCF into the emergent complexities of project delivery. The excellent work done by OGP, whom we were delighted to be a partner with, for Hong Kong Development Bureau shows how a hybrid approach can improve dynamic final cost and time forecasting.”

Get started!

Whatever your project organisation’s technical maturity, you can use data to improve your project forecasts, make better use of your resources and break the Iron Law. If you would like to learn more about RCF from the global leaders, Flyvbjerg and Budzier, you can access hours of free resources at OGP Academy Archive or join a course at RCF Course. 

Listen to APM’s podcast with Professor Flyvbjerg, part of the ‘Project Innovators’ series.

Six tips for RCF

1. Uniqueness bias is a trap to be avoided. The DNA of your project may be different from others, but it shares the same basic building blocks with many projects.
2. You need a robust and objective reason to assume your performance will differ substantially from the reference class. ‘We’re a great team’ is not sufficient, but ‘this is the 15th time we’ve done this type of project and our performance is better according to tests for statistical significance’ is.
3. If in doubt, draw on a wider reference class, rather than a more specific one.
4. Give careful consideration to the quality of the sources used. Data from academic journals is high calibre because of the independent peer review process.
5. Don’t go to the effort of creating an RCF and then leave it to metaphorically gather dust in your digital archives. As time progresses and additional relevant projects are completed, continue to update your RCF when you get access to credible data.
6. If you’re in a large organisation with a history of project delivery, it is likely that you can get access to a treasure trove of data. Set yourself the challenge to build a robust dataset of external projects and compare the performance. Transform the data into information that helps you break the Iron Law over and over again.

 

Karlene Agard is a senior consultant at Oxford Global Projects.

THIS ARTICLE IS BROUGHT TO YOU FROM THE AUTUMN 2021 ISSUE OF PROJECT JOURNAL, WHICH IS FREE FOR APM MEMBERS.