Cost models are used in many different areas of analysis and decision-making. At Naiad, we focus on models of infrastructure, engineering and the environment, but models can be built that are relevant to many different sectors such as finance, services and manufacturing.
Fundamentally, a model is a useful representation of something else, typically some part of the real world or a system of interest. Sometimes a model tries to project a view of the future, and we call these type of models forecasts. Models can be empirical (data driven) or mechanistic (process driven), or in many cases contain aspects of both. Even a detailed ‘cost build-up’ or ‘bill of material’ used for construction estimates are a kind of model, combining a large number of mechanistic assumptions about the sequence and construction processes to be followed, together with empirical cost rates for material, plant and labour.
Note that the diagram above is simply about classifying different types of model in terms of their approach and granularity, rather than about different approaches being right or wrong. Whether a particular approach is appropriate for generating accurate and useful results depends on many other factors, some of which are discussed below.
Models will typically be comprised of a set of inputs (generally real world data or assumptions), a number of algorithms which represent processes or relationships, and outputs which the client is interested in. Cost models fall within a branch of modelling known as econometrics, which is the area of modelling and mathematics concerned with representing economic systems. However, the modeller may also need to represent other aspects of interest – such as a construction programme in the case of an estimator.
What makes a good model?
Accuracy is generally the most important requirement when building a model or making a forecast. However, it can be worth distinguishing between accuracy and precision, as accuracy refers to how close your prediction is to the truth, overall, while precision is only about how detailed and repeatable your prediction is. The picture below illustrates this distinction, showing how a model can be accurate but imprecise (a large scatter around the bullseye) or precise but inaccurate (a tight grouping far from the bullseye).
The needs of the client are vital to establishing what makes a good model. If the client is interested only in the predictions made by the model, then aspects such as accuracy, speed and computational cost will be highest. However, models can also be used for scenario testing, risk and sensitivity analysis, value engineering and even as ‘digital testbeds’ to test novel ideas before or instead of prototyping. In these cases, the flexibility of the model to accept a range of inputs and parameters, and its robustness to a wide range of scenarios, may be more important than pinpoint accuracy.
How accurate should a model be?
Any modeller must make a decision on whether to have a simple, cheap and fast model or a complicated and therefore potentially expensive and slow model. A more complicated model may be more accurate, but if it is only needed for low risk purposes then the investment needed may not be justified. This trade-off principle is called ‘parsimony’, and means that a modeller should generally aim to build the simplest model they can which gives acceptable accuracy.
What accuracy is acceptable is therefore an important question which a modeller should discuss with clients and decision-makers. It depends on the client organisation’s risk appetite, and the consequences that could follow if the model is incorrect. For example, a model that sets pricing that will be contractually relied upon for a number of years will need to be highly robust, defensible and accurate.
How can I quantitatively assess model quality?
If model quality is considered to mainly be about accuracy, then the modeller will need to assess the ‘goodness of fit’ of their model. This is an attempt to compare the model’s predictions with actual data.
There are many statistical tests and interpretations which can be used to assess goodness of fit, but an appreciation of the aim of the model and the context of the sector are always important. For example, hydrological models often attempt to show river flows against time, and so the model’s ability to accurately predict the magnitude and timing of flow peaks with minimal lag or error may be most important. A cost model on the other hand may not have a time component at all, and its accuracy is purely assessed in terms of the closeness of all predictions to actual observations.
At Naiad we use many different approaches to quantifying goodness of fit, and ensure we spend the necessary time explaining the results to stakeholders in plain and simple language.
Clearly there are other aspects to model quality beyond pure accuracy. Evidence of processes such as quality assurance, verification and validation and peer review are also overarching factors which will give stakeholders and decision-makers confidence in the robustness of a model and the validity of its outputs.
What frameworks and standards are available to guide modellers?
Depending on the sector or industry, modellers will be guided by frameworks and standards that are relevant to their field. In the infrastructure and environment field, models are commonly used for public-sector decision-making and investment. As a result, frameworks such as the NAO’s 2016 ‘Framework to review models‘, are a good starting point for common terminology and best practices for model review procedures. Similarly, the Department for Business, Energy and Industrial Strategy have produced a framework for model quality assurance.
These frameworks are useful, but tend not to be prescriptive about either how models are built (such as platform specifications, programming language) or about how to make the selection of key parameters. We therefore also use more detailed standards (such as the FAST standard for spreadsheet-based models) and guidance (such as DBEIS’s guidance on the monetisation of carbon) in building and assuring models for our clients.
You can see how approaches to model building should be carefully thought out, and draw on the most relevant and current guidance available. Naiad Infrastructure can help guide clients through this process, whether reviewing and developing existing models or building new ones from scratch.