Model design and forecasting errors

Modelling errors

In a recent post I discussed the fact that errors in our transport models are practically inevitable. Our current models have thousands of links, millions of cells in our trip matrices, hundreds of parameters and thousands of lines of script defining the structure and processes to be followed. Moreover, many of these have been inherited from earlier incarnations of the model and reflect different coding practices and assumptions that are never fully documented. The opportunities for errors in the model are legion; the chances of eliminating all of them minuscule. Of course we should try but why make our task more difficult than necessary?

It makes sense, therefore, to design a model system bearing in mind the opportunities for error and avoid the over-specification risk. In order to do this we must look at the dimensions of our proposed model and the errors that are more likely to occur; these will not be just modelling errors but more general forecasting errors.

Forecasting errors

I consider four main sources of errors and uncertainty in model based forecasting:

1. Errors in the base year data (we do know they exist, small sample, blended data, mythical “average day”, etc.) They can be reduced with better data collection methods and perhaps new data sources like anonymised mobile phone traces.
2. Two kinds of errors in the model. First, simply coding, scripting or parameter errors as discussed before; they can be reduced with better quality assurance but never fully eliminated. Second, just the limitations due to the simplified nature of our models (behavioural responses omitted, trips rather than activities, ignoring the dynamics of congestion, the mode on the trip back home different from that to work, etc.) They can be reduced with better models but as time passes these errors will be magnified, as tastes and decisions structures evolve.
3. Errors in the future inputs to any model as depend on imperfect independent forecasts for the economy, population, fuel prices and so on. These will be greater than on the base year data and will grow with time as the further we look into the future the less certain we are. I suspect that the more detailed and refined the future data needs the greater the potential for these errors.
4. Finally, scenario changes. Semi-expected changes that affect the forecasts. These are clear for transport concessions (what if another untolled bridge is built, what if the feeder modes do not materialise, what would the ferries do when we build the Channel Tunnel….). For transport plans they would be major disruptions like Self Driving Vehicles (SDVs) and peak-car or a major change in energy prices.

The figure heading this post illustrate this; the vertical axis is only a notional uncertainty/error index. It suggests that errors in our forecasts will be inevitably much greater than errors in the base year.

There is a level of uncertainty in our model outputs that cannot be eliminated. There is a limit to the forecastability of the future of transport whatever the level of effort and sophistication adopted. Therefore, it is fair to question what should be the appropriate level of detail for a model.

Model granularity

By granularity I mean the level of detail in a number of different dimensions:

• Zones (large or small, or even addresses)
• Links (and their delay functions, with or without junctions and strict capacity constraints)
• Time periods (AM, PM, OP, Week-end, Winter, Summer, etc.)
• Person types (gender, age, car availability, income, HH type, type of job FT/PT, ticket type, place of birth, etc.)
• Purposes for trips/tours/activities
• Behavioural responses considered (Assignment only, Mode and destination choice, time of travel choice, etc.)
• Special generators/attractors (airports, stations, ports)

My concern in this respect is that as we increase the granularity or precision of our models we also increase future year input errors and therefore reduce their forecasting accuracy. This issue is often referred to as “the curse of dimensionality” or “the overfitting problem”, two related issues discussed from slightly different perspectives; searching for these terms in the internet delivers some interesting insights. Strangely, it is little discussed in transport model development.

As a starting point we could try to establish a reasonable metric for granularity. This metric should be good enough to serve as proxy for model complexity and facilitate comparisons between models in these terms. This is an interesting but very difficult task; I think that only a very rough approximation would be possible.

Ideally, we would like to know how much each increase in granularity, for example adding one journey purpose, adds to the realism of our models in the base year and also how much it increases the uncertainty in future year forecasts. In addition to this, we would like to know how much it adds to data collection and processing costs and the opportunity for coding errors; in the case of a journey purpose the cost is marginal on conventional data collection but more significant in processing and forecasting. Other additions to granularity may be more valuable and also more expensive, for example moving from link to junction delay functions.

It would be possible to produce a generic list of the likely impacts of different components of the granularity of a model to provide some guidance to the model design task.

Probably the best paper I know that treat this issue is by Zhao and Kockelman “Propagation of uncertainty through demand models: an exploratory analysis” at Annals of Regional Science 36 (1):145-163, 2002. They suggest that the greater the granularity the greater the uncertainty propagation.

I welcome the opportunity to discuss these issues