REPLY Ensuring Quality of Models (SD6662)

[SDMAIL Tom Fiddaman]

Posted by  Tom Fiddaman <tom at ventanasystems.com>

> Ultimately, the ONLY fundamental criterion of SD model validity is "what
> did it change, in what direction, and by how much?"

I'd call this influence, not validity. A completely spurious model could be 
involved in a process that changed things a lot. Influence and validity are 
separate, individually necessary but not sufficient, conditions for a successful 
project. Validity is about the "in what direction" part - i.e. do the contingent 
predictions yielded by the model direct change in the right direction.

> "Goodness of fit" would seem to be the single most important measure.  By
> that, I mean two things.  First, does the model as conceived seem a good
> fit with the variables and relationships making up the system being modeled?
> Second, does the behavior of the model produce a good fit with the data
> generated by the real system?

This is a useful elaboration, because frequently people only think about the second 
criteria - fit to data - which is by itself a weak test. The first test, determining 
whether the model is a good match for the system under consideration, is much more 
important and difficult. Client involvement with the model is one key component of 
such a test, e.g., for establishing the face validity of the model structure and key 
relationships, and for proposing and evaluating experiments that must pass a laugh 
test. Other components include all the things we normally check - units balance, 
nonnegativity of physical stocks, robustness in extreme conditions, agents' use of 
available information, etc.

Typically the application of such checks is informal, in part because modelers work 
in small teams. The software development world has far more elaborate formal 
processes for building in model quality or at least enforcing conformance of product 
to spec. I think many are not appropriate for use because modeling is not the same as 
programming. But most likely we could benefit from routinization of some aspects of the 
validation process.

The only example I know of is Reality Check in Vensim, which lets you build up a library 
of behavior tests that can be automatically applied to a model. Unfortunately it hardly 
ever gets used, even by us. I'm not sure whether that's because the implementation is 
deficient or because modeler incentives favor quantity of equations over quality. But I 
would love to see reality checks made a standard part of every conversation about models.

Tom

****************************************************
Tom Fiddaman
Ventana Systems, Inc.
Posted by  Tom Fiddaman <tom at ventanasystems.com>
posting date  Mon, 12 Nov 2007 10:25:38 -0700