REPLY Ensuring Quality of Models (SD6660)

[SDMAIL Jack Homer]

Posted by  "Jack Homer" <jhomer at comcast.net>

Kim Warren cites Geoff Coyle:  ".. A model should do what the real world does, 
and for the same reasons."  That's still not quite enough.  The SD approach to 
model evaluation (as in Forrester and Senge 1980) also considers whether the 
equations are robust to extreme conditions, the constants and table functions 
have plausible values, the boundary is adequate for testing all relevant 
questions, the outputs are plausible under all test conditions, and whether the 
model is transparent, has explanatory power, and provides results that provide 
novel insight.  In other words, we require that our models not only look 
realistic, but also that they be robust and useful.

Kim asks how one can establish confidence in a model that is highly aggregated 
and may therefore seem abstract and distant from real-world data. This raises a 
good question about one of the sometimes-overlooked requirements of good modeling: 
the ability to find (and cite) lots of real-world data, and then to summarize or 
capsulize it in the form of a single variable.  Often, an existing metric exists 
that will serve as a good representative proxy for all of the real-world details, 
that will persuasively "sum up" all of the details.  Every social science needs 
and rests upon such summary measures, and so do we.  If a commonly accepted summary 
measure does not already exist, the modeler will have to find or create one that 
seems to fit the requirements, and make a good case for it.

For example, in a high-level model of health and health care (see current issue 
of SDR), we wanted an overall measure of ill health prevalence, not the 
prevalence of any disease in particular, but all significant symptomatic disease, 
and especially chronic disease, combined.  Time series exist on the prevalence of 
dozens of individual diseases, but one cannot just add them up because of (a) 
different levels of severity and (b) significant overlaps among them (if you simply 
add them all up, it appears that over 100% of the population has chronic disease).  
Instead, we looked to data on self-reported health status: excellent, very good, 
good, fair, poor.  These self-report measures have been tracked annually since 1982 
and have been found to be good predictors of individual health care utilization. 
"Fair+poor" is typically 10% of those surveyed, while "good+fair+poor" is over 30%.  
We ended up using "good+fair+poor", because a sum of the prevalences of just a handful 
of mostly non-overlapping symptomatic diseases (heart disease, cancer, diabetes, asthma, 
bronchitis, arthritis) gives at least 30%, and much exceeds 10%.  Thus, we have a 
metric that (a) reflects the notion of "at least moderately symptomatic disease" and 
(b) rests upon accepted data with a long history.  It is not a perfect metric, but it 
is the best one available, and health professionals who have read our work have for 
the most part accepted our use of the metric.  That's what I think you look for in a 
good summary measure.

Jack Homer 
Posted by  "Jack Homer" <jhomer at comcast.net>
posting date  Mon, 12 Nov 2007 09:37:41 -0500