REPLY Ensuring Quality of Models (SD6660)

[SDMAIL nickols at att.net]

Posted by  "nickols at att.net" <nickols at att.net>

Richard Dudley, prompted by Forrester's article "The Next Fifty Years," 
comments about ways and means of ensuring the quality of models used.  
That's probably a good idea but I can envision a snag or two.  While 
head of strategic planning and management services at ETS, I tried a 
number of times to interest senior managers and execs in system 
dynamics.  One of them involved some real SD modeling.  I did this on 
my own and I referred the SD practitioner (Jack Homer) to some others.  
Jack did what I thought was good work for us (including clearing up 
some wrongheaded thinking on our part about how to use SD).  Here's 
the snag:  I doubt those for whom the work was done would want their 
models exposed to public scrutiny.  For one thing, they no doubt view 
them as proprietary - as works done for hire.  For another, assuming 
the models did indeed provide value, they wouldn't want that made freely 
available to competitors.  For a third, if the models were a "bust" which, 
in this case they  weren't, why put a misstep on public display?

As for how we ourselves evaluated such models, I can speak only for myself.  
"Goodness of fit" would seem to be the single most important measure.  By 
that, I mean two things.  First, does the model as conceived seem a good 
fit with the variables and relationships making up the system being modeled?  
Second, does the behavior of the model produce a good fit with the data 
generated by the real system?  (When there is no system, as in a situation 
in which you are designing a system to behave a certain way, "goodness of 
fit" becomes a less tenable measure.)

That said, I'm sure there are some arenas where the models developed could 
be publicly reviewed, analyzed and critiqued.  Perhaps others have some 
ideas on that score.

Regards,


--
Fred Nickols
Posted by  "nickols at att.net" <nickols at att.net>
posting date  Fri, 09 Nov 2007 13:14:33 +0000