REPLY Do marginal models marginalize modeling? (SD6739)

[SDMAIL Jack Harich]

Posted by  Jack Harich <register at thwink.org>

Richard Dudley wrote:
>  
> I see that the topic has switched from my question about questioning poor 
> models, to a discussion of how we proceed with a good modeling process.  
> Well, that's OK, but it does not really answer my question. 

My point is that "good modeling practices," good enough to reliably solve 
difficult problem, do not yet exist. Solving tough problems with SD is still 
way too dependent on the brilliance of the modeler. But bright modelers are 
in short supply.

The field is unable to consistently solve the great social problems of our 
time, as Forrester alludes to in "System dynamics: The next fifty years" when 
he asks: "Why is there so little impact of system dynamics in the most important 
social questions?" My answer is because the foundation every field needs for 
success is not yet mature. In particular, SD's foundation lacks a repeatable 
process. Therefore "more widespread use" of existing modeling practices will 
not lead to SD success.

Hypothesis: Without a repeatable process, most modelers cannot solve difficult 
social problems.

Corollary: Without a repeatable process, most difficult problem models will 
be low quality.

This hypothesis can be tested. One way would be to writeup a case, on a problem 
with difficulty level similar to the urban decay problem (Forrester 1969). The 
case has all the clues and data you need to build a rough model to solve the 
problem. But the case also has a lot of chaff or noise: irrelevant and attractive 
data that leads to symptomatic solutions that will of course fail. It should not 
be at all obvious what the backbone of the model should be.

Then you run an experiment with the case. The control group spends 30 minutes (more? 
a lecture Q&A? a course?) reading an irrelevant short article. The treatment group 
spends that time reading an article on how to execute a repeatable process, such as 
the System Improvement Process. Then both groups receive the case and try to solve 
the problem with SD. The treatment group is allowed to refer to the written process 
as they go. This of course may require separate rooms for the two groups.

Such an attack would allow the process to be iteratively improved until the 
experimental results were stunning, and the hypothesis was proven to satisfaction. 
The output would be (1) a formal process, a best practice that we could spread, and 
(2) a series of cases that can be used to teach/test this best practice.

Democracy succeeds because it relies on the rule of law, not men. SD too can succeed 
if it comes to rely on the use of process, not men.


> My recent post asks if we should be commenting more on articles which claim to use 
> SD modeling but may be using questionable/or incomplete approaches.  Such commentary 
> may be worthwhile, but needs to be done carefully, and in a constructive way.  It is 
> not easy.

A sort of self-policing by peers. This will be difficult to implement and maintain. 
How about the alternative of self-policing by the the modelers themselves?

Perhaps this could be done if modelers had a standard test to apply to a model. The 
test would objectively determine if the model had met its objectives. A standard 
objective is that it be a high quality model.

Better yet would be if the model was produced with the process the above experimental 
program produced. Then we are not trying to test defects out, which is poor practice. 
We are trying to prevent defects at the source.


> I am working on such a comment now, and would like to have a reviewer look at it for me... 
> any volunteers?
  
I'd be glad to help. But I'm not an academic. I'm an engineer/consultant. So I have some 
limitations.
  
> A few months ago I asked a similar question in a more narrow sense: Do organizations 
> ensure that the models their employees produce meet certain quality standards, and what 
> review processes are used, if any.  There were not many responses.
   
Hmmm, a nice question. Perhaps I've addressed it above.

Jack
Posted by  Jack Harich <register at thwink.org>
posting date  Sun, 10 Feb 2008 09:54:31 -0500