REPLY Ensuring Quality of Models (SD6666)

[SDMAIL Jean-Jacques Laublé]

Posted by  Jean-Jacques Laublé <jean-jacques.lauble at wanadoo.fr>

Hi Richard

About your question:

What procedures do people use to ensure that the model formulation, testing,
evaluations cycle is done in a rigorous manner?  How could that be improved?

Like Fabian, I think that there is no real answer to your question. 

For the simple reason that there seems to be different methods of building models. 
The process of validation will then be very different, depending on the method. 

For instance, if you look at the web site of Ventana systems, you will see a 
very simplified process of building and you will notice that it is relying heavily 
on reality checks, right from the start. 

They verify all along the process that the model conforms to a predefined set 
of standard behaviour of reality plus a set of past data. 

On the other side there does not seem to be a lot of qualitative analysis with 
diagramming. 

If you study the Vensim user guide, there is too little mention of diagramming, and 
the modelling process starts immediately with a simple quantitative model with a single 
stock. 

>From all the models used, there is only one that refers to past reference modes, and the 
other ones do not. 

Logically all the models from the modelling guide should be constructed with a set of 
reality check equations. No one has it. 

That means that what is prescribed is not done. 

There is only one model with past references and no models with reality checks. 

This could be interpreted as if in the reality, reality checks or past references are rarely 
useful. 

But when one sees the Ventana systems web site, one sees that data and reality check are 
heavily used. 

I can add that Vensim being one of the most used software, I have never seen any model 
with reality checks in published articles or papers. 

All this seems rather incoherent. 

I think that one of the reasons of not using reality checks, is that it needs a lot of work 
and experience, especially in choosing the right equations and a very good knowledge of 
reality. You can choose hundreds of reality check equations, even for a relatively simple model. 

But going further on, some methods recommend building a complete conceptual qualitative 
diagram, eventually with the stock and flows, prior to building  a quantitative model (Hines, 
Coyle). 

Sterman advocates too building a diagram, but thinks that it is better to quantify and get
data to compare them with reality, to verify the validity of the diagram. 

But the big difference comes from the Hines and Sterman way of building a diagram and the 
Coyle's way. 

I have never personally understood the Hines way (I can see him nodding his head across 
the ocean!). 

I have tried to use it and it never worked for my cases. 

To resume it too simply, I do not believe that the study of reference modes can give a precise 
indication on the structure of a model and can give a hint on an eventual hypothesis about the 
structure. 

It works with simple models or if the problem has a dominant loop and no exogenous data, 
otherwise the past behaviour can be the result of multiple loops and external influences. 

When one studies Coyle's way to build a diagram, he just does not use reference modes at 
all. Is it for the same reason than the one I described? 

More on that he does not use reference modes to verify his models too!

At least not in the models he studies in his book 'System dynamics modelling a practical approach'. 

I have not yet finished studying the book that is extraordinary dense. 

With the time I like less and less using reference modes, because the only thing that it 
can tell you, is that the model is wrong, (which is not so bad) but it cannot tell you were it is wrong. 

It can eventually show you that the model behaves relatively like the reality, but at the condition 
that you know how much variable you have to survey, how much independent they must from one 
another and how much close the models data must be from the real ones.  

The thing I do not like in reference modes, is that they act like a block box, they give you 
results, but do not explain why? 

To resume Coyle's way to analyze things relies heavily on the structure and on its deep 
understanding and not very much on the past behaviour. 

One of the particularities of Coyle's method, is to show how to analyze qualitative 
diagrams in a very original way. He recommends building diagrams at different levels 
of abstraction, which is the best way I know to really understand deeply a model.  

Of course I like Geoff's method because it is based on deep understanding of the 
problematic situation and not on past data that can be biased, difficult to get, 
representative of a past situation and not of the future. 

It is the only complete and coherent published (there might be better unpublished 
one's) method of building a model, which starts from the problem definition to the 
finished model, training the reader all the way with about 8 models, through all 
the steps of the modelling process. 

The only think lacking is how to build a sound written definition of the problem. 
By interviews, using cognitive map techniques? 

The book is unfortunately not very appealing, written with little characters, based 
on an old software, no more published: cosmic and cosmos. 

I personally do not mind because I always do myself the modelling before reading 
the solution of the author directly in Vensim. 

After this too long ( I had not the time to make it shorter) explanation, 
you will agree that there is still a long way to an unified method of building models. 

Regards. 

Jean-Jacques Laublé. Eurli Allocar 

Strasbourg, France
Posted by  Jean-Jacques Laublé <jean-jacques.lauble at wanadoo.fr>
posting date  Tue, 13 Nov 2007 12:39:03 +0100