REPLY Who wants to share models (SD6957)

[SDMAIL Tom Fiddaman]

Posted by  Tom Fiddaman <tom at ventanasystems.com>

Mabel Fong raises a good point:
> First, is it not a general commonplace of complex non-linear
> dynamic systems analysis that one does not even claim to have a model 
> until it
> has produced the same stimulus/response behaviors when programmed in 
> different
> languages and run on different CPUs? Is there something special about SD
> training, SD software, or the problems addressed by SDers that 
> obviates such
> precautions? Or, is it possible that there are ‘validated’ SD models 
> that might
> be called into question if they were subjected to such tests?

It's certainly a good idea to verify numerical sensitivity of models, 
but I doubt that it's commonplace, at least outside the natural 
sciences. Verification more commonly focuses on the numerical algorithm, 
rather than on models implemented using a given algorithm. One might 
argue that most SD software, having been around for a while, is well 
verified, though I'm reluctant to take that on faith. Given the 
counterintuitive nature of systems under study, it would be easy for 
errors to persist, with users rationalizing them away. However, if one 
follows good SD practice by exploring the model's sensitivity to it's 
time step (DT) and solution method (Euler, Runge-Kutta, ...), most 
problems are likely to surface. Most will be due to formulations, not 
the underlying software.

There is definitely potential for problems though. I have run across 
models that behaved differently in single- and double-precision versions 
of Vensim and on different CPUs. In those few cases, the differences 
were due to what I would consider to be bad formulations, and thus cast 
doubt on the model rather than the software. I also know of one model - 
Mosekilde et al.'s model of chaos in rat kidneys - that is good, but 
still numerically sensitive. I replicated the model from a paper, using 
a different numerical scheme (Vensim's FIND ZERO function) to solve a 
simultaneous equation. My replication generates the same behavior modes, 
but with chaotic transitions at slightly different points in the 
parameter space. The model is at
http://metasd.com/models/index.html#Science

I suspect that most SD problems are not really testing the limits of 
numerics. Most models of an organization do not track many periods of an 
oscillation, or experience wild shifts in feedback loop gains. Even if a 
model is chaotic, generally the horizon of interest is short compared to 
the time scale over which numerical errors would cause results to 
diverge. In general, and especially with chaotic or PDE systems like a 
turbulent fluid, one can't count on such benign behavior, and one 
shouldn't count on it even when expected. So, again, one should:
- Vary the model time step (a good rule of thumb is to keep halving it 
until the model behavior stops changing). Bear in mind that, if the time 
step gets too small, increasing roundoff error offsets gains in 
precision. A time step that is a power of 2 minimizes roundoff error. 
Increasing the time step may also be a useful test, for robustness if 
nothing else.
- Vary the integration method. This may not work well for models with 
discrete events. Oscillatory models should generally not be using 
Euler's method, which tends to amplify oscillations.
- Explore the model's parameter space extensively
Running the model on different software or hardware provides an 
additional test, but of these four, I would expect it to be the least 
likely strategy to reveal errors.

Stochastic models pose a particular challenge, because experimental 
changes (e.g., in the time step) will also cause changes in the random 
draws realized. This makes results non-comparable. My personal 
preference is to build models such that all stochastic inputs, and 
perhaps also data drivers, can be turned off, so that a clean baseline 
is available for numerical and equilibrium experiments.



****************************************************
Tom Fiddaman
Ventana Systems, Inc.  
Posted by  Tom Fiddaman <tom at ventanasystems.com>
posting date  Mon, 21 Apr 2008 17:12:39 -0600