REPLY Ensuring Quality of Models (SD6661)

[SDMAIL Keith Linard]

Posted by  "Keith Linard" <klin4960 at bigpond.net.au>

The fundamental step in ensuring the quality of models is ensuring that we
are modelling the correct problem.  All too often I have seen consultants
(whether internal or external to the organisation and whether from the SDM
stable or other OR) coming up with technically brilliant quality solutions
to the "wrong" problem.

Echoing previous respondents and the extensive literature on this topic I
would suggest the following are also critical to quality modelling:

1.  Empathy with the client:  Modelling supports decision making.  If I do
not understand the client's needs and the socio-political environment within
which s/he operates, the most technically brilliant quality model is likely
to fail.

2.  Engagement with the client:  SDM is fundamentally directed to changing
structures which affect behaviour.  Changing structures almost inevitably
means challenging "the way we do things around here" ... culture, power,
influence, procedures, networks etc.  The journey to the solution, the
process of modelling, is as important as, if not more important than the
technical quality of the model. 

3.  Building client confidence:  The model structure should be modularised
so that, however complex the underlying mathematics, the top level stocks
and flows structure can be 'read' by the subject area expert.  Three key
advantages of SDM for me have been first the fact that I can use a (top
level simplified) stock-flow model to feed back to the client my
understanding of the current process;  secondly, that I can use a
'simplified' stock- flow picture with causal loops superimposed to
communicate to the client how the dynamic behaviour of the system is
producing the typically unforeseen consequences;  and thirdly, I can use a
'simplified' stock-flow model to show build the confidence of senior
executive that I have indeed included all key relationships.

In regard to the last point, I have found that line operators, technical
experts and management staff have no difficulty in 'reading' their areas and
challenging my understandings of 'how the system works'.

4.  Key modelling techniques for ensuring quality:  (in addition to all the
ideas in the literature on model validation & verification)
    4.1  Rigorously apply units to all stocks, flows & auxiliaries.  With
the introduction of Powersim Studio for student projects, with its utterly
frustrating (initially) pedantic approach to units, the quality of student
projects jumped to a much higher level.  This particularly imposes a
salutary reality check on willy-nilly use of arbitrary 'unitless' modifying
auxiliaries. 
    4.2  Always run a 'mass balance' sub-model in parallel with the model
building which is continually verifying that 'what goes in equals what comes
out'.
    4.3  Start the model at, eg, time minus 10, and have 10 time periods of
historical data, so that you can check actual against projections.  This, I
have found, gives the client a profound sense of comfort with the next 20+
time periods of projection (whether it is warranted or not).  Refer, eg,
Oliva, R. 1995. A VensimR Module to Calculate Summary Statistics for
Historical Fit. System Dynamics Group, MIT. Memo D-4584. Cambridge, MA.
(For Powersim users I can send an example where this approach is used.


The following are just some of the texts & papers which have an excellent
focus on both model conceptualisation & the modelling process.

Andersen, D.F. and Richardson, G.P. 1997, 'Scripts for group model
building.' In System Dynamics Review, Vol. 13, No. 2, Summer 1997. 
Checkland, P.B. and Scholes, J. 1999. 'Soft Systems Methodology In Action.'
John Wiley and Sons, Chichester, UK.
Coyle, R.G. 1996. 'System Dynamics Modelling: A Practical Approach.' Chapman
and Hall, London.
Richardson & Pugh's 1981 book "Introduction to System Dynamics" 
Vennix, J.A.M, Andersen, D.F. and Richardson, G.P. 1997, 'Group model
building, art and science.' In: System Dynamics Review, Vol. 12, No. 2, (
Summer) 1997: 103-106.
Vennix, J.A.M. 1996. 'Group Model Building: Facilitating Team Learning Using
System Dynamics', John Wiley and Sons, Chichester, UK.
Wolstenholme, E. 1990.  'System Enquiry - A System Dynamics Approach.'
Wiley, Chichester.



Keith Linard
Ankie Consulting Pty Ltd
134 Gisborne Road
Bacchus Marsh
VIC 3340
Posted by  "Keith Linard" <klin4960 at bigpond.net.au>
posting date  Thu, 15 Nov 2007 09:54:59 +1100