REPLY Ensuring Quality of Models (SD6660)

[SDMAIL Alan Graham]

Posted by  "Alan Graham" <Alan.Graham at paconsulting.com>

Keith Linard is so very right:

"The fundamental step in ensuring the quality of models is ensuring that we
are modelling the correct problem."  

These days I'm telling people that there are three validations of a
model's fitness for purpose, all of them necessary, and they require
quite different validation processes (i.e. tests to disprove):

1.  Validating the purpose

2.  Validating the model

3.  Validating the results

In practice, validating a purpose is done more effectively if the
modelers go quite a bit beyond the relationship hygiene that Keith
Linard discussed--a specific enumeration of the purpose, validated by
thorough discussion with the stakeholders (e.g. clients) is immensely
valuable later in the modeling process.

A one-sentence purpose of the form "to model X" or "to understand X" is
usually a good sign that the modeling effort will end up being
ineffectual--it would take some extremely focused and effective thinking
and persuasion elsewhere to make up for this vagueness.  Such abstract
purposes imply no criteria for demonstrating that a model has filled its
purpose;  there's nothing specific enough to be able to tell whether a
modeling effort will be useful for anyone.  And that's a big problem
waiting to happen.  (And not just for commercial modelers--when can one
know when a thesis is satisfactorily done?)

A tangible and agreed-on model purpose will typically have at least a
couple of these elements:

1.  Actions or policies "on the table"--what real actions or policies
(in the SD sense) is the model supposed to be influencing?  (Some
high-level issues will be "above the pay grade" of the client, while
others will be too detailed to justify SD modeling.)  This ultimately
translates fairly directly into model policy levers.  Even if the client
is really at the stage of bewilderment and can't articulate actions,
it's very helpful to state something like "to evaluate actions feasible
for the X organization, such as A, B, C" to start focusing the client's
thinking.

2.  What defines a better outcome?  Often, a client's organization will
have specified metrics (e.g. IRR or 5-year cumulative profit).  It's a
good idea to be computing an NPV (or a social present value metric of
some kind) in addition, if for nothing else, to sharpen the analysis of
tradeoffs.

3.  What contingencies might change the analysis' results?  E.g., is a
given policy still an improvement if the economy enters a recession?  Or
if people are less responsive than the market research indicates?  These
should translate directly into exogenous inputs or parameter changes for
policy sensitivity tests.

4.  What are client hopes, fears, puzzles and disagreements?  Often,
clients will have very specific concerns, such that the modeling effort
can't succeed in creating an improvement in the real system until the
clients get past their concerns.  Sometimes these are simply
contingencies or uncertainties about actions, which fit into categories
1 or 3.  But other times they've been, e.g. sharp disagreement over
whether a standard industry metric was really appropriate (given that a
favored alternative policy decreased the standard measure while
increasing NPV).  Or disagreement or uncertainty about the strength of
two opposing impacts of a given policy.  This is perhaps close to a
"relationship hygiene factor", but client concerns are often quite
specific, and are much more useful to the analysis if addressing those
concerns is explicitly part of the model purpose.  The (SD) classic
dynamic hypothesis may also codify a client concern (feared behavior and
a cause for it), and the description usually contains metrics of
goodness as well.

The model purpose is something to start codifying and validating with
the client/stakeholder before even causal diagramming.  And "codifying"
means a written, presented, discussed and distributed document.
(Usually Powerpoint, but client cultures differ.)

The model purpose also needs revisiting at important milestones
thereafter:  Often, modeling insights will shift the perception of the
problem, and unless the clients likewise share the changed perception,
there will be problems down the road.

And of course validation of model and results also needs to be done (to
the satisfaction of both the modeler and the stakeholders) and also
making sure that whatever organizational convincing and change
management is needed will in fact get done.  But those are other
stories.

Cheers,

Alan
Posted by  "Alan Graham" <Alan.Graham at paconsulting.com>
posting date  Thu, 15 Nov 2007 09:48:01 -0500