Why simulation?

[Corey Lofdahl]

Earlier this week I was asked a question that I thought I should have been able to answer easily but was in fact hard. The question is this: What empirical evidence is there that Modeling and Simulation (M&S) generally and System Dynamics (SD) specifically helps decision makers in dynamically complex environments? I quickly came up with three examples, but each one is flawed in its own way.

First, there is Beer Game, but the lesson here is that people respond similarly and non-optimally within a dynamic system rather than their performance is improved by the application of M&S.

Second, there are the examples in the second chapter of Sterman's Business Dynamics -- GM auto leasing, Ingalls shipbuilding and the rework model, DuPont and BP maintence -- but these are anecdotes rather than studies.

Third, there's John's stock/flow customers in a department store example with which he opens a lot of talks to get people familiar with everyday complexity. After getting people confused he tells them not to worry, MIT students don't do too well either.

Are there other or better examples to hook skeptics? Specifically, are there studies that show M&S/SD help people make better decisions?

Corey

[Jim Duggan]

Hi Corey,

Interesting question, and as you said, a challenging one to answer. I think there are a number of perspectives you could take:

(1) The general, which explores the need for modelling in complex systems, where cause and effect are separated in time & space, and people are not aware of feedback loops. There are many excellent SD texts that make the case for modelling, and a recent related paper by Joshue Epstein adds to this as he discussed reasons why you might model (see http://jasss.soc.surrey.ac.uk/11/4/12.html)

(2) Specific examples where modelling has led to impact. One recent example is the Forrester Award winner from 2008, Kimberley Thompson:

Using system dynamics to develop policies that matter: global management of poliomyelitis and beyond

By Kimberly M. Thompson, Radboud J. Duintjer Tebbens

Abstract
We offer an example of modeling that influenced global health policy related to polio and we provide some insights about the modeling process. Although system dynamics lies at the heart of our modeling, we emphasize that downplaying the modeling to focus on the policy questions played a critical role in the use of the results by decision makers. Here we provide details of the system dynamics modeling that underlies our research on polio, and discuss the process we followed to bring scientific modeling of a highly complex issue to policymakers. We hope that providing a better understanding of our efforts and describing our continuing analytical journey provides useful insights for others who seek to use system dynamics to develop policies that matter. Copyright © 2009 John Wiley & Sons, Ltd

best regards,
Jim.

[Etiënne Rouwette]

Hello Corey,

The question you pose (what empirical evidence is there that Modeling and Simulation (M&S) generally and System Dynamics (SD) specifically helps decision makers in dynamically complex environments?) is a very interesting one and one that seems to me fundamental to our field. The second type of evidence you mention - the examples in the second chapter of Sterman's Business Dynamics such as GM auto leasing, Ingalls shipbuilding and the rework model, DuPont and BP maintence - you think of as anecdotes rather than studies.

I agree to the last point and would add that if the examples concern constructing a model to answer a client's problem, from a research perspective these could be called 'field studies' or 'case studies'. This type of research scores high on realism, less so on precision and generality (cf. McGrath, 1982): they show that SD can lead to changes in real issues, but it is not exactly clear why or how (precision) or whether the same would work elsewhere (generality). A couple of papers in the past have addressed these last two questions. David Andersen, George Richardson and Jac Vennix wrote in 1997 that the effects of SD interventions in real life problems can be due to a number of reasons, for instance: these interventions bring the managers and doers together in one room, or modeling generates chunks of insight (while details are forgotten), or this is effective because David/ George/ Jac/ John are doing it (they could have used another approach), or even it's just a matter of giving the participants extra attention (the Hawthorne hypothesis). To test which of these hypotheses explains (most of) the effect of SD interventions, we would need a research program that ideally is realistic, precise and general.

I do think since 1997 we have made some progress in testing these and other hypotheses. For instance, Stephen Huz and Krys Stave have in their studies used field experiments and shown that SD modeling scores better than nonsupported groups or groups facilitated in a traditional way (not using models). In my own research, I have looked at theories from social psychology to try to explain effects. In brief, these theories propose that on matters that are important and for which people are able to process available information, new and relevant information will lead to changes in their mental models. These changed mental models will in turn lead to changes in intentions and actions. Other theories and studies show that in complex problems, individuals and groups do a poor job of gathering and processing relevant information. In this line of argumentation, modeling is beneficial because it helps people to gather and process information. In seven case studies of SD interventions, support for some of these relations was found. So this seems one likely route via which SD modelling may help people deal with dynamically complex problems: if the problem is important, SD enables people to process information, which will lead to changes in mental models and actions. We are now trying to look at the role of information sharing and processing in controlled studies and hopefully pinpoint more exactly if and how modelling contributes.

I might have reduced your question 'what evidence do we have that SD/ modeling in general help decision makers in dynamically complex environments' to 'how does SD help in interventions in managerial problems'. Nevertheless, the broader and the more specific question seem to me quite important to our field. I would be really interested to hear other people's reactions and answers to these questions.

Thanks,

Etiënne Rouwette

[Jean-Jacques Lauble]

Hi Etienne
The question was ‘what empirical evidence is there that Modeling and Simulation (M&S) generally and System Dynamics (SD) specifically helps decision makers in dynamically complex environments?’
The question above can only be answered by asking decision makers what they think about it.
I have read plenty of books and papers about SD during years and none have ever answered that question to me.
The only way to answer the question is self experimentation.
Regards.
A decision maker. Jean-Jacques Laublé.

[Thomas Fiddaman]

I understand the spirit in which one would ask decision makers about the value of simulation, but that might be a weak test of the actual value of simulation. I know people who speak enthusiastically about their use of channeled communications from ancient gods and dead pets (really), but that doesn't mean those methods really work.

Another way to think about this might be to shift the burden of proof, and ask, what makes you think you can predict the behavior of a high-order system in your head? Is there empirical evidence that successful decisions in dynamically complex environments, unsupported by simulation, are anything other than lucky?

In practice, the choice is not binary. Confronted with a problem, people can attack it with anything from seat-of-the-pants intuition to extensive simulation and data analysis, with many gradations in between. For any particular problem, it's not necessarily easy to identify the right amount of formalization, or the right mix of methods, but it's unlikely that less thinking is better than more.

I think the nugget of wisdom in the question arises from skepticism about past modeling and forecasting failures. Those arise most often from things like too-narrow boundaries, failure to define relevant concepts, and open-loop thinking, rather than the choice of analytical method (though the latter may influence the likelihood of the former).

Perhaps what the questioner should be asking, then, is whether a careful qualitative approach with broad boundaries might sometimes beat a process involving simulation. I think there are probably some cases where the answer is yes, but mostly one should seek balance - the right amount of dynamic modeling can greatly improve most decision processes.

[James Thompson]

Simulation can satisfy different needs in the learning process. Imagine a continuum of client-learners that runs from "realist" to "idealist". The realist needs to see lots of measured data as a condition of accepting model output. The idealist needs to understand the structure of an argument (system) as a condition of accepting model output. Some simulation model forms are very good at producing tight fits of simulated output to measured time-series data and thus satisfy the realists. Some simulation forms are very good at illucidating an argument and thus satisfy the idealists. My research doesn't suggest the distribution of the general population (or even potential client population) but, if past simulation consumption patterns are an indication, there may be many more realists than idealists. In group dynamics, the demands of realists for historical evidence may be acceptable to idealists. In those cases, it may be well advised to employ more than one model form, say, system dynamics for policy development and some other simulation method for data modelling.

[Thomas Fiddaman]

Interesting comment from Jim. Shooting from the hip a bit here, so take this with a grain of salt:

One possible reason for greater prevalence of realists is that it's easier to assess fit to data than it is to assess the logical consistency of a formalized argument, and that prevalent training "out there" (e.g., in intro economics courses) emphasizes the realist perspective more than the idealist.

Those who were at one of the Ventana gatherings a few years back, where Martha Miller ran the Meyers Briggs personality inventory, may recall that modelers are generally in a corner of the personality space that represents a tiny slice of the aggregate population. All those "other" folks may hardly respond to simulation at all, hence the need for mixed approaches. My guess is that those others are more responsive to the kinds of qualitative stories that emerge from the idealist perspective than to the argument from authority ("the model is right") that emerges from the realist perspective.

There may be many data-oriented methods that are appropriate for various tasks, e.g. visualization and datamining techniques for exploring large datasets or networks. However, SD, with the right statistics, is the best simulation method for data modeling of dynamic problems.

[Eric Stiens]

I find this conversation interesting when combined with the talk about a moribund forum, the "aimless plateau", the lack of successful SDS marketing, etc.

It is one thing to explain to people why their current way of thinking about things is inadequate, another entirely to suggest that this particular other way of thinking is better, and that it is possible. The most common response I have seen to presenting people with SD models are skepticism that they will work "over here" as well as "over there" (particularly in public policy as well as management/business issues) or a skepticism that they or their organization will ever be able to invest the time/energy into learning modeling.

The lack of quantitative studies backing up modeling as an effective decision making aid may be problematic in some cases, but the lack of easily digestible 1-2 page success stories "Here's how this group of decision makers was thinking about this problem, here's the insights that they gained while modeling it, here's the actions they took as a result, here's what resulted from those actions" across many disciplines and fields is even more problematic. It seems like generating the latter would be easier than generating the former, even if they were anecdotal.

[Jean-Jacques Lauble]

Hi Eric

To my opinion, the virtue of simulation is too much depending on the context to be able to have any general opinion about it. It is then a highly individual question. For me for instance, I think that to make simulation useful, the solution is to set easy to reach objectives at least relative to the capacity of the modeler and the client and avoid transforming a model into a chimera like unfortunately most publicized models are or look like especially for common people (who have a good common sense).
See the Colbert’s report.
Regards.
Jean-Jacques Laublé

[Richard Dudley]

Not sure I understand this question.

Cory, are you saying that there are only a handful of real applications of SD that have been useful in a practical sense? That there are few examples available for inspection? Or that SD applications have not been shown to be clearly better than possible alternatives?

I read some articles in SDR and elsewhere about what I assume are real world applications. I have assumed that there are many more, but that unlike academics (who have to publish) businesses tend not to publish, especially when the subject concerns an intractable problem they have been having. There should be 'closet' examples floating around. In addition to these and published articles there are those examples 'exposed' by the System Dynamics Applications Award. The presentation about the Fluor Corp., for example mentioned that the SD approach developed has been used at over 100 large projects.

Am I missing something in the question? Or is it just that 'we' feel there should be more examples of good SD application.

Does someone have a list of successful interventions? What constitutes success? Better understanding? Better decisions? Better bottom line?