Your question on whether there is evidence 'out there that decisions and change has happened in a significantly better way using SD that it would have if SD had not been used' has been of interest to me for quite some time. I think there is some evidence. In 2002 Jac Vennix, Theo van Mullekom and I published a paper in System dynamics review that looked at 107 published cases on group model building. Group model building (GMB) is a generic term for building SD models with experts and/ or stakeholders. There were few studies comparing GMB to other interventions, and naturally those that do use a comparison group tend to build models with students. In those modeling projects there is no actual implementation and some authors (e.g. Colin Eden) argue that the decision making process in a student group is uncomparable to that of 'real' clients. So ideally you would like to have both a control group *as well* as real clients. The only examples that come to my mind are the studies by Steve Huz and Krys Stave (see references below).
In short, the results of the 2002 study for decisions and change are:
- of 107 cases, 35 report on commitment to a decision taken, in 31 of these the impact is positive;
- of 107 cases, 46 report on system changes. Of these, 42 out of 46 report implemented changes at the system level.
The paper also goes into the question to what extent the 107 cases are representative of 'all' GMB projects with clients. The overwhelming majority of cases reports positive results, which raises some suspicion. On the other hand, many studies are co-authored by clients.
You raise the point thar the client's stated needs do not necessarily reflect the real needs. Interestingly, a similar process seems to be operating in the client's perception of learning. In several projects, a client would state not to have learned (much) of participating in modeling sessions. Comparison of answers to a questionnaire before and after the sessions would however indicate clear changes in opinions on the problem modeled. When this was pointed out to them, most would say something like "now that you mention it, I did not know X which was mentioned in session Y. And yes, that might have changed my idea on what to do about this problem". So the conclusion was that people have little insight into their own learning - which turns out to be a finding that was widely known in cognitive psychology. See the reference below: Telling more than we can know: verbal reports on mental processes.
You also mention the case repository (http://cases.systemdynamics.org/
). The idea in launching this was to add cases on the basis of publications in System dynamics review and conference papers, and to invite all discussion list participants to contribute their own cases. So far we have 36 cases but that there are many more examples not yet written up. So I would encourage anyone who has an idea on a case to send it in - see the website on how and where to do so.
Huz S, Andersen DF, Richardson GP, Boothroyd R. 1997a. A framework for evaluating systems thinking interventions: an experimental approach to mental health system change. System Dynamics Review 13(2): 149–169.
Nisbett, R., & Wilson, T. (1977). Telling more than we can know: verbal reports on mental processes. Psychological Review, 84(3), 231-259.
Rouwette E.A.J.A, Vennix J.A.M., Mullekom T. van. (2002). Group model building effectiveness: a review of assessment studies. System Dynamics Review 18(1): 5-45.
Dwyer, M., & Stave, K. (2008). Group model building wins: the results of a comparative analysis. Paper presented at the 2008 System Dynamics Conference.