Using Theory to Improve Evaluation Practice:

Resources to Enhance Your Knowledge and Skills

 Stewart I. Donaldson

Claremont Graduate University

Updated July 2009

   The discipline and profession of evaluation is growing by leaps and bounds.  Societies all around the globe seem to be embracing the values of accountability, professionalism, and evidence-based practice, and are commissioning evaluations of all shapes and sizes.  While there is little disagreement about the need for, and value of, evaluation, there remain trenchant disagreements in the field about how best to practice evaluation today (see Donaldson, Christie, & Mark, 2008; Donaldson & Christie, 2005; Donaldson & Scriven, 2003;).  Roles for theory in evaluation practice remain one of the most confusing and controversial topics in the profession.  Reference to theory is widespread in the contemporary evaluation literature, but what is meant by theory encompasses a confusing mix of concepts related to evaluators’ notions about how evaluation should be practiced, explanatory frameworks for social phenomena drawn from social science, and assumptions about how programs function, or are supposed to function. A newcomer to evaluation, and even a grizzled veteran, could have a difficult time sorting  through the closely related and sometimes interchangeable terms that litter the evaluation landscape—theories of practice, theory-based evaluation, theory-driven evaluation, program theory, evaluation theory, theory of change, logic models, and the like (Donaldson, 2007; Donaldson & Lipsey, 2006).  In addition to the confusion, distinguished evaluators as Scriven (1998, 2004a, 2004b) and Stufflebeam (2001, 2004) continue to assert that there is little need for theory in evaluation. For example, Scriven (2004a) claimed, “It's possible to do very good program evaluation without getting into evaluation theory or program theory.”  Scriven (2004b) declared that “the most popular misconception amongst currently politically correct program evaluators is the evaluation of a program (a) requires that you have, or (b) is much benefited by having, a logic model or program theory.  Similarly, Stufflebeam (2001), in a review of evaluation models and theories of evaluation practice, remarked that “there really is not much to recommend theory-based evaluation, since doing it right is usually not feasible and since failed or misrepresented attempts can be counterproductive.”  More recently, Stufflebeam (2004) claimed Braybrooke & Lindblom (1963) discredited the notion of theory-driven evaluation 40 yeas ago, and their message clearly needs to be revisited by modern day evaluators (p. 253).

     However, the overwhelming major of evaluators today seem to argue that theory does, and should, play important roles in modern program evaluation (e.g., Alkin, 2004; Chen, 2004; Christie, 2003; Donaldson, 2003, 2007; Donaldson & Lipsey, 2006; Fetterman, 2003; Mark, 2003; Rossi, Lipsey, & Freeman, 2004; Shadish, Cook, & Campbell, 2004; Weiss, 2004a, 2004b). 

     The purpose of this brief note is to share with you some of latest resources available for helping you to use theory to improve your evaluation practice.  In my opinion, the readings, workshops, and courses listed below in the Appendix are effective ways to enhance your knowledge and understanding about how to use theory in contemporary evaluation practice. 

Happy evaluating! 

Stewart

References

Alkin, M. C. (Ed.) (2004).  Evaluation roots. Thousand Oaks: Sage.

Baybrooke, D., & Lindblom, C. E. (1963). A strategy decision. New York: Free Press.

Chen, H. T. (2004).  Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Newbury Park, CA: Sage.

Christie, C. A. (2003).  The practice-theory relationship in evaluation. New Directions for Program Evaluation, 97.  San Francisco, CA: Jossey-Bass.

Donaldson, S. I. (2003).  Theory-driven program evaluation in the new millennium. In S. I. Donaldson & M. Scriven (Eds.) Evaluating social programs and problems: Visions for the new millennium (pp. 111-142).  Mahwah, NJ: Erlbaum.

Donaldson, S.I. (2005). Using program theory-driven evaluation science to crack the Da Vinci Code. New Directions for Evaluation, 106, 65-84.

Donaldson, S.I. (2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum.

Donaldson, S. I,  & Christie, C. A. (2005).  The 2004 Claremont Debate: Lipsey versus Scriven.  Determining causality in program evaluation and applied research: Should experimental evidence be the gold standard? Journal of Multidisciplinary Evaluation.

Donaldson, S.I., Christie, C. A., & Mark, M.M. (2008). What counts as credible evidence in applied research and evaluation practice? Newbury Park, CA: Sage.

 Donaldson, S. I., & Gooler, L. E. (2003). Theory-driven evaluation in action: Lessons from a $20 million statewide work and health initiative. Evaluation and Program Planning, 26, 355-366.

Donaldson, S. I., Gooler, L. E., & Scriven, M. (2002). Strategies for managing evaluation anxiety: Toward a psychology of program evaluation. American Journal of Evaluation, 23(3) 261-273.

Donaldson, S.I., & Lipsey, M.W. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In I. Shaw, J.C. Greene, & M.M. Mark (Eds.), The Handbook of Evaluation: Policies, Programs, and Practices (pp. 56-75). London: Sage.

Donaldson, S. I., & Scriven, M. (2003b). Diverse visions for evaluation in the new millennium: Should we integrate or embrace diversity? In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium (pp. 3-16). Mahwah, NJ: Erlbaum.

Fetterman, D. (2003).  Empowerment evaluation strikes a responsive cord.  In S. I. Donaldson & M. Scriven (Eds.) Evaluating social programs and problems: Visions for the new millennium (pp. 63-76).  Mahwah, NJ: Erlbaum.

Gargani, J. (2003). The history of theory-based evaluation: 1909 to 2003.  Paper presented at the American Evaluation Association annual conference, Reno, NV.

Mark, M. M. (2003).  Toward a integrative view of the theory and practice of program and policy evaluation.  In S. I. Donaldson & M. Scriven (Eds.) Evaluating social programs and problems: Visions for the new millennium (pp. 183-204).  Mahwah, NJ: Erlbaum.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004).  Evaluation: A systematic approach (7^th Ed.).  Thousand Oaks, CA: Sage

Scriven, M. (1998).  Minimalist theory: The least practice requires.  American Journal of Evaluation, 19, 57-70.

Scriven, M. (2003).  Evaluation in the new millennium:  The transdisciplinary vision.  In S. I. Donaldson & M. Scriven (Eds.) Evaluating social programs and problems: Visions for the new millennium (pp. 19-42).  Mahwah, NJ: Erlbaum.

Scriven, M. (2004a).  Practical program evaluation: A checklist approach.  Claremont Graduate University Annual Professional Development Workshop Series.

Scriven, M. (2004b). April 26, 2004 EvalTalk posting.

Shadish, W. R. (1998). Evaluation theory is who we are.  American Journal of  Evaluation, 19(1), 1-19.

Shadish, W. R., Cook, T. D.,  & Campbell, D. T. (2001).  Experimental and quasi-experimental designs for generalized causal inference.  Boston: Houghton-Mifflin.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991).  Foundations of program evaluation: theories of practice.  Newbury Park, CA: Sage.

Stufflebeam, D. L. (Ed.) (2001). Evaluation models (New Directions for Evaluation, No.89). San Francisco, CA: Jossey-Bass.

Stufflebeam, D. L. (2004).  The 21st-Century CIPP Model: Origins, Development, and Use.  In M. C. Alkin (Ed.), Evaluation roots (pp. 245-266). Thousand Oaks: Sage.

Weiss, C. H. (2004a).  On theory-based evaluation: Winning friends and influencing people.  The Evaluation Exchange, IX, 4, 1-5.

Weiss, C. H. (2004b).  Rooting for evaluation: A cliff notes version of my work. In M. C. Alkin (Ed.), Evaluation roots (pp. 153-168). Thousand Oaks: Sage.

Appendix - Resources for Learning about Theory

Recommended Readings

Overview

Donaldson, S.I., & Lipsey, M.W. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In I. Shaw, J.C. Greene, & M.M. Mark (Eds.), The Handbook of Evaluation: Policies, Programs, and Practices (pp. 56-75). London: Sage.

Donaldson, S.I. (2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum.

Donaldson, S.I., Christie, C. A., & Mark, M.M. (2008). What counts as credible evidence in applied research and evaluation practice? Newbury Park, CA: Sage.

Evaluation Theory

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991).  Foundations of program evaluation: theories of practice.  Newbury Park, CA: Sage. (Classic)

Alkin, M. C. (Ed.) (2004).  Evaluation roots. Thousand Oaks: Sage.

Donaldson, S. I., & Scriven, M. (Eds.) (2003). Evaluating social programs and problems: Visions for the new millennium. Mahwah, NJ: Erlbaum

Christie, C. A. (2003).  The practice-theory relationship in evaluation. New Directions for Program Evaluation, 97.  San Francisco, CA: Jossey-Bass.

Program Theory and Theory-driven Evaluation

Bledsoe, K., & Graham, J. A. (2005).  The use of multiple evaluation approaches in program evaluation.  American Journal of Evaluation, 26(3), 302-319.

Bledsoe, K. L. (2005). Using theory-driven evaluation with underserved communities: Promoting program development and program sustainability. In S. Hood, R. Hopson, and H. Frierson (Eds.). The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice. Greenwich. CT: Information Age Publishing. V. 3, 175-196.

Chen, H. T. (2004).  Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Newbury Park, CA: Sage.

Donaldson, S.I. (2001). Mediator and moderator analysis in program development. In S. Sussman (Ed.), Handbook of program development for health behavior research and practice (pp. 470-496). Newbury Park, CA: Sage.

Donaldson, S. I. (2003).  Theory-driven program evaluation in the new millennium. In S. I. Donaldson & M. Scriven (Eds.) Evaluating social programs and problems: Visions for the new millennium (pp. 111-142).  Mahwah, NJ: Erlbaum.

Donaldson, S.I. (2005). Using program theory-driven evaluation science to crack the Da Vinci Code. New Directions for Evaluation, 106, 65-84.

Donaldson, S.I. (2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum.

Donaldson, S. I., & Gooler, L. E. (2003). Theory-driven evaluation in action: Lessons from a $20 million statewide work and health initiative. Evaluation and Program Planning, 26, 355-366.

Gargani, J. (2003). The history of theory-based evaluation: 1909 to 2003.  Paper presented at the American Evaluation Association annual conference, Reno, NV.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004).  Evaluation: A systematic approach (7^th Ed.).  Thousand Oaks, CA: Sage. (Chapter on Expressing and Assessing Program Theory)

Weiss, C. H. (2004).  On theory-based evaluation: Winning friends and influencing people. The Evaluation Exchange, IX, 4, 1-5.

Weiss, C. H. (2004).  Rooting for evaluation: A cliff notes version of my work. In M. C. Alkin (Ed.), Evaluation roots (pp. 153-168). Thousand Oaks: Sage.

Professional Development Workshops

The American Evaluation Association(usually in November; http://www.eval.org/Training/training.htm) and Claremont Graduate University (usually in August; http://www.cgu.edu/pages/465.asp) sponsor a week of relatively low-cost professional development workshops that provide some of the latest information about how to use evaluation theory, program theory, and social science theory in evaluation practice.