A guest post by Ferdinand C Mukumbang
Realist evaluation is a theory-driven approach to evaluating programmes (interventions and policies). Its focus on explaining how, why, for whom, and under what circumstances a programme works (or not) makes it attractive to researchers and policymakers. With increasing emphasis on considering philosophical foundations while conceptualising research projects (Nichol et al., 2023), operationalising realist evaluation highlighted concerns around the ontological and epistemological foundations underpinning its methodology and adopted methods.
Realism generally underpins realist evaluation—reality exists and is knowable. With several forms of Realism found in the extant literature, there is confusion among novices regarding the foundation of realist evaluation. Consequently, we will focus on two predominant forms: scientific realism and critical realism. Scientific realism is a form of Realism focusing on the aim of science and the very nature of scientific knowledge. Scientific realism supports the claim that the objects referenced in theories, including unobservable entities, exist objectively. Conversely, critical realism focuses on the social activity of developing theories. It is more about the methods employed to explain outcomes and events in natural settings.
Pawson and Tilley (1997) situated realist evaluation in the scientific realist philosophy of science. Nevertheless, the approaches applied to develop programme theories (how and why programmes are hypothesised to work) in realist evaluation harness principles predominantly espoused by critical realism. Due to their discrete fundamental goals, scientific and critical realism offer different but complementary perspectives to inform realist evaluation. Indeed, they share similar understandings relating to the existence of a mind-independent reality, (in)transitive entities, generative causation, stratified reality, emergence, open systems and retroductive theorising. This post sheds light on these essential philosophical principles as guiding realist evaluation. For a more detailed version of the argument see our recent paper (Mukumbang et al., 2023).
Mind-independence is a principle shared by scientific and critical realism and stipulates that reality exists whether we perceive it or not. For example, many people deny that smoking causes lung cancer, but the objective reality is its effects on the lungs. The mind-independent nature of reality applies not only to the physical dimensions (cancerous lung lesions) but also to the social and cultural aspects (Sturgiss & Clark, 2020). For instance, the mind-independent nature of a religion means that human perceptions of that religion remain that and cannot be considered to represent that religion as practiced. Non-believers can have personal understandings, but these perceptions do not change the state of that religion. Reality also does not depend on the different languages and conceptualisations we use to understand a phenomenon; they do not determine what is real. Programmes are what they are, irrespective of the programme theories we use to describe them.
Although reality is independent of our human thought, language, conceptual activity, and perceptual experience, realists believe it can be captured using theories and models—the semantic notion of scientific realism. For instance, theories can be used to explain why and how COVID-19 vaccination levels remain lowest in low- and middle-income countries. Still, our best and most credible scientific theories represent only an estimation of reality—the epistemic notion of scientific realism. After applying our best research methods, using our best research tools, whatever theories we obtain are considered only an approximation of reality. Our elicited programme theories only approximate how and why the programme works (or not).
One of the reasons why our theories can only approximate reality is because not everything that exists manifests as events, and not all the events that take place are experienced. Consequently, three levels of reality are identified in critical realism—stratified reality. Evaluation activities can predominantly capture those events or outcomes experienced (‘Empirical’ level). Events are situated at the next level, the ‘Actual’, as they represent things that ‘actually’ happened, even though not all are experienced. Entities found at the ‘Real’ cause events to occur. These entities possess causal powers, which can generate an outcome (event) when activated.
These causal entities are usually unseen and are considered intransitive entities—human actions cannot change them. They represent laws and properties of the world independent of our knowledge of (and efforts to understand) them. Nevertheless, through research activities, we can formulate theories and develop models to estimate their existence. These models and theories, like programme theories developed through realist evaluation, are considered transitive entities—amenable to alteration by human action. Consequently, programme theories are always provisional and partial, but the entities that cause the observations described by the programme theories remain unchanged.
Indeed, causal entities can combine in different ways, depending on other conditions and interactions, to form new outcomes—emergence. The emerging outcomes are not the sum of the parts of the interacting entities. Thus, the emerged outcomes cannot be reduced to the entities that combined to form them. Emergence is a property of open systems, such as the social world, with a constellation of structures, mechanisms, and other entities constantly interacting with each other to form outcomes. Programmes are parts of open systems embedded in multiple social systems.
Within open systems are the activities of agents (thoughts and actions taken by people) interacting within structures (organised social institutions and patterns of institutionalised relationships). Mechanisms of programmes are only activated when agents act upon them. Nevertheless, the mechanisms introduced by a programme are not the only ones in operation. Other conditions (context in which the programme is introduced) play a role in (dis)activating the introduced mechanisms—generative causation. Following this understanding, Pawson and Tilley (1997) proposed the context + mechanism = outcome (CMO) as a tool for formulating programme theories. Other authors have proposed a modification, Contextual Mechanisms + Program Mechanisms + Agency = Outcome (CM+PM+A = O). This modification proposes that the evaluation should consider the mechanisms embedded in the social context (CM), the programme mechanisms (PM), and how agents interpret and respond to these mechanisms (A), leading to the outcome (O).
Formulating theories in realist evaluation is achieved through retroductive theorising. Retroduction is a form of retrospective theorising (working backwards). It entails developing explanations using information obtained from the ‘Empirical’ through interpreting particular actions and events situated in the ‘Actual’ to uncover the structures and mechanisms that will account for the outcome found in the ‘Real’. (Mukumbang et al., 2021)
Cited works
Mukumbang, F. C., De Souza, D. E., & Eastwood, J. G. (2023). The contributions of scientific realism and critical realism to realist evaluation. In Journal of Critical Realism. https://doi.org/10.1080/14767430.2023.2217052
Mukumbang, F. C., Kabongo, E. M., & Eastwood, J. G. (2021). Examining the Application of Retroductive Theorizing in Realist-Informed Studies. International Journal of Qualitative Methods, 20, 1–14. https://doi.org/10.1177/16094069211053516
Nichol, A. J., Hastings, C., & Elder-Vass, D. (2023). Putting philosophy to work: developing the conceptual architecture of research projects. Journal of Critical Realism. https://doi.org/10.1080/14767430.2023.2217054
Pawson, R., & Tilley, Nick. (1997). Realistic Evaluation. Sage Publications. https://uk.sagepub.com/en-gb/eur/realistic-evaluation/book205276
Sturgiss, E. A., & Clark, A. M. (2020). Using critical realism in primary care research: An overview of methods. Family Practice, 37(1), 143–145. https://doi.org/10.1093/fampra/cmz084
Bio
Ferdinand C Mukumbang is an Assistant Professor at the Department of Global Health, University of Washington. He is an implementation scientist and specialises in realist-informed methodologies.