President's Message - Fall 2011

Evidence-based Practices and Practice-based Evidence: A Union of Insufficiencies

Bryan Cook
University of Hawaii

Advocates of evidence-based practice (i.e., evidence-based special education) suggest that instructional practice should be based on research evidence, which certainly seems like a worthy goal in special education - so long as the evidence is trustworthy. As I wrote in the previous issue of Focus on Research, evidence-based practices (EBPs) are important because they are supported by multiple, rigorous, experimental studies - thereby yielding the most trustworthy evidence available regarding whether a practice works. It would be wonderful if we could package EBPs such that they could be applied and maintained successfully over time in any variety of environments, with any variety of teachers, and for any variety of students. But the real world is far too complicated and nuanced for such simple solutions. Most EBPs are supported by only a few internally valid experimental studies, which were typically conducted with artificially high levels of support (e.g., grant funding, specially trained interventionists). Additional evidence is needed to know whether EBPs will work in real world conditions, with whom they work, and how they work.

Practice-based evidence (PBE) represents an alternative perspective for examining and identifying effective practices that is gaining popularity in fields such as medicine (Green, 2006), psychiatry (Walker & Bruns, 2006), and psychology (Barkham & Mellor-Clark, 2003). Rather than emphasizing the derivation of practice from evidence, PBE focuses on mining evidence from practice. The hallmark of PBE is that it involves and reflects the realities of typical practitioners. PBE can take many forms, from anecdotes based on personal experiences (e.g., discussions and advice in teacher lounges), to informal assessments of consumer outcomes (e.g., student performance on teacher-made tests), to more formal research that does not involve randomized assignments to groups (e.g., case studies, single-case research, quasi-experiments). It addresses issues such as where and under what conditions a practice works, with whom the practice works, how a practice can be adapted and maintained successfully, and how practitioners feel about a practice.

EBPs and PBE both have their strengths and shortcomings. EBPs are highly trustworthy due to the rigor of their supporting research, but such research efficacy often fails to translate into practical effectiveness. In contrast, although PBE is grounded in the real world of teachers and therefore highly relevant, it is less trustworthy than EBPs due to the lack of controls for common threats to validity. Unfortunately, these two perspectives have been conceptualized as antithetical and competing; greater rigor suggests less relevance, and greater relevance suggests less rigor (Barkham & Margison, 2007). Researchers tend to focus on rigor and EBPs, oftentimes devaluing and denigrating the importance of PBE and practice utility; and practitioners tend to champion the relevance of PBE while disregarding the significance of rigor and EBPs.

As suggested by Stokes (1997), however, EBPs/rigor and PBE/relevance are not mutually exclusive considerations. Rather than accept the traditional paradigm in which research evidence is either rigorous or relevant, we should strive for practices that are supported by evidence that is both rigorous and relevant. That is, we need to know that a practice works, as demonstrated by rigorous experimental research; but we also need to know that it can be adopted and maintained in common settings by typical teachers, for whom it will work, and how it can be adapted successfully. EBPs and PBE are both insufficient in isolation. But they are perfect complements to each other, forming a powerful union of insufficiencies in which the shortcomings of one approach are compensated by the strengths of the other (Shulman, 2004). 

Truly effective practices should be both (a) shown by rigorous experimental research to work and (b) understood and supported by practitioners as “doable” in the real world. The Society for Prevention Research’s Standards Committee addressed this issue by proposing separate sets of standards for efficacious and effective practices; whereas efficacious practices must meet typical EBP standards, effective practices must meet standards for both EBPs and PBE (Flay et al., 2005). Practices can be established as effective, then, by being supported by two separate bodies of literature (i.e., rigorous experimental studies, non-experimental research focused on relevance). Ideally, however, researchers and practitioners should collaboratively develop a new approach to conducting research studies that are both rigorous and relevant (e.g., explore questions of importance to practitioners, involve practitioners with typical supports as interventionists, use an experimental design), what Stokes (1997) called use-inspired research.

As one educator opined, it is often true that “research is not the real world” (Nelson, Leffler, & Hansen, 2009, p. 28). Indeed, many special educators do not trust research or researchers (Boardman et al., 2005), in large part because research seldom speaks to their concerns regarding whether and how a practice will work in their classes, with their students (Simons, Kushner, Jones, & James, 2003). Convincing practitioners that a practice works will not be accomplished by having them read more research reports or presenting more effect sizes. Rather, to sell effective practices, special education researchers and policymakers need to speak the language of practice (i.e., PBE). I’m not saying that experimental research should not play a role in determining what works; it must. But when disseminating information about instructional practices to practitioners, such evidence should take a backseat to the kind of PBE that practitioners value—for example, stories from other teachers who teach students similar to their own (Smith, Richards-Tutor, & Cook, 2010).

Traditional, rigorous, experimental research is important. But as Hambrick (1994) lamented, researchers typically operate in a “closed incestuous loop” (as cited in Aguinis et al., 2010, p. 516) disconnected from the world of practice. Bridging the longstanding research-to-practice gap in special education will require that researchers think in new ways and embrace new forms of research and dissemination, such as PBE. The Division for Research’s invited Showcase Session for next year’s CEC conference, in which leading researchers will co-present with practitioners about effective practices, highlights this new kind of thinking. I’m looking forward to other new approaches for merging EBPs and PBE to better identify and promote effective instructional practices for students with disabilities, which is, after all, the common goal of both special education researchers and practitioners.

Aguinis, H., Werner, S., Abbott, J. L., Angert, C., Park, J. H., Kohlhausen, H. (2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13, 515-539.

Barkham, M., & Margison, F. (2007). Practice-based evidence as a complement to evidence-based practice: From dichotomy to chiasmus. In C. Freeman & M. Power (Eds.), Handbook of evidence-based psychotherapies: A guide for research and practice (pp. 443-476). West Sussex, England: John Wiley & Sons.

Barkham, M., & Mellor-Clark, J. (2003). Bridging evidence-based practice and practice-based evidence: Developing a rigorous and relevant knowledge for the psychological therapies. Clinical Psychology and Psychotherapy, 10, 319–327.

Boardman, A. G., Arguelles, M. E., Vaughn, S., Hughes, M. T., & Klingner, J. (2005). Special education teachers’ views of research-based practices. Journal of Special Education, 39, 168-180.

Flay, B., Biglan, A., Boruch, R., Castro, D., Gottfredson, D., Kellam, S., Moscicki, E, … Ji, P. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151-175.

Green, L. W. (2006). Public health asks of systems science: To advance our evidence-based practice, can you help us get more practice-based evidence. American Journal of Public Health, 96, 406-409.

Nelson, S. R., Leffler, J. C., & Hansen, B. A. (2009). Toward a research agenda for understanding and improving the use of research evidence. Portland, OR: Northwest Regional Educational Laboratory. Retrieved from http://educationnorthwest.org/webfm_send/311

Shulman, L. S. (2004). A union of insufficiencies: Strategies for teacher assessment in a period of educational reform. In S. M. Wilson (Ed.), The wisdom of practice: Essays on teaching, learning and learning to teach (pp. 352-361). San Francisco: Jossey-Bass.

Simons, H., Kushner, S., Jones, K., & James, D. (2003). From evidence-based practice to practice-based evidence: The idea of situated generalization. Research Papers in Education, 18, 347-364.

Smith, G. J., Richards-Tutor, C., & Cook, B. G. (2010). Using teacher narratives in the dissemination of research-based practices. Intervention in School and Clinic, 46, 67-70.

Stokes, D. E. (1997). Pasteur’s quadrant: Basic science and technological innovation. Washington DC: Brookings Institution Press.

Walker, J. S., & Bruns, E. J. (2006). Building on practice-based evidence: Using expert perspectives to define the wraparound process. Psychiatric Services, 57, 1579-1585.