Meta-Analysis is a powerful process, which allows a researcher to statistically aggregate results from multiple studies. Using advanced meta-analytic techniques, the process combines the results of these studies creating a larger sample, possible effect size, and allows the researcher to look at a specific subject or school related variable across multiple research studies. Researchers can identify malleable factors that can lead to development of more effective interventions. Another relevant theme may be an attempt to identify not only the most effective interventions, but those that seem to be most readily implemented in actual school settings for particular grades, students, etc.  

As one can imagine, the larger sample size can be advantageous while at the same time come at a considerable cost savings.  Without any data to collect, subjects to monitor, and/or treatment to administer, a significant amount of time and effort can be saved on the part of the research.  While meta-analytic techniques are powerful with tremendous benefits, they do not come without problems.  

  1. No two studies are exactly the same. When studies sample sizes and results are combined the researcher must assume that all included study were conducted with similar processes and standards. For example, certain students may have been excluded from a particular study, and while noted, their demographic is not included in results.  It is critical for high quality meta-analysis to make careful note of differences in studies both in the selection process as well
  2. Not all data is the same.  When you collect a test score, evaluate a questionnaire or interview a student, each individual piece of data is different in some way.  Many times data can be influenced by the data collection, recording, storing and then evaluation. Over the course of a study and then the meta-analysis process, the nuances created are often impossible to account for in analysis.  However, understanding and noting their presence is critical.  
  3. Not all studies are published.  It has been long held that studies with significant results are published and available, but what about the other studies that are not published?   Typically, studies that are published report positive results.  That is, a treatment effect was found indicating an intervention worked, providing a solution (or part of a solution) to a given problem.  However, the unpublished studies can be important, however difficult to find. Critics have long argued the need to explicitly address the “file drawer” problem (Rosenthal, 1979), and the importance of writing about and reporting a lack of expected results. This problem is an important one to note.  
  4. If one or some or all of the studies were poorly conducted. It is often assumed that research published in peer-reviewed journals has been conducted and evaluated according to rigorous standards.  But, as we know, sometimes even the most well-intentioned researchers arrive at incorrect conclusions due to flaws which introduce bias and mistakes created during the research process. For example, use of implementer-made measures, small sample sizes, and failure to account for clustering, introduce bias.

These dangers should not dissuade researchers from conducting a study. Rather the limitations should serve as precautions when conducting a meta-analysis.

How Corcoran Lab selects studies for meta analysis is by using a rigorous process.   After writing a concise thesis and conducting an exhaustive search of the literature, we first look for high quality research.  Related studies pass through an initial phase, screening title and abstracts.  For selected works, we then code for specifics including germaneness and methodological characteristics such as control groups and duration of study.  This process then goes through an extensive review to ensure that the right studies have been selected for inclusion.  

Examples of Meta Analysis Conducted by Corcoran Lab:

An example of the type of meta analysis we conduct is a recent project titled: “Effective school-based social and emotional learning programs for improving academic achievement: A systematic review and meta-analysis of 50 years of research.” The goal was to evaluate school-based social and emotional learning (SEL) interventions for pre-kindergarten through high school students. The studies selected represent a variety of SEL interventions evaluated in rigorous school based research for evaluation.  Additionally across studies we can evaluate what variables moderate the impact of school-based SEL programs.  

Meta-analytic techniques are an excellent way to make sense of rapidly expanding research literature.