When conducting research to determine whether positive outcomes are caused by an intervention, randomized experiments are preferred. Say we want to measure the impact of a specific social and emotional learning (SEL) program on student test scores. For simplicity, let’s say we have two groups, a control group who is going to receive no treatment. Our experimental group is going to participate in a new SEL program designed to increase reading achievement. A reading pre-test will be given at the beginning of our study and a reading post-test after the SEL program is over.
Our goal is to randomly assign subjects (or units) to the treatment and control group. But if our program is going to be run during the day in a school we have a problem. School officials do not randomly assign students to classrooms. Classroom assignments are based on several factors. Suppose we have two second grade classrooms in an elementary school. Academic achievement, age, behavior and gender may all be balanced out so the classrooms are similar on those variables. A set of twins may have to be divided, as well as children who do not get along, and those that are best friends. Sometimes parents make requests and demands for specific teachers that are honored.
Therefore, when researchers aim to conduct a randomized experiment using this existing classroom structure, their ability to randomize is compromised. Further, when treatment and control students are in the same school, it is more difficult to minimize what researchers call ‘cross-overs’—that is, when control students actually receive treatment.
However, randomization is still achievable, the study will just have to get larger. You could randomly select elementary schools in a specific school district, but then you need to factor in how certain neighborhoods are assigned to those schools which could trigger issues of socioeconomic status. A step larger would be districts in county or counties in a state. Of course, as you increase your sample size you require more budget, permissions, staff, and time. Obtaining any or all of these can slow down your project. Most studies do start off with a few schools or classrooms, hopefully with at least two teachers/classrooms in each treatment group. If positive results are found, the study is replicated with many more diverse students so that we can build up a large and diverse evidence base for the program.
Educational research is by and large a game of controlling what you can and accounting for what you can not. In certain circumstances, the researcher must annotate all results when there are factors outside of their control and they are unable to statistically control for them during analysis. This would include variables that could both positively and negatively impact test results such as IQ, motivation, and the influence of parents and peers.