The only way to really know if an educational intervention is working is to test the effectiveness of the program. In an educational setting, this can be difficult. As we have discussed earlier, there are issues with educational research especially when conducting Large-Scale Randomized Controlled Trials (RCTs). Another important factor is the selection of an internal or external evaluator. One key factor often overlooked, that can complicate any research, is the selection of either an independent/external or an internal evaluator.
An internal evaluator is an individual who is already working in some capacity within the organization as the program developer (i.e., the developer and evaluator are from the same institution). This can include a teacher, guidance counselor, or other staff member.
By contrast, an independent evaluator is the person or organization responsible for carrying out the evaluation independent of the program developer. Independence of evaluation means that the evaluator independently conducts key aspects of the evaluation, including random assignment (for RCTs), the collection of any data used in the impact analysis (e.g., outcome measures), the conduct of the impact analyses, and the reporting of study findings. This individual or organization does not have a current affiliation with the same organization as the program developer.
It may be helpful to use an analogy. In medicine, for example, would you take a drug that a pharmaceutical company develops when that same company tests the effectiveness of the drug? They may have a conflict of interest if they stand to profit as a result of positive findings.
There are advantages and disadvantages to each type of evaluator, but for the evaluation of educational interventions, an external or independent evaluator is preferred.
Internal evaluators already know the inner workings of the organization, school culture and political structure. While this familiarity may be convenient, it comes with a hefty price tag. Often internal evaluators are invested in the program and want to show that something is working to help students. The influence of the project developer/implementer on the evaluation findings are reasons for concern. Given their existing relationship with stakeholders, they may feel pressure (or be pressured) to produce results that are in the favor of those in charge. They may also have direct conflict with objectivity during the evaluation given their positions and roles in the school.
External evaluators are the preferred choice as they can provide the necessary objectivity in collection, analysis and dissemination of information. Additionally, experienced external evaluators have technical expertise in the particular type of evaluation design and methods.
Independent external evaluation is critical to high quality research. For further information, see some examples of my independent evaluation projects.