Relevance for Complex Systems Knowledge
The publication is not a description of an intervention but a collection of intervention that were evaluated to assess their quality and categorized using a theoretical framework. The framework seems to be the most interesting part of the study and could be used as a reference in the development of the protocol. It is divided into several sets of factors mediating and moderating the process of making educational choices and persisting in STEM education.
- The social context domain, which encompasses the national educational policy, cultural environment, and economy. The social context influences the academic choices and persistence in STEM education by influencing the school context, social environment and/or student characteristics.
- The school context refers to the educational policy on the school level, school climate and teaching pedagogy. The school context is assumed to influence the student characteristics.
- The social environment, which includes family and peers.
- The student characteristics (motivation, attitude, preferences, performances, and behavior) which influence the educational outcomes.
- The educational outcomes, which are the persistence in STEM education and educational choices.
The theoretical framework is used to sort the reviewed studies into categories.
The 119 publications remaining after the initial selection undergo a quality assessment, which shows some interesting information. Most of the studies used a pre/ post-test design without a control group, which is at risk of incurring in selection bias. For example, many studies describe the evaluation of a summer science camp in which participants were asked to fill an attitude survey before and after the experiment, not considering that willing participants are more likely to be positively influenced by the experience. Therefore, those designs aren’t considered useful to draw reliable conclusions. Only 30 studies use a quasi-experimental or experimental design, and are assessed using the following criteria:
- Trustworthiness of the executed design,
- Sampling and assignment of the intervention group and control group,
- Instrumentation used to measure the dependent variable.
All the requirements narrow the selection to 9 papers that can be used to draw conclusions about the effectiveness of the intervention. Then the studies are categorized using the theoretical framework.
- Studies are categorized in the social context when the interventions are aimed at altering educational policy and/or cultural and social views. There are 2 publications in this category, attempting to influence the student characteristics. In particular, they are aimed at enhancing women’s motivation and attitude by providing them with female role models.
- Studies in the school context are about interventions in pedagogy, school climate and school-level educational policies. 6 studies are in this category. 4 interventions are focused on improving or renewing the STEM pedagogy, by using more ICT (information and communication technology) or more hand-on experiences inside and outside the classroom. The 2 remaining studies are about training programs for women and minorities. All studies reported a positive effect of the intervention.
- The last study is inserted into the social environment category. It is focused on an intervention on the familiar environment as it is expected to influence students’ motivation to pursue an educational career in STEM. In details, the intervention is a randomized experiment in which parents are informed about the positive aspects of choosing STEM courses in upper high school by means of brochures and websites. The objective is to enhance parents’ view of STEM and increase communications on the subject between them and adolescents. The conclusion is that the intervention positively influenced both the parents’ perception of the value of STEM courses and the number of STEM courses chosen by the adolescents. Moreover, it seems that gender doesn’t have a significant effect whereas the parents’ educational level is a predictor for the number of STEM courses taken by the children.
The publication remarks that systematic reviews always suffer from publication bias as it’s easier to publish an intervention with a positive effect even if negative effects are as much important for the development of new interventions. Moreover, the heterogeneity of the studies makes it difficult comparing the results. Despite these limitations, it seems that interventions focused on “role models” aimed at enhancing students’ motivation and belonging in STEM, STEM-oriented summer camps, and raising awareness about the importance of STEM education among parents and students are effective. However, the small number of successful interventions and the limitations mentioned above make it difficult to draw any general conclusion about what seems to work best.
