Acknowledging complexity in evaluation of gender equality interventions

A collection of evidence on gender equality published in EClinicalMedicine discusses the systemic nature of restrictive gender norms in science and medicine, and calls attention to the importance of institutional interventions to overcome restrictive gender norms[[1]Darmstadt G.L. Gender equality: framing a special collection of evidence for all.EClinicalMedicine. 2020; 20https://doi.org/10.1016/j.eclinm.2020.100307Summary Full Text Full Text PDF Scopus (1) Google Scholar]. Inequality is deep-rooted in science and medicine while evaluation of interventions show a very slow progress and often unintended consequences [[2]Moughalian C. Täuber S. When gender equality initiatives risk doing more harm than good.EClinicalMedicine. 2020; 22https://doi.org/10.1016/j.eclinm.2020.100330Summary Full Text Full Text PDF PubMed Scopus (1) Google Scholar]. For example, women often undertake a disproportionate amount of gender equality work, institutional gender equality plans can become box-ticking exercises, men can feel discriminated against, and gender can take prevalence over race and class [[3]Kalpazidou Schmidt E. Ovseiko P.V. Henderson L.R. Kiparoglou V. Understanding the Athena SWAN award scheme for gender equality as a complex social intervention in a complex system: analysis of Silver award action plans in a comparative European perspective.Health Res Policy Syst. 2020; 18https://doi.org/10.1186/s12961-020-0527-xCrossref PubMed Scopus (19) Google Scholar]. While the analysis of the reasons for inequality and the need for policy interventions is well established, evaluation of gender equality interventions is challenging due to the complex nature of gender norms and many interacting factors. In particular, recent studies of gender equality interventions draw attention to the difficulties in attributing effect and in evaluating their overall impact [4Kalpazidou Schmidt E. Cacace M. Addressing gender inequality in science: the multifaceted challenge of assessing impact.Research Evaluation. 2017; 26: 102-114https://doi.org/10.1093/reseval/rvx003Crossref Scopus (17) Google Scholar, 5Rosser S.V. Barnard S. Carnes M. Munir F. Athena SWAN and ADVANCE: effectiveness and lessons learned.The Lancet. 2019; 393: 604-608https://doi.org/10.1016/S0140-6736(18)33213-6Summary Full Text Full Text PDF PubMed Scopus (26) Google Scholar, 6Kalpazidou Schmidt E. Graversen E.K. Developing a conceptual evaluation framework for gender equality interventions in research and innovation.Eval Program Plann. 2020; 79101750https://doi.org/10.1016/j.evalprogplan.2019Crossref PubMed Scopus (9) Google Scholar]. We concur that evaluating the impact of such complex interventions is indeed problematic when impact evaluation is solely predicated on attribution and linear causality. Here, we argue that the solution lies in acknowledging and operationalising complexity as a frame of reference and call for a paradigm shift in evaluating gender equality interventions. Key considerations of complexity in gender equality interventions involve [[3]Kalpazidou Schmidt E. Ovseiko P.V. Henderson L.R. Kiparoglou V. Understanding the Athena SWAN award scheme for gender equality as a complex social intervention in a complex system: analysis of Silver award action plans in a comparative European perspective.Health Res Policy Syst. 2020; 18https://doi.org/10.1186/s12961-020-0527-xCrossref PubMed Scopus (19) Google Scholar,[4]Kalpazidou Schmidt E. Cacace M. Addressing gender inequality in science: the multifaceted challenge of assessing impact.Research Evaluation. 2017; 26: 102-114https://doi.org/10.1093/reseval/rvx003Crossref Scopus (17) Google Scholar,[7]Kalpazidou Schmidt E. Cacace M. Setting up a dynamic framework to activate gender equality structural transformation in research organizations.Sci Public Policy. 2019; 46: 321-338https://doi.org/10.1093/scipol/scy059Crossref Scopus (14) Google Scholar]. First, multiple actions and areas of intervention. For example, Athena SWAN gender equality action plans in the UK have on average more than 30 actions addressing five major areas (organisation and culture, career development, self-assessment and monitoring, key career transition points, flexible working and career breaks)[[3]Kalpazidou Schmidt E. Ovseiko P.V. Henderson L.R. Kiparoglou V. Understanding the Athena SWAN award scheme for gender equality as a complex social intervention in a complex system: analysis of Silver award action plans in a comparative European perspective.Health Res Policy Syst. 2020; 18https://doi.org/10.1186/s12961-020-0527-xCrossref PubMed Scopus (19) Google Scholar]. In the USA, the ADVANCE program provides institutions with competitive grants, which commonly support multiple interventions including gender-disaggregated data collection, mentoring schemes, work-life balance policies, and guidance on enhancing faculty careers for women in STEM [[5]Rosser S.V. Barnard S. Carnes M. Munir F. Athena SWAN and ADVANCE: effectiveness and lessons learned.The Lancet. 2019; 393: 604-608https://doi.org/10.1016/S0140-6736(18)33213-6Summary Full Text Full Text PDF PubMed Scopus (26) Google Scholar]. Second, focus on the local dynamics. Interventions are tailored to the local contexts of specific institutions in order to disrupt local self-organisation processes maintaining gender norms – one size does not fit all. Third, non-linear nature of interventions. Due to the high number of variables involved in gender equality interventions and their constantly emergent character, effects cannot be directly attributed to interventions. Fourth, dynamic adaptation to constantly emerging conditions. A continuous monitoring and adaptation of interventions in response to implementation feedback, new emerging conditions, unintended consequences, and changes in the wider social, economic, and political context is crucial to achieve structural and cultural change. Fifth, probabilistic nature of change. Impact of gender equality interventions is expected in terms of contribution to change, improved conditions to foster change, and working to increase the probability that change can occur. Such considerations of complexity emerge from a number of studies published in the recent collection on gender equity in EClinicalMedicine. For example, a national survey of Canadian medical students' experiences of sexual harassment demonstrates that when official policies regarding sexual harassment in medical education fail to disrupt societal gender norms, such policies are not only ineffective, but can also inadvertently cause harm to victims [[8]Phillips S.P. Webber J. Imbeau S. et al.Sexual harassment of canadian medical students: a national survey.EClinicalMedicine. 2019; 7: 15-20https://doi.org/10.1016/j.eclinm.2019.01.008Summary Full Text Full Text PDF PubMed Scopus (16) Google Scholar]. Several association studies also highlight the complex causality and non-linear impact of gender equality interventions. For example, a nationally-representative study from India suggests that current policy efforts focused on affecting sex ratio imbalance are unlikely to succeed without challenging social norms regarding son preference and reduced care for infant girls [[9]Raj A. Johns N.E. McDougal L. et al.Associations between sex composition of older siblings and infant mortality in India from 1992 to 2016.EClinicalMedicine. 2019; 14: 14-22https://doi.org/10.1016/j.eclinm.2019.08.016Summary Full Text Full Text PDF PubMed Scopus (3) Google Scholar]. A study based on data from 97 countries shows that greater gender parity in education and work is associated with better health outcomes not only for females, but also for males [[10]Gadoth A. Heymann J.. Gender parity at scale: examining correlations of country-level female participation in education and work with measures of men's and women's survival.EClinicalMedicine. 2020; 20100299https://doi.org/10.1016/j.eclinm.2020Summary Full Text Full Text PDF PubMed Scopus (4) Google Scholar]. Overall, we argue for acknowledging and operationalising complexity as a frame of reference in gender equality interventions and for a paradigm shift towards impact evaluation models open to context-sensitivity and emergent causality, non-linearity, and a probabilistic nature of change. EKS and PVO co-wrote this commentary. EKS and PVO have nothing to disclose. EKS was supported by the Aarhus University Research Foundation. PVO is supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre, grant BRC-1215–20008 to the Oxford University Hospitals NHS Foundation Trust.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.