By Rai Sengupta
Ecorys UK
Globally, governments and development institutions are gradually moving from ‘issue-based’ to ‘systems-based’ approaches in child protection. While the former focuses narrowly on specific child protection issues (often addressing the “low hanging fruit”), the latter looks to deliver child protection outcomes at scale, while involving a range of stakeholders at the national and sub-national levels. In essence, a child protection system is a structural framework consisting of “human resources, finance, laws and policies, governance, monitoring and data collection as well as protection and response services and care management” (UNICEF, 2012).
This blog examines ways in which the effectiveness of child protection systems can be innovatively evaluated, by focusing on the principles and methods that can be employed to assess the extent to which child protection systems meet their desired objectives. Evaluation of child protection systems is crucial, for it generates vital evidence to inform policy and programming, in pursuit of SDG 16.2 (protect children from abuse, exploitation, trafficking, and violence).
How can we evaluate the effectiveness of child protection systems?
The OECD Development Assistance Committee (DAC) highlights six key criteria for evaluating interventions, one of which is effectiveness. Effectiveness is defined as the extent to which an intervention meets its intended objectives, the process through which this was accomplished, and the factors that influence intended and unintended consequences. Evaluating effectiveness is essential for improving programme performance through subsequent course correction (OECD, 2021).
Evaluating whether an entire child protection system – comprising a range of individuals and institutions – is effective involves a complex task. At one level, it requires clarity on intended objectives and measures of effectiveness across various sub-systemic components. Further, it requires analysis of outcomes achieved for all sub-groups of children, not just vulnerable children (Wessells, 2014). The following sections highlight evaluation principles and methods through which the effectiveness of child protection systems can be evaluated.
Evaluation principles
A child protection system consists of a range of services including ‘evidence-based programmes, practices, processes and workforce development’ (Molloy et al., 2017). Rigorously evaluating the effectiveness of such a complex, multidimensional system requires the perspectives of various stakeholders to be incorporated. This can be accomplished through a participatory evaluation. Participatory evaluation involves the relevant stakeholders in a policy/programme across various stages of the evaluation process - from developing the evaluation framework to data collection and validation of findings and recommendations to facilitating the use of the evaluation. In the context of evaluating child protection systems, it is imperative to promote participatory methodologies, wherein administrators, practitioners, and beneficiaries engage throughout the evaluation process to provide inputs, validate findings and co-develop policy recommendations (INTRAC, 2017).
Adopting a participatory approach is likely to improve the quality of information gathered. Further, such an inclusive evaluation design is more likely to provide true estimates of systemic effectiveness. For instance, public records on child protection are likely to contain outcomes associated with the establishment of child protection infrastructure (for instance, child welfare committees). However, direct and indirect beneficiaries (for instance, the children and their families/communities) would be more reliable sources of information on child welfare outcomes (Joynes and West, 2018). Analyzing only the first kind of outcomes would provide a superficial impression of systemic effectiveness, while triangulating both forms of evidence through a participatory process would ensure more reliable estimates.
The need to include children’s voices in evaluating child protection systems resonates across various child rights institutions. UNICEF (2021) asserts that as service users, children’s feedback and complaints must be incorporated in any assessment of effectiveness across child protection systems. This is vital since pre-existing power dynamics may mask the perspectives of children, while underestimating the evidence gleaned from them. In a similar vein, Save the Children (2019) highlights the need for evaluations to reflect children’s voices, while disaggregating outcomes for gender, ethnicity, and race. This is important for assessing the distribution of intended and unintended outcomes across various socio-economic and demographic categories.
Measuring effectiveness for an entire child protection system requires understanding how effectiveness may best be captured for all sub-systemic components.
Evaluation methods
The methodology involved in evaluating the effectiveness of child protection systems is diverse and highly context specific. However, evaluating effectiveness in most cases begins with examining and/or reconstructing the theory of change – the theoretical pathways linking the various sub-systemic components to their intended outcomes. Doing so helps to develop consensus on ‘what good looks like’ (Molloy et al., 2017) - which sets the expectation on what results are to be measured in any effectiveness analysis. In the development of indicators and targets for each outcome, adherence to global conventions may be a useful starting point. For instance, the Convention on the Rights of the Child (CRC) provides a comprehensive list of principles underpinning effectiveness in a child protection system, which can help frame targets and measure systemic effectiveness, across logical frameworks (Bruning and Doek, 2021).
Further, measuring effectiveness for an entire child protection system requires understanding how effectiveness may best be captured for all sub-systemic components. For instance, ‘singling out’ the effect of a specific intervention (say the establishment of a children’s helpline number) may involve the conduct of a Randomized Controlled Trial (RCT), wherein judgements of effectiveness are based on quantitative outcomes, measured across treatment and control groups. However, for components such as practices to enhance parent sensitivity, or laws enacted to promote child rights – effectiveness measurements may require the inclusion of qualitative insights as well.
Building upon the participatory and consultative evaluation design discussed above, effectiveness at a systemic level must inevitably leverage evidence – not only from various stakeholders, but also of different types. In essence, any evaluation at the systems level requires a mixed-methods approach, to account for the forms of data available and the formats they are available in. A mixed-methods approach combines qualitative and quantitative evidence in data collection and analysis, thereby building on the strength of each data type, while minimizing the weaknesses associated with any one form of evidence. Such an approach – which may be sequential, parallel or multilevel -- also lends support to the practice of data triangulation, thereby generating more robust estimates of systemic effectiveness (UNICEF, 2017).
Evaluating the effectiveness of a child protection system requires a multiplicity of stakeholder perspectives and methodologies.
Conclusion
In sum, evaluating the effectiveness of a child protection system requires a multiplicity of stakeholder perspectives and methodologies. Adopting participatory principles and mixed-methods analysis is key, particularly for evaluating complex sub-systemic components within this sector. Going forward, it is imperative to develop a common understanding (and a common set of indicators) to evaluate child protection systems globally. Further, leveraging partnerships with development partners, and enhancing government capacity to manage evidence will be crucial for facilitating systemic evaluations.
Bibliography
Bruning, M. & Doek, J. (2021). Characteristics of an Effective Child Protection System in the European and International Contexts. International Journal on Child Maltreatment: Research, Policy and Practice.
Child Protection Systems Task Group. (2019). Strengthening Child Protection Systems: Guidance for Country Offices. Save the Children.
INTRAC. (2017). Participatory Evaluation. Author.
Joynes, C., & West, H. (2018). Evidence of Effective Child Protection Systems in Practice.
Molloy et al. (2017). Improving the Effectiveness of the Child Protection System. Local Government Association.
OECD. (2021). Applying Evaluation Criteria Thoughtfully. Author.
Wessells, M. (2014). The Case for Population Based Tracking of Outcomes for Children Toward a Public Health Approach in Child Protection System Strengthening. Columbia University.
UNICEF East Asia & Pacific. (2012). Measuring and Monitoring Child Protection Systems. Author.
UNICEF. (2017). Child Protection Resource Pack: How to Plan, Monitor and Evaluate Child Protection Programmes. Author.
UNICEF. (2021). The UNICEF Child Protection Systems Strengthening Approach. Author.
Rai Sengupta, a young and emerging evaluator, brings over five years of international experience in monitoring and evaluating large scale development programmes. She presently works as a Senior Monitoring and Evaluation Consultant at Ecorys UK, an international development consulting firm in London, wherein she is part of evaluation teams for large-scale development programmes across India, Viet Nam, Nigeria, and Zimbabwe. Rai has an MSc in Evidence-Based Social Intervention and Policy Intervention from the University of Oxford where she was a fully funded Weidenfeld Hoffmann Scholar. Follow Rai on Twitter and LinkedIn.
