What is the Self-Reliance Index and why is it important?

The Self-Reliance Index is a holistic, scored survey tool assessing 12 domains of a displaced family's life. It's important because it provides a broad view of a household's well-being, helps identify gaps in services, and reflects what refugees themselves desire: moving from dependency to self-sufficiency.

How do Randomized Control Trials (RCTs) work for humanitarian programs?

RCTs involve randomly assigning eligible individuals to receive a program or not (or a delayed receipt) to rigorously measure impact. This methodology helps understand under what conditions interventions are effective, even when faced with limited resources and complex contexts.

What are the challenges in measuring the long-term impact of refugee programs?

Challenges include the impatience of funders for quick results, the time it takes for impacts to materialize, and the limited program funding cycles (often 12 months) which don't allow for post-program follow-up to assess durability.

What is 'decent work' and why is it important for evaluating refugee programs?

Decent work, as defined by the ILO, includes fair working conditions, sufficient pay, formal employment, and reasonable working hours. Measuring this is crucial because it reflects the quality of employment, not just the quantity, and aligns with broader UN goals for economic growth.

Why are ethical considerations crucial when evaluating refugee programs?

Ethical considerations, including informed consent, privacy, and mitigating potential harm or distress, are paramount. Robust institutional review processes are essential before data collection to ensure participants' rights and well-being are protected.

How can programs be better designed to address the needs of displaced populations?

Programs should incorporate the voice of the beneficiaries, understand their context, and use holistic assessment tools. Communication is key: clearly explain the purpose of assessments and be flexible in how topics are approached and sequenced to build trust.

Are there efforts to standardize indicators in education for emergency contexts?

Yes, the Inter-Agency Network for Education in Emergencies (INEE) has launched a reference group to streamline indicators, aiming for a reliable set of measures to be used globally in education in emergencies.

What are the limitations of focusing evaluation solely on short-term results?

Stopping evaluations at six months or one year can miss significant impacts, such as participants transitioning to formal work later, productivity gains, or shifts to better quality employment. This can lead to underestimating a program's true effectiveness.

Key Moments

How do we know what works? Research, Measurement and Evaluation

Q: Do funding agencies keep pace with evaluating long-term outcomes for refugees?

There's been positive movement with increased flexibility in funding cycles, multi-year guarantees, and less focus on simple cost-per-beneficiary metrics. However, the prevalence of 12-month program cycles and difficulty in funding post-program assessments remain challenges.

MIT OpenCourseWare

Education3 min read74 min video

May 19, 2022|235 views|2

Migration Summit MIT ReACT Higher Education Workforce Development Displacement Refugees

Save to Pod

Key Moments

TL;DR

Evaluating refugee programs requires holistic, long-term measurement beyond immediate outcomes.

Key Insights

Measuring the 'quality' and 'long-term impact' of refugee education and livelihood programs is crucial, not just access or immediate employment.

Holistic measurement tools, like the Self-Reliance Index, capture multiple life domains (education, health, shelter) for a comprehensive understanding.

Randomized Control Trials (RCTs) can provide robust evidence on program efficacy but require careful design and long-term follow-up.

Decent work indicators, beyond mere employment, assess fair conditions, sufficient pay, and sustainable employment for beneficiaries.

Effective evaluation requires ethical considerations, informed consent, and clear communication with participants about program goals and limitations.

Donor funding cycles and reporting requirements often favor short-term results, hindering the assessment of long-term impact and sustainability.

THE CHALLENGE OF MEASURING IMPACT IN REFUGEE PROGRAMS

The session addresses the critical need to evaluate the effectiveness of refugee education and livelihood programs. Speakers highlight that traditional evaluations often focus on immediate outcomes like access to education or employment, neglecting the quality and long-term sustainability of these interventions. The challenge lies in determining if programs truly contribute to refugees' self-reliance and well-being beyond initial metrics.

HOLISTIC MEASUREMENT AND THE SELF-RELIANCE INDEX

Ned Meerdink introduced the Self-Reliance Index, a scored survey tool developed by a multi-stakeholder coalition. This index assesses a displaced family's life across 12 domains, including education, health, food, and shelter, to gauge their self-reliance holistically. It moves beyond single-sector focus to understand how basic needs are met, through what means, and assesses the sustainability of those means, offering a comprehensive view for practitioners.

THE ROLE OF RANDOMIZED CONTROL TRIALS (RCTS)

Emma Smith discussed the application of Randomized Control Trials (RCTs) in humanitarian programming, particularly with Syrian refugees in Jordan. RCTs involve randomizing program beneficiaries to assess impact rigorously. While complex to implement, they provide strong evidence on what works under specific conditions, helping to guide decisions on resource allocation and scale-up, even when initial analysis reveals mixed or surprising results.

ASSESSING 'DECENT WORK' AND LONG-TERM OUTCOMES

Christina Peebles presented Paper Airplanes' evaluation of their 'Women in Tech' program, emphasizing the importance of assessing 'decent work' beyond just employment. This includes indicators like sufficient pay, formal employment status, and avoiding underemployment or excessive working hours. The evaluation also stressed the need for long-term impact studies, as outcomes like productivity gains and transitions to formal work may only materialize well after the initial program ends.

METHODOLOGICAL CONSIDERATIONS AND ETHICAL IMPLICATIONS

Joanna Hofman shared insights from an evaluation of an emergency education program in Jordan, highlighting the use of mixed methods, including participatory photography, to capture participants' experiences. The discussion underscored the practical challenges in designing robust evaluations during emergencies and the trade-offs between different methodologies. Ethical considerations, informed consent, and ensuring participants' voices are central to the evaluation process were stressed by all panelists.

ALIGNING DONOR EXPECTATIONS AND FUNDING CYCLES

A significant challenge identified is the misalignment between evaluation needs and donor funding structures. Donors are increasingly open to multi-year funding and flexible programming cycles, showing less emphasis on simple cost-per-beneficiary metrics. However, the persistent prevalence of 12-month funding cycles and the difficulty in securing funds for post-program follow-up assessments hinder the ability to track durable and sustainable change in beneficiaries' lives.

Mentioned in This Episode

●Organizations

●Concepts

●People Referenced

Common Questions

Evaluations often focus on access to education and basic employment status or income. However, there's a recognized need to measure quality of education, 'decent work' indicators (like fair pay, formal employment, reasonable hours), and long-term impacts beyond the immediate program completion.

Topics

Health & Longevity Human Performance Mindset & Self-Improvement Society & Philosophy Program Evaluation Refugee Support Humanitarian Aid Impact Measurement Livelihood Programs Education Quality Measurement Tools Long-term Outcomes

Mentioned in this video

personLorraine Charles

Executive Director of Normal and co-chair of the MIT Migration Summit, moderating the panel.

organizationNormal

Organization where Lorraine Charles is the Executive Director.

organizationMIT Migration Summit

A month-long global convening designed to build bridges between diverse communities around challenges and opportunities for refugee and migrant communities.

organizationUCL

University where Christina Pebble is pursuing a second master's in education planning, economics, and international development.

conceptSyrian refugees in Jordan

A population group studied by Emma Smith using RCTs and also mentioned in the context of an emergency education response evaluation.

organizationInternational Labour Organization (ILO)

An organization whose definition of 'decent work' is used as a framework for evaluation, and which has country-specific profiles for indicators.

organizationUNHCR

Possibly referring to the UN Refugee Agency, mentioned in relation to program funding cycles.

organizationRefuge Point

An organization where Ned Mercek works, co-chairing the Refugee Self-Reliance Initiative.

studyHarvard University

University where Emma Smith is a PhD candidate in Public Policy.