Evaluation of JPF Funded Projects in South Sudan (JPF-SS-22-004) Consultancy

1. Background:

The Japan Platform (hereinafter referred to as “JPF”) is an international emergency humanitarian aid organization which offers the most effective and prompt emergency aid in response to humanitarian needs, focusing on issues of refugees and natural disasters. JPF conducts such aid through a tripartite cooperation system where NGOs, business communities, and the government of Japan work in close cooperation, based on equal partnership, and making the most of the respective sectors’ characteristics and resources.

Since 2006, JPF has been funding Japanese NGOs responding to the chronic humanitarian crisis in South Sudan, predating South Sudan’s independence in 2011. JPF remained engaged in the face of the political unrest of 2013 and the ensuing period of instability. The 2018 Revitalized Agreement on the Resolution of the Conflict in the Republic of South Sudan (R-ARCRSS) paved the way for the formation of the Revitalized Transitional Government of National Unity in February 2020. Nonetheless, the humanitarian situation in South Sudan remains dire. In 2022, the humanitarian community in South Sudan estimates that more than two-thirds of South Sudan’s population, 8.9 million people, are in need of humanitarian assistance, an increase of 600,000 since 2021. The country continues to experience the cumulative effects of years of conflict, a surge in sub-national violence, unprecedented flooding and hyperinflation, further compounded by the COVID-19 pandemic. JPF repeatedly renewed its annual funding commitment to the South Sudan refugee programme in South Sudan and its neighboring countries. In FY 2021, JPF funded 8 projects in WASH, protection and education implemented by 6 agencies across 5 countries. JPF’s South Sudan refugee program set the following priority objectives that are identified by its member agencies in line with the 2021 Humanitarian Response Plan (HRP).

1. Address urgent humanitarian needs of the crisis affected people that have been aggravated and compounded by COVID 19
2. Strengthen resilience of people against sudden and complex risks

JPF annually conducts evaluations on selected projects under the South Sudan program. Some of the key issues emerged from the past evaluations on WASH and Protection as well as the meta-evaluation of 2016-2020 are as follows. Those are learning needs for JPF in conducting this evaluation.

Multi-year Evaluation:

• Although over 50 projects had been implemented during the multi-year programs (2016-2018, 2018-2020), the evaluation was unable to verify a cumulative impact conclusively, due to lack of quantified data. It is important to streamline an evaluation framework for a multi-year program in the future, with clearly defined strategic goals and verifiable indicators.

WASH:

• Insufficient resources: Access to water was improved for the target population but fell short of meeting the Sphere standards given the population in target areas much exceeded the water points provided by the project.
• Maintenance & repair: There had been greater needs for repair of water points than the project could have responded to. Consider ease of repair and locally manageable operations and maintenance in choice of water systems.
• Scope of WASH services: A project met critical WASH needs that no other agencies had
• addressed such as solid waste management and desludging of latrines.
• Behavior change: Hand washing behavior, use and cleanliness of latrines required continuous follow up.
• Sustainability: Capacity building for the project-installed committees required ongoing follow up support. Lack of incentives for committee members threatened their continuous engagement beyond the project period. Communities could not self-finance activities funded by a project.

Protection:

• Alignment with the existing protection system: It is important to coordinate with other protection agencies to strengthen the existing system and structure for community-based protection. Effectiveness and sustainability are undermined, where different agencies create their own sub structures and parallel systems without adequate engagement of relevant government offices and trusted community leaders who may not have formally assigned roles in protection but are counted on by community members.
• Clarity of roles and effective relationships among key community workers and focal points were important to the community based protection system. Inclusion of male and female case workers, the youth and the elder in leadership for conflict resolution, accompanied by tailored support to address gender and inter-generational barriers, were particularly effective in breaking the entrenched norms and reinforcing recognition and respect for each one’s roles. There is a need for ongoing technical support to strengthen their capacity.
• Effectiveness in case management: Effective response to cases may require closer coordination between the protection structures in camps and host communities, and capacity building for police, medical personnel and local authorities. Even if referral pathways are developed, the scarcity of protection services available and access to transport limit its effectiveness.
• Insufficient resources: Capacity was built for case management but the caseload far exceeded the desired ratio of cases per caseworker in some camps.
• Sustainability: At times, child protection committees and/or para-social workers worked voluntarily alongside community volunteers who played a similar role but with incentives. Lack of consistency in payment of incentives demotivated the former.

In addition, JPF is keen to explore how it may advance the localization agenda called for since the 2016 World Humanitarian Summit. Although JPF has not consciously taken the localization agenda into account in developing the program strategy, its portfolio or financing modalities, it hopes to revisit the current ways of working vis-à-vis the localization agenda and explore how it delivers on the localization agenda in moving forward.

2. Purpose of the evaluation

The purpose of the evaluation is to evaluate two of the JPF-funded projects in Juba in line with the OECD DAC evaluation criteria and the Core Humanitarian Standards (CHS), and draw recommendations and lessons learnt for the respective implementing agencies as well as for JPF’s program strategy. The past evaluations of these projects exclusively focused on one of the successive phases of the projects under JPF’s annual funding scheme, and assessed the projects’ achievements against the predetermined outcomes and outputs. Given that JPF is committing to a multi-year funding scheme for South Sudan for the period of FY 2022-2024, this evaluation will assess the long-term results of the two projects funded annually for multiple years, and seek to draw relevant lessons and recommendations for JPF and its member implementing agencies in strategizing multi-year programming.

The evaluation also retrospectively review the current implementation and financing modalities in light of the localization agenda. It explores opportunities and potential for advancing localization agenda, and hence informs JPF’s dialogue with member agencies and other stakeholders on its positioning and strategy towards localization.

The primary audience of the evaluation are JPF and the implementing agencies, REALs and PWJ. The evaluation report will be also communicated to the wider audience in South Sudan and in Japan to share relevant lessons learnt for the broader sector and as a means to hold JPF accountable to its supporters and stakeholders.

3. Scope of the evaluation

The evaluation will address the following areas of inquiry with regard to the two projects respectively.

1. Assess the extent to which the newly identified needs of the target population under the current phase (Mangaten camp in REALs’ project and Juba, Mahad, and Don Bosco IDP camps in PWJ’s project), has been addressed in a timely and coordinated manner. In doing so, assess the level of satisfaction with the current project interventions among the vulnerable groups identified as such in the given project contexts, such as women, children, youth, the disabled and/or SGBV victims. Where there are tensions and competing interests between different social groups, assess if and how these groups perceive they have benefitted from the projects equitably. (in line with OECD DAC criteria of effectiveness and relevance)
2. Assess long-term results of the two projects over different phases, both intended and unintended, on the everyday lives of the target population (REALs’ project), and on the local capacity and systems to sustain the project achievements (PWJ’s project) (in line with OECD DAC criteria of impact and/or sustainability as well as CHS)
3. Analyze how intended and unintended results have occurred, and identify contributing and constraining factors to the changes. Assess the extent to which these factors have been attributable to different approaches taken by the projects in different phases, and if and how these approaches have been complemented by ongoing efforts of other actors. In doing so, validate the assumptions and hypothesis underpinning the project designs and positioning vis-à-vis other actors, and appropriateness and timeliness of the project responses to the past evaluations. (in line with OECD DAC criteria of effectiveness and coherence).
4. Identify actual and potential alignment of the project with the core responsibilities 4A Reinforce local systems & 5A Invest in local capacities under the Agenda for Humanity, with regard to the following questions (in line with OECD DAC criteria of efficiency and sustainability as well as CHS).
   1. Reinforce local systems: International actors should enable people to be the central drivers in building resilience and be accountable to them through consistent community engagement and ensuring their involvement in decision-making. The international community should respect, support and strengthen local leadership and capacity in crises and not put in parallel structures that may undermine it.
      1. How have the projects involved local actors?
      2. How have the projects allowed communities and local actors to become more active in decision making?
      3. How have the projects increased the capacity of communities and local actors?
      4. Have the capacity building of these actors enables them to sustain project achievements beyond the projects’ lifetime?
   2. Invest in local capacities: Local actors are the best placed to know the underlying risks and priorities of communities. While those factors place them in the ideal position to provide humanitarian assistance, local actors can struggle to scale high-volume delivery and sustain adequate resources to support a lasting organizational presence. Direct and predictable financing, where possible, should be provided to national and local actors along with long-term support to develop their capacity to prevent, respond and recover from crises.
      1. How has JPF’s annual funding scheme and financing modalities facilitated or hindered REALs’ and PWJ’s efforts at localization,
5. Draw lessons learnt from the evaluation for REALs, PWJ, JPF and other JPF member agencies active in South Sudan.
   1. Recommendations for the two implementing agencies on the respective project designs, implementation and stakeholder engagement.
   2. Replicable lessons learnt on gender and conflict transformative programming, behavior change communications, and community engagement that are of broader relevance beyond these projects.
   3. Recommendations for JPF on the program priorities and financing modalities.
   4. Areas for further discussion between JPF and its member agencies on the localization agenda

4. Methodology

Given the nature of this evaluation, the consultant will primarily make use of participatory data collection as main methods, complemented by secondary and quantitative data. The evaluation will consider the different focus of REALs and PWJ’s projects in assessing long-term results of the respective projects (the scope 2). By tracing the process and factors that led to the intended and unintended changes, the evaluation will validate the theory of change underpinning the projects (the scope 3) and retrospectively assess the projects’ alignment with localization (the scope 4). The evaluation will use consistent methodology in assessing the two projects’ alignment with localization agenda so the findings can be synthesized. In responding to the scope of the evaluation, the consultant will undertake the following types of assessment and analysis. Analytical tools and methods referred to are only indicative, and the consultant is requested to develop appropriate methods to collect required data.

5. Deliverables

• An inception report, describing the methodology and plan for the evaluation and annexing the evaluation tools.
• A debriefing meeting with REALs and PWJ’s field teams
• A presentation of the preliminary findings and recommendations to JPF, REALs & PWJ
• A final report, responding to the Terms of Reference