Designing and measuring the progress and impact of health research capacity strengthening initiatives

Abstract

Strengthening capacity in poorer countries to generate multi-disciplinary health research and to utilise research findings, is one of the most effective ways of advancing the countries' health and development. This paper explores current knowledge about how to design health research capacity strengthening (RCS) programmes and how to measure their progress and impact. It describes a systematic, evidence-based approach for designing such programmes and highlights some of the key challenges that will be faced in the next 10 years. These include designing and implementing common frameworks to facilitate comparisons among capacity strengthening projects, and developing monitoring indicators that can capture their interactions with knowledge users and their impact on changes in health systems.

Measuring and learning about health research capacity strengthening for development

Strengthening health research capacity in low and middle income countries (LMICs) is one of the most effective ways of advancing their health and development. Over the next 10-20 years the current trend towards more emphasis on capacity strengthening within health research projects is likely to accelerate. This is because lack of capacity to utilise research findings, to integrate them into health systems and embed them in institutions, is a major bottleneck in our ability to maximize the benefits of research for development.

Definition of research capacity strengthening - By health RCS we mean a ‘process of individual and institutional development which leads to higher levels of skills and greater ability to perform useful research.’

Finding the best ways to do capacity strengthening and to measure its effectiveness requires a much better understanding of the processes involved than we have at present. It will involve critically reviewing and sharing lessons about what works and what doesn't work, and using much more robust ways to measure the impact of research capacity strengthening (RCS) efforts. Common frameworks to guide health RCS efforts will be needed[1] to facilitate comparison among different projects and for real-time lesson learning (for examples see table 1).

Table 1 Example of using a common framework based on ‘phases’ of progress to systematically compare case studies of sustainable capacity strengthening programmes[11]

Health RCS takes many forms and operates at three different levels, often simultaneously – individual (e.g. PhD training), institutional (e.g. establishing research governance systems) and national/international (e.g. international collaborations). The complexity and heterogeneity of RCS initiatives[2] makes it difficult to evaluate their effectiveness.

Our Capacity Research Unit at LSTM does research to inform better design and measurement of RCS programmes. From our work with health research funders, research institutions and national research systems, and through our own experience of managing capacity strengthening programs in LMICs we have developed an evidence-based 5-step approach to designing RCS programmes which is applicable to diverse contexts. We have collated information about different approaches to RCS and summarized what works best in which type of context and why. We have analysed evaluators' reports to bring together a comprehensive list of indicators that have been used to measure RCS processes and impact.

Our research has inevitably unearthed some critical priorities, challenges and tensions that need to be addressed over the next 10-20 years to ensure we make significant progress in bridging the gap between research generation and population health and development in LMICs. Integration of different research disciplines, as well as enhanced research capacity, is essential to make a significant impact on strengthening health systems and to support effective policies and decision-making. In this paper we outline some of the key findings of our research and highlight what has emerged from our research as priorities for the next 10 years.

Designing RCS programmes: 5-step approach

Evidence to guide the design and monitoring of RCS efforts in LMICs is limited and formulating a general approach is complex because each programme is unique. Capacity strengthening programmes usually start off with a ‘needs assessment’ but by doing this, two critical steps are missed out – defining the expected outcomes and identifying the capacity needed to achieve the outcomes. Both these steps are important in developing a robust and comprehensive action plan and in promoting ownership and sustaining change. Using published evidence, including literature on organisational management, we have developed a five step pathway for designing and evaluating RCS programmes and tested it in a variety of contexts in Africa[3–5].

The pathway involves:

• a) defining the goal of the capacity strengthening effort,

• b) describing the optimal capacity needed to achieve the goal,

• c) determining the existing capacity gaps compared to the optimum,

• d) devising an action plan to fill the gaps and associated indicators of change, and

• e) adapting the plan and indicators as the programme matures.

The pathway starts with a clear goal and objectives, and it make the capacity required to achieve the goal explicit. We derive a description of the optimal capacity needed to achieve the goal from a comprehensive search of the literature and include inputs by international experts relevant to the goal of the RCS, and a ‘reality check’ by LMIC partners. The unique aspect of this 5-step approach is the production of this evidence-based ‘benchmark’ (i.e. definition of optimal capacity needed to achieve the goal) which is designed specifically for each project and against which existing capacity is compared. This approach differs from most needs assessments of research capacity which do not use a rigorous benchmark and therefore are not able to provide reassurance that significant areas have not been overlooked.

Using the optimal capacity ‘benchmark’ we identify capacity gaps and develop the action plan to fill the gaps in collaboration with LMIC partners. The plan is designed to be flexible enough to generate and utilise ongoing learning; strategies for promoting sustainability are incorporated from the outset. The plan takes into account not only technical, managerial and financial processes within an organisation, but also the individuals who work in the organisation and the wider system within which the organisation operates and is financed and managed. Interestingly we have consistently found when using this five-step approach in practice, that 60-70% of activities in the plan are ‘no cost’. For example, establishing a new committee, drafting protocols and strategies, or providing in-house training.

Five step pathway for designing health research capacity strengthening programmes

1. Define the goal of the capacity strengthening project.

This necessitates harmonising the expectations and objectives of the most critical stakeholders including developing country partners, people involved in ensuring sustainability of the activities in the long term and the funding organisation.

2. Describe the required capacity needed to achieve the goal.

This will require a search for the best evidence to describe the ‘optimal’ capacity collated from, for example, peer-reviewed published papers or expert groups, including evidence from outside the health sector.

3. Determine the existing capacity and identify any gaps compared to the required capacity.

The evidence from step two is formatted into a set of qualitative and quantitative data collection tools to identify existing capacity and capacity gaps. Data is collected from stakeholders with different perspectives; discrepancies are highlighted and resolved through further discussion.

4. Devise and implement an action plan to fill the gaps.

The prioritised list of capacity gaps is transformed into an action plan which includes objectives, activities, deliverables and monitoring indicators, and measures to facilitate sustainability

5. Learn through doing; adapt the plan and indicators regularly.

Results from experimentation and learning, and regular discussions with those responsible for monitoring progress are used to refine the plan. Progress indicators become more sophisticated as the programme matures and capacity is strengthened

10 year perspective for designing RCS programmes to enable measurements of progress and impact and allow comparisons

In order to facilitate measurements of progress and impact of individual RCS programmes, and to allow comparisons between programmes, much more effort needs to be made to develop and use common principles, and if possible, a generic approach, for designing RCS programmes.

Investing in learning from good quality RCS evaluations

Understanding what indicators or metrics can be used for health RCS and their appropriateness, or not, in different contexts, is critical to monitor the performance of RCS programmes, to measure achievement and to demonstrate accountability. Ideally RCS funders would like a few common measurable, reliable indicators for measuring RCS but they also need to demonstrate project impact on, for example, policies and poverty, for which direct attribution is difficult to prove. Our research has shown that the majority of evaluations of health RCS meet very few of the generally accepted quality standards. Most of the evaluations we have reviewed were retrospective so they lack comparative baseline data, and the purpose and indicators for outcomes and impact were often not stipulated.

We were unable to find information in the published literature about indicators for RCS that were SMART (Specific, Measurable, Attainable, Realistic and Time Bound). We therefore collaborated with a group of global health research funders and collated all the indicators we could find in their evaluation reports of RCS evaluations[6]. We extracted the indicators into a matrix across individual, institutional and national/international levels. Critical to making sense of RCS outcomes is the need to be explicit about the pathway by which change is to be brought about (i.e. the theory of change) and to use indicators that reflect different stages on the pathway. We therefore synthesized indicators across potential impact pathways (activities to outputs to outcomes) and iteratively verified our findings with key health RCS evaluation stakeholders. The indicators we identified were primarily of activities, outputs or outcomes with a strong bias towards the more easy-to-measure individual level indicators. There was very little information on the inter-relationships along a ‘change pathway’ so measurements of progress in developing research capacity over time were scarce.

Define the underlying ‘pathway to change’

RCS is a complex process with a long intervention pathway and examples of good quality RCS evaluations are hard to find. Such evaluations involve theory-driven evaluative thinking, identification of underlying assumptions and an explicit rationale for any actions[7]. It is important to consider long-term impact across different levels throughout the whole project cycle using clear conceptual frameworks, multiple data sources and valid standards to enhance quality.

RCS evaluations should incorporate a theory of change as this helps to identify impact trajectories, strengthen evaluation rigour, foster assessment of generalisability to other contexts, and guide influence on policy and practice. However, most evaluations currently do not have time and resources to incorporate theory-informed indicators of impact and sustainability, or to collect data against these indicators. This means opportunities to enhance knowledge and learning among funders and funding recipients about how to improve planning, monitoring and evaluation of health RCS initiatives are being missed[8].

Sharing lessons among research funders

Some funder groups (e.g. UK Collaborative on Development Sciences, the ESSENCE on Health Research Initiative) are now coming together to share experiences and learn from health RCS initiatives but they are hampered by a lack of common approaches and of systematic methods for documenting RCS. Ultimately, timely sharing information about RCS will help to avoid repeating mistakes, and enhances knowledge about how to do better health RCS. Our research highlighted some key challenges concerning learning from RCS efforts that will need to be addressed in future. Health RCS activities are often ‘bolted on’ to research projects but expectations that researchers themselves will spend a significant amount of time documenting and using learning to improve the RCS activities at the same time as delivering their primary research, may be unrealistic. Research funders face a trade-off between their need to show accountability and value for money, and the extent to which they should invest in facilitating and sharing learning.

Engagement of stakeholders in evaluations of RCS

Our research suggests that funding recipients and other stakeholders should be actively involved in all stages of the RCS evaluation process as this ‘empowerment’ makes RCS initiatives more likely to become self-sustaining[9]. It also helps to detect and correct problems early, makes the decisions underlying the evaluations more visible, and encourages sharing and use of the evaluation results. Stakeholders such as funding recipients have greater in-depth knowledge about the project and its context compared to external evaluators, which is vital for solving problems and sustainability. However there is a tension between this approach, which takes time and needs an additional, ear-marked budget, and the traditional summative model of evaluation carried out by an independent external team, which many funders' perceive as better value for money, for demonstrating accountability and for producing rapid results[10].

10 year perspective on using learning from research to design more effective RCS programmes

10 year perspective for measuring RCS progress and impact

Tools and methods for measuring RCS are in their infancy. There is huge scope for progress in this area over the next decade particularly in defining better indicator measurement properties and in developing indicators that encompass relationships with knowledge users. Indicators need to be more sophisticated so they can provide information not only on knowledge production and capacity development at individual level, but also on changes in health system policies, programs and practices. Greater attention to evaluation design, prospective indicator measurement, and systematic linkage of indicators in keeping with theories of change could provide more robust evidence on outcomes of health RCS. The impact and expected outcomes of RCS programmes should be defined. A paradigm shift will be needed to accept that the focus should be on assessing the ‘contribution’ of a programme, to this impact goal rather than trying to directly attribute impact to an individual programme.

In future each RCS project should develop comprehensive, prospective system for health RCS evaluation. Donors and policy makers could develop supporting guidance, tools or training for leading or participating in health RCS evaluations. Adequate funding will need to be committed to ensure high quality and meaningful evaluations of health RCS projects. To avoid the tensions faced by researchers and funders in prioritizing learning from RCS efforts, specialized evaluation teams could work closely with the researchers and funders, to take responsibility for the learning aspects of evaluations (i.e. to undertake ‘developmental’ evaluations).

How to minimize future tensions in designing and measuring RCS efforts

Funders of health RCS initiatives and those involved in determining policies concerning RCS can play a key role in minimizing tensions that threaten the effectiveness of RCS efforts. They should:

1. Promote a critical debate on the tensions inherent in evaluation of health RCS.

2. Develop supporting guidance, tools and training for leading and participating in health RCS evaluations.

3.Allocate adequate funding to the evaluation of health RCS projects.

4. Systematically document lessons learned in M&E of health RCS projects and engage funders, implementers and evaluators in an effective learning process.

5. Develop a community of practice to share lessons and experiences and consider joint ways forward.