TPI Working Paper No. 1/2008
Current practice in the evaluation of cross-sector
partnerships for sustainable development
RAFAL SERAFIN1, DARIAN STIBBE
The Partnering Initiative (TPI)
CLAUDIA BUSTAMANTE and CAROLIN SCHRAMM
London School of Economics
November 2008
1
Please direct correspondence to:
Rafal Serafin, Development Director, The Partnering Initiative
International Business Leaders Forum, 15-16 Cornwall Terrace, Regent’s Park, London NW1 4QP, England UK.
E-mail: rafal.serafin@iblf.org
1
ABOUT THE PARTNERING INITIATIVE (TPI)
The Partnering Initiative (TPI) is concerned with developing the art and science of cross-sector partnerships. TPI
works with individuals, organisations, and systems to promote and improve the effectiveness of partnerships for
sustainable development - between business, government and civil society
TPI is a global programme of the International Business Leaders Forum (IBLF) operated in association with the
University of Cambridge Programme for Industry, Environmental Partnership for Sustainable Development, the
UN Staff College and the Overseas Development Institute.
www.ThePartneringInitiative.org
ABOUT TPI WORKING PAPERS
TPI working papers are intended to make important ideas, information, research and experience of partnership
practitioners more widely available for discussion, verification and scrutiny.
TPI working papers present the work of partnership practitioners who want to contribute to the development of a
more effective global partnership movement.
The views, observations, conclusions and recommendations presented in TPI working papers are those of the
authors and are intended to inform and stimulate debate. They do not necessarily represent the views or formal
position of TPI, IBLF or their supporters and partners.
More information:
Rafal Serafin,
TPI Development Director
rafal.serafin@iblf.org
ABOUT THE IBLF
The Prince of Wales International Business Leaders Forum (IBLF) is a not-for-profit organisation established in
1990 to ‘put business at the heart of sustainable development’ by promoting responsible business leadership and
partnerships for sustainable development.
With a membership of over 80 international companies from around the world, IBLF works in more than 90
countries mobilising visionary leadership and engaging the capabilities of companies in creating innovative
partnerships and sustainable development solutions.
www.iblf.org
2
CONTENTS
SUMMARY .........................................................................................................................................4
INTRODUCTION ...............................................................................................................................5
RESEARCH GOALS & OBJECTIVES ...............................................................................................5
RESEARCH DESIGN ........................................................................................................................5
FINDINGS ..........................................................................................................................................6
CONCLUSIONS ...............................................................................................................................10
BIBLIOGRAPHY ..............................................................................................................................13
3
SUMMARY
Without a more systematic and comprehensive approach to evaluating cross-sector partnerships, further
investment in the development and implementation of partnership approaches is likely to be curtailed in coming
years. This is because the effectiveness and desirability of the cross-sector partnership approach is likely to be
judged not just in relation to the impact of well-designed and well-run partnerships, but on the results of a myriad
of initiatives and activities labelled as partnerships. Many of these underperform and do not achieve what is
expected of them because they are badly conceived or poorly implemented. This is why more emphasis needs
now to be placed on evaluation of the performance, effectiveness and impact of cross-sector partnerships.
With this motivation in mind, The Partnering Initiative (TPI) canvassed the views of partnership practitioners on
current practice through a combination of desk research, literature review, questionnaire surveys and face-toface interviews sought to answer the following questions:
1. DEFINITIONS: Is there a consensus on the terminology related to evaluating cross-sector partnerships?
KEY FINDING: There is no consensus both within and across the civil society, business and public sectors
as to definitions of what constitutes “evaluation” and what does not. A variety of terms are used including
evaluation, tracking, assessing, monitoring, reviewing with no consistency over the terminology used.
2. ASPECTS OF EVALUATION: Which aspects of partnering are considered to be the most important in
evaluations of cross-sector partnerships?
KEY FINDING: The focus on ‘producing tangible results’ or assessing impacts dominates current practice in
evaluating cross-sector partnership performance. More intangible or unexpected outcomes resulting from
cross-sector partnering are not well addressed and are often ignored altogether. Partnership performance is
seldom monitored and evaluated in relation to the potential advantages or benefits, which can be achieved.
3. PLANNING EVALUATION: In what ways do partnership practitioners plan to evaluate their partnerships and
what is the focus of such evaluations?
KEY FINDING: Few cross-sector partnerships are subjected to formal evaluation. Of these, only a minority
are evaluated in a systematic or comprehensive way in terms of their overall performance and impact.
Alternatives to partnership approaches are seldom considered in evaluations. Most partnerships are
evaluated from the perspective of one of the partners in relation to financial investment and related
reputation risks/benefits.
4. TOOLS: What tools are used for evaluating cross-sector partnerships?
KEY FINDING: Evaluations of cross-sector partnerships most commonly rely on the judgement of specialist
consultants, who make use of a wide range of specialized tools, frameworks, techniques and approaches.
There is no single most favoured or accepted tool, framework or approach. Evaluators opt for the evaluation
tools, which are most appropriate or relevant to meeting the needs, circumstances, purposes and
organizational culture of specific sectors. Frameworks and tools are typically selected by the agency, partner
or funding commissioning the evaluation.
5. IMPROVING EVALUATION: What are the most important barriers to improving evaluations of cross-sector
partnerships?
KEY FINDING: The most frequently cited barrier to undertaking evaluations of cross-sector partnerships
relates to securing adequate resources. The availability of resources is closely related to the way evaluations
are organized and carried out. In other words, who decides on their scope, who funds them, who carries
them out and who uses and interprets the results are crucial questions that must be dealt with by the
partners working together in a cross-sector partnership. A key aspiration for partnership practitioners relates
to finding ways of designing evaluations of cross-partnerships as a whole in ways, which draw on or include
all partners, as well as those who have been affected by the activities of the partnership. The aspiration in
this regard is to ensure evaluation results contribute to improving partnership performance and impact.
Partnership practitioners are increasingly interested in access to methods and approaches focused on evaluating
the totality of partnership performance, benefit and impact. Such evaluations need to be distinguished from
evaluations undertaken from the perspective of a funder or a single partner, which will continue to be favoured by
those wishing to extract more value from their investments of time, money and reputation. But single-partner
oriented evaluations will not capture the degree to which a cross-sector partnership is realizing its promise and
potential. The priority for research and development is to develop more effective tools, methods, frameworks and
approaches for evaluating the totality of performance, benefit and impact of cross-sector partnerships.
4
INTRODUCTION
The Partnering Initiative (TPI) has been working for more than a decade with partnership practitioners from the
business, public and civil society sectors from around the world to increase the use and improve the
effectiveness of cross-sector partnerships for sustainable development. Focused on the practical end of
designing, implementing and evaluating partnerships, TPI has adopted the following definition of cross-sector
partnerships to guide its work:
“Cross-sector partnerships are voluntary collaborations between organisations from different sectors
(business, not for profit, government, academia, media) where each shares benefits, costs and risks with
others to achieve a jointly defined sustainable development goal.”
Since the World Summit on Sustainable Development (WSSD) held in Johannesburg in 2002, cross-sector
partnerships have become increasingly popular. They have been adopted by many governments, businesses,
civil society groups and international agencies as a mechanism or means for dealing with environmental,
economic and social development challenges. The growing world-wide commitment to cross-sector partnership
has been driven in part by the assumption that partnerships are self-evidently a ‘good thing’ and therefore should
be encouraged.2 Although this attitude has served to promote cross-sector collaboration, the potential and actual
impacts of such collaborations in terms of delivering tangible results have seldom been subjected to scrutiny.3
If cross-sector partnerships are to achieve their promise in terms of delivering on sustainable development, more
emphasis needs now to be placed on evaluation of their performance, effectiveness and impact. With this
motivation in mind, TPI invited a group of students from the London School of Economics (LSE) 4 to work with the
TPI team to research what constitutes current practice with evaluation of cross-sector partnership for sustainable
development.
This paper reports the results of research undertaken in the spring and summer of 2008 and discusses its
implications for ongoing TPI research and development on tools for improving the effectiveness of cross-sector
partnerships in delivering sustainable development – work still very much in progress.
RESEARCH GOALS & OBJECTIVES
The goal of the project was to assess the ‘how and what’ of what concerns partnership practitioners in evaluating
the cross-sector partnerships in which they are involved. The motivation was to identify the ingredients of a
successful partnership evaluation and to identify priorities for further research and development of tools for
evaluating cross-sector partnerships.
By canvassing the views of partnership practitioners, the project sought to answer the following questions:
1. DEFINITIONS: Is there a consensus on the terminology related to evaluating cross-sector partnerships?
2. ASPECTS OF EVALUATION: Which aspects of partnering are considered to be the most important in
evaluations of cross-sector partnerships?
3. PLANNING EVALUATION: In what ways do partnership practitioners plan to evaluate their partnerships
and what is the focus of such evaluations?
4. TOOLS: What tools are used for evaluating cross-sector partnerships?
5. IMPROVING EVALUATION: What are the most important barriers to improving evaluations of crosssector partnerships?
RESEARCH DESIGN
The project was designed as an initial exploration of the views of partnership practitioners related to evaluation of
cross-sector partnerships and involved:
•
A literature review;
•
An on-line web-based questionnaire survey;
2
See for example, thepartnershipdeclaration.org
There is now growing interest in developing theory of cross-sector partnership. One recent publication in this regard is: Glasbergen et al.
(2007), which is listed in the bibliography. The post-graduate course on Cross-Sector Partnerships for Sustainable Development convened by
the University of Cambridge and The Partnering Initiative of The International Business Leaders Forum, which is now in its 7th year has
generated considerable research aimed at linking theory and practice of cross-sector partnership. See www.thepartneringinitiative.org
3
4
Claudia Bustamante, Hongyi Liu, Carolin Schramm, Yinjun Wang.
5
•
Personal face-to-face interviews, and;
•
Review of TPI partnership evaluations and related documents.
As the study was primarily concerned with identifying issues, trends and ideas for more detailed consideration,
no sampling was involved in the questionnaire survey nor in organizing the face-to-face interviews. This means
that the survey results should not be interpreted in terms of their statistical significance and should not be seen
as representative of all partnership practitioners.
The project canvassed the views of partnership practitioners associated with the TPI through the Partnership
Brokers Accreditation Scheme (PBAS), the University of Cambridge Post-Graduate Course on Cross-Sector
Partnership (PCCP), UN Staff College partnership training5, as well as partnership practitioners from selected
organisations from the public, private and civil society sectors which have made a public commitment to using
cross-sector partnership approaches.
The on-line survey was sent to the 1500+ partnership practitioners associated with TPI who receive TPI mailings,
as well as to organizations and individuals who have declared an interest in the study through personal
connections of TPI staff and the LSE student research team. As a result, it is not possible to estimate the total
number of partnership practitioners who received the survey.
A total of 72 individuals completed the on-line survey. Their
responses were analyzed and are presented in this paper. In
addition, 12 face-to-face interviews were completed with
partnership practitioners in organisations who have used crosssector partnerships for sustainable development to deliver on their
mission.6 Their views have been taken into account here.
To gain an understanding of the differences and similarities in
approaches to partnership evaluation used by practitioners working
in different sectors, sector affiliation was also taken into account in
analysing responses. Figure 1 shows the range of respondents of
the survey by sector affiliation. The majority of questionnaire
respondents were from civil society organisations (37%) and
business (28%).
Figure 1: Sector Affiliations
Governement
14%
Academia/Research
8%
Independent/Others
6%
NGO/Not-for-profit
37%
Business
28%
UN/Multi-lateral
Development Agency
7%
The survey and interview research for the project was undertaken in the winter and spring of 2008 with follow-up
work in the summer of 2008.
FINDINGS
The findings of the research reported here are presented in relation to the five questions posed at the outset.
DEFINITIONS
Partnership practitioners participating in the study were first asked if they saw a difference between the terms
“review” and “evaluation” and secondly to report on other terms which they felt were appropriate for them.
Partnership practitioners responses revealed that:
•
there is no consensus about definitions relating to evaluating cross-sector partnerships;
•
there is a lack of consensus as to definitions within sectors but the differences are even more
prevalent across sectors;
•
the terms “review” and “evaluation” are typically used interchangeably. In many instances,
“reviewing” was used to refer to understanding the ongoing dynamic or performance of a crosssector partnership, whereas “evaluation” was used to refer to a retrospective look at a cross-sector
partnership once it had been terminated or had reached some watershed or climax. Another
5
Partnership practitioners associated with TPI are partnership brokers accredited under the Partnership Brokers Accreditation Scheme
(PBAS), graduates of the University of Cambridge Post-Graduate Course on Cross-Sector Partnerships (PCCP) and the UN Staff College
partnership training, as well as other partnership practitioners actively cooperating with TPI and IBLF. See www.thepartneringinitiative.org
6
Partnership practitioners were included also from Oxfam GB, Microsoft, Plan International, The Catholic Agency for Overseas Development
(CAFOD), The International Institute for Environment and Development (IIED), One World Action, CARE UK, Conservation Leadership
Program (CLP), Barkers Global, China Dialogue, The BBC World Service Trust China Projects Office, NGO Management Group/NGO
Management School.
6
distinction encountered was that “reviewing” related often to specific phases of a cross-sector
partnership, whereas “evaluation” referred to the totality of a cross-sector partnership.
•
the most common distinction (36% respondents) between review and evaluation was on the level of
formality:
o
Review: a more informal process, usually done internally;
o
Evaluation: a more formal process, often done in collaboration with an external consultant;
•
other terms with reference to evaluating cross-sector partnerships included: monitoring (17%
respondents), assessment (17% respondents) and impact assessment (8% respondents);
•
other suggested terms were: benchmarking, diagnosis, tracking and analysis.
KEY RESEARCH FINDING ON DEFINITIONS
There is no consensus both within and across the civil society, business and public sectors as to definitions of
what constitutes “evaluation” and what does not. A variety of terms are used including evaluation, tracking,
assessing, monitoring, reviewing with no consistency over the terminology used.
ASPECTS OF EVALUATION
TPI experience7 with evaluating the benefits, performance and impact of cross-sector partnerships suggests that
there are at least four key aspects or elements to consider in any evaluation:
1. the quality and effectiveness of the way a partnership is organized, including the quality of relationships
between the partners involved (Operations);
2. impact or progress towards achieving anticipated sustainable outcomes (Impacts);
3. added value of using the partnership approach as compared to alternative approaches (Alternatives);
4. benefits accruing to partners in line with expectations (Benefits).
When asked about the most important aspects or features in evaluation, respondents reported the following:
•
Partnership’s Impact: Respondents from all sectors identified the achievement of the project or
activity goals as the most important aspect to be assessed and most already have some form of
measuring this for their specific projects.
•
Partner Relationship and Operations: The evaluation of the partnership relationship was seen as the
second most important aspect to evaluate but most partnerships did not have formal process for this
aspect.
•
Partnership as a Mechanism: Responses revealed that measuring whether partnership was the best
available option to achieve the objectives compared with non-partnering alter-natives was considered
to be the least important aspect. This aspect was seldom considered in evaluations.
KEY RESEARCH FINDING ON ASPECTS OF EFFECTIVE EVALUATIONS
Impacts, operations and alternatives are all perceived as important aspects or features of effective partnership
evaluation by the majority of partnership practitioners from across all sectors. But the focus on ‘producing tangible
results’ or assessing impacts dominates current practice in evaluating cross-sector partnership performance. More
intangible or unexpected outcomes resulting from cross-sector partnering are not well addressed and are often
ignored altogether, despite the fact that the added-value of partnership is often cited as a reason for choosing the
mechanism.
PLANNING EVALUATION
There appears to be no universally accepted approach to planning evaluations of cross-sector partnership and
evaluations are typically not planned at the outset. When it comes to actually carrying out evaluations of cross7
TPI has carried out over 50 evaluations of cross-sector partnerships in a range of geographical, cultural , thematic and institutional
contexts.
7
sector partnerships, only 20% of respondents reported that they are evaluating or planning to evaluate the
partnerships in which they are involved. In part, this is because only a minority of partnership agreements include
a specific provision and budget item for evaluation. Indeed, several respondents pointed out that evaluation of
their partnerships was important, but that the main barrier to undertaking evaluation was the lack of sufficient
financial resources. This is consistent with the findings of a recent survey of UN CSD-registered partnerships,
which found that few partnerships had conducted an evaluation of their programmes and fewer than one in four
had put aside resources specifically for evaluation8.
In many cases, evaluations were seen as something to be undertaken informally within the framework and
activities of the partnership. Nonetheless, more than half (58%) of respondents were of the opinion that a formal
evaluation is a preferred option and should be undertaken with an external and independent consultant where
possible. A frequent caveat was that the way an evaluation is carried out has implications for the development
and future performance of the cross-sector partnership.
Who decides on evaluating the cross-sector partnership, with what purpose and on what basis are crucial
considerations. In the words of one respondent:
“Evaluation, like the partnership itself, requires equity. Therefore evaluation should not be lead by one
partner alone but jointly by all partners. The best idea would be a shared leadership model.”
In practice, most evaluations of cross-sector partnerships are not planned or budgeted for at the outset. When
their organizations do plan for evaluation, respondents reported that the focus is primarily on evaluating results,
impacts or outcomes in terms of benefits resulting from the organization’s engagement, involvement or
investment (i.e. the ‘business case’).
Some of the more experienced partnership practitioners did note, however, that their organizations are
increasingly interested in considering also the performance of the cross-sector partnership as a whole as
opposed to focusing just on benefits accruing to the individual organizations involved in the partnership. But such
considerations are still typically focused costs and related reputation risks/benefits.
No-one reported planning evaluations of cross-sector partnerships in relation to non-partnering alternatives.
Surprisingly, only a third of respondents saw evaluation of the cross-sector partnerships in which they are
involved in as a priority for the future
KEY RESEARCH FINDING ON PLANNING EVALUATIONS
Few cross-sector partnerships are subjected to some form of formal evaluation. Of these, only a minority are
evaluated in a systematic or comprehensive way in terms of their overall performance and impact. Alternatives to
partnership approaches are seldom considered in evaluations. Most partnerships are evaluated from the perspective
of one of the partners in relation to financial investment and related reputation risks/benefits. When it comes to
actually carrying out evaluations, the preferred option is for using external and independent consultants.
TOOLS
Evaluations of cross-sector partnerships are most typically undertaken in an informal way. Approaches relying on
the judgement of a specialist consultant were the most common. Although partnership practitioners refer to a
wide range of specific tools, techniques and approaches, the research revealed that there is no single most
favoured or accepted tool or approach.
There have been few tools or guidance developed for practitioners specifically for evaluating partnerships.
Existing tools are typically focussed on one particular aspect of a partnership (for example governance Partnership Governance and Accountability tool developed by Accountability and the many tools for evaluating
financial performance). The bibliography contains references to those cited or referred to by partnership
practitioners contributing to the research.
Preferences as to the application of specific tools and approaches for evaluating cross-sector partnership varied
in relation to sector affiliation. More specifically, the research found that:
•
UN/multilateral development agencies mainly use tools designed by similar agencies, such as the
United Nations Development Programme (UNDP) or the International Organisation for Migration (IOM).
8
Survey of UN CSD-registered partnerships with an environmental flavour conducted by The Partnering Initiative on behalf of the
OECD. See OECD (2006).
8
•
NGOs indicate in many cases that they rely on universities or other academic institutions to undertake
evaluations. They favour evaluation tools and approaches, which incorporate stakeholders/
beneficiaries.
•
In the business sector, the preference is for evaluation tools which are already familiar and widely used
in the business sector, such as Return on Investment (ROI), SWOT analysis and Present Value
Objective (PVO). Practitioners from the business sector also reported that their evaluations usually
focused on progress against predetermined objectives. Some had developed their own customized
frameworks and approaches consistent with the organizational culture of the business.
The differing preferences for evaluation tools and approaches might be explained by the fact that each sector
has different circumstances, interests, motivations, expectations and needs when it comes to evaluation. The
differences might also attributed to the fact that each sector differs in organisational culture, work practices,
resources and needs. As a result, partnership practitioners opt for the evaluation tools, which are most
appropriate or relevant to meeting the needs, circumstances and purposes of specific sectors.
The value of undertaking an evaluation of the totality of a cross-sector partnership is seldom seen as a priority
and few tools were seen as being available for such evaluations.
In terms of priorities for the future, respondents emphasized the need for improved evaluation tools and
approaches, which recognize the uniqueness of each and every partnership, as well as the differing needs,
circumstances and organizational culture of the partners involved. Tools are needed for evaluating a cross-sector
partnership in terms of:
•
the impact on beneficiaries;
•
the impact on partners (what did they loss and gain by participating in the partnership);
•
unexpected outcomes (both positive and negative);
•
costs and benefits achieved.
Although much needed, development of a universally-accepted tool or approach for evaluating cross-sector
partnerships is likely to be difficult since cross-sector partnerships and the partners involved in them differ so
widely in their focus, needs, circumstances and organizational culture.
KEY RESEARCH FINDING ON TOOLS
Evaluations of cross-sector partnerships most commonly rely on the judgement of specialist consultants, who make
use of a wide range of specific tools, techniques and approaches. There is no single most favoured or accepted tool
or approach for conducting evaluations of cross-sector partnerships. Evaluators opt for the evaluation tools, which
are most appropriate or relevant to meeting the needs, circumstances, purposes and organizational culture of
specific sectors. Tools and approaches are needed for evaluating cross-sector partnerships in their totality, as well
as in terms of the needs of individual partners.
IMPROVING EVALUATION
According to partnership practitioners, the most frequent barrier to undertaking evaluations of cross-sector
partnerships is the availability of adequate resources. The availability of resources is closely related to the way
evaluations are organized and carried out. This means who decides on their scope, who funds them, who carries
them out and who uses and interprets the results are crucial questions for the partners working together in a
cross-sector partnership.
A key aspiration for partnership practitioners relates to finding ways of designing evaluations in ways, which draw
on or include all partners, as well as those who have been affected by the activities of the partnership. The
challenge relates here to dealing with the fact that the more comprehensive and complex an evaluation, the more
time-consuming and costly it becomes. This is because the needs for monitoring and evaluation data increase
exponentially in ever more comprehensive evaluations. The practical implication is that the more complex and
comprehensive an evaluation, the less likely it is to be carried out.
If evaluations are to become more frequent in partnership practice, they must be thought not as a means of
searching for or admitting mistakes, but as interventions aimed at bringing about improvements in performance
of a partnership and increasing its impact. For partnership practitioners this means finding practical ways of
incorporating evaluation in the ongoing activities of a cross-sector partnership and undertaking evaluations in all
stages of the partnership cycle.
9
KEY RESEARCH FINDING ON IMPROVING EVALUATION
The most frequent barrier to undertaking evaluations of cross-sector partnerships relates to securing adequate
resources. The availability of resources is closely related to the way evaluations are organized and carried out. In
other words, who decides on their scope, who funds them, who carries them out and who uses and interprets the
results are crucial questions for the partners working together in a cross-sector partnership. A key aspiration for
partnership practitioners relates to finding ways of designing evaluations of cross-partnerships as a whole in ways,
which draw on or include all partners, as well as those who have been affected by the activities of the partnership.
The challenge relates here to dealing with the fact that the more comprehensive and complex an evaluation, the
more time-consuming and costly it becomes. Evaluations need to be incorporated – where possible – into the
ongoing activities of a cross-sector partnership.
CONCLUSIONS
Despite the growing popularity of cross-sector partnership approaches among business, civil society and public
sector organizations, few appear to have been evaluated in relation to their overall impact and benefit. Most
evaluations centre on the perspective of a single partner or funder and the degree to which narrowly-defined
project objectives are achieved.
Partnership operations, comparison to alternative non-partnering approaches and additional benefits and addedvalue partnerships bring those involved and affected by them are seldom dealt with in a systematic or
comprehensive way. This means that the effectiveness and desirability of the cross-sector partnership approach
is likely to be judged not just in relation to the impact of well-designed and well-run partnerships, but on the
results of a myriad of initiatives and activities labelled as partnerships. Many of these underperform and do not
achieve what is expected of them because they are badly conceived or poorly implemented. Without a more
systematic and comprehensive approach to evaluating cross-sector partnerships, further investment in the
development and implementation of partnership approaches is likely to be curtailed in coming years.
For these reasons, there is a growing interest among partnership practitioners and partnership advocates in
undertaking evaluations of the totality of partnership performance, benefit and impact. Such evaluations need to
be distinguished from evaluations undertaken from the perspective of a funder or a single partner. This is not to
say that single partner or funder perspectives should not be pursued. On the contrary, they should and no doubt
will be pursued with increased vigour as those investing in cross-sector partnerships will seek to extract more
value from their investments of time, money and reputation. The point here is simply that such evaluations will
not capture the degree to which a cross-sector partnership is realizing its full potential. The priority should be to
ensure that the two categories or types of evaluation are not confused. The two figures, which follow summarize
the key features of evaluation by “Partnership as a Whole” from evaluation by an individual partner.
To be of practical use to partnership practitioners, frameworks, tools and approaches for evaluating cross-sector
partnerships need to be able to accommodate many different types of partnerships, as well as the many different
needs, circumstances and motivations of organisations participating in them. They must offer the means for all
those involved in and affected by a cross-sector partnership an opportunity to gain an appreciation for the full
impact and benefits of a cross-sector partnership, not simply its stated objectives. This means systematically
asking systematically and consistently throughout all the phases of the partnering cycle:
•
Goals/objectives: Is the key problem or challenge being addressed in the right way? Is the right problem
being addressed?
•
Operations/implementation: Is the partnership operating to its potential? Are things being done right?
How can they be done better?
•
Learning/improving: How can we use experience gained to extract more benefit for partners, improve
performance and increase impact?
Posing such questions can provide the basis for the development of universally applicable or generic tools for
evaluating cross-sector partnerships. These are desirable as they will allow partnership practitioners and
partnership advocates to learn more systematically from different applications and experiences of organizing
cross-sector partnerships. The development of such tools will, however, require greater consensus among
partnership practitioners and funders as to terminology and concepts related to evaluation. Such consensus is
needed both within and across the civil society, business and public sectors.
10
Evaluation by Partnership as Whole vs Evaluation by an Individual Partner
Evaluation by Partnership as a Whole
ANALYSIS OF PARTNERSHIP ITSELF
Evaluation by an Individual Partner
COST BENEFIT/ANALYSIS
COSTS
PARTNER ASPECTS
Staff costs
Other in-kind resource
contributions
Financial contributions
Degree of institutional
buy-in
Negative impact
Comparison with non-partnering alternatives
(opportunity cost)
11
Degree of internal
learning / change
Degree of knowledge
capture
Delivery of
commitments
Based on the research undertaken in this project, improving the practice of evaluation will require partnership
practitioners to:
1. advocate carrying out formal evaluations of cross-sector partnerships as interventions aimed at
improving their performance and helping those involved in them realize their promise and potential.
2. build in evaluation into activities undertaken in all stages of the partnership cycle, ensuring where
possible that a provision is made for evaluation in the early formative stages of a cross-sector
partnership.
3. distinguish between evaluations undertaken from the perspective of a single partner or funder from those
seeking to assess the performance, benefits and impact of the cross-sector partnership as a whole.
4. focus evaluations of cross-sector partnerships not just on their impacts or results, but also on their
design and operation, benefits to partners, unexpected consequences and value-added and
appropriateness or relevance in a particular situation as compared to other non-partnering approaches.
5. select tools and approaches for evaluating cross-sector partnerships, which take into account the needs,
circumstances, organizational culture and context of all the partners involved.
12
BIBLIOGRAPHY
DEFINITIONS
Brinkerhoff, J. M. 2002. “Assessing and improving partnership relationships and outcomes: a proposed
framework.’ Evaluation and Program Planning 25(3): 215-31.
Caplan, K. 2003. “The Purist’s Partnership: Debunking the Terminology of Partnerships” Partnership Matters:
Current Issues in Cross-sector Collaboration 1: 31-35
Caplan, K; Gomme, J; Mugabi, J and L. Stott. 2007. Assessing Partnership Performance: Understanding the
Drivers for Success. London: Building Partnerships for Development in Water and Sanitation (BPD).
Glasbergen, P. 2007. “Setting the scene: The partnership paradigm in the making.” In: Glasbergen, P.,
Biermann, F. and A. P. Mol (eds) Partnerships, governance and sustainable development. Reflections on theory
and practice. Cheltenham, UK: Edward Elgar, pp. 6-25.
Hansen, H. F. 2005. “Choosing Evaluation Models – a Discussion on Evaluation Design”. Evaluation 11: 447462.
Lasker, R. D., Weiss, E. S. And R. Miller. 2001. “Partnership Synergy: a practical framework for studying and
strengthening the collaborative advantage.” The Milbank Quarterly 79(2): 179 – 205.
Mattesich, P. W. And B. R. Monsey. 1992. Collaboration: What makes it work? St. Paul, Minn.: Amherst H.
Wilder Foundation.
Pawson, R. And N. Tilley. 1997. Realistic Evaluation. London: Sage.
Van Hujstee, M. M., M. Fancken and P. Leroy. 2007. Partnerships for sustainable development: a review of
current literature. Environmental Sciences. 4(2): 75-89.
ASPECTS OF EVALUATION
Clayson, Z.C.; Castaneda, X.; Sancheze. and C. Brinedis. 2002. “Unequal Power—Changing Landscapes:
Negotiations between Evaluation Stakeholders in Latino Communities. Evaluation Stakeholders in Latino
Communities” American Journal of Evaluation 23: 33-44.
Hirsch, B. And M. Meyer. 2005. “Accounting for trust in cooperations.” In: Gossling, T, Jansenn, R. J. And L. A.
Oerlemans (eds) Coalitions and collisions. Nijmegen: Wolf Publishers, pp. 421-432.
IBLF. 2006a. Facing the Facts, realities of Partnering Practice: Survey of Views and Experiences of Frontline
Partnership Practitioners. London: The Partnering Initiative, IBLF.
IBLF. 2006b. Making a Difference, Exploring Issues in partnering Practice: Rio Tinto: Tackling the Cross-sector
Partnerships Challenge. London: The Partnering Initiative, IBLF.
Lorenzen, K. 2006. “Enhancing the ‘Value Add’ in the Partnership Review Process.” Partnership Matters, Current
Issues in Cross-sector Collaboration 4: 31-33.
OECD. 2006. Evaluating effectiveness and efficiency of partnerships involving governments from OECD
countries. ENV/EPOC (2006) 15/REV1. Paris: OECD.
Rochin, S., Zadek, S. and M. Forstater 2008. Governing collaboration: making partnerships accountable for
delivering development. London: Accountability (www.accountability21.net)
Watkins, F. and S. Watson. 2005. Evaluation of the DFID / CAFOD Partnership Programme Agreement - 20012005. www.dfid.gov.uk/aboutDFID/DFIDwork/ppas/cafod-evaluation04.pdf (accessed April 23, 2008).
PLANNING EVALUATION
American Evaluation Association. 2001. The Program Evaluation Standards. Summary f the Standards: Utility
Standards. www.eval.org/evaluationdocuments/progeval.html
Backer, T. E. 1999. Innovation in Context: new foundation approaches to evaluation, collaboration and best
practices. Encino, California: Human Interaction Institute.
Binkerhoff, J. M. 2002. “Assessing and improving partnership relationships and outcomes: a proposed
framework.” Evaluation Programme Planning 25: 215 – 231.
13
Francisco, V. T., A. L. Paine and S. B. Fawcett. 1993. “A Methodology for Monitoring and Evaluating Community
Health Coalitions.” Health Education Research 8: 403-416.
Gujit, I. And J. Woodhill. 2002. Managing for impact in rural development – a guide for project monitoring and
evaluation. Rome, Italy: International Fund for Agricultural Development.
Hennessy, M. 1995. “What works in program evaluation.” Evaluation practice 16(3): 331–344.
Kosak, J. Z. And R. C. Rist. 2004. Ten steps to a results-based monitoring and evaluation system. Washington,
D. C.: The World Bank.
Kreuter, M. W., N. A. Lezin and L. A. Young. 2000. “Evaluating community-based collaborative mechanisms:
implications for practitioners.” Health Promotion Practice. 2: 49-63.
OECD 1991. Principles for evaluation of development assistance. Paris, France: OECD.
Office of the Chief Researcher Scottish Executive. 2002. Partnership Working: Key Issues Around Evaluation.
http://www.scotland.gov.uk/Publications/2004/05/19378/37361 (accessed April 23, 2008).
Rossi, P. H. and H. Freeman 1985. Evaluation: a systematic approach. Beverley Hills, California: Sage.
Scriven, M. 1996. “Types of evaluation and evaluator.” Evaluation Practice 17(2): 151-161.
Taylor-Powell, E., B. Rossing and J. Geran. 1998. Evaluating Collaboratives: reaching the potential. Madison,
WI: University of Wisconsin-Cooperative Extension.
Wagner, D. A., Day, B., James, T., Kozma, R. B., Miller, J. and T. Unwin. 2005. Monitoring and evaluation of ICT
in education projects: a handbook for developing countries. Washington, D.C.: InfoDev/World Bank.
TOOLS
Caplan, K and D. Jones. 2002. “Partnership Indicators: Measuring the Effectiveness of Multi-sector Approaches
to Service Provision”. Practitioner Note Series - BPD Water and Sanitation Cluster, London, July 2002.
Center for the Advancement of Collaborative Strategies in Health. 2006. The Partnership Self-Assessment Tool:
2006-2007. http://www.cacsh.org/psat.html (accessed April 23, 2008).
Chmielewski, T. L. and J. J. Phillips. 2002. “Measuring Return-on-investment in Government: Issues and
Procedures”
Public
Personnel
Management,
http://www.entrepreneur.com/tradejournals/article/print/160542375.html (accessed April 23, 2008).
Earl, S.; Carden, F. and T. Smutylo. 2001. Outcome Mapping: Building Learning and Reflection into
Development Programs. Canada: International Development Research Centre (IDRC).
Employers Organization for Local Government 2008. Smarter Partnerships. Tool kit developed by EDuce Ltd.
http://www.lgpartnerships.com/digging.asp (accessed April 23, 2008).
Fetterman, D., Kaftarian, S. J. And A. Wandersman. 1996. “Empowerment evaluation.” In: Knowledge and Tools
for Self Assessment and Accountability. California: Sage.
Field,
J.
2008.
Evaluation
Through
Storytelling.
The
Higher
Education
Academy.
http://www.heacademy.ac.uk/assets/York/documents/resources/resourcedatabase/id473_valuation_through_stor
ytelling.pdf. (accessed April 23, 2008).
Hersen, M. and A. S. Bellack (eds) 1981. Behavioral Assessment: a practical handbook. New York: Pergamon
Press (2nd ed).
Hardy B., Hudson, B., and E. W. Waddington 2000. What makes a good partnership? Partnership Assessment
Tool. Leeds: Nuffield Institute of Health.
Hardy, B., Hudson, B. And E. Waddington. 2003. Assessing Strategic Partnership: The Partnership Assessment
Tool. Strategic Partnering Task Force and the Nuffield Institute for Health. London: Office of the Deputy Prime
Minister, UK Government.
Independent Evaluation Group/World Bank. 2007. Sourcebook for evaluating global and regional partnership
programs: indicative principles and standards. Paris: OECD/DAC.
Melaville, A. I. 1997. A guide to selecting results and indicators: implementing results-based budgeting.
Washington, D. C.: Finance Project.
The Partnering Initiative. 2006. Case Study Project, Rationale: What are the ingredients of a good partnership?
http://thepartneringinitiative.org/mainpages/case/examples/RTCaseSt.pdf (accessed April 23, 2008).
14
Strategic Management 2007. SWOT Analysis. Quick MBA Knowledge to Power Your Business.
http://www.quickmba.com/strategy/swot (accessed April 23, 2008).
USAID. 2000. Inter-sectoral Partnering: Tools for Implementation & Evaluation. USAID Development Dialogue
Notes. Washington D.C.: US AID.
Waddock, S. A. and B. D. Bannister. 1991. “Correlates of effectiveness and partner satisfaction in social
partnersips”. Journal of Organizational Change Management 4: 64-79.
IMPROVING EVALUATION
Ansari, W. E., Phillips, C. J. And M. Hammick. 2001. “Collaboration and partnerships: developing the evidence
base” Health and Social Care in the Community 9(4): 215-227.
Biermann, F., Mol, A. P. and P. Glasbergen. 2007. “Conclusion: partnerships for sustainability – reflections on a
future research agenda.” In: Glasbergen, P., Biermann, F. and A. P. Mol (eds) Partnerships, governance and
sustainable development: reflections on theory and practice. Cheltenham, UK: Edward Elgar, pp. 288-299.
Liebenthal, A, Feinstein, O. N., and G. K. Ingram (eds) 2004. Evaluation and Development: the Partnership
Dimension. World Bank Series on Evaluation and Development, volume 6. Washington, D. C.: The World Bank.
Mitchell, J., Shankleman, J. and M. Warner. 2004. “Measuring the added value of partnerships” In: M. Warner
and R. Sullivan (eds) Putting Partnerships to Work: Strategic Alliances for Development between Government,
the Private Sector and Civil Society. Sheffield, UK: Greenleaf, pp. 191-200.
Selsky, J. W. and B. Parker 2005. “Cross-sector partnerships to address social issues: challenges to theory and
practice”. Journal of Management 31(6): 849-873.
15