SOCIAL WATCH DEVELOPING AN EVALUATION
the Social Watch Coordinating Committee and Reference Group
New York, 22 May 1999
Peter van Tuijl
Social Watch was established
in November 1995 to provide a global platform for NGOs to
monitor and promote the effective implementation of the important
commitments which were made by national governments during
the United Nations World Summit on Social Development (WSSD,
Copenhagen, March, 1995) and the Fourth World Conference for
Women (WCW, Beijing, September 1995). The founders of Social
Watch decided to plan for an initial time span of five years
to pursue the initiative, up and to the official high level
evaluation of the WSSD in the year 2000.
We are now four years
into Social Watch. The official five-year review of the WSSD
is coming up in June 2000 in Geneva, and is preceded by two
PrepComs in New York in May 1999 and April 2000. The moment
is approaching for Social Watch to take stock of the initiative,
to look back and learn, in support of an assessment of the
opportunities and constraints to improve the project, and
as an input for decision making on what to do after the first
The complexity of Social
Watch is enormous, the specific objectives of an evaluation
of Social Watch are far from evident and the available methodologies
for evaluating a transnational NGO advocacy effort like Social
Watch are at best still experimental. The process of formulating
a terms of reference for an evaluation of Social Watch is
thus an important issue in itself.
This document has been
written with the purpose to facilitate a systematic discussion
of different aspects of a Social Watch evaluation among a
number of key stakeholders. It is partly based on an informal
exchange of views which has taken place between the Social
Watch secretariat and Novib as of early this year. It should
not be seen as a draft terms of reference and has no other
intention than being a discussion paper. The document will
be presented in a meeting of the Social Watch coordinating
committee- and reference group on May 22, 1999, in New York.
The document has four
main sections. First, I have tried to outline some basic concepts
and questions, including a number of considerations why it
is difficult to evaluate Social Watch (section II). This is
not intended to discourage the reader, but rather to help
constructing more of a common understanding before entering
into a discussion on what we should evaluate and how we could
do it (sections III and IV). I then propose some ideas on
further developing the evaluation after our meeting in New
York (section V). I have also included a list of references
The document has been
written, assuming a general knowledge and understanding of
Social Watch, its history, context, structure and activities.
I hope it will help to support a fruitful discussion, leading
to a useful and effective evaluation of Social Watch.
Peter van Tuijl
SOCIAL WATCH: BASIC CONCEPTS AND QUESTIONS
Watch: measuring the impact of an NGO advocacy effort
An evaluation tries to
perceive change, and whether changes as identified match with
those intended. Social Watch has tried to provide a global
platform for NGOs to monitor and promote the implementation
of the official commitments made in Copenhagen and Beijing.
As such, Social Watch provides a platform for NGO advocacy.
Many definitions of NGO advocacy focus on NGOs’ efforts to
"influence policy" (usually public policy). However, this
leaves out the element of strengthening civil society and
changing power relations or democratization, which is often
at the heart of NGO advocacy.
If we refer to the first
brochure which came out of the founding workshop of Social
Watch in November 1995, it is clear that the intentions of
Social Watch were aimed both at influencing policy and at
strengthening civil society. One of the angles used to formulate
the objective of Social Watch in the brochure is "concentrating
the lobby capacity of NGOs to influence (local and national)
governments, donor agencies and international organisations
in order to realise policy changes on behalf of social development."
The definition of "social development" as used in the first
brochure contains three basic elements: a) the provision of
basic social services; b) income-generating and income-supporting
activities for the poor (especially women); and c) strengthening
social organisations. Finally, the three main qualitative
indicators for the kind of monitoring which Social Watch wished
to promote were identified as: a) a critical analysis of the
current national social policies; b) the political environment,
national and international; c) the involvement of civil society(1).
The approach of Social
Watch as outlined in 1995 fits with current thinking on the
definition of NGO advocacy and NGO advocacy impact assessment.
For the sake of brevity, we will not further elaborate these
conceptual discussions, but rather propose to use the following
overall framework for NGO advocacy and advocacy impact assessment
in the evaluation of Social Watch (2).
NGO advocacy is to
organise the strategic use of information to democratise
unequal power relations and to improve the condition of
those living in poverty or who are otherwise discriminated
NGO advocacy is multi-dimensional.
Three levels are distinguished for assessing the impact
of NGO advocacy.
1. Policy gains:
this include legislative and policy changes and their impact
(or the lack of it) on the lives of poor people;
2. Civil Society
gains: strengthening of NGOs and other civil society actors
in ways that help to hold governments and international
organisations accountable over the long run;
3. Political and
democratic gains: increased democratic space where NGOs
and other civil society actors can operate.
Social Watch is primarily
a platform of NGOs. To perceive of the functioning of NGOs
in a broader perspective of civil society is perhaps nowadays
more common than when Social Watch was created in 1995. We
therefore may want to add definitions of "NGOs" and "civil
society" to our basic framework. The latter in particular
to take distance from an understanding of civil society which
includes the private for-profit business sector.
Organisations (NGOs) are self-governing, private, not-for-profit
organisations that are geared toward improving the quality
of life of disadvantages people(3).
Civil Society is
a space or arena between households and the state which
affords possibilities of concerted action and self-organisation.
It describes and envisages a complex and dynamic ensemble
of NGOs and other organisations that tend to be non-violent,
self-organising, self-reflexive and permanently in tension
with each other and with the state institutions that "frame"
constrict and enable their activities (4).
If we can agree on this
approach, it leads us to three overall questions for an evaluation
of Social Watch:
1. Has Social Watch
contributed to legislative and/or policy changes in support
of the actual implementation of the commitments made in
Copenhagen and Beijing, leading to changes in the lives
of poor people?
2. Has Social Watch
strengthened NGOs and/or other civil society actors to hold
governments (local and national) and international organisations
accountable for the implementation of commitments made in
Copenhagen and Beijing?
3. Has Social Watch
contributed to opening-up political space for NGOs and/or
other civil society actors, in particular to raise issues
concerning social development and gender inequity?
is it difficult to evaluate Social Watch?
To establish an agreement
on the above conceptual framework is important but can only
be a starting point for developing an evaluation of Social
Watch. We need to further unpack the complexity of Social
Watch to arrive at operational objectives for an evaluation.
Below, we first try to list a number of factors explaining
why evaluating Social Watch is difficult.
- A broad geographical
scope: A principal claim of Social Watch is to be a
global initiative, that is to say, Social Watch is open
for participation by NGOs from any country in the world.
In reality, the participation in Social Watch has been stronger
in some countries/regions than in others and has also changed
over time. When reviewing Social Watch, it will not be easy
to fully grasp the broad scope of participants and activities,
and the effect of changing dynamics therein on the overall
impact of Social Watch.
- Bridging the South-North
divide: A particular aspect of Social Watch is that
it bridges the South-North gap. As a point of departure
governments in the South and in the North are equally monitored
and hold accountable for the implementation of the Copenhagen
and Beijing commitments. On the one hand, this equates the
relationships among Southern and Northern NGO participants
in Social Watch. On the other hand, the South-North divide
is not insignificant for Social Watch, because Northern
countries and Northern dominated international organisations
are playing a significant role in setting the agenda and
funding social development in the South. To use this Northern
leverage, for example by supporting the 20/20 compact, has
clearly been a part of Social Watch advocacy strategies.
Moreover, Social Watch itself has predominantly been funded
by Northern NGOs, above all by Novib as a key founder and
supporter. To capture the South-North dynamics in Social
Watch is thus a difficult question, which will require the
unraveling of processes at the crossroads of the relationships
between Social Watch participants themselves and their advocacy
strategies vis-á-vis their own governments and international
- Multiple level target
audiences: The idea of Social Watch starts from a level
of international commitments by governments, but great emphasis
has always been put on the need to promote national level
activities by Social Watch participants, primarily to ensure
national level implementation of the Copenhagen and Beijing
agreements. For the evaluation, it will be important to
understand the interaction between international level and
national level advocacy. Has the government of country x
acted in a certain way because of concerted international
and/or national-level pressure which can reasonably be related
to Social Watch? These dynamics may also be hard to establish.
It is already difficult to establish a reasonable insight
into what is happening at one level, but combining and cross-checking
assessments from different national and/or international
levels can be like trying to force pieces from different
puzzles into one frame. How do you reasonably connect the
observation that some good media work was done in the local
language press in a certain country with the action or inaction
of an UNDP official 10.000 miles away?
- Multiple advocacy
tools: In the context of Social Watch, different advocacy
strategies and tools have been used. The most outstanding
tool is obviously the annual Social Watch report, probably
the heart of the whole initiative. Yet, NGOs participating
in Social Watch have also written separate position papers,
done research, participated in various official meetings,
undertaken lobby meetings and written lobby letters, organised
public seminars and used the media in a variety of ways
to articulate Social Watch concerns. How to measure the
impact of different advocacy tools individually as well
as in relation to each other?
- Social Watch versus
other civil society advocates: The objectives of Social
Watch are shared by various civil society actors. NGOs not
participating in Social Watch, other social actors or networks
have in one way or the other monitored and promoted the
implementation of the Copenhagen and Beijing agreements,
sometimes in cooperation with Social Watch, sometimes not.
It will not always be evident to establish the contribution
of Social Watch versus the results of advocacy by other
civil society advocates. This is not only a question of
attribution, it also relates to the distinction made above
between policy gains and civil society gains. Has Social
Watch indeed managed to be a mobilizing force in civil society?
- How to move from
output to impact?: A regularly occurring weakness in
NGO advocacy impact assessment is too much emphasis on output
indicators, such as the number of press releases or newsletters
distributed, or the number of lobby meetings with officials
and whether they were high-up in the bureaucracy or not.
Although it may be relevant to establish these facts, impact
assessment will usually require one or more additional steps.
As in other NGO advocacy efforts, the creation and distribution
of information has been essential in Social Watch. The difficulty
is to establish what was done with it. What kind of indications
do we have that the Social Watch report was read? Can we
somehow see or hear the echo of information released by
Social Watch? In some cases, Social Watch lobby messages
could be delivered in official meetings. But has there been
any follow-up? Establishing high-level access is no guarantee
for high-level impact.
- The politics of
impact assessment: Impact assessment is not a value
free activity. This is true in general, but needs to be
emphasized for advocacy. Different stakeholders may each
have their own reasons to either exaggerate or downplay
the impact of advocacy related to Social Watch. An official
may say that NGOs have done a great job, while in reality
the person is trying to keep the NGOs in a marginal position
by making them believe they are successfully lobbying. The
reverse can also happen. The impact of NGO advocacy might
be ridiculed, precisely because it’s influence is felt and
to prevent it from becoming more successful. All parties
may suffer from a dubious self-perception. A recent study
on the political influence of NGOs on the Climate and Biodiversity
Conventions concludes that NGOs consistently tend to overestimate
the impact of their advocacy(5).
- Balancing different
results: A multi-level and complex NGO advocacy effort
as Social Watch is likely to achieve a mixed bag of results.
Approaches have to be found to balance gains in one country
with losses in another, or how to match intended with unintended
results or drawbacks. The purpose of advocacy impact assessment
is to learn and to improve future action. But the effectiveness
of NGO advocacy is hardly ever achieved through a standard
sequence of events. The causality in an advocacy effort
may look more like a pinball machine, with actions and reactions
constantly crisscrossing and influencing each other in different
ways. There is an issue of balancing different types of
results when constructing overall assessments. Last but
not least, when evaluating Social Watch, it has to be realized
that five years should be seen as a short period of time,
in view of the ambitious objectives of Social Watch.
SELECTING OBJECTIVES: WHAT SHOULD BE EVALUATED?
There are a great number
of angles which can be used to formulate different operational
objectives for an evaluation of Social Watch. The list we
propose here is certainly not exhaustive. While the list can
be expanded, priorities also need to be set. We have tried
to indicate what we see as possible main elements to be dealt
with in the evaluation. There is a rough order in the list
of suggestions, from broader to more specific questions or
subjects for the evaluation.
- Have the main assumptions
of Social Watch been correct?: Social Watch is based
on a few key assumptions, notably: a) the commitments made
in Copenhagen and Beijing are sufficiently taken serious
nationally and internationally to provide for a useful advocacy
pressure point, b) NGOs are legitimate, credible and capable
actors to take up this advocacy and c) information will
be the key tool to achieve results, notably by establishing
"social watch" as a self-standing, recognizable and credible
source of information. Have these assumptions proven to
- How has Social Watch
impacted on the strategic position of NGOs? This is
directly related to the overall issues of political space
and civil society as identified under II.A. If we look at
the discourse on the role of NGOs with respect to the United
Nations Conference on Sustainable Development (UNCED), there
are clear voices contending that the ultimate impact of
the NGOs’ involvement has merely resulted in NGOs resources
and potential influence being sucked into time-consuming
UN processes aimed at achieving corrective steps only, without
fundamentally challenging the established development paradigm
(6). Where has Social Watch left
its NGO participants on a scale from co-optation to opening-up
political space? Indicators to approach this question could
be: has Social Watch engaged the UN-agenda only or expanded
upon it?; has Social Watch managed to come up to the broad
agenda it spelled out itself in 1995?; and/or has Social
Watch supported legitimate voices on social development
which otherwise would not have been heard?
- Has Social Watch
influenced policy or legislative changes at international
levels? How visible and effective has Social Watch been
at the level of different international target audiences:
the UN, the European Union, the World Bank and the IMF etc.
(all of these are mentioned in the first Social Watch brochure).
What kind of access did Social Watch have, at what levels
and how consistent over the four-five year period? Have
proposals/language as articulated by Social Watch been accepted,
or is their impact otherwise visible? Perhaps, an additional
specific focus on regional levels should also be
considered. At least at the level of NGO networking and
presenting Social Watch, several regional efforts were made.
- Has Social Watch
influenced policy or legislative changes at national levels?
The initial outline of Social Watch focused very much at
promoting activities at national levels, for example related
to influencing the establishment and/or formulation of National
Poverty Eradication Plans or World Bank Country Assistance
Strategies. What has happened at national levels in different
countries and did it impact on the lives of poor people?
To explore this question obviously calls for the selection
of a number of country-case studies. Criteria for selecting
countries could be: a) regional and North-South variety;
b) variety between conducive and difficult environments
for NGO advocacy; c) variety in specific issues advocated
upon at national level (gender, health, education etc.);
d) involvement in Social Watch for at least three out of
the five year period.
- How effective has
been the relationship between Social Watch and the media?
This question perhaps needs to be broken down for different
national and international levels. In any case, the role
of the media is of great importance for any advocacy effort.
One element which deserves to be highlighted is the relationship
between Social Watch and Inter Press Service.
- How has the agenda
of Social Watch developed? The annual Social Watch report
has always put forward a particular theme, which was intended
to give a certain direction to national-level advocacy as
well. In combination with the country-case studies, what
kind of picture can we construct about the development of
the Social Watch agenda at national and international levels,
and how they interacted? Is it possible to distinguish between
the impact of different themes?
- Who has participated
in Social Watch, how active and how consistent? Some
picture has to be drawn of the dynamics in the overall participation
in Social Watch. In this regard, the Social Watch secretariat
probably has a lot of information ready-available.
- What kind of alliance
building has taken place related to the Social Watch?
This tackles the broader question on the relationship between
Social Watch participants (NGOs) and other civil society
actors, nationally, regionally and internationally. The
ambition to be a "mobilizing force" is clearly spelled out
in the founding document of Social Watch. The evaluation
could explore what kind of alliances have been build because
of Social Watch, or what the impact of Social Watch has
been on existing NGO coalitions/alliances. How has Social
Watch globally related to like-minded initiatives, such
as WEDO’s attempts to monitor the implementation of Beijing?
- What is the impact
of the Social Watch report? The annual Social Watch
report is the one outstanding articulation of the Social
Watch initiative. This supports a specific focus on the
report in the evaluation. How effective has the preparation,
production, distribution/ dissemination and use of the Social
Watch report been? How did the contents develop? What kind
of innovations did it introduce, etc.?
- How effective has
been the operational structure of Social Watch? Initially,
the coordination of the founding workshop of Social Watch
was based in Novib, acting upon discussions among NGOs at
the development caucus during the WSDD process. The first
meeting decided to start publishing an annual report, a
task which was mandated to the Third World Institute in
Montevideo, which subsequently took on the function of global
coordinating secretariat for Social Watch. In addition there
has been a Social Watch reference group and a smaller coordinating
committee, each meeting in different configurations and
at varying intervals. Regional contact points of the coordinating
committee were created at a later stage. The effectiveness
of these structures needs to be assessed.
- How effective has
been the involvement of Novib and the Third World Institute?
This is mostly an elaboration of the previous question.
However, given the vital role of both Novib and the Third
World Institute in Social Watch, they deserve specific attention
in the evaluation, perhaps in a case-study format.
- How has the necessary
mobilization of financial resources impacted on Social Watch?
An evaluation of Social Watch has to assess the financial
resources which has been involved, their origin, whether
they have been used effectively and what their possible
impact has been on the initiative as such (patterns of relationships,
specific conditions attached to certain resources etc.).
A number of the above
mentioned possible operational objectives for an evaluation
of Social Watch can be packaged from a methodological point
of view. Especially the country- or actor (Novib/Secretariat)
case study perspective can be used as an angle to gather information
about different questions. In addition, a few additional steps
can be taken.
- Agree on a baseline:
This is usually a serious problem in advocacy impact assessment,
but perhaps relatively easy in the case of Social Watch,
at least for the initiative overall. First, the Copenhagen
and Beijing agreements offer a very clear reference in terms
of change objectives, albeit a long list. Second, the founding
document of Social Watch of November 1995, already quoted
several times in this paper, offers quite a clear base-line
for what Social Watch has intended to do. Problems may occur
at the level of evaluating more specific situations. What
was the situation of education for girls in country x in
1995 and in 1999?
- Reconstruct the
story: Based on previous experiences with evaluating
complex NGO advocacy efforts, it would perhaps be advisable
to ask someone to write up the basic history of Social Watch.
The more complex an advocacy campaign has been, the more
it will be necessary to start with composing an overview
of what has happened at different levels of the campaign
and where interactions took place. It is likely that many
individual NGOs who participated in Social Watch have far
from a complete picture of what they have been involved
in. They may not be aware of activities in other parts of
the world, or that certain events took place, or worse,
they may even have lost track of their own involvement because
the urgency of other problems or activities have pushed
them forward. To present a basic narrative of Social Watch
is therefore likely to be an important service to the NGOs
involved in Social Watch, and necessary to set the stage
for going into other more detailed questions. It can be
done by means of a desk study.
- Use multiple sources
of information: generally, the quality of the evaluation
is likely to be enhanced if multiple sources of information
can be cross-checked with each other. In particular the
problem of causality calls for pursuing NGO advocacy impact
assessment with the curiousity and persistence of investigative
journalism. Desk-studies should preferably be complemented
by focused interviews, either by phone or in person, for
example with representatives of advocacy target audiences.
The distribution of questionnaires can also be considered,
but the response rate is usually low.
- Promote cross comparability:
It seems likely that any Social Watch evaluation will have
different components. Yet, in the further development of
the evaluation it could be tried to promote the cross comparability
of different parts. To establish one framework of key concepts
and one set of overall issues or indicators that everyone
should try to respond to is one step. But it could also
be considered to agree upon one standard checklist of questions
and indicators for country case-studies.
In the earlier discussions about the Social Watch evaluation,
it was suggested that self-evaluation(s) could play an important
role, also because there is still the final year 1999/2000
to do some additional focused monitoring which can feed
into the evaluation. This is probably a good idea. Country
case-studies, for example, could be pursued by key NGOs
who participated in Social Watch. Or key actors as Novib
or the Third World Institute could be asked to submit reflections
on their role in Social Watch.
- Who else can be
involved?: As useful as self-evaluations can be, they
will have their limitations. One could think of various
other ways of ensuring that different eyes look at Social
Watch. For sure, various academic institutions will be happy
to join in and observe/evaluate (parts of) Social Watch.
The downside of it can be that they do not know the NGO
community and the complicated field of transnational advocacy
well enough. We are also likely to end up with Northern
rather than Southern academic institutions. A somewhat different
suggestion would be to invite a journalist to follow/monitor
Social Watch in 1999/2000 (attending different meetings
etc.) and ask her to write up her story for us. We can also
consider to invite NGOs or other resource person known to
be critical of Social Watch to present their views. The
whole point is to create enough density of different views.
No evaluation of Social Watch will be complete or perfect.
One may note that I do not suggest to involve consultants.
Those capable in this field are very hard to find and I
believe that the evaluation should be more engaging.
TO MOVE FORWARD?
In order to proceed after
our meeting in New York, I would suggest the following steps.
- Establish coordination:
In whatever way we will agree on different components for
the evaluation, the pieces of the puzzle do have to come
together at one point. Some form of coordination for the
development as well as for the implementation of the evaluation
seems inevitable, but in a facilitating rather than in a
- Seek clarity on
available resources: The evaluation can be bigger or
smaller, and more or lesser ambitious. Much will depend
on the available resources. This again relates to the question
how the evaluation will be done. Self-evaluations, for example,
are likely to be cheaper and have a bigger component of
- Involve other stakeholders:
Not everyone will be present in New York. It seems that
at least the members of the Social Watch reference group
and coordinating committee should have a chance to contribute
to the development of the evaluation. The simple way would
be to distribute this paper and a summary report of the
New York meeting with a deadline for responses.
- Develop TOR:
A draft TOR should perhaps be ready by July, and a final
version in August.
Bas Arts, The Political Influence
of Global NGOs. Case Studies on the Climate and Biodiversity
Conventions. Ìnternational Books, Utrecht, 1998.
Pratap Chatterjee and Mathias Finger,
The Earth Brokers. Power, Politics and World Development.
London and New York, 1994.
Ann Doherty, The role of Non-Governmental
Organization in UNCED. Unpublished research paper. Amsterdam,
John Keane, Civil Society. Old
Images, New Visions. Oxford. 1998
Percy B. Lehning, Towards a Multi-Cultural
Civil Society: The Role of Social Capital and Democratic Citizenship.
In: Civil Society and International Development. Ed/
Amanda Bernard, Henny Helmich and Percy B. Lehning. Paris,
Development Center of the OECD, 1998, p. 27-42.
Marie Manson, NGOs on the Move.
A study of NGOs at the NGO Forum ’95 in connection with
the UN World Summit for Social Development in Copenhagen 1995.
The Swedish Institute of International Affairs, Occasional
Papers 7. Stockholm, April 1996.
Marie Manson, NGOs, Women and
Beijing. A study of NGOs at the NGO Forum on Women
’95 in connection with the Fourth World Conference on Women
in Beijing 1995. The Swedish Institute of International
Affairs, Occasional Papers 9. Stockholm, May 1996.
Valerie Miller, NGOs and Grassroots
Policy Influence: What is success? Institute for Development
Research (IDR), Boston, 1994. This paper was reproduced as
the final chapter of: Policy Influence: NGO Experiences.
Published by the Ateneo Center for Social Policy and Public
Affairs, IDR and the Konrad Adenauer Stiftung.
Quezon City, 1997.
Chris Roach, Impact Assessment
and Advocacy. In: Impact Assessment: seeing the wood and
the trees (working title), Chapter 6. Novib/Oxfam GB (forthcoming).
Anna Vakil, Confronting the Classification
Problem: Toward a Taxonomy of NGOs. In World Development,
25, no12 (1997).
Social Watch International. The Hague,
1-All quotes from Social
2-The following draws from Chris Roach and Valerie
3-The Definition is from Ana Vakil, p.2060.
4-This language draws on Percy B. Lehning and
5-See: Bas Arts, The Political Influence of
6-See for example Pratap Chatterjee and Mathias