SOCIAL WATCH DEVELOPING AN EVALUATION (1995-2000)

Discussion paper

Submitted to the Social Watch Coordinating Committee and Reference Group
New York, 22 May 1999

Peter van Tuijl
Novib
May 1999

I INTRODUCTION

Social Watch was established in November 1995 to provide a global platform for NGOs to monitor and promote the effective implementation of the important commitments which were made by national governments during the United Nations World Summit on Social Development (WSSD, Copenhagen, March, 1995) and the Fourth World Conference for Women (WCW, Beijing, September 1995). The founders of Social Watch decided to plan for an initial time span of five years to pursue the initiative, up and to the official high level evaluation of the WSSD in the year 2000.

We are now four years into Social Watch. The official five-year review of the WSSD is coming up in June 2000 in Geneva, and is preceded by two PrepComs in New York in May 1999 and April 2000. The moment is approaching for Social Watch to take stock of the initiative, to look back and learn, in support of an assessment of the opportunities and constraints to improve the project, and as an input for decision making on what to do after the first five-year period.

The complexity of Social Watch is enormous, the specific objectives of an evaluation of Social Watch are far from evident and the available methodologies for evaluating a transnational NGO advocacy effort like Social Watch are at best still experimental. The process of formulating a terms of reference for an evaluation of Social Watch is thus an important issue in itself.

This document has been written with the purpose to facilitate a systematic discussion of different aspects of a Social Watch evaluation among a number of key stakeholders. It is partly based on an informal exchange of views which has taken place between the Social Watch secretariat and Novib as of early this year. It should not be seen as a draft terms of reference and has no other intention than being a discussion paper. The document will be presented in a meeting of the Social Watch coordinating committee- and reference group on May 22, 1999, in New York.

The document has four main sections. First, I have tried to outline some basic concepts and questions, including a number of considerations why it is difficult to evaluate Social Watch (section II). This is not intended to discourage the reader, but rather to help constructing more of a common understanding before entering into a discussion on what we should evaluate and how we could do it (sections III and IV). I then propose some ideas on further developing the evaluation after our meeting in New York (section V). I have also included a list of references (section VI).

The document has been written, assuming a general knowledge and understanding of Social Watch, its history, context, structure and activities. I hope it will help to support a fruitful discussion, leading to a useful and effective evaluation of Social Watch.

Peter van Tuijl
Novib
May 1999

II EVALUATING SOCIAL WATCH: BASIC CONCEPTS AND QUESTIONS

II.A Social Watch: measuring the impact of an NGO advocacy effort

An evaluation tries to perceive change, and whether changes as identified match with those intended. Social Watch has tried to provide a global platform for NGOs to monitor and promote the implementation of the official commitments made in Copenhagen and Beijing. As such, Social Watch provides a platform for NGO advocacy. Many definitions of NGO advocacy focus on NGOs’ efforts to "influence policy" (usually public policy). However, this leaves out the element of strengthening civil society and changing power relations or democratization, which is often at the heart of NGO advocacy.

If we refer to the first brochure which came out of the founding workshop of Social Watch in November 1995, it is clear that the intentions of Social Watch were aimed both at influencing policy and at strengthening civil society. One of the angles used to formulate the objective of Social Watch in the brochure is "concentrating the lobby capacity of NGOs to influence (local and national) governments, donor agencies and international organisations in order to realise policy changes on behalf of social development." The definition of "social development" as used in the first brochure contains three basic elements: a) the provision of basic social services; b) income-generating and income-supporting activities for the poor (especially women); and c) strengthening social organisations. Finally, the three main qualitative indicators for the kind of monitoring which Social Watch wished to promote were identified as: a) a critical analysis of the current national social policies; b) the political environment, national and international; c) the involvement of civil society(1).

The approach of Social Watch as outlined in 1995 fits with current thinking on the definition of NGO advocacy and NGO advocacy impact assessment. For the sake of brevity, we will not further elaborate these conceptual discussions, but rather propose to use the following overall framework for NGO advocacy and advocacy impact assessment in the evaluation of Social Watch (2).

NGO advocacy is to organise the strategic use of information to democratise unequal power relations and to improve the condition of those living in poverty or who are otherwise discriminated against.

NGO advocacy is multi-dimensional. Three levels are distinguished for assessing the impact of NGO advocacy.

1. Policy gains: this include legislative and policy changes and their impact (or the lack of it) on the lives of poor people;

2. Civil Society gains: strengthening of NGOs and other civil society actors in ways that help to hold governments and international organisations accountable over the long run;

3. Political and democratic gains: increased democratic space where NGOs and other civil society actors can operate.

Social Watch is primarily a platform of NGOs. To perceive of the functioning of NGOs in a broader perspective of civil society is perhaps nowadays more common than when Social Watch was created in 1995. We therefore may want to add definitions of "NGOs" and "civil society" to our basic framework. The latter in particular to take distance from an understanding of civil society which includes the private for-profit business sector.

Non Governmental Organisations (NGOs) are self-governing, private, not-for-profit organisations that are geared toward improving the quality of life of disadvantages people(3).

Civil Society is a space or arena between households and the state which affords possibilities of concerted action and self-organisation. It describes and envisages a complex and dynamic ensemble of NGOs and other organisations that tend to be non-violent, self-organising, self-reflexive and permanently in tension with each other and with the state institutions that "frame" constrict and enable their activities (4).

If we can agree on this approach, it leads us to three overall questions for an evaluation of Social Watch:

1. Has Social Watch contributed to legislative and/or policy changes in support of the actual implementation of the commitments made in Copenhagen and Beijing, leading to changes in the lives of poor people?

2. Has Social Watch strengthened NGOs and/or other civil society actors to hold governments (local and national) and international organisations accountable for the implementation of commitments made in Copenhagen and Beijing?

3. Has Social Watch contributed to opening-up political space for NGOs and/or other civil society actors, in particular to raise issues concerning social development and gender inequity?

II.B Why is it difficult to evaluate Social Watch?

To establish an agreement on the above conceptual framework is important but can only be a starting point for developing an evaluation of Social Watch. We need to further unpack the complexity of Social Watch to arrive at operational objectives for an evaluation. Below, we first try to list a number of factors explaining why evaluating Social Watch is difficult.

1. A broad geographical scope: A principal claim of Social Watch is to be a global initiative, that is to say, Social Watch is open for participation by NGOs from any country in the world. In reality, the participation in Social Watch has been stronger in some countries/regions than in others and has also changed over time. When reviewing Social Watch, it will not be easy to fully grasp the broad scope of participants and activities, and the effect of changing dynamics therein on the overall impact of Social Watch.
2. Bridging the South-North divide: A particular aspect of Social Watch is that it bridges the South-North gap. As a point of departure governments in the South and in the North are equally monitored and hold accountable for the implementation of the Copenhagen and Beijing commitments. On the one hand, this equates the relationships among Southern and Northern NGO participants in Social Watch. On the other hand, the South-North divide is not insignificant for Social Watch, because Northern countries and Northern dominated international organisations are playing a significant role in setting the agenda and funding social development in the South. To use this Northern leverage, for example by supporting the 20/20 compact, has clearly been a part of Social Watch advocacy strategies. Moreover, Social Watch itself has predominantly been funded by Northern NGOs, above all by Novib as a key founder and supporter. To capture the South-North dynamics in Social Watch is thus a difficult question, which will require the unraveling of processes at the crossroads of the relationships between Social Watch participants themselves and their advocacy strategies vis-á-vis their own governments and international organisations.
3. Multiple level target audiences: The idea of Social Watch starts from a level of international commitments by governments, but great emphasis has always been put on the need to promote national level activities by Social Watch participants, primarily to ensure national level implementation of the Copenhagen and Beijing agreements. For the evaluation, it will be important to understand the interaction between international level and national level advocacy. Has the government of country x acted in a certain way because of concerted international and/or national-level pressure which can reasonably be related to Social Watch? These dynamics may also be hard to establish. It is already difficult to establish a reasonable insight into what is happening at one level, but combining and cross-checking assessments from different national and/or international levels can be like trying to force pieces from different puzzles into one frame. How do you reasonably connect the observation that some good media work was done in the local language press in a certain country with the action or inaction of an UNDP official 10.000 miles away?
4. Multiple advocacy tools: In the context of Social Watch, different advocacy strategies and tools have been used. The most outstanding tool is obviously the annual Social Watch report, probably the heart of the whole initiative. Yet, NGOs participating in Social Watch have also written separate position papers, done research, participated in various official meetings, undertaken lobby meetings and written lobby letters, organised public seminars and used the media in a variety of ways to articulate Social Watch concerns. How to measure the impact of different advocacy tools individually as well as in relation to each other?
5. Social Watch versus other civil society advocates: The objectives of Social Watch are shared by various civil society actors. NGOs not participating in Social Watch, other social actors or networks have in one way or the other monitored and promoted the implementation of the Copenhagen and Beijing agreements, sometimes in cooperation with Social Watch, sometimes not. It will not always be evident to establish the contribution of Social Watch versus the results of advocacy by other civil society advocates. This is not only a question of attribution, it also relates to the distinction made above between policy gains and civil society gains. Has Social Watch indeed managed to be a mobilizing force in civil society?
6. How to move from output to impact?: A regularly occurring weakness in NGO advocacy impact assessment is too much emphasis on output indicators, such as the number of press releases or newsletters distributed, or the number of lobby meetings with officials and whether they were high-up in the bureaucracy or not. Although it may be relevant to establish these facts, impact assessment will usually require one or more additional steps. As in other NGO advocacy efforts, the creation and distribution of information has been essential in Social Watch. The difficulty is to establish what was done with it. What kind of indications do we have that the Social Watch report was read? Can we somehow see or hear the echo of information released by Social Watch? In some cases, Social Watch lobby messages could be delivered in official meetings. But has there been any follow-up? Establishing high-level access is no guarantee for high-level impact.
7. The politics of impact assessment: Impact assessment is not a value free activity. This is true in general, but needs to be emphasized for advocacy. Different stakeholders may each have their own reasons to either exaggerate or downplay the impact of advocacy related to Social Watch. An official may say that NGOs have done a great job, while in reality the person is trying to keep the NGOs in a marginal position by making them believe they are successfully lobbying. The reverse can also happen. The impact of NGO advocacy might be ridiculed, precisely because it’s influence is felt and to prevent it from becoming more successful. All parties may suffer from a dubious self-perception. A recent study on the political influence of NGOs on the Climate and Biodiversity Conventions concludes that NGOs consistently tend to overestimate the impact of their advocacy(5).
8. Balancing different results: A multi-level and complex NGO advocacy effort as Social Watch is likely to achieve a mixed bag of results. Approaches have to be found to balance gains in one country with losses in another, or how to match intended with unintended results or drawbacks. The purpose of advocacy impact assessment is to learn and to improve future action. But the effectiveness of NGO advocacy is hardly ever achieved through a standard sequence of events. The causality in an advocacy effort may look more like a pinball machine, with actions and reactions constantly crisscrossing and influencing each other in different ways. There is an issue of balancing different types of results when constructing overall assessments. Last but not least, when evaluating Social Watch, it has to be realized that five years should be seen as a short period of time, in view of the ambitious objectives of Social Watch.

III SELECTING OBJECTIVES: WHAT SHOULD BE EVALUATED?

There are a great number of angles which can be used to formulate different operational objectives for an evaluation of Social Watch. The list we propose here is certainly not exhaustive. While the list can be expanded, priorities also need to be set. We have tried to indicate what we see as possible main elements to be dealt with in the evaluation. There is a rough order in the list of suggestions, from broader to more specific questions or subjects for the evaluation.

1. Have the main assumptions of Social Watch been correct?: Social Watch is based on a few key assumptions, notably: a) the commitments made in Copenhagen and Beijing are sufficiently taken serious nationally and internationally to provide for a useful advocacy pressure point, b) NGOs are legitimate, credible and capable actors to take up this advocacy and c) information will be the key tool to achieve results, notably by establishing "social watch" as a self-standing, recognizable and credible source of information. Have these assumptions proven to be correct?
2. How has Social Watch impacted on the strategic position of NGOs? This is directly related to the overall issues of political space and civil society as identified under II.A. If we look at the discourse on the role of NGOs with respect to the United Nations Conference on Sustainable Development (UNCED), there are clear voices contending that the ultimate impact of the NGOs’ involvement has merely resulted in NGOs resources and potential influence being sucked into time-consuming UN processes aimed at achieving corrective steps only, without fundamentally challenging the established development paradigm (6). Where has Social Watch left its NGO participants on a scale from co-optation to opening-up political space? Indicators to approach this question could be: has Social Watch engaged the UN-agenda only or expanded upon it?; has Social Watch managed to come up to the broad agenda it spelled out itself in 1995?; and/or has Social Watch supported legitimate voices on social development which otherwise would not have been heard?
3. Has Social Watch influenced policy or legislative changes at international levels? How visible and effective has Social Watch been at the level of different international target audiences: the UN, the European Union, the World Bank and the IMF etc. (all of these are mentioned in the first Social Watch brochure). What kind of access did Social Watch have, at what levels and how consistent over the four-five year period? Have proposals/language as articulated by Social Watch been accepted, or is their impact otherwise visible? Perhaps, an additional specific focus on regional levels should also be considered. At least at the level of NGO networking and presenting Social Watch, several regional efforts were made.
4. Has Social Watch influenced policy or legislative changes at national levels? The initial outline of Social Watch focused very much at promoting activities at national levels, for example related to influencing the establishment and/or formulation of National Poverty Eradication Plans or World Bank Country Assistance Strategies. What has happened at national levels in different countries and did it impact on the lives of poor people? To explore this question obviously calls for the selection of a number of country-case studies. Criteria for selecting countries could be: a) regional and North-South variety; b) variety between conducive and difficult environments for NGO advocacy; c) variety in specific issues advocated upon at national level (gender, health, education etc.); d) involvement in Social Watch for at least three out of the five year period.
5. How effective has been the relationship between Social Watch and the media? This question perhaps needs to be broken down for different national and international levels. In any case, the role of the media is of great importance for any advocacy effort. One element which deserves to be highlighted is the relationship between Social Watch and Inter Press Service.
6. How has the agenda of Social Watch developed? The annual Social Watch report has always put forward a particular theme, which was intended to give a certain direction to national-level advocacy as well. In combination with the country-case studies, what kind of picture can we construct about the development of the Social Watch agenda at national and international levels, and how they interacted? Is it possible to distinguish between the impact of different themes?
7. Who has participated in Social Watch, how active and how consistent? Some picture has to be drawn of the dynamics in the overall participation in Social Watch. In this regard, the Social Watch secretariat probably has a lot of information ready-available.
8. What kind of alliance building has taken place related to the Social Watch? This tackles the broader question on the relationship between Social Watch participants (NGOs) and other civil society actors, nationally, regionally and internationally. The ambition to be a "mobilizing force" is clearly spelled out in the founding document of Social Watch. The evaluation could explore what kind of alliances have been build because of Social Watch, or what the impact of Social Watch has been on existing NGO coalitions/alliances. How has Social Watch globally related to like-minded initiatives, such as WEDO’s attempts to monitor the implementation of Beijing?
9. What is the impact of the Social Watch report? The annual Social Watch report is the one outstanding articulation of the Social Watch initiative. This supports a specific focus on the report in the evaluation. How effective has the preparation, production, distribution/ dissemination and use of the Social Watch report been? How did the contents develop? What kind of innovations did it introduce, etc.?
10. How effective has been the operational structure of Social Watch? Initially, the coordination of the founding workshop of Social Watch was based in Novib, acting upon discussions among NGOs at the development caucus during the WSDD process. The first meeting decided to start publishing an annual report, a task which was mandated to the Third World Institute in Montevideo, which subsequently took on the function of global coordinating secretariat for Social Watch. In addition there has been a Social Watch reference group and a smaller coordinating committee, each meeting in different configurations and at varying intervals. Regional contact points of the coordinating committee were created at a later stage. The effectiveness of these structures needs to be assessed.
11. How effective has been the involvement of Novib and the Third World Institute? This is mostly an elaboration of the previous question. However, given the vital role of both Novib and the Third World Institute in Social Watch, they deserve specific attention in the evaluation, perhaps in a case-study format.
12. How has the necessary mobilization of financial resources impacted on Social Watch? An evaluation of Social Watch has to assess the financial resources which has been involved, their origin, whether they have been used effectively and what their possible impact has been on the initiative as such (patterns of relationships, specific conditions attached to certain resources etc.).

IV DEVELOPING THE METHODOLOGY

A number of the above mentioned possible operational objectives for an evaluation of Social Watch can be packaged from a methodological point of view. Especially the country- or actor (Novib/Secretariat) case study perspective can be used as an angle to gather information about different questions. In addition, a few additional steps can be taken.

1. Agree on a baseline: This is usually a serious problem in advocacy impact assessment, but perhaps relatively easy in the case of Social Watch, at least for the initiative overall. First, the Copenhagen and Beijing agreements offer a very clear reference in terms of change objectives, albeit a long list. Second, the founding document of Social Watch of November 1995, already quoted several times in this paper, offers quite a clear base-line for what Social Watch has intended to do. Problems may occur at the level of evaluating more specific situations. What was the situation of education for girls in country x in 1995 and in 1999?
2. Reconstruct the story: Based on previous experiences with evaluating complex NGO advocacy efforts, it would perhaps be advisable to ask someone to write up the basic history of Social Watch. The more complex an advocacy campaign has been, the more it will be necessary to start with composing an overview of what has happened at different levels of the campaign and where interactions took place. It is likely that many individual NGOs who participated in Social Watch have far from a complete picture of what they have been involved in. They may not be aware of activities in other parts of the world, or that certain events took place, or worse, they may even have lost track of their own involvement because the urgency of other problems or activities have pushed them forward. To present a basic narrative of Social Watch is therefore likely to be an important service to the NGOs involved in Social Watch, and necessary to set the stage for going into other more detailed questions. It can be done by means of a desk study.
3. Use multiple sources of information: generally, the quality of the evaluation is likely to be enhanced if multiple sources of information can be cross-checked with each other. In particular the problem of causality calls for pursuing NGO advocacy impact assessment with the curiousity and persistence of investigative journalism. Desk-studies should preferably be complemented by focused interviews, either by phone or in person, for example with representatives of advocacy target audiences. The distribution of questionnaires can also be considered, but the response rate is usually low.
4. Promote cross comparability: It seems likely that any Social Watch evaluation will have different components. Yet, in the further development of the evaluation it could be tried to promote the cross comparability of different parts. To establish one framework of key concepts and one set of overall issues or indicators that everyone should try to respond to is one step. But it could also be considered to agree upon one standard checklist of questions and indicators for country case-studies.
5. Self-evaluation(s): In the earlier discussions about the Social Watch evaluation, it was suggested that self-evaluation(s) could play an important role, also because there is still the final year 1999/2000 to do some additional focused monitoring which can feed into the evaluation. This is probably a good idea. Country case-studies, for example, could be pursued by key NGOs who participated in Social Watch. Or key actors as Novib or the Third World Institute could be asked to submit reflections on their role in Social Watch.
6. Who else can be involved?: As useful as self-evaluations can be, they will have their limitations. One could think of various other ways of ensuring that different eyes look at Social Watch. For sure, various academic institutions will be happy to join in and observe/evaluate (parts of) Social Watch. The downside of it can be that they do not know the NGO community and the complicated field of transnational advocacy well enough. We are also likely to end up with Northern rather than Southern academic institutions. A somewhat different suggestion would be to invite a journalist to follow/monitor Social Watch in 1999/2000 (attending different meetings etc.) and ask her to write up her story for us. We can also consider to invite NGOs or other resource person known to be critical of Social Watch to present their views. The whole point is to create enough density of different views. No evaluation of Social Watch will be complete or perfect. One may note that I do not suggest to involve consultants. Those capable in this field are very hard to find and I believe that the evaluation should be more engaging.

V HOW TO MOVE FORWARD?

In order to proceed after our meeting in New York, I would suggest the following steps.

1. Establish coordination: In whatever way we will agree on different components for the evaluation, the pieces of the puzzle do have to come together at one point. Some form of coordination for the development as well as for the implementation of the evaluation seems inevitable, but in a facilitating rather than in a substantive sense.
2. Seek clarity on available resources: The evaluation can be bigger or smaller, and more or lesser ambitious. Much will depend on the available resources. This again relates to the question how the evaluation will be done. Self-evaluations, for example, are likely to be cheaper and have a bigger component of self-funding.
3. Involve other stakeholders: Not everyone will be present in New York. It seems that at least the members of the Social Watch reference group and coordinating committee should have a chance to contribute to the development of the evaluation. The simple way would be to distribute this paper and a summary report of the New York meeting with a deadline for responses.
4. Develop TOR: A draft TOR should perhaps be ready by July, and a final version in August.

VI REFERENCES

Bas Arts, The Political Influence of Global NGOs. Case Studies on the Climate and Biodiversity Conventions. Ìnternational Books, Utrecht, 1998.

Pratap Chatterjee and Mathias Finger, The Earth Brokers. Power, Politics and World Development. London and New York, 1994.

Ann Doherty, The role of Non-Governmental Organization in UNCED. Unpublished research paper. Amsterdam, November 1992.

John Keane, Civil Society. Old Images, New Visions. Oxford. 1998

Percy B. Lehning, Towards a Multi-Cultural Civil Society: The Role of Social Capital and Democratic Citizenship. In: Civil Society and International Development. Ed/ Amanda Bernard, Henny Helmich and Percy B. Lehning. Paris, Development Center of the OECD, 1998, p. 27-42.

Marie Manson, NGOs on the Move. A study of NGOs at the NGO Forum ’95 in connection with the UN World Summit for Social Development in Copenhagen 1995. The Swedish Institute of International Affairs, Occasional Papers 7. Stockholm, April 1996.

Marie Manson, NGOs, Women and Beijing. A study of NGOs at the NGO Forum on Women ’95 in connection with the Fourth World Conference on Women in Beijing 1995. The Swedish Institute of International Affairs, Occasional Papers 9. Stockholm, May 1996.

Valerie Miller, NGOs and Grassroots Policy Influence: What is success? Institute for Development Research (IDR), Boston, 1994. This paper was reproduced as the final chapter of: Policy Influence: NGO Experiences. Published by the Ateneo Center for Social Policy and Public Affairs, IDR and the Konrad Adenauer Stiftung.

Quezon City, 1997.

Chris Roach, Impact Assessment and Advocacy. In: Impact Assessment: seeing the wood and the trees (working title), Chapter 6. Novib/Oxfam GB (forthcoming).

Anna Vakil, Confronting the Classification Problem: Toward a Taxonomy of NGOs. In World Development, 25, no12 (1997).

Social Watch International. The Hague, November 1995.

1-All quotes from Social Watch International.
2-The following draws from Chris Roach and Valerie Miller.
3-The Definition is from Ana Vakil, p.2060.
4-This language draws on Percy B. Lehning and John Keane.
5-See: Bas Arts, The Political Influence of Global NGOs.
6-See for example Pratap Chatterjee and Mathias Finger.