TOWARDS AN OUTCOME BASED POLICY EVALUATION FRAMEWORK

Since 1994, South Africa has experienced the policy cycle in a fast-forward mode due to the transition phase from the apartheid era. As a result, policy design, legislation and policy implementation have proceeded rapidly in all sectors. This transition meant that many of the policies that were inherited from the apartheid era were inappropriate for the democratic dispensation. To this effect Roux (2002:420) notes that constitutional reform has led to change and transformation in almost all spheres of government and administration. Such changes affected virtually all the functional fields of government, and consequently redefined the role of policy- and decision-makers. Echoing this sentiment Brynard (2005:3) states that an extraordinary degree of intellectual and political energy was harnessed to generate public policies that would suit the current needs of the State. South Africa, in a policy context, went through a major review of policies especially between 1995 and 1996; Brynard (2005:3) terms this period the ‘White Paper Era’.

Brynard (2000:164-165) further states that after 1994, the democratic government embarked on an aggressive process of policy formulation with a view to remove discrimination in the government’s public policy and statute. This continued until the end of 1990s. The second democratic government (1999 to 2004) shifted focus more towards implementation of policies of a democratically elected government, which still continues.

The focus of this paper is on the performance of such policies. Goldfrank (1998:1) highlights the importance of looking beyond the euphoria that comes with the installation of new governments when assessing such government’s performance. He contends that, in studying the relatively recently democratised countries, scholars have largely moved beyond the theme of transitions to democratically-elected governments and have begun to ask questions regarding the kind of democracies that have arisen and how to sustain democratic practices. Almost uniformly, political analysts and actors deplore the quality of the new democracies, pointing to one or another deficiency, including ineffective legislatures, inefficient public bureaucracies, corrupt judiciaries, and, perhaps most strikingly governments’ inability to deliver their mandates.

Sanderson (2002:2) support this view when he points out that with increasing questioning and scrutiny of public intervention in economic and social spheres, governments are turning to evidence of performance for legitimacy since it is no longer guaranteed solely by democratic political processes.

This paper argues that for the government to be able to provide evidence of performance of its policies, it must institutionalise an outcomes-based evaluation system. An Outcomes-based Policy Evaluation system is presented in this paper as a tool through which the government can objectively demonstrate achievements of its policies while at the same time accounting about the performance of its policies. However, for such a framework to be successful it must be embedded on a well crafted evidence based system. Thus, the researcher will argue that Evidence-based practice is a cornerstone for an outcomes-based policy performance system. Hence a saying that the system will only be as good as the data that it is based on holds true for this paper.

In support of this exposition, Rosanbalm, Owen, Rosch and Harrison (2009:6) contend that evidence-based policy provides an effective mechanism to establish, in a scientifically valid way, what works or does not work, and for whom it works or does not work. With this structured approach to evaluation, knowledge can be used to improve practice, allowing successful programs to develop iteratively over time. Without this approach, interventions go in and out of practice, little is learned about what works, and the effectiveness of social programs does not advance significantly over time. Rigorous evaluation can end the spinning of wheels and bring rapid progress to social policy as it has to the field of medicine.

This paper, though critical of the emerging policy evaluation framework in SA, it acknowledges the efforts made in the policy arena since 1994. Further, in identifying challenges, this paper seeks to take a forward-looking approach that would outline the issues which government must grapple with in order to develop an outcomes based policy evaluation framework.

Research Problem

After fifteen years of policy implementation, questions on whether or not such policies are delivering the intended outcomes are continuously being raised by different stakeholders including the government and the ruling party, African National Congress (ANC). For instance, since its landmark victory in 1994, the ANC government has introduced several policies with the aim of improving the living conditions of the South Africans. Now the dilemma that is facing the ruling party is its inability to objectively determine the extent to which the implemented policies are adding value to the lives of the previously disadvantaged communities. The ANC has reiterated this concern in its Strategy and Tactics document of 2002 where it argues that, ‘…policy leadership responsibility is compromised by the general absence of reliable and appropriate information that will evaluate policy performance and the impact of government policy decisions. Where there is information available it is compiled and communicated by those responsible for implementation, which raises the question as to the reliability and validity of the evidence that is being presented to the Executive, Parliament and the ruling party.

This suggests that performance measurement systems in government require serious rethinking. The biggest challenge is that most performance measurement systems in government are still input-based and, at the most, report on outputs without justifying input-output ratio (Sangweni 2006:6).

Schacter (1993:1) is very accurate in his diagnosis of the problem when he contends that public sector performance has often been measured in terms of what the government has done, meaning an amount of funding provided, number of kilometers of road tarred, number of new hospital beds and so forth. Such measures focus on how “busy” the government has been rather than on what it has achieved. They highlight means rather than ends.

Schacter (1993:2) further argues that this is not to say that keeping track of means, as opposed to ends isn’t important. Governments need to measure how much they spend and “do”. But when performance measurement focuses too heavily or exclusively on how much is spent -“inputs” – or done – “outputs” — as opposed to impact on society — “outcomes” — the result is often that public sector organisations lose sight of why they were created in the first place. Public organisations may be very “busy” but be accomplishing little from society’s perspective. For example, it would be futile for the Department of Transport to build thousands of kilometers of roads to places where no one travels. The danger of this approach, as noted by Radebe and Pierre (2007:110) is that organisations take their own implementation decisions which may not be in line with national priorities. One of the consequences of the apparent absence of strategic leadership was pointed out as inappropriate infrastructure developments such as building new parking facilities at Durban International Airport while the airport would be decommissioned in 2009.

The purpose of the Study and Research Questions

The purpose of this study is to explore the extent to which Evidence based and Results based management approaches are being applied in SA in the area of public policy with an aim of improving policy performance feedback (performance information or evidence of whether policies are successful or not). In order to achieve this purpose the researcher will be guided by two main research questions. The first question to be addressed is: Why does the increased attention to outcomes and accountability intersect with the growing demand for evidence-based policies and programs? In other words, how does the advancement of connections between science (evidence-based policy making approach) and policy making improve policy evaluation? The researcher will contend that a government that basis its policy decisions on scientific evidence enhances its chances not only of implementing sound policies but also of executing effective performance evaluation of its policies. According to Lasswell (quoted by Hoppe 1999:1), policy science is about the production and application of knowledge of and in policy. Policymakers, who desire to successfully tackle problems on the political agenda, should be able to mobilise the best available knowledge. This requires high quality knowledge in policy. Policymakers and, in a democracy, citizens, also need to know how policy processes really evolve. This demands precise knowledge of policy. There is an obvious link between the two: the more and better knowledge of policy, the easier it is to mobilise knowledge in policy.

Hartig, DePinto, Stone and McIntyre (2003:1) observed that informing public policy with sound science has long been recognized as a vital need for effective policy management… However, delivering scientific findings to policy-makers in a useful manner has been problematic. Policy-makers have often lacked timely access to scientific information. And when they do have access, this information is often too technical and needs interpretation to be truly useful for decision-making. Clearly, there is a need to strengthen science-policy linkages in order to improve policy performance.

The second question to be addressed is: what strides have been made by SA towards an outcomes-based policy performance evaluation framework – “Where are we” and what are the gaps? To this end, Scott (2006:87) argues that South African government need to be able to determine whether government policies, interpreted into government programmes and projects, are causally linked to policy outcomes. We need to be able to determine whether progress, or lack of it, is due to (or happening despite) government policies and activities. Thus this paper will carefully examine the extent to which the South African government is able to objectively report on the performance of its policy interventions and also whether policy evaluation data is utilised to improve future policy interventions.

Objectives of the Study

This paper has three main objectives:

Firstly, the study aims to examine the extent to which departments apply the Government-wide Policy Framework on Monitoring and Evaluation which was published by government in 2007; this is an overarching policy framework that ushers a new culture on monitoring and evaluation and is predicated on a RBM approach (The Presidency 2007:1). Secondly, the study aims to assess the manner in which government departments generate and use evidence throughout the policy lifecycle (policy formulation, policy implementation and policy closure or redesign). Thirdly, the study aims to assess the impact of the existing accountability mechanisms on the utilisation of scientifically generated evidence in government.

Theoretical Framework

This paper employs a dynamic analysis approach of the systems theory as a basis for understanding the interrelationship between policy making and policy evaluation. Dynamic analysis examines interdependent effects among variables over time, with time lags on effects and feedback loops as part of the analysis. Dynamic analysis differs significantly from static analysis which assumes unidirectional relationships between the independent and dependent variables in the analysis. While static analysis assumes that a change in some independent variables will result in change in one or more dependent variables, dynamic analysis introduces two-way relationship or feedback loops into the system of relationships being investigated (that is, in the two-way relationship, a change in one variable affects the second, which in turn affects the first – changes in both variables continue until equilibrium or system collapse occurs (Melcher A and Melcher B, 1980:235-239).

Thus this paper moves from the premise that if policies are based on tested theories (theories that have been subjected to vigorous scientific procedures); examination of their performance during and after implementation is made easy. Subsequently, evidence of whether policies work or not will be feedback to the initial phase of policy formulation for policy redesign where necessary. This is premised on the fact that public policies are not eternal truths but rather hypotheses subject to alteration and to devising of new and better ones until these in turn are proved unsatisfactory (Wildavsky 1979:16). To this end, this paper ventures into assessing which procedures are in place in SA and which processes, according to literature, ought to be in place in order for government to be able to account to its citizens on the implementation of public policies.

Literature is very rich on how government’s ability to account on the implementation of public policies can be improved. The focus of study is limited to two interventions. They are Evidence-Based Policy Making and Results-Based Management approaches. Evidence-Based Policy Making approach finds its expression through policy science which can be summarised as the intersection between scientific research and public policy.

Davies as cited by Segone (2004:27) defines evidence-based policy as an approach which helps people make well informed decisions about policies, programmes and projects by putting the best available evidence at the heart of policy development and implementation. Segone (2004:27) points out that this definition matches that of the UN in the MDG guide where it is stated that “Evidence-based policy making refers to a policy process that helps planners make better-informed decisions by putting the best available evidence at the centre of the policy process”. Evidence may include information produced by integrated monitoring and evaluation systems, academic research, historical experience and “good practice” information. This approach stands in contrast to opinion-based policy, which relies heavily on either the selective use of evidence (e.g. on single studies irrespective of quality) or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices, or speculative conjecture.

Proponents of evidence-based policy and practice acknowledge that not all sources of evidence are sufficiently sound to form the basis of policy making. Much research and evaluation is flawed by unclear objectives; poor design; methodological weaknesses; inadequate statistical reporting and analysis; selective use of data; and, conclusions which are not supported by the data provided (Davies 2003:54).

On the other hand, Results-Based Management (RBM) is defined as a management strategy aimed at achieving important changes in the way organisations operate, with improving performance in terms of results as the central orientation. RBM provides the management framework and tools for strategic planning, risk management, performance monitoring and evaluation. Its primary purpose is to improve efficiency and effectiveness through organisational learning, and secondly to fulfill accountability obligations through performance reporting. Key to its success is the involvement of stakeholders throughout the management lifecycle in defining realistic expected results, assessing risk, monitoring progress, reporting on performance and integrating lessons learned into management decisions (Meier 2003:6)

Scott, Joubert and Anyogu (2006:11) concur with Meier when they contend that RBM is a management strategy or approach by which an organization ensures that its processes, products and services contribute to the achievement of clearly stated results. RBM provides a coherent framework for strategic planning and management by improving learning and accountability. It is also a broad management strategy aimed at achieving important changes in the way agencies operate, with improving performance and achieving results as the central orientation, by defining realistic expected results, monitoring progress towards the achievement of expected results, integrating lessons learned into management decisions and reporting on performance.

Key RBM concepts central to this paper include; theory of change, causal chain, programme theory and logic model. According to Bickman (1987:2) program theory can be defined as “a plausible and sensible model of how a program [policy] is supposed to work.” A good program theory logically and reasonably links program activities to one or more outcomes for participants. Program theories can often be captured in a series of “if-then” statements – IF something is done to, with, or for program participants, THEN theoretically something will change. Figure 1 below illustrates how a program theory can be captured in a logframe.

Figure 1: Logframe

On the other hand logic model is a tool for illustrating an underlying program theory. A logic model illustrates the linkages between program components and outcomes (Wilder Research Center1987:2-4). It is this theory that must be backed-up by sound evidence as discussed in chapter 2. Figure 2 below illustrates how a logic model can be captured.

Figure 2: Example of a RBM Logic Model

ACTIVITIES OUTPUTS RESULTS IMPACT

Research, monitoring, analysis of information

Dissemination of information to health workers and population

Informed Stakeholders

Population assumes responsibility to protect, maintain, improve its health

Improved general health. Reduced variances between segments of the population.

Research Design

This is a qualitative research which is located within the evaluation field of study. A qualitative research methodology has been chosen because of its approach towards finding the truth which bodes very well with the requirements of this study. Qualitative methods draw up an interpretive paradigm where there are multiple truths regarding the social world. In qualitative methods knowledge gathering is always partial, and the researcher is encouraged to be on the same plane as the researched in an effort to promote a co-construction of meaning Try to link this statement to your study to make what you are saying clearer to the reader. (Hesse-Biber and Leavey 2006:320).

The researcher will use literature review to achieve three objectives. Firstly, this paper will examine literature on the application of evidence-based and RBM approaches throughout the policy lifecycle in order to construct a framework of analysis for the study. Secondly, the paper will identify critical variables that may help government to institutionalise an outcomes-based policy evaluation framework. Thirdly, the paper will examine strides that have been made by the SA government towards an outcomes based policy evaluation framework. Comparative views on the achievement of other developing and developed countries will be included in this study in order to augment theoretical exposition of this study with empirical evidence.

The literature review will further be augmented with empirical findings arising from the semi-structured interviews. The interviews will be carried out with a sample of respondents from government whose jobs’ functions entail research, policy analysis and evaluation. The nature of the study requires (experts focused input) that the sample be stratified; as a result the paper will use a non-probability sampling technique (judgmental sample).

With regard to data analysis, the researcher will use content analysis method which has been credited for its versatility to both quantitative and qualitative research enquiries. For instance, Creswell (2003:289) contends that content analysis has historically been conducted quantitatively; however, now there is a rich tradition of qualitative content analysis. The primary difference in these two broad applications is in research design. Quantitative approaches to content analysis are largely deductive and follow a linear model of research design. Qualitative approaches are mainly inductive and follow what is termed a spiral model of research design. When using a linear design the researcher has a preconceived set of steps to follow in a linear (vertical) path through each phase of the research process. A spiral design, employed by qualitative researchers, allows the investigator to, metaphorically, drive in and out of the data. In this model a researcher generates new understandings, with varied levels of specificity (Hesse, et al 2003:289).

This paper will employ the spiral model together with the memo writing approach in analysing the findings. By writing memos one can raise a code to the level of a category. The idea of a grounded theory approach is to read carefully through the data and to uncover the major categories and concepts and ultimately the properties of these categories and their interrelationships. Memo writing is an integral part of the grounded theory process and assists the researcher in elaborating on their ideas regarding their data and code categories. Reading through and sorting memos can also aid the researcher in integrating his or her ideas and may even serve to bring up new ideas and relationships within the data. (Hess, et al 2003:349)

As the process of analysis continues the researcher may begin to see more developed codes – focused codes especially through the process of writing memos. Coding is a central part of a grounded theory approach and involves extracting meaning from non-numerical data such as text and multimedia such as audio and video. Coding is the analysis strategy many qualitative researchers employ in order to help them locate key themes, patterns, ideas, and concepts that may exist within their data (Hesse, et al: 2003, 349).

To conclude, Karp (2003:356) notes that after pondering the ideas in the memos and coding interviews – when you think you have been able to “grab onto a theme” – it is time to begin what he term “data memo”. By this he means a memo that integrates the theme with data and any available literature that fits; something that begins to look like a paper.

Importance of the Study

Even though the focus of this paper is on performance evaluation, it ultimately addresses a very critical issue of an accountable government. Thus the researcher will also argue that a performance evaluation system should enable the government to account to its citizens about the effective and efficient use of their resources. This paper will thus contribute to the growing body of knowledge of policy making and performance evaluation in the South African literature, which aims at strengthening the accountability mechanisms of government.

Summary of Literature Review

The second chapter of this study focuses on the evolution of the policy analysis with specific focus on policy making and evaluation as well as on the progress made by SA towards an outcomes-based policy evaluation framework. A brief outline of some of the sections covered in the literature is provided below:

Role of theories in policy making

While policy could be defined in several ways, the point of departure for this paper is that policy is viewed as a theory. The proposition of this paper is that theories that underlie policies must be backed up by scientific evidence so that measures of success for policy performance will be effective. This proposition is backed up by scholars such as Pressman and Wildavsky (1973, 1979), Bardach (1977) and more recently by Pawson (2002). For instance, Pressman and Wildavsky (1973) described any policy as a ‘hypothesis’ containing initial conditions and predicted consequences. That is, the typical reasoning of the policy-maker is along the lines of ‘if x is done at time t(1) then y will result at time t(2)’. Hill (1998) concludes that thus every policy incorporates a theory of cause and effect (normally unstated in practice) and, if the policy fails, it may be the underlying theory that is at fault rather than the execution of the policy.

Role of Evidence in Policy Making – Evidence Based Policy Making (EBPM) Approach

As stated above, the proposition of this paper is that theories that underlie policies must be backed up by scientific evidence so that measures of success for policy performance will be effective. This view is supported by scholars such as Gray (1997), Davies (1999, 2003), Nutley (2003) and Segone (2004). Arguments presented by these scholars are discussed in detail in chapter two, which is the literature review chapter. These scholars concur that evidence-based decision making draws heavily upon the findings of scientific research, including social scientific research that has been gathered and critically appraised according to explicit and sound principles of scientific inquiry.

Framework for an accountable and learning Government

Recently, we have observed a growing interest in performance measurement or evaluation in the public sector. The question is, what drives this interest in performance measurement and evaluation, in the public sector? In answering this question Schacter (2002:5) argues that the fundamental reason why performance measurement matters to us is that it makes accountability possible, and accountability goes to the heart of our system of political governance. Schacter further contends that citizens grant their governments a high degree of control over their lives. Citizens allow governments to take part of their income through taxes for instance, and to limit their freedom through enforcement of laws and regulations. In return citizens expect their governments to be accountable to them for the ways in which they exercise power.

Performance evaluation is not only beneficial to citizens but to government as well. A government that utilises findings on the performance of its policies is able to improve on new policies as well as on the implementation of such policies. Wildavsky (1984:255) echoes this point when he contends that learning evaluation strives to unearth faulty assumptions, reshape misshapen policy designs, and continuously refine goals in light of new information derived during implementation.

Previous research on Policy Making and Evaluation in South Africa

Literature reviewed indicates that a significant amount of work has been done on policy making and evaluation in SA. Key topics covered in the reviewed literature include transition from apartheid to democratic era, Public policy making in a post-apartheid South Africa, policy evaluation, Electoral system and political accountability. These topics are addressed in chapter 2 where I discuss the work of scholars like Van Niekerk, Van Der Waldt and Jonker (2001) Roux (2002), Cloete and Wissink (2004), Scott (2006 and 2007), Radebe and Pierre (2007), Christo de Coning (2008), Gumede (2008), Carter (2008). Government reports, covering framework and performance documents, are also used in this study to present the side of government.

Notwithstanding the existence of literature on policy making and evaluation, more work is still needed on how evidence-based approach improves policy performance as well as quality of performance data; this is the area this study seeks to address.

Limitations of the Study

This paper will not venture to quantifiably assess the extent to which the introduction of Evidence-Based Policy Making and Results-Based Management approaches have improved policy performance feedback in SA. Such an enquiry will require more time and a different strategy; this will be a subject for further research. Nevertheless, this paper will explore scholarly literature so as to identify main arguments on how policy evaluation could be improved. Themes emanating from the literature will then be tested through an interview with a sample of policy and evaluation practitioners.

The other limitation of this paper is that, no matter how relevant it may be, it does not represent the official position of government. Hence there is no guarantee for the implementation of the recommendations of this paper. Finally, the timeframe as well as the financial resources will limit the researcher from doing an in-depth analysis of key variable of the study, i.e. the relationship between policy making and policy performance measurement approaches.

Chapter Outline

Abstract: presents an overview of the paper and introduces contents of each chapter.

Introduction: presents a background to the study, the motivations for embarking on the proposed study as well as the purpose of the study are also presented.

Chapter one: This chapter provides a reader with a methodology to be employed in search for the answers to the research questions. It also presents a brief summary of the theoretical framework which includes concepts and theories.

Chapter two: looks into the literature that is already available on the evolution of the policy analysis with specific focus on policy making and evaluation. Key variables for an effective policy performance assessment framework will also be identified in this chapter

Chapter three: examines strides that have been made by the government towards an ideal (evidence-outcomes based framework as espoused in chapter two) policy performance evaluation framework

Chapter four: provides analysis and interpretation of the research findings based on the reviewed literature and interview outcomes. The purpose of this chapter is to present solid descriptive data and to lead the reader to understand the meaning of the phenomenon that is being studied. Content analysis approach and memo writing approach are utilised to analyse and interpret the findings of the study.

Chapter five: presents conclusions and recommendations that have been drawn from the reviewed literature, constructed theoretical framework, as well as the interview results. The chapter also proposes research areas requiring further research in the field of policy assessment.

Conclusion

In this paper the researcher intends to assess critical strides made by SA towards an outcome-based policy evaluation framework. The researcher will use evidence emanating from literature and interviews to highlight weaknesses in the SA public sector performance evaluation system.