Policy research: surveys work best if balanced by in-depth qualitative studies

Agrarian reform

In conversation with Tim Hart

The purpose of policy research is to enable policy makers and officials to formulate sound policies to address areas within society that require much needed intervention – including the economy. It should also enable the provision of adequate information that allows for appropriate conceptualisation of intervention programmes by indicating why certain things are or are not happening.

Unfortunately, most policy makers, officials and even some scientists believe that statistics derived from fairly regular official national surveys are adequate for these purposes. I believe surveys have the best impact on policy when they go hand in hand with in-depth qualitative case studies that complement the information obtained from surveys, particularly national and localised surveys that tend to aggregate people into unrealistic categories. On their own, neither of these two approaches (surveys or case studies) is adequate for policy makers and officials attempting to ensure social and economic improvement.

To illustrate this point, South Africa’s Science, Technology and Innovation (STI) policy is based on national STI and research and development (R&D) surveys. However, while these enable us to compare across countries at a national level, they do not enable us to understand geographical and regional nuances within a country. They also don’t tell us about enterprise dynamics, and why some enterprises are included in these surveys and therefore have a better chance of receiving support, while others are not.

The Rural Innovation Assessment Toolbox study, supported by the Department of Science and Technology, is looking at developing tools to overcome some of these problems and to identify the challenges in the innovation system at various levels. Case studies and local surveys are two of the important tools being developed.
National surveys give us a national picture of what is currently happening but not how or why it has happened. Nor do they enlighten us on how to intervene where this might be required. In fact, without knowing how and why such a state of affairs exists, can we really provide guidance on what policy interventions are required in order to bring about positive change?

Of course, when regularly conducted, some of these surveys can enable us to identify changes over time, such as the Quarterly Labour Force Survey, which repeatedly tries to include the original respondent. Other surveys enable us to obtain a snapshot of the situation in a province or in a district, if we are lucky. Unfortunately, beyond that level, they become meaningless due to sample size constraints and we have no real understanding of why diversity exists.

More localised surveys might be considered an option but these too are limited in that they only provide a picture of what is happening at a particular point in time in specific sites. Like their larger companions, they are unable to tell us the why and how.

But perhaps the most serious constraint with surveys is their inability to pick up important information about the realities of the day-to-day lived experiences of the respondents. This is of course vital for effective policy making, but it is obscured in the nature of the survey process with its tendency to ignore diversity and place people into fairly neat categories. For example, it becomes difficult for a respondent in a food security survey to answer questions about plot yield of specific crops when in fact they plant multiple crops on a plot and harvest produce as and when needed for consumption throughout the season as opposed to simply at the end.

If research is to seriously contribute to policy making it must consider the use of in-depth case studies to indicate the reality of what is being experienced. More specifically, why and how this has come about in different locations, and also indicating possible ways to intervene and improve the situation (these are often location specific). All this needs to be based on empirical evidence.

The inclusion of relevant case studies may well increase research costs, but should this be a concern if more effective policies and interventions are a result? It is within the country’s interests that future policy research makes provision for including case studies as part of the methodology.

It was very welcoming to see that a recent call by the Department of Performance Monitoring and Evaluation, based in the Presidency, required in-depth case studies to be a part of a national impact evaluation of the Comprehensive Agricultural Support Programme. Now all we need is for other line departments to adopt this style and to ensure that there are sufficient funds and time frames to permit comprehensive and effective policy research.

Dr Tim Hart, senior research manager, Economic Performance and Development (EPD) research programme, HSRC.