Methodology wars in the measuring and evaluation of innovation

The typical instruments and manuals used to evaluate a country’s level and extent of research and development and science, technology and innovation, are not appropriate for many developing countries as they overlook the glaring differences between developed and developing economies, and the innovation (products) that do exist. Alexandra Mhula, Tim Hart and Peter Jacobs discuss alternative measurements employed by other developing countries and ask whether these should not also be adapted to local circumstances.
Globally, innovation and the recognition to be considered innovative appear to be increasingly popular ambitions among so-called developed and developing countries, with most striving for improved and increased innovative capacity, capability and output. To enable comparisons between countries, standardised survey tools are used to measure and evaluate a country’s level and extent of science, technology and innovation (STI), using indicators such as capacity, outputs, patents and intellectual property rights (IPR).

Since the early 1960s, the Organisation for Economic Co-operation and Development (OECD) has been compiling research and development (R&D) and STI indicators. The development of these indicators took years of experimentation and scrutinising by various government agencies. As a result, the OECD developed various manuals and survey instruments, such as the Frascati and Oslo Manuals for measuring the level and extent of R&D and STI.

The use of such standardised instruments overlooks the glaring differences between developed and developing economies.

These manuals, along with their measurement instruments, quickly became internationally accepted as reference points for the development of STI indicators and the comparative measurement of innovation. They are currently used to measure innovation activities in both the developed and developing countries. However, OECD indicators are largely based on experience and circumstances within the OECD countries, rather than developing countries. Furthermore, these indicators were developed for the purpose of using national surveys focusing on formal enterprises (i.e. firms).
However, the use of such standardised instruments overlooks the glaring differences between developed and developing economies, including the diversity of reasons why specific innovation activities are selected and others not. Consequently, there is a failure to recognise the subsequent innovations (products and processes) determined by these choices.

The need for developing more relevant indicators is perhaps most urgent in Africa.

There is a need for a more bottom-up approach to developing indicators, given that national innovation choices, outputs and capacity differ from those at the more localised level, while national comparisons do not take into account the inherent and structural differences between developing and developed countries and their economies. The need for developing more relevant indicators is perhaps most urgent in Africa.

In Latin America and the Caribbean, these indicators have been adapted to some extent to capture the innovation diversity found in many of these countries. This pioneering work was compiled into what is known as the Bogota Manual. Unfortunately, these important contributions to measuring innovation in developing countries are simply noted in the appendix of the Oslo Manual. Consequently, much of the innovation activities occurring in informal enterprises are not captured by these instruments. Even if they did attempt to consider informal activities, the indicators currently used (outputs, commercial/market value, tertiary education, IPRs etc.) are insufficient.

Local innovations occurring in the informal sector cannot be neglected, especially in developing countries where they play a crucial role in the local development of rural communities and their livelihood strategies. In many cases such innovations address local social and economic challenges, including unemployment, food and water scarcity, and inadequate health, social and education services. While it makes sense to adapt the best and most relevant OECD indicators when compiling STI and R&D assessment instruments for developing countries, including South Africa, it is imperative to also develop and include indicators that would accommodate the local diversity of these countries.

The international PROLINNOVA (Promoting Local Innovation in Ecologically-oriented Agriculture and Natural Resource Management) network has made significant strides in emphasising the importance of local innovations, particularly in African agriculture and natural resource use. Where appropriate, the programme attempts to strengthen the linkages between farmers, users, researchers, NGOs and other more formal stakeholders in agricultural R&D.

Partnerships are largely directed by farmers and increase the capacities of all partners to address their challenges in an ever-changing world. From this work PROLINNOVA partners have started working towards the development of locally relevant indicators, offering a pool of local level indicators that considers local realities in rural areas of developing countries. Without doubt, these realities should be incorporated into innovation decision-making at the national level.

It is critical to develop an approach that combines national and internationally comparative indicators... with indicators generated by innovators and actors active at the local level.

This participatory approach emphasises the importance of involving local communities in innovation activities to stimulate social action, while allowing for self-learning, reflection, appropriate action and improved understanding by all actors involved. It also provides additional benefits to both researchers and the local innovators, because it is not only an additional source of valuable information for the researchers, but also an important source of self-learning and understanding for both parties.

To have a coherent and informed picture of innovation activities in a developing country, it is critical to develop an approach that combines national and internationally comparative indicators, such as those developed by the OECD, together with indicators generated by innovators and actors active at the local level. Top-down development of instruments and indicators is inadequate. Such an approach needs to be well balanced to ensure local needs and circumstances that direct local innovation activities and ultimately, outputs, are not ignored.

Figure 1 is an illustration of the indicator pyramid, an approach worth experimenting with when developing a methodology for STI indicators for African and other developing countries.

Figure 1: Indicator Pyramid

Source: Adapted from UNAIDS (2010); Sheu and Lo (2005); Letty, Shezi and Mudhara (2012)

The indicator pyramid consists of three levels of indicators: global, national and local. The top of the pyramid comprises global indicators such as the OECD indicators found in the Frascati, Oslo and other manuals that allow for international comparability among countries. National level indicators are those developed by national statistical agencies and research institutes, and are measured by means of large-scale surveys. Indicators at this level may also be developed by various research and academic institutions.

Often these indicators and measurement instruments do not differ from the ones used for global comparative purposes. At the bottom of the pyramid there are indicators that can be extracted from case studies and small sample

surveys. These are especially important to collect information about innovation at the local level. The indicator pyramid suggests that rather than standing alone, the local level indicators should be used to develop relevant indicators for the national and global levels.

Despite the involvement of numerous research organisations, the development of local level indicators that could contribute to a framework such as the pyramid indicator framework is lacking, or slow, at best.

Strengthening these contributions would enable the further development of stronger national and global level indicators, while ensuring that locally developed innovations were not ignored and their relevance to local people as part of their own attempts to improve their socioeconomic circumstances were acknowledged and given value.

Authors: Alexandra Mhula, junior researcher, Economic Performance and Development programme (EPD), HSRC; Tim Hart, senior research manager, EPD, HSRC; Peter Jacobs, senior research specialist, EPD, HSRC.