Use what you have to get what you need
The appropriateness of TIMSS to appraise and inform policy
Education is a key priority in the country, and outcome one in the presidential delivery agreements. It is therefore important to use reliable data to appraise the well-being of the system to date. Shawn Rogers and Lolita Winnaar explore how the TIMSS design and framework generate data that provides invaluable information on the nature and extent of learner achievement, and measure improvement, quality and outcomes of the education system.
The Trends in International Mathematics and Science Study (TIMSS) has been undertaken in South Africa in 1995, 1999, 2002 and more recently, in 2011. TIMSS is one of the first studies to provide international and national learner assessment trend data for mathematics and science, and places quality of education in the eye of politicians, policy makers and the public.
Why do we need assessments?
Although South Africa has made significant progress towards rectifying the issues of its past, its society remains highly unequal with limited access to quality public education. Government has stated it intends to improve the quality of the education system and provide the younger generation with better opportunities and a brighter future.
The Education Action Plan 2014 Towards the Realisation of Schooling in 2025 is a policy response by the Department of Basic Education (DBE) to strengthen the education system. The action plan focuses on short-term goals, which it aims to reach by 2014, towards the realisation of longer-term goals in schools by 2025.
To achieve these objectives, DBE has established goals related to learning, enrolment and other areas identified in the system that require improvement.
- Improving the minimum quality standards and increasing the number of learners in the different grades (grades 3, 6, 9, 12);
- Improving the average performance for literacy and numeracy;
- Improving the access and progression of learners through the system;
- Improving teacher training and teacher supply and demand, upskilling teachers and ensuring that they are prepared and able to cover the curriculum; and
- Improving access to quality learning materials/resources, improving school infrastructure and support services, and engaging with the community to improve the learning environment.
TIMSS provides invaluable and reliable data on factors directly linked to learning that could improve learner outcomes by benchmarking within the country and comparing internationally.
Accountability through assessment
Studies such as the International Association for the Evaluation of Educational Achievement’s (IEA) TIMSS study are used to evaluate the improvements and outcomes of policy-implemented changes to the education system. TIMSS is one of the largest nationwide achievement studies, and provides invaluable and reliable data on factors directly linked to learning that could improve learner outcomes by benchmarking within the country and comparing internationally.
TIMSS has a very robust methodological design and framework that allow for both valid and reliable data. This makes it an excellent study to rely on when trying to establish the level of learner achievement in the country so as to compare this to the international standards of approximately 60 other countries. The objective of the study is to determine the impact underlying contextual factors have on learner achievement, with the hope of informing policy and offering a positive change.
The curriculum model grounds the TIMSS framework and links the data collection to national ideals of what the learner is expected to learn (intended), and the actual opportunities offered by schools and teachers (implemented) with what skills the learner masters (attained). Therefore, the intended, implemented and attained curriculum model shows the relationship between learner achievement and the impact of national educational policy, teacher and classroom practices, and the learner’s home environment.
The South African TIMSS 2011 selected 298 schools with approximately 12 000 learners. The data provided information on national and provincial performance, performance by language of learning and teaching (English, Afrikaans), performance by school type (public, Dinaledi and independent schools), and by school poverty index (quintile ranking).
The learner achievement scores were gathered using achievement instruments, which the learners had to complete. These included mathematics and science subject matter, and differentiated between specific cognitive domains. The cognitive domains assessed the learners’ thinking processes, specifically knowing, applying and reasoning.
The TIMSS assessment booklets are developed using a matrix sampling design, which evenly distributes the different subject and cognitive domain blocks across 14 different booklets. This complex design increases the reliability of the data, ensuring that learners are unable to replicate each other’s work, as the booklets are all varied and it is unlikely that two learners in close proximity will have the same booklets.
Mathematics items cover numbers, algebra, geometry, and data and chance. Science includes biology, chemistry, physics and earth science.
More notably, each of these booklets contains the all-important trend items. These items are never released to the public and remain consistent throughout every TIMSS cycle. These enable countries to get the same measure of learners’ achievement rates over time that are used to gauge improvements and the effectiveness of implemented policies and programmes.
TIMSS gathers information about each learner’s social and educational environment using student background, teacher, school and curriculum questionnaires. The aim of this is to determine what specific contextual factors (according to the curriculum model) have the greatest impact on learner achievement, with a view to evaluate government goals and improve learning.
These questionnaires cover four broad areas, namely the national and community context (cultural, social, political and economic factors), school context (indicators of school quality and effectiveness), classroom context (teachers, classroom characteristics and resources) and student characteristics and attitudes (experiences, expectations, demographics as well as attitudes towards learning).
During the data collection phase, the international TIMSS team provides detailed operational manuals, which include strict guidelines on all procedures and preparations for administrating the instruments to ensure the data collected is valid, reliable and standardised for accurate comparability across countries.
TIMSS is dynamic and beneficial, placing countries in a position to quantify the impact interventions and policies have had on the quality of education and the performance of the learners over time.
All procedures are monitored by IEA-trained national and international representatives in selected and previously undisclosed schools. The quality assurance continues throughout the scoring of the assessment booklets, the double-marking of selected scripts, and exchanges between northern and southern hemisphere countries. This is to validate the scorer reliability within and across countries, once again strengthening the quality of the TIMSS data.
Additional verification is done at the data processing stage, whereby all of South Africa’s collected data is captured twice and compared to the original data, to confirm the information is accurate and not contaminated by human error. The rigorously cleaned data set is then sent to the Data Processing and Research Center (DPC) in Germany for further cleaning and verification. Once released, countries are able to undertake analysis and evaluate the changes that have occurred in their countries.
Overall, TIMSS is dynamic and beneficial, placing countries in a position to quantify the impact implemented interventions and policies have had on the quality of education and the performance of the learners over time.
Authors: Shawn Rogers, PhD intern, and Lolita Winnaar, Chief Programmer, Education and Skills Development research programme, HSRC.