Learner Performance in South Africa pdf

Free download from www.hsrc
p
ress.ac.za
Compiled by the Research Programme on Human Resources Development,
Human Sciences Research Council
Commissioned by JET Education Services and funded by the Business Trust
Published by HSRC Press
Private Bag X9182, Cape Town, 8000, South Africa
www.hsrcpress.ac.za
© 2005 Human Sciences Research Council
First published 2005
All rights reserved. No part of this book may be reprinted or reproduced or utilised in
any form or by any electronic, mechanical, or other means, including photocopying
and recording, or in any information storage or retrieval system, without permission
in writing from the publishers.
ISBN 0-7969-2041-9
Cover by Flame Design
Produced by comPress
Distributed in Africa by Blue Weaver Marketing and Distribution,
P.O. Box 30370, Tokai, Cape Town, South Africa, 7966, South Africa.
Tel +27 +21 701-4477
Fax: +27 +21 701-7302
email:
Distributed worldwide, except Africa, by Independent Publishers Group,
814 North Franklin Street, Chicago, IL 60610, USA.
www.ipgbook.com
To order, call toll-free: 1-800-888-4741
All other enquiries, Tel: +1 +312-337-0747
Fax: +1 +312-337-5985

email:


Free download from www.hsrc
p
ress.ac.za
Contents
List of figures and tables iv
Executive summary v
Acknowledgements ix
Abbreviations x
1 Introduction 1
1.1 Structure of this report 2
2 Features of the Quality Learning Project 5
3 Data preparation, statistical procedures
and methodology 7
3.1 Introduction to the statistical analysis 7
3.2 Data preparation 7
3.3 Data reliability problems 9
3.4 Analytical strategy 13
3.5 School effects 15
3.6 Household effect variables 19
3.7 Removing the household effects from the school environment
and language and mathematics interest and experience variables 21
3.8 The set of explanatory variables 24
3.9 The language and mathematics test scores 24
3.10 Do richer communities have better schools? 27
3.11 Regression of language and mathematics scores on the
explanatory variables 28
4 Commentary on the findings 33

Appendices
Appendix 1 Districts participating in the QLP 35
Appendix 2 The QLP Model 36
Appendix 3 The programmes of the QLP 39
Appendix 4 Sampling methodology 41
Appendix 5 Developing and administering the instruments 46
Appendix 6 Conversion of data from the learner questionnaire into the
list of variables used in this study 52
Appendix 7 Learner background questionnaire 55
Appendix 8 List of schools ordered by general factor, Grade 11 factor
and Senior Certificate 73
References 76


Free download from www.hsrc
p
ress.ac.za
Figures
Figure 1: Survey sample 5
Figure 2: Education of mothers 11
Figure 3: Education of fathers 11
Figure 4: Household wealth vs household income 12
Figure 5: General school factor on household wealth 27
Figure 6: School test factor on household wealth 28
Figure A3.1: QLP programmes 40
Tables
Table 1: Total sample obtained for the baseline fieldwork in 2000 6
Table 2: Language most often spoken at home, by population group 9
Table 3: Education of mother and father, by grade 10
Table 4: Expected level of the household wealth score at different levels of income 13

Table 5: Analysis of variance of school environment, interest and learning experience
scores 16
Table 6: Correlation matrix and factor analysis of the regression coefficients of school-
level variables 17
Table 7: Regression coefficients for Senior Certificate pass rates on general and Grade
11 factors, 1999 and 2000 18
Table 8: Distribution of the use of the language of instruction at home 19
Table 9: Distribution of household scores 20
Table 10: Regression of study aids, meals, parental support, time use and home
reading scores on whether an African language is spoken at home, logarithm
of household wealth, parental support score (time use and home reading
only) and a Grade 11 dummy variable 21
Table 11: Regression of interest and experience residuals on home effects variables 22
Table 12: Correlation matrix and factor analysis of the regression coefficients of school-
level variables after the school and household effects have been removed 23
Table 13: Distribution of language and mathematics marks 25
Table 14: Factor analysis of test scores and Senior Certificate results 26
Table 15: Percentage test scores regressed on explanatory variables 29
Table 16: Language scores by quintile 31
Table 17: Mathematics scores by quintile 32
Table A 2.1: Outcomes and covariates at district level 37
Table A 4.1: Number of schools sampled per district 42
Table A 4.2: Number of learners assessed by school district 44
Table A 4.3: Total sample obtained for the baseline fieldwork in 2000 45
Table A 5.1: List of instruments 46
Table A 5.2: Test topics – mathematics, Grades 9 and 11 48
Table A 5.3: Reading and writing skills tested, Grades 9 and 11 49
Table A 5.4: Testing duration of the assessment instruments in minutes 50
iv
©HSRC 2005

List of figures and tables
v


Free download from www.hsrc
p
ress.ac.za
Introduction
Despite all the difficulties associated with the expansion of its educational system during
the twentieth century, South Africa has done well in systematically lengthening the
average education of each successive age cohort. By the 1990s, more than 90 per cent of
the 7–16 age group was enrolled in school, although not every learner was putting in a
full day’s attendance. But the quality of the output from the school system has left much
to be desired. For many years, close to half of the Senior Certificate candidates have
failed the examination outright. And international comparisons, such as the Third
International Mathematics and Science Study of 1995, have given no comfort. South Africa
was bottom of both the mathematics and science tables, even though a number of similar,
middle-income countries were included in the study.
In seeking to identify the reasons for this situation, it is important to relate educational
outputs (competencies, as measured for instance by Senior Certificate examinations or
standardised tests) to inputs. The most obvious input is that of the school itself – the
quality of teachers, facilities and management. Another input is that of the household –
the education of the parents, household income and wealth, and support for learners.
Education is a joint product of school and home, and learners who are backed by strong
household resources have an advantage. The abilities and proclivities of individual
learners is a third input; learners from the same household and the same school can end
up with very different profiles of achievement.
Determining the relative contributions of these inputs to educational outputs is not
straightforward. Partly this is a data problem. The information necessary to carry out a
comprehensive analysis is extensive and usually not fully available. Partly it is a statistical

problem: many of the explanatory variables are themselves related in complex ways, so
identifying the true drivers of the situation under analysis is difficult. Moreover, in South
Africa very little educational production function analysis has been undertaken, so there
are few landmark results from which one can take one’s bearings. It is no exaggeration to
say that educational production function analysis in South Africa is in a preliminary
exploration phase. The results of this study must be interpreted in that light.
Up until the Quality Learning Project’s (QLP) baseline study in 2000, no South African
data set had ever included test results, school characteristics and information on the
household circumstances of individual learners. Before that one could, as in the analysis
of Senior Certificate results, relate results to schools and schools to the communities
within which they were located. But one could not relate individual learners to the
households from which they came. The QLP data set therefore offers a new analytical
opportunity. The research question posed for this study was: what are the effects of
socio-economic variables on educational outcomes in the QLP schools?
Limitations of the study
Before describing the methods and the findings of the analysis, it is important to note two
limitations of this study. The first is that, within the time and resources available, it was
not possible to use the full QLP data set. This study, therefore, concentrates on the
learner background questionnaire (which elicited information on households) and the
learner achievement questionnaire. It did not use (with one small exception) the data
v
©HSRC 2005
Executive summary


Free download from www.hsrc
p
ress.ac.za
vi
©HSRC 2005

from the school or educator questionnaires. Of course, the schools attended by learners
were identified in the background and achievement questionnaires, so that school effects
could be identified. But they were not correlated with a wider set of characteristics of
schools. Thus schools appear in this study through the eyes of learners, rather than
through those of teachers or principals.
The second limitation of this study is the reliability of the information about households.
The information collected covered:
• Education of parents;
• An indicator of household wealth, as evidenced by items within the home;
• Language mostly spoken at home;
• How often the language of instruction is spoken at home;
• Frequency of meals;
• Reading at home;
• Parental practices supporting education;
• Time use.
Unfortunately, the data on the education of parents turned out to be unusable. Preliminary
regression results produced incoherent results and checking of average educational
levels in the immediate environment of the school from the 1996 Census suggested that
educational levels had been exaggerated, particularly by Grade 9 pupils. The exaggeration
was probably worse at the bottom than at the top. It was also not systematic, so that there
was no way of working back from reported to actual education. The variable had to be
discarded – a serious blow, since it is a very important one. Its absence makes the
findings of this study less certain than they would otherwise be. There may also have
been some (but less fatal) exaggeration of items contained within the home.
The conclusion from this experience is that household variables really need to be
collected from adults within the households rather than learners at schools. The reporting
of educational levels in population censuses since 1960 has been reasonably accurate and
is certainly much more accurate than the QLP data.
The explanatory variables
A great many questions in the learner background questionnaire dealt with the school

environment, and the interest and experience of learners in the language of instruction
and mathematics in both Grade 9 and Grade 11. These data were combined into nine
indices (one for school environment and eight covering interest and experience by grade
and subject). Each index measured positive or negative orientation of the learner towards
the topic in question.
The first question becomes: how much of the variance in indices could be explained by
the schools themselves? The answer turned out to be between 10 and 30 per cent. The
correlations between indices were such that one could identify the most generally
favoured schools and relate them to performance in the 1999 and 2000 Senior Certificate
examinations. Schools approved by pupils generally did better in the Senior Certificate
examinations, though the correlation was not perfect. So one can conclude that the
indices, in part, measured something about the school. But the 70 per cent or more of
unexplained variance meant that they measured other things as well. Some of the
remaining variance was explained by differences between households. The rest can be
Learner Performance in South Africa


Free download from www.hsrc
p
ress.ac.za
ascribed to individual learner motivation and morale. The statistical analysis enables one
to separate out which portion of the indices should be ascribed to which levels.
Turning now to the household – the main entity of interest in this study – it quite rapidly
emerged that language use affected test results. If the language of instruction was used a
lot in the home, test results rose. This really was an issue for households in which an
African language was mostly spoken at home. For others, the language mostly used at
home and the language of instruction was likely to be the same. The other fundamental
household variable is household wealth. One could then start to relate other household
variables to these two. Thus:
• The study-aid score rises with household wealth as does the meal score (though at a

given level of household wealth, this is lower for African-language households);
• Interestingly, parental support for study is stronger among African-language
households and rises with household wealth. It drops from Grade 9 to Grade 11;
• Time-use patterns conducive to study are likely to be better if an African language is
spoken at home, better as household wealth and parental support rises, but worse
in Grade 11 than in Grade 9. Reading at home rises with household wealth and
parental support, but is less extensive in African-language households.
By using the household explanatory variables one could remove the effects of households
as well as schools on the school environment and interest/experience indices. The
correlation among the purged indices threw up three patterns:
• A general orientation to school;
• An interest in mathematics;
• A positive interest in subjects, but negative experience of them.
The analytical approach enabled one to separate out the data into individual-level,
household-level and school-level information, in such a way as to reduce the correlations
between the explanatory variables. Other variables were added, namely: gender and age
for grade in the case of learners and urban or rural location in the case of schools.
The test results
The test results had the following characteristics:
• They were higher for the language of instruction than mathematics;
• There was a considerable deterioration in language scores between Grade 9 and
Grade 11. This finding is significant because experts, at grade appropriate levels, set
the tests. The achieved improvement (off a low base: the median mark in Grade 9
was 38 per cent) is not equal to the expected improvement;
• The level of mathematics performance in Grades 9 and 11 was approximately the
same and extremely poor (median in Grade 9 was 7 per cent and in Grade 11 was
8 per cent). Eighty per cent of pupils scored less than 15 per cent in Grade 9 and
16 per cent in Grade 11. Even at the 95th percentile, pupils achieved a mark of less
than 30 per cent in both grades.
There is a significant but not perfect correlation between school achievement in the tests

and its achievement in the Senior Certificate examinations of 1999 and 2000. Tests and
Senior Certificate results are measuring approximately the same thing. There are also
significant correlations between:
vii
©HSRC 2005
Executive summary


Free download from www.hsrc
p
ress.ac.za
• The school general quality factor perceived by pupils and the mean of the logarithm
of household wealth derived from pupils at the school;
• The school achievement quality factor (derived from test results and Senior
Certificate results) and the mean of the logarithm of household wealth.
Not surprisingly, richer communities tend to have better schools.
The relation between test results and the explanatory variables
Language of instruction
Individual level variables: the ones which matter are general positive orientation (which
has a positive effect) and the over-age for grade variable (which has a negative effect).
The effects are stronger at Grade 9 than Grade 11. The gender coefficient is not
significantly different from zero at the 5 per cent level.
Household level variables: African home language is a marked disadvantage at the Grade
11 level. Whether the language of instruction is spoken at home often is a marked
advantage. Household wealth exerts a positive effect. Study aids and meals exert a
positive influence over and above the expected level.
School level variables: The school general factor exerts a positive influence. Other things
being equal, rural schools do better than urban schools.
Mathematics
Because the average result is so poor, school and household variables explain

considerably less of the mathematics score; individual ability and proclivity is the main
cause of such variation in marks. This is confirmed by the result when one adds in the
language mark residual (after school and home effects have been removed).
Conclusions
The findings of this study can be generalised only for QLP schools and not the national
school system. Suburban (as opposed to township) schools are entirely unrepresented in
the QLP universe. QLP schools are drawn from the bottom 70–80 per cent of schools, but
they have not been randomly drawn. Inferences to the wider school system would be risky.
Nonetheless, the general finding in this study – that social and economic variables do not
play an enormous role in determining performance at the individual level, with the
exception of language variables – deserves further investigation. Moreover, included in
the set of social and economic variables are behavioural variables that can change at
existing levels of household incomes and wealth. The rules to be followed are simple:
• Feed learners as well as possible;
• Equip them with a full range of inexpensive study aids;
• Talk to them often in the language of instruction.
By contrast, household wealth does not give much of an edge in school performance.
And competence in the language of instruction is valuable not only in itself, but as a
means to improved mathematics performance.
viii
©HSRC 2005
Learner Performance in South Africa


Free download from www.hsrc
p
ress.ac.za
Acknowledgements
Nick Taylor and Mark Orkin played a pivotal role in conceiving this research project in
discussion at a meeting between the HSRC and Joint Education Trust (now JET Education

Services) staff in early 2001.
We thank the following people for their various contributions to the development of this
report, in alphabetical order:
• Nicolaas Claassen and Godwin Khoza for their inputs;
• Helen Perry for her useful comments and suggestions on the draft and for providing
an introduction to production functions;
• Jacques Pieterse for his advice on statistical aspects;
• Phumudzo Singo for administrative assistance;
• Elsie Venter for data cleaning, preparation and for running the procedures;
• Penny Vingevold for her support and critical input;
• Mariette Visser for preparing the Senior Certificate data;
• All those attending the presentation of the first full draft of this paper, including
Carol Deliwe.
ix
©HSRC 2005


Free download from www.hsrc
p
ress.ac.za
Abbreviations
FRC Free reponse question
JET Joint Education Trust
MCQ Multiple choice question
NBI National Business Initiative
NGO Non-governmental organisation
QLP Quality Learning Project
SCE Senior Certificate Examination
x
©HSRC 2005

Free download from www.hsrc
p
ress.ac.za