Short abstract

Are A levels predictive of outcome?

Many beliefs are strongly held about undergraduate student selection but without any “visible means of support”⁴: one is that A levels are not predictive of outcome at university. The opposite is true. A study of 79 005 18 year old students entering university in 1997-8 and followed through until 2000-1 (www.hefce.ac.uk) shows a clear relation between A level grades and university outcome (fig 1). The result is compatible with many other studies of students in general,⁵w5 w6 and of medical students in particular.^5-8w7-w10 Small studies of individual students in individual years at individual institutions are unlikely to find such correlations—the reasons being statistical, including lack of power, restriction of range, and attenuation of correlations caused by unreliability of outcome measures (see bmj.com).

Fig 1

Outcome of students in relation to A level grades in all subjects. Grades are based on best three. A=10, B=8, C=6, D=4, and E=2 points

Why are A levels predictive?

The three broad reasons why A levels may predict outcome in medicine are: cognitive ability—A levels are indirect measures of intelligence, and intelligence correlates with many biological and social outcomes⁹; substantive content—A levels provide students with a broad array of facts, ideas, and theories about disciplines such as biology and chemistry, which provide a necessary conceptual underpinning for medicine; and motivation and personality—achieving high grades at A level requires appropriate motivation, commitment, personality, and attitudes, traits that are also beneficial at medical school and for lifelong learning.

Cognitive ability alone cannot be the main basis of the predictive ability of A levels, because measures of intelligence and intellectual aptitude alone are poor predictors of performance at university. This is not surprising, as an old axiom of psychology says that “the best predictor of future behaviour is past behaviour,” here meaning that the future progress in passing medical school examinations will best be predicted by performance in past examinations. Of course success in medicine and being a good doctor are not identical nor are either of these the same as passing medical school examinations; but those who fail medical school examinations and have to leave medical school never become doctors of any sort. The predictive value of A levels most likely results either from their substantive content, their surrogate assessment of motivation to succeed, or both.

Separating the substantive and motivational components of A levels is straightforward in principle. If the substantive content of A levels is important for prediction then there will be a better prediction of outcome from disciplines underpinning medical science, such as biology and chemistry, than there will be from other subjects—for example, music, French, or economics. Alternatively if motivational factors are the main basis for the predictive power of A levels, indicating pertinent personality traits and attitudes such as commitment, the particular subject taken will be less relevant and an A grade in music, French, or economics will be found to predict performance at medical school as well as an A grade in biology or chemistry. Few analyses, however, differentiate between these factors. But evidence is increasing that A level chemistry is a particularly good predictor of performance in basic medical science examinations^6,10 (although not all studies find an effect^{w7 w11}), and A level biology also seems to be important.^6,11 As almost all medical school entrants take at least two science A levels, however, one of which is chemistry, this leaves little variance to partition. It should also be remembered that there are probably true differences in the difficulty of A levels, with A grades easier to achieve in subjects such as photography, art, Italian, and business studies, than in chemistry, physics, Latin, French, mathematics, biology, and German.¹² A clear test of the need for substantive content will occur should medical schools choose to admit students without A level chemistry, as has been suggested,^w12 or perhaps without any science subjects. If students with only arts A levels perform as well as those with science A levels, then the substantive content of A levels is unimportant—and there are suggestions that arts and humanities subjects independently predict outcome.^{13 w13}

What do intellectual aptitude tests do?

Aptitude has many meanings,¹⁴ but the glossary to the Schwartz report says that aptitude tests are “designed to measure intellectual capabilities for thinking and reasoning, particularly logical and analytical reasoning abilities.”³ Aptitude, however, also refers to noncognitive abilities, such as personality. We therefore talk here of “intellectual aptitude,” both in the sense described by Schwartz and in the meaning used in the United States for what used to be known as scholastic aptitude tests (SATs; but now the “A” stands for assessment) and which are largely assessments of intellectual ability.^w14

Most intellectual aptitude tests assess a mixture of what psychologists call fluid intelligence (logic and critical reasoning, or intelligence as process¹⁵) and crystallised intelligence (or intelligence as knowledge, consisting of general culturally acquired knowledge of vocabulary, geography, and so on). We thus consider intelligence tests and intellectual aptitude tests as broadly equivalent, but distinguish them from achievement tests, such as A levels, which assess knowledge of the content and ideas of academic subjects such as chemistry and mathematics.

Although purveyors of tests such as the biomedical admissions test (table) argue that they are not measures of intelligence but of “critical thinking,” there is little agreement on what critical thinking means.^w15-w17 Critical thinking is related more to aspects of normal personality than it is to IQ and reflects a mixture of cognitive abilities (for example, deductive reasoning) and dispositional attitudes to learning (for example, open mindedness).¹⁶ Evidence also shows that critical thinking skills, not dispositions, predict success in examinations¹⁷ and that critical thinking may lack temporal stability.¹⁸ The content and the timed nature of the biomedical admissions test suggest it will correlate highly with conventional IQ tests.

What do intellectual aptitude tests predict?

We know of three studies that have compared intellectual aptitude tests with A levels (see bmj.com).

The investigation into supplementary predictive information for university admissions (ISPIUA) project⁵ studied 27 315 sixth formers in 1967, who were given a three hour test of academic aptitude (TAA); 7080 entered university in 1968 and were followed up until 1971. The results were clear: “TAA appears to add little predictive information to that already provided by GCE results [A levels and O levels] and school assessment in general.”

The Westminster study¹⁴ followed up 511 entrants to the Westminster Medical School between 1975 and 1982 who had been given a timed IQ test, the AH5.^w18 Intellectual aptitude did not predict outcomes measured in 2002, whereas A level grades were predictive of both academic and career outcomes.

The 1991 cohort study looked at 6901 applicants to UK medical schools in 1990, of whom 3333 were admitted^{w19 w20} and followed up.^{w21 w22} An abbreviated version of the timed IQ test was given to 786 interviewees.^w18 A levels were predictive of performance in basic medical science examinations, in final clinical examinations, and in part 1 of a postgraduate examination, whereas the intellectual aptitude test was not predictive (see bmj.com).

In the United States (presently outside the hegemony of the A level system) a recent study of dental school admissions¹⁹ evaluated an aptitude selection test, carefully founded in the psychology of cognitive abilities and skill acquisition. The scores were of no predictive value for clinical achievement at the end of the course.

Do aptitude tests add anything to what A levels already tell us?

In interpreting the validity of aptitude tests it should be acknowledged that some aptitude tests are not content free and do assess substantive components. For instance, the biomedical admissions test seeks to test aptitude and skills (problem solving, understanding argument, data analysis, and inference) in section 1 and scientific knowledge and applications in section 2. Section 2 contains questions on biology, chemistry, physics, and mathematics (www.bmat.org.uk) at the level of key stage 4 of the UK national curriculum (www.nc.uk.net). Because most applicants for medical school are studying many of those subjects for A level, section 2 may be a better predictor of university outcome than is section 1, because it is indirectly assessing breadth of scientific background (and in all likelihood it will be found to correlate with A level grades, but not necessarily to add any predictive value).

On balance, A levels predict university achievement mainly because they measure the knowledge and ideas that provide the conceptual scaffolding necessary for building the more advanced study of medicine.^{20 w23} As with building a house, the scaffolding will later be taken down and its existence forgotten, but it will nevertheless have played a key part in construction. Motivation and particular personality traits may also be necessary, just as a house is built better and faster by an efficient, conscientious builder. However, intellectual aptitude tests assess neither the fundamental scientific knowledge needed to study medicine nor the motivation and personality traits. Pure intellectual aptitude tests only assess fluid intelligence, and empirically that is a weak predictor of university performance. Intelligence in a builder, although highly desirable, is no substitute for a professional knowledge of the craft of house building.

What are the problems of using A levels for selection?

Reported problems in using A levels in selection are threefold: the increasing numbers of candidates with three A grades at A level; social exclusion, and type of schooling.

A continuing concern of the UK government is that entry to medical school is socially exclusive.²¹ The class distribution of entrants has been unchanged for over half a century, with a preponderance of applicants and entrants from social class 1.^w24 It is not clear whether the entrant profile reflects bias by selectors,^w25 an active choice not to apply for medicine by those from social classes IV and V,^w26 or an underlying distribution of ability by social class.^w27

Although most children in the United Kingdom attend state schools, the minority attending private (independent) schools are over-represented among university entrants, possibly because as a group they achieve higher A level scores.

Can these problems be fixed?

Type of schooling

To tackle possibly unfair over-representation of entrants from independent schools, a case has been made for university selectors taking into account type of school and the relative performance of the school: achieving three grade Cs at A level in a school where most pupils gain three grade Es may predict university achievement better than gaining three grade Cs in a school where most pupils gain three grade Bs. Detailed analyses by the Higher Education Funding Council for England, however, show that after taking into account the grades achieved by an individual student, the aggregate achievement of the school from which the student has come provides no additional prediction of university outcome (see bmj.com). The funding council has good evidence to show that on aggregate, pupils from independent schools under-perform at university compared with those with the same grades from state schools (fig 2).

Fig 2

Predictive value for gaining first class or upper second class degree of A levels obtained at different types of school (see bmj.com for definition of school types)

Intellectual aptitude tests are not a solution to this problem. A solution might be to upgrade the A level grades of applicants from state schools so that, say, one A grade and two B grades are treated as equivalent to two A grades and one B grade from an independent school applicant (that is, increasing by 20 points on the new tariff of the Universities and Colleges Admissions Service (www.ucas), or by 2 points on the older scheme shown in figure 2). Any system should also take into account that many pupils are at independent schools until age 16 and then transfer to (state) sixth form colleges for their A levels. A proper, holistic assessment of each student will, however, require more information than is readily available on the present admissions form.

What is the potential value of non-cognitive aptitude tests?

The aptitude tests we have considered are those that assess cognitive skills. Other skills, however, are needed by doctors, such as the ability to communicate and to empathise, having appropriate motor and sensory skills, and having the appropriate attitudes and ethical standards necessary for professionalism (see for example, www.pqa.net.au). None of these is disputed, and there are strong arguments that selectors can and should take such measures into account at the same time as they are assessing the intellectual skills necessary for coping with the course.^{w28 w29} We have not considered such aptitudes in detail here because few UK medical schools are as yet using them in selection (as opposed to research and validation), although initiatives are in progress, both in the United Kingdom and elsewhere.^{w30 w31} Situational selection tests have been used in five Flemish medical and dental schools and were found to predict performance in the final first year's examinations better than tests of cognitive ability.²³

It is also the case, however, that if selection is to be made on the basis of several independent characteristics, then the extent of selection on each is inevitably lower than if there is selection only on any one of them,²⁴ until eventually the selective power of each is so reduced that “if you select on everything you are actually selecting on nothing.”^w32 An attractive argument is that if most students with three grade As at A level (or indeed even lower grades) can cope with a medical course, then instead of looking for selection using A+ and A++ grades, medical schools should be selecting from the existing pool more systematically on non-cognitive measures. That will require validation of the measures, but it might result in a cohort of doctors who are not only academically able but also well suited to medicine because of temperament and attitude. Whether the slight potential decrease in the academic qualifications of entrants will be offset by their increased suitability on non-cognitive measures will depend on a precise knowledge of the relation between academic ability, suitability, and examination performance, and in particular whether these are linear or non-linear and show thresholds. It is an important question that must be answered by empirical study.

Supplementary Material

Notes

References