A-Levels in England

I have not been following this closely enough to know accurately what’s going on, but let me try. On twitter there’s been some discussion that I entered in to, but Twitter’s format is too short to be good for discussing this, so I’ll post a reply to some questions here.

This kerfuffle started because in the UK (maybe just England; I don’t know if Scotland and Northern Ireland, and Ireland, for that matter, all use the same system), they decided to cancel the A-level end-of-high-school tests. These are subject matter tests, e.g. Math, Physics, Spanish, Chemistry, that are uniform across schools. You get a grade of A*, A, B, C, etc., not a whole lot of different possible grades, and each student decides which subjects he wants to test in. These are used by colleges to decide who to admit. They function like the American SAT and ACT tests that way, and like the tests used in China and Japan (though in Japan, at least, I think top places like the U. of Tokyo and U. of Kyoto have their own tests too).

In America, state bar exams for new lawyers faced the same decision, and many of them were cancelled, and there’s been discussion of whether to just let students who finished their JD degrees become lawyers without having to take any test.

It’s unclear to me why anybody has cancelled the tests in the first place. It seems entirely hysterical. These are highly important tests which don’t, I think, require more than a couple of days of sitting in a giant room, each student with his own desk, and these are students, so practically none of them are at risk of serious illness even if the room or stadium or open field they were in had bad ventilation and they allowed people sick with covid and coughing to take the test without masks or special seating, which they wouldn’t. This is one of the least dangerous group activities I can think of— they’ll all be sitting there utterly silent, typing away or scribbling away, or filling in dots with a pencil, not talking or cheering or singing or hugging each other or dancing or even panting (except for the really nervous ones). But there are a lot of fear-crazed people out there.

Anyway, back to A-levels. They were cancelled. What should we do about admission to UK colleges and universities? The main criterion is gone. I think — though please correct me– that what they’ve done is to use other things to replace A-level grades, but rather than use them directly, they’ve used them to generate an index, which they call an estimated A-level. In particular, they take (a) the student’s score on a practice A-level taken before covid, probably without much security against cheating and without the student thinking it would “count” for anything, and (b) the high school teachers’ opinion of how well each student would do if he were to take the real A-level test. Crucially, they also use (c) what high school the student went to.

So far, this is uncontroversial, up to (a) and (b). Rather than just let everyone into Oxford, they’re trying to figure out which students would have gotten A* on their chemistry A-level. And they’re using reasonable data for that estimate.

But many people don’t think they use use data (c)— the high school the student went to. They think all schools should be treated the same.

Why not treat all schools the same? Because getting an A* on the practice exam and an A* teacher opinion at School Uno doesn’t mean the same thing as A*’s from School Duo. Suppose School Uno takes the practice exam really seriously and teaches to the test, drilling relentlessly. Suppose School Uno tells its teachers to say all the students will get A*’s, because it wants to get its students into Oxford, or the teachers there are systematically overoptimistic about how well their students will do on the A-level just because some people are systematically overoptistic about life generally. Then treating the A*’s from schools Uno and Duo the same is bad in every way. It rewards bad behavior in preparing for the practice test and biased opinions and outright cheating on the teacher reports.

So, from what I can tell, the official agency has used a regression model, a standard, totally objective, and completely basic (not fancy) statistical technique to estimate the A-levels. This lets the data determine the answer, with no opinion input by the analyst. Essentially it looks at what happens on average. If in 2019, the average Uno teacher rec of A* has become a B on the actual chemistry test, the regression spits out an B for Uno, but if the Duo recommendation of A* always comes out A* the Duo student will get a predicted grade of A*. To be sure, even though the average Uno A* became B, some of them become A*, some A, some B, some C, and some D. Thus, the estimate of B at Uno will be unfair, one might say, to the students who would have gotten A* or A. Of course, they will also be unfairly generous to the students who would have gotten C or D. They’re only an estimate. But it’s fairer than if schools Uno and Duo were treated the same.

The complaint, though, has another element to it. Apparently the schools with poor kids tend to be Uno and with rich kids tend to be Duo. Thus, a teacher’s opinion of A* at a poor school carries less weight, because on average the teachers at poor schools tend to oversell their students. This is thought to be unfair to the students at the poor schools. Actually, it’s hard for to me to see why, if the estimates are good, which I think nobody is denying. Really, the complaint is that kids at poor schools do worse on the A-levels than kid at rich schools.

A separate question is what the university should do with the A-level grades once it receives them. I have heard mention that a student from a poor school with a chemistry B does better on average than a student from a rich school with a chemistry B. That is quite plausible (the reverse is also plausible; it’s just a fact, which could go either way). In that case, maybe Oxford should accept applicants with a B from a poor school when thy reject applicants with a B from a rich school. This is actually another application of statistics, maybe even formally using that same technique of linear regression: the A-level grade means something different depending on which high school you went to.

Am I wrong on any of this?