*B. Spolsky, Measured Words
Language tests and the UK Border Agency
In response to a parliamentary question, Damian Green, UK Minister of State (Immigration), is reported to have stated:
As part of its current exercise to revise its list of secure English language test providers, the UK Border Agency conducted a verification exercise of the mapping of prospective providers of secure English language tests to the CEFR. Dr North (one of the CEFR co-authors) was invited to play the role of an independent expert on 10 December 2010 and the consultation took place during the second week of March 2011.
Quite deservedly, since this news was announced, there’s been some very heated discussion on the Language Testing Research and Practice (LTEST-L) mailing list. In brief, the facts are as follows:
- The UK Border Agency requires that immigrants (including students) produce evidence of competence in the English language at a stated CEFR level.
- Acceptable evidence is in the form of a test score or certificate from an approved test provider.
- Whether a particular test score or certificate proves that the holder is at the required CEFR level can be determined via a “verification exercise of the mapping” of the test to the CEFR; this verification exercise can be in the form of a consultation with an expert, as long as the expert is “one of the CEFR co-authors”
The CEFR and the Use of Language Tests
These are rather disturbing developments for the ELT community as well as society at large. Here is why:
- The UKBA’s decisions on what minimum CEFR levels are acceptable for different types of migrants (Tier 1 workers, Tier 2 workers, students, etc) are quite arbitrary – this was proven in quite a spectacular manner when the UKBA had to change their minds last summer about whether B1 or B2 was a more appropriate threshold for foreign students.
- The UKBA’s decisions on which examinations and which scores constitute proof that one has reached the arbitrarily determined level are equally arbitrary: some of the approved test providers have produced evidence of having completed a CEFR mapping project as suggested in the Council of Europe’s Manual for Relating language exams to the CEFR, while others have not; in addition, there are examination providers who have produced such evidence, but their examinations are not on the UKBA approved list.
- Even when such mapping has been undertaken, it is questionable that a link has been established between the CEFR levels and the levels that the test tasks reflect or the levels of the cut scores of the various “CEFR-mapped” examinations. This is mainly because the CEFR-mapping procedure descibed in the Council of Europe’s manual is actually very much a fool-proof exercise, which guarantees that once the stages of the “linking” process have been completed, some sort of link will have been established; in other words, it is not possible to do the mapping exercise and conclude that there is no link between your exam and the CEFR!
- The CEFR levels themselves do not represent the nature of communication or the stages of acquisition of a foreign language; in the words of Dr North (the same Dr North that was involved in the UKBA consultation that started the heated discussion on LTEST-L), one of the co-authors of the CEFR, the CEFR levels are merely “teachers’ perceptions of language proficiency (appropriate for a common framework of reference) not validated descriptions of SLA processes.”
- Even if the CEFR levels did represent actual stages of language acquisition, rather than the collective impressions of teachers, linking an examination to the CEFR does not make the examination valid. To prove that a test is valid, you have to provide evidence that the decisions based upon the test are rational (cf. McNamara, T and C Roever. 2006. Testing: the social dimension, Chapter 2). Unfortunately, what often happens is that examination bodies use CEFR-mapping procedures in lieu of validation rather than in addition to validation of their tests.
- In actual fact, the uses to which the UK Border Agency puts some of the tests on its “approved” list constitute, in themselves, an invalidating argument: a test designed to measure competence in Academic English cannot be a valid measure of an unskilled worker’s ability to use English for occupational purposes, even if that test has hastily been claimed to be linked to the CEFR!
The CEFR as Shibboleth
In the critical language testing literature, mention is often made of the biblical pronunciation test of the Shibboleth (Judges, 12:6): in the biblical account, those who could pronounce “shibboleth” correctly passed the test, proved their origin and were spared their lives; those who failed were instantly killed, all forty two thousand of them. The present-day Shibboleth is, it appears, not the language test itself, but the CEFR and its test alignment procedures: should a language exam prove to be “aligned” to the CEFR, it can live and prosper, as CEFR mapping has become all that is required for the exam to gain recognition and therefore be commercially viable; if an exam is not linked to the CEFR, or cannot, or does not, claim to be, it does not seem to matter whether it is valid, whether it is fair, whether it serves the purpose for which it was designed: it will not be recognised and the examinees that have successfully taken the exam will not be given, for instance, the opportunity to study, work or live in the country of their choice.
