Aptitude Tests: when will they have worked?

I remain agnostic about aptitude tests. If they truly are good indicators of aptitude, and if they can encourage greater diversity in the profession, then there is significant merit in them especially if they inform decisions by chambers (which will be the real practical test of their ultimate success).

There are, however, significant, but not impossible to meet, ifs. The Bar Standards Board’s announced delay in their potential implementation got me thinking anew about this (see Neil Rose here). As did Rachel Rothwell’s piece in the Gazette which is a somewhat premature pat on the back for the BSB. It is premature principally because we would need to know the answer to these questions if we are to be sure they have mastered their brief:

1. What data is the BSB collecting to validate aptitude tests (e..g. what data on existing characteristics and achievements are being collected)?
2. What sophistication of analysis is being undertaken to assess the tests?
3. Is the research to be subject to peer review?
4. Is the data to be made available (suitably anonymised) for others to test their analysis and to conduct further research in the area (to the benefit of all)?
5. What are the key thresholds that they expect to see passed if the test is to have succeeded?

This last one may seem overly precious but it is actually quite important: one would expect there to be some correlation between an aptitude test and performance on the BPTC but how strong does it need to be to ‘work’? How have they decided that level or are they going to rely on the level of correlation they find? And similarly, is there existing, alternative information available which would perform as good, or almost as good, a basis for deciding who has the relevant aptitude (such as degree award)?

One could think of other things which may also be important. to know what research design is used to inform the testing (for instance do they intend to compare groups who do and don’t take the tests? How are they controlling for potential biases in who do and don’t sit the test?).

It would be interesting to know now what the answers to these questions are so that, come decision day, we can judge aptitude tests on their merits.