Aptitude Tests: the BSB gives some answers

A little while back I blogged (again) on aptitude tests.suggesting that if we were to be convinced the Bar Standards Board were on top of the testing of aptitude tests  they should be able answer to these questions:

1. What data is the BSB collecting to validate aptitude tests (e..g. what data on existing characteristics and achievements are being collected)?
2. What sophistication of analysis is being undertaken to assess the tests?
3. Is the research to be subject to peer review?
4. Is the data to be made available (suitably anonymised) for others to test their analysis and to conduct further research in the area (to the benefit of all)?
5. What are the key thresholds that they expect to see passed if the test is to have succeeded?

This last, I pointed out, may seem overly precious but it is actually quite important: one would expect there to be some correlation between an aptitude test and performance on the BPTC but how strong does it need to be to ‘work’? How have they decided that level or are they going to rely on the level of correlation they find? And similarly, is there existing, alternative information available which would perform as good, or almost as good, a basis for deciding who has the relevant aptitude (such as degree award)?

One could think of other things which may also be important.  For instance, to know what research design is used to inform the testing (for instance do they intend to compare groups who do and don’t take the tests? How are they controlling for potential biases in who do and don’t sit the test?).

I then said, it would be interesting to know now what the answers to these questions are so that, come decision day, we can judge aptitude tests on their merits.  You can judge the answers for yourselves.

Well, I am pleased to say, the BSB have responded to my questions and – in a the spirit of openness and engagement – I give you their replies (with permission):

1. What data is the BSB collecting to validate aptitude tests (e..g. what data on existing characteristics and achievements are being collected)?

As part of the Wood Review of the BVC (2008-09) we looked in detail at Aptitude testing – including but not exclusively, the LNAT test and various testing systems used in medical training.  The BSB cannot collect  data on existing characteristics  since new forms of the test are being written especially for our purpose. The purpose of the first pilot (2010) was to test basic viability. The purpose of the current (ie 2^nd) much wider pilot is not only to test the test itself, but also the individual questions to ensure that they are effective. The comparison between the Aptitude Test and students’ final results on the Bar Course should provide the BSB with the information necessary to confirm whether the test is valid or not.

2.       What sophistication of analysis is being undertaken to assess the tests?

The BSB has engaged the services of an independent consultant (since 2009) to undertake specific and detailed analyses of the pilot results and students’ final BPTC grades. The analysis will, inter alia, consider:

·         performance on the test as against performance on the course

·         determination of the ‘cut score’ (or minimum threshold that should be required)

·         the reliability and the validity of the test instrument (ie the extent to which the test would yield the same result during independent, repeated trials)

·         the  relationship between test scores and performance which would allow the test to be used as a predictor for success on the BPTC.

·         whether there might be any unintended consequences of the aptitude test and whether it may disadvantage any group (ie analysis of performance for different groups, or against students’ earlier study results if available)

3.       Is the research to be subject to peer review?

Wide consultation will take place. This will also help feed into discussions with the LSB

4.       Is the data to be made available (suitably anonymised) for others to test their analysis and to conduct further research in the area (to the benefit of all)?

The report will be made available to the Providers of the course so that they can form a view on how their own cohorts performed. It is not yet decided if it will be published more widely.

1. What are the key thresholds that they expect to see passed if the test is to have succeeded?

The BSB  will discuss the required correlation in consultation with our independent consultant prior to the results of the pilot being made available (see also above).