Robust learning and assessment approaches

4.122     Effective instructional design needs close alignment between outcomes, learning approaches and assessment. Assessment in particular serves to demonstrate that student or trainee learning outcomes have been achieved. Assessment judgments provide the primary means of ensuring competence to progress to the next stage of training, or achievement of the standard expected of a newly qualified lawyer, and hence perform an important public protection function. However there are risks that over-prescription in setting assessment methods, and teaching and learning processes can stifle innovation, and limit opportunities for quality enhancement.[1] In this section the primary focus is on assessment, recognising that effective assessment should also contribute to learning.

4.123     There is a legitimate regulatory interest in ensuring that assessment is robust. It must be capable of assessing that which it sets out to assess (valid); it must produce consistent and replicable results (reliable); and it must assess against the syllabus and learning outcomes that have been set out (fair) (Bone, 1999; Stuckey et al, 2007:235-6). The proposal for greater use of standardisation for high stakes assessment in the previous section responds to that need. A process of standardisation improves the quality of assessment by increasing confidence in its reliability and validity. This can be achieved by extending the number of credible judgments in respect of a task, and by ensuring that those judgments are collected and evaluated systematically. Vocational training courses should be encouraged to adopt such approaches for at least a proportion of high stakes assessment tasks. Standardisation can also be used to move away from fixed pass marks, which tend to assume a high degree of inter-test reliability (ie, that two tests created at entirely different times with different questions can reliably be marked to the same pass mark).[2]

4.124     The remainder of this section focuses on a number of specific assessment issues and approaches, chiefly in respect of ensuring validity of assessment, and developing both the range of outcomes assessed, and the range of appropriate assessment tools used.

Practice validity

4.125     A number of comments in respect of the LPC, BPTC and CILEx qualifications identify dissonance between classroom-based activity and what happens in practice. For example, one barrister respondent emphasised the difference between a ‘real’ particulars of claim drafted in chambers and ‘a bar school particulars of claim’. This has a number of undesirable consequences. It creates a burden of unlearning and re-learning once in the practice environment, and reduces the value of the training in the eyes of the profession, and quite possibly the student. The extent to which this is a significant problem is, however, difficult to gauge from the data gathered. It is raised often enough by trainees and practitioners alike to suggest the problem is not isolated to one institution or one set of skills or transactions, but it is also not clear to what extent differences are permissible variations in style, an inevitable consequence of the fragmentation or simplification of a practice task for teaching purposes, or genuinely poor simulation of effective practice.

4.126     The difficulties of getting a range of practitioners involved in training are well known, at least anecdotally, nonetheless there may be a case for involving practitioners more closely with materials design, moderating assessments and setting assessment standards. Moves towards more integrated training (discussed in the next section) may also facilitate more authentic training experiences.

Centralised assessment: validity vs consistency?

4.127     The BPTC, CILEx and CLC qualifications all make use of centralised assessment.[3] This is seen by those involved in their delivery as ensuring consistency and rigour in assessment, and may limit risks of declining standards in a highly competitive market.[4] It may also, as two of the academic respondents observed, better meet public assumptions about professional qualifications than local assessment.

4.128     However, the suggestion of centralised assessment for the LPC divided opinion:

We would approve of centralised examinations, not for the academic stage of legal education, but for the LPC…. The vocational stage of education for solicitors is currently characterised by varying standards of provision across those institutions that offer the programme. While the providers are currently ‘policed’ by SRA external examiners, LPC assessments are nevertheless of variable standard and content. This can be confusing for employers and students.

Kaplan response to Discussion Paper 02/2012

The reasons why the centrally-set Solicitors Final Examination was replaced by the LPC remain as valid now as when the change was made some 20 years ago. The curriculum was driven by the requirements of a centrally set examination, thus teachers felt no ownership of it. This led to unimaginative teaching and rote learning. A return to that system would remove the flexibility that enables LPC programmes to be tailored to the needs of individual firms or types of legal practice. A centrally set examination cannot cater for the assessment of practical skills. A major criticism of the former Final Examination, and of the programmes leading to it, was that trainees entered employment with wholly inadequate skills. To the extent that the Final Examination addressed skills at all, it was by expecting students to write about them, rather than to demonstrate their possession.

There are no benefits to the re-adoption of such a system, and significant risks to the quality and relevance of the LPC.

University of Law response to Discussion Paper 02/2012

4.129     Both arguments have merits. Differences in assessment methods (eg, between open-book and closed-book assessment) may also result in a marked difference in pass rates between different institutions and weaken the ‘face validity’[5] of the assessment process as a whole. A move to centralised assessments, however, may also increase risks to assessment validity and fairness. By definition, it tends to separate the assessment from the learning, and, unless the curriculum is very tightly defined, there tends to be a higher risk that some element of the test might not meet the reasonable expectations of either teacher or learner. This is also likely negatively to impact perceptions of the fairness and face validity of the assessment. In the end the choice may be both needs- and cost-based, between consistency and flexibility. The move to more flexible, bespoke, provision may limit face validity problems, provided trainees and employers are satisfied with the standard. Validity problems with centralised assessment may also be reduced by more rigorous monitoring processes which raise the standard of validity to be met.[6] This does not resolve the lack of ownership problem identified above, but it would tend to reduce doubts about the fairness of any resulting assessment.

Validity in assessing communication and client-facing skills

4.130     Client communication forms a large part of the work of legal services providers. The primacy of client communication and related skills should be better recognised in mainstream assessment practices within LSET. Some current courses use trained actors in summative client interviewing or client conferencing assessments. It is also necessary to assess the trainee’s capacity to get inside the client’s world and the client’s experience.

4.131     The use of actor-clients contrasts with the recent development of ‘standardised clients’. Standardised clients (SCs) are lay people who are trained to do two things well: role-play a client in an interview or client conference, and assess the client-facing skills of the learner. It is important to recognize from the outset that SCs do not assess the legal competence of lawyers or law students or the quality of their advice; but they are uniquely placed as a role-playing client to comment upon the communications skills and relational attitudes of lawyers and law students in front of them, and, as Fry et al (2012:145) point out, prioritise the consumer experience within the training and assessment process.

4.132     The SC model is drawn from medical education. The use of standardised patients has been extensively researched and documented over the last half century in the health sciences. Following a literature review, and with assistance from the Clinical Skills Unit of the Medical Faculty at the University of Dundee and Ninewells Hospital, a correlative research project was designed and implemented by Barton et al (2006) on Strathclyde Law School’s professional programme, the (then) Diploma in Legal Practice. It proved that the use of properly trained SCs to assess interviewing skills was as reliable and valid as practitioner-tutor assessments of the interviews. At Strathclyde now, SCs assess student interviewing skills and values: tutors are used only to second-mark what the SCs grade as not yet competent or borderline competent. Their decisions are rarely overturned.

4.133     In the six years since this original study, SCs have been tested in the simulation of business clients for niche accreditation programmes for solicitors (Writers of the Signet Society, Edinburgh), in a CPD programme for experienced solicitors (Law Society, Ireland), in professional legal education in the USA (New Hampshire) and Australia (Australian National University and Adelaide University), and in undergraduate legal education at Northumbria Law School, as part of preparation for mandatory clinical modules. They are also a key component of the QLTS assessment. Formal evaluations of many of these initiatives are at an early stage, but early indicators suggest that the approach has been successful in improving standards (see Fry et al, 2012; Garvey, 2010).

4.134     Three points should be noted regarding the training and use of SCs:

• SCs do not act in the normal sense of the word: they enact the situation of a person, according to standardised criteria, and with as full an understanding of the person’s situation as they require. This involves improvisation, but it also involves knowing the difference between what can be said or not said, according to legal points that may be at issue, and exploring the emotional range that can be enacted.

• The training of SCs includes standardisation of the act of assessment. This means that not only do SCs need to understand thoroughly and memorise the basic assessment criteria, but they also need to be able to apply the criteria in a consistent way across the range of persons they are assessing; and each SC performance has to be standardised to a specific set of standards. This process of standardisation is what makes the simulation both a powerful tool for learning through formative assessment, and a precise tool for the assessment of person-centred skills, values and attitudes. This is particularly important because such skills are not easy to assess robustly.

• The medical literature refers to the power of standardised patients to change the behaviour of medical staff in a fundamental way. This is one reason why simulation is used so extensively in the primary training and education of doctors and health workers, and why it is used in the national Doctors in Difficulty programme. More recently it has also been developed for the National Skills Academy in Social Care, where it was piloted with registered managers of care homes. In this aspect of its use, the approach can blend what might be termed technical professional practice with ethical practice, based upon client-centred activities. To that extent it also has the potential to link the ethical and client communication outcomes of LSET.

4.135     The use of SCs requires more preparation time for staff and SCs than actors, and a larger financial investment in assessment, though the experience at Strathclyde indicates that this can be achieved in a legal services education setting.[7]

Assessing the range of outcomes

4.136     Not all outcomes of an education or training module need to be assessed, though ‘key’ outcomes and a representative range should be. This is particularly true of the academic stage, where there is considerable flexibility regarding the teaching and assessment of the Benchmark skills. Competence in IT skills or in writing, for example, is often integrated within predominantly knowledge-based assessments, rather than as a free-standing skill, and may be inferred from the completion of a task – eg, word-processing competence may be inferred from submitting an essay in word-processed form, without direct assessment of word-processing ability. This approach means that some important skills may never be subject to an assessment in their own right.

4.137     In thinking about the robustness of the range of assessments, in the context of the knowledge and skills gaps identified above, it is proposed:

• First, that legal values should be assessed in (some combination of) Foundation subjects at the academic stage. If adopted, ‘professionalism’ outcomes, akin to those used for PEAT 1 and 2, would similarly ensure that professionalism provides a critical context within which other knowledge and skills are developed as part of vocational or professional training. Accordingly professionalism should be assessed pervasively rather than as a discrete subject. (see Maharg, 2013:122-3; Stronach et al, 2002).

• Secondly, there should be a discrete terminal assessment of legal writing and critical thinking skills, at least at level 5 of the academic stage. Academic stage providers should retain discretion in setting the context and parameters of the task, provided that it is sufficiently substantial to give students a reasonable but challenging opportunity to demonstrate their competence.

• Thirdly, reflection should be developed and assessed through a portfolio or personal development plan.

4.138     The use of portfolios may require additional justification. Portfolio-based assessment of knowledge and skills is developing in undergraduate (legal) education, and, with the consent of the SRA, is being tested by the University of Law as an assessment tool for the LPC. Portfolio assessment has also been piloted in the SRA work-based learning scheme and forms one of the tools used in the IPS work-based learning scheme for periods of supervised practice. In a meta-analysis of the ‘higher quality’ research studies, chiefly in medical education, Buckley et al (2009) identify improvements in knowledge and understanding, increased self-awareness and engagement in reflection and improved student-tutor relationships as the main benefits arising from portfolios. Whilst portfolios encourage students to engage in reflection, Buckley et al also found that the quality of those reflections cannot be assumed and the time commitment required for portfolio completion may impact on other learning, or deter students from engaging with the process unless they are required to do so by the demands of assessment. These challenges are consistent with those reported in legal education (eg, Webb, 2002; Polding, 2010). Consequently, if portfolios are to be effective, they must be properly embedded in both learning and assessment processes.

4.139     Nonetheless, the assessment of reflection can be challenging and is controversial in its own right, given the personal and developmental function that such reflection often performs (see, eg, Maughan and Webb, 1996; Moon, 2004: 149-57). It may therefore need time to develop in each course and providers should have the option initially to experiment with either formally assessing or simply signing-off the portfolio without grading it.

Extending assessment

4.140     The examples included here offer some indications of ways in which more or different assessment should contribute to the development of professional competence. It serves also to point up the relatively narrow base of assessment methods in use, and the fairly limited range of potential competencies that are being assessed, particularly outside the cognitive domain. Much that falls in the affective/moral domain or habits of mind may not be widely assessed or evaluated, for example: independence, emotional intelligence, respect for clients, or attention to detail; similarly, relationship matters such as teamwork and handling conflict may be addressed in the workplace, but not as part of some consistent professional standard.[8] As Epstein and Hundert (2002:233) conclude: ‘educators, professional societies, and licensing boards should view professional competence more comprehensively to improve the process of assessment’.

 

---