The objective structured clinical examination (OSCE) has developed a reputation as the "gold standard" for the assessment of competence in medical education.¹ The value in the OSCE lies in its ability to accurately determine whether a student can “do things” rather than just remember, write, and talk about them.²

In well-designed OSCEs, students' grades are affected by their performance alone – with minimal influence by other variables.³ If you currently run OSCEs in your medical education program, you likely know that taking steps to quality-assure your exams can help you maintain the consistency and standardization that are key to a successful OSCE.

In this article, we discuss 3 areas worth focusing on for quality assurance of your OSCEs. 

When it comes to ensuring your OSCEs' reliability, you know that consistent marking by your examiners is paramount.⁴ Some research found that even when an OSCE is well-designed and valid, examiner factors are a major contributor to overall examination error.⁵ This suggests that examiner training plays an important role in safeguarding reliability. 
 
A 2020 study found that a structured training program consisting of a model assessment, training webinars, practice assessments, and facilitated discussion contributed to higher inter- and intra-rater reliability.⁶ The training program helped the evaluators to develop a shared mental model, or "common ground" about what competence looks like. Developing this shared mental model is critical to making assessments fair and consistent – but can be quite difficult to accomplish.⁷

It's important to remember that effective training will take time, and your examiners will benefit from repeated practice. You may want to consider using video recordings of real OSCEs and letting examiners practice assessing the recordings. When undergoing her training, one examiner noted that "The more videos I reviewed, the better I became at noticing the criteria I was evaluating."⁸

Beyond initial training, it’s important to keep an eye on how your reliability is tracking. Software like SimCapture can make this often daunting task easier to manage by providing reporting on inter- and intra-rater consistency so you can easily identify any issues that need attention.

If you’ve already tried rigorous training for your examiners and are still struggling with reliability, it may be a matter of diving deeper into your individual examiners. You might find "outliers" that are throwing off your numbers. This could be a "hawk" that always grades too harshly, a "dove" that grades too leniently, or an examiner that might lack consistency in their grading. It’s a common assumption that all clinical instructors are well-prepared to accurately assess students.⁹ But sometimes, for various reasons, not everyone is cut out to be an examiner.¹⁰

Standardized patients (SPs) are an essential element of the OSCE. They help provide a realistic scenario for the student to demonstrate competence and they often serve as a source of valuable feedback for the student. Comprehensive SP training is a key step in creating a bank of optimally-performing SPs.

Covering all these areas in your training will clearly take time and effort. One study found it may take up to 15 hours to adequately train an SP, depending on their role, experience, and adaptability.¹² However, ensuring each SP is well-trained will be worth the pay-off, as it will help protect the quality of your OSCEs.

After training completes, you may want to quality assure each SP’s performance before they participate in an OSCE. One way to do this is to record their performance and have it evaluated by a group of trainers.¹³

Once the SP has started participating in OSCEs, it’s important to provide them with feedback on their performance. The National Board of Medical Examiners recommends creating a checklist of desired behaviors, and then reviewing it with the SP either immediately after the encounter or later using video recordings. If using video, the time of the issue on the checklist should be noted for easy location during the review.¹⁴ Alternatively, using video debriefing software like SimCapture, which allows you to annotate videos, can make this process easier.