Siirry sisältöön
Avaa navigointi

3 Tips for Ensuring Quality OSCEs

Learn some ways to safeguard the assessment power of your exams.

The objective structured clinical examination (OSCE) has developed a reputation as the "gold standard" for the assessment of competence in medical education.1 The value in the OSCE lies in its ability to accurately determine whether a student can “do things” rather than just remember, write, and talk about them.2

In well-designed OSCEs, students' grades are affected by their performance alone – with minimal influence by other variables.3 If you currently run OSCEs in your medical education program, you likely know that taking steps to quality-assure your exams can help you maintain the consistency and standardization that are key to a successful OSCE.

In this article, we discuss 3 areas worth focusing on for quality assurance of your OSCEs. 

1. Achieve high reliability with thorough examiner training

high-reliability-training-subhero.jpg

When it comes to ensuring your OSCEs' reliability, you know that consistent marking by your examiners is paramount.4 Some research found that even when an OSCE is well-designed and valid, examiner factors are a major contributor to overall examination error.5 This suggests that examiner training plays an important role in safeguarding reliability
 
A 2020 study found that a structured training program consisting of a model assessment, training webinars, practice assessments, and facilitated discussion contributed to higher inter- and intra-rater reliability.6 The training program helped the evaluators to develop a shared mental model, or "common ground" about what competence looks like. Developing this shared mental model is critical to making assessments fair and consistent – but can be quite difficult to accomplish.7

It's important to remember that effective training will take time, and your examiners will benefit from repeated practice. You may want to consider using video recordings of real OSCEs and letting examiners practice assessing the recordings. When undergoing her training, one examiner noted that "The more videos I reviewed, the better I became at noticing the criteria I was evaluating."8

Beyond initial training, it’s important to keep an eye on how your reliability is tracking. Software like SimCapture can make this often daunting task easier to manage by providing reporting on inter- and intra-rater consistency so you can easily identify any issues that need attention.

If you’ve already tried rigorous training for your examiners and are still struggling with reliability, it may be a matter of diving deeper into your individual examiners. You might find "outliers" that are throwing off your numbers. This could be a "hawk" that always grades too harshly, a "dove" that grades too leniently, or an examiner that might lack consistency in their grading. It’s a common assumption that all clinical instructors are well-prepared to accurately assess students.9 But sometimes, for various reasons, not everyone is cut out to be an examiner.10

2. Build a pool of competent SPs.

build-competent-sps-subhero.jpg

Standardized patients (SPs) are an essential element of the OSCE. They help provide a realistic scenario for the student to demonstrate competence and they often serve as a source of valuable feedback for the student. Comprehensive SP training is a key step in creating a bank of optimally-performing SPs.

According to the Association of Standardized Patient Educators (ASPE) Standards of Best Practice, your training program should effectively prepare SPs to provide:11
  • consistent and accurate role portrayal
  • their unique and valuable perspective in feedback to learners
  • accurate and consistent completion of assessment tools (if assessment is part of their role)

Covering all these areas in your training will clearly take time and effort. One study found it may take up to 15 hours to adequately train an SP, depending on their role, experience, and adaptability.12 However, ensuring each SP is well-trained will be worth the pay-off, as it will help protect the quality of your OSCEs.

After training completes, you may want to quality assure each SP’s performance before they participate in an OSCE. One way to do this is to record their performance and have it evaluated by a group of trainers.13

Once the SP has started participating in OSCEs, it’s important to provide them with feedback on their performance. The National Board of Medical Examiners recommends creating a checklist of desired behaviors, and then reviewing it with the SP either immediately after the encounter or later using video recordings. If using video, the time of the issue on the checklist should be noted for easy location during the review.14 Alternatively, using video debriefing software like SimCapture, which allows you to annotate videos, can make this process easier.

3. Maximize station quality with peer review, piloting, and monitoring

maximize-station-quality-subhero.jpg

Quality-assuring a new station will play an important role in protecting the reliability and validity of your OSCE scores.15 One way to accomplish this is by conducting a peer review workshop. Participants can break into small groups, making sure there are no issues with clinical accuracy or appropriateness of the tasks.16 After peer-reviewing, piloting your station can help you identify and correct any issues.17

 

Once you’ve started using your new station, you may want to consider monitoring question quality on an ongoing basis to ensure there are no issues. SimCapture can provide you with a report that utilizes point-biserial correlations to easily tell you if a specific question may need refinement.

simcapture-elearning-subhero1.jpg

simcapture-central-control-subhero.jpg

Free up more time for quality assurance initiatives

You know that running your OSCEs requires a substantial amount of time and effort. If you’re wondering where you’ll find the time for more quality assurance, you may want to consider seeking help from software that can lessen the burden of orchestrating your OSCEs. A program like SimCapture Cloud can free up time, effort, and resources by allowing you to:

  • see all rooms and control them at once
  • eliminate the need for preparing papers, transcribing them, and loading the patient cases manually
  • automate and computerize your OSCE flows
  • access rubrics, evaluations, and recordings at any time, in any location
  • have seamless collaboration between your organizations and campuses
  • free up IT resources by eliminating the need to maintain and upgrade your own systems

If you’re already a SimCapture Enterprise user, you may want to consider making the switch to SimCapture Cloud to maximize the power of SimCapture in streamlining your program and giving you time back to focus on maintaining high-quality OSCEs.

 

Learn more

References

  1. Furmedge, D. S., Smith, L., & Sturrock, A. (2016). Developing doctors: What are the attitudes and perceptions of year 1 and 2 medical students towards a new integrated formative objective structured clinical examination? BMC Medical Education, 16(1). https://doi.org/10.1186/s12909-016-0542-3
  2. Khan, K., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013). The Objective Structured Clinical Examination (OSCE): Amee guide no. 81. part I: An historical and theoretical perspective. Medical Teacher, 35(9). https://doi.org/10.3109/0142159x.2013.818634
  3. Ibid.
  4. Khan, K., Gaunt, K., Ramachandran, S., & Pushkar, P. (2013). The Objective Structured Clinical Examination (OSCE): Amee guide no. 81. part II: Organisation & Administration. Medical Teacher, 35(9). https://doi.org/10.3109/0142159x.2013.818635
  5. Mortsiefer, A., Karger, A., Rotthoff, T., Raski, B., & Pentzek, M. (2017). Examiner characteristics and Interrater Reliability in a communication OSCE. Patient Education and Counseling, 100(6), 1230–1234. https://doi.org/10.1016/j.pec.2017.01.013
  6. Holland, A., Tiffany, J., Blazovich, L., Bambini, D., & Schug, V. (2020). The effect of evaluator training on inter- and intrarater reliability in high-stakes assessment in simulation. Nursing Education Perspectives, 41(4), 222–228. https://doi.org/10.1097/01.nep.0000000000000619
  7. Zavertnik, J. & Holland, A. (2017). Are you using high-stakes assessment? do you have intra/interrater reliability? NLN Nursing EDge. Retrieved from https://nursingedge.nln.org/2017/10/24/are-you-using-high-stakes-assessment-do-you-have-intrainterrater-reliability/
  8. Ibid.
  9. Kardong-Edgren, S., Oermann, M. H., Rizzolo, M. A., & Odom-Maryon, T. (2017). Establishing inter- and intrarater reliability for high-stakes testing using simulation. Nursing Education Perspectives, 38(2), 63–68. https://doi.org/10.1097/01.nep.0000000000000114
  10. Ibid.
  11. Lewis, K. L., Bohnert, C. A., Gammon, W. L., Hölzer, H., Lyman, L., Smith, C., Thompson, T. M., Wallace, A., & Gliva-McConvey, G. (2017). The Association of Standardized Patient Educators (ASPE) standards of best practice (SOBP). Advances in Simulation, 2(1). https://doi.org/10.1186/s41077-017-0043-4
  12. Khan, K., Gaunt, K., Ramachandran, S., & Pushkar, P. (2013). See reference #4.
  13. Ibid.
  14. Objective Structured Clinical Examinations III: Quality Assurance of Standardized Patients. (2019). National Board of Medical Examiners (NBME). Retrieved from https://www.nbme.org/sites/default/files/2020-01/OSCE_III_Quality_Assurance_of_Standardized_Patients_Lesson.pdf
  15. Khan, K., Gaunt, K., Ramachandran, S., & Pushkar, P. (2013). See reference #4.
  16. Ibid.
  17. Ibid.

Helping you build high-quality OSCEs

simcapture-pro-listing.jpg
SimCapture mahdollistaa simulaatioiden tehokkaan hallinnan, tallentamisen, debriefingin ja arvioinnin sekä... SimCapture mahdollistaa simulaatioiden tehokkaan hallinnan, tallentamisen, debriefingin ja arvioinnin sekä koulutustiloissa että in-situ.

SimCapture