P-39 Determining the minimum number of OSCE stations required for valid and reliable assessment in the final MBBS Examination
Author(s):
A Kumar , K Krishnamurthy , S Motilal , M H Campbell , K Connell , M Fernandes , E Morris , J Paul-Charles , B Sa , M A A Majumder
Year of Presentation:
2026
Objective: To determine the minimum number of Objective
Structured Clinical Examination stations (OSCEs) needed to achieve acceptable reliability and validity in the final
Bachelor of Medicine and Bachelor of Surgery (MBBS)
clinical examination at the University of the West Indies.
Methods: Cross-sectional psychometric analysis of the June 2019 OSCEs across Mona, St Augustine, and Cave Hill campuses. A total of 485 final-year students completed 17 stations; balanced subsets of 6, 8, 10, and 12 stations were compared with the full set. Internal consistency reliability was estimated with Cronbach’s alpha. Generalizability Theory was used to quantify reliability of scores across stations and examiners. Validity evidence included blueprint-based content review and Pearson correlations with total scores. Analysis of variance tested differences in subset mean scores.
Results: Internal consistency reliability increased with station count: at 6 stations, 0.32 (St. Augustine) to 0.61 (CHILL); at 17, Mona 0.76, St. Augustine 0.64, CHILL 0.71. G-Theory coefficients showed similar gains: 6 stations Mona 0.21, St. Augustine 0.18, CHILL 0.42; 17 stations Mona 0.43, St. Augustine 0.36, CHILL 0.70. Subset mean scores differed by size at Mona (F(4,1290)=9.046, p<0.001) and St. Augustine (F(4,960)=11.53, p<0.001) but not at Cave Hill (F(4,145)=0.463, p=0.763). Pearson correlations between subset and total scores increased with station count, from 0.799, 0.791, and 0.836 (6 stations) to 1.000 at 17 stations across all campuses. Construct validity was robust for subsets ≥10 stations.
Conclusion: Under the current design, 17 stations did not achieve reliability targets of ≥.80; Cave Hill performed best. A practical minimum of 10–12 stations preserves construct validity, although achieving high-stakes reliability will likely require additional stations and/or multiple raters with stronger standardization of blueprints, scoring instruments, and simulated patient training.