Medical Licensing Exam Scores Sent to Residency Programs After Switch to Pass/Fail

Numerical scores for the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 were mistakenly made visible to ob/gyn programs despite the test being pass/fail. On top of that, the students themselves never saw their own scores.

The National Board of Osteopathic Medical Examiners (NBOME), which runs the COMLEX, notified students in an email that it had learned on October 7 of “a technical issue that impacted the delivery of your COMLEX-USA Level 1 performance results to programs using ResidencyCAS [Centralized Application Service] for residency applications.”

This technical issue resulted in ob/gyn programs receiving the three-digit exam score for each applicant.

However, COMLEX Level 1 became pass/fail in 2022, 2 years after the United States Medical Licensing Examination (USMLE) for allopathic medical students made its Step 1 exam pass/fail.

The email to students said that “NBOME does not report 3-digit scores for Level 1, but a score was transmitted by NBOME to ResidencyCAS in error, and was available to program directors until this week.”

In the email, Gretta Gross, DO, MEd, executive vice president for assessment and chief assessment officer at NBOME, apologized for the error and said the board has reached out to ob/gyn program directors to alert them to the error and advised them not to consider anything but the pass/fail result.

A spokesperson for the American College of Obstetricians and Gynecologists (ACOG) told MedPage Today that “Liaison, who hosts ResidencyCAS, was notified by NBOME that NBOME had transmitted both numeric and pass/fail scores to ResidencyCAS,” and that “Liaison immediately notified ACOG.”

Additionally, “Liaison immediately addressed this issue and removed the numerical scores from the Program Director portal and all portals” and programs can now only see the pass/fail result, as intended, the spokesperson said.

As of press time, neither Liaison nor NBOME had replied to MedPage Today’s request for comment.

On the r/comlex Reddit forum, impacted students shared their experiences. User MoreScallions, an ob/gyn applicant, told MedPage Today that this situation “causes a lot of chaos and distrust towards NBOME since they are the ones who released the scores.”

“We were told there is no way to access our three-digit scores, even for our own knowledge, yet here they are storing the data and releasing it without our consent,” they said.

MoreScallions also noted that “DO applicants are already at a disadvantage because most programs do not universally accept [COMLEX] Level 2 scores and highly recommend [USMLE] Step 2 scores.” In their view, this mishap is another knock to DO applicants since their three-digit score was visible to program directors for an entire week while MD applicants were marked pass/fail, as designed.

To mitigate some of the harm, they said that NBOME should “release our three-digit scores so we know what program directors saw and refund our score release fee.”

Harris Ahmed, DO, MPH, a vitreoretinal surgery fellow at Weill Cornell Medicine in New York City, told MedPage Today that because Level 1 is pass/fail, students allocate a different level of time and attention to it than if it was scored.

“They’re taking the test with the goal to just pass, not to score their highest, which is what they’re going to do for the test that they know is scored,” like COMLEX Level 2 or USMLE Step 2, he said. “So their score is not going to be that great. It’s not going to be a reflection of their best effort.”

Bryan Carmody, MD, of Eastern Virginia Medical School in Norfolk, who shared a screenshot of the email on X, pointed out that a lot of program directors don’t know how to interpret COMLEX scores, since they’re different than USMLE scores. Furthermore, DO applicants should be compared against other DO applicants — not to MD applicants.

“Paradoxically, I think the NBOME’s score leak mostly hurts DO applicants applying to ‘DO friendly’ programs,” Carmody wrote. “If a [program director] who’s used to evaluating/stratifying DO applicants sees the secret COMLEX Level 1 score — especially if that score is very low — they may not be able to ‘unsee’ it.”

Ahmed also noted that most DO students take the USMLE in addition to the COMLEX in order to maximize their chances of matching. (Back in 2020, the Student Osteopathic Medical Association passed a resolution calling for a unified licensure exam.) He said that he worries that seeing an applicant’s score will color the program’s view of applicants.

“It’s kind of putting somebody’s career in life up in the air due to a mistake, a typo, or however this happened,” Ahmed said.

Please enable JavaScript to view the comments powered by Disqus.