Medical Education Flamingo, MD, PhD

Our study published in @JSurgEduc today!

1st-year med students using the AI tool, I vibe-coded through @Replit, outperformed 2nd-years in surgical diagnosis.

No human-written content. No human in the loop.

Scalable, low-cost diagnostic training is no longer hypothetical.

Researchers tested 15 LLMs by generating 27800 medical MCQs and found structural flaws in 23.4% of questions.

Seven models placed the correct answer at A in 61–94% of items.

Item-writing guidelines reduced flaws to 13.6%, while template changes largely fixed answer-position bias. #MedEd

If you still use ChatGPT and Claude via web interface, not from their apps, what you're missing is huge.

Download the apps and ask most difficult tasks to do in your computer. If it can't do, slightly lower the bar and try again.

The things it does will surprise you. #MedEd

AI use for academic promotion and tenure guidance.

AI is able to and should help us on this kind of boring work. #MedEd

A REALLY INTERESTING finding is that having more diagnoses in the list was related to better accuracy for humans, but lower accuracy for non-reasoning LLMs.

This important and newly published paper aims to analyze the concept of diagnostic updating when new evidence contradicts the initial diagnosis. #MedEd

The authors compare 12 LLMs and 75 medical students and physicians on 4 clinical cases, where new information is given step by step.

The empirical evidence given in this work shows that LLMs were more successful in identifying and ranking the correct diagnosis first (77–88% vs. 50% for humans).

However, LLMs often kept incorrect diagnoses in their lists, even after changing their rankings. Humans, on the other hand, mostly decreased their confidence without changing their diagnosis rankings.

I consider this submission an interesting study because it shows that we should not focus only on final accuracy of LLMs. How they change their decisions when new evidence is provided is also important.