Pinned post
Replying to @_jasonwei
Great post…thanks for sharing your thoughts. If I may opine on this as a radiologist with some technical understanding of LLMs/VLMs: I agree radiologists will not be automated away anytime soon. Your elastic demand point is mostly right (some modalities impart ionizing radiation so there is a good reason to not scan as frequently as you want), and I’d add a more fundamental one.
The current transformer-based AI algorithms are extremely sample-inefficient compared to humans, especially when it comes to vision tasks (as opposed to text-based tasks). When I was a radiology resident, I read maybe 3000 CTs of the chest, abdomen and pelvis to reach expert/attending level. I can also make a rare-disease diagnosis after having seen 1 example study (or none; I can make a real-life diagnosis purely from reading and seeing an example case from a textbook). Currently, the best radiology VLM models, such as Nvidia’s NV-Reason-CT model, do not reliably call even some of the most common pathologies, despite having trained on >70,000 CTs that surely contain many examples of a common finding (macro-F1 of 0.614). I can think of several reasons for this: these models usually leverage weakly supervised learning such as CLIP, pairing radiology reports with the images, since wide-scale expert annotation is impractical for every study for every finding (Radiology’s Bitter Lesson, if you will; however, the signal that can scale is the noisy one). To scale, the model(s) will need to learn to associate image findings with words in the radiology reports. However, using radiology reports as ground truth can substantially limit learning. (1) As any radiologist will tell you, we don’t mention everything that we see. Otherwise, your radiology report would be like 30 pages long and not clinically useful. We mostly call the things that matter (for the most part). Sometimes, we don’t even call things that matter. Thus, some reports may mention “diverticulosis” when there is diverticulosis, while other reports don’t, even if there are findings of diverticulosis in the images. (2) Another limitation is that the actual real ground truth is never confirmed, which I suspect is a much bigger problem and harder to correct. When I was a radiology resident, I saved many cases in which my attendings and I made a certain call, and I was curious if those were correct. Right before I was about to graduate, I checked every case I had logged. Many calls were incorrect (and I trained at a major academic center in which subspecialized radiologists read their part of the body, so the attendings making those calls are above-average radiologists; real-world data is likely much worse). (1) and (2) make training very difficult when there is so much incorrectness, ambiguity and noise in the training data.
The above observations are pretty self-evident when you look at other vision tasks, such as self-driving, for which there is (1) ample data with predominantly correct actions (labels), and (2) known ground truths (the task of driving is, for the most part, unambiguous. You see a red light, you stop. You want to make a left turn? You signal and yield to passing cars before turning. Those are universally agreed-upon “correct” actions). To use Tesla FSD as an example: even with the best compute, engineers and data, it took over a decade to reach limited, geofenced driverless service. And this is despite the fact that driving has a much shorter tail than radiology does. Further, self-driving usually has one correct action at a time. In radiology, that’s not true. There are so many different radiological appearances of one disease, as well as the same appearances of different diseases. And various diseases can involve the same anatomical location (for example, look up adrenal collision tumor).
To “automate radiology” is also a huge simplification. The vast breadth of radiological studies is seldom mentioned. For a single modality of a body part, there are so many different variants. Say, for example, CT of the abdomen and pelvis: there are at least 10 different flavors of this. To name a few, there is the non-contrast CT, the portal venous phase CT (bread-and-butter workhorse), and then there are specialized versions such as the liver mass protocol, CT angiogram GI bleed protocol, pancreatic mass protocol CT, adrenal mass protocol CT, kidney mass protocol CT, and CT urogram, each of which contains 3-4 phases and is tailored to answer a specific question. All of this is just one modality for one section of the body. Now do ultrasound, X-rays, gamma studies, PET studies, and MRI. And then consider all the other body parts (brain, head and neck, chest, etc.).
I’ve only touched upon the surface of all the nuances, but I think the current capability gap and your demand argument point the same way: radiologists will be busy for a long time.
The current transformer-based AI algorithms are extremely sample-inefficient compared to humans, especially when it comes to vision tasks (as opposed to text-based tasks). When I was a radiology resident, I read maybe 3000 CTs of the chest, abdomen and pelvis to reach expert/attending level. I can also make a rare-disease diagnosis after having seen 1 example study (or none; I can make a real-life diagnosis purely from reading and seeing an example case from a textbook). Currently, the best radiology VLM models, such as Nvidia’s NV-Reason-CT model, do not reliably call even some of the most common pathologies, despite having trained on >70,000 CTs that surely contain many examples of a common finding (macro-F1 of 0.614). I can think of several reasons for this: these models usually leverage weakly supervised learning such as CLIP, pairing radiology reports with the images, since wide-scale expert annotation is impractical for every study for every finding (Radiology’s Bitter Lesson, if you will; however, the signal that can scale is the noisy one). To scale, the model(s) will need to learn to associate image findings with words in the radiology reports. However, using radiology reports as ground truth can substantially limit learning. (1) As any radiologist will tell you, we don’t mention everything that we see. Otherwise, your radiology report would be like 30 pages long and not clinically useful. We mostly call the things that matter (for the most part). Sometimes, we don’t even call things that matter. Thus, some reports may mention “diverticulosis” when there is diverticulosis, while other reports don’t, even if there are findings of diverticulosis in the images. (2) Another limitation is that the actual real ground truth is never confirmed, which I suspect is a much bigger problem and harder to correct. When I was a radiology resident, I saved many cases in which my attendings and I made a certain call, and I was curious if those were correct. Right before I was about to graduate, I checked every case I had logged. Many calls were incorrect (and I trained at a major academic center in which subspecialized radiologists read their part of the body, so the attendings making those calls are above-average radiologists; real-world data is likely much worse). (1) and (2) make training very difficult when there is so much incorrectness, ambiguity and noise in the training data.
The above observations are pretty self-evident when you look at other vision tasks, such as self-driving, for which there is (1) ample data with predominantly correct actions (labels), and (2) known ground truths (the task of driving is, for the most part, unambiguous. You see a red light, you stop. You want to make a left turn? You signal and yield to passing cars before turning. Those are universally agreed-upon “correct” actions). To use Tesla FSD as an example: even with the best compute, engineers and data, it took over a decade to reach limited, geofenced driverless service. And this is despite the fact that driving has a much shorter tail than radiology does. Further, self-driving usually has one correct action at a time. In radiology, that’s not true. There are so many different radiological appearances of one disease, as well as the same appearances of different diseases. And various diseases can involve the same anatomical location (for example, look up adrenal collision tumor).
To “automate radiology” is also a huge simplification. The vast breadth of radiological studies is seldom mentioned. For a single modality of a body part, there are so many different variants. Say, for example, CT of the abdomen and pelvis: there are at least 10 different flavors of this. To name a few, there is the non-contrast CT, the portal venous phase CT (bread-and-butter workhorse), and then there are specialized versions such as the liver mass protocol, CT angiogram GI bleed protocol, pancreatic mass protocol CT, adrenal mass protocol CT, kidney mass protocol CT, and CT urogram, each of which contains 3-4 phases and is tailored to answer a specific question. All of this is just one modality for one section of the body. Now do ultrasound, X-rays, gamma studies, PET studies, and MRI. And then consider all the other body parts (brain, head and neck, chest, etc.).
I’ve only touched upon the surface of all the nuances, but I think the current capability gap and your demand argument point the same way: radiologists will be busy for a long time.
