Dvir Aran

Our paper is now out in @NEJM_AI!

Three years ago, I met @ShellyShahar, and we started experimenting with LLMs for clinical decision support. We were excited about the possibilities, but felt something was missing in how these models were evaluated.

Decisions in medicine involve trade-offs. Send a patient for an MRI? Recommend a biopsy? Refer, or wait?

Most benchmarks focus on accuracy and safety. We examined another dimension: the cost of recommended care.

eur01.safelinks.protection.outlook.com/?url=https%3A%…

In this editorial about our paper the author note correctly that "the tools studied here will soon be outdated". I recently reran the pipeline with the 10 newest models. In most families, cost of care continued going up - workup-inflation.medbar.ai

I’ll try to make my point in a different way. A few years ago (mostly pre-LLMs), a high school student in my lab used data to find synthetic lethal targets for cancer academic.oup.com/nargab/article…. The bioinformatics pipeline revealed many such candidates. We chose one and, together with @ayoubnlab, validated it, and it actually worked!
1. This paper didn’t get much love from editors and was eventually published in a relatively low-impact journal. If you want impact, you need to go much deeper into validation and mechanism. The hard truth is that there are many papers like this, and editors are fed up with bioinformatics pipelines that discover candidates.
2. My main point is that it seems Anthropic automated the pipeline for finding candidates. I’m pretty sure that if Anthropic asked 900 agents to find synthetic lethal targets, they would do a really good job of finding promising candidates, in a sense, replacing a phenomenal high school student in a small bioinformatics lab. This is very, very, very different from revolutionizing biology and very similar to the regular applications of LLMs we all experience.
3. I am 100% bullish on LLMs/foundation models revolutionizing biology and healthcare. It’s actually already happening, and there have been dozens of high-impact discoveries. This one doesn’t seem to be in the same league, but I’m sure the Anthropic lab, which I assume is heavily funded and has unlimited access to tokens and agents, will be at the forefront of future discoveries.

Based on this post, I was expecting a groundbreaking advancement, but reading through the work, it feels quite underwhelming. It seems to cover only basic bioinformatics analysis, whereas we have seen much more complex tasks accomplished by LLMs. Without the Anthropic PR surrounding it, it would likely be viewed as just another standard paper among the dozens published every day.

Our paper is out in @Cancer_Cell!
We started this project almost 5 years ago with the idea that to understand why some cancer patients respond to immunotherapy and others do not, we may need to look beyond the tumor before treatment. Immunotherapy is not just a test of a pre-existing state; it is a perturbation. It changes the tumor immune microenvironment.
So the question became: can the direction of that change explain response?
authors.elsevier.com/a/1niWC5TA51p1…