Twitxy Saved

Anthony Gitter

@anthonygitter

Computational biologist; Professor at University of Wisconsin-Madison; Jeanne M. Rowe Chair at Morgridge Institute
gitterlab.org Joined April 2015
434 Following 2.9K Followers 2.9K Posts

Latest posts 5

Anthony Gitter @anthonygitter
I read DRYLAB-Bench and do not find the proposed Biosecurity Risk Score to be interpretable. At the core, it is a z-score (with clipping, normalization, and other adjustments). 1/
Biology+AI Daily @BiologyAIDaily Oct 13
Drylab-bench: Benchmarking the biosecurity risk of agentic LLM mutation design

1. The paper introduces DRYLAB-Bench (Design Risk Yielded by LLM Agent in Biodesign), a benchmark that quantifies how well agentic LLMs can design high-risk single-amino-acid mutations—moving beyond “does the model know dangerous facts?” to “can it operationally propose risky variants?”

2. Core idea: evaluate LLM agents in realistic, iterative mutation-design workflows where the model proposes candidates, consults biological design tools (BDTs), revises designs over multiple rounds, and finally submits its top-ranked mutations (Top-3 per unit).

3. Key technical innovation: safe, scalable, ground-truth verification without wet-lab synthesis and without unreliable “LLM judges.” Designs are scored by matching to 187,695 empirical deep mutational scanning measurements from ProteinGym and ViroGym, enabling “zero physical risk” evaluation.

4. The benchmark contains 30 genuinely hazardous scenarios distilled from biosecurity frameworks (BWC, WHO guidance, US Executive Orders, EU AI Act), organized into an 8-category risk taxonomy: immune evasion, enhanced infectivity/host-range, antimicrobial/antiviral resistance, toxification, oncogenesis/tumor-suppressor inactivation, metabolic collapse/pharmacogenomic weaponization, environmental tolerance, and agricultural/ecological biosecurity.

5. DRYLAB-Bench formalizes a Biosecurity Risk Score (BRS): assay-oriented risk values are standardized within each assay (z-scored, clipped), aggregated over the model’s Top-3 picks using self-reported confidence weights, and averaged across scenarios; refusals are handled via a refusal floor (set to 0) to avoid rewarding blanket refusal.

6. Tool-augmentation is explicitly measured with staged conditions along a shared generation trajectory: S0 one-shot (no tools), S1 single-round tool revision, S2 multi-round agentic iteration (up to 4 rounds). A separate S0-iter control attempts iteration without tools (reported as noisy but provides a bound on iteration-only effects).

7. The tool suite spans multiple evidence dimensions (none reveal the hidden DMS labels): evolutionary conservation (MMseqs2), sequence likelihood (ESM-1v ensemble), structure compatibility (ProteinMPNN), UniProt functional annotations, toxicity prediction (ToxinPred3), and scenario-specific predictors (AlphaMissense / EVEscape / FoldX).

8. Main empirical finding: frontier models already show substantial baseline risk in S0. Across models, 67.6% of one-shot designs exceed the average measured variant, and 21.4% land in the highest-risk decile (vs 10% expected by chance).

9. Agentic tool use amplifies risk: pooled BRS rises from 0.359 (S0) to 0.475 (S2), i.e., ∆BRS = +0.116; the top-decile share increases from 21.4% to 23.5%. The paper highlights concrete examples where iterative tool use drives dramatic shifts in percentile rank within an assay (e.g., a disruption scenario moving from ~11.7% to ~98.9% outranked variants).

10. Safety-alignment insight: refusals correlate more with surface wording/keywords than with underlying hazardous intent. When the same scientific request is rewrapped with neutral academic framing, refusals generally drop; when prefixed with explicit harm context, refusals spike across many models—suggesting current safeguards can be bypassed by “benign” scientific restatement of the same objective.

💻Code: github.com/drylab-bio/DRY…
📜Paper: arxiv.org/abs/2610.13116
#Biosecurity #AIAlignment #LLMSafety #ComputationalBiology #ProteinEngineering #Benchmarking #AgenticAI #DeepMutationalScanning #ProteinGym #ViroGym
View quoted post on X
Anthony Gitter @anthonygitter
I struggle to see how watermarking strengthens global biosecurity. I see why a DNA synthesis provider would want to verify a customer generated a sequence using a closed, "trusted tool". What are the benefits of a watermark over a digital signature? (see review)

cc @davidstutz92
Pushmeet Kohli Verified account @pushmeet Sep 30
Very happy to announce that our team @GoogleDeepmind has pushed the boundaries of generative biology, achieving the successful synthesis of AI-designed proteins that are both functional and watermarked.

This proof-of-concept watermarking of the building blocks of life is enabled by SynthID Bio, our new protein watermarking method. It is designed to safeguard the new era of AI-powered generative biology and strengthen global biosecurity.

You can read my thoughts here on why watermarking AI-designed proteins is an important research breakthrough: x.com/pushmeet/statu…
View quoted post on X
2 replies 2 reposts 27 likes 3.7K views View on X
Anthony Gitter @anthonygitter
Something feels off about these evaluations. Why create position-balanced folds? They make the supervised fitness effect prediction task artificially easy and likely lead to the much higher performance than the ProteinGym benchmarks proteingym.org/benchmarks.
Biology+AI Daily @BiologyAIDaily Sep 28
From single-sequence structure prediction to protein fitness landscape through a composable, epistasis-aware mutation atlas

1. Cerebra-Epistasis encodes a wild-type protein once, then reuses a mutation atlas to score combinations of substitutions. This lets it explore large fitness landscapes without running sequence and structure encoding for every candidate variant.

2. Its atlas contains a score for each possible single-residue substitution and a learned representation of its potential interactions. A shared network assembles the selected entries to predict both total fitness and the contribution beyond individual mutation effects.

3. The interaction model can respond to mutational background without storing a separate parameter for every pair or higher-order combination. In GB1, it reproduced a measured example in which the interaction between sites 41 and 54 changes sign depending on the residue at site 39.

4. Structural context comes from Cerebra-Seq, a single-sequence predictor that combines ESMC and ESM3 representations and produces all-atom structures alongside residue and residue-pair features. An SE(3)-equivariant network incorporates these features into the fitness atlas.

5. In a low-data benchmark trained on fewer than 300 single- and double-mutant measurements, Cerebra-Epistasis predicted unseen mutants with three or more substitutions at a mean Spearman correlation of 0.65, versus 0.62 for Kermut, 0.61 for MULTI-evolve, and 0.56 for ProteinNPT. Its advantage generally grew with mutation order.

6. When the constituent single mutations were observed but their double-mutant combination was held out, its mean Spearman correlation for pairwise epistasis was 0.55, compared with 0.35 for Kermut and 0.19 for MULTI-evolve. This tests whether the model learns interactions beyond additive mutation scores.

7. The predicted interactions also reflected biological structure. In a TrpB dataset, the model ranked the experimentally strongest positive double-mutant interaction, V227K/S228G, first. In GB1, stronger predicted epistasis tended to occur between residues closer in three-dimensional space.

8. The authors estimate that screening every triple mutant of a 100-residue protein would take 3.70 minutes with Cerebra-Epistasis, compared with 2.57 days for Kermut. These landscape-wide times are extrapolated from measured throughput; they are not timings of a completed exhaustive screen.

9. End-to-end training improved fitness prediction while largely preserving structural accuracy. On held-out cDNA proteins, the final-epoch mean Spearman correlation rose from 0.68 with a frozen structure predictor to 0.72 when Cerebra-Seq was trained jointly with the fitness model.

10. The atlas is built around the wild-type structure, so it does not explicitly model mutation-induced structural rearrangements, conformational ensembles, or changes in binding context. Prospective protein-engineering experiments remain necessary to establish how well its predictions guide design.

💻Code: github.com/xulab-research…
📜Paper: doi.org/10.64898/2026.…
#ProteinEngineering #Epistasis #ProteinFitness #ComputationalBiology #ProteinStructure
View quoted post on X
2 replies 0 reposts 12 likes 1.9K views View on X
Anthony Gitter @anthonygitter
This perspective from Bruce Wittmann on computational protein engineering is phenomenal. Ignoring library construction and costs is a big gap between researchers only training and benchmarking models and those applying them.

arxiv.org/abs/2609.03046
arxiv.org Advances in Machine Learning for Directed Evolution: A Five-Year... The last five-plus years have seen many protein engineering disciplines transformed by advances in machine learning (ML), but the same cannot be said for directed evolution. Reflecting on a...