Twitxy Saved

Roi Cohen

Verified account This account is verified. Learn more

@roicohen9

PhD candidate working on cognitively-inspired AI models | HPI, currently at @stanford |
Bay Area
Tel Aviv roicohen.org Joined January 2014
147 Following 199 Followers 36 Posts

Latest posts 5

Roi Cohen Verified account @roicohen9
What if fine-tuning took a paragraph and a few seconds?

We trained a meta-model that writes the weight update for Qwen3.5-2B from a one-paragraph task description, in seconds on one small GPU, no gradient step.

Across 75 public benchmarks, the update written from just one paragraph improved the base model's score on 59 of them and lowered it on 8. For example, ComVE jumped from 50.0 to 91.1 percent, three-object logical deduction went from 46.4 to 82.0, and on GSM8K with the final number only, from 5.7 to 18.6, ahead of a LoRA trained on up to 1,400 rows of GSM8K (12.7).

Post, figures and every benchmark: roicohen.org/blog/compress-…
0:30
30 replies 47 reposts 217 likes 23K views View on X
Roi Cohen Verified account @roicohen9
New work on evaluating the "Ripple Effects" of KE methods!

We propose a new benchmark, 'RippleEdits', and show current KE methods have yet to reach satisfying performance - suggesting that we still have a long way to go!

Link to github (code+data): github.com/edenbiran/Ripp…
github.com GitHub - edenbiran/RippleEdits: Evaluating the Ripple Effects of Knowledge Editing in Language... Evaluating the Ripple Effects of Knowledge Editing in Language Models - edenbiran/RippleEdits
AK Verified account @_akhaliq Jul 25, 2023
Evaluating the Ripple Effects of Knowledge Editing in Language Models

paper page: huggingface.co/papers/2307.12…

Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple effect'' in the form of additional facts that the model needs to update (e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we propose a novel set of evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct , a diagnostic benchmark of 5K factual edits, capturing a variety of types of ripple effects. We evaluate prominent editing methods on , showing that current methods fail to introduce consistent changes in the model's knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.
View quoted post on X
0 replies 1 reposts 15 likes 1.1K views View on X
Roi Cohen Verified account @roicohen9
LLMs know much about the world. But what do they know precisely? Could we explicitly extract this factual knowledge?

In our new paper “Crawling the internal Knowledge-Base of Language Models” we propose a simple mechanism to address these questions.