Twitxy Saved

Howard Chen

Verified account Engram

@__howardchen

Research @EngramLab
Prev: PhD @Princeton
howardbchen.com Joined August 2013
1.5K Following 1.5K Followers 233 Posts

Latest posts 5

Pinned post

Howard Chen Verified account @__howardchen
Text memory is great. But how can the agents stay familiar with these ever-growing notes? In this blog, we explore the dynamics of learning and text memory -- we find that training is a compelling option to use notes better and can endure in a changing world!
Engram @EngramLab Nov 4
In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes.

It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.
View quoted post on X
1 replies 1 reposts 20 likes 1.2K views View on X
Howard Chen Verified account @__howardchen
AI agents now seem to solve the hardest problems in math and science, but they haven't yet deepened our understanding at the same rate. They are great problem-solving machines when the objective and optimization process are clear, but discovering insights is a lot fuzzier!

We build EurekaBench to measure this gap using 6 natural science domains (neuroscience, geophysics, plasma physics, astrophysics, computer science, and chemistry). We test if AI agents can complete the whole scientific discovery loop from understanding the observations, explaining the data by describing the mechanism (e.g., equations), and eventually discovering insights.

Results show that the agents don't advance on both improving accuracy and finding insights at the same pace! A lot more details in the thread. Fantastic work led by @JiayiiGeng.
Jiayi Geng Verified account @JiayiiGeng Oct 2
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?

As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?

To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
View quoted post on X
4 replies 7 reposts 28 likes 2.7K views View on X
Howard Chen Verified account @__howardchen
The paradigm in the last few years has been scaling reasoning to get better performance. It will remain to be extremely important (yet costly).
There's another axis to scale: through study, our agent internalizes the env and navigates it with familiarity and intuition. We are seeing sparks of something different, and it's just the beginning.
Engram @EngramLab Aug 17
Today we're publishing our first research blog, Understanding a Law Firm through Study.

We're sharing a glimpse of a future where agents are trained with native memory:
View quoted post on X