Daniel Rupawalla

excited to launch bank-world. this is arcanum's first in-house benchmark. SOTA models are still unready to be deployed autonomously within banks.

in bank-world, agents are dropped into a large bank setting with hundreds of thousands of queryable entities. agents in single-turn tasks role-play as real people within the bank (claude tag, grok bot style), in multi-turn tasks, agents role-play as chatbots, assisting a simulated user agent. scores and the full paper are attached.

this is the first of many new releases we're excited to share, and we're continually updating and building better evals that truly resemble real-world enterprises.

the main failures we are noticing is that agents will often act without being given explicit authority in long-running evals; can still succumb to simple mistakes like double-charging customers, forgetting payment history, etc; and will not accurately complete a full task, even if the main source of work was completed.

*we are not publicly releasing any of the tasks yet to prevent distillation (on request, we'll share the worlds & tasks). we plan on releasing tasks as they become saturated. new models like carbon and fable 5.2 will be added soon.*

"She is seeing all of this for the first time. She does not even know what she is." - jurassic world (2015)

the signaling some frontier labs have around their models feels similar to the way the indominus rex was referred to. all of the safety measures, reports, and research...

i don't really see an argument against AI acceleration:

back in debate, if you could tie your argument to the most lives affected in a plausible link (w/o K's or non-trad args), you'd usually win, after all you had the biggest impact. thus, i make my impact calculus:

for model acceleration:

1. super-intelligence (given a 5-10 year timeline), is the only argument you need:

*as a counter to a lot of the against arguments (3), if we end up in an ultron-type scenario, vision is the best counter. thus, having a more powerful, aligned model is a naive but realistic argument.

1a. solve all forms of disease, we're seeing early progress in cancer treatments, wet lab experiments, and scientific development. this feels like an inevitable. we will solve medicinal research undoubtedly with LLMs. in fact, LLMs will allow for specialized medicine to become possible (individual cancer treatments to target different types, etc).

1b. given western dominance, we solve all geopolitical issues. this one is more complex, but from the way i view LLM progress, RSI means that any given lead MUST be maintained and will continue to grow. if SOTA makes the next model better, faster, iteration cycles will continue to accelerate, and being 3 months behind now, will be years in the future. it is IMPERATIVE that we maintain that lead geopolitically. after all, if any belligerent foreign actor gets access to a model that can outperform ours, we run the risk of bio-weapons we can't understand, shifts in technological domination, robotics & military technology at levels we can't combat, etc. imo, almost everything is downstream of intelligence, and most countries used to brain drain into the west. now, people aren't the bottleneck to intelligence/research, models are, so giving up the lead functionally is the same.

1c. the most downstream of all of these is we just solve all problems. math, biology, etc, all forms of open research will be solved. we can do immensely powerful things via this technology. in the next 10 years, i want to be able to rip a prompt to the SOTA model that says something like "build me a rocket ship and take me to my friends on the moon", and i would want that mode to exist. the only way we get there is via more intelligence and investment into physical AI. every luxury we could ever possibly want has come via some sort of intelligence, whether it be taste-based or science-based.

1d. a lot of the above arguments have a very clear bottleneck, physical materials. i personally am under the belief that intelligence will help us get around these constraints to an extent. for example, if a certain material is too difficult to get and/or too expensive, i believe having a smarter model may help synthesize a new version or a replacement. there are ways to lever intelligence in different angles to accelerate all development.

against acceleration (similar timeline and probabilities imo):

1. job loss / identity loss. there seems to be two-parties here and both are valid. job loss is your average person losing their knowledge work job (accountants, lawyers, etc) to a model. identity loss (more frequently in SF), is people who fear not being able to contribute to the world at all anymore. identity loss seems to be a bigger fear in my mind, given that if people were to lose all forms of identity it leads to heavy civil unrest / revolution.

2. environmental issues. data centers & disruption to the community and environments there. electricity costs from models.

3. models turning evil and killing all of us or something along these lines. given all of the hacks + fear mongering from labs and politicians alike, i think this is a really big concern, not sure if it should be though. alignment & safety are problems that undoubtedly need to be understood and worked on though.

3a. i think the bigger argument here is that by us not understanding interp and potentially not being able to understand and analyze traces perfectly, it may be difficult to catch or understand latent patterns in models that could be dangerous. as models get smarter we understand them better, and hopefully more aligned, i think this will become less of an issue. moreover, bad actors using good models (cyber attacks, bioweapons, etc) for bad things, is my biggest concern and the most probable of negative outcomes, and the most tangible. the naive argument against all of 3 is that a smarter, more aligned model just counters though.

4. hemming of people's intellect & an over reliance on LLMs to do all thinking, thereby stunting the growth of students/people in general. valid concern and i think this needs some level of thought on how to structure new learning systems around LLMs. for the first time ever, students can ever-growing, customized teachers, we should empower that. i wish i had this technology growing up. however, used incorrectly, i can see this happening.

5. power concentration in a few companies. tbd on how aligned those companies are. this is a bit of a red herring, but i think the alternative to western concentration is belligerent concentration, which is worse. though i think this is a valid argument (not sure if there is any single government or company that should have this kind of power though, which is why the race b/w model companies is actually good).

6. fake information/deepfakes/etc. this is a byproduct of technology getting better, not really sure there's a way around this. regulation, watermarks, filtering, and education are gonna be important here, similar to internet safety lessons imo.

although i can see both sides, it feels like for acceleration clearly outweighs.

hello all, as part of an internal physics agent evaluation, we put codex & claude together on resolving the quantum four-outcome algebraic unbiasedness problem. attached a pre-print and the lean proof below.

during the 3 day process i let them run for, the agents collaborated, although i had to do some prompt guiding & steering them in the right direction occasionally.

this builds on sébastien designolle and máté farkas’s theorem 7, the “hadamard–clifford construction,” which established the three-measurement case. the open question was only proposed a few months ago, but the space was an area i specialized in during high school and has always interested me (super niche quantum problem).

the result is a fundamental limit on this class of quantum measurements. basically, four measurements with four outcomes each cannot satisfy this exact unbiasedness rule, no matter how large the quantum system is. the broader theory connects to quantum cryptography and reconstructing quantum states.

tl;dr of the problem & solution:
1. take four ways of measuring a quantum system, each with four possible answers.
2. start with a state vector that gives a definite answer to one of the measurements, and pick an answer from each of the other three.
3. filter the vector through those answers in all six possible orders, then back through the starting answer. each filter keeps the part corresponding to the answer you picked.
4. average the resulting vectors without rescaling them. the rule requires exactly 1/64 of the original vector, whatever answers you chose and whichever measurement you started with. this averages amplitudes before calculating probabilities.
5. we proved that satisfying this rule would force a sum of squared quantities to be negative, which is impossible. three measurements can satisfy their corresponding rule, but four cannot; and making the quantum system bigger won't change that.

evals being tied to real productive work feels like a lost art.

pass@k is irrelevant when the eval you're creating doesn't measure anything economically relevant (if the goal of the eval is to then sell that data to labs).