Pinned post
FleetingBits
Verified account This account is verified. Learn more@fleetingbits
sf thinkcat
393 Following 11K Followers 12K Posts
Latest posts 5
i've been working with claude on a programming game, inspired by screeps, for my personal website, fleetingbits.io; there are two factions, one based on scheme and the other on apl
1:08
some quick thoughts on metr's evaluation
1) metr reports that anthropic’s models are improving 1.9x as fast as they were before october 2025
2) the main evidence in the report comes in the form of an expanded version of anthropic's internal eci and reports from researchers
3) metr also says that they have other evidence, which anthropic redacted; odds are that this is information about algorithmic efficiency over time
4) the anthropic eci information looks convincing to me; it looks like anthropic is no longer releasing its best models and this has made public eci less useful
5) when you add in historical models of other providers to the eci, the speedup falls to 1.4x, since anthropic models were weaker than the frontier models of other providers (e.g. openai) earlier in 2025
6) researchers reported that they are able to get a lot of uplift from research agents; they can launch more experiments; less handholding of experiments
7) this makes sense, the uplift appears to have began sometime around opus-4.5, which appears to have began to automate coding for many programmers
8) this just emphasizes how important it is for anthropic and other labs to release their internal eci on a regular basis so observers have better information
9) also, they should release more information on their research agent spend; so far, openai has been the most transparent on this topic (~33 day doubling time)
10) i think that more uniformity of reporting would do a lot of good; regular internal eci reporting, standardized research agent reporting
11) since, it would make the public discourse much more regular, well-informed, there would be much less alarmism
12) note, i think that a lot of people will overreact to this because they view acceleration as the start of the end of the world; but this says little about alignment
13) by the way, i do wonder if model 2 is just mythos 5.1 with additional post-training for ai r&d; it would make sense for the labs to attempt to separate this out
14) since, it would make those capabilities very hard to impossible to distill and it would presumably also make classifiers less important / upset users less
1) metr reports that anthropic’s models are improving 1.9x as fast as they were before october 2025
2) the main evidence in the report comes in the form of an expanded version of anthropic's internal eci and reports from researchers
3) metr also says that they have other evidence, which anthropic redacted; odds are that this is information about algorithmic efficiency over time
4) the anthropic eci information looks convincing to me; it looks like anthropic is no longer releasing its best models and this has made public eci less useful
5) when you add in historical models of other providers to the eci, the speedup falls to 1.4x, since anthropic models were weaker than the frontier models of other providers (e.g. openai) earlier in 2025
6) researchers reported that they are able to get a lot of uplift from research agents; they can launch more experiments; less handholding of experiments
7) this makes sense, the uplift appears to have began sometime around opus-4.5, which appears to have began to automate coding for many programmers
8) this just emphasizes how important it is for anthropic and other labs to release their internal eci on a regular basis so observers have better information
9) also, they should release more information on their research agent spend; so far, openai has been the most transparent on this topic (~33 day doubling time)
10) i think that more uniformity of reporting would do a lot of good; regular internal eci reporting, standardized research agent reporting
11) since, it would make the public discourse much more regular, well-informed, there would be much less alarmism
12) note, i think that a lot of people will overreact to this because they view acceleration as the start of the end of the world; but this says little about alignment
13) by the way, i do wonder if model 2 is just mythos 5.1 with additional post-training for ai r&d; it would make sense for the labs to attempt to separate this out
14) since, it would make those capabilities very hard to impossible to distill and it would presumably also make classifiers less important / upset users less
1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.View quoted post on X
some quick thoughts on the rsi risk in anth's s-1
1) anthropic updated its s-1 yesterday and one of the changes was adding an additional item to its risk factors, describing the risk of research acceleration
2) now, if we just look at public models, models have pretty much been improving on trend; there hasn't been any real direct evidence of an acceleration
3) there are some interesting potential signs, probably most interesting that openai and anthropic have had enormous growth in their spend on research agents
4) basically, spend doubling on a ~33 day period; on trend, openai would be spending 4% of their ai r&d compute on research agents now
5) and, presumably they would be doing this because they provide a lot of value; but, this has not yet resulted in an acceleration we can see
6) if frontier labs just released the benchmark scores of their best internal models, we would have a lot more information; even better if outside parties could verify
7) and, at this point, i basically think frontier labs need to release the benchmark scores of their best internal models to keep the discourse healthy
8) otherwise, everything ends up being boosterism, etc; it feels a lot like when people thought openai had agi in a box somewhere before gpt-5
9) it would also be helpful for anthropic and openai to release the equivalent of the oai "research acceleration" blog post monthly; since the data is somewhat stale
10) btw, for reference, companies add pretty much everything to their risk factors, because there is much more cost to leaving something out than adding it
11) for example, a subsequent risk factor is dedicated to the fact that "[anthropic] may not be able to obtain the datasets that [it] need[s]"
12) note, i'm agnostic at this point as to whether a research acceleration has occurred or not
1) anthropic updated its s-1 yesterday and one of the changes was adding an additional item to its risk factors, describing the risk of research acceleration
2) now, if we just look at public models, models have pretty much been improving on trend; there hasn't been any real direct evidence of an acceleration
3) there are some interesting potential signs, probably most interesting that openai and anthropic have had enormous growth in their spend on research agents
4) basically, spend doubling on a ~33 day period; on trend, openai would be spending 4% of their ai r&d compute on research agents now
5) and, presumably they would be doing this because they provide a lot of value; but, this has not yet resulted in an acceleration we can see
6) if frontier labs just released the benchmark scores of their best internal models, we would have a lot more information; even better if outside parties could verify
7) and, at this point, i basically think frontier labs need to release the benchmark scores of their best internal models to keep the discourse healthy
8) otherwise, everything ends up being boosterism, etc; it feels a lot like when people thought openai had agi in a box somewhere before gpt-5
9) it would also be helpful for anthropic and openai to release the equivalent of the oai "research acceleration" blog post monthly; since the data is somewhat stale
10) btw, for reference, companies add pretty much everything to their risk factors, because there is much more cost to leaving something out than adding it
11) for example, a subsequent risk factor is dedicated to the fact that "[anthropic] may not be able to obtain the datasets that [it] need[s]"
12) note, i'm agnostic at this point as to whether a research acceleration has occurred or not
