(working on variations of this point) it is not currently possible to assess "RSI" as a phenomenon in frontier AI companies in a scientific manner because data (pre and post!) is such a massive confounder to any simple 2D analysis of possible "RSI indicators"
Nick Vincent
Verified account This account is verified. Learn more@nickmvincent
Assistant professor @SFU_CompSci. Data-centric AI and HCI; "data leverage" for healthy data markets/ecosystems; better AI and less power concentration.
962 Following 1.1K Followers 1.9K Posts
Latest posts 5
A practice that could improve AI discourse, especially around "RSI": AI companies should share info about how they allocate {$, compute, human labor} to different domains for post-training.
Could be voluntary or enforced as part of new data transparency laws. Would help a ton!
Could be voluntary or enforced as part of new data transparency laws. Would help a ton!
It would probably be good as an emergency measure to just add a (configurable) "no judgement" Pangram integration to OpenReview and PCS and other review platforms ASAP as it's very obvious from many conversations that "community goodwill" is rapidly being torched.
And of course communities and conferences should continue to iterate on their own policies (IMO, it's obviously good to have *some* venues where everyone must solemnly swear to type every character by hand live into the text editor, but doesn't need to be 100%).
But totally disruptive to community, and frankly a bit dissociating, to be having discussion/rebuttal periods where some of the reviewers are 95% sure on vibes a review or rebuttal is 100% AI, some have Pangrammed it (or used some other tool) themselves, some have submitted 100% AI and feel bad about doing so, and others have submitted 100% AI and are totally happy with their 100% AI submission!
Noting again too -- for people involved in these debates on here, it seems super obvious that pasting in 100% AI reviews and rebuttals is disrespectful, but I think there's a larger-than-some-might guess population of people who are consuming AI marketing materials -- a PhD in your pocket, etc. pushed by the AI developers but bolstered by some universities, other actors in the tech space, some discourse on social media -- and basically feel that pasting in AI reviews is prosocial behavior!
Of course, would need to tell participants ahead of time. (And it'd be nice if this could be funded via some kind of consortium of AI developers via the various "we want to help science" programs).
And of course communities and conferences should continue to iterate on their own policies (IMO, it's obviously good to have *some* venues where everyone must solemnly swear to type every character by hand live into the text editor, but doesn't need to be 100%).
But totally disruptive to community, and frankly a bit dissociating, to be having discussion/rebuttal periods where some of the reviewers are 95% sure on vibes a review or rebuttal is 100% AI, some have Pangrammed it (or used some other tool) themselves, some have submitted 100% AI and feel bad about doing so, and others have submitted 100% AI and are totally happy with their 100% AI submission!
Noting again too -- for people involved in these debates on here, it seems super obvious that pasting in 100% AI reviews and rebuttals is disrespectful, but I think there's a larger-than-some-might guess population of people who are consuming AI marketing materials -- a PhD in your pocket, etc. pushed by the AI developers but bolstered by some universities, other actors in the tech space, some discourse on social media -- and basically feel that pasting in AI reviews is prosocial behavior!
Of course, would need to tell participants ahead of time. (And it'd be nice if this could be funded via some kind of consortium of AI developers via the various "we want to help science" programs).
There's two recurrently popular memes about AI use that pretty incisively capture what I believe to be grand challenges for AI x HCI:
- "Everyone using AI more than me is an over-reliant cognitive offloading slop machine; everyone using AI less than me is totally going to be left behind"
- "Every app should offer MCP or a nice agent-friendly API so I can interact with that app via my agent or bespoke workflow; my app should have beautiful interfaces and handcrafted buttons, as the experiences and flows I've crafted are critical to achieving user goals and producing meaningful outputs; if my app is reduced to an API, what's the point?"
(My slightly embellished interpretation.)
Here are some WIP thoughts on my these matter a lot of some remedies:
The first dilemma is very resolvable at the domain level (basically, just by doing more evaluation). That is, we absolutely can produce somewhat "scientific" recommendations about an optimal level of AI use for a given domain / task set / organizational context. (If people are willing to really surveil their own work, can get meta and use AI to help apply such recommendations).
Everybody is racing to figure this out in the context of their own organization. But right now we see relatively little sharing of in public. Model providers are kind of bailing us out in terms of some bad behaviors (because they're mining failure cases and updating the model) but we're relying on a lot of lagged social science to try to figure out the impacts.
Incentives are also currently set up to make sharing feel risky: we haven't established new Data Rules that would allow people to publicly share information without advancing their own replacement.
The grand HCI challenge here is to figure out how to enable much larger scale peer production-y and open-science-y style sharing of AI use, while mitigating the fact that going "fully open" creates serious privacy challenges and kind requires organizations to do much more digital surveillance and/or requires individuals to apply scientific management to themselves (not fun!)
I think this dilemma will eventually be resolved, but thoughtful research and design can greatly increase how quickly.
The second dilemma is also a really big deal (and has a greater risk of "spiralling out of control").
Many pieces of software rely on the shared structure imposed by opinionated interfaces to the data they produce meaningful at all. There's been a lot of discussion about LLMs might homogenize language, culture, and meaning more generally -- this is related.
Hyper-personalized and malleable software that adapts to your changing context and needs is really promising, and just plain cool. I've been playing with these ideas a lot and it brings a lot of joy to computing. I love the idea of being able to customize every button I click or form I engage with going forward.
However! A world in which everyone insists on using API/MCP to engage with all the software they use throughout a day means we could lose a lot of shared meaning.
With time, we'll likely see some social diffusion of common patterns by which people set up their own personalized malleable software (basically, lots of people will continue to use device defaults, some people will use opinionated packages, there will be communities around this just like with Linux distros and browser extensions and mobile apps); and patterns of use will reveal some information about the interfaces that people used, so that post-hoc meaning can still be assigned to records and traces that people produce.
But in the short term, if you're a data scientist with a set of user logs or form entries, your ability to confidently impute meaning will be actually very heavily diminished compared to a dataset from the "old days".
(In the extreme, a world of absolute maximal digital freedom and agency -- interfaces changing every day, data being retractable at will, etc. -- basically makes data science and machine learning impossible -- useful to think about this!)
This is all maybe obvious from a Communication-heavy lens on HCI (connective goods, etc.), but is certainly not being taken super seriously in AI discourse or in product design.
This means there is probably some need for figuring out the most core interaction types needed to maintain some stability of interaction meaning in a world with malleable interfaces.
- "Everyone using AI more than me is an over-reliant cognitive offloading slop machine; everyone using AI less than me is totally going to be left behind"
- "Every app should offer MCP or a nice agent-friendly API so I can interact with that app via my agent or bespoke workflow; my app should have beautiful interfaces and handcrafted buttons, as the experiences and flows I've crafted are critical to achieving user goals and producing meaningful outputs; if my app is reduced to an API, what's the point?"
(My slightly embellished interpretation.)
Here are some WIP thoughts on my these matter a lot of some remedies:
The first dilemma is very resolvable at the domain level (basically, just by doing more evaluation). That is, we absolutely can produce somewhat "scientific" recommendations about an optimal level of AI use for a given domain / task set / organizational context. (If people are willing to really surveil their own work, can get meta and use AI to help apply such recommendations).
Everybody is racing to figure this out in the context of their own organization. But right now we see relatively little sharing of in public. Model providers are kind of bailing us out in terms of some bad behaviors (because they're mining failure cases and updating the model) but we're relying on a lot of lagged social science to try to figure out the impacts.
Incentives are also currently set up to make sharing feel risky: we haven't established new Data Rules that would allow people to publicly share information without advancing their own replacement.
The grand HCI challenge here is to figure out how to enable much larger scale peer production-y and open-science-y style sharing of AI use, while mitigating the fact that going "fully open" creates serious privacy challenges and kind requires organizations to do much more digital surveillance and/or requires individuals to apply scientific management to themselves (not fun!)
I think this dilemma will eventually be resolved, but thoughtful research and design can greatly increase how quickly.
The second dilemma is also a really big deal (and has a greater risk of "spiralling out of control").
Many pieces of software rely on the shared structure imposed by opinionated interfaces to the data they produce meaningful at all. There's been a lot of discussion about LLMs might homogenize language, culture, and meaning more generally -- this is related.
Hyper-personalized and malleable software that adapts to your changing context and needs is really promising, and just plain cool. I've been playing with these ideas a lot and it brings a lot of joy to computing. I love the idea of being able to customize every button I click or form I engage with going forward.
However! A world in which everyone insists on using API/MCP to engage with all the software they use throughout a day means we could lose a lot of shared meaning.
With time, we'll likely see some social diffusion of common patterns by which people set up their own personalized malleable software (basically, lots of people will continue to use device defaults, some people will use opinionated packages, there will be communities around this just like with Linux distros and browser extensions and mobile apps); and patterns of use will reveal some information about the interfaces that people used, so that post-hoc meaning can still be assigned to records and traces that people produce.
But in the short term, if you're a data scientist with a set of user logs or form entries, your ability to confidently impute meaning will be actually very heavily diminished compared to a dataset from the "old days".
(In the extreme, a world of absolute maximal digital freedom and agency -- interfaces changing every day, data being retractable at will, etc. -- basically makes data science and machine learning impossible -- useful to think about this!)
This is all maybe obvious from a Communication-heavy lens on HCI (connective goods, etc.), but is certainly not being taken super seriously in AI discourse or in product design.
This means there is probably some need for figuring out the most core interaction types needed to maintain some stability of interaction meaning in a world with malleable interfaces.
This seems awesome; it would be great for the world if more people do experiments like this, and Claude and similar products enable that. (Also shoutout @imbue_ai’s Bouncer and many spiritual predecessors).
However — I think there is a pretty important narrative element missing from the discussion here (mainly responding to the many excited quote tweets and not trying to put words in the mouth of the original post).
This is really not “vibe coding an alternative recommender” in a sustainable sense. That’s just something that coding agents cannot do alone, because collaborative filtering systems have a dependency on sustained user labor/effort (even just thoughtfully engaging with infinite scroll could count).
(It’s possible now to use model weights directly as a filtering intelligence, but this will similarly require human effort to update those weights over time! Such an approach would just trade out the HN data labor for data labor from data vendor contributors, opted in LLM users, etc)
Arguably, the biggest source of value in this project is not the coding agent weights, it’s the curation labor performed by HN users! The same type of “volunteered structuring work” that made pretraining possible in the first place.
So yes, it’s great to bring attention to still-pressing issues with our recommendation paradigm and to highlight how coding agents can help,
but any solution that uses this particular approach will only be sustainable if very proactive efforts are made to sustain HN-like online communities (note that just recently Reddit announced increasing “walling of the garden”).
At the moment, many of the product and policy choices from the operators of the best AI models are mostly disrupting the ecology of online communities that made AI possible.
However — I think there is a pretty important narrative element missing from the discussion here (mainly responding to the many excited quote tweets and not trying to put words in the mouth of the original post).
This is really not “vibe coding an alternative recommender” in a sustainable sense. That’s just something that coding agents cannot do alone, because collaborative filtering systems have a dependency on sustained user labor/effort (even just thoughtfully engaging with infinite scroll could count).
(It’s possible now to use model weights directly as a filtering intelligence, but this will similarly require human effort to update those weights over time! Such an approach would just trade out the HN data labor for data labor from data vendor contributors, opted in LLM users, etc)
Arguably, the biggest source of value in this project is not the coding agent weights, it’s the curation labor performed by HN users! The same type of “volunteered structuring work” that made pretraining possible in the first place.
So yes, it’s great to bring attention to still-pressing issues with our recommendation paradigm and to highlight how coding agents can help,
but any solution that uses this particular approach will only be sustainable if very proactive efforts are made to sustain HN-like online communities (note that just recently Reddit announced increasing “walling of the garden”).
At the moment, many of the product and policy choices from the operators of the best AI models are mostly disrupting the ecology of online communities that made AI possible.
Weekend hack: HackerTube. I was annoyed with YouTube's recommendations so I had Claude filter down 43M comments over 20 years of HN to 6,454 videos that keep being recommended. *Significantly* better viewing experience, ~2 hours with Opus 5.5:View quoted post on X
hackertube.org
- Excellent recs without any logins (state is kept in local browser session)
- Thumbnail is a representative frame from middle of the video instead of the super duper A/B test hyper-optimized user uploads
- Underlying data from the comment analysis as one zip file, point your Claude at it and ask for your own version
- HackerTube Longs ✊
A bit more on the process:
I use Claude Code (desktop app for Mac), Opus 5.5 on Medium effort as my default. I only talk to Claude and have it do everything, the only "manual work" I helped with in tandem was in the browser (reminiscent of MenuGen): 1) I had to create some kind of API key from Google Cloud UI to query YouTube for video metadata, 2) I booted up a VPS box for serving and 3) I bought the domain name. Once the box was up I gave Claude the IP and told it update, secure, harden, deploy and optimize the code, and it did the whole thing.
Many awesome projects that would have normally taken weeks/months even can now be few-shotted for breakfast. What else are we not thinking of.