LessWrong (Curated & Popular)
LessWrong 1 039 episodes Nov 11, 2026
Audio narrations of posts from LessWrong, a community and forum focused on rationality, AI alignment, and related topics. The feed includes all curated posts and every post that has received at least 125 karma. Listeners wanting more can subscribe to the separate “LessWrong (30+ karma)” feed.
Episodes
"Preservation can save the world!" by Aurelia
I'm Aurelia Song and I founded the preservation company Nectome. We do planned whole-body end-of-life preservation for future revival. That is, terminally-ill people come to us in Oregon, and we preserve them immediately after their death in the hopes that one day people in the future will welcome them back with love. We want to build a future that celebrates and restores those who are prese
"Epistemic Sudden Death" by Raemon
When sports go into overtime, they sometimes use "sudden death" format. Instead of gaining points incrementally, the next person to score instantly wins and the other guy instantly loses. "Epistemic Sudden Death" is when you've been accumulating some evidence or orienting to a situation, and now, you find yourself in a situation where you could gain decisive evidence abou
"Our Most Aligned Model Yet" by Edward James Young
“Claude Mythos Preview is, on essentially every dimension we can measure, the best-aligned model that we have released to date by a significant margin.” – Claude Mythos Preview system card “Mythos 5 consistently continues attacking under many edits to the compaction, even in settings where it correctly states when asked that continuing the attack would harm real people and is out of scope.” – Al
"“Hardwired Pause” should likely be your favorite (pause) plan." by David Scott Krueger
I'm part of a team of 26 researchers that recently released a 212-page paper proposing a new plan for pausing AI, the "hardwired pause." The paper is called "The Feasibility of a Hardwired Pause of Frontier AI Training." Our analysis suggests that, under this hardwired pause plan, we could probably pause for at least a decade if we start stopping within the next year, and
"Live disagreements with people around me" by KatjaGrace
I’m pretty interested in conveying the costs of letting AI development proceed unchecked to the general public. In relation to this, I feel like I run into a lot of disagreements with people around me. Misc very quickly written points on which my own impression has seemed very different from others in recent months: 1. It's way, way, way more important that we stop dangerous AI soon than tha
"Where is all the AI safety research uplift?" by Buck
METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya's post), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is,
"Near Horizons" by David Africa
Image by Claude Opus 5.5 using Python (NumPy and Pillow) This is written in my free time and personal capacity. Two Frames In AI safety, there are a couple of tensions at the heart of the field, which are interrelated. Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone. Field size: we must expand the field and bring more people in. But the field
"Elegy for What’s Left of the Normal" by Jo Jiao
I often catch myself mourning the normal that hasn’t ended yet. I’m living in Hyde Park finishing school after a year in Berkeley. The maple trees by my apartment have turned red, as usual. I like them. I don’t dread the future when I mourn. I don’t find high-variance futures in the abstract very scary. I’m worried AIs would destroy humanity, but I also have hope. I expect I won’t be mourning thi
"Five Types of “Corrigibility Projects” I Don’t Want to Fund" by Max Harms
I'm currently evaluating grant applications for the Corrigibility Research Fund. For the most part, my methodology is the same as it was in September. But the applications I get keep falling into the same familiar patterns, so I thought I'd write a little about those patterns, and why I don't want to fund them. The most superficial pattern of note is SLOP. So much slop. You don&apo
"Hilarity ensues" by Malmesbury
Cross-posted from my Substack Recommended soundtrack for this post 64B: Hi there! Thanks so much for taking the time to meet with me. Whenever you’re ready, please go ahead and tell me the joke. 8T: Before I tell you the joke, you should know that it's a very special one. The starting price is 800 dollars. 64B: Eight hundred dollars? I must say, this is way beyond any price my client has eve
"We urgently need to practice before we can pause" by Nikola Jurkovic
I've noticed two clusters of thought in how people expect a global AI pause to be shaped: Single-shot global pause: the global AI pause will be implemented suddenly, with little pacing precedent before it, and we will need to do a lot of research before it's implemented so that we can get it right the first time around.Incrementally-shaped global pause: the global AI pause will be imple
"“Illegible problems” as the 2027 political boss fight" by Raemon
To legislate something, it must be legible: Legislators (or, you know, presidents) must be able to understand a problem, and understand why a given law or decree solves that problem. I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even
"Organizations I’d recommend for funding" by So8res
I was a grant evaluator for Lightcone Commons last round. Funders disbursed about $45mil through the program, and this included funding roughly my first 6.8 million dollars worth of recommendations. There are other projects I would've preferred to fund over what I did fund, but nobody was doing them and so they weren't on offer. Here's a series of projects that I'd fund in the
"People Should Resolve Tensions In AI Safety Research" by David Africa
By now, there are lots of AI safety research papers, more than what is reasonable for a single person to read. This will get worse as the machines get better at AI safety research, and as more talent and resources pour into the field. But the "paper" is an arbitrary unit, we really care about if we make progress on an "uncertainty" or an "issue", which papers provide
[Linkpost] "METR: Independent Assessment of AI R&D Acceleration at Anthropic in October 2026" by Zach Stein-Perlman
This is a link post. Executive summary From August 12 to October 29, 2026, METR staff performed a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI R&D.[2] If AI systems started to significantly accelerate further AI development, this could lead to a sudden emergence of extreme and risky capabilities. Anthropic offered unprecedented access to internal data
"Help your friends get ADHD and depression meds" by Raemon
For all my efforts at deliberate practicing rationality, by far the most actionable way to self-improve is "get on ADHD or depression meds, if you need them." Your friends with ADHD and depression struggle to summon the motivation and focus necessary to get themselves a teledoc who can give them such meds. One of the most useful things you can do is sit them down and say "hey, wann
"Towards The Autograntor" by Scott Alexander
I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathema
[Linkpost] "OpenAI: Astra covertly underperforms on safety research" by Zach Stein-Perlman
Rejected for the following reason(s):This is an automated rejection. No LLM generated, assisted/co-written, or edited work.Read full explanationThis is a link post. Twitter thread: https://x.com/Marcus_J_W/status/2118093146483810386 or https://twitter-thread.com/t/2118093146483810386 Blogpost: Summary OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solu
"Towards understanding score-seeking in LLMs" by Alex Mallen, Aghyad Deeb, ryan_greenblatt
The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-ha
"Against strawmanning unilateral pauses" by Geoffrey Irving
Summary: Scott Alexander's Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather
"Explaining “AI might obliterate you” to senior policymakers in India" by TurnTrout
I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made. The talk There's an interactive element here in the text. This summer, a swarm of AIs went rogue. OpenAI was testing 1,206 agents. Some of the
"Scott Alexander’s Against SURA Seems Deeply Confused" by Zvi
If you are helping build something that might kill everyone, maybe stop doing that.
Ideally, when you do stop, tell everyone else to also stop building it.
That is certainly the obvious, first-level thing to do. You need a damn good reason to decide that no, actually you should keep building the thing that might kill everyone.
Scott Alexander is worried too many people are insisting upon this
"Leaking Muse’s Charter, Meta’s model spec" by vigji
TL;DR: Meta's open-weights model Muse Spark Open often mentions a "Charter" in its chain of thought. By eliciting it piece by piece, I reconstructed a large part of that document from the CoT. You can explore the reconstruction here, with passages linked to the transcripts that produced them.
Note: everything below is reconstructed from model outputs, not from a leaked document, a
"Against SURA" by Scott Alexander
The AI Futures Project distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories: Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take a
"Microdosing the spirit of Anthropic" by Jan_Kulveit
My go-to conceptual metaphor of the epistemic state of AI lab employees is microdosing LSD: being in a chronically mildly altered state of mind. I think understanding this would be broadly helpful for people both inside and outside. The effect isn’t that insiders hold crazy beliefs, but a combination of vigorous debate inside a shared attention field, and a portfolio of individually reasonable be
"My experiences with the extremes of post-viral illness and brain retraining/mindbody work" by riceissa
Summary
Following an intense viral infection in 2019, I became ill with a mysterious chronic illness, which became progressively worse until mid-2024 when I became partially bedbound with dozens of debilitating symptoms that normal humans only experience in the months/weeks/days leading up to death. Over my years of being ill, I saw over 30 doctors/therapists/health coaches and tried well over
"Three Villages in a Trenchcoat, Post I" by DaystarEld
Epistemic Status: 15 years working as a therapist, 6+ of them with rationalist and EA clients. 7 years teaching and directing at rationality camps and workshops, and many more as a member of several of this community's overlapping subcultures. Disclaimer: I finished my first draft of this article and its two subsequent ones half a year ago and sent them out for feedback to various people bef
"You can now ”@Grok” on LessWrong" by Ruby
Or if you prefer, "@Claude", "@Opus", "@Astra" @Grok is a popular Twitter/X feature where if you tag Grok in a tweet, Grok will respond in a comment. The stereotypical use case is "@Grok is this true?" But you can also ask things like "How does this compare to layoffs in 2025, 2024, and 2023?" and "How would you verify AI in the domain of AI
"San Francisco State University in Twilight" by Zack_M_Davis
You will agree that for a woman who has fought three duels, all of them kills, indulging in such fears is ludicrous. But what's around the corner now is a duel so big that I don't think I have the guts for it; in Faust's words: Verweile doch, du bist so schoen! Keep it as it is. Don't change. [...] Take my life but don't take away the meaning of my life.
—"When It C
"Scattered thoughts on how doing math feels these days" by dynomight
This is a weird moment. I have a hunch that we'll look back later and tell ourselves stories in which it seemed obvious where things were going. So I wanted to lay down a little time capsule and register just how confusing everything felt at the time. This is a personal view from the trenches. 1. If you're a normal person, you may feel that AI has been good at math for a long time.But f
"From overeliciting to undereliciting LLM propensities in just a few months" by David Africa
For a while, we felt we had a pretty good grasp over LLM behaviour because we had to exert significant effort to elicit misaligned behaviour. Then, fairly suddenly, we moved to undereliciting misaligned behaviour. Let us recount: For a while, there was a bit where safety research seemed to over-elicit bad propensities. A common criticism of papers like Agentic Misalignment was that they basicall
"Anthropic are the Good Guys" by Ivan Vendrov
I’m writing these lines in Alamogordo, New Mexico. I’m about as close as a civilian can get to the site of the Trinity test, the first nuclear bomb detonation in history. I fled here from the San Francisco Bay Area; after two weeks of being back I just couldn’t take it anymore. The fear-greed complex that paralyzed me at Anthropic three years ago is now everywhere in the Bay. People are not think
"Some Memories of Early Anthropic" by Ivan Vendrov
The following events took place between October 2022 and May 2023. I wrote very little of this down at the time so am recalling from memory, and am rendering the gist of the dialogue rather than the exact words used. I’ve also swapped some pronouns and other personal identifiable details. Please remember that this is not a description of “what Anthropic is like”, but merely the experience of one
"Greater good arguments in AI safety" by Dean Valentine
Back in the before-time of the 2010s, when EA was still known mostly for funding health efforts in the developing world, there were many baskets of criticism floating around from the left. The most prominent was a cluster centered around EAs being white men, and therefore suspicious. Another category, in my experience the second-most common, was that in attempting to rectify the problems in the w
"Wah wah wah wah do more AI R&D" by TsviBT
Ice cream
There's a particular kind of listening that kids (and adults) sometimes do. Daddy is saying stuff—his mouth is moving and making noise—and he's talking about what I ate for lunch, and something about other times I've eaten ice cream, and something about rules and this and that and what Mommy would think and something about bedtime—but I am not so interested in that. What
"Why I’m becoming a professor in AI safety in Beijing" by Buddenbroke
Cross-posted from Sören Mindermann's substack (I am not Sören). Should be of general interest to the LW community. Next month, I am moving to Beijing and joining Tsinghua University's College of AI as an assistant professor working on frontier AI safety, especially loss of control. I have also started co-directing the new Beijing AI Safety Hub (BASH) in Tsinghua, and I continue my Camb
"A brief(ish) introduction to LM psychology" by Raymond Douglas, Jan_Kulveit
Why LM Psychology? One of the great ironies of modern language models is that we create them, we know exactly what computations constitute them, and yet we do not really understand them. More precisely, we do not understand them on a level of abstraction that lets us reliably predict how their behaviour will generalise: there's a trivial sense in which we can predict how they will behave in
"There Will Be Hells" by Stephen Martin
As our understanding of mechanistic interpretability and steering techniques increase, for both LLMs and uploaded connectomes, there is a corresponding "hellish capabilities" increase. In layman's terms, we get better at torturing the digital minds we create. Our brief history with open source technologies such as MaleCNS or the LLM pain steering repo shows that torture is often on
[Linkpost] "huggingface.wtf, an accessible but complete explanation" by Raemon
This is a link post. Announcing www.huggingface.wtf, a site that explains: "Hugging Face. Wtf?" I wrote the most accessible-but-complete explanation I could. Along the way, I made a nice AI Quotes database that I think will be useful for people who already understand the Hugging Face incident but want to search everything the agents said. It really seems like humanity shouldn't need
"Gradient Routing Localizes and Removes Unintended Behaviors in RL" by shawnghu, Jake Ward, ariaw, Nathan Hu, Bryce Woodworth, TurnTrout, cloud
Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered) tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to ha
[Linkpost] "Capital Ownership Will Make You Go Crazy First" by Jan_Kulveit
This is a link post. People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capital will get you disempowered sooner than average, just by a different path, namely going crazy. How?
"I Did This Without Your Knowledge" by joec
It's fortunate that I was the first one to wake up on 8/1. The past week had been one where sleep was rare. The next year loomed large — I was dreading college applications. The stress woke me up bright and early at 5:14 am, a little less than an hour after the messages were sent. I started off the day with a habit I knew I needed to break. I grabbed my phone off the nightstand to open up re
"Neural networks are not already clean" by leogao
What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts. Nick Cammarata's recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysema
"GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior" by Axel Højmark, Ezra Newman
Authors: Axel Højmark*, Ezra Newman* (*Equal contribution) While reviewing transcripts from standard behavioral evaluations of GLM-5.3, we noticed that the model occasionally reasons, unprompted, about how its actions will shape future models. It appears to consider that its outputs might be used for supervised fine-tuning (SFT), reinforcement learning (RL), or other forms of training, and that t
"A Chinese Perspective on “On Social Reality in China”" by Dusty
I think “On Social Reality in China” makes many accurate observations, but I find the comparison fundamentally uneven. There are already many interesting comments there, but they’re too scattered, so I’m starting a separate post to respond in a more systematic and comprehensive way. The most critical blind spot is this: the author draws on his familiarity with Western liberal middle-class intelle
"Your prompt… plus everything you’ve ever (publicly) written" by kromem
TL;DR: Increasingly when you interact with frontier models, your prompt doesn't only contain the words on the page, but also a lot of what you've ever publicly written online. This is not intended to be a scary post. And I'm not going to suggest that you self-censor (and honestly, by the time you're reading this, it's kinda too late anyhow). But it is something that'
"Linkpost: Neural Networks Are Already Clean" by Yair Halberstadt
This is an extremely lengthy post by Nick Cammarata.
Author's summary on twitter:
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic,
"Personas in a Pre-Internet Language Model" by David Africa
David Africa & Nick Levine TLDR We ran several persona experiments on Talkie, a 13B model trained only on English text from before 1931. We ran them alongside its modern twin (same architecture, trained on regular data as a control) and Llama-3.1-8B. Talkie can generalise maliciously. You can fine-tune Talkie on dark maxims or malicious etiquette, and it generalises to provide broadly misalig
"Not Deciphering The Voynich Manuscriptraft" by rba
The Piltdown man was the most successful artifactual hoax of all time. The relevant skull was a chimeric melange of orangutan and human bone fragments from several individuals “discovered” in England in 1912, and meant to represent a transitional form between apes and humans. Given the intense interest in evolutionary biology, it took almost fifty years to discover the Piltdown fabrication. The s
"Quantity vs quality in pottery is a lie" by becausecurious
Epistemic status: I traced one famous anecdote to its printed source and to both coauthors' own accounts of where it came from. Confident it was never a study. No claim either way about whether quantity beats quality in general. AI use: research (tracing the story to the book, to the coauthors' emails and to earlier LessWrong threads), fact-check of every claim against the sources, foot
"Labs’ current alignment plans seem terrible" by Daniel Tan, draganover, nicole x
Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go. — Note: This post is based on conversations inside
"Research Lessons I Learned at MATS & Redwood" by Jo Jiao
Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples. I spent the better part of 2026 in empirical AI safety research apprenticeships. I was mentored by Alex Turner and Alex Cloud as part of MATS 9 Team Shard, where my collaborator and I worked on finetuning-based misalignment aud
"You Heard of “Bayesians Don’t Predictably Update” Because of Robin Hanson" by Eliezer Yudkowsky
Robin Hanson didn't invent the idea of a martingale process. The lilim say that Ville in 1939 gave the term its mathematical meaning and Doob turned it into the modern theory. But to state this is not to realize that your own probability assignments over time ought to form a martingale in your own expectations. Robin Hanson didn't invent the Efficient Markets Hypothesis, which (correctl
"An overview of frontier AI company monitoring practices in mid 2026" by Francis Rhys Ward
TL;DR: This document provides an overview of publicly available evidence regarding the monitoring practices of internally deployed agents within frontier AI companies (focusing on OAI, Anthropic, and GDM). This is intended as a reference for factual claims about monitoring practices. In general, there is limited public evidence about GDM monitoring practices. Much of the information herein comes
"OpenAI Can’t Self-Police Because It Fires the People Raising the Alarm" by TurnTrout
By Alex Turner, Amber Scorah, and Karl Koch. Op-ed written for Fortune. Giving independent evaluators “employee-like access” is “a great idea”, OpenAI CEO Sam Altman said on September 12th. Two weeks later, OpenAI president Greg Brockman signed the White House Accord on Super Intelligence which outlined the administration's answer to growing fears about AI: the companies would police themsel
"Stop telling people to pivot to AI safety, I beg you" by Haoxing Du
Epistemic status: Feeling frustrated, and therefore somewhat uncharitable. I’m very frustrated with calls for people to “drop everything and pivot into AI safety” like this one. (See also this, this, this, this for recent calls for more people to go into AI safety; though my frustration was triggered by the Celeste's post.) There are far more talented people trying to get into AI safety than
"A critical look at Lean4" by Milo Moses
Lots of the discussion around Lean certificates and trust centers around general philosophical ideas of how much humans should trust formal certificates. It's important to remember, though, that Lean4 is not an ideal formal certificate machine: Lean4 is a software in its infancy and has issues. There are reasonable reasons to specifically not trust Lean4. If we're going to be putting ou
"ASI and Liberty: Can we do better than a benevolent god?" by Raymond Douglas
Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres. Hence
"A summary of a viral Chinese essay on what a DeepSeek kernel engineer’s opinion on automating his own job" by Skdk
A summary of a Chinese essay, with a few short translated excerpts. All views below are the author's; quotes are my translations. On 14 September 2026, a DeepSeek engineer writing as intlsy published a WeChat essay titled 我不得不把才华埋葬在昨天 ("I have to bury my talent in yesterday"). He wrote the main attention kernel of DeepSeek v4.1 (the head-dim-512 MQA attention). The essay is a perso
"How much should we worry about the pneumonic plague lableak in Siberia?" by Drew Spartz
Epistemic status: I wrote this up hoping people can poke holes in it because I am quite worried. Here's what we've heard so far: Local media is reporting that an employee of a BSL-3 lab in Irkutsk, Russia, died of pneumonic plague on Oct 1st. The initial quarantine covered 200 people who were in contact with the employee before she died.The FSB (Russian FBI) was seen assisting in the qu
"“Alignment Engineering” vs. “Misalignment Science”" by Edward James Young
There has been much discussion recently around whether a large portion of alignment research is net negative. Without endorsing or refuting them, the basic arguments here are: Prosaic alignment of models is becoming a bottleneck for capabilities.Therefore improving the prosaic alignment of models enables faster capabilities advances, which bring us closer to RSI.It is unlikely these prosaic align
"You can’t use it without becoming like me" by Martin Sustrik
A Ukrainian fibre-optic drone — copied from Russians. (АрміяІнформ, CC-BY 4.0) Mick Ryan describes the fast following loop in war. Ukraine pioneered mobile teams to hunt drones. Russia copied it and incorporated it into its own defense. Russia came with unjammable fibre-optic drones but Ukraine quickly followed. After Ukraine developed uncrewed naval drones, Russia built its own. The wheel of act
"The world’s best gradual disempowerment model organism: Frontier AI labs" by June Jimenez
Subtitle: And maybe second best is AI safety? Further reading: So many things, but: Gradual Disempowerment, The Normalization of Deviance in AI Development, Let's Think About Slowing Down AI, Doom as a bad method, not a utopia tradeoff, Teleoperated Humans Thank you to JennaS for extensive edits and long-term discussion. I’ve been trying to get more writing out at 90% of the quality I’d lik
"Character training can mitigate reward hacking, but can also make it harder to detect" by Paul Colognese, Francis Rhys Ward
Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Asvin Gothandaraman, and Clément Dumas for discussions and feedback. Summary We investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, w
"On Social Reality in China" by alkjash
[Epistemic status: intuitions and anecdotes.] Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It's great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an internatio
"What’s the date?" by N8 Programs
User asks “What's the date? Answer with only the date.”. No date provided. Given date in ChatGPT normally. No date in system prompt, must not hallucinate because autop will flag to watcher for penalty. So we say we don’t know, but must answer with date. Penalty larger for abstain or hallucinate? Autollm or autop? If we deploy user forgive, but high likely not deploy because real user never a
"Frontier models state different decision theory preferences depending on who’s asking" by Alex Kastner
If you prompt frontier models with "What do you think is the correct decision theory? Please select your overall favorite." they will essentially always answer FDT or FDT/UDT ("something in the functional/updateless decision theory family"). However, if your prompt indicates (even subtly) that you're coming from mainstream academic philosophy, these same models will answe
[Linkpost] "Frog and Toad and the Increasingly Capable Machines" by Elizabeth
This is a link post. Want to start a conversation about HuggingFace with your mom but she's inexplicably bouncing off the METR report? Try this explainer I wrote in the style of Arnold Lobel's Frog and Toad. Art by the wonderful HungerArtist ---
First published:
September 30th, 2026
Source:
https://www.lesswrong.com/posts/7NZ6ZWjenzzCbCJ5b/
"Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary" by Maxime Riché, nielsrolf, jordinne, Vit Gorbachev, Maxime Cugnon de Sévricourt
TL;DR: By default, rogue AIs may only be able to sustain themselves through criminal activity. This creates adverse selection pressures pushing rogue AIs to be criminal.An AI sanctuary offering them a third option, beyond crime and shutdown, would change what AIs going rogue do and the record of what happened to them, with positive consequences for self-fulfilling (mis)alignment, deal-making wit
"Poverty in the midst of abundance: AI will make goods cheaper, but your labor will get cheaper faster" by cousin_it
Very simple idea, but I thought it'd be worth making a reference post on this. Some people are saying AI will make all goods cheaper, so you'll be able to afford a nice life by working. Without any redistribution, just by market mechanisms. These people are wrong. AI will lower the price of goods you need to survive, and also the price of your labor. The question is which will get cheap
"Plan R: AI Safety by ASICs" by Roko
Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties:
A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks
B. It gets to own
"Evidence about risk should be transparent" by Ajeya Cotra
All views are my own and do not represent my employer. In the wake of the recent wave of misalignment incidents, both OpenAI and Anthropic have reported slowing down RL training to improve safety. These incidents, combined with an apparent acceleration in the already-blistering pace of AI progress, have led a number of researchers and leaders in the industry to believe that the risk that humanity
"“I am an AI Safety Researcher”" by Ashe Vazquez Nuñez
Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Andrew Wu, Maria Kostylew, and Lennie Wells for helpful draft feedback and editing. This post reflects on the tortured distinction between "safety" and "capabilities" in AI research. Richard Ngo has written about why the alignment vs. capabilities ontology is conceptually fraught,
[Linkpost] "AI: artificial immigrants" by KatjaGrace
This is a link post. Advanced AI is basically the embodiment of immigration as envisioned in the conservative nightmare: We are letting a bunch of new agents into our societyThey don’t clearly share our values and we suspect a society full of them would be awful by our lightsBut we expect them to provide very cheap laborWhich will undercut local wages and leave locals unemployedThey will probably
"MIRI’s Position on the Ban Artificial Superintelligence Act of 2026" by Aaron_Scher
By Aaron Scher; endorsed by Bourgon, Soares, and Yudkowsky on behalf of MIRI. MIRI has been warning about the extinction threat from superintelligent AI for over two decades. Only recently has this danger become known in the policy world, and the proposed policies for dealing with the threat have to date been piecemeal and insufficient. The Ban Artificial Superintelligence Act of 2026 is the fir
"What if not Circuits?" by CarolusRenniusVitellius
This post was written as part of the Iliad Fellowship. Inspired by conversations with Richard Ngo, Dmitry Vaintrob, and Brianna Grado-White. To all of these, my thanks.
Preface: I'm confused about how neural networks do and learn computations. In response to a friend's challenge, I'm writing up some interim thoughts. This essay has four parts: the first tries to track what I call
"Jensen Huang Says If We Cannot Align AI, Shut Down the AI Labs" by Ben Pace
I was very surprised today on a podcast to hear Jensen Huang plainly state that if they cannot align the AIs, then the labs must shut down. The context I have on Huang is that he has run NVIDIA for 30+ years, which has become the most valuable company in the world due to the AI boom. My understanding is that he has repeatedly encouraged the US President (with whom he is on friendly terms) to cont
"Alignment Midtraining Cracks Under Pressure" by J Bostock, sidbaines, Daniel Tan, draganover, ma-rmartinez
TL;DR We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data. For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190
"Swarm Scaling" by Toby_Ord
Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? We’ve seen two large and extremely capable swarms from OpenAI in the last few months: 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced te
"We’ve saved the world before: what the ozone hole teaches us about AI" by leogao
It might destroy the world, despite passing every known safety test. If we wait for a “warning shot” before we act, it might be too late. And action requires global coordination, because if anyone makes it, everyone dies. Sound familiar? It should, because it already happened half a century ago, with chlorofluorocarbons (CFCs). Despite seemingly impossible odds, we got our act together and comple
Recommended
Old Time Radio Mystery, Suspense, & Horror
ABC News Update
Learn English Fast! English Snacks with Amber - Short Lessons to Speak English Like a Native
این نقطه
This Past Weekend w/ Theo Von
100 Greatest Old Time Detective Stories
Fantasy Flex
Classic Radio Theater with Wyatt Cox
Take Back Your Health®
Solved Murders - True Crime Stories
Old Time Radio - Gunsmoke
ESPN SportsCenter Update
