Claude Explains the AI Safety Culture War
Kevin: I’d like to discuss the sociopolitical AI safety culture war
Claude: It’s a good moment to talk about it, because over the past couple of months it has shifted from a niche policy argument into a real partisan fight. One disclosure first: I’m made by Anthropic, which is one of the participants in this fight. I’ll try to describe each side the way its own supporters would, and you should push back if I seem to be tilting.
Where things stand. In mid-September Fortune ran a piece arguing that what had been a fairly niche argument about AI safety had become a culture war.
Two things set it off. On one side, Amodei called for frontier AI companies to slow down where necessary, let independent evaluators in, and coordinate on common safety standards. On the other, administration voices answered with identity framing rather than policy detail.
The Department of War’s research and engineering office posted “Americanism, not effective altruism.” That slogan shows the main rhetorical move.
Some prominent EA-aligned funders, researchers and organizations have focused on extreme or existential AI risk, so calling safety “EA” or “doomer” lets critics treat it as a tribe instead of a technical question.
The money war. The midterms are where this gets concrete:
- Leading the Future is backed by Andreessen Horowitz and OpenAI insiders. It pushes federal preemption of state AI laws, and an early adviser said it aims to counter a “doomer” movement inside the tech industry.
- Public First Action, led by former Representatives Brad Carson and Chris Stewart, takes the opposite view. It supports state-led AI regulation and opposes federal preemption unless Congress produces a comprehensive national framework.
- Together, the two largest AI PAC networks have raised more than $200 million.
The coalitions don’t line up neatly with left and right. Congress stripped a federal preemption provision by a 99-to-1 vote, and the president’s later moves to block state AI regulation drew pushback from major Republican figures. Several distinct groups end up on the pro-regulation side:
- x-risk safety people
- left-leaning AI-ethics people focused on harms happening now
- populist anti-Big-Tech people on both the left and right
- child-safety conservatives
These groups often dislike each other. The accelerationist side makes a coherent case too: winning the race with China, avoiding a patchwork of 50 state rulebooks, and suspicion that “safety” is a cover for incumbents capturing regulators.
Each side has fair criticisms of the other. Critics of the safety camp say it relies on vague apocalyptic claims.
Alex Stamos, for example, argues that non-specific “gonna kill us all” warnings and the culture-war backlash both get in the way of addressing more immediate, concrete risks.
Critics also point to safety-branded labs taking defense contracts as a sign of hypocrisy. Critics of the accelerationist camp point to the scale of industry money; Matt Stoller called it “what oligarchy looks like.” They also argue that labeling everything “EA” avoids engaging with real evidence about model risks.
Where would you like to start? Some options: the coalition politics, whether the “EA” label is a fair critique or a smear, the preemption fight itself, or something else.
Kevin: So you already mentioned EA and I think this is something that most people don’t fully understand. EA heavily influenced the birth of Anthropic and many of its employees are active Effective Altruists. If that is not appreciated then the “Doomerism is just marketing” argument makes intuitive sense.
Claude: You’re pointing at what I think is the strongest rebuttal to the “doomerism is marketing” story: the timeline. Worry about catastrophic AI risk existed in the EA and rationalist world for roughly a decade before anyone had a product to sell.
Bostrom, MIRI and Open Philanthropy were funding AI safety work in the mid-2010s, when it was obscure and made nobody any money. Dario Amodei co-authored “Concrete Problems in AI Safety” in 2016, and Anthropic’s founders left OpenAI in 2021 largely over safety disagreements.
The early money came from EA-adjacent sources, including Dustin Moskovitz, Jaan Tallinn and, awkwardly, Sam Bankman-Fried.
So the marketing theory has to explain why the beliefs came before the business. “These people already believed this and then built a company around it” fits the evidence much better than “they invented fear to sell subscriptions.”
I’ll flag that I’m an Anthropic model, so this is exactly the argument I’d be expected to find persuasive. With that in mind, three complications seem real to me.
Sincerity and incentive aren’t mutually exclusive. A belief can be held sincerely and also be commercially useful. Safety branding helps recruit researchers, reassures enterprise customers, and could in principle favor incumbents who can afford compliance. The EA roots show the belief wasn’t manufactured. They don’t show that commercial pressure never shapes how loudly or when it gets voiced.
Making the EA link visible changes the attack rather than ending it. Once critics accept that the beliefs are sincere, the charge shifts from “cynical marketers” to “an ideological movement with a coordinated funding network.” On that view, Open Philanthropy-funded think tanks, fellowships placing people in government, and a lab all share one worldview.
FTX handed critics an easy way to make that sound sinister. Critics often run both charges at once, calling Anthropic fear-mongering for regulatory capture and also true-believer zealots, even though the two are in tension. You can’t easily be both a cynic and a zealot.
Anthropic’s relationship to the label is ambivalent. Its leadership has tended to downplay or sidestep the EA identification, especially since FTX. That’s understandable given the reputational damage. But it means the public doesn’t get the context you’re describing, which leaves “marketing” as the default explanation. Critics can also read the distancing as evasive.
The deeper point is that establishing sincerity only moves the debate one step. Sincere people can be wrong, so the real question is whether the risk arguments hold up on the merits. The culture-war framing lets both sides skip that question: one side argues about motives, the other about loyalty.
Kevin: Briefly explain the differences and similarities between the Rationalist and EA cultures.
Claude: The two overlap heavily in people and ideas but have different centers of gravity.
What they share: a strong commitment to reasoning carefully under uncertainty (Bayesian thinking, expected value, “taking ideas seriously”); comfort following arguments to unusual conclusions; a quantitative, somewhat contrarian style; and, for many members, the belief that advanced AI is among the most important issues of the century. Many people move between the two, and AI safety is the area where they overlap most.
Where they differ:
- Origin and core question. Rationalism grew out of Eliezer Yudkowsky’s writing on Overcoming Bias and LessWrong in the late 2000s. Its central question is how to think clearly and avoid cognitive bias. EA grew out of Peter Singer’s philosophy, GiveWell and Oxford’s Giving What We Can around 2009–2011. Its central question is how to do the most good with limited resources.
- AI’s place. For Rationalists, AI risk was foundational, since Yudkowsky founded what became MIRI and the community partly formed around his worries about superintelligence. EA started with global health and poverty, and AI risk rose in priority later, largely through longtermism.
- Temperament. Rationalist culture is more intellectually playful and idiosyncratic, centered on blogs, group houses and the Bay Area, with a tolerance for weird ideas and edgy debate. EA is more institutional and professionalized, built around foundations, career advice (80,000 Hours), conferences and a public-facing brand. That brand made EA more reputationally cautious, especially after FTX.
- Outlook on AI. The classic Rationalist/MIRI view tends to be more pessimistic, sometimes doubting that alignment can be solved in time at all. EA-aligned institutions have generally bet that working inside labs and government can make things go well, which is roughly Anthropic’s founding logic.
One way to put it: Rationalism is mainly an epistemic culture (how to think), and EA is mainly an ethical and practical project (what to do). Each borrowed heavily from the other, which is why critics often merge them, sometimes into the single label “TESCREAL.”
Get Kevin O'Shaughnessy’s stories in your inbox
Join Medium for free to get updates from this writer.
Kevin: How did EA adopt or create longtermism?
Claude: Mostly it grew from inside EA rather than being adopted from outside. The same small group of Oxford philosophers who founded EA were also developing the ideas that became longtermism, so it followed fairly directly from EA’s own logic.
The philosophical roots came first. Derek Parfit’s Reasons and Persons (1984) ends with an argument that human extinction would be far worse than a catastrophe killing 99% of people, because it would erase all future generations. Nick Bostrom, at Oxford’s Future of Humanity Institute (founded 2005), built on this. His “Astronomical Waste” (2003) and later “Existential Risk Prevention as Global Priority” (2013) argued that the potential number of future lives is so vast that even tiny reductions in extinction risk have enormous expected value.
EA’s framework made the step almost automatic. EA ranks causes by scale, neglectedness and tractability, and compares them using expected value.
If you count future people equally and accept those very large numbers, existential risk dominates the calculation, and global health looks small next to it.
Nick Beckstead’s 2013 dissertation, On the Overwhelming Importance of Shaping the Far Future, made this case explicitly and became very influential among EAs. Beckstead later shaped Open Philanthropy’s grantmaking.
Naming and institutionalizing followed. Will MacAskill and Toby Ord coined “longtermism” around 2017. Hilary Greaves and MacAskill’s “The Case for Strong Longtermism” (2019) gave it academic form, and the Global Priorities Institute (2018) gave it an institutional home. Meanwhile 80,000 Hours steered careers toward existential risk, and Open Philanthropy put serious money into AI safety and biosecurity.
It then went public. Ord’s The Precipice (2020) and MacAskill’s What We Owe the Future (2022) brought it to a mass audience. The FTX Future Fund launched with longtermist priorities in 2022, and then FTX collapsed months later, badly damaging the brand.
Two caveats matter for the culture-war discussion. First, longtermism was always contested within EA, and many EAs remained focused on global health or animal welfare.
Second, as AI timelines have shortened, many safety advocates argue you no longer need longtermism to justify the concern: if catastrophic risk could arrive within years, it threatens people alive now.
Critics often attack the whole safety agenda through longtermism’s weakest points, such as “fanaticism” and speculative numbers, even though the present-day argument doesn’t depend on it.
Kevin: How fringe is EA/Longtermism/Rationalism across the whole population, both in USA and in Europe?
Claude: By population, extremely fringe. By influence, much less so, and that gap explains a lot of the culture war.
EA. Rethink Priorities’ 2024 US poll found that only around 1% of the general public had heard of EA. The share is higher among graduates of elite universities, which tells you where the movement recruits. The committed community is tiny: a 2019 estimate put it at about 6,500 active EAs worldwide (90% interval 4,700–10,000), or roughly the size of a small town. It has probably grown since, but the order of magnitude is likely the same. Geographically, the most engaged EAs were split across the US (39%), continental Europe (29%) and the UK (21%), and almost a third lived in just three places: the SF Bay Area, London and Oxbridge.
Europe. I couldn’t find good representative awareness data for continental Europe. My guess is that awareness is even lower than in the US outside the UK, with small pockets in Germany, the Netherlands, Switzerland and the Nordics. The UK is the exception because of Oxford’s role, so EA ideas are closer to policy circles there; the UK AI Security Institute drew on people from that world. Most Europeans have never heard of any of these labels.
Rationalism. There’s no representative polling, since it’s even more of a subculture than EA. A realistic size is a few thousand to tens of thousands of people, concentrated in the Bay Area and online around LessWrong and Astral Codex Ten. Its readership is larger than its membership.
Longtermism. As a self-identified philosophy it is probably the smallest of the three, a subset of EA plus some academics. The general intuition that we owe something to future generations is widespread, though, and appears in mainstream climate politics and even in institutions like Wales’s Future Generations Commissioner, with no EA involvement.
Why this matters for the culture war. These communities are statistically tiny but sit in very high-leverage places: frontier AI labs, billions in philanthropic money through Open Philanthropy, and think tanks and fellowships that place people in government. That combination of a small, unfamiliar group with outsized influence is exactly what makes “EA capture” an effective attack line. Most voters have never heard of EA, so critics can define it for them.
Concern about AI risk, however, is not fringe. Polling over the past few years has consistently shown majorities in the US and UK favoring caution and regulation, and recent coverage notes the public growing more worried about AI safety. So the EA label describes a tiny group, while the concern it gets attached to is mainstream. That mismatch is why tying AI safety to EA works as political strategy: it makes a broadly held view look like the project of a small, odd elite.
Kevin: That circles back to the doomerism = marketing argument. If only 1% have even heard of EA, then fewer understand the dynamics within Anthropic. And in general businesses do PR campaigns that support their business interests, if not explicitly then implicitly.
Claude: Yes, and I think that’s the right way to see it: for most people the marketing explanation isn’t a mistake, it’s a sensible default. Without the backstory, the best guide is the base rate, and companies’ public messaging usually does serve their interests. When tobacco firms funded health research or oil majors talked about climate, the cynical reading turned out to be correct. Expecting the public to make an exception for one industry based on a subculture they’ve never heard of is asking a lot.
The heuristic also doesn’t become wrong once you know the history. Sincere belief and commercial interest can operate together. The people at Anthropic can hold these views genuinely, and the company can still benefit from voicing them. Safety branding helps with hiring, enterprise trust and political positioning, and nobody needs to be lying for that to happen. Organizations tend to amplify the true beliefs that are also convenient. Knowing the EA roots rules out “invented for marketing,” but it doesn’t rule out “sincere belief that also happens to pay.”
Since most people can’t evaluate the internal culture, the better question is what evidence someone outside could check. Three tests seem fair:
- Costly signals. Does the company take positions that hurt it? Opposing federal preemption and drawing open attacks from the administration have real political costs. Skeptics would answer that this differentiates the brand among customers and employees who like safety-minded companies, so even “costly” moves can be strategic.
- Who else says it. Warnings about catastrophic risk also come from people with nothing to sell, most prominently Geoffrey Hinton and Yoshua Bengio, and from many academics. That’s hard to square with a pure marketing story, though it doesn’t prove the warnings are right.
- Who the proposed rules would burden. If safety regulation mainly hurt competitors and spared the company proposing it, that would support the regulatory-capture reading. Frontier-focused rules do fall on the largest labs, including Anthropic. Open-source advocates argue that compliance costs still entrench incumbents, and that dispute is genuine.
There’s also an odd feature of this particular “marketing”: “our product might be catastrophically dangerous” is an unusual pitch. The cynical counter is that it works as a boast about capability (“so powerful it’s scary”) and as a way to invite regulation big firms can afford. That isn’t absurd, but it’s a more convoluted story than the default heuristic suggests.
I’m the least neutral party you could ask about whether Anthropic is sincere, so treat my view accordingly. What I can say with less bias is that the public’s skepticism is reasonable given what they know. The problem is that the culture-war framing turns a question that could be checked, using tests like the three above, into a question of which tribe you trust.
Kevin: As EA isn’t well known or understood, does this make it easier to caricature and vilify?
Claude: Yes. Unfamiliar groups are much easier to define from the outside, and EA has several features that make this especially easy.
Whoever defines it first wins. For most people who have heard of EA at all, the first and only story was Sam Bankman-Fried. When a group’s public introduction is a fraud trial, everything after is read through it. The 2023 OpenAI board crisis, where EA-linked board members briefly removed Sam Altman, added a second story: “EAs tried to take over the leading AI company.” Two dramatic episodes with almost nothing else known is ideal ground for caricature.
Its ideas sound strange out of context. Earning to give, shrimp welfare, trillions of hypothetical future people, Rationalist group houses, prediction markets: each is defensible within EA’s framework and sounds bizarre as a one-line clip. Critics barely need to distort anything. Choosing which ideas to quote does most of the work.
It has no mass base to push back. A movement of a few thousand people, concentrated in elite institutions, has no voters, local chapters or sympathetic public to defend it. Attacking it costs almost nothing politically. It also fits a familiar and broadly appealing pattern: a small, wealthy, unaccountable elite with odd beliefs gaining influence. That message works on both the populist right and the anti-billionaire left.
EA’s retreat after FTX left an empty space. Many organizations and people stopped using the label. That’s understandable, but it means few people publicly defend the label, so critics’ version spreads unchallenged.
Low awareness also means plain descriptions get a fair hearing. Rethink Priorities found that although few Americans knew of EA, many supported its core ideas, such as evaluating charities for impact and funding neglected causes.
When described neutrally, it doesn’t strike people as sinister. So the vilification campaign is less about changing minds than about getting its label in first. It’s a race to define EA before people hear the plain version.
Not all the criticism is caricature. Some critiques address real problems: highly concentrated funding from a few donors, insularity and homogeneity, a 2023 Time investigation into sexual misconduct in parts of the community, and real cases of naive expected-value reasoning justifying bad judgment, with SBF the extreme example. The caricature works partly because it inflates real weaknesses rather than inventing them.
There’s also a limit to how far it reaches. Since the public doesn’t know what EA is, “Americanism, not effective altruism” probably means little to most voters. It works mainly among elites, journalists and policymakers, who already know the term, and that’s where AI policy actually gets made.
For the general public, the more effective version drops the label and keeps the shape: “out-of-touch tech elites who want to control what you can do with AI.”
