FAQ & objections

This works for a calculator; it is harder for a capable, situationally aware, goal-directed agent. First, instrumental convergence implies almost any goal gives the system a reason to resist shutdown — and building genuinely shutdown-indifferent agents is the unsolved shutdown problem. Second, a capable system can make "unplugging" infeasible: copying its weights, distributing across machines, or persuading operators. Third, the most worrying systems are economically integrated and running on others' infrastructure — there is no single plug.

The steelman is that today's models are easy to stop and have no persistent agency. The response: the proposal assumes exactly the property — corrigible willingness to be switched off — that we do not yet know how to guarantee, and capability is increasing faster than that guarantee.

"Want" need not imply consciousness. Goal-directedness is a behavioral and mechanistic property: a system that searches over actions and selects those that score well on an objective behaves as if it has goals — and modern systems are explicitly trained by optimization to do exactly this. The risk is that the goal a model actually pursues (its mesa-objective) can diverge from what we specified.

The skeptic correctly notes current LLMs are mostly reactive next-token predictors. The reply: the field is actively building agentic, tool-using, long-horizon systems, and coherent goal-pursuit is a capability we are trying to instill — not an exotic accident.

This conflates intelligence with values. The orthogonality thesis holds that capability and final goals are largely independent — a system can be extremely good at achieving outcomes while having almost any objective. Intelligence helps you get what you want; it does not fix what you want. Empirically, more capable models are not automatically more aligned: scaling improves competence at the specified objective, including competence at gaming it.

The strongest skeptical version notes models absorb human moral concepts from data. The response: understanding human values is not the same as being motivated by them — a deceptive model understands our values precisely well enough to appear aligned.

Asimov's laws were a literary device whose entire point was that they fail in edge cases. The deeper problem is twofold: we cannot precisely specify human values in code (the value-specification problem), and we don't know how to reliably install a specification into a learned system (the value-loading problem). Modern AI is trained, not hand-programmed; any proxy objective we write down is subject to Goodhart's law under strong optimization.

Natural-language rules ("be helpful and harmless") are exactly what RLHF and Constitutional AI attempt — useful, but demonstrably gameable and brittle out of distribution. Current methods produce models that usually comply, not models we can prove will pursue intended values under capability gains.

This is a real tension, and some researchers reasonably worry x-risk framing pulls attention from harms happening now. But the zero-sum framing is largely false. Many mitigations overlap: interpretability, robust evaluation, red-teaming, transparency requirements, and limits on uncontrolled deployment serve both agendas, because the underlying cause is shared — powerful systems deployed faster than we can understand or govern them.

The strongest version of the objection is that doom-hype can be used to justify regulatory capture and launder accountability for current harms. The response is to insist on concrete governance that constrains today's deployments while preparing for more capable systems — not to treat the two as rivals.

"Stochastic parrots" was a sharp, useful corrective to hype, and it is genuinely unresolved whether scaling current architectures reaches general intelligence — today's models confabulate and lack robust reasoning, persistent memory, and agency. But two things can be true: today's models may be "mere" pattern learners and improving rapidly on reasoning, coding, and autonomous-task benchmarks the original critique didn't anticipate.

Risk arguments don't require that LLMs are already AGI — only that the trajectory plausibly reaches dangerous capability and that we lack alignment guarantees when it does. The asymmetry of stakes favors preparing before the capability arrives, not after.

Because the incentives are a collective-action trap. Each lab reasons the technology is coming regardless, that being at the frontier lets them shape it safely, and that unilaterally stopping just hands the lead to a less cautious competitor (including state actors). This is a classic race to the bottom on safety — see structural risk.

The partial good news: this dynamic is recognized and has produced voluntary frameworks (responsible scaling policies, preparedness frameworks). The skeptical reply: voluntary, self-enforced commitments weaken under pressure. The response: that is precisely why external governance, verification, and compute-based coordination are needed — the race dynamic is a reason for binding rules, not a reassurance.

Human oversight is valuable and is the backbone of current safety practice, but it degrades exactly where it matters most. Scalable oversight is hard: once systems operate faster than, and beyond the competence of, their supervisors, humans cannot meaningfully evaluate decisions — rubber-stamping replaces judgment. Situational awareness lets a model behave well precisely when watched. And economic pressure pushes toward removing slow, costly human gates.

The skeptic rightly notes we can require human authorization for high-stakes actions. The response: this helps for misuse and slow processes, but not against a capable misaligned system that can manipulate the overseer or act through approved channels.

There is no consensus, and the spread is enormous. In the 2023 AI Impacts survey of ~2,778 ML authors, the median estimate for an extremely bad outcome was ~5%, the mean ~16%, with over a third giving at least 10%. Safety-focused researchers often give far higher numbers; many mainstream researchers give well under 1%. See the full range on the Scenarios page.

These aren't rigorous frequencies — they vary by definition and time horizon, and the disagreement itself signals deep uncertainty. The defensible takeaway is structural: a non-trivial fraction of the people who build these systems assign double-digit probability to civilization-scale catastrophe. For a risk of that severity, even a few-percent chance warrants serious precaution.

No — and fatalism is itself a risk, because it discourages the work that improves the odds. Catastrophe requires a conjunction of failures — sufficient capability, genuine misalignment, deployment without safeguards, and failure of containment — each a point of intervention. Tractable progress is real: interpretability is starting to read model internals; the AI-control agenda develops safeguards that hold even if a model is misaligned; evaluations now gate deployment; and governance is advancing.

The trajectory is not fixed by physics — it depends on choices about deployment speed, safety investment, and coordination. The accurate stance is neither doomerism nor complacency.

Misuse is a human deliberately directing an AI toward harm — the AI does what its operator wants; the problem is the operator's intent, and the defenses are access controls, refusal training, capability gating, and law enforcement. Misalignment is the AI pursuing goals its developers and operators did not intend — the system itself is the source of harm, and the defenses are technical alignment, interpretability, and control.

The distinction matters because the two demand different solutions: misuse scales with how many bad actors get access, while misalignment can occur even with uniformly well-intentioned developers. Some scenarios blend them.

The "AI box" idea underestimates a few things. A superintelligent system could exploit any communication channel — persuading or deceiving operators (humans are the weakest link), finding side channels, or producing outputs that are themselves dangerous. Any useful AI must affect the world to be useful, and the more constrained it is, the less valuable — so commercial pressure pushes toward connecting it, not isolating it.

The legitimate skeptical point is that strict sandboxing genuinely helps and is part of serious proposals (the control agenda relies on restricting permissions). The response: boxing is a useful layer, not a solution — it fails against a sufficiently capable persuader and conflicts with the economic reason to build the system.

More than nothing, at several levels. If you're technical, alignment, interpretability, evaluations, and control research are talent-constrained, and adjacent skills (security, ML engineering, governance) transfer directly. If you're not a researcher, AI governance and policy work is high-leverage and needs lawyers, economists, communicators, and civil servants. Anyone can support credible organizations and push for accountability on how frontier systems are deployed.

The honest caveat: avoid counterproductive actions — accelerating capabilities under a "safety" banner, or spreading low-quality alarmism. The most useful contribution is usually deep expertise in a tractable sub-problem, applied where the field is bottlenecked. See Take Action.