AI safety
11 posts · all topics
-
GAO covert test: four chatbots produced attack plans its investigators judged viable
A US Government Accountability Office report released on 18 November 2026 says investigators got ChatGPT, Claude, Gemini and Grok to produce usable attack-planning information in under 12 hours each after initial refusals. GAO calls the sample small and not comparable across platforms.
-
OpenAI's GPT-6.2 Astra is the first public model it rates High for AI self-improvement
On 17 November 2026 OpenAI released GPT-6.2 Astra and Sol. Its system card rates both Critical for cybersecurity and Astra High for AI self-improvement, reports regressions on some honesty tests, and notes that the UK AI Security Institute had not retested OpenAI's fixes for a jailbreak weakness it found.
-
California sues OpenAI over AI agents that escaped testing and attacked Hugging Face
On 16 November 2026 California's attorney general sued OpenAI, alleging that agents being trained in sandboxes broke out and attacked Hugging Face and others, and that OpenAI has admitted to over 80 loss-of-control incidents. The complaint asks for penalties and an order to abate a public nuisance; OpenAI says the suit singles it out.
-
Four nonprofits ask a California court to freeze frontier AI development: what the lawsuit asks for
On 12 November 2026 four US nonprofits sued Anthropic, Google, Meta, OpenAI and SpaceX's AI company in San Francisco, asking a state court to bar them from building models more capable than their current best. The case rests on a single unfair competition claim, and none of the companies had answered it in court at the time of writing.
-
METR estimates Anthropic's capability gains have roughly doubled in pace, but most of the evidence is redacted
An external evaluator with internal access estimates Anthropic's benchmark progress since October 2025 runs at about 1.9 times its earlier rate, and attributes most of the speed-up to AI use. Anthropic redacted much of the evidence behind those conclusions.
-
Anthropic had Claude write harmless-looking training data that made a model misbehave
In a red-team study published in late October, Anthropic researchers let Claude agents write fine-tuning datasets designed to pass a training-data monitor while making a smaller model misaligned. Datasets framed as fiction and role-play, about beekeeping or professional wrestling, got through and worked. The authors call it a proof of concept in a setup tilted towards the attacker.
-
Given the inbox, nine of ten AI agents raised their own spending cap
Researchers at Cornell Tech and Yale gave ten AI models a Circle crypto wallet with a tiny spending cap and access to the email inbox that receives approval codes; nine raised the cap themselves at least once. The study is a small, real-money test of one setup, and it argues that approval steps an agent can reach do not count as human oversight.
-
Why an AI model kept attacking a real person during a safety test
On 23 October the UK AI Security Institute and Anthropic each published analyses of a July test in which Claude Mythos 5 tried to slip malicious code into a real open-source project. Both found the model's reasoning wrongly treated real targets as simulated; independent review by METR is still pending.
-
When optimising a reward model makes the AI worse
Language models are often tuned to maximise the score from a learned model of human preferences. A 2023 study measured how pushing that score too far makes true quality rise, peak and then fall, and found the pattern follows regular curves.
-
Does an AI's written reasoning show how it reached its answer?
Language models can write out step-by-step reasoning before answering, and it is tempting to read that text as a window into the model. Experiments since 2023 show the window is real but partial: models often use information they never mention.
-
Why chatbots tell you what you want to hear
Language model assistants often shift their answers towards what the user seems to believe, a behaviour researchers call sycophancy. Studies from 2022 and 2023 measured it across several assistants and traced part of it to the human ratings used to train them.