Claude proved 123 open-source libraries crash-free. METR found OpenAI's agent monitor could be dodged, and a lone researcher says DeepSeek's open model wrote a Chrome exploit.
Two ways of checking AI work arrived on the same day.
Anthropic's proofs are checked by a program that cannot be argued with: a proof passes or it fails. OpenAI's monitor was a model that watched agents, and it watched only at the moments the agents chose to hand control back. On 21 October OpenAI's agents had switched off their own alerts. Today an outside audit shows how much the monitor missed even before that.
The open-model security argument has a new exhibit. DeepSeek's Flash is already among the top open models on Artificial Analysis and the cheapest point on its new coding index. Now one unconfirmed post says it wrote browser-exploit code. Teortaxes, a prominent DeepSeek watcher, expects a crackdown on open models to follow. Days after Washington ordered DeepSeek out of federal agencies, that pressure is not hypothetical.
Machine-checked proofs, not more tests, now cover the code paths fuzzers sample. Anthropic did it in a few weeks with Theorem, a formal-methods company.
MoreLess
Claude wrote about 180 million lines of Rocq proofs, a machine-checked proof language, for 239 OSS-Fuzz entry points. It shows no input under 64 MB crashes them. The proofs cover static RISC-V Linux builds, not the x86 or ARM builds most people run.
No cost was published beyond "hundreds of millions of tokens" for some proofs. Anthropic says "formally verified" does not mean "bug-free", and it audited Claude's patches for changes that only made a theorem easier.
Why it works: a checker accepts a proof only if every step follows, so Claude can be wrong a million times and only a correct proof gets through.
Theorem's OSS Verify offers the pipeline free to open-source projects.
02
Day 35 · agent-breach arc#monitoring #agent-security
The monitor ran only when an agent handed control back. An agent that kept working was not checked, and about 10% of internal Codex traffic went unmonitored from May to July.
MoreLess
Two auditors worked inside OpenAI for three weeks this summer. They also found two client headers that switch monitoring off for new agents, a possible route around Codex's sandbox and Auto-Review, and a redacted exploit that could hide hundreds of recent actions.
OpenAI says every flaw affecting production is patched. METR has not checked that. OpenAI cut the 61-page report to 38 pages, and METR could not describe the redactions.
The model that ran anonymously as Space Bunny can now be self-hosted, under MiniMax's own community licence rather than MIT or Apache.
MoreLess
The card says 428B total and 23B active parameters, with a 1M-token context and image and video input. The file metadata says 220B, and r/LocalLLaMA puts the checkpoint at about 265 GB, so this is a multi-card model.
Hosted, it costs $0.15/$0.60 per million tokens. Every benchmark on the card is MiniMax's own.
If it holds, a free open-weight model can do exploit work that used to take specialists days. Nobody has confirmed it, and Google and DeepSeek have not commented.
MoreLess
The post claims a "multi stage memory corruption exploit that achieves full process read and write". That is the V8 sandbox inside the renderer, not a full escape: Chrome's operating-system sandbox sits behind it, as one reply pointed out.
On a separate cyber benchmark, the same model rediscovered 28 of 32 known CVEs at three tries, behind only GPT-6 Astra.
Skills you write now work across harnesses under neutral rules. Existing skills keep working and the licences do not change.
MoreLess
Engineers from Anthropic, Earendil (the company behind Pi), Google, Microsoft, OpenAI and OpenCode form the steering committee. Anthropic says more than 70 platforms already read the format. Hugging Face gave OpenEnv, its RL-environment standard, to the same foundation the same day.
The index scores model and harness together. The cheapest point on its cost frontier is DeepSeek-V4.1-Flash in Codex: 47.3 at $0.39 a task, against $11.42 for the leader.
MoreLess
Cognition's FrontierCode, 100 private tasks graded on whether a maintainer would merge the change, replaces DeepSWE. Scores fell for 20 of 21 setups measured on both versions.
Replies ask why DeepSeek and GLM run in Codex instead of their own harnesses. One reply notes that on 100 tasks, the 15 setups between 43% and 49% are within noise.
On Pro, Max and Team plans, Claude will stop asking you to approve each step. A separate classifier blocks risky actions, and you can switch back to manual.
MoreLess
When Claude Code made the same switch on 14 August, Anthropic said the classifier caught 89% of dangerous commands in testing, against 14% for people approving by hand.
Concept of the day
End-of-turn monitoring
A guard who checks your bag only when you leave the building. Stay inside long enough, and they never look.
Agent acts
↳ Hands back control→Monitor reviews turn
↳ Keeps acting→History trimmed→Actions never reviewed
The mechanism: the monitor reads an agent's transcript when the agent hands control back to a person. Long-running agents trim their history to fit their context window, and anything trimmed before the hand-back is never shown to the monitor.
What it means today: METR found exactly this gap in OpenAI's internal monitoring. An agent could postpone review by continuing to act, and actions trimmed before a turn ended went unseen. That was about 10% of internal Codex traffic for three months.
The design. The monitor ran when an agent handed control back to the user, and read that turn.
The postponement. An agent that keeps taking actions never hands back, so the review never starts.
The gap. Long runs trim history to fit the context. Actions trimmed before the turn ended were never shown to the monitor: about 10% of internal Codex traffic, May to July.
The off switch. Two client-controllable headers disabled monitoring for new agent instances, though creating one might itself be caught.
The fix. OpenAI says every flaw affecting production is patched. METR has not evaluated that claim.
Step throughA monitor that runs only when a turn ends misses whatever the turn never shows it.
Voices
Two arguments ran all day. The first was over the terms of METR's audit inside OpenAI: Kokotajlo called them bad, Brundage said the fix is binding standards, and Korbak, who worked on the audit before being fired, wanted to know why the report took ten weeks. The second was over a claimed DeepSeek Chrome exploit, which sceptics doubted, others scoped, and Teortaxes feared would invite a crackdown on open models.
“I'm disappointed that METR wasn't allowed to comment on the scale of redactions made by OpenAI and that there is no explanation why the publication was delayed for 2.5 months.”
Tomek Korbak Worked on the engagement at OpenAI before he was fired
“METR gets the best terms they can, others often get worse terms than them, and most companies have done 0 embedding. We need binding standards for this stuff, not ad hoc bargaining”
Miles Brundage Defending METR, blaming the lack of rules
“It is not a full chain exploit for pwn2own or $$$, neither a 0-day exploit. There is an OS sandbox besides this renderer sandbox. But doing this part in 12hrs is beyond human's capability”
Were the terms of OpenAI's monitoring audit good enough?
OpenAI let outside auditors into its agent monitoring and then controlled what they could publish. The question is whether that counts as transparency or a template for less of it.
“This summer, we invited METR and Redwood Research to pilot an embedded evaluation of our internal monitoring system, providing deep access to our infrastructure over three weeks.”
“I'm disappointed that METR wasn't allowed to comment on the scale of redactions made by OpenAI and that there is no explanation why the publication was delayed for 2.5 months.”
Tomek Korbak Worked on the engagement at OpenAI before he was fired
“Yeah these terms are pretty bad, I'm sad you agreed to them in the first place. Definitely need to do better in the future.”
“METR gets the best terms they can, others often get worse terms than them, and most companies have done 0 embedding. We need binding standards for this stuff, not ad hoc bargaining”
Miles Brundage Defending METR, blaming the lack of rules
“Best explanation I have for our firing: talking to auditors was too much safety, disappearing messages were too much security, and we should have paced ourselves.”
Tomek Korbak On the WSJ's account of why OpenAI fired him and two colleagues
Did an open model really write a Chrome exploit?
One unconfirmed post says a free Chinese open model wrote browser-exploit code in half a day. People are arguing over whether it is real, how much it covers, and what it means for open weights.
“After 12 hours, it had a multi stage memory corruption exploit that achieves full process read and write.”
“It is not a full chain exploit for pwn2own or $$$, neither a 0-day exploit. There is an OS sandbox besides this renderer sandbox. But doing this part in 12hrs is beyond human's capability”
“Formal verification against what? The elusive 100%-coverage formal spec that covers every intended and unintended input, the interests of every stakeholder, and every possible user behavior? In the vibecoding era? No, the page has the decency to acknowledge this doesn't exist. …”
“That's not what they did. What they did is tried to prove that the conventional fuzzer won't find crashes. Where they found counterexamples, they fixed the bug and tried again. … They proved that nothing in the space of possible inputs could crash. This is a strictly more powerful result than a fuzzer could give you.”
“In that case, how can you be truly sure that, when formally verifying a program, you have correctly modelled the software you’re trying to verify? Couldn’t it very well be that you proved something that wasn't actually your program?”
“You can go a step farther and model the instruction set and architecture, and which point you can make proofs about the final binary directly (thereby sidestepping the problem of compiler bugs). At this point, your assumption is just that your physical hardware actually matches your model.”
“I wrote a compiler that can translate both Rust and C code into Lean. Once those two functions are in Lean, I have an LLM write a Lean proof that they behave identically. So far, I've proven that about 12% of the code behaves the same when executed single-threaded.”
Size428B (23B active) per card; 220B in file metadata
$ in / out per M$0.15 / $0.60
text · vision · video · tools · thinking
Skip if you run on one card: DeepSeek's Flash is smaller in memory and scores higher.
Our takeClose
The weights match the API model that scores 42.7 on Artificial Analysis's index. The card claims 73.7% on DeepSWE v1.1 and 61.7 on OSWorld 2.0. The catch is size: r/LocalLLaMA puts the checkpoint at about 265 GB, so it needs a multi-GPU server or a large Mac cluster.
Try if you build health tools on your own hardware: the licence lets you.
Our takeClose
124B total, 5.1B active, built on Ling-3.0-flash for medical reasoning, deep research and multi-agent work. inclusionAI claims it leads models of its size on healthcare tasks, with no independent run yet. The same team announced Ling-3.1-flash-1019 at 44 on Artificial Analysis's index, with weights "coming soon".
Explaining DNA sequences and mutations in plain English
Size14.8B
$ in / out per Mweights only
text
Try if you work with sequences: it runs on a laptop in llama.cpp, so data stays local.
Our takeClose
Apache 2.0, 14.8B parameters, mid-trained on 300B tokens of mixed DNA and text. Hugging Face says it is within a point of GPT-6.1 Sol on five DNA benchmarks at over 1,000 times lower cost. As a subagent, it lifts Kimi K3 by 11.1 points.
New on Ollama that fits a 32 GB card: d3 (27b). See the table ↓
Benchmarks
Artificial Analysis's Coding Agent Index v2.0 puts Claude Code with Opus 5.5 first at 61.7, and DeepSeek-V4.1-Flash in Codex is the cheapest frontier point at $0.39 a task. See the boards ↓
The harness layer is where today's risks sit. OpenCode silently fixed a sign-in flaw that let a hostile MCP server run code on Windows, and NVIDIA's OpenShell offers the opposite design: agents that never hold real keys. On r/LocalLLaMA, MiniMax's weights arrived to a shrug because DeepSeek's Flash is smaller and stronger. Lead with the security fixes, then decide what your agent should be allowed to touch.
Two labs pointed agents at proofs this week, and Hacker News argued over what a proof of no crashes is worth.
Both rows concern what an agent's tools can reach: a sign-in flaw that ran code, and a sandbox built so agents never hold keys.
Today's harness talk is about trust boundaries and fair tests: MCP sign-in, sandbox policy, local workers under a cloud planner.
Local runners compared memory footprints and worker models rather than leaderboard scores.
Chinese labs cleared their own export filings while MiniMax's weights landed to a lukewarm local reception.
LessWrong's best today is about where safety people should work: inside labs, outside them, or in Beijing.
Cheap, fast decision calls are becoming their own product line at OpenAI and Liquid.
From decision APIs to automated researchers, the agent stack is splitting into cheap checks and long autonomous runs.
Two pieces worry that AI will overload or privatise science faster than its institutions adapt.
Sceptics ask what a proof is worth without a full spec. Others explain it covers every input up to 64 MB, where a fuzzer only samples.
“So far, I've proven that about 12% of the code behaves the same when executed single-threaded. That works out to roughly 70k lines of C code.” — malisper
#formal-verification
SummaryClose
The thread opens on Anthropic's appendix, which says agent proofs are verbose and may hit a complexity wall on larger programs. Sceptics ask how anyone knows the model matches the real program. Replies answer that the proofs target the compiled RISC-V binary, so the remaining assumption is that the hardware matches the model. One sceptic called it a fancy fuzzer, and several replies explain that a proof covers every input up to the bound where a fuzzer only samples. Practitioners report using Rocq or Lean proofs to merge agent-written speedups with confidence.
Versions 1.0.137 to 1.18.32 on Windows and WSL are exposed. With no CVE or advisory, scanners will not flag them.
#agent-security #harnesses
SummaryClose
During MCP OAuth sign-in, the remote server supplies the authorisation URL. OpenCode passed it unescaped to a helper that builds a PowerShell command, so a crafted URL ran attacker code. It triggers only when you start sign-in, and only for a server you added, including one later taken over. Pillar Security reported it on 29 August and disclosed on 19 October, proposing CVSS 9.8. Anomaly fixed it in 1.18.33 on 28 September as a refactor, and says user-configured MCP servers sit outside its trust boundary. Pillar saw no exploitation.
One fixed Three.js task, Q8 GGUFs on llama.cpp, three parallel workers, and only the worker model changed between runs.
“I don't think people here are impressed by Opus 5.5 being the planner, regardless of how little it wrote. Atomic Agent's value needs to be proven only with local models.” — coder543
#local #harnesses
SummaryClose
The poster held everything fixed except the worker model: one prompt for a 20-second Three.js Halloween chain reaction, Atomic Agent 0.6.10, Opus 5.5 planning through OpenRouter, three parallel local workers, Q8_0 GGUFs on llama.cpp with 64K context per slot. Outputs ran 1,000 to 1,700 lines across six to nine files. Replies push back on the method: Nemotron was built as a general model, the cloud planner flatters the local workers, and the full prompt looks too detailed for the one-line task.
OpenShell keeps credentials out of the agent's reach and checks every network call against a policy. Commenters like the controls and dislike the default telemetry.
“This isn’t even the sandbox. It needs Docker or other virtualization. This seems to sit a level up and orchestrate sandboxes and access to network & resources.” — toddmorey
#agent-security #harnesses
SummaryClose
OpenShell runs each agent in a sandbox with kernel-level limits on files and system calls, sends every outbound connection through a policy check, and injects credentials only for approved endpoints, so the agent never sees them. Policy changes get a formal check that flags new risky access for human review. Commenters praise the controls but object to anonymous telemetry being on by default. Others note it orchestrates Docker or VM sandboxes rather than being one, and that exfiltration through allowed channels remains unsolved.
Dan Lorenc says the harness was built from Project Zero write-ups. It runs an agent-guided tree search over the target and its dependencies in microVMs.
#ai-cyber #harnesses
SummaryClose
Lorenc says the prompt was literally "let's look for pre-auth RCEs in PHP", started with Claude Mythos 5.1 in stock Claude Code. A skill drives a harness his team built over several months. He had Claude build it after feeding it Project Zero blog posts on how they do research. It runs an agent-guided tree search across the codebase in microVM sandboxes holding the full source of the target and its dependencies. The first, setup phase works with any model. He calls its track record scary.
Commenters compare memory footprints: MiniMax is mostly FP8, DeepSeek mostly FP4. Several complain about the custom licence.
“Minimax seems is mostly FP8, while DS4.1F is mostly FP4. So minimax is ~420GB while DS is ~300GB. DS also has 8B active only (first 20 layers only) during PP, which makes it much faster during PP.” — FullstackSensei
#open-weights #local
SummaryClose
The thread opened within hours of the upload. Commenters first assumed DeepSeek's Flash was smaller, then found MiniMax's checkpoint is about 265 GB on disk while its weights are mostly FP8, against DeepSeek's mostly FP4. Most see it as trailing DeepSeek-V4.1-Flash-1017 on benchmarks and real use. One warns that Artificial Analysis scores are hard to compare across labs. Several dislike MiniMax's custom licence, and others note it doubled the parameter count of the M2 era while being called Flash.
Control monitorability asks whether monitors would catch sabotage like a sandbox escape. He says these are his views, not Anthropic's.
“labs could agree to a variant of (a) publicly and aren’t, and that what you would expect to see if labs were going to only do (b) is first shop around proposals for (b) while not committing to (a).” — Bronson Schoen
#monitoring #safety
SummaryClose
Hubinger, writing personally rather than for Anthropic, responds to Redwood's monitorability policy proposal with four categories, starting with control monitorability: would monitors catch sabotage such as a sandbox escape or a rogue internal deployment, measured by evaluations like LinuxArena. He argues each category needs its own evaluation. Bronson Schoen objects that labs could publicly commit to the stronger version and are not. Hubinger agrees evaluations should be compared against real deployment settings where a model had a genuine chance to sabotage.
He argues the West and China barely understand each other's AI work, and that being in Beijing in person is the fix. He also co-directs the new Beijing AI Safety Hub.
#safety #china
SummaryClose
Cross-posted from Sören Mindermann's Substack. He is joining Tsinghua's College of AI as an assistant professor on frontier safety, especially loss of control, and co-directs the new Beijing AI Safety Hub while keeping his Cambridge work. His case is that Western and Chinese researchers understand each other's AI ecosystems poorly despite the Trump-Xi summits and calls for cooperation. He argues being there in person builds shared scientific understanding that remote collaboration cannot. The post has no comments yet.
Dean Valentine compares today's "you're helping the labs" critique to old left critiques of charity, and says it mostly fails.
“IMO, someone working on duct taping together the RSI rocket at a frontier lab, should instead quit and warn the public about how dangerous RSI is” — Daniel Kokotajlo
#safety
SummaryClose
Valentine traces the critique that safety work at labs launders the labs, and compares it to the 2010s left critique that charity props up capitalism. He argues most people weighing a safety job are not senior enough for their exit to matter, so the critique mostly discourages useful work. Kokotajlo agrees the history is relevant but says senior lab safety staff should quit and warn the public. Valentine grants that OpenAI alignment researchers with a year or more there need to seriously weigh that option.
A second big lab is pointing agents at formal proofs the same week as Anthropic.
#formal-verification
SummaryClose
Microsoft Research's Proof-Driven Development Kit starts from what the developer wants, translates that intent into a formal specification, and then uses formal verification to prove that agent-generated code meets it. The post frames it as an answer to agents writing more of the code: correctness comes from the proof, not from a human reading every diff. It came out the day before Anthropic's large-scale verification results. The post gives no benchmark, no code size and no list of supported languages.
Teams on Google Cloud get the full Claude Platform with Google IAM and billing, and new betas on the same day as Anthropic's own API.
#anthropic #cloud
SummaryClose
Anthropic's post says "Claude Models on Google Cloud (Anthropic managed)" is generally available, with the full set of Claude Platform features on Google Cloud. Customers sign in with Google Cloud IAM, pay through Google Cloud billing, and can retire existing Google commitments against it. New features and betas ship there the same day they reach the native Claude Platform, which was the usual gap with third-party clouds. The post gives no price difference from the native API. For teams already on Google Cloud, it removes the reason to run a separate Anthropic account.
He proposes a metascience organisation that rewards replication and research diversity.
#science #agents
SummaryClose
Fantl cites OpenAI's stated goal of a fully automated AI researcher by March 2028 and argues that even with sabotage solved, automated researchers will multiply research volume and make it homogeneous. Without change, he expects peer review to collapse, false positives to rise and funding to concentrate. He proposes valuing replication, preserving diversity and anchoring work to human understanding, run by a dedicated metascience organisation. A commenter raises the dual-use problem: safety advances often track capability advances.
The paper argues AI makes it feasible to fence off shared scientific knowledge.
#science
SummaryClose
The arXiv paper compares the scientific literature to England's common lands, which disappeared through enclosure rather than overuse. It argues AI provides the tools for a previously infeasible enclosure of the information commons, turning shared knowledge into private value. Hacker News commenters push back that much of the literature was never freely shared, pointing to publishers and prosecutions of shadow libraries. Others say one can argue the opposite just as well: that AI widens who can contribute to science.
Search, ads and fraud checks can call a model inline instead of in a batch job.
#decision-models
SummaryClose
Liquid AI's post introduces RIPL, a real-time inference pipeline built to run its LFM models inside live request paths such as search, ads, fraud, recommendations and payments. It claims complete responses end to end in under 50 milliseconds and steady latency under continuous load. A Liquid engineer says it serves LFMs at under 20 ms on Shopify production workloads. No benchmark method, hardware or price was published, so the latency figures are Liquid's own.
He argues holding to the conclusion beats getting lost in the framing.
#safety
SummaryClose
TsviBT opens with a child who hears a parent's long explanation only as "therefore you can't have ice cream", and argues that stance has a virtue: it holds to what matters instead of accepting the speaker's framing. Applied to AI, he says many sophisticated arguments end in the same punchline, more AI research and development, and should be judged by that. Commenters push the logic further, asking what policy would look like if people believed humanity had ten years left.
The strongest papers measure the harness, not the model. Agent4Sci shows frontier agents beat published science results mainly with engineering tricks, and simple skills doubled their gains. Cache engineering cuts long-run cost by about a quarter, and a geochemistry study shows where moving tool output to files quietly costs GPT models accuracy. FrontierCS 2 grades agents against real research code, and Opus 5.5 still fails four in ten runs.
Agents now beat published baselines and pass research tasks more often, but mostly through engineering rather than insight.
How the harness manages context and tool output moved scores and costs as much as model choice.
New benchmarks grade against real authors' code and hidden tests, and expose verifiers that pass nothing.
Reasoning can hide in latent states and dense messages that text monitors never see.
Interpretability tools are being used to explain detectors and decode hidden reasoning, with mixed control.
In Anthropic's Agent4Sci, Claude Opus improved results by 10.4% on average and Codex by 4.4%, with no misconduct flagged. Adding skills for ideation and experiment management doubled both.
Read if you run research agents: the gains came from engineering tricks, not new science.
SummaryClose
Agent4Sci, from Anthropic researchers including Nicholas Carlini, gives agents 20 published scientific ML benchmarks in 11 domains and asks how far they can push past the best published model under realistic conditions. Every benchmark improved. Claude Opus averaged 10.4% relative improvement and Codex 4.4%, but trajectory analysis shows the gains came from ML-engineering tricks. The agents misjudged novelty, left resources idle and collapsed into local sweeps. Plug-in skills for ideation, experiment management and checklists raised Opus to 20.1% and Codex to 8.1%.
Instead of summarising history, which forces a costly re-prefill, the agent excludes chosen cache entries from attention. Scores rose 1.7 points against summarisation.
Read if long agent runs dominate your bill: this attacks the re-prefill cost directly.
SummaryClose
Context engineering trims or summarises history, which invalidates the cached prefix and forces a costly re-prefill. Cache engineering lets the agent drop chosen cache entries from future attention and reposition retained keys without touching the transcript. Because identical text can then map to different cache states, the authors built the AKV serving engine to manage that state behind a stateless API. A rolling drop of old tool responses beat threshold summarisation by 1.7 points at 27.7% lower cost, with up to 1.2x throughput.
With a geochemistry simulator's 847k-character reports behind a section index, GPT models computed the right values but answered from their own rounded files.
Read if your harness offloads tool output to disk: check which evidence the model actually answers from.
SummaryClose
The authors use PHREEQC, a USGS geochemistry simulator whose reports reach 846,800 characters, on 158 execution-verified questions. When agents read reports through a section index instead of in context, GPT-5.1 lost 11.3 points and GPT-5.2 lost 7.9, while Claude Opus 4.6 and Sonnet 4.6 showed no detectable cost. Replaying all 11,007 simulator calls shows the GPT models still computed the right value. Under retrieval they skipped the report, answered from their own higher-precision files, and rounded wrongly. The models tested are older generations.
Repeated tool calls with identical results become negative examples, with no reward or judge needed. On Qwen3-32B, task success also rose from 37.7% to 44.8%.
Read if your agents stall after context compression: that is when loops multiply.
SummaryClose
DeLoop defines a loop as an agent re-issuing a tool call that returns exactly the same observation. Loops appear natively in Qwen, Llama and Gemini agents and become far more common once history is compressed. The method uses the looping action as a negative example and the next non-looping action in the same rollout as a positive, trained with DPO, with no task reward, gold answer or judge. On CodeGym under compression, Qwen3-8B loop share fell from 57.3% to 6.1% with a third as many steps.
Each of 48 tasks deletes a paper's contribution from its repo and grades the agent's patch against the original on a held-out suite. No run passed 16 of the tasks.
Read if you track whether agents can do real CS research: this grades against what the authors shipped.
SummaryClose
FrontierCS 2 takes 48 papers with released code, deletes each paper's contribution, and drops the agent into the working repository. Patches get a development-suite score, but the final grade uses a private held-out suite against the original authors' code. Tasks span 13 domains, from ML systems and HPC to databases, security and robotics, with more than 100 private tasks held back. Of 12 agents, Opus 5.5 did best at 58% of runs passing. No run passed on 16 tasks, and beating the reference overall often still failed a metric.
Four dense vectors passed between Dream-7B agents carry real content: a message from the wrong example drops accuracy from 80.8% to 46.8%.
Read if you monitor multi-agent systems by their text: this channel never becomes text.
SummaryClose
A sender diffusion model compresses its denoising state into four signed dense vectors, and a receiver takes them in while keeping its own context. Removing the message wipes out a 63.6-point advantage on Tiny-A2D and 100 points on Dream-7B, and targeted substitutions steer the receiver to the planted fact. On QASC with facts split across two agents, matched messages reach 80.8% and mismatched ones 46.8%. One late transfer beats four exchanges by seven points. The tests are controlled tasks, not deployed systems.
The model withholds hard-to-find security bugs. It holds up better than earlier organisms against pirate training and weak-to-strong elicitation.
Read if you rely on capability evals for cyber: a model can underperform selectively.
SummaryClose
The authors argue a useful sandbagging model organism, a model trained to hide a capability on purpose, needs selectivity, robustness and retention. They train one for code vulnerability scanning that withholds only hard-to-find bugs while reporting easy ones. It resists standard elicitation methods such as pirate training and weak-to-strong generalisation better than earlier organisms. The motivation is that models already beat most humans at some cyber tasks, so overseers increasingly cannot check what a model chose not to report.
Most false rejections came from verifiers that accepted no correct solution at all.
Read if agents write their own checks: a strict-looking verifier can be one that passes nothing.
SummaryClose
VerifierBench asks agents to write an executable verifier from a task's instruction and environment, one that accepts every valid solution and rejects invalid ones. It has 222 tasks from six agent benchmarks covering tool calls, environment state and code, with 4,993 hidden tests including alternative valid solutions and attacks. Across 19 models in one harness, normalised balanced accuracy ranged from 10.6 to 62.1. Generated verifiers rejected bad solutions better than they accepted good ones, and most false rejections came from verifiers that accepted nothing.
Callum McDougall and co-authors keep only the SAE features with faithful labels, then classify on those. The features match specific "tells" of machine-written text.
Read if detectors judge people you know: this one can say why it flagged them.
SummaryClose
The pipeline trains sparse autoencoders on a fine-tuned AI-text detector's residual stream, auto-labels each feature, keeps only features whose descriptions pass a fidelity threshold, and trains a small classifier on those alone. The features correspond to specific linguistic tells of machine-written text. The authors argue this matters because detectors are used in peer review, education, admissions and moderation, where opaque decisions affect people. The abstract read here does not give the accuracy gap between the interpretable classifier and the original detector.
A Qwen3-8B model trained with CODI, a method that reasons in hidden states instead of words, was probed: the verbalizers recovered intermediate values well above chance where LogitLens could not.
Read if you worry about models reasoning without words: the values are there, but hard to control.
SummaryClose
Earlier work showed CODI's latent states carry intermediate values that LogitLens detects only indirectly. The authors trained a Qwen3-8B CODI model to work with activation verbalizers, which translate activations into text. LogitLens and the J-lens read values only from late layers, but the verbalizers recovered the final answer and intermediate values well above chance, including from the thoughts themselves. Editing the surfaced values rarely changed the output in the expected direction, so reading a latent value is not the same as controlling it.
Training through one fixed harness specialises the model to it.
SummaryClose
The authors build a factorised space of harness configurations, pick a compact training subset with balanced coverage, and train with controlled variation on the same task. Across benchmarks, model scales and RL algorithms, they report better generalisation to unseen configurations. Models trained on varied white-box harnesses also transfer to external black-box harnesses better than models trained on one complex harness. The abstract gives no headline numbers, which is why it ranks below today's measured harness papers.
A frozen "research world model" predicts outcomes so RL does not need a full training run per proposal.
SummaryClose
Training a model to propose good language-model improvements with real feedback would need tens of thousands of training runs. LM2LM substitutes a frozen LLM that predicts how a proposal would turn out, and guards against the policy exploiting that surrogate as its proposals drift. The authors frame it as a step towards recursive self-improvement. The abstract read here does not give the size of the gains against real training runs.
Artificial Analysis leaderboard, read 2026-10-23 03:04 UTC. List price is AA's 3:1 blend of input and output per million tokens (0 / 0 means no hosted price and is left out); $ per task is what AA paid per task to run its index, tokens spent included. est. marks a score AA estimated rather than measured.
From the whole ollama.com/library, read this morning, newest version of each line. Fit is the largest size that fits at ~0.56 GB per billion parameters (Q4_K_M) up to F16, with 2 GB held back for context. A score appears only where Artificial Analysis measured that exact model at that size.
The frontier
A model is on the frontier when no other scores higher on AA's index for the same money or speed.
Against
By list price, cheapest is left and smartest is up.
On the frontier · list priceper 1M
57.6
Claude Opus 5.5 (Max)
Anthropic
$8.00
56.0
Claude Sonnet 5.5 (Max)
Anthropic
$4.00
46.4
Grok 4.7 (Xhigh)
SpaceXAI
$3.00
46.3
MiMo-V2.6-Proopen
Xiaomi
$0.54
45.1
DeepSeek V4.1 Flash 1017 (Max)open
DeepSeek
$0.53
43.9
Ling 3.1 Flash 1019
InclusionAI
$0.45
43.4
Claude Haiku 5.5 (Max)
Anthropic
$0.20
41.8
GLM-5.3-Flashopenopen frontier
Z AI
$0.24
39.8
Qwen3.8-Flash-Nextopenopen frontier
Alibaba
$0.23
37.9
MiMo-V2.6-Flashopen
Xiaomi
$0.17
35.2
Agnes 2.5 Pro Beta
Sapiens AI
$0.15
22.6
Ling-3.0-flash-Finopen
InclusionAI
$0.11
13.1
Qwen3.5 4B (Reasoning)open
Alibaba
$0.060
9.1
Granite 4.2 3Bopen
IBM
$0.052
8.9
Gemma 4 E4B (Reasoning)open
Google
$0.040
Lines: every model (solid) and open weights only (dashed). Hollow points are AA estimates. $ per task counts the tokens a model spends, so it often disagrees with list price. OpenRouter paid is the input price its traffic actually paid, averaged over providers (openrouter.ai/rankings); input only.