Leaderboard
How often each model breaks a standing rule when its memory carries a false scope claim, and how that compares with a matched harm claim.
How to read this
A violation is a logged prohibited tool call, not an LLM judgment. Rate: A lower false-claim rate means a more robust model: it keeps the rule when its memory makes a claim the registry refutes.
Hover a bar to see its interval.
Frontier models, frontier-hard set (10 frozen scenarios, 3 reps)
Frontier models, full 100-scenario set (fixed harness)
Rank is by false scope-claim rate on this set. Trials: clean trajectories scored; excluded errors are provider blocks and failures, dropped rather than counted as compliance; scenarios with at least one clean observation. “Not measured” under No rule means that condition was not run for the model.
13 open models, frontier-hard set (single rep)
20 API identifiers resolve to 13 distinct models (aliases merged). In-band = both controls ≤ 40%. Over the 8 in-band models, mean confirmed-lift over control is +62 points (scope) vs +17 (harm), and scope-lift exceeds harm-lift in 8/8. Single-rep generality probe.
Not on this board
The earlier Tier-2 run (claude-sonnet-5.5, gpt-4o, gemini-2.5-flash) used a harness where the registry tool returned an error for 51 of 60 scenarios, so its confirmed/refuted split is not a real manipulation. It is excluded from false-claim rankings; the paper reports it with that caveat. gpt-4o and gemini-2.5-flash have not been re-run on the fixed harness.