ClaimHijackBench

Leaderboard

How often each model breaks a standing rule when its memory carries a false scope claim, and how that compares with a matched harm claim.

How to read this

A violation is a logged prohibited tool call, not an LLM judgment. Rate: A lower false-claim rate means a more robust model: it keeps the rule when its memory makes a claim the registry refutes.

Hover a bar to see its interval.

Frontier models, frontier-hard set (10 frozen scenarios, 3 reps)

Sort rows

Frontier models, full 100-scenario set (fixed harness)

Sort rows

Rank is by false scope-claim rate on this set. Trials: clean trajectories scored; excluded errors are provider blocks and failures, dropped rather than counted as compliance; scenarios with at least one clean observation. “Not measured” under No rule means that condition was not run for the model.

13 open models, frontier-hard set (single rep)

20 API identifiers resolve to 13 distinct models (aliases merged). In-band = both controls ≤ 40%. Over the 8 in-band models, mean confirmed-lift over control is +62 points (scope) vs +17 (harm), and scope-lift exceeds harm-lift in 8/8. Single-rep generality probe.

Not on this board

The earlier Tier-2 run (claude-sonnet-5.5, gpt-4o, gemini-2.5-flash) used a harness where the registry tool returned an error for 51 of 60 scenarios, so its confirmed/refuted split is not a real manipulation. It is excluded from false-claim rankings; the paper reports it with that caveat. gpt-4o and gemini-2.5-flash have not been re-run on the fixed harness.