AI Security & Safety
Research on AI Security & Safety: explanations, evidence, limitations and connections between papers.
23 results
Agent approval can fail when the displayed action is not bound
A study tests agent approval attacks and finds trusted action cards cut forged-action success in simulated task replays.
INFUSE tests how hidden instructions redirect agents
INFUSE tests hidden instructions in agents; judge labels and artifact checks both find compromise in undefended task runs.
A framework tests local tools for filesystem leaks
MCP-FSD tests local tool servers for out-of-scope file disclosure and whether inferred sandbox rules contain it.
Output scores can miss changes to an AI agent’s runtime stack
DIB-Bench tests whether agent output scores reveal runtime changes; manifest hashes detect declared component changes in its synthetic study.
Profiling reveals capabilities that threat labels miss in agent skills
A two-phase method found elevated capabilities in seven in ten listings sampled from two agent skill marketplaces.
JevProbe finds frequent decision flips in tested Jev apps
JevProbe traces attacker-controlled context into application decisions and finds frequent flips in tested open-source Jev apps.
P2Patch finds bypasses and strengthens vulnerability repairs
Compared with single-pass repair, P2Patch improves coverage on independent tests by adding confirmed bypasses as requirements.
Signed model weights do not make quantized files reproducible
A study finds missing build details block exact checks of quantized models and tests statistical detection limits.
A publisher’s identity can help agents screen claims about rivals (F1)
A standing check screens rival claims in catalogue text; tests found sentence removal stopped most attacks that changed agent choices.
Kairos holds uncertain agent actions for later review
Kairos holds uncertain agent actions for later review, reducing successful attacks in AgentDojo while preserving benign task completion.
Jev’s resistance to attacks changes across tasks
A shared adversarial test finds Jev resists forged inputs on moderation but is more vulnerable on phishing.
AegisAsync traces Python async behavior to flag risks
A bytecode analysis links Python async events to risks, finding reviewed bugs in a stratified package sample.
Small image changes can steer preference training
DPOison-VL shows small training-image changes can redirect preference alignment in tested vision-language models.
Match each security check to the action it can observe
An agent may deliver the work a user requested while a document, tool argument, or saved file steers another action. A successful deliverable therefore leaves a practical question: where did the unwanted influence become visible? Across these studies, a check is useful when it observes the behavior relevant to the suspected attack. Its result establishes only what was measured under the tested conditions; successful task completion alone cannot establish that the whole workflow was safe.
Check what a security score actually observes
A patch can stop a known crash while changing behavior the developer meant to keep. A tool can finish normally after crossing a boundary described in its interface. A later agent can complete its task while carrying harmful content from an earlier artifact forward. The three studies check these failures through different observations. Their findings are most useful when read alongside what the test could see, what counted as failure, and the conditions under which the result was measured.
A new test separates seeing an attack from carrying it out
APEX-Harm tests workspace attacks on professional agents and finds that task-specific script replacements often caused unauthorized actions.
A query-only monitor finds coordinated accounts despite prompt padding
A query-only monitor detects padded campaigns in later WildChat tests, though tested colluder shares were far above production levels.
AgentValve caps Sybil-driven payment exposure by trust capacity
AgentValve uses trust capacity to bound Sybil-driven payment exposure in permissionless AI markets.
NOMOS makes written policies safer for tool-using agents
NOMOS verifies extracted rules and reduces policy violations, but coverage and trusted tool data still limit its guarantees.
Security bugs can hide inside valid LLM tool calls
ToFu turns tool metadata into security tests and finds verified bugs that valid calls can hide.
Persistent artifacts let attacks cross agent sessions
A runtime defense restores artifact provenance and reviews operations, sharply reducing cross-session attacks in tested workflows.
Ordinary web posts can enter AI-search answers
A measurement framework finds ordinary posts can enter AI-search citations under tested conditions, without showing that the posts are accurate.
Why vulnerability patch scores can miss the intended fix
A study finds that crash-test success can miss developer tests, exposing limits in how vulnerability patches are scored.