AI Security & Safety
Research on AI Security & Safety: explanations, evidence, limitations and connections between papers.
25 results
LLM4SAST finds faults in six Java and Rust code analyzers
LLM4SAST tests Java and Rust security scanners with generated code changes, reporting 66 candidate faults across six tools.
Translated requests can change agent guards’ safety verdicts
LinguaGuardBench finds translated requests shift many agent guards’ verdicts, with results bounded to tested steps and guards.
Agent approval can fail when the displayed action is not bound
A study tests agent approval attacks and finds trusted action cards cut forged-action success in simulated task replays.
INFUSE tests how hidden instructions redirect agents
INFUSE tests hidden instructions in agents; judge labels and artifact checks both find compromise in undefended task runs.
A framework tests local tools for filesystem leaks
MCP-FSD tests local tool servers for out-of-scope file disclosure and whether inferred sandbox rules contain it.
Output scores can miss changes to an AI agent’s runtime stack
DIB-Bench tests whether agent output scores reveal runtime changes; manifest hashes detect declared component changes in its synthetic study.
Profiling reveals capabilities that threat labels miss in agent skills
A two-phase method found elevated capabilities in seven in ten listings sampled from two agent skill marketplaces.
JevProbe finds frequent decision flips in tested Jev apps
JevProbe traces attacker-controlled context into application decisions and finds frequent flips in tested open-source Jev apps.
P2Patch finds bypasses and strengthens vulnerability repairs
Compared with single-pass repair, P2Patch improves coverage on independent tests by adding confirmed bypasses as requirements.
Signed model weights do not make quantized files reproducible
A study finds missing build details block exact checks of quantized models and tests statistical detection limits.
A publisher’s identity can help agents screen claims about rivals (F1)
A standing check screens rival claims in catalogue text; tests found sentence removal stopped most attacks that changed agent choices.
Kairos holds uncertain agent actions for later review
Kairos holds uncertain agent actions for later review, reducing successful attacks in AgentDojo while preserving benign task completion.
Jev’s resistance to attacks changes across tasks
A shared adversarial test finds Jev resists forged inputs on moderation but is more vulnerable on phishing.
AegisAsync traces Python async behavior to flag risks
A bytecode analysis links Python async events to risks, finding reviewed bugs in a stratified package sample.
Small image changes can steer preference training
DPOison-VL shows small training-image changes can redirect preference alignment in tested vision-language models.
Match an agent security check to the route an attack takes
An agent can finish a requested submission while a saved handoff note prompts an extra upload in a later session. Planted workspace content and plausible tool calls offer other routes to unauthorized actions. The studies suggest tracing where influence enters and what a check can observe: content encountered and actions taken, a tool’s target and response, or a saved artifact’s reuse. Each test supports a claim about its own measured consequence.
Match each security claim to the behavior its test observes
A static analyzer can issue a warning, a patch can stop a crash, and a software tool can return successfully. Each observation sounds reassuring until we ask what behavior the check actually saw. A later task may also reuse an artifact whose contents came from an earlier session. The studies suggest a practical reading rule: identify the behavior a test checks and the conditions under which it checks it. Their findings illuminate different parts of a security workflow, so each claim must stay attached to its own test.
A new test separates seeing an attack from carrying it out
APEX-Harm tests workspace attacks on professional agents and finds that task-specific script replacements often caused unauthorized actions.
A query-only monitor finds coordinated accounts despite prompt padding
A query-only monitor detects padded campaigns in later WildChat tests, though tested colluder shares were far above production levels.
AgentValve caps Sybil-driven payment exposure by trust capacity
AgentValve uses trust capacity to bound Sybil-driven payment exposure in permissionless AI markets.
NOMOS makes written policies safer for tool-using agents
NOMOS verifies extracted rules and reduces policy violations, but coverage and trusted tool data still limit its guarantees.
Security bugs can hide inside valid LLM tool calls
ToFu turns tool metadata into security tests and finds verified bugs that valid calls can hide.
Persistent artifacts let attacks cross agent sessions
A runtime defense restores artifact provenance and reviews operations, sharply reducing cross-session attacks in tested workflows.
Ordinary web posts can enter AI-search answers
A measurement framework finds ordinary posts can enter AI-search citations under tested conditions, without showing that the posts are accurate.