LLM4SAST finds faults in six Java and Rust code analyzers

If tests generate narrow or invalid code changes, a scanner’s missed warning or extra warning can be mistaken for a defect in the test. How can testing cover different kinds of code changes while checking that each changed program really fits the intended change?

LLM4SAST has a language model create three kinds of code changes, checks whether they fit the intended change, then tests scanners against the resulting warnings.

LLM4SAST records a scanner’s warnings, then uses a language model to preserve behavior, repair a vulnerability, or add one. Compilation, execution, and dynamic-analysis checks screen variants before the scanner’s new warnings are compared with expected patterns.

Across six Java and Rust tools, researchers reported 66 distinct faults. The detailed status table lists 12 fixed and 24 confirmed, but the abstract and contribution text report 32 confirmed or fixed, so the source gives conflicting figures.

Another 30 reports were pending. The results show the approach surfaced candidate faults in the tested tools.

The evaluation covered six selected Java and Rust tools and their security-related rules. Some invalid variants may remain, and clustering left some variants unanalyzed, so false reports or missed faults remain possible.