AI agents can build specialized software and verify its quality
ForgeSystem: An Architecture for Demand-Specific Artefact Production with AI Agents · arXiv:2611.12288
In plain words
General-purpose software has to serve many needs, which can limit how well it works for any one of them. Software tailored to a specific need can perform better, but building and checking it takes work.
ForgeSystem is a proposed process in which AI agents build specialised software, test it against requirements and keep records to help with later builds.
In three kinds of software, its implementations met task-specific quality checks and showed performance gains against the references used in those settings.
The research
General-purpose software must accommodate different needs, while tailored software can use a demand’s specific conditions but takes work to build and maintain.
ForgeSystem offers a way to organise that trade-off: the reported comparisons show what specialised agent-built software achieved when paired with checks for each task’s quality.
ForgeSystem organises AI agents to build software for specific demands, verify each candidate, and reuse construction records to guide later builds.
The system was implemented for training frameworks, decoding megakernels and individual computational kernels, each compared with references suited to that task.
In the tested settings, the specialised artefacts met their corresponding quality requirements and showed performance gains against those references.
A training comparison also found lower MFU when the explicit training-quality requirement was removed. This supports the role of verification requirements in guiding acceptable optimisation, alongside the performance results.
The evaluations cover three specific software settings, and the quality checks differ by artefact type. The results do not show whether the benefits outweigh construction, verification, integration and maintenance costs for other demands.
What the evidence shows
- A repeatable way to specialise
Forge is a production paradigm in which AI agents construct specialised software artefacts for explicit demands. ForgeSystem organises that work with a control layer that selects demands and starting points, a loop that builds and verifies candidates, and a loop that reuses archived artefacts and construction records.
The central claim is that specialisation can be made repeatable while keeping verification in the production process.
- Three kinds of software
The paper evaluates three implementations: ForgeTrain for training frameworks, ForgeMegakernel for decoding, and ForgeKernel for individual computational kernels. Each is compared with references appropriate to its artefact and setting, and each has task-specific quality requirements.
This scope matters because a performance comparison alone would not show that a candidate met the requirements for the work it performs.
- Performance against references
ForgeTrain’s MFU was 1.047–1.332 times its corresponding reference across seven model–hardware settings; ForgeMegakernel’s geometric-mean decoding speedup was 1.21 times over SGLang across thirteen configurations.
These comparisons show gains against each system’s own reference in the tested settings, not a direct comparison between the systems.
- Speed with quality checks
For individual kernels, ForgeKernel matched or exceeded pinned expert-library speed in 18 of 22 configurations, with three more within 1% of reference. Candidates had to pass the operator’s correctness tolerance checks before timing.
This links the speed comparison to an output-quality condition, though the result applies to the tested kernels and GPUs.
- Requirements guide optimisation
On MiniCPM4 0.5B, the process without an explicit training-quality requirement stayed around 37% MFU within 33 rounds, while the complete process reached 46.1%; the source does not say these results were measured at the same round count.
The authors report that the requirement let the agent move beyond exact bitwise agreement while preserving a final training-quality judgement, showing how it guided acceptable optimisation.
Deep
- Original paper authors
- Qingfeng He, Zhui Zhu, Yaojian Chen, Shangzhan Li, Leshan Li
- Submitted to arXiv
- License
- CC BY 4.0