PrairieFire handles frequent changes in large vector indexes
In plain words
PrairieFire is a system for searching large collections of vectors while the collection keeps changing. It stores new items in buffers, groups changes into batches, and rebuilds only affected parts of its index instead of repeatedly moving individual vectors.
In tests with 100 million vectors from SIFT and SPACEV, PrairieFire handled updates and searches at higher rates than three compared systems, with comparable or better search accuracy. Those results apply to the tested workloads.
The research
A recommendation index can become stale as its vectors change, but refreshing a 100-million-vector index may require up to 30,000 updates per second, each consisting of one insertion and one deletion. Existing SSD-based systems fell short of that estimated rate even with more CPU cores and SSD bandwidth. Can an index keep up without so much CPU work?
PrairieFire buffers vector changes and rebuilds affected partitions in batches to support faster updates alongside search.
In comparisons against Greator, OdinANN, and SPFresh, PrairieFire achieved higher update and search rates on 100-million-vector SIFT and SPACEV workloads, with comparable or better recall@10.
The systems ran concurrent searches and updates, using thread allocations chosen to target roughly balanced rates.
At one billion vectors on SIFT, PrairieFire also had higher update and search throughput than the compared systems. These results support the system's approach in the tested settings: batching changes and rebuilding locally can sustain fast updates while keeping search performance.
These results come from specified workloads and an evaluation platform, and the systems' resource allocations were not identical. The tests do not establish the same performance for other datasets, systems, or production settings.
Analysis
Deep
- Original paper authors
- Yuchen Huang, Baiteng Ma, Yiping Sun, Xiao Chen, Rui Ma
- Submitted to arXiv
- License
- CC BY-SA 4.0