MetaDB reshapes databases around file-system metadata
MetaDB: Putting File-System Structure Back into Distributed Databases · arXiv:2610.32247
In plain words
File systems organize files and folders in directories. General-purpose databases may split related directory records across machines, make many changes compete over one parent record, or slow down folder listings after large batches of deletions.
MetaDB is a distributed database built around those patterns. It keeps directory data together and handles some updates asynchronously. In a test of concurrent folder creation in one directory, that update method achieved 10.7 times the throughput of lock-based transactions, under the tested setup.
The research
File systems create and remove entries through directories, yet a general-purpose database may scatter related records and require strict coordination for updates that could be delayed.
Can a database designed around these patterns handle directory contention and deletion-heavy scans more efficiently?
MetaDB aligns data placement and update handling with directory structure and metadata needs, then tests the design under contention, deletion, and production workloads.
In a production-cluster test, the authors used mdtest to issue five metadata operations to MetaDB and a general-purpose database on one 10-backend-node cluster. With 20 clients, 50 threads each, and 2,000 requests per client, MetaDB achieved 2.2 to 4.1 times GP-DB’s throughput across the five operations. This shows higher throughput in that test.
The result supports the case for tailoring database design to metadata workloads, while the separate production latency observations do not isolate the database effect because their clusters differed substantially in scale.
The reported production latency observations came from two differently-sized clusters: GP-DB had 4 backend nodes managing 1 billion files and directories, while MetaDB had 50 nodes managing 50 billion. The authors say these observations are not a strict performance comparison.
What the evidence shows
- A database for metadata
MetaDB is a metadata-aware distributed database used as a shared metadata layer for Baidu storage systems. The paper asks how to handle file-system metadata when directory structure, update consistency needs, and deletion patterns differ from the assumptions of general-purpose databases.
- Directory updates contend
One test illustrates the contention problem: in the previous system, create throughput fell from about 60,000 queries per second without contention to about 6,000 when all requests targeted one directory.
The requests then competed to update the same parent-directory record, motivating a way to handle those updates without locking each one in the foreground.
- Aggregation eases contention
For concurrent mkdir requests under the same directory, the authors compared lock-based transactions with asynchronous aggregation. Each client sent four concurrent requests as client counts increased.
Aggregation achieved 10.7 times the throughput of lock-based execution in this experiment, evidence for this approach under the tested contention pattern.
- Storage and production results
A separate deletion test compared four storage configurations during bulk deletion, with one concurrent folder listing. The configuration combining in-memory MVCC and locality-aware compaction had a 0.7-second peak listing latency, lower and more stable than the other tested configurations.
A production-cluster test also found higher throughput than GP-DB across five metadata operations, within that test’s scope.
Deep
- Original paper authors
- Zebin Ren, Biao Cao, Kang Chen, Zhihui Yin, Qiushi Chen
- Submitted to arXiv
- License
- CC BY 4.0