Stored page vectors can reveal the document they represent
In plain words
Document search systems can store a page as many vectors, one for each image patch. The paper asks whether someone could recover the page from those vectors alone, without the original page or its details.
The authors train an attack to generate page images from stored vectors. On a fixed set of public pages, its raw-index images recovered more source words than returning the nearest training page, and the source page usually ranked first when the images were searched against the stored pages.
The research
When a visual document search system stores page vectors separately from the page images, can someone who gets only those vectors reconstruct the documents? The paper tests whether an index that people cannot read directly still preserves enough content and layout to reveal the page it represents.
The authors build an attack that infers missing page details and generates document images from stored multi-vector indices.
The authors tested whether patch vectors that look unreadable can still reveal the document page they index. Their attack generates a page from the stored vectors, inferring the encoder and page shape and restoring vector order when needed.
On a fixed public-page set, raw-index reconstructions recovered a mean 47.4% of source words, versus 23.4% for the nearest-training-page baseline.
The retriever ranked the source first for 98.4% of reconstructions, and an independent encoder did so for 97.7%, evidence that the index itself retained recoverable page information.
The tests cover two public retrievers from one model family and an encoder-identification pool of 13 public encoders. The weak results for pooled indices apply only to the attacks tested; the authors did not try to recover their layout.
Analysis
Deep
- Authors
- Zhuchenyang Liu, Yao Zhang, Yu Xiao
- License
- CC BY 4.0