DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering

Answer grounding in visual question answering over documents remains an open challenge: most existing benchmarks lack grounding annotations or provide annotations of limited quality, and the development of answer grounding datasets is still bounded to expensive manual effort. We introduce DocAttriBench (DAB), an improved large-scale benchmark for fine-grained, element-level source attribution in Document VQA, grounding each answer to specific layout elements such as text blocks, tables, and images. To build DAB, we propose a Mask-based Perplexity-Derived Attribution method (MAPPET) that leverages document layout and language modeling to assign each answer to its most informative document element. MAPPET measures the increase in perplexity when masking candidate elements, attributing the answer to the element that contributes most to model confidence. Applying MAPPET to multiple existing Document VQA datasets yields DAB, which contains 237k documents and 296k question-answer pairs, where each answer is grounded to the corresponding layout elements. We benchmark a range of Multimodal Large Language Models (MLLMs) with grounding capabilities on DAB, evaluating answer accuracy, attribution accuracy, and overall answer quality. Our results show that while larger models tend to achieve higher answer accuracy, even the most capable models often fail to correctly localize the supporting elements. DAB thus provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models.