Visual Text Grounding: New Challenges for Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) have made impressive progress in recent years. They can generate texts, understand images, and even combine different modalities. Despite these developments, a significant challenge remains in visual text grounding, particularly with text-rich images such as documents, forms, and infographics. These documents are characterized by complex layouts and dense textual information, which makes the precise linking of text and visual elements difficult.

Existing benchmarks mostly focus on visual grounding in natural images and neglect the specific challenges of text-rich documents. To address this gap, TRIG (Text-Rich Image Grounding) was developed – a new task with a specifically created dataset that evaluates and improves the capabilities of MLLMs in the field of document question answering.

TRIG: A New Benchmark for Text-Rich Images

The TRIG dataset was created using a combination of OCR (Optical Character Recognition), LLM interaction, and human annotation. It comprises 800 manually annotated question-answer pairs that serve as a benchmark. In addition, a comprehensive training dataset with 90,000 synthetically generated examples based on four different datasets was created. The questions in the dataset directly relate to the textual content of the images and require the MLLMs to have a precise understanding of the visual and textual information.

Evaluation and Improvement of MLLM Performance

A comprehensive evaluation of various MLLMs on the TRIG benchmark revealed significant weaknesses in their ability to perform visual text grounding in text-rich images. The models had difficulties identifying the relevant text passages in the complex layouts and linking them with the corresponding visual areas. To improve the performance of the MLLMs, two new methods were developed: a general Instruction Tuning and a plug-and-play method for efficient embedding. By fine-tuning the MLLMs on the synthetic TRIG dataset, significant improvements in spatial reasoning and grounding ability were achieved.

Future Research and Application Possibilities

The development of TRIG and the associated research results open new perspectives for the further development of MLLMs. The improved ability for visual text grounding in text-rich documents is relevant for numerous applications, including:

Automatic document analysis and processing

Improved search functions in digital archives

Development of intelligent assistants for working with documents

Creation of accessible technologies for people with visual impairments

Research in the field of visual text grounding is far from complete. Further studies are necessary to improve the robustness and generalizability of the developed methods and to further explore the limits of MLLMs in this area. The TRIG dataset and the associated methods provide a solid foundation for future research and development.

Bibliographie: Li, M., Zhang, R., Chen, J., Gu, J., Zhou, Y., Dernoncourt, F., Zhu, W., Zhou, T., & Sun, T. (2025). Towards Visual Text Grounding of Multimodal Large Language Model. arXiv preprint arXiv:2504.04974. Samdani, R., & Lee, K. H. (2024). Flamingo-2: An Open-Source Multimodal Language Model. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 1023-1040). Alayrac, J. B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., ... & Mensch, A. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35, 23716-23736. Liu, Y., Li, C., Wu, F., Zhang, S., Hu, H., & Zhang, Y. (2024). Visual instruction tuning. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 5918-5936). Huang, J., Dai, A., & Anandkumar, A. (2024). Towards Reasoning in Large Language Models: A Survey. arXiv preprint arXiv:2403.19322. Xiao, L. Awesome-Visual-Grounding. GitHub repository. Yang, Y., Hu, H., & Xing, E. P. (2024). Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 37. Wang, S., Xie, Y., Wang, M., Yang, J., Liang, X., & Yan, S. (2024). PromptCAL: Contrastive Affinity Learning for Few-Shot CLIP-Adapter. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 1279-1289). Springer, Cham. Li, Y., Liu, H., & Liu, Z. (2024). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv preprint arXiv:2404.13013.