AI-Powered Chart Analysis: ChartCitor Enables Precise Source References

Analyzing charts and answering related questions is a complex task requiring both visual understanding and the ability to interpret data. While Large Language Models (LLMs) can already analyze charts and answer questions, they often generate unverified and hallucinated responses. Existing methods for verifying these answers struggle to trace them back to specific areas within the charts. This is due to limited visual-semantic context, complex image-text alignment requirements, and the difficulty of predicting bounding boxes in complex layouts.

A new approach to address this challenge is ChartCitor, a multi-agent framework that provides detailed source references in the form of bounding boxes within charts. By identifying supporting evidence within the visuals, ChartCitor enables more precise traceability of the generated answers. The system orchestrates multiple LLM agents that perform various tasks:

- Extracting tables from charts - Rephrasing questions - Augmenting tables with additional information - Retrieving relevant information through pre-filtering and re-ranking - Mapping table content back to the chart

ChartCitor follows a multi-stage process. First, an agent extracts the relevant data from the chart and converts it into a tabular format. Another agent rephrases the original question to optimize it for table analysis. The table is then enriched with additional information to broaden the context. In the next step, relevant information is extracted from the table by filtering out irrelevant entries and ranking the remaining ones by relevance. Finally, the retrieved information is mapped back to the corresponding area in the chart and represented as a bounding box.

Improved Transparency and User Trust

Tests have shown that ChartCitor achieves higher accuracy in mapping answers to chart areas compared to existing methods. This holds true for various chart types, highlighting the robustness of the approach. Qualitative user studies indicate that ChartCitor increases user trust in generative AI by improving the explainability of LLM-powered chart question-answering systems. By precisely linking answers to the underlying data in the chart, users can better understand and verify the generated responses. This allows professionals to work more efficiently and make more informed decisions.

The development of ChartCitor represents a significant advancement in the field of AI-powered chart analysis. By providing detailed source references, the system contributes to increasing the transparency and traceability of LLM-generated answers and strengthens user trust in this technology. The ability to accurately process and interpret complex visual information opens up new possibilities for applying AI in various fields, from data analysis to scientific research.

Bibliographie: Goswami, K., Mathur, P., Rossi, R., & Dernoncourt, F. (2025). ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution. arXiv preprint arXiv:2502.00989. Autonomous Agents on Github. https://github.com/tmgthb/Autonomous-Agents ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution. https://www.researchgate.net/publication/388658278_ChartCitor_Multi-Agent_Framework_for_Fine-Grained_Chart_Visual_Attribution Dernoncourt, F. (2025). ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution. Hugging Face. https://huggingface.co/papers/2501.11067 Hugging Face Papers. https://huggingface.co/papers?date=2025-02-07 Su, Y. et al. (2024). ChatCoT: Exploring Human-Chatbot Collaborative Reasoning on Complex Open-Domain Tasks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 239-248). https://aclanthology.org/2024.emnlp-demo.26.pdf Youtube Video: I built an AI Agent with NO CODE! https://www.youtube.com/watch?v=AwnltW8n74A&pp=ygUJI2Fp56CU55m8