A curated list of recent and past chart understanding work based on our survey paper: From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models.
The repository will be continuously updated 📝. Don't forget to hit the ⭐️ and stay tuned!
If you find this resource beneficial for your research, please do not hesitate to cite the paper referenced in the Citation section. Thank you!
Factoid Questions
-
DVQA: Understanding Data Visualizations via Question Answering.
-
FigureQA: An Annotated Figure Dataset for Visual Reasoning.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, Yoshua Bengio.
-
LEAF-QA: Locate, Encode & Attend for Figure Question Answering.
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, Ajay Joshi.
-
STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering.
-
PlotQA: Reasoning over Scientific Plots.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Pratyush Kumar.
-
MapQA: A Dataset for Question Answering on Choropleth Maps.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, Ningchuan Xiao.
-
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, Enamul Hoque.
-
SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs.
-
MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning.
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong Yu.
-
MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao.
-
ChartBench: A Benchmark for Complex Visual Reasoning in Charts.
Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, Jian Guo.
-
Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, Qi Liu.
Long-form Questions
-
OpenCQA: Open-ended Question Answering with Charts.
Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, Shafiq Joty.
-
Figure Captioning with Relation Maps for Reasoning.
Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Ryan Rossi.
-
Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model.
-
What Will You Tell Me About the Chart? – Automated Description of Charts.
Karolina Seweryn, Katarzyna Lorenc, Anna Wróblewska, Sylwia Sysko-Romańczuk.
-
SciCap: Generating Captions for Scientific Figures.
-
Chart-to-Text: A Large-Scale Benchmark for Chart Summarization.
Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty.
-
LineCap: Line Charts for Data Visualization Captioning Models.
-
ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries.
Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tahmid Rahman Laskar, Md. Hamjajul Ashmafee, Abu Raihan Mostofa Kamal.
-
VisText: A Benchmark for Semantically Rich Chart Captioning.
-
FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback.
Ashish Singh, Prateek Agarwal, Zixuan Huang, Arpita Singh, Tong Yu, Sungchul Kim, Victor Bursztyn, Nikos Vlassis, Ryan A. Rossi.
-
Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, Qi Liu.
-
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning.
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji.
-
Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking.
-
ChartCheck: An Evidence-Based Fact-Checking Dataset over Real-World Chart Images.
Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, Elena Simperl.
-
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning.
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji.
-
A Simple Neural Network Module for Relational Reasoning.
Adam Santoro, David Raposo, David G.T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Timothy Lillicrap.
-
DVQA: Understanding Data Visualizations via Question Answering.
-
MapQA: A Dataset for Question Answering on Choropleth Maps.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, Ningchuan Xiao.
-
DVQA: Understanding Data Visualizations via Question Answering.
-
Answering Questions about Data Visualizations using Efficient Bimodal Fusion.
Kushal Kafle, Robik Shrestha, Brian Price, Scott Cohen, Christopher Kanan.
-
LEAF-QA: Locate, Encode & Attend for Figure Question Answering.
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, Ajay Joshi.
-
PlotQA: Reasoning over Scientific Plots.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Pratyush Kumar.
-
TaPas: Weakly Supervised Table Parsing via Pre-training.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, Julian Eisenschlos.
-
STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering.
-
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, Enamul Hoque.
-
SciCap: Generating Captions for Scientific Figures.
-
Figure Captioning with Relation Maps for Reasoning.
Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Ryan Rossi.
-
Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model.
-
Tackling Hallucinations in Neural Chart Summarization.
Saad Obaid ul Islam, Iza Škrjanec, Ondřej Dušek, Vera Demberg
-
Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs.
Mingyang Zhou, Yi Fung, Long Chen, Christopher Thomas, Heng Ji, Shih-Fu Chang.
-
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Eisenschlos.
-
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning.
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, Shafiq Joty.
-
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding.
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova.
-
Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA.
-
DePlot: One-shot visual language reasoning by plot-to-table translation.
Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun.
-
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning.
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji.
-
Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization.
Xuan Long Do, Mohammad Hassanpour, Ahmed Masry, Parsa Kavehzadeh, Enamul Hoque, Shafiq Joty.
-
DOMINO: A Dual-System for Multi-step Visual Language Reasoning.
Peifeng Wang, Olga Golovneva, Armen Aghajanyan, Xiang Ren, Muhao Chen, Asli Celikyilmaz, Maryam Fazel-Zarandi.
-
StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding.
Renqiu Xia, Bo Zhang, Haoyang Peng, Hancheng Ye, Xiangchao Yan, Peng Ye, Botian Shi, Yu Qiao, Junchi Yan.
Tailored for Chart Understanding
-
ChartLlama: A Multimodal LLM for Chart Understanding and Generation.
Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, Hanwang Zhang.
-
MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning.
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong Yu.
-
ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning.
Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, Ping Luo.
-
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning.
Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty.
-
ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning.
Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, Yu Qiao.
-
FigurA11y: AI Assistance for Writing Scientific Alt Text.
-
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs.
Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, Abhanshu Sharma.
General-purpose
-
Visual Instruction Tuning.
-
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, Fei Huang.
-
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, Jingren Zhou.
-
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou.
-
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao.
-
Gemini: A Family of Highly Capable Multimodal Models.
-
GPT-4V.
-
Introducing the next generation of Claude (Claude 3).
-
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning.
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji.
@misc{huang-etal-2024-chart,
title = "From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models",
author = "Huang, Kung-Hsiang and Chan, Hou Pong and Fung, Yi R. and Qiu, Haoyi and Zhou, Mingyang and Joty, Shafiq and Chang, Shih-Fu and Ji, Heng",
year={2024},
eprint={2403.12027},
archivePrefix={arXiv},
primaryClass={cs.CL}
}