shiva-sankaran / lineex

Data Extraction from Scientific Line Charts

LineEX: Data Extraction from Scientific Line Charts

This repo contains code and models for the LineEX system, (link paper), which extracts data from scientific line charts. We adapt existing vision transformers and pose detection methods and showcase significant performance gains over existing SOTA baselines. We also propose a new loss function and present its effectiveness against existing loss functions.

The LineEX pipeline consists of three modular stages, which can be used independent from each other. They are :

• Keypoint Extraction
• Chart Element Detection and Text Extraction
• Keypoint Grouping, Legend Mapping and Datapoint Scaling

Usage

Clone this repository:

git clone https://github.com/Shiva-sankaran/LineEX.git
cd LineEX

Install the dependencies:

conda env create -f environment.yml
conda activate LineEX

Download weights and data

Weights and data will be placed at the correct folders

Set corresponding DATA_flag(True/False) to download a particular data set.

chmod +x download.sh
./download.sh -T False -V False  -L True  # To download only the test data 

UPDATE: Dataset moved to here.

UPDATE: Weights can be found here

Testing

Each of the modules can be used separately, or the entire pipeline can be called at once to extract the desired information. Output is stored in the corresponding directory

Overall

python pipeline.py --input_path = sample_input/

Keypoint detection

cd modules/KP_detection
python run.py

Chart element detection

cd modules/CE_detection
python run.py

Evaluation

Refer to the paper for more information about the metrics

Overall

Overall metrics is essentially the metric for grouping and legend mapping

cd modules/Grouping_legend_mapping
python eval.py

Keypoint detection

cd modules/KP_detection
python eval.py

Chart element detection

cd modules/CE_detection
python run.py

Training

Keypoint Extraction

cd modules/KP_detection
python -m torch.distributed.launch --nproc_per_node=3 --node_rank=0 train.py --vit_arch xcit_small_12_p16 --batch_size 42 --input_size 288 384 --hidden_dim 384 --vit_dim 384 --num_workers 24 --vit_weights https://dl.fbaipublicfiles.com/xcit/xcit_small_12_p16_384_dist.pth --alpha 0.99

Chart Element Detection and Text Extraction

cd modules/CE_detection
python -m torch.distributed.launch train.py --coco_path path_to_data

TBA

Need to change data paths

Citation

Shivasankaran, V. P., Muhammad Yusuf Hassan, and Mayank Singh. "LineEX: Data Extraction from Scientific Line Charts." 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2023.

lineex's Issues

Grouping_legend_mapping code for training

Could you please provide the Grouping_legend_mapping code for training? thanks！

Could you pls provide the results of ChartOCR?

Thank you for your contribution. Could you pls provide the results of ChartOCR that as a argment for KP evaluation.

LineEX/modules/KP_detection/eval.py

Line 66 in 173343c

parser.add_argument('--chartOCR_pred',default="/home/vp.shivasan/ChartIE/val_synth.json",help = "Path to chartOCR predictions on the dataset")

Grouping_legend_mapping权重文件

你好，请问可以提供modules/Grouping_legend_mapping/ckpts/mlp_ckpt.t7、modules/Grouping_legend_mapping/ckpts/ckpt_30.t7两个权重文件吗，感谢

unrecognized argument--input_path and attribute error for new test image prediction

I tried to detect the objects in a random line chart diagram with "python pipeline.py --input_path = sample_input/"
But it resulted in unrecognized arguments error(screenshot attached).
Then I just ran the command as "python pipeline.py" , the sample image files were processed, but when it came to parsing the new diagram that I uploaded in the sample_input folder, it resulted in an attribute error. The diagram I used looked similar to the diagrams that had been used for training the model.

Output variation with Input

Hi @Shiva-sankaran,

I used the pretrained weights and data and got the output as same as that is given in this repository, but facing a wide variation in the generated JSON output values with the input image.

How to check those values with the input image?

Could you please provide the sample file annotation for training?

Thank you for your contribution. Could you pls provide the file annotation for training: https://github.com/Shiva-sankaran/LineEX/blob/173343ceb92e7352427a4d937ee872b920a8d76f/modules/KP_detection/train.py#LL100C88-L100C88
Thank you so much!

unable to download weights on google colab

getting the following error:

Cannot retrieve the public link of the file. You may need to change the permission to 'Anyone with the link', or have had many accesses. You may still be able to access the file from the browser: https://drive.google.com/uc?id=176BjH_6W-HRoU9RvsysSNW27GVIHPlHV

Thanks

I used your model for map testing and found that the accuracy was high on the data you provided but very low on the other data in the paper (chart element detection module), I wonder if you can provide your data for this part?

weights files of modules/Grouping_legend_mapping

Could you please share the weight files of modules/Grouping_legend_mapping/ckpts/mlp_ckpt.t7 and modules/Grouping_legend_mapping/ckpts/ckpt_30.t7, thanks