carolinegao / lora-dataset

[NeurIPS2023] LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering

LoRA Dataset Generation

LoRA Project Website

This is the code used to generate the LoRA Dataset.

You can use this code to generate complex logical questions and related images, like this:

Q: If we do not have milk, is there another dairy product that does not necessarily contain fat but is rich in protein that can be subsituted for breakfast?
A: Yogurt

Q: Can we use the food between eggs and bread to make a meal for vegetarians?
A: No

Step 1: Constructing Ontology

First, we construct a knowledge base by using Owlready2.

cd ontology
python ontology.py

Step 2: Generating Questions

Next we generate questions based on the ontology and logical operators. You can generate logical questions with 2 logical operators and 3 logical operators.

You can generate basic logical questions by using:

cd question_generation
python generate_questions_logic2.py

The detailed steps can also be found in the basic_logic_questions_generation.ipynb file.

More complex logical reasoning questions are generated by using:

cd question_generation
python generate_questions_logic3.py

The detailed steps can also be found in the complex_reasoning_questions_generation.ipynb file.

After generating the questions using the template, you can convert these questions into natural language questions using the following script:

cd question_generation
python generate_questions_natural_language_logic2.py
python generate_questions_natural_language_logic3.py

Step 3: Generating Answers

The answers are batch generated by using SparQL_query algorithm.

The algorithm can be found here:

cd answer_generation
SparQL_query.ipynb

To generate the answers for logic 2 questions, run:

cd answer_generation
python generate_answers_logic2.py

To generate the answers for logic 3 questions, run:

cd answer_generation
python generate_answers_logic3.py

The details can also be found in the answer_generation.ipynb file.

Step 4: Generating Images

The images are generated based on answers and noise objects. Each image contains the answers objects and the noise objects. We use Blender script to generate the realistic kitchen scene.

Firstly, install blender v2.93.

Secondly, you need to generate visible_objects.csv file by using:

cd image_generation
python visible_object_processing.py

The generated visible objects file are saved in the folder:

cd image_generation
logic2_and_visible.csv

You can choose different blender background. The blender background file is Vqa_Background_1.blend. We provided four kitchen background scenarios in the file image_generation/blender_background/Vqa_Backgrounds_Full.blend.

You can then render some images by using:

cd image_generation
blender Vqa_Background_1.blend --background --python blend_image_generation.py

Download LoRA Dataset:

https://drive.google.com/drive/folders/1H4msZ4ae1msEDApxOT7cr1A1QwQ33Pkn

Citation:

@inproceedings{gao2023lora,
  title={LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering},
  author={Gao, Jingying and Wu, Qi and Blair, Alan and Pagnucco, Maurice},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}

lora-dataset's Issues

Upload dataset to Huggingface

Now the dataset is hosted on Google Drive and a virus scan warning will prompt when downloading, resulting failure when attempting to download them with gdown.pl and other tools. Thus, I have to download manually on my laptop and then upload the the server. It is a bit bothering.

Can you upload the dataset to huggingface for public downloading?

There are fixed path in code

In almost every python file there is a fixed path /home/jingying/LoRA/, and the code reads data from files under this path.

Please remove them and use a relative path.