Error trying to train custom model. about pure

[a3616001]

Hi, this seems because the program run out of the GPU memory.
If possible, could you try a GPU with a larger memory? (Our experiments are based on a NVIDIA 2080Ti GPU, which has 11GB memory.) Also, try to train the model without adding cross-sentence context first, i.e., setting --context_window 0.

[AlanQuille]

Of course, I'll try tomorrow. Also, is my setting for task correct: --task ace05? Also, will your code accepts my relations:

"relations": [[[0, 0, 10, 11, "Founder"], [0, 0, 13, 14, "Founder"]], [[33, 33, 39, 39, "CEO"]], [[46, 46, 51, 54, "Located in"]]]}

Thanks again.

[a3616001]

The ACE05 doesn't include these relations. You can define relation types for new task in shared/const.py and add your task name in the entity model and the relation model.

[AlanQuille]

I have altered the files as you suggested, I am using this command to train the model:

python run_entity.py --do_train --learning_rate=1e-5 --task_learning_rate=5e-4 --train_batch_size=16 --context_window 0 --data_dir data_dir --model bert-base-uncased --output_dir output_dir --task inception

using RTX 3060 with 12 GB RAM, NVIDIA-SMI 460.91.03, and cuda==11.1

The command is working but it is taking a very long time, it seems to be stuck on "Moving to CUDA...". I know the GPU is working, the fan is at 78% when running the code. Is it supposed to take long?

My train.json, test.json and dev.json files are all the same. Perhaps it is because they all have the same doc_key? The content is as follows:

{"doc_key": "Test_001", "sentences": [["Google", "was", "founded", "on", "September", "4", ",", "1998", ",", "by", "Larry", "Page", "and", "Sergey", "Brin", "while", "they", "were", "Ph.D", ".", "students", "at", "Stanford", "University", "in", "California", "."], ["On", "December", "3", ",", "2019", ",", "Pichai", "also", "became", "the", "CEO", "of", "Google", "."], ["In", "March", "1999", ",", "the", "Google", "moved", "its", "offices", "to", "Palo", "Alto", ",", "California", "."]], "ner": [[[0, 0, "ORG"], [10, 11, "PER"], [13, 14, "PER"]], [[33, 33, "PER"], [39, 39, "ORG"]], [[46, 46, "ORG"], [51, 54, "LOC"]]], "relations": [[[0, 0, 10, 11, "Founder"], [0, 0, 13, 14, "Founder"]], [[33, 33, 39, 39, "CEO"]], [[46, 46, 51, 54, "Located in"]]]}

[a3616001]

It is weird that the code gets stuck on "Moving to CUDA". Did you run the code on the datasets we used (e.g., SciERC) for a sanity check?
Also, please use different doc_key, as we use the doc_key when generating the final predictions.

[AlanQuille]

Hi, I ran the code on the datasets you used and I am still running into the same issue, the following is showing up on my screen:

09/15/2021 19:03:28 - INFO - root - Moving to CUDA...

I have some questions:

1.) What is the time required to wait for this command to execute?

2.) I do not have any clusters defined in the train, dev and test JSON files. Would this make any difference?

3.) I am using --model bert-base-uncased. Is this for relation extraction? I only need the model for relation extractions.

Just to clarify, my task is to make a custom relation extraction model from the pretrained model using my domain data and domain relations. Thank you very much for all your help.

[AlanQuille]

Hi, I think I have resolved this issue. It takes 10 minutes but it moves on from "Moving from CUDA..." to the next stage. Thank you very much.