mandarjoshi90 / coref Goto Github PK
View Code? Open in Web Editor NEWBERT for Coreference Resolution
License: Apache License 2.0
BERT for Coreference Resolution
License: Apache License 2.0
A couple of really minor typos in README:
Thanks for releasing all of this!
I want to test & evaluate performance of this model on GAP data-set. Using its below google colab notebook to avoid machine dependency issues.
https://colab.research.google.com/drive/1SlERO9Uc9541qv6yH26LJz5IM9j7YVra#scrollTo=H0xPknceFORt
How can I view the results of evaluation metrics as shown in the mentioned research paper?
Secondly, I tried to run cmd ! GPU=0 python evaluate.py $CHOSEN_MODEL
in colab, assuming it would generate the evaluation results, but getting below error:
..
..
..
W0518 20:39:24.163360 140409457641344 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/optimizer_v2/learning_rate_schedule.py:409: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0518 20:39:24.184890 140409457641344 deprecation_wrapper.py:119] From /content/coref/optimization.py:64: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
bert:task 199 27
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2020-05-18 20:39:34.733225: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-05-18 20:39:34.735705: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-05-18 20:39:34.735782: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fcef0d45f3e2): /proc/driver/nvidia/version does not exist
2020-05-18 20:39:34.736239: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-18 20:39:34.750671: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200160000 Hz
2020-05-18 20:39:34.751024: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x78bf9c0 executing computations on platform Host. Devices:
2020-05-18 20:39:34.751074: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
Restoring from ./spanbert_base/model.max.ckpt
2020-05-18 20:39:40.322311: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
W0518 20:39:47.165590 140409457641344 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "evaluate.py", line 26, in <module>
model.evaluate(session, official_stdout=True, eval_mode=True)
File "/content/coref/independent.py", line 538, in evaluate
self.load_eval_data()
File "/content/coref/independent.py", line 532, in load_eval_data
with open(self.config["eval_path"]) as f:
FileNotFoundError: [Errno 2] No such file or directory: './dev.english.384.jsonlines'
Any idea what could be the possible reason? I am new to this area/environment and following the colab code right now.
Looking for some suggestions / steps to do evaluation part.
Thanks in advance!
Hi Mandar,
I wonder if the model can be finetuned using the pytorch version of BERT? I saw that there are tf_checkpoint and init_checkpoint in the config file. Is it possible to modify them in order to support pytorch BERT models.
Hi, I tried to run python train.py <experiment>
following the instruction in README. I tried <experiment>=bert_base
and <experiment>=spanbert_base
. and set export data_dir=/data/coref
. I met the following errors:
Restoring from: /data/coref/bert_base/model-57000
W1010 16:06:31.544241 140159242708736 deprecation.py:323] From /home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "train.py", line 41, in <module>
saver.restore(session, ckpt.model_checkpoint_path)
File "/home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1278, in restore
compat.as_text(save_path))
ValueError: The passed save_path is not a valid checkpoint: /data/coref/bert_base/model-57000
The bert_base directory created by ./download_pretrained.sh bert_base
contains:
bert_config.json checkpoint events.out.tfevents.1551148806.learnfair2008 events.out.tfevents.1551148825.learnfair0213 events.out.tfevents.1551148826.learnfair0213 model.max.ckpt.data-00000-of-00001 model.max.ckpt.index stdout.log vocab.txt
Restoring from: /checkpoint/danqi/coref_eval/final/base_pair_external_sl384_blr2e-05_tlr0.0001/model-57000
W1010 15:52:59.750342 139657016100608 deprecation.py:323] From /home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "train.py", line 41, in <module>
saver.restore(session, ckpt.model_checkpoint_path)
File "/home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
if not checkpoint_management.checkpoint_exists(compat.as_text(save_path)):
File "/home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_management.py", line 372, in checkpoint_exists
if file_io.get_matching_files(pathname):
File "/home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 363, in get_matching_files
return get_matching_files_v2(filename)
File "/home/jiezhong/anaconda3/envs/coref/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 384, in get_matching_files_v2
compat.as_bytes(pattern))
tensorflow.python.framework.errors_impl.NotFoundError: /checkpoint/danqi/coref_eval/final/base_pair_external_sl384_blr2e-05_tlr0.0001; No such file or directory
It seems that the path /checkpoint/danqi/coref_eval/final/base_pair_external_sl384_blr2e-05_tlr0.0001
is hard-coded somewhere. Moreover, the spanbert_base directory created by ./download_pretrained.sh spanbert_base
contains:
bert_config.json checkpoint events.out.tfevents.1561596094.learnfair1413 model.max.ckpt.data-00000-of-00001 model.max.ckpt.index stdout.log vocab.txt
Any help would be greatly appreciated!
Thank you in advance.
Hi,
I wanted to ask about subtoken_map
in jsonlines file. In trial.jsonlines
{"doc_key": "bn/voa/02/voa_0210_0",
"sentences": [["[CLS]", "Meanwhile", "Prime", "Minister", "E", "##hu", "##d", "Bar", "##ak", "told", "Israeli", "television", "he", "doubts", "a", "peace", "deal", "can", "be", "reached", "before", "Israel", "'", "s", "February", "6th", "election", ".", "He", "said", "he", "will", "now", "focus", "on", "suppress", "##ing", "Palestinian", "violence", ".", "[SEP]"]],
"speakers": [["[SPL]", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "[SPL]"]],
"clusters": [[]],
"sentence_map": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2],
"subtoken_map": [0, 0, 1, 2, 3, 3, 3, 4, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 30, 31, 32, 33, 33]
}
The subtoken_map
is same (33) for "."
and "[SEP]"
tokens while in README example it differs for full-stop and [SEP] token.
So I wanted to ask which of the convention is expected?
Thank you for open-sourcing the code. I am wondering if you can release of TensorFlow checkpoints of the spanBERT other than the pytorch version.
experiments.conf:
tf_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
init_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
I am running train.py,wrong indication:Failed to find any matching file for bert_base/model.max.ckpt.
I'm not sure where the file came from or was it generated in the middle.
Is there any good way?
Hi,
Thanks for releasing the code. While running predict.py
I was getting
tensorflow.python.framework.errors_impl.NotFoundError: ./coref_kernels.so: undefined symbol: _ZN10tensorflow12OpDefBuilder4AttrESs
I searched and found this link. It would be nice to either check the GCC version or keep this as a warning in the script to help anyone else stuck here!!
I am using Windows 10 OS. What environment is required to setup & run this project on my machine?
PS: Anaconda3 is already Installed. What else is required to execute the given python/shell cmds in Setup section?
Hi,
I tried getting this going on a new macbook pro but installation didn't seem possible. I just wanted to use your pretrained model to make cluster predictions from a set of documents. I'm wondering if this use case is supported yet. It seems like maybe this is windows/linux + GPU only at this point.
I have a question about a Independent.
If I use Independent
and max_segment_len = 128
, It divide all documents into 128 tokens. After 128 tokens enter BERT, It will make an embedding for them.
In this situation,
(1) Is it looking for a antecedent only within the 128 tokens?
OR
(2) Put all segments in BERT and get embedding for all tokens.
and after that, Is it looking for a antecedent within all token?
Lines 444 to 448 in 6db8cac
Hi, is there dropout wrong thing above?
So when i pass a text in the required json format.Its predicting for some examples properly.but it throws assertion error for other examples. And i found out its due to larger length of text.cant we use span bert coref for a paragraph having large sentences having total more than 512 words .
InvalidArgumentError (see above for traceback): assertion failed: [] [Condition x <= y did not hold element-wise:x (bert/embeddings/strided_slice_3:0) = ] [679] [y (bert/embeddings/assert_less_equal/y:0) = ] [512] [[Node: bert/embeddings/assert_less_equal/Assert/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT32, DT_STRING, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](bert/embeddings/assert_less_equal/All, bert/embeddings/assert_less_equal/Assert/Assert/data_0, bert/embeddings/assert_less_equal/Assert/Assert/data_1, bert/embeddings/strided_slice_3, bert/embeddings/assert_less_equal/Assert/Assert/data_3, bert/embeddings/assert_less_equal/y)]]
Hello,
After creating test jsonlines files and changing eval_path to one of the test files in experiments.conf, I only can run bert_base with test.english.128.jsonlines and the F1 score is 73.38, others(only test on bert_base and bert_large) were all failed by tensorflow.python.framework.errors_impl.OutOfRangeError: Read less bytes than requested
[[{{node checkpoint_initializer_337}}]]
[[{{node checkpoint_initializer_3}}]]
could you help me to have check what's wrong on it? If current repo can run successfully, please?
Many thanks~
Is here anyone using google colab to train this model? I have faced many problems such as:
1- The model was unable to create gold_conll files in colab, I have built them in my system locally and then uploaded it to my drive to solve this.
2- I am unable to train the model, it reports a problem related to pyhocon such as follows:
Traceback (most recent call last):
File "train.py", line 18, in <module>
config = util.initialize_from_env()
File "/content/coref/util.py", line 37, in initialize_from_env
config = pyhocon.ConfigFactory.parse_file("experiments.conf")[name]
File "/usr/local/lib/python3.6/dist-packages/pyhocon/__init__.py", line 32, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename))
File "/usr/local/lib/python3.6/dist-packages/pyhocon/__init__.py", line 60, in parse_string
return ConfigParser().parse(content, basedir)
File "/usr/local/lib/python3.6/dist-packages/pyhocon/__init__.py", line 198, in parse
ConfigParser._resolve_substitutions(config, substitutions)
File "/usr/local/lib/python3.6/dist-packages/pyhocon/__init__.py", line 241, in _resolve_substitutions
is_optional_resolved, resolved_value = ConfigParser._resolve_variable(config, substitution)
File "/usr/local/lib/python3.6/dist-packages/pyhocon/__init__.py", line 223, in _resolve_variable
col=col(substitution.loc, substitution.instring)))
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${data_dir} (line: 95, col: 16)
any help with this?
I run: pip install -r requirements.txt
Got following error.
ERROR: mxnet 1.4.0 has requirement numpy<1.15.0,>=1.8.2, but you'll have numpy 1.18.1 which is incompatible.
ERROR: bert-embedding 1.0.1 has requirement numpy==1.14.6, but you'll have numpy 1.18.1 which is incompatible.
Installing collected packages: PyYAML, awscli, certifi, jpype1, cython, mmh3, future, pystanforddependencies, soupsieve, beautifulsoup4, cort
Found existing installation: PyYAML 5.1.2
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Is the project depend on specific version of packages mentioned or higher version will also work ?
The Setup for training section says:
./setup_training.sh <ontonotes/path/ontonotes-release-5.0> $data_dir
What does this path <ontonotes/path/ontonotes-release-5.0> refer? How can I access it?
Hi, I'm trying to retrain the coref model starting from another BERT model trained on different data. It seems the loss values are not going down but another issue is that training seems slow and the GPU is underutilized (screenshot below). Any tips on how to speed up the training or fit more data in the gpu?
TITAN V | 40'C, 0 % | 316 / 12036 MB
I0905 12:20:54.217084 140321920943872 train.py:59] [100] loss=2071.04, steps/s=0.20
I0905 12:21:45.037879 140321920943872 train.py:59] [110] loss=837.87, steps/s=0.20
I0905 12:22:37.386365 140321920943872 train.py:59] [120] loss=1475.69, steps/s=0.20
I0905 12:23:34.424523 140321920943872 train.py:59] [130] loss=1111.34, steps/s=0.20
I0905 12:24:26.693988 140321920943872 train.py:59] [140] loss=1088.69, steps/s=0.20
I0905 12:25:14.780310 140321920943872 train.py:59] [150] loss=792.43, steps/s=0.20
I0905 12:26:08.272615 140321920943872 train.py:59] [160] loss=1597.89, steps/s=0.20
I0905 12:26:55.389269 140321920943872 train.py:59] [170] loss=1087.88, steps/s=0.20
Thank you for your comment.
${data_dir}/*.jsonlines, ${data_dir}/*.v4_gold_conll
were made successfully.
I tried GPU=0 python train.py train_1
but, AssertionError occurred in evaluation.
It's my experiments.conf
data_dir = data
best {
# Edit this
data_dir = data
model_type = independent
# Computation limits.
max_top_antecedents = 50
max_training_sentences = 5
top_span_ratio = 0.4
max_num_speakers = 20
max_segment_len = 256
# Learning
bert_learning_rate = 1e-5
task_learning_rate = 2e-4
num_docs = 2802
# Model hyperparameters.
dropout_rate = 0.3
ffnn_size = 1000
ffnn_depth = 1
num_epochs = 20
feature_size = 20
max_span_width = 30
use_metadata = true
use_features = true
use_segment_distance = true
model_heads = true
coref_depth = 2
coarse_to_fine = true
fine_grained = true
use_prior = true
# Other.
train_path = train.english.jsonlines
eval_path = dev.english.jsonlines
conll_eval_path = dev.english.v4_gold_conll
single_example = true
genres = ["bc", "bn", "mz", "nw", "pt", "tc", "wb"]
eval_frequency = 1000
report_frequency = 100
#log_root = ${data_dir}
log_root = models
adam_eps = 1e-6
task_optimizer = adam
}
train_1 = ${best}{
num_docs = 2802
bert_learning_rate = 1e-05
task_learning_rate = 0.0002
max_segment_len = 128
ffnn_size = 200
model_heads = false
train_path = ${data_dir}/train.english.128.jsonlines
eval_path = ${data_dir}/dev.english.128.jsonlines
conll_eval_path = ${data_dir}/dev.english.v4_gold_conll
max_training_sentences = 5
bert_config_file = data/cased_L-12_H-768_A-12/bert_config.json
vocab_file = data/cased_L-12_H-768_A-12/vocab.txt
tf_checkpoint = data/cased_L-12_H-768_A-12/bert_model.ckpt
init_checkpoint = data/cased_L-12_H-768_A-12/bert_model.ckpt
}
and result
I0829 11:00:12.977631 139776037099264 train.py:58] [1000] loss=nan, steps/s=5.96
Loaded 343 eval examples.
2019-08-29 11:00:18.389048: W tensorflow/core/kernels/queue_base.cc:277] _0_padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
File "train.py", line 64, in <module>
eval_summary, eval_f1 = model.evaluate(session, tf_global_step)
File "/data/BERT-coref/coref/independent.py", line 559, in evaluate
coref_predictions[example["doc_key"]] = self.evaluate_coref(top_span_starts, top_span_ends, predicted_antecedents, example["clusters"], coref_evaluator)
File "/data/BERT-coref/coref/independent.py", line 524, in evaluate_coref
predicted_clusters, mention_to_predicted = self.get_predicted_clusters(top_span_starts, top_span_ends, predicted_antecedents)
File "/data/BERT-coref/coref/independent.py", line 499, in get_predicted_clusters
assert i > predicted_index, (i, predicted_index)
AssertionError: (0, 0)
Is there anything I missed?
A pair of vectors walked in 2 a bar....
It's original code.
bert_base = ${best}{
num_docs = 2802
bert_learning_rate = 1e-05
task_learning_rate = 0.0002
max_segment_len = 128
ffnn_size = 3000
train_path = ${data_dir}/train.english.128.jsonlines
eval_path = ${data_dir}/dev.english.128.jsonlines
conll_eval_path = ${data_dir}/dev.english.v4_gold_conll
max_training_sentences = 11
bert_config_file = ${best.log_root}/bert_base/bert_config.json
vocab_file = ${best.log_root}/bert_base/vocab.txt
tf_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
init_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
}
train_bert_base = ${bert_base}{
tf_checkpoint = ${best.log_root}/cased_L-12_H-768_A-12/bert_model.ckpt
init_checkpoint = ${best.log_root}/cased_L-12_H-768_A-12/bert_model.ckpt
}
I think below is right.
If I want to train at first time.
train_bert_base = ${bert_base}{
bert_config_file = ${best.log_root}/cased_L-12_H-768_A-12/bert_config.json
vocab_file = ${best.log_root}/cased_L-12_H-768_A-12/vocab.txt
tf_checkpoint = ${best.log_root}/cased_L-12_H-768_A-12/bert_model.ckpt
init_checkpoint = ${best.log_root}/cased_L-12_H-768_A-12/bert_model.ckpt
}
Is it the one in experiment.conf
? but the paper said setting max_seq_len
to 128 is always the best for bert_base and spanbert_base.
Hi Mandar,
Thanks for this repo.
I am using the pretrained model. If I have a set of sentences in txt format how should I predict the coref. Is there any method to directly convert my sentences in txt format into the required format that model need as an input?
Publishing your paper on arxiv is great.
But please, you achieve state of the art results, so publish your result on the World reference for state of the art which is:
Paperswithcode.com
Add your results here
https://paperswithcode.com/task/coreference-resolution
Someone already published your paper
https://paperswithcode.com/paper/bert-for-coreference-resolution-baselines-and
in independent.py
top_fast_antecedent_scores = util.batch_gather(fast_antecedent_scores, top_antecedents) # [k, c]
sometimes return [NaN, NaN ...]
I tried to print the value of tensors using tf.Print()
batch_gather in util.py
def batch_gather(txt, emb, indices):
batch_size = shape(emb, 0)
seqlen = shape(emb, 1)
if len(emb.get_shape()) > 2:
emb_size = shape(emb, 2)
else:
emb_size = 1
flattened_emb = tf.reshape(emb, [batch_size * seqlen, emb_size]) # [batch_size * seqlen, emb]
offset = tf.expand_dims(tf.range(batch_size) * seqlen, 1) # [batch_size, 1]
gathered = tf.gather(flattened_emb, indices + offset) # [batch_size, num_indices, emb]
gathered = tf.Print(gathered, [gathered], message='gathered')
if len(emb.get_shape()) == 2:
gathered = tf.squeeze(gathered, 2) # [batch_size, num_indices]
gathered = tf.Print(gathered, [gathered], message=txt+'gathered2')
return gathered
The results are as follows.
emb == [[-inf -inf -inf...]...]
flattened_emb == [[-inf][-inf][-inf]...]
indice + offset == [[808 809 810...]...]
gathered == [[[nan][nan][nan]]...]
gathered2 == [[nan nan nan...]...]
Sometimes it doesn't happen(well done), but it happens frequently(training stops because Loss is NaN).
Can you help me..?
Hello author:
i want to change the config para max_training_sentences to 5, origin 3.
but i was received a error
ValueError: Shape of variable coref_layer/segment_distance/segment_distance_embeddings:0 ((5, 20)) doesn't match with shape of tensor coref_layer/segment_distance/segment_distance_embeddings ([3, 20]) from checkpoint reader.
Where should I change?
best wish
Look forward to your reply
hello author:
I want to use this res code to train my model with another dataset.
But i find original spanbert is a pytorch.bin file. config file is ckpt model loader.
I can't load spanbert.
best wish
Line 231 in bdd1525
Hi, I guess an offset
can only be the first term, not the subtraction.
Hi Mandar,
When I finetune BERT base, I get an OntoNotes dev F1 of 73.69. I was wondering if this is within the variance that you saw for BERT base, or could there be some problem with my setup?
I'm using the requirements versions from requirements.txt
(except with the MarkupSafe version changed to 1.1.1, #40, and psycopg2 changed to psycopg2-binary), and am training on a V100 32GB, with these commands:
python train.py train_bert_base
python evaluate.py train_bert_base
When evaluating your finetuned BERT base model on dev (python evaluate.py bert_base
), I get an F1 of 74.05. This is closer to the 74.3 dev F1 number from Table 4, but should it match exactly? I'm wondering if there could be some difference in my setup which affects eval a bit but gets magnified during training.
Thanks,
Daniel
pip install requirements.txt
I'm really really sorry to ask you a trivial question.
train.py train_bert_base
works well!
I have Korean data in conll 2012 form.
they are work in e2e-coref model.
I want to apply Korean data to your model.
So I did run minimize.py
, and train.py
But Loss is 0 in step 200 and assertion error in eval time.
assertion error log
Traceback (most recent call last):
File "train.py", line 64, in <module>
eval_summary, eval_f1 = model.evaluate(session, tf_global_step)
File "/data/BERT-coref/coref/independent.py", line 559, in evaluate
coref_predictions[example["doc_key"]] = self.evaluate_coref(top_span_starts, top_span_ends, predicted_antecedents, example["clusters"], coref_evaluator)
File "/data/BERT-coref/coref/independent.py", line 524, in evaluate_coref
predicted_clusters, mention_to_predicted = self.get_predicted_clusters(top_span_starts, top_span_ends, predicted_antecedents)
File "/data/BERT-coref/coref/independent.py", line 499, in get_predicted_clusters
assert i > predicted_index, (i, predicted_index)
AssertionError: (0, 0)
It's my experiments.conf
train_kor = ${best}{
num_docs = 2411
bert_learning_rate = 1e-05
task_learning_rate = 0.0002
max_segment_len = 128
ffnn_size = 800
train_path = ${data_dir}/train.kor.128.jsonlines
eval_path = ${data_dir}/dev.kor.128.jsonlines
conll_eval_path = ${data_dir}/dev.kor.v4_gold_conll
max_training_sentences = 8
bert_config_file = ${best.log_root}/multi_cased_L-12_H-768_A-12/bert_config.json
vocab_file = ${best.log_root}/multi_cased_L-12_H-768_A-12/vocab.txt
tf_checkpoint = ${best.log_root}/multi_cased_L-12_H-768_A-12/bert_model.ckpt
init_checkpoint = ${best.log_root}/multi_cased_L-12_H-768_A-12/bert_model.ckpt
}
I changed the BERT model to the multi_cased.
I editted num_docs to 2411.
(It's result of minimize.py. Is this correct?)
and default ffnn_size and max_training_sentences
cause memory error.
so I editted ffnn_size and max_training_sentences.
Is there anything I miss?
Hi!
I see you beated state of the art on coreference resolution (which is a fundamental NLP task!!) by using and tuning BERT.
Yet, BERT has been "obscoleted" by XLnet.
So would you like to be the first to use and fine tune it for coreference resolution just like you did with BERT? That would be very helpful for all those people that needs highly accurate coref.
https://github.com/zihangdai/xlnet
BTW I suggested the same idea to a researcher in dependency/constituency parsing a few weeks ago and they are now number 1 on the paperswithcode.com leaderboards!
Hi,
I am going to restore your checkpoint and finetune the model on a new dataset, but I failed to do this and received the following errors:
Traceback (most recent call last):
File “train.py”, line 23, in
model = util.get_model(config)
File “/home/chaiha/bert-e2e-coref/util.py”, line 21, in get_model
return independent.CorefModel(config)
File “/home/chaiha/bert-e2e-coref/independent.py”, line 61, in init
init_from_checkpoint(config[‘init_checkpoint’], assignment_map)
File “/home/chaiha/anaconda3/envs/e2e-gpu/lib/python3.7/site-packages/tensorflow/python/training/checkpoint_utils.py”, line 190, in init_from_checkpoint
_init_from_checkpoint, args=(ckpt_dir_or_file, assignment_map))
File “/home/chaiha/anaconda3/envs/e2e-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py”, line 1516, in merge_call
return self._merge_call(merge_fn, args, kwargs)
File “/home/chaiha/anaconda3/envs/e2e-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py”, line 1524, in _merge_call
return merge_fn(self._distribution_strategy, *args, **kwargs)
File “/home/chaiha/anaconda3/envs/e2e-gpu/lib/python3.7/site-packages/tensorflow/python/training/checkpoint_utils.py”, line 229, in _init_from_checkpoint
tensor_name_in_ckpt, str(variable_map[tensor_name_in_ckpt])
ValueError: Shape of variable coref_layer/slow_antecedent_scores/hidden_bias_0:0 ((1000,)) doesn’t match with shape of tensor coref_layer/slow_antecedent_scores/hidden_bias_0 ([3000]) from checkpoint reader.
Here is my configuration:
finetune_bert_base = ${best}{
num_docs = 2802
bert_learning_rate = 1e-05
task_learning_rate = 0.0002
max_segment_len = 128
ffnn_size = 3000
train_path = ${data_dir}/train.english.128.jsonlines
eval_path = ${data_dir}/test.english.128.jsonlines
conll_eval_path = ${data_dir}/test.english.v4_gold_conll
max_training_sentences = 11
bert_config_file = ${best.log_root}/bert_base/bert_config.json
vocab_file = ${best.log_root}/bert_base/vocab.txt
tf_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
init_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
}
I also made a copy of 'bert_base' folder into the folder called 'finetune_bert_base' and that still fails... Thank you for any thoughts!
For pretrained models, it says "speanbert_base" instead of "spanbert_base".
This was slightly confusing when using the download.sh script, as I could not tell if it was an error w/ the script config or something else.
Hi,
First, thank you for providing this valuable resource.
According to Table 4 of the Bert paper, for long sequences with length 1152+ the performance declined.
I wonder if I want to do the coref for my dataset in which average sequence length is 1500+, do you suggest using 'spanbert' on my data as it is. Or it is better to segment the data into pieces of length 512?
Of course both has it's drawbacks in negatively effecting the performance of pertained model but which approach do you suggest?
Hi Mandar ,
While running spanbert large model , if is not able to capture some of words such as Degree .
The same text if I use other brand like Kurkure it will detect . Can I train coref model with my own dataset .
How do I do it ? I have access to onto notes 5. I'm new in the field on AI . Should I convert my dataset to ontonotes format? How can I train on custom dataset.
command : ./download_pretrained.sh bert-base
error : HTTP request sent, awaiting response... 404 Not Found
command : GPU=0 python train.py best
error :
Traceback (most recent call last):
File "train.py", line 22, in
model = util.get_model(config)
File "/data/BERT-coref/coref/util.py", line 21, in get_model
return independent.CorefModel(config)
File "/data/BERT-coref/coref/independent.py", line 32, in init
self.bert_config = modeling.BertConfig.from_json_file(config["bert_config_file"])
File "/home/fairy_of_9/anaconda3/envs/bert/lib/python3.6/site-packages/pyhocon/config_tree.py", line 366, in getitem
val = self.get(item)
File "/home/fairy_of_9/anaconda3/envs/bert/lib/python3.6/site-packages/pyhocon/config_tree.py", line 209, in get
return self._get(ConfigTree.parse_key(key), 0, default)
File "/home/fairy_of_9/anaconda3/envs/bert/lib/python3.6/site-packages/pyhocon/config_tree.py", line 151, in _get
raise ConfigMissingException(u"No configuration setting found for key {key}".format(key='.'.join(key_path[:key_index + 1])))
pyhocon.exceptions.ConfigMissingException: 'No configuration setting found for key bert_config_file'
best {
# Edit this
data_dir = data_set
model_type = independent
# Computation limits.
max_top_antecedents = 50
max_training_sentences = 5
top_span_ratio = 0.4
max_num_speakers = 20
max_segment_len = 64 #256
# Learning
bert_learning_rate = 1e-5
task_learning_rate = 2e-4
num_docs = 2802
# Model hyperparameters.
dropout_rate = 0.3
ffnn_size = 500 #1000
ffnn_depth = 1
num_epochs = 20
feature_size = 20
max_span_width = 30
use_metadata = true
use_features = true
use_segment_distance = true
model_heads = false #true
coref_depth = 2
coarse_to_fine = true
fine_grained = true
use_prior = true
# Other.
train_path = data_set/train.english.jsonlines
eval_path = data_set/dev.english.jsonlines
conll_eval_path = data_set/dev.english.v4_gold_conll
single_example = true
genres = ["bc", "bn", "mz", "nw", "pt", "tc", "wb"]
eval_frequency = 1000
report_frequency = 100
log_root = logs
adam_eps = 1e-6
task_optimizer = adam
}
bert_base = ${best}{
num_docs = 2802
bert_learning_rate = 1e-05
task_learning_rate = 0.0002
max_segment_len = 128
ffnn_size = 3000
train_path = data_set/train.english.128.jsonlines
eval_path = data_set/dev.english.128.jsonlines
conll_eval_path = data_set/dev.english.v4_gold_conll
max_training_sentences = 11
bert_config_file = ${best.log_root}/bert_base/bert_config.json
vocab_file = ${best.log_root}/bert_base/vocab.txt
tf_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
init_checkpoint = ${best.log_root}/bert_base/model.max.ckpt
}
...
Is there anything I missed?
BERT will divide word into many subwords, do you make them together in some way to a entire representation, or leave the cluster larger and predict more tokens? And where are the codes in this rp? It bother me for a long time ......
All the links won't open, is the server crashed
Hi, thanks for the implementation. I'm trying to train my first model but got this error:
$ GPU=0 python train.py train_spanbert_base
...
tensorflow.python.framework.errors_impl.NotFoundError: ../output/bert/data/spanbert_base/bert_config.json; No such file or directory
../output/bert/data
is an empty folder before I ran setup_all.sh
and setup_training.sh
. Am I missing some step here?
I need results of a coreference resolution consisting of an original index (not bert token index).
kentonl's jsonlines_to_json.py
don't work, because of indices change by bert tokenizer.
Do you have a jsonlines_to_json.py
for bert coreference?
or
Do you have an idea about this problem?
Thanks!
I simply want to use state of the art coreference resolution.
it should take an arbitrary input text made of valid english and output the resolutions of pronouns.
How can I achieve this basic usage ?
I tried both evaluate and predict
GPU=0 python evaluate.py spanbert_base give =>
...
UNKNOWN ERROR (303)
2020-06-02 15:17:49.112473: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (stephane): /proc/driver/nvidia/version does not exist
2020-06-02 15:17:49.112826: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-06-02 15:17:49.143598: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2901210000 Hz
2020-06-02 15:17:49.143965: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x565419b6ea50 executing computations on platform Host. Devices:
2020-06-02 15:17:49.144000: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
Restoring from /home/stephane/b_data/spanbert_base/model.max.ckpt
2020-06-02 15:17:55.723471: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
W0602 15:17:58.632378 140016442066752 deprecation.py:323] From /home/stephane/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "evaluate.py", line 26, in
model.evaluate(session, official_stdout=True, eval_mode=True)
File "/home/stephane/coref/independent.py", line 538, in evaluate
self.load_eval_data()
File "/home/stephane/coref/independent.py", line 532, in load_eval_data
with open(self.config["eval_path"]) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/stephane/b_data/dev.english.384.jsonlines'
Could I ask where I can find cased_config_vocab directory, please?
Hi,
Thanks for your opening your source code!
In your repo, spanbert is finetuned on OntoNotes (http://nlp.cs.washington.edu/pair2vec/spanbert_base.tar.gz) while you said "The non-finetuned version of SpanBERT weights will be released soon." in readme.
In https://github.com/facebookresearch/SpanBERT, there is also a spanbert model. Is the model in facebookresearch's repo finetuned on OntoNotes? Or it is the non-finetuned version?
Thanks,
Deming Ye
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.