rloganiv / kglm-model Goto Github PK

Hi, Thanks for the great work!
I want to try the inference step by the subgraphs I made.
However, I'm not sure if the code for the inference step is provided.
Am I missing the code? If not, do you have any plan of releasing it?

Hi, I wanted to annotate some data using the same format as you annotated linked-wikitext 2 dataset. Can you share the entity linker and the Stanford CoreNLP coreference code which was used to create that dataset?

Docstrings for models/modules are currently not very descriptive, and often incomplete or not up-to-date.

Hello, I would like to ask where your [path to config], I use json file which under the folder of "kglm-model/kglm/tests/fixtures/training_config/ ", but it seem to go wrong with " RuntimeError: Expected object of scalar type Bool but got scalar type Byte for argument #2 'other' ".

Hi,

Thanks for the awesome work. I have a small question about evaluation:

In the following command, I only have model_archive_file after training, how can I get a sampler_archive_file?

python -m kglm.run evaluate-perplexity \ [model_archive_file] \ [sampler_archive_file] \ [input_file]

The error showed "cloze is not a registered Predictor". I can't find any information related to "cloze", this predictor might be written by the author. I only find two files related to predictor in their /kglm/predictor folder. One is called complete-the-sentence and the other is called kglm.

I tried to replace "cloze" to "complete-the-sentence" and "kglm", but there was a new error "json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1". Now I'm stuck at this step.

Hi, I am quite interested in the fact completion task in your paper. Can you provide the data that you used for this particular experiment? Thanks!

Thanks for the author's great work.
When I trained the kglm model, the validation evaluation always failed.
I use the following config:

{
"dataset_reader": {
"type": "enhanced-wikitext-kglm",
"alias_database_path": "data/linked-wikitext-2/alias.pkl"
},
"iterator": {
"type": "fancy",
"batch_size": 128,
"split_size": 70,
"splitting_keys": [
"source",
"target",
"mention_type",
"raw_entity_ids",
"entity_ids",
"parent_ids",
"relations",
"shortlist_inds",
"alias_copy_inds"
]
},
"model": {
"type": "kglm",
"alias_encoder": {
"type": "lstm",
"hidden_size": 400,
"input_size": 400
},
"cutoff": 30,
"entity_embedder": {
"token_embedders": {
"entity_ids": {
"type": "embedding",
"embedding_dim": 256,
"pretrained_file": "data/linked-wikitext-2/embeddings.entities.txt",
"trainable": false,
"vocab_namespace": "entity_ids"
}
}
},
"hidden_size": 1150,
"initializer": [
[
"token_embedder.weight",
{
"a": -0.1,
"b": 0.1,
"type": "uniform"
}
],
[
"decoder.bias",
{
"type": "constant",
"val": 0
}
]
],
"knowledge_graph_path": "data/linked-wikitext-2/knowledge_graph.pkl",
"num_layers": 3,
"relation_embedder": {
"token_embedders": {
"relations": {
"type": "embedding",
"embedding_dim": 256,
"pretrained_file": "data/linked-wikitext-2/embeddings.relations.txt",
"trainable": true,
"vocab_namespace": "relations"
}
}
},
"tie_weights": true,
"token_embedder": {
"token_embedders": {
"tokens": {
"type": "embedding",
"embedding_dim": 400,
"trainable": true
}
}
},
"use_shortlist": false
},
"train_data_path": "data/linked-wikitext-2/train.jsonl",
"validation_data_path": "data/linked-wikitext-2/valid.jsonl",
"test_data_path": "data/linked-wikitext-2/test.jsonl",
"trainer": {
"type": "lm",
"cuda_device": [
2,
3
],
"num_epochs": 1,
"optimizer": {
"type": "adam",
"lr": 0.0003,
"weight_decay": 1.2e-06
},
"patience": 10,
"validation_metric": "-ppl"
},
"vocabulary": {
"type": "extended",
"directory_path": "data/linked-wikitext-2/vocab",
"extend": false
},
"evaluate_on_test": true,
"validation_dataset_reader": {
"type": "enhanced-wikitext-kglm",
"alias_database_path": "data/linked-wikitext-2/alias.pkl"
},
"validation_iterator": {
"type": "fancy",
"batch_size": 128,
"split_size": 70,
"splitting_keys": [
"source",
"target",
"mention_type",
"raw_entity_ids",
"entity_ids",
"parent_ids",
"relations",
"shortlist_inds",
"alias_copy_inds"
]
}
}

but the evaluation on validation set after each epoch always failed. The metric result as follows:

{
"best_epoch": 0,
"peak_cpu_memory_MB": 8197.112,
"peak_gpu_0_memory_MB": 10,
"peak_gpu_1_memory_MB": 10,
"peak_gpu_2_memory_MB": 957,
"peak_gpu_3_memory_MB": 10,
"training_duration": "00:20:00",
"training_start_epoch": 0,
"training_epochs": 0,
"epoch": 0,
"training_ppl": 995.1405480875865,
"training_upp": 1254.7674796458832,
"training_kg_ppl": 193.04238067628987,
"training_bg_ppl": 1197.0806547174739,
"training_type": 0.46333958114065776,
"training_new": 1.0831616436562888,
"training_kg": 0.329283076343013,
"training_vocab": 0,
"training_new_p": 0.1187648456057007,
"training_new_r": 0.0007927447995941146,
"training_new_f1": 0.0015749767690913384,
"training_kg_p": 0.0,
"training_kg_r": 0.0,
"training_kg_f1": 0.0,
"training_new_ent_acc": 0.0,
"training_new_ent_acc_20": 0.0,
"training_parent_ppl": 4.633376004674751,
"training_relation_ppl": 310.0364866986571,
"training_loss": 8.898253054735138,
"training_cpu_memory_MB": 8197.112,
"training_gpu_0_memory_MB": 10,
"training_gpu_1_memory_MB": 10,
"training_gpu_2_memory_MB": 957,
"training_gpu_3_memory_MB": 10,
"validation_ppl": 1.0,
"validation_upp": 1.0,
"validation_kg_ppl": 1.0,
"validation_bg_ppl": 1.0,
"validation_type": 0,
"validation_new": 0,
"validation_kg": 0,
"validation_vocab": 0,
"validation_new_p": 0.0,
"validation_new_r": 0.0,
"validation_new_f1": 0.0,
"validation_kg_p": 0.0,
"validation_kg_r": 0.0,
"validation_kg_f1": 0.0,
"validation_new_ent_acc": 0.0,
"validation_new_ent_acc_20": 0.0,
"validation_parent_ppl": 1.0,
"validation_relation_ppl": 1.0,
"validation_loss": 0.0,
"best_validation_ppl": 1.0,
"best_validation_upp": 1.0,
"best_validation_kg_ppl": 1.0,
"best_validation_bg_ppl": 1.0,
"best_validation_type": 0,
"best_validation_new": 0,
"best_validation_kg": 0,
"best_validation_vocab": 0,
"best_validation_new_p": 0.0,
"best_validation_new_r": 0.0,
"best_validation_new_f1": 0.0,
"best_validation_kg_p": 0.0,
"best_validation_kg_r": 0.0,
"best_validation_kg_f1": 0.0,
"best_validation_new_ent_acc": 0.0,
"best_validation_new_ent_acc_20": 0.0,
"best_validation_parent_ppl": 1.0,
"best_validation_relation_ppl": 1.0,
"best_validation_loss": 0.0
}

So I want to know how to config to get the evaluation on validation set work?

Make use of features added to AllenNLP that address problems encountered when writing this model.

• Replace LmTrainer with CallbackTrainer.
• Use StraightThroughIterator in place of custom iterators. Update dataset readers accordingly.
• Replace SequentialArrayField with ArrayField (which now supports dtype argument).

I was interested in annotating new sentences with the KG. I saw the KG pkl file in this google drive link. However, there is no README or specifications about those files. Can you show me the code which reads these files? or just the specifications of these files?

@rloganiv Nice work.

1. Is there any pre-trained model available? How can I use it for sentence completion task.

I tried using the model in test folder but then I get this error :

E:\GitHub\kglm-model>python -m kglm.run predict --predictor cloze kglm\tests\fixtures\kglm.model.tar.gz input.txt
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
D:\Programs\Python\Python37\lib\site-packages\sklearn\utils\linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  DeprecationWarning)
2019-10-23 18:10:59,191 - INFO - allennlp.models.archival - loading archive file kglm\tests\fixtures\kglm.model.tar.gz
2019-10-23 18:10:59,194 - INFO - allennlp.models.archival - extracting archive file kglm\tests\fixtures\kglm.model.tar.gz to temp dir C:\Users\anand\AppData\Local\Temp\tmphjgljrs7
2019-10-23 18:10:59,216 - WARNING - allennlp.common.params - _jsonnet not loaded, treating C:\Users\anand\AppData\Local\Temp\tmphjgljrs7\config.json as json
Traceback (most recent call last):
  File "D:\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "E:\GitHub\kglm-model\kglm\run.py", line 24, in <module>
    'complete-the-sentence': CompleteTheSentence(),
  File "D:\Programs\Python\Python37\lib\site-packages\allennlp\commands\__init__.py", line 101, in main
    args.func(args)
  File "D:\Programs\Python\Python37\lib\site-packages\allennlp\commands\predict.py", line 187, in _predict
    predictor = _get_predictor(args)
  File "D:\Programs\Python\Python37\lib\site-packages\allennlp\commands\predict.py", line 99, in _get_predictor
    overrides=args.overrides)
  File "D:\Programs\Python\Python37\lib\site-packages\allennlp\models\archival.py", line 156, in load_archive
    cuda_device=cuda_device)
  File "D:\Programs\Python\Python37\lib\site-packages\allennlp\models\model.py", line 322, in load
    return cls.by_name(model_type)._load(config, serialization_dir, weights_file, cuda_device)
  File "D:\Programs\Python\Python37\lib\site-packages\allennlp\common\registrable.py", line 58, in by_name
    raise ConfigurationError("%s is not a registered name for %s" % (name, cls.__name__))
allennlp.common.checks.ConfigurationError: 'kglm is not a registered name for Model'
2019-10-23 18:10:59,257 - INFO - allennlp.models.archival - removing temporary unarchived model dir at C:\Users\anand\AppData\Local\Temp\tmphjgljrs7

2. Error in training:

(env) λ allennlp train E:\GitHub\kglm-model\experiments\kglm.jsonnet -s E:\GitHub\kglm-model\experiments\checkpoint --include-package kglm
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
e:\github\kglm-model\env\lib\site-packages\sklearn\utils\linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  DeprecationWarning)
2019-10-23 18:46:47,867 - WARNING - allennlp.common.params - _jsonnet not loaded, treating E:\GitHub\kglm-model\experiments\kglm.jsonnet as json
Traceback (most recent call last):
  File "E:\GitHub\kglm-model\env\Scripts\allennlp-script.py", line 11, in <module>
    load_entry_point('allennlp===0.8.2-unreleased', 'console_scripts', 'allennlp')()
  File "e:\github\kglm-model\env\lib\site-packages\allennlp\run.py", line 18, in run
    main(prog="allennlp")
  File "e:\github\kglm-model\env\lib\site-packages\allennlp\commands\__init__.py", line 101, in main
    args.func(args)
  File "e:\github\kglm-model\env\lib\site-packages\allennlp\commands\train.py", line 103, in train_model_from_args
    args.force)
  File "e:\github\kglm-model\env\lib\site-packages\allennlp\commands\train.py", line 135, in train_model_from_file
    params = Params.from_file(parameter_filename, overrides)
  File "e:\github\kglm-model\env\lib\site-packages\allennlp\common\params.py", line 442, in from_file
    file_dict = json.loads(evaluate_file(params_file, ext_vars=ext_vars))
  File "D:\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "D:\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "D:\Programs\Python\Python37\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 99 column 9 (char 2996)

The sampling code is currently very difficult to understand, move to a step-by-step approach.

Hi, thanks for open-sourcing the code and release the preprocessed data.

yet i find something strange in the released data. The @@EnD@@ sometimes appears in unreasonable places.

For example, in valid.jsonl, I found a case:

... Queen Elizabeth II and Prince Philip were guests of honor , making it the second Bond film premiere to be attended by the Queen , the first one being You @@EnD@@ Only Live Twice in 1967 ...

please note the @@EnD@@ is the middle of You Only Live Twice, which is an entity span.

and the entity is annotated with {'source': 'WIKI', 'id': 'Q107761', 'relation': ['@@NEW@@'], 'parent_id': ['Q107761'], 'span': [2281, 2286]}, resulting in an entity span You @@EnD@@ Only Live Twice according to the span indices.

there are many cases like the above.

I would like to thank you @rloganiv for such great work. I am quite new in the area and intend to apply your model for my data. I did implement the model successfully but didn't manage to apply it for my data.
Is there any guideline showing how to start or how I can use your model on my data? How can your pre-trained model be used for my data? I would appreciate if you can provide an example of usage on any sentence/text.

Thank you