rloganiv / kglm-model Goto Github PK
View Code? Open in Web Editor NEWCode for KGLM paper
Code for KGLM paper
Hello, I would like to ask where your [path to config], I use json file which under the folder of "kglm-model/kglm/tests/fixtures/training_config/ ", but it seem to go wrong with " RuntimeError: Expected object of scalar type Bool but got scalar type Byte for argument #2 'other' ".
I was interested in annotating new sentences with the KG. I saw the KG pkl file in this google drive link. However, there is no README or specifications about those files. Can you show me the code which reads these files? or just the specifications of these files?
Thanks for the author's great work.
When I trained the kglm model, the validation evaluation always failed.
I use the following config:
{
"dataset_reader": {
"type": "enhanced-wikitext-kglm",
"alias_database_path": "data/linked-wikitext-2/alias.pkl"
},
"iterator": {
"type": "fancy",
"batch_size": 128,
"split_size": 70,
"splitting_keys": [
"source",
"target",
"mention_type",
"raw_entity_ids",
"entity_ids",
"parent_ids",
"relations",
"shortlist_inds",
"alias_copy_inds"
]
},
"model": {
"type": "kglm",
"alias_encoder": {
"type": "lstm",
"hidden_size": 400,
"input_size": 400
},
"cutoff": 30,
"entity_embedder": {
"token_embedders": {
"entity_ids": {
"type": "embedding",
"embedding_dim": 256,
"pretrained_file": "data/linked-wikitext-2/embeddings.entities.txt",
"trainable": false,
"vocab_namespace": "entity_ids"
}
}
},
"hidden_size": 1150,
"initializer": [
[
"token_embedder.weight",
{
"a": -0.1,
"b": 0.1,
"type": "uniform"
}
],
[
"decoder.bias",
{
"type": "constant",
"val": 0
}
]
],
"knowledge_graph_path": "data/linked-wikitext-2/knowledge_graph.pkl",
"num_layers": 3,
"relation_embedder": {
"token_embedders": {
"relations": {
"type": "embedding",
"embedding_dim": 256,
"pretrained_file": "data/linked-wikitext-2/embeddings.relations.txt",
"trainable": true,
"vocab_namespace": "relations"
}
}
},
"tie_weights": true,
"token_embedder": {
"token_embedders": {
"tokens": {
"type": "embedding",
"embedding_dim": 400,
"trainable": true
}
}
},
"use_shortlist": false
},
"train_data_path": "data/linked-wikitext-2/train.jsonl",
"validation_data_path": "data/linked-wikitext-2/valid.jsonl",
"test_data_path": "data/linked-wikitext-2/test.jsonl",
"trainer": {
"type": "lm",
"cuda_device": [
2,
3
],
"num_epochs": 1,
"optimizer": {
"type": "adam",
"lr": 0.0003,
"weight_decay": 1.2e-06
},
"patience": 10,
"validation_metric": "-ppl"
},
"vocabulary": {
"type": "extended",
"directory_path": "data/linked-wikitext-2/vocab",
"extend": false
},
"evaluate_on_test": true,
"validation_dataset_reader": {
"type": "enhanced-wikitext-kglm",
"alias_database_path": "data/linked-wikitext-2/alias.pkl"
},
"validation_iterator": {
"type": "fancy",
"batch_size": 128,
"split_size": 70,
"splitting_keys": [
"source",
"target",
"mention_type",
"raw_entity_ids",
"entity_ids",
"parent_ids",
"relations",
"shortlist_inds",
"alias_copy_inds"
]
}
}
but the evaluation on validation set after each epoch always failed. The metric result as follows:
{
"best_epoch": 0,
"peak_cpu_memory_MB": 8197.112,
"peak_gpu_0_memory_MB": 10,
"peak_gpu_1_memory_MB": 10,
"peak_gpu_2_memory_MB": 957,
"peak_gpu_3_memory_MB": 10,
"training_duration": "00:20:00",
"training_start_epoch": 0,
"training_epochs": 0,
"epoch": 0,
"training_ppl": 995.1405480875865,
"training_upp": 1254.7674796458832,
"training_kg_ppl": 193.04238067628987,
"training_bg_ppl": 1197.0806547174739,
"training_type": 0.46333958114065776,
"training_new": 1.0831616436562888,
"training_kg": 0.329283076343013,
"training_vocab": 0,
"training_new_p": 0.1187648456057007,
"training_new_r": 0.0007927447995941146,
"training_new_f1": 0.0015749767690913384,
"training_kg_p": 0.0,
"training_kg_r": 0.0,
"training_kg_f1": 0.0,
"training_new_ent_acc": 0.0,
"training_new_ent_acc_20": 0.0,
"training_parent_ppl": 4.633376004674751,
"training_relation_ppl": 310.0364866986571,
"training_loss": 8.898253054735138,
"training_cpu_memory_MB": 8197.112,
"training_gpu_0_memory_MB": 10,
"training_gpu_1_memory_MB": 10,
"training_gpu_2_memory_MB": 957,
"training_gpu_3_memory_MB": 10,
"validation_ppl": 1.0,
"validation_upp": 1.0,
"validation_kg_ppl": 1.0,
"validation_bg_ppl": 1.0,
"validation_type": 0,
"validation_new": 0,
"validation_kg": 0,
"validation_vocab": 0,
"validation_new_p": 0.0,
"validation_new_r": 0.0,
"validation_new_f1": 0.0,
"validation_kg_p": 0.0,
"validation_kg_r": 0.0,
"validation_kg_f1": 0.0,
"validation_new_ent_acc": 0.0,
"validation_new_ent_acc_20": 0.0,
"validation_parent_ppl": 1.0,
"validation_relation_ppl": 1.0,
"validation_loss": 0.0,
"best_validation_ppl": 1.0,
"best_validation_upp": 1.0,
"best_validation_kg_ppl": 1.0,
"best_validation_bg_ppl": 1.0,
"best_validation_type": 0,
"best_validation_new": 0,
"best_validation_kg": 0,
"best_validation_vocab": 0,
"best_validation_new_p": 0.0,
"best_validation_new_r": 0.0,
"best_validation_new_f1": 0.0,
"best_validation_kg_p": 0.0,
"best_validation_kg_r": 0.0,
"best_validation_kg_f1": 0.0,
"best_validation_new_ent_acc": 0.0,
"best_validation_new_ent_acc_20": 0.0,
"best_validation_parent_ppl": 1.0,
"best_validation_relation_ppl": 1.0,
"best_validation_loss": 0.0
}
So I want to know how to config to get the evaluation on validation set work?
I would like to thank you @rloganiv for such great work. I am quite new in the area and intend to apply your model for my data. I did implement the model successfully but didn't manage to apply it for my data.
Is there any guideline showing how to start or how I can use your model on my data? How can your pre-trained model be used for my data? I would appreciate if you can provide an example of usage on any sentence/text.
Thank you
Hi, I am quite interested in the fact completion task in your paper. Can you provide the data that you used for this particular experiment? Thanks!
Hi, Thanks for the great work!
I want to try the inference step by the subgraphs I made.
However, I'm not sure if the code for the inference step is provided.
Am I missing the code? If not, do you have any plan of releasing it?
Hi, thanks for open-sourcing the code and release the preprocessed data.
yet i find something strange in the released data. The @@EnD@@ sometimes appears in unreasonable places.
For example, in valid.jsonl
, I found a case:
... Queen Elizabeth II and Prince Philip were guests of honor , making it the second Bond film premiere to be attended by the Queen , the first one being You @@EnD@@ Only Live Twice in 1967 ...
please note the @@EnD@@ is the middle of You Only Live Twice, which is an entity span.
and the entity is annotated with {'source': 'WIKI', 'id': 'Q107761', 'relation': ['@@NEW@@'], 'parent_id': ['Q107761'], 'span': [2281, 2286]}
, resulting in an entity span You @@EnD@@ Only Live Twice according to the span indices.
there are many cases like the above.
Docstrings for models/modules are currently not very descriptive, and often incomplete or not up-to-date.
Hi, I wanted to annotate some data using the same format as you annotated linked-wikitext 2 dataset. Can you share the entity linker and the Stanford CoreNLP coreference code which was used to create that dataset?
The sampling code is currently very difficult to understand, move to a step-by-step approach.
Make use of features added to AllenNLP that address problems encountered when writing this model.
LmTrainer
with CallbackTrainer
.StraightThroughIterator
in place of custom iterators. Update dataset readers accordingly.SequentialArrayField
with ArrayField
(which now supports dtype
argument).@rloganiv Nice work.
I tried using the model in test folder but then I get this error :
E:\GitHub\kglm-model>python -m kglm.run predict --predictor cloze kglm\tests\fixtures\kglm.model.tar.gz input.txt
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
D:\Programs\Python\Python37\lib\site-packages\sklearn\utils\linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
DeprecationWarning)
2019-10-23 18:10:59,191 - INFO - allennlp.models.archival - loading archive file kglm\tests\fixtures\kglm.model.tar.gz
2019-10-23 18:10:59,194 - INFO - allennlp.models.archival - extracting archive file kglm\tests\fixtures\kglm.model.tar.gz to temp dir C:\Users\anand\AppData\Local\Temp\tmphjgljrs7
2019-10-23 18:10:59,216 - WARNING - allennlp.common.params - _jsonnet not loaded, treating C:\Users\anand\AppData\Local\Temp\tmphjgljrs7\config.json as json
Traceback (most recent call last):
File "D:\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "D:\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "E:\GitHub\kglm-model\kglm\run.py", line 24, in <module>
'complete-the-sentence': CompleteTheSentence(),
File "D:\Programs\Python\Python37\lib\site-packages\allennlp\commands\__init__.py", line 101, in main
args.func(args)
File "D:\Programs\Python\Python37\lib\site-packages\allennlp\commands\predict.py", line 187, in _predict
predictor = _get_predictor(args)
File "D:\Programs\Python\Python37\lib\site-packages\allennlp\commands\predict.py", line 99, in _get_predictor
overrides=args.overrides)
File "D:\Programs\Python\Python37\lib\site-packages\allennlp\models\archival.py", line 156, in load_archive
cuda_device=cuda_device)
File "D:\Programs\Python\Python37\lib\site-packages\allennlp\models\model.py", line 322, in load
return cls.by_name(model_type)._load(config, serialization_dir, weights_file, cuda_device)
File "D:\Programs\Python\Python37\lib\site-packages\allennlp\common\registrable.py", line 58, in by_name
raise ConfigurationError("%s is not a registered name for %s" % (name, cls.__name__))
allennlp.common.checks.ConfigurationError: 'kglm is not a registered name for Model'
2019-10-23 18:10:59,257 - INFO - allennlp.models.archival - removing temporary unarchived model dir at C:\Users\anand\AppData\Local\Temp\tmphjgljrs7
(env) λ allennlp train E:\GitHub\kglm-model\experiments\kglm.jsonnet -s E:\GitHub\kglm-model\experiments\checkpoint --include-package kglm
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
e:\github\kglm-model\env\lib\site-packages\sklearn\utils\linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
DeprecationWarning)
2019-10-23 18:46:47,867 - WARNING - allennlp.common.params - _jsonnet not loaded, treating E:\GitHub\kglm-model\experiments\kglm.jsonnet as json
Traceback (most recent call last):
File "E:\GitHub\kglm-model\env\Scripts\allennlp-script.py", line 11, in <module>
load_entry_point('allennlp===0.8.2-unreleased', 'console_scripts', 'allennlp')()
File "e:\github\kglm-model\env\lib\site-packages\allennlp\run.py", line 18, in run
main(prog="allennlp")
File "e:\github\kglm-model\env\lib\site-packages\allennlp\commands\__init__.py", line 101, in main
args.func(args)
File "e:\github\kglm-model\env\lib\site-packages\allennlp\commands\train.py", line 103, in train_model_from_args
args.force)
File "e:\github\kglm-model\env\lib\site-packages\allennlp\commands\train.py", line 135, in train_model_from_file
params = Params.from_file(parameter_filename, overrides)
File "e:\github\kglm-model\env\lib\site-packages\allennlp\common\params.py", line 442, in from_file
file_dict = json.loads(evaluate_file(params_file, ext_vars=ext_vars))
File "D:\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "D:\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\Programs\Python\Python37\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 99 column 9 (char 2996)
Hi,
Thanks for the awesome work. I have a small question about evaluation:
In the following command, I only have model_archive_file after training, how can I get a sampler_archive_file?
python -m kglm.run evaluate-perplexity \ [model_archive_file] \ [sampler_archive_file] \ [input_file]
The error showed "cloze is not a registered Predictor". I can't find any information related to "cloze", this predictor might be written by the author. I only find two files related to predictor in their /kglm/predictor folder. One is called complete-the-sentence and the other is called kglm.
I tried to replace "cloze" to "complete-the-sentence" and "kglm", but there was a new error "json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1". Now I'm stuck at this step.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.