benkrause / dynamic-evaluation Goto Github PK
View Code? Open in Web Editor NEWDynamic evaluation for pytorch language models, now includes hyperparameter tuning
License: BSD 2-Clause "Simplified" License
Dynamic evaluation for pytorch language models, now includes hyperparameter tuning
License: BSD 2-Clause "Simplified" License
Hi,
Thanks for sharing this work.
I tried to reproduce the PTB results (the first two bold rows in Table 1):
architecture | val | test |
---|---|---|
LSTM(ours) | 88.0 | 85.0 |
LSTM+dynamic eval | 73.5 | 71.7 |
However, I am getting results:
python main.py --cuda --tied #valid ppl 90.69 and test ppl 87.08 python dynamiceval.py --model model.pt --val #perplexity of 79.7848 python dynamiceval.py --model model.pt #perplexity of 78.9188
where main.py (the baseline) is from https://github.com/pytorch/examples/tree/master/word_language_model, and
dynamiceval.py is from this repository.
There is over 13 ppl difference (from 85.0 to 71.7) on the test data reported in the paper. However, my results show that there is 7.3 ppl difference (from 87.08 to 79.7848).
Did I do anything wrong?
Thanks,
Hi there,
Has anyone tried to update this code for pytorch 0.4? awd-lstm-lm
repo recently upgraded to make the models work in 0.4.
I made some attempts but got stuck at gradstat()
, since in this method the code calls model.eval()
, I encountered the following error while running with a trained model. See code here
Traceback (most recent call last):
File "dynamiceval.py", line 299, in <module>
gradstat(args, corpus, model, train_data, criterion)
File "dynamiceval.py", line 83, in gradstat
loss.backward()
File "/usr/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 89, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: backward_input can only be called in training mode
Looks like I cannot call loss.backward()
when the model is set to eval
mode. I've not tried running in with previous versions of pytorch. Is there a workaround for this in version 0.4
?
Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.