neubig / lamtram Goto Github PK

hi,
I am trying to use function average(), but it seems something wrong with it, my input is a list of expression which shape remains the same. c++ version ~
thanks ,neubig~

Good morning Graham,

Actually, I don't have an issue to report. I just wanted to say thank you for your research und providing this system.

I've followed your instructions in nmt tips and I'm currently setting up an English-German NMT system based on Europarl and News Commentary. I'm using a Google VM with 8 vCPUs, 52 GB Ram and 1TB hard disk. I started the training process yesterday at ppl ~1700. I'm still at epoch 1 and the ppl has dropped to ~360.

This is the command I used after pre-processing the corpus:

lamtram/src/lamtram/lamtram-train --cnn-mem 30000 --model_type encatt --train_src /home/mehmedes777/data/train.unk.en --train_trg /home/mehmedes777/data/train.unk.de --dev_src /home/mehmedes777/data/dev.en --dev_trg /home/mehmedes777/data/dev.de --attention_lex prior:file=/home/mehmedes777/lexicon/train.ende.prob:alpha=0.0001 --layers lstm:200:1 --trainer adam --learning_rate 0.001 --minibatch_size 256 --rate_decay 1.0 --epochs 10 --motdel_out models/encatt-unk-stop-lex-w200.mod

I'm looking forward to the outcome!

Is there a straightforward way to output something like confidence from this model?
That is, given a sentence, not only translate but return a signal that the given translation is probably right or wrong.

I was trying to use lamtram to train an encoder-decoder model from Chinese to English. However, when completing the first epoch and saving the model to a file, a runtime error will be always thrown out:
terminate called after throwing an instance of 'std::runtime_error'
what(): ERROR: Could not open output file: models/encdec.mod

The command used is:
./src/lamtram/lamtram-train --model_type encdec --train_src data/train.zh --train_trg data/train.en --trainer sgd --learning_rate 0.1 --rate_decay 1.0 --epochs 10 --model_out models/encdec.mod --minibatch_size 80 --dynet_mem 3096

Please help me with this issue. Thanks.

As far as I see, this is mentioned in the dynet tutorial page.
I am not sure though, I tried to follow the instruction there but still failed.

I think it is a good idea to have a page tutorial about using GPU (or, is it already available?)

Many of the models fall back to GlobalVars::layer_size if a value is not specified (mlp:0, multilayer:0:full, etc.). When training models, if the command line flag --layer_size is used, it sets this global value. At test time, GlobalVars::layer_size is never set so defaults to 512 and reading in models crashes.

Please excuse my question but where do I enter the source text for translation:

Decoding

You can also generate the most likely translation for the input sentence. Here is a typical use case.

src/lamtram/lamtram
--operation gen
--models_in encdec=transmodel.out
--beam 5 \ # Perform beam search with a beam of 5
--word_pen 0.0 \ # The word penalty can be used to increase or decrease
# the average length of the sentence (positive for more words)
> results.txt

Should I use src_in ...?

How can I make lamtram resort to multiple cpu cores instead of one?

when I change the layer_size to another value, it gives this error for generating sentences. Any idea? I checked the source code. It seems this is not implemented.

what(): Dimensions of parameter /_3 looked up from file ({128,128}) do not match parameters to be populated ({512,128})

Is it possible to interrupt training and have a model written out before the first iteration has been completed?

here is my error info

here is my train.sh file

And here is my question.
I don't know what's wrong with my program :)

Currently there is no way to generate random sentences from the command line. This would be nice to have for language models, etc.

I used this command
"src/lamtram/lamtram-train --dynet_mem 1024 --model_type encdec --layer_size 6 --trainer adam --learning_rate 0.001 --seed 0 --train_src train-src.unk --train_trg train-trg.unk --dev_src dev-src.txt --dev_trg dev-trg.txt --model_out transmodel.out" for training the translation model ans used below for testing

"src/lamtram/lamtram --dynet_mem 1024 --operation ppl --src_in test_src.txt --models_in encdec=transmodel.out > results.txt"

But, I am not getting any output. Please help

It would be great to shard minibatches across multiple GPUs to speed up training. Any pointers for how I get started? I'm just reading the dynet docs now.

razavi@razavi-G20AJ:/dynet-base/lamtram$ make
Making all in src
make[1]: Entering directory '/home/razavi/dynet-base/lamtram/src'
Making all in lamtram
make[2]: Entering directory '/home/razavi/dynet-base/lamtram/src/lamtram'
make all-am
make[3]: Entering directory '/home/razavi/dynet-base/lamtram/src/lamtram'
depbase=echo lamtram-train.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||';
/bin/bash ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I/usr/include -I/dynet-base/eigen -I~/dynet-base/dynet -I~/usr/local/cuda/include -DHAVE_CUDA -DEIGEN_USE_GPU -DEIGEN_HAS_CUDA_FP16 -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram-train.lo -MD -MP -MF $depbase.Tpo -c -o lamtram-train.lo lamtram-train.cc &&
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I/usr/include "-I~/dynet-base/eigen" "-I~/dynet-base/dynet" "-I~/usr/local/cuda/include" -DHAVE_CUDA -DEIGEN_USE_GPU -DEIGEN_HAS_CUDA_FP16 -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram-train.lo -MD -MP -MF .deps/lamtram-train.Tpo -c lamtram-train.cc -fPIC -DPIC -o .libs/lamtram-train.o
In file included from lamtram-train.cc:2:0:
./../lamtram/lamtram-train.h:4:26: fatal error: dynet/tensor.h: No such file or directory
compilation terminated.
Makefile:657: recipe for target 'lamtram-train.lo' failed
make[3]: *** [lamtram-train.lo] Error 1
make[3]: Leaving directory '/home/razavi/dynet-base/lamtram/src/lamtram'
Makefile:443: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '/home/razavi/dynet-base/lamtram/src/lamtram'
Makefile:369: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/razavi/dynet-base/lamtram/src'
Makefile:400: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

With the most recent version, training dies when using GPUs on the following error:

lt-lamtram-train: /home/gneubig/usr/local/eigen/unsupported/Eigen/CXX11/src/Tensor/TensorExecutor.h:267: static void Eigen::internal::TensorExecutor<Expression, Eigen::GpuDevice, Vectorizable>::run(const Expression&, const Eigen::GpuDevice&) [with Expression = const Eigen::TensorAssignOp<Eigen::TensorChippingOp<1l, Eigen::TensorMap<Eigen::Tensor<float, 2, 0, long int>, 0> >, const Eigen::TensorCwiseBinaryOp<Eigen::internal::scalar_sum_op<float, float>, const Eigen::TensorChippingOp<1l, Eigen::TensorMap<Eigen::Tensor<float, 2, 0, long int>, 0> >, const Eigen::TensorCwiseBinaryOp<cnn::FSoftmaxBackward, const Eigen::TensorChippingOp<1l, const Eigen::TensorMap<Eigen::Tensor<float, 2, 0, long int>, 0> >, const Eigen::TensorChippingOp<1l, const Eigen::TensorMap<Eigen::Tensor<float, 2, 0, long int>, 0> > > > >; bool Vectorizable = false]: Assertion `cudaGetLastError() == cudaSuccess' failed.

This seems to happen particularly when using mini-batching, and lamtram hits a wide mini-batch with a large number of sentences in the mini-batch.

In the README section:

$ export LD_LIBARY_PATH="$DYNET_DIR/build/dynet:$LD_LIBRARY_PATH"

I think it should be "LD_LIBRARY_PATH"

Hi Graham,

I like lamtram! Thanks for building this toolkit! I was using it before cnn is updated to dynet, but now I have a question: the readme says that I should use v2 branch of dynet, however, it seems that dynet doesn't have a v2 branch anymore. Am I supposed to use the master branch of it? Thank you!

Best,
Yuchen

Following to the clab/dynet#695 , trainer->update_epoch() in src/lamtram/lamtram-train.cc should be removed.

Is it ok to simply insert the following line?

trainer->learning_rate /= (1 - rate_decay_)

What is the difference between trainer->learning_rate and learning_rate declared at https://github.com/neubig/lamtram/blob/master/src/lamtram/lamtram-train.cc#L929 ?

I feel puzzled when reading lamtram code , so is there any detailed annotations when lamtram will be updated next time?

Hello,

I am trying to compile the lamtram and when doing the make, the following error is logged:

Making all in src
make[1]: Entering directory '.../lamtram/src'
Making all in lamtram
make[2]: Entering directory '.../lamtram/src/lamtram'
make  all-am
make[3]: Entering directory '.../lamtram/src/lamtram'
depbase=`echo lamtram-train.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/bash ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.    -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram-train.lo -MD -MP -MF $depbase.Tpo -c -o lamtram-train.lo lamtram-train.cc &&\
mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram-train.lo -MD -MP -MF .deps/lamtram-train.Tpo -c lamtram-train.cc  -fPIC -DPIC -o .libs/lamtram-train.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram-train.lo -MD -MP -MF .deps/lamtram-train.Tpo -c lamtram-train.cc -o lamtram-train.o >/dev/null 2>&1
depbase=`echo lamtram.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/bash ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.    -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram.lo -MD -MP -MF $depbase.Tpo -c -o lamtram.lo lamtram.cc &&\
mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram.lo -MD -MP -MF .deps/lamtram.Tpo -c lamtram.cc  -fPIC -DPIC -o .libs/lamtram.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram.lo -MD -MP -MF .deps/lamtram.Tpo -c lamtram.cc -o lamtram.o >/dev/null 2>&1
depbase=`echo ensemble-decoder.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/bash ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.    -I/usr/include -I/.../eigen3 -I/.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT ensemble-decoder.lo -MD -MP -MF $depbase.Tpo -c -o ensemble-decoder.lo ensemble-decoder.cc &&\
mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/include -I.../eigen3 -I.../dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT ensemble-decoder.lo -MD -MP -MF .deps/ensemble-decoder.Tpo -c ensemble-decoder.cc  -fPIC -DPIC -o .libs/ensemble-decoder.o
In file included from ensemble-decoder.cc:1:0:
./../lamtram/ensemble-decoder.h: In function 'bool lamtram::operator<(const EnsembleDecoderHypPtr&, const EnsembleDecoderHypPtr&)':
./../lamtram/ensemble-decoder.h:38:30: error: 'assert' was not declared in this scope
   assert(lhs.get() != nullptr);
                              ^
Makefile:645: recipe for target 'ensemble-decoder.lo' failed
make[3]: *** [ensemble-decoder.lo] Error 1
make[3]: Leaving directory '.../lamtram/src/lamtram'
Makefile:430: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '.../lamtram/src/lamtram'
Makefile:357: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '.../lamtram/src'
Makefile:388: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

Note: I have removed the full path just simplicity, so the ... implies full path.

Eigen, DyNet and other dependencies are installed correctly so that does not seems to be the issue.

hi,I am using lamtram recently and I found something puzzling me. When calculating loss of batched sentence, I change the code and add loss.value() it runs ok , what's the meaning of .value()? but the program shut down if I don't add .value symbol.

A clean compile from the latest commit (cc05380) gives the following compilation error with GCC 4.9.2 on Debian stable:

[...]
mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/include -I/home/myhome/src/eigen -I/home/myhome/src/dynet/ -fopenmp -I./.. -g -O2 -std=c++0x -MT lamtram-train.lo -MD -MP -MF .deps/lamtram-train.Tpo -c lamtram-train.cc  -fPIC -DPIC -o .libs/lamtram-train.o
lamtram-train.cc: In member function 'lamtram::LamtramTrain::TrainerPtr lamtram::LamtramTrain::GetTrainer(const string&, dynet::real, dynet::Model&)':
lamtram-train.cc:1069:12: error: 'struct dynet::Trainer' has no member named 'sparse_updates_enabled'
   trainer->sparse_updates_enabled = false;
            ^

While generating sentences , lamtram cost a lot of memory. If beam-size is setting to 100, out of memory occurs , and memory increases with larger test data . Any solutions to my problem ?