alex-berard / seq2seq Goto Github PK
View Code? Open in Web Editor NEWAttention-based sequence to sequence learning
License: Apache License 2.0
Attention-based sequence to sequence learning
License: Apache License 2.0
Hello!
Great work!
But when I train a baseline model on the Example model(WMT14), the termination shows "./seq2seq.sh line 4 4353 killed /usr/bin/env python3 -m translate "$@"". When I try again, the termination shows "./seq2seq.sh line 4 5335 killed /usr/bin/env python3 -m translate "$@"". So I want to know whether other dependencies such as memory or CPU, except python3,YAML and Matplotlib.
According to the Librispeech AST config, a deep layer is used prior to output projection (https://github.com/eske/seq2seq/blob/master/config/LibriSpeech/AST.yaml#L42) and that layer size is 512 (https://github.com/eske/seq2seq/blob/master/config/LibriSpeech/AST.yaml#L45). However, running ./seq2seq.sh config/LibriSpeech/AST.yaml --train -v --purge
shows that the actual deep layer size is 128 (the target embedding size). In https://github.com/eske/seq2seq/blob/master/translate/models.py#L776, decoder.pred_deep_layer_size
is None
so decoder.embedding_size
is used. Would you be able to confirm which size is used for https://github.com/eske/seq2seq/blob/master/config/LibriSpeech/model-outputs.tar.xz ? Thanks in advance.
Hi,
I am trying to do audio in English to text in Russian translation using the Alpha release(0.1) code.
I am working on experiments/btec_speech folder of Alpha release code.
I am giving the audio and text aligned chapter wise (Each chapter contains around 1500 russian words or 50000 MFCC values ).
I have separated the chapters in such a way that 16 chapters are given for train, 2 for dev and 5 for test.
When I generate the MFCC for each set (ie, train,dev or test) I am concatenating them. ie, the First 4 bytes of the train MFCC feature file will be 16, that of dev will be 2 and that of test will be 5.
I am generating the vocab files from russian text.
Please find attached the prepare.sh file that I am using.
I modified baseline-mono.yaml for training the network.
The modifications that I made in baseline-mono.yaml are
I am pasting the modified baseline-mono.yaml file.
I modified max_len field in both encoder and decoder because, the number of words in russian text for 1 chapter or number of MFCC coefficients in one frame (one frame of MFCC coefficients corresponds to one line in Russian text ) are much more than the max_seq_len already given (It was 25 and 600 ). (In function read_dataset() there is a check for max_seq_len )
So In order to avoid the checking of max_seq_len I modified in the init() of class TranslationModel as 'self.max_len=False' instead of 'self.max_len = dict(zip(self.extensions, self.max_input_len + self.max_output_len)) '.
When I try to train the network with these modifications I am getting the below mentioned error.
11/26 15:56:17 files: experiments/btec_speech/data/hbfn.dev.en experiments/btec_speech/data/hbfn.dev.ru
11/26 15:56:18 size: 2
11/26 15:56:31 starting training
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/sharvinv/work/speec_proc/seq2seq-0.1/translate/main.py", line 229, in
main()
File "/home/sharvinv/work/speec_proc/seq2seq-0.1/translate/main.py", line 221, in main
model.train(sess=sess, **config)
File "/home/sharvinv/work/speec_proc/seq2seq-0.1/translate/translation_model.py", line 368, in train
self.train_step(sess=sess, loss_function=loss_function, use_baseline=use_baseline, **kwargs)
File "/home/sharvinv/work/speec_proc/seq2seq-0.1/translate/translation_model.py", line 412, in train_step
update_baseline=True)
File "/home/sharvinv/work/speec_proc/seq2seq-0.1/translate/seq2seq_model.py", line 200, in step
res = session.run(output_feed, input_feed)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1096, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (32, 0) for Tensor 'encoder_en:0', which has shape '(?, ?, 41)'
Could you please tell me whether I am doing it in correct way,or should I make any other modification in order to train the network to translate the English audio to Russian Text.
Thanks and Regards,
Olga Strizhko
baseline-mono.yaml
label: 'baseline-mono'
description: "mono-speaker baseline on BTEC"
dropout_rate: 0.5
cell_size: 256
attn_size: 256
embedding_size: 256
layers: 2
bidir: True
use_lstm: True
weight_scale: null
data_dir: experiments/btec_speech/data
model_dir: experiments/btec_speech/model
batch_size: 64
train_prefix: hbfn.train # 'easy' mono-speaker settings
dev_prefix: hbfn.dev
optimizer: 'adam'
learning_rate: 0.001
steps_per_checkpoint: 1000
steps_per_eval: 1000
max_gradient_norm: 5.0
max_steps: 30000
batch_mode: 'standard'
read_ahead: 10
vocab_prefix: vocab
encoders:
decoders:
prepare.sh
raw_data_dir=data/raw/btec.en-ru
raw_audio_dir=${raw_data_dir}/speech_en
speech_dir=experiments/btec_speech
data_dir=${speech_dir}/data # output directory for the processed files (text and audio features)
mkdir -p ${raw_audio_dir} ${data_dir}
scripts/speech/extract-audio-features.py ${raw_audio_dir}/hbfn_wav16_en/train/* --output ${data_dir}/hbfn.train.en
scripts/speech/extract-audio-features.py ${raw_audio_dir}/hbfn_wav16_en/dev/* --output ${data_dir}/hbfn.dev.en
scripts/speech/extract-audio-features.py ${raw_audio_dir}/hbfn_wav16_en/test/* --output ${data_dir}/hbfn.test.en
scripts/prepare-data.py ${data_dir}/hbfn.train ru ${data_dir} --max 0 --lowercase --output vocab --mode vocab
Hi
I'd like to replicate the experiments of the paper
Aléxandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu and Olivier Pietquin, "End-to-End Automatic Speech Translation of Audiobooks" in submitted to ICASSP 2018
using the LibriSpeech corpus, but the link to download the pre-processed corpus is broken. Please can you let me know where I can find the corpus?
Best Regards
Marco
Hi, Thanks for you good work. I found you code support reinforcement learning. Can you give more detail? What paper you implement?
There seems to be some kind of encoding problem when training a new model (en-fr). I'm pretty sure it's because of the french alphabet (e.g. é, è ...).
To make sure the problem wasn't caused by me, I followed the instructions provided. However I was running the code on a GPU cluster in a docker container. See the files attached for a complete list of apt, pip and pip3 packages available in my container (I provided python 2 and 3, since I wasn't sure if 2 is still needed).
I downloaded the LibriSpeech Data set and used the LibriSpeech AST config file to train a new model, the error occured within 30s after starting training.
HI,sir! Your open source code is very exciting, its features are powerful,I am currently doing end-to-end speech translation tasks(chinese speech to english text),I want to refer to your code.I have a few questions that would like to ask you.1、extract.py and extract-new.py What is the difference between these two codes?What are their respective functions?
I've read the paper LIG-CRIStAL System for the WMT17 Automatic Post-Editing Task. As the paper said, the code for automatic post-editing is at https://github.com/eske/seq2seq/tree/APE. But it seems to be lost...
When a parameter "scope='....' " was there previously, the error message was following:
Traceback (most recent call last):
File "/usr/python354/lib/python3.5/runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "/usr/python354/lib/python3.5/runpy.py", line 85, in run_code
exec(code, run_globals)
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/main.py", line 225, in
main()
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/main.py", line 177, in main
model = MultiTaskModel(name='main', checkpoint_dir=checkpoint_dir, decode_only=decode_only, **config)
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/multitask_model.py", line 26, in init
model = TranslationModel(checkpoint_dir=None, keep_best=keep_best, **kwargs)
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/translation_model.py", line 118, in init
**kwargs)
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/seq2seq_model.py", line 139, in init
**parameters
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/decoders.py", line 472, in beam_search_decoder
linear_unsafe(initial_state, cell.state_size, False, scope='initial_state_projection')
File "/home/olgastrizhko/cuda-workspace/seq2seq-speech_test/translate/rnn.py", line 131, in fun
return fun(*args, **kwargs)
TypeError: _linear() got an unexpected keyword argument 'scope'
Could you please help me to solve this issue
Thank you in advance! Looking forward to hear from you.
Best Regards,
Olga.
Can someone verify that specifying a model directory in the config works as it's supposed to?
My checkpoints seem to be defaulting to /model/.
I think your work is better than the Google models -
Thank you very much,
I'm always following your updates.
def look
in https://github.com/eske/seq2seq/blob/master/translate/models.py
I see it in https://github.com/eske/seq2seq/blob/master/translate/models.py#L851
where a lot of init_var here
initial_context, _ = look(0, initial_output, initial_input, pos=initial_pos, prev_weights=initial_weights,
context=zero_context)
@Eske Thank you!!
Hi,I want to use your pretrained model from the following link but I get an error. Could you help me figure out this?
https://drive.google.com/file/d/1XM_76xP5kg8--BvykF3uJdCSYtdLFGX_/view?usp=sharing
I use tensorflow 1.6,Python 3.6.4 with Anaconda on Ubuntu 14.04
run command
./seq2seq.sh models/WMT14/config.yaml --decode -v
and part of the error message
Traceback (most recent call last):
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
return fn(*args)
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [500,30000] rhs shape= [620,30000]
[[Node: save_1/Assign_14 = Assign[T=DT_FLOAT, _class=["loc:@decoder_fr/softmax1/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](decoder_fr/softmax1/kernel, save_1/RestoreV2/_15)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/xuchongyang/inference_evaluation/workloads/code/seq2seq/translate/main.py", line 306, in
main()
File "/home/xuchongyang/inference_evaluation/workloads/code/seq2seq/translate/main.py", line 278, in main
model.initialize([best_checkpoint], **params)
File "/home/xuchongyang/inference_evaluation/workloads/code/seq2seq/translate/translation_model.py", line 720, in initialize
load_checkpoint(sess, None, checkpoint, blacklist=blacklist, whitelist=whitelist, **params)
File "/home/xuchongyang/inference_evaluation/workloads/code/seq2seq/translate/translation_model.py", line 796, in load_checkpoint
tf.train.Saver(variables).restore(sess, filename)
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1755, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/home/xuchongyang/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
I have run --train, --decode, --eval options successfully. However, when I run --align option using the command as follows, I can get the attention heatmap but it isn't correct because it is with the reference file not with the prediction file. The align option didn't do the translation.
Here is my command:
python -m translate experiments/WMT14/substr.yaml \
--align experiments/WMT14/data/bpe_en-de/test2.en \
experiments/WMT14/data/bpe_en-de/test2.de \
--checkpoints experiments/WMT14/beam10_en-de/checkpoints/best \
--output experiments/WMT14/data/bpe_en-de/test2.raw \
--beam-size 1 \
--gpu-id 0
I want to know if I use the wrong command. Thank you very much!
Will there be any access to Examples
or Pre-trained
models in the future?
When I run this command in terminal,
./seq2seq.sh config/APE17/chained.yaml --train -v
an error is raised:
Traceback (most recent call last):
File "/home/devops/.conda/envs/tensorflow_py35/lib/python3.5/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/devops/.conda/envs/tensorflow_py35/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/devops/ape/seq2seq/translate/main.py", line 298, in
main()
File "/home/devops/ape/seq2seq/translate/main.py", line 230, in main
model = TranslationModel(**config)
File "/home/devops/ape/seq2seq/translate/translation_model.py", line 67, in init
ref_ext=ref_ext, binary=self.binary, **kwargs)
File "/home/devops/ape/seq2seq/translate/utils.py", line 217, in get_filenames
shutil.copy(src, dest)
File "/home/devops/.conda/envs/tensorflow_py35/lib/python3.5/shutil.py", line 241, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/devops/.conda/envs/tensorflow_py35/lib/python3.5/shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'data/APE17/vocab.mt'
It seems the script can't find the APE17 data files. Where can I get these files?
Hi all,
For the multi-task training, what kind of architecture did you use to accomplish this task? Did you use a separate encoder/decoder for each task, or did you use shared parameters between the encoder and/or the decoder? Could you make available an article about the implemented architecture.
Than you in advance.
Could you please make the trained model available?
Hi, I've just tried the latest seq2seq with tensorflow 1.2. The bleu score for my test set decoding suddenly become 1++ (even though during training with development set, bleu score is around 20+) using the latest version. Previously I used the seq2seq with tensorflow 1.0 without any problem, test set is around 20+. Did I miss something?
By the way, thanks for the great work!
Hi
thanks for sharing your code with us. Is this model good for feature extraction or it has been designed only for the translation purpose?
Thanks:)
I want to know, is the encoder shared between ASR and AST task and decoder shared between MT and AST task when we run the LibriSpeech Multitask recipe. Or the ASR, AST and MT tasks use different encoders and decoders and update the parameters turn by turn.
./seq2seq.sh config/LibriSpeech/AST.yaml --train -v --purge
an error is raised:
06/11 06:44:24 step 177000 epoch 101 learning rate 0.001 step-time 1.064 loss nan
06/11 06:44:24 starting evaluation
06/11 06:45:50 dev bleu=0.00 ter=100.00 wer=100.00 bleu1=0.00 loss=nan penalty=0.000 ratio=0.071
06/11 06:45:50 saving model to models/LibriSpeech/AST/checkpoints
06/11 06:45:50 finished saving model
Traceback (most recent call last):
File "/home/zd/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "/home/zd/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 85, in run_code
exec(code, run_globals)
File "/home/zd/speech_translation/seq2seq/translate/main.py", line 306, in
main()
File "/home/zd/speech_translation/seq2seq/translate/main.py", line 300, in main
model.train(**config)
File "/home/zd/speech_translation/seq2seq/translate/translation_model.py", line 472, in train
self.train_step(loss_function=loss_function, use_baseline=use_baseline, **kwargs)
File "/home/zd/speech_translation/seq2seq/translate/translation_model.py", line 527, in train_step
update_baseline=True)
File "/home/zd/speech_translation/seq2seq/translate/seq2seq_model.py", line 231, in step
encoder_inputs, targets, input_length = self.get_batch(data)
File "/home/zd/speech_translation/seq2seq/translate/seq2seq_model.py", line 330, in get_batch
for encoder, inputs in zip(self.encoders, inputs)]
File "/home/zd/speech_translation/seq2seq/translate/seq2seq_model.py", line 330, in
for encoder, inputs in zip(self.encoders, inputs)]
ValueError: setting an array element with a sequence.
*** Error in `python3': free(): invalid pointer: 0x000055f69f8a4b00 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fb347e0d7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fb347e1637a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fb347e1a53c]
/home/zd/anaconda3/envs/tensorflow/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2e757)[0x7fb34604c757]
python3(+0xf1532)[0x55f5f1c2a532]
python3(+0xf1532)[0x55f5f1c2a532]
python3(+0xf1532)[0x55f5f1c2a532]
python3(+0xf12b7)[0x55f5f1c2a2b7]
python3(+0xf1147)[0x55f5f1c2a147]
python3(+0x198aa2)[0x55f5f1cd1aa2]
python3(PyObject_CallFinalizer+0xf9)[0x55f5f1cd1f19]
python3(PyObject_CallFinalizerFromDealloc+0x1b)[0x55f5f1cd1f9b]
python3(+0x19902e)[0x55f5f1cd202e]
python3(+0xf18a8)[0x55f5f1c2a8a8]
python3(+0x1993b1)[0x55f5f1cd23b1]
python3(+0xf12b7)[0x55f5f1c2a2b7]
python3(+0xf1147)[0x55f5f1c2a147]
python3(+0xf115d)[0x55f5f1c2a15d]
python3(+0xf115d)[0x55f5f1c2a15d]
python3(+0xf115d)[0x55f5f1c2a15d]
python3(PyDict_SetItem+0x3da)[0x55f5f1c6fe7a]
python3(PyDict_SetItemString+0x4f)[0x55f5f1c7878f]
python3(PyImport_Cleanup+0x99)[0x55
https://github.com/eske/seq2seq/blob/master/config/WMT14/download.sh#L18-L19
We have the original downloaded data:
ntst14.en
ntst14.fr
ntst1213.en
ntst1213.fr
What is ntst1213.fr-en
and ntst14.fr-en
?
@Eske Thank you!
Does this code work without GPU ? thanks
Hi,
Great work!
But did you have a plan to implement additional input features?
Like the paper:
Linguistic Input Features Improve Neural Machine Translation (Senrich et al. 2016)
and implement in torch:
harvardnlp/seq2seq-attn@cb18ac3
Thank you very much for sharing such great code!
在输入命令./seq2seq.sh CONFIG --train -v时报错No such file or directory: CONFIG
Hi,
Thanks Eske for this amazing stack.
I am trying to train the model to convert the audio in one language to the test in another language.
Looking at the config folder I can see BTEC,ICASSP folders with each having prepare.sh files.
If I need to train the network with my own wav files and text files, How can I do that?
Even if I try to modify the prepare.sh, which folder in ./config/. is to be selected(BTEC or ICASSP) etc.
Thanks in advance!!!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.