pytorch / translate

Translate - a PyTorch Language Library

---

NOTE

PyTorch Translate is now deprecated, please use fairseq instead.

---

Translate - a PyTorch Language Library

Translate is a library for machine translation written in PyTorch. It provides training for sequence-to-sequence models. Translate relies on fairseq, a general sequence-to-sequence library, which means that models implemented in both Translate and Fairseq can be trained. Translate also provides the ability to export some models to Caffe2 graphs via ONNX and to load and run these models from C++ for production purposes. Currently, we export components (encoder, decoder) to Caffe2 separately and beam search is implemented in C++. In the near future, we will be able to export the beam search as well. We also plan to add export support to more models.

Quickstart

If you are just interested in training/evaluating MT models, and not in exporting the models to Caffe2 via ONNX, you can install Translate for Python 3 by following these few steps:

1. Install pytorch
2. Install fairseq
3. Clone this repository git clone https://github.com/pytorch/translate.git pytorch-translate && cd pytorch-translate
4. Run python setup.py install

Provided you have CUDA installed you should be good to go.

Requirements and Full Installation

Translate Requires:

• A Linux operating system with a CUDA compatible card
• GNU C++ compiler version 4.9.2 and above
• A CUDA installation. We recommend CUDA 8.0 or CUDA 9.0

Use Our Docker Image:

Install Docker and nvidia-docker, then run

sudo docker pull pytorch/translate
sudo nvidia-docker run -i -t --rm pytorch/translate /bin/bash
. ~/miniconda/bin/activate
cd ~/translate

You should now be able to run the sample commands in the Usage Examples section below. You can also see the available image versions under https://hub.docker.com/r/pytorch/translate/tags/.

Install Translate from Source:

These instructions were mainly tested on Ubuntu 16.04.5 LTS (Xenial Xerus) with a Tesla M60 card and a CUDA 9 installation. We highly encourage you to report an issue if you are unable to install this project for your specific configuration.

• If you don't already have an existing Anaconda environment with Python 3.6, you can install one via Miniconda3:

  wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
  chmod +x miniconda.sh
  ./miniconda.sh -b -p ~/miniconda
  rm miniconda.sh
  . ~/miniconda/bin/activate
  

• Clone the Translate repo:

  git clone https://github.com/pytorch/translate.git
  pushd translate
  

• Install the PyTorch conda package:

  # Set to 8 or 9 depending on your CUDA version.
  TMP_CUDA_VERSION="9"
  
  # Uninstall previous versions of PyTorch. Doing this twice is intentional.
  # Error messages about torch not being installed are benign.
  pip uninstall -y torch
  pip uninstall -y torch
  
  # This may not be necessary if you already have the latest cuDNN library.
  conda install -y cudnn
  
  # Add LAPACK support for the GPU.
  conda install -y -c pytorch "magma-cuda${TMP_CUDA_VERSION}0"
  
  # Install the combined PyTorch nightly conda package.
  conda install pytorch-nightly cudatoolkit=${TMP_CUDA_VERSION}.0 -c pytorch
  
  # Install NCCL2.
  wget "https://s3.amazonaws.com/pytorch/nccl_2.1.15-1%2Bcuda${TMP_CUDA_VERSION}.0_x86_64.txz"
  TMP_NCCL_VERSION="nccl_2.1.15-1+cuda${TMP_CUDA_VERSION}.0_x86_64"
  tar -xvf "${TMP_NCCL_VERSION}.txz"
  rm "${TMP_NCCL_VERSION}.txz"
  
  # Set some environmental variables needed to link libraries correctly.
  export CONDA_PATH="$(dirname $(which conda))/.."
  export NCCL_ROOT_DIR="$(pwd)/${TMP_NCCL_VERSION}"
  export LD_LIBRARY_PATH="${CONDA_PATH}/lib:${NCCL_ROOT_DIR}/lib:${LD_LIBRARY_PATH}"
  

• Install ONNX:

  git clone --recursive https://github.com/onnx/onnx.git
  yes | pip install ./onnx 2>&1 | tee ONNX_OUT
  

If you get a Protobuf compiler not found error, you need to install it:

conda install -c anaconda protobuf

Then, try to install ONNX again:

yes | pip install ./onnx 2>&1 | tee ONNX_OUT

• Build Translate:

  pip uninstall -y pytorch-translate
  python3 setup.py build develop
  

Now you should be able to run the example scripts below!

Usage Examples

Note: the example commands given assume that you are the root of the cloned GitHub repository or that you're in the translate directory of the Docker or Amazon image. You may also need to make sure you have the Anaconda environment activated.

Training

We provide an example script to train a model for the IWSLT 2014 German-English task. We used this command to obtain a pretrained model:

bash pytorch_translate/examples/train_iwslt14.sh

The pretrained model actually contains two checkpoints that correspond to training twice with random initialization of the parameters. This is useful to obtain ensembles. This dataset is relatively small (~160K sentence pairs), so training will complete in a few hours on a single GPU.

Training with tensorboard visualization

We provide support for visualizing training stats with tensorboard. As a dependency, you will need tensorboard_logger installed.

pip install tensorboard_logger

Please also make sure that tensorboard is installed. It also comes with tensorflow installation.

You can use the above example script to train with tensorboard, but need to change line 10 from :

CUDA_VISIBLE_DEVICES=0 python3 pytorch_translate/train.py

to

CUDA_VISIBLE_DEVICES=0 python3 pytorch_translate/train_with_tensorboard.py

The event log directory for tensorboard can be specified by option --tensorboard_dir with a default value: run-1234. This directory is appended to your --save_dir argument.

For example in the above script, you can visualize with:

tensorboard --logdir checkpoints/runs/run-1234

Multiple runs can be compared by specifying different --tensorboard_dir. i.e. run-1234 and run-2345. Then

tensorboard --logdir checkpoints/runs

can visualize stats from both runs.

Pretrained Model

A pretrained model for IWSLT 2014 can be evaluated by running the example script:

bash pytorch_translate/examples/generate_iwslt14.sh

Note the improvement in performance when using an ensemble of size 2 instead of a single model.

Exporting a Model with ONNX

We provide an example script to export a PyTorch model to a Caffe2 graph via ONNX:

bash pytorch_translate/examples/export_iwslt14.sh

This will output two files, encoder.pb and decoder.pb, that correspond to the computation of the encoder and one step of the decoder. The example exports a single checkpoint (--checkpoint model/averaged_checkpoint_best_0.pt but is also possible to export an ensemble (--checkpoint model/averaged_checkpoint_best_0.pt --checkpoint model/averaged_checkpoint_best_1.pt). Note that during export, you can also control a few hyperparameters such as beam search size, word and UNK rewards.

Using the Model

To use the sample exported Caffe2 model to translate sentences, run:

echo "hallo welt" | bash pytorch_translate/examples/translate_iwslt14.sh

Note that the model takes in BPE inputs, so some input words need to be split into multiple tokens. For instance, "hineinstopfen" is represented as "hinein@@ stop@@ fen".

PyTorch Translate Research

We welcome you to explore the models we have in the pytorch_translate/research folder. If you use them and encounter any errors, please paste logs and a command that we can use to reproduce the error. Feel free to contribute any bugfixes or report your experience, but keep in mind that these models are a work in progress and thus are currently unsupported.

Join the Translate Community

We welcome contributions! See the CONTRIBUTING.md file for how to help out.

License

Translate is BSD-licensed, as found in the LICENSE file.

translate's Issues

NameError: name 'LevenshteinTransformerModel' is not defined

When running bash pytorch_translate/examples/train_iwslt14.sh

Traceback (most recent call last):
  File "pytorch_translate/train.py", line 835, in <module>
    _main()
  File "pytorch_translate/train.py", line 831, in _main
    main(args)
  File "pytorch_translate/train.py", line 809, in main
    single_process_main(args, trainer_class, **train_step_kwargs)
  File "pytorch_translate/train.py", line 719, in single_process_main
    **train_step_kwargs,
  File "pytorch_translate/train.py", line 639, in train
    checkpoint_manager=checkpoint_manager,
  File "/Users/erippeth/Code/independent/onnx_junk/translate/pytorch_translate/evals.py", line 395, in save_and_eval
    averaged_params=averaged_params,
  File "/Users/erippeth/Code/independent/onnx_junk/translate/pytorch_translate/evals.py", line 181, in evaluate_bleu
    model_params=averaged_params,
  File "/Users/erippeth/Code/independent/onnx_junk/translate/pytorch_translate/evals.py", line 275, in calculate_bleu_on_subset
    modify_target_dict=False,
  File "/Users/erippeth/Code/independent/onnx_junk/translate/pytorch_translate/generate.py", line 101, in generate_score
    modify_target_dict=modify_target_dict,
  File "/Users/erippeth/Code/independent/onnx_junk/translate/pytorch_translate/generate.py", line 178, in _generate_score
    translator = build_sequence_generator(args, task, models)
  File "/Users/erippeth/Code/independent/onnx_junk/translate/pytorch_translate/generate.py", line 136, in build_sequence_generator
    elif isinstance(models[0], LevenshteinTransformerModel):
NameError: name 'LevenshteinTransformerModel' is not defined

The culprit is this fella: https://github.com/pytorch/translate/blob/master/pytorch_translate/generate.py#L48

Because the import is conditional, there's a name error when the import fails.

How to train a translator from English to Chinese?

Just wonder the properly data preparation to train a translation model

Generating with replace-unk is broken

I get the following error:

  File "../../../generate.py", line 200, in _iter_first_best_bilingual
    target_str = task.dataset(dataset_split).dst.get_original_text(sample_id)
AttributeError: 'LanguagePairDataset' object has no attribute 'dst'

Most likely an artifact of the fairseq update

Hybrid model performance

Hello! I would like to ask if somebody have training results for the hybrid model implementation? Also please confirm, did i understand correctly that this model uses RNMT+ decoder, as in Chen et al. paper?

No more libcaffe2.so to link

PyTorch Translate uses libcaffe2.so, but it seems that it no longer exists in the latest PyTorch build.
See

translate/pytorch_translate/cpp/CMakeLists.txt

Line 51 in c0c8867

target_link_libraries(translation_decoder PUBLIC caffe2)

While running pretrained model(IWSLT 2014) , observed below errors

Traceback (most recent call last):
File "pytorch_translate/generate.py", line 705, in
main()
File "pytorch_translate/generate.py", line 609, in main
generate(args)
File "pytorch_translate/generate.py", line 634, in generate
args.path.split(":")
File "/root/pytorch/fairseq/pytorch-translate/translate/pytorch_translate/utils.py", line 116, in load_diverse_ensemble_for_inference
task = tasks.setup_task(checkpoints_data[0]["args"])
File "/root/pytorch/fairseq/fairseq/tasks/init.py", line 19, in setup_task
return TASK_REGISTRY[args.task].setup_task(args, **kwargs)
File "/root/pytorch/fairseq/pytorch-translate/translate/pytorch_translate/tasks/pytorch_translate_task.py", line 124, in setup_task
args.source_vocab_file
File "/root/pytorch/fairseq/fairseq/data/dictionary.py", line 184, in load
d.add_from_file(f, ignore_utf_errors)
File "/root/pytorch/fairseq/fairseq/data/dictionary.py", line 201, in add_from_file
raise fnfe
File "/root/pytorch/fairseq/fairseq/data/dictionary.py", line 195, in add_from_file
with open(f, 'r', encoding='utf-8') as fd:
FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/dictionary-de.txt'

train_with_tensorboard.py

Hi, there is no such file in your repo. Can you please add ?

Export a fairseq trained transformer (base) model

I trained a translation model based on scaling neural machine translation and wanted to convert into into caffe2 but when used docker image it is throwing an error ::

According to #60 only RNN trained models are supported ?
How about the transformer trained models .
Should I retrain again to export to caffe2 ?

Python version degrades

while using this conda install -y -c caffe2 "pytorch-caffe2-cuda${TMP_CUDA_VERSION}.0-cudnn7"
python version degrades to 2.7.15 and hence is not able to export model using https://github.com/pytorch/translate#exporting-a-model-with-onnx

Can this conversion be done on a CPU
Does pytorch1.0 does not need many of the above steps and facilitate the conversion process..

Error when exporting checkpoint: 'Namespace' has no 'residual_level'

I used the docker image to train a model, but when I tried to export it I got this error message:

Ignoring @/caffe2/caffe2/contrib/aten:aten_op as it is not a valid file.
Traceback (most recent call last):
  File "pytorch_translate/onnx_component_export.py", line 124, in <module>
    main()
  File "pytorch_translate/onnx_component_export.py", line 86, in main
    dst_dict_filename=args.dst_dict,
  File "/home/translate/pytorch_translate/ensemble_export.py", line 236, in build_from_checkpoints
    dst_dict_filename,
  File "/home/translate/pytorch_translate/ensemble_export.py", line 62, in load_models_from_checkpoints
    dst_dict,
  File "/home/translate/pytorch_translate/rnn.py", line 218, in build_model
    residual_level=args.residual_level,
AttributeError: 'Namespace' object has no attribute 'residual_level'

I also got this error message previously when I tried to export checkpoints from models I'd trained with fairseq-py - I assumed it just meant this wasn't supported, but since it happens on models trained with this project as well it's maybe a bug.

The command I used to export was:

#!/bin/bash

export NCCL_ROOT_DIR="$(pwd)/nccl_2.1.15-1+cuda8.0_x86_64"
export LD_LIBRARY_PATH="${NCCL_ROOT_DIR}/lib:${LD_LIBRARY_PATH}"
python3 pytorch_translate/onnx_component_export.py \
    --checkpoint mounted/checkpoints/checkpoint_best.pt \
    --encoder_output_file mounted/export/encoder.pb \
    --decoder_output_file mounted/export/decoder.pb \
    --src_dict mounted/checkpoints/dictionary-in.txt \
    --dst_dict mounted/checkpoints/dictionary-out.txt \
    --beam_size 6 \
    --word_penalty 0.25 \
    --unk_penalty -0.5 \
    --batched_beam && \
  echo "Finished exporting encoder as ./encoder.pb and decoder as ./decoder.pb"

The command I used to train this model was:

#!/bin/bash

export NCCL_ROOT_DIR="$(pwd)/nccl_2.1.15-1+cuda8.0_x86_64"
export LD_LIBRARY_PATH="${NCCL_ROOT_DIR}/lib:${LD_LIBRARY_PATH}"
CUDA_VISIBLE_DEVICES=0 python3 pytorch_translate/train.py \
   "" \
   --arch fconv_iwslt_de_en \
   --lr-scheduler reduce_lr_on_plateau \
   --lr-shrink 0.1 \
   --max-epoch 20 \
   --optimizer nag \
   --lr 0.3 \
   --clip-norm 0.1 \
   --criterion cross_entropy \
   --batch-size 256 \
   --max-tokens 3000 \
   --save-dir /home/translate/mounted/checkpoints \
   --beam 5 \
   --source-lang in \
   --target-lang out \
   --source-vocab-file /home/translate/mounted/checkpoints/dictionary-in.txt \
   --target-vocab-file /home/translate/mounted/checkpoints/dictionary-out.txt \
   --train-source-text-file /home/translate/mounted/bpe/train.in \
   --train-target-text-file /home/translate/mounted/bpe/train.out \
   --eval-source-text-file /home/translate/mounted/bpe/valid.in \
   --eval-target-text-file /home/translate/mounted/bpe/valid.out \
   --log-interval 1000 \
   2>&1 | tee -a /home/translate/mounted/checkpoints/log

I notice the error is in rnn.py, but I don't use an rnn, but an fconv model. Are these models not yet supported for export?

Cannot import 'latent_var_criterion'

Hi,
After installing translate, when I try:
bash pytorch_translate/examples/train_iwslt14.sh
It shows this error:
Traceback (most recent call last): File "pytorch_translate/train.py", line 22, in <module> from pytorch_translate import latent_var_criterion # noqa ImportError: cannot import name 'latent_var_criterion' from 'pytorch_translate'

How to solve it?
Thanks.

How can I export a single Onnx?

hi,

Sorry to bother you, I want to export the Onnx model using fairseq, and I see that you have implemented this function in your code.
I see your code that use onnx as the medium and them export to caffe2. But the result are divided into encoder and decoder. May I ask why it needs to divided into 2 parts? And how can I output a single entire model?

thanks

RNN decode results are batch-size dependent

Using an RNN model (similar to the example architecture), smaller batch sizes produce better results with generate.py. It looks like a padding error, the best results occur with a batch size of 1 (no batching).

I attempted to reproduce with the example scripts but that model seems out of date and would not run.

Cannot load pretrained model

When I try to load the pretrained iwslt de-en model, I see an exception:

>>> import torch
>>> from fairseq.models.transformer import TransformerModel
>>> model = TransformerModel.from_pretrained("./", "averaged_checkpoint_best_0.pt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/erippeth/miniconda3/envs/fairseq-dev/lib/python3.6/site-packages/fairseq/models/fairseq_model.py", line 218, in from_pretrained
    **kwargs,
  File "/Users/erippeth/miniconda3/envs/fairseq-dev/lib/python3.6/site-packages/fairseq/hub_utils.py", line 73, in from_pretrained
    arg_overrides=kwargs,
  File "/Users/erippeth/miniconda3/envs/fairseq-dev/lib/python3.6/site-packages/fairseq/checkpoint_utils.py", line 202, in load_model_ensemble_and_task
    state = load_checkpoint_to_cpu(filename, arg_overrides)
  File "/Users/erippeth/miniconda3/envs/fairseq-dev/lib/python3.6/site-packages/fairseq/checkpoint_utils.py", line 175, in load_checkpoint_to_cpu
    state = _upgrade_state_dict(state)
  File "/Users/erippeth/miniconda3/envs/fairseq-dev/lib/python3.6/site-packages/fairseq/checkpoint_utils.py", line 366, in _upgrade_state_dict
    registry.set_defaults(state["args"], tasks.TASK_REGISTRY[state["args"].task])
KeyError: 'pytorch_translate'

Do I need to retrain this model because of API changes or am I doing something wrong in loading?

adv_train.py can't work well

I found that there are some errors when I run adv_train.py, some functions used in adv_train.py do not match the current version.

build translate fail under gcc 5.3.1

when i build translate with cuda 9.0, i meet below issues, did anyone meet the same issue?

-- The C compiler identification is GNU 5.3.1
-- The CXX compiler identification is GNU 5.3.1
-- Check for working C compiler: /opt/rh/devtoolset-4/root/usr/bin/cc
-- Check for working C compiler: /opt/rh/devtoolset-4/root/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/rh/devtoolset-4/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/devtoolset-4/root/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2: Found gflags with new-style gflags target.
-- Caffe2: Cannot find glog automatically. Using legacy find.
-- Found glog: /usr/include
-- Caffe2: Found glog (include: /usr/include, library: /usr/lib64/libglog.so)
-- Caffe2: Found protobuf with new-style protobuf targets.
-- Caffe2: Protobuf version 3.5.0
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "7.0")
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 9.0
-- Found CUDNN: /usr/local/cuda/include
-- Found cuDNN: v7.1.2 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s): 6.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60
-- Summary:
-- CMake version : 3.11.1
-- CMake command : /home/wliao2/local/bin/cmake
-- System name : Linux
-- C++ compiler : /opt/rh/devtoolset-4/root/usr/bin/c++
-- C++ compiler version : 5.3.1
-- CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing
-- Caffe2 version : 0.8.2
-- Caffe2 include path : /home/wliao2/anaconda3/envs/translate/include
-- Have CUDA :
-- Configuring done
-- Generating done
-- Build files have been written to: /home/wliao2/translate/pytorch_translate/cpp/build
(translate) [xxx]$ make 2>&1 | tee MAKE_OUT
Scanning dependencies of target translation_decoder
[ 16%] Building CXX object CMakeFiles/translation_decoder.dir/Decoder.cpp.o
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:38:18: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_string(encoder_model, "", "Encoder model path");
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:39:18: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_string(decoder_step_model, "", "Decoder step model path");
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:40:18: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_string(source_vocab_path, "", "Source vocab file");
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:41:18: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_string(target_vocab_path, "", "Target vocab file");
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:43:15: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_int(beam_size, -1, "Beam size");
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:44:18: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_double(
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:50:15: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_int(
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:57:16: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_bool(
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:61:16: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_bool(
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:65:16: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_bool(
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:69:18: error: expected constructor, destructor, or type conversion before ‘(’ token
C10_DEFINE_double(
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp: In function ‘int main(int, char**)’:
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:78:7: error: ‘FLAGS_source_vocab_path’ is not a member of ‘c10’
if (c10::FLAGS_source_vocab_path.empty() ||
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:79:7: error: ‘FLAGS_target_vocab_path’ is not a member of ‘c10’
c10::FLAGS_target_vocab_path.empty() ||
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:80:7: error: ‘FLAGS_encoder_model’ is not a member of ‘c10’
c10::FLAGS_encoder_model.empty() ||
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:81:7: error: ‘FLAGS_decoder_step_model’ is not a member of ‘c10’
c10::FLAGS_decoder_step_model.empty()) {
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:85:38: error: ‘FLAGS_source_vocab_path’ is not a member of ‘c10’
<< "(source_vocab_path='" << c10::FLAGS_source_vocab_path
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:86:40: error: ‘FLAGS_target_vocab_path’ is not a member of ‘c10’
<< "', target_vocab_path='" << c10::FLAGS_target_vocab_path
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:87:36: error: ‘FLAGS_encoder_model’ is not a member of ‘c10’
<< "', encoder_model='" << c10::FLAGS_encoder_model
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:88:41: error: ‘FLAGS_decoder_step_model’ is not a member of ‘c10’
<< "', decoder_step_model='" << c10::FLAGS_decoder_step_model << "')";
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:92:41: error: ‘FLAGS_source_vocab_path’ is not a member of ‘c10’
std::make_sharedpyt::Dictionary(c10::FLAGS_source_vocab_path);
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:94:41: error: ‘FLAGS_target_vocab_path’ is not a member of ‘c10’
std::make_sharedpyt::Dictionary(c10::FLAGS_target_vocab_path);
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:96:7: error: ‘FLAGS_beam_size’ is not a member of ‘c10’
c10::FLAGS_beam_size,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:97:7: error: ‘FLAGS_max_out_seq_len_mult’ is not a member of ‘c10’
c10::FLAGS_max_out_seq_len_mult,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:98:7: error: ‘FLAGS_max_out_seq_len_bias’ is not a member of ‘c10’
c10::FLAGS_max_out_seq_len_bias,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:101:7: error: ‘FLAGS_encoder_model’ is not a member of ‘c10’
c10::FLAGS_encoder_model,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:102:7: error: ‘FLAGS_decoder_step_model’ is not a member of ‘c10’
c10::FLAGS_decoder_step_model,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:103:7: error: ‘FLAGS_reverse_source’ is not a member of ‘c10’
c10::FLAGS_reverse_source,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:104:7: error: ‘FLAGS_stop_at_eos’ is not a member of ‘c10’
c10::FLAGS_stop_at_eos,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:105:7: error: ‘FLAGS_append_eos_to_source’ is not a member of ‘c10’
c10::FLAGS_append_eos_to_source,
^
/home/wliao2/translate/pytorch_translate/cpp/Decoder.cpp:106:7: error: ‘FLAGS_length_penalty’ is not a member of ‘c10’
c10::FLAGS_length_penalty);
^
make[2]: *** [CMakeFiles/translation_decoder.dir/Decoder.cpp.o] Error 1
make[1]: *** [CMakeFiles/translation_decoder.dir/all] Error 2
make: *** [all] Error 2

beam search

According to README:

Currently, we export components (encoder, decoder) to Caffe2 separately and beam search is implemented in C++. In the near future, we will be able to export the beam search as well. We also plan to add export support to more models.

Where can I find this C++ code or exportable (jit) beam search already in repo?

Replacing Hardswish to alternative activation function

torch.onnx seems to have an error in converting
Error:Exporting the operator hardsigmoid to ONNX opset version 11 is not supported.

Unk replacement doesn't work with the transformer

I get the following error when decoding using the transformer with --replace-unk:

Traceback (most recent call last):
  File "../../../generate.py", line 569, in <module>
    main()
  File "../../../generate.py", line 474, in main
    generate(args)
  File "../../../generate.py", line 555, in generate
    models=models, args=args, task=task, dataset_split=args.gen_subset
  File "../../../generate.py", line 142, in _generate_score
    args, task, dataset_split, translations, align_dict
  File "../../../generate.py", line 220, in _iter_first_best_bilingual
    remove_bpe=args.remove_bpe,
  File "/home/pmichel1/.local/lib/python3.6/site-packages/fairseq-0.5.0-py3.6-linux-x86_64.egg/fairseq/utils.py", line 293, in post_process_prediction
    hypo_str = replace_unk(hypo_str, src_str, alignment, align_dict, tgt_dict.unk_string())
  File "/home/pmichel1/.local/lib/python3.6/site-packages/fairseq-0.5.0-py3.6-linux-x86_64.egg/fairseq/utils.py", line 283, in replace_unk
    src_token = src_tokens[alignment[i]]
IndexError: list index out of range

ONNX export of RNN failed, Cannot export individual pack_padded_sequence/pad_packed_sequence

The new RNN model I trained didn't export properly, I got this error message:

Ignoring @/caffe2/caffe2/contrib/aten:aten_op as it is not a valid file.
Traceback (most recent call last):
  File "pytorch_translate/onnx_component_export.py", line 124, in <module>
    main()
  File "pytorch_translate/onnx_component_export.py", line 89, in main
    encoder_ensemble.save_to_db(args.encoder_output_file)
  File "/home/translate/pytorch_translate/ensemble_export.py", line 213, in save_to_db
    self.onnx_export(tmp_file)
  File "/home/translate/pytorch_translate/ensemble_export.py", line 204, in onnx_export
    output_names=self.output_names,
  File "/home/translate/pytorch_translate/ensemble_export.py", line 44, in onnx_export_ensemble
    output_names=output_names,
  File "/home/miniconda/lib/python3.6/site-packages/torch/onnx/__init__.py", line 20, in _export
    return utils._export(*args, **kwargs)
  File "/home/miniconda/lib/python3.6/site-packages/torch/onnx/utils.py", line 207, in _export
    proto, export_map = graph.export(params, _onnx_opset_version, defer_weight_export, export_raw_ir)
RuntimeError: ONNX export failed: Cannot export individual pack_padded_sequence or pad_packed_sequence; these operations must occur in pairs.

... followed by a long stacktrace.

The model was trained with this command line/script, based on the example:

#!/bin/bash

export NCCL_ROOT_DIR="$(pwd)/nccl_2.1.15-1+cuda8.0_x86_64"
export LD_LIBRARY_PATH="${NCCL_ROOT_DIR}/lib:${LD_LIBRARY_PATH}"
CUDA_VISIBLE_DEVICES=0 python3 pytorch_translate/train.py \
   "" \
   --arch rnn \
   --lr-scheduler fixed \
   --force-anneal 200 \
   --max-epoch 100 \
   --stop-time-hr 72 \
   --optimizer sgd \
   --lr 0.5 \
   --clip-norm 5.0 \
   --criterion label_smoothed_cross_entropy \
   --batch-size 256 \
   --lenpen 0 \
   --unkpen 0.5 \
   --word-reward 0.25 \
   --max-tokens 3000 \
   --encoder-layers 2 \
   --encoder-embed-dim 256 \
   --encoder-hidden-dim 512 \
   --decoder-layers 2 \
   --decoder-embed-dim 256 \
   --decoder-hidden-dim 512 \
   --decoder-out-embed-dim 256 \
   --save-dir /home/translate/mounted/checkpoints \
   --attention-type dot \
   --sentence-avg \
   --momentum 0 \
   --generate-bleu-eval-avg-checkpoints 10 \
   --generate-bleu-eval-per-epoch \
   --beam 6 \
   --no-beamable-mm \
   --source-lang in \
   --target-lang out \
   --source-vocab-file /home/translate/mounted/checkpoints/dictionary-in.txt \
   --target-vocab-file /home/translate/mounted/checkpoints/dictionary-out.txt \
   --train-source-text-file /home/translate/mounted/bpe/train.in \
   --train-target-text-file /home/translate/mounted/bpe/train.out \
   --eval-source-text-file /home/translate/mounted/bpe/valid.in \
   --eval-target-text-file /home/translate/mounted/bpe/valid.out \
   --source-max-vocab-size 14000 \
   --target-max-vocab-size 14000 \
   --log-interval 500 \
   2>&1 | tee -a /home/translate/mounted/checkpoints/log

(I see now that I forgot to make it an LSTM, wow. How did I even get good results?)

The export command was exactly the same as the example, only the path being different and the download being removed. I tried exporting many different checkpoints, averaged and not.

nccl no more downloadable

Following your README instructions I get an error downloading nccl

wget "https://s3.amazonaws.com/pytorch/nccl_2.1.15-1%2Bcuda10.0_x86_64.txz"
--2019-09-27 14:51:33--  https://s3.amazonaws.com/pytorch/nccl_2.1.15-1%2Bcuda10.0_x86_64.txz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.97.133
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.97.133|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2019-09-27 14:51:33 ERROR 403: Forbidden.

please help

Training Knowledge distillation with model trained on Fairseq

Hi ,
I have trained a En-De model using fairseq and now want to try the knowledge_distillation task to train the student model on reduced transformer architecture.I was however unable to do so. I wanted to know whether this is possible i.e., training teacher on fairseq and student in translate libraries. Or should the teacher be also trained on translate library. Kindly clarify. Also I am trying to do knowledge_distillation task and I am confused on how to give inputs.
Say I have model , binaries and dictionaries generated from walkthrough in fairseq. How can I use them from knowledge distillation. Are they enough or should I also need the logits saved in file and how should I use them please elaborate.
Thanks

Exporting model status

What is the current roadmap and status for export to Caffe2?
Currently I cannot build as missing files, build scripts.
Also in other issue it is said that next commit, the export will be using TorchScript instead?

pytorch

What is the version of pytorch

Do you include pre-trained translate model for popular languages ?

Do you have any plans to include the pre-trained models for popular languages ?

Syntax Error while running the training command

Hi @jmp84
While running the command bash pytorch_translate/examples/train_iwslt14.sh, an invalid syntax error is thrown. Any help on this would be appreciated.
https://user-images.githubusercontent.com/40763103/54600117-1751d480-4a62-11e9-849b-914ff3d9587d.png

During Build Translate step i am getting this error, should i start all over again?

Btw this notice that caffe installed never failed but it got successfully ended at 18% . it was shocking to me but i let is go on ..

Current error is below

`(base) test@dc-isb-ds-001:~/Downloads/translate/pytorch_translate/cpp/build$ cmake \

-DCMAKE_PREFIX_PATH="${CONDA_PATH}/usr/local"
-DCMAKE_INSTALL_PREFIX="${CONDA_PATH}" ..
2>&1 | tee CMAKE_OUT
CMake Error at CMakeLists.txt:8 (find_package):
By not providing "FindCaffe2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "Caffe2", but
CMake did not find one.

Could not find a package configuration file provided by "Caffe2" with any
of the following names:

Caffe2Config.cmake
caffe2-config.cmake

Add the installation prefix of "Caffe2" to CMAKE_PREFIX_PATH or set
"Caffe2_DIR" to a directory containing one of the above files. If "Caffe2"
provides a separate development package or SDK, be sure it has been
installed.

-- Configuring incomplete, errors occurred!
See also "/home/test/Downloads/translate/pytorch_translate/cpp/build/CMakeFiles/CMakeOutput.log".
`

obtain alignment/attention information

I would like to access information on alignment or even the full attention matrix. This is possible with the fairseq CLI (with the print-alignment flag) and to some extent also in the python interface to fairsec (as logging messages).
Is there a way to obtain this information with pytorch-translate?

unrecognized arguments: --batched-beam

It looks like --batched-beam has been removed from the ONNX export:

$ bash pytorch_translate/examples/export_iwslt14.sh

...

WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
usage: onnx_component_export.py [-h] [--path FILE]
                                [--encoder-output-file ENCODER_OUTPUT_FILE]
                                [--decoder-output-file DECODER_OUTPUT_FILE]
                                --source-vocab-file SOURCE_VOCAB_FILE
                                --target-vocab-file TARGET_VOCAB_FILE
                                [--beam-size BEAM_SIZE]
                                [--word-reward WORD_REWARD]
                                [--unk-reward UNK_REWARD] [--char-source]
onnx_component_export.py: error: unrecognized arguments: --batched-beam

Unable to export model

Hi,
After following instructions given in https://github.com/pytorch/translate#install-translate-from-source
I ran https://github.com/pytorch/translate/blob/master/pytorch_translate/examples/train_iwslt14.sh and also succesfully exported the model using https://github.com/pytorch/translate/blob/master/pytorch_translate/examples/export_iwslt14.sh to generate .pb files. But when I try to run echo "hallo welt" | bash pytorch_translate/examples/translate_iwslt14.sh
I get error saying that there is no translation_decoder in cpp/build directory. Did I do any mistake please elaborate the procedure so that the export to Caffe2 is easily reproducable.
Thanks in advance.

Are there any example of deep fusion?

class DeepFusionStrategy(MultiDecoderCombinationStrategy) seems that it's an implementation of deep fusion, but I'm not sure how to use it.

Could you please show me an example of deep fusion parameters exactly, like train_iwslt14.sh?

Following instructions to install gives compiler errors when building python on Ubuntu

Hello,

I tried to follow the instructions in the readme to install. First, I used my existing anaconda environment. However, it failed during the build of pytorch, the linker complaining about missing -pthreads among other things. I did some searching and found that maybe the anaconda compilers weren't installed, so I tried to install them with

conda install gcc_linux-64 gxx_linux-64 gfortran_linux-64

However, this was apparently the wrong compilers, since the build immediately failed with two unrecognized command line options: -fstack-protector-strong and -fno-plt

After that I thought I'd try using a fresh miniconda environment rather than my existing anaconda environment, so I did it over with following the miniconda installation steps in the README too. This time, it fails with linker being unable to find -lgcc_s, which again sounds a lot like a missing compiler install. Trying to install them with the same command line as above gives the same error.

Installing FAIR's tools - which are fantastic once I get them to work, thank you! - appear to be extremely sensitive to C++ compiler versions. The same with nvidia's tools... I stranded on this problem last time I tried to install pytorch from source, however then the new nightly build distributions of pytorch saved me then.

If you could provide similar bleeding edge binaries for ONNX and Caffe2, that'd be terrific! If not, if the build files could make sure the right compiler was used (and give sensible error messages if not) that would be a big improvement too.

ONNX export not working

pytorch_translate/examples/export_iwslt14.sh

This example doesn't work for me.
Would you be able to help me to run the example script?

I tried to fix the errors one by one, by removing parameters, but couldn't succeed to export.

1. onnx_component_export.py: error: unrecognized arguments: --batched-beam
2. File "/data/repos/translate/pytorch_translate/rnn.py", line 526, in build_single_decoder
   fp16=args.fp16,
   AttributeError: 'Namespace' object has no attribute 'fp16'
3. Exception has occurred: RuntimeError
   Error(s) in loading state_dict for RNNModel: Missing key(s) in state_dict: "encoder.bilstm.layers.0.weight_ih_l0", "encoder.bilstm.layers.0.weight_hh_l0", "encoder.bilstm.layers.0.bias_ih_l0", "encoder.bilstm.layers.0.bias_hh_l0", "encoder.bilstm.layers.0.weight_ih_l0_reverse", "encoder.bilstm.layers.0.weight_hh_l0_reverse", "encoder.bilstm.layers.0.bias_ih_l0_reverse", "encoder.bilstm.layers.0.bias_hh_l0_reverse", "encoder.bilstm.layers.1.weight_ih_l0", "encoder.bilstm.layers.1.weight_hh_l0", "encoder.bilstm.layers.1.bias_ih_l0", "encoder.bilstm.layers.1.bias_hh_l0". Unexpected key(s) in state_dict: "encoder.layers.0.weight_ih_l0", "encoder.layers.0.weight_hh_l0", "encoder.layers.0.bias_ih_l0", "encoder.layers.0.bias_hh_l0", "encoder.layers.0.weight_ih_l0_reverse", "encoder.layers.0.weight_hh_l0_reverse", "encoder.layers.0.bias_ih_l0_reverse", "encoder.layers.0.bias_hh_l0_reverse", "encoder.layers.1.weight_ih_l0", "encoder.layers.1.weight_hh_l0", "encoder.layers.1.bias_ih_l0", "encoder.layers.1.bias_hh_l0".
   File "\opt\conda\lib\python3.6\site-packages\torch\nn\modules\module.py", line 771, in load_state_dict
   File "\opt\conda\lib\python3.6\site-packages\fairseq\models\fairseq_model.py", line 66, in load_state_dict
   File "D:\repos\translate\pytorch_translate\ensemble_export.py", line 122, in load_models_from_checkpoints
   File "D:\repos\translate\pytorch_translate\ensemble_export.py", line 414, in build_from_checkpoints
   File "D:\repos\translate\pytorch_translate\onnx_component_export.py", line 116, in export
   File "D:\repos\translate\pytorch_translate\onnx_component_export.py", line 88, in main
   File "D:\repos\translate\pytorch_translate\onnx_component_export.py", line 156, in

depricated functions

p3 instance on AWS
Ubuntu 18.04
torch: 1.5.0.dev20200304
fairseq: 0.9.0
pytorch-translate: 0.1.0

from ~/translate
I run

bash pytorch-translate/examples/train-transformer.sh

where train-transformer.sh is:

#!/bin/bash

NCCL_ROOT_DIR="$(pwd)/nccl_2.1.15-1+cuda-10.1"
export NCCL_ROOT_DIR
LD_LIBRARY_PATH="${NCCL_ROOT_DIR}/lib:${LD_LIBRARY_PATH}"
export LD_LIBRARY_PATH
wget https://download.pytorch.org/models/translate/iwslt14/data.tar.gz
tar -xvzf data.tar.gz
rm -rf checkpoints data.tar.gz && mkdir -p checkpoints
CUDA_VISIBLE_DEVICES=0 python3 pytorch_translate/train.py
""
--arch ptt_transformer
--lr-scheduler inverse_sqrt
--log-verbose
--max-epoch 100
--stop-time-hr 72
--stop-no-best-bleu-eval 5
--optimizer adam
--lr 5e-4
--clip-norm 5.0
--criterion label_smoothed_cross_entropy
--label-smoothing 0.1
--batch-size 128
--length-penalty 0
--unk-reward -0.5
--word-reward 0.25
--max-tokens 4096
--save-dir checkpoints
--adam-betas '(0.9, 0.98)'
--num-avg-checkpoints 10
--beam 2
--no-beamable-mm
--source-lang de
--target-lang en
--train-source-text-file data/train.tok.bpe.de
--train-target-text-file data/train.tok.bpe.en
--dropout 0.3
--attention-dropout 0.3
--relu-dropout 0.3
--encoder-embed-dim 256
--encoder-ffn-embed-dim 256
--encoder-layers 4
--encoder-attention-heads 4
--decoder-embed-dim 256
--decoder-ffn-embed-dim 256
--decoder-layers 4
--decoder-attention-heads 4
--decoder-layerdrop 0.3
--eval-source-text-file data/valid.tok.bpe.de
--eval-target-text-file data/valid.tok.bpe.en
--source-max-vocab-size 14000
--target-max-vocab-size 14000
--log-interval 10
--seed "${RANDOM}"
2>&1 | tee -a checkpoints/log

I get:

...
[2020-03-08 18:16:15.546057] | Finished removing old checkpoint checkpoints/checkpoint100_end.pt.
pytorch_translate/train.py:693: UserWarning: Trainer.get_meter is deprecated. Please use fairseq.metrics instead.
  meter = trainer.get_meter(k)
/opt/conda/conda-bld/pytorch_1583309282142/work/torch/csrc/utils/python_arg_parser.cpp:739: UserWarning: This overload of add_ is deprecated:
	add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
	add_(Tensor other, Number alpha)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda/lib/python3.7/site-packages/fairseq-0.9.0-py3.7-linux-x86_64.egg/fairseq/logging/progress_bar.py", line 299, in _close_writers
    for w in _tensorboard_writers.values():
NameError: name '_tensorboard_writers' is not defined

fairseq version 0.5.0 requirement

It seems that on 12 jun there was a major update to fairseq, changing the library structure, and Translate has already started using this. The fairseq package installed in my miniconda environment following the installation instructions is 0.4.0. Running any translate command fails with

from fairseq import (
ImportError: cannot import name 'tasks'

I think this old version of fairseq was installed as part of the combined PyTorch and Caffe2 Conda package. pip/conda don't know about the newer version.

Could you update the installation instructions so that the current version of Translate can be run?

Unable to import onxx

When trying to export PyTorch model to a Caffe2 graph via ONNX, the script throws an error on importing onxx.
Import Error: undefined symbol: _ZNK6google8protobuf7Message11GetTypeNameEv
Kindly help explain the issue and also provide solution for the same.

unexpected keyword argument 'max_tokens'

I followed "Quickstart" section to install the framework.
When I run default training example (bash pytorch_translate/examples/train_iwslt14.sh) I get following error:

Traceback (most recent call last):
  File "pytorch_translate/train.py", line 974, in <module>
    main(args, single_process_main)
  File "pytorch_translate/train.py", line 945, in main
    return single_process_train(args)
  File "pytorch_translate/train.py", line 332, in single_process_main
    extra_state, trainer, task, epoch_itr = setup_training(args)
  File "pytorch_translate/train.py", line 285, in setup_training
    shard_id=args.distributed_rank,
TypeError: __init__() got an unexpected keyword argument 'max_tokens'

I am not quite sure what is exact problem here...
Please tell me if you need any additional information.

Unable - pull the docker

HI Team,

I am not pull Translate image from Docker. it is throwing error "unauthorized: authentication required". Can you please help us on this?

Here is Log

docker pull pytorch/translate

Using default tag: latest
latest: Pulling from pytorch/translate
297061f60c36: Pull complete
e9ccef17b516: Pull complete
dbc33716854d: Pull complete
8fe36b178d25: Pull complete
686596545a94: Pull complete
f611dfbee954: Pull complete
c51814f3e9ba: Pull complete
5da0fc07e73a: Pull complete
97462b1887aa: Extracting 64.06MB/477MB
924ea239f6fe: Downloading 469.4MB/585.2MB
f4a08faeb019: Download complete
d2a2cd1e3b42: Download complete
d714aef7a6eb: Download complete
a525fe235307: Downloading 62.29MB/69.73MB
6657161af193: Downloading
1839b2f91735: Waiting
a6034776fbac: Waiting
d1470d258ff1: Waiting
cff0a26dca2d: Waiting
4d6715d99c6a: Waiting

unauthorized: authentication required