SWBD Recipe Error about espresso

[freewym]

Does temporally unset LC_ALL as LC_ALL= python3 ../../scripts/spm_encode.py help?

[annamine]

still same error unfortunately

[freewym]

what if you set LC_ALL= around snippet:
LC_ALL=
cut -f 2- -d" " $text | \ python3 ../../scripts/spm_encode.py --model=${sentencepiece_model}.model --output_format=piece | \ paste -d" " <(cut -f 1 -d" " $text) - > $token_text cut -f 2- -d" " $token_text > $lmdatadir/$dataset.tokens
LC_ALL=C

[jinpoon]

I had the same issues for librispeech, I did LANG="" cut -f 2- -d" " $text | \ python3 ../../scripts/spm_encode.py .... and it worked for me.

[freewym]

Hmm... $LANG in my environment is en_US.UTF-8, and I don't have this problem. Maybe you can check your default $LANG value

[annamine]

My $LANG environment is en_GB.UTF-8 and I also tried to set LANG="" cut -f 2- -d" " $text | \ python3 ../../scripts/spm_encode.py .... but both still returning same error message

[annamine]

I have the same error for Librispeech recipe too

[freewym]

Sorry I am not in your environment so it's not easy for me to debug. I just googled the error message, and all I could find is export LANG=en_US.UTF-8 or export LC_ALL=en_US.UTF-8, or export PYTHONIOENCODING=utf-8.

[marthayifiru]

Hi,
Thanks a lot for the wonderful tool. I tried to built a model for African language using the WSJ recipe. Language and acoustic model training finished without error after setting LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, PYTHONIOENCODING=utf-8.

I have now error during decoding, the error message is
UnicodeEncodeError: 'ascii' codec cant' encode characters in position 11-14: ordinal not in range(128)

I tried to include various online recommendations for the problem in speech_recognize.py, but could not solve the problem.

Could you help.

[freewym]

Hi,
Thanks a lot for the wonderful tool. I tried to built a model for African language using the WSJ recipe. Language and acoustic model training finished without error after setting LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, PYTHONIOENCODING=utf-8.

I have now error during decoding, the error message is
UnicodeEncodeError: 'ascii' codec cant' encode characters in position 11-14: ordinal not in range(128)

I tried to include various online recommendations for the problem in speech_recognize.py, but could not solve the problem.

Could you help.

which line does it happen at?

[marthayifiru]

Hi,
Thanks for your prompt reply.

which line does it happen at?

The lines are 297, 293, 39 and 191 as shown in the following message.

loading model(s) from exp/lstm/checkpoint_best.pt:exp/lm_lstm/checkpoint_best.pt
LM fusion with Subword LM
using LM fusion with lm-weight=0.70
0%| | 0/26 [00:00<?, ?it/s]/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
Traceback (most recent call last):
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 297, in
cli_main()
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 293, in cli_main
main(args)
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 39, in main
return _main(args, h)
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 191, in _main
print('T-{}\t{}'.format(utt_id, detok_target_str), file=output_file)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 11-14: ordinal not in range(128)

Best regards,

[freewym]

I would first print T-{}\t{}'.format(utt_id, detok_target_str) to the screen to see if the string is displayed normally. If yes, then the problem may be when it gets written out to output_file, then I would try add encoding argument at line 38 as open(output_path, 'w', buffering=1, encoding='utf-8')

[annamine]

Hi just to give an update, I managed to run the section of code without error now by changing the global paths. Thanks for your help & advice!

[marthayifiru]

Thanks a lot. Adding the encoding argument at line 38 solved the problem. I can now decode without a problem. Thanks a lot.

Do you have a recipe for multilingual training?

Best regards.

[freewym]

Hi just to give an update, I managed to run the section of code without error now by changing the global paths. Thanks for your help & advice!

Cool. What do you mean by "global paths"?

[freewym]

Thanks a lot. Adding the encoding argument at line 38 solved the problem. I can now decode without a problem. Thanks a lot.

Do you have a recipe for multilingual training?

Best regards.

No. I don't have one yet.

[annamine]

Hi just to give an update, I managed to run the section of code without error now by changing the global paths. Thanks for your help & advice!

Cool. What do you mean by "global paths"?

I just needed to modify the path script, for my environment, to use the same en_US-UTF8 to stop the sorting error