Comments (17)
Does temporally unset LC_ALL
as LC_ALL= python3 ../../scripts/spm_encode.py
help?
from espresso.
still same error unfortunately
from espresso.
what if you set LC_ALL=
around snippet:
LC_ALL=
cut -f 2- -d" " $text | \ python3 ../../scripts/spm_encode.py --model=${sentencepiece_model}.model --output_format=piece | \ paste -d" " <(cut -f 1 -d" " $text) - > $token_text cut -f 2- -d" " $token_text > $lmdatadir/$dataset.tokens
LC_ALL=C
from espresso.
I had the same issues for librispeech, I did LANG="" cut -f 2- -d" " $text | \ python3 ../../scripts/spm_encode.py ....
and it worked for me.
from espresso.
Hmm... $LANG
in my environment is en_US.UTF-8
, and I don't have this problem. Maybe you can check your default $LANG
value
from espresso.
My $LANG environment is en_GB.UTF-8 and I also tried to set LANG="" cut -f 2- -d" " $text | \ python3 ../../scripts/spm_encode.py .... but both still returning same error message
from espresso.
I have the same error for Librispeech recipe too
from espresso.
Sorry I am not in your environment so it's not easy for me to debug. I just googled the error message, and all I could find is export LANG=en_US.UTF-8
or export LC_ALL=en_US.UTF-8
, or export PYTHONIOENCODING=utf-8
.
from espresso.
Hi,
Thanks a lot for the wonderful tool. I tried to built a model for African language using the WSJ recipe. Language and acoustic model training finished without error after setting LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, PYTHONIOENCODING=utf-8.
I have now error during decoding, the error message is
UnicodeEncodeError: 'ascii' codec cant' encode characters in position 11-14: ordinal not in range(128)
I tried to include various online recommendations for the problem in speech_recognize.py, but could not solve the problem.
Could you help.
from espresso.
Hi,
Thanks a lot for the wonderful tool. I tried to built a model for African language using the WSJ recipe. Language and acoustic model training finished without error after setting LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, PYTHONIOENCODING=utf-8.I have now error during decoding, the error message is
UnicodeEncodeError: 'ascii' codec cant' encode characters in position 11-14: ordinal not in range(128)I tried to include various online recommendations for the problem in speech_recognize.py, but could not solve the problem.
Could you help.
which line does it happen at?
from espresso.
Hi,
Thanks for your prompt reply.
which line does it happen at?
The lines are 297, 293, 39 and 191 as shown in the following message.
loading model(s) from exp/lstm/checkpoint_best.pt:exp/lm_lstm/checkpoint_best.pt
LM fusion with Subword LM
using LM fusion with lm-weight=0.70
0%| | 0/26 [00:00<?, ?it/s]/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
Traceback (most recent call last):
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 297, in
cli_main()
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 293, in cli_main
main(args)
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 39, in main
return _main(args, h)
File "/home/myt_002/espresso/examples/Tigrigna_E2E_ASR/../../espresso/speech_recognize.py", line 191, in _main
print('T-{}\t{}'.format(utt_id, detok_target_str), file=output_file)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 11-14: ordinal not in range(128)
Best regards,
from espresso.
I would first print T-{}\t{}'.format(utt_id, detok_target_str)
to the screen to see if the string is displayed normally. If yes, then the problem may be when it gets written out to output_file
, then I would try add encoding
argument at line 38 as open(output_path, 'w', buffering=1, encoding='utf-8')
from espresso.
Hi just to give an update, I managed to run the section of code without error now by changing the global paths. Thanks for your help & advice!
from espresso.
Thanks a lot. Adding the encoding argument at line 38 solved the problem. I can now decode without a problem. Thanks a lot.
Do you have a recipe for multilingual training?
Best regards.
from espresso.
Hi just to give an update, I managed to run the section of code without error now by changing the global paths. Thanks for your help & advice!
Cool. What do you mean by "global paths"?
from espresso.
Thanks a lot. Adding the encoding argument at line 38 solved the problem. I can now decode without a problem. Thanks a lot.
Do you have a recipe for multilingual training?
Best regards.
No. I don't have one yet.
from espresso.
Hi just to give an update, I managed to run the section of code without error now by changing the global paths. Thanks for your help & advice!
Cool. What do you mean by "global paths"?
I just needed to modify the path script, for my environment, to use the same en_US-UTF8 to stop the sorting error
from espresso.
Related Issues (20)
- Stream ASR HOT 1
- TypeError: get_asr_dataset_from_json() got an unexpected keyword argument 'combined' HOT 1
- Different WERs when decoding with different batch size (--max-sentences) HOT 5
- ONNX exportation of speech_lstm based model HOT 1
- Build PyChain with CPU HOT 6
- WER difference when decode with different batch size HOT 5
- SpecAug slows down training time HOT 4
- WSJ Recipe: "wsj_data_prep.sh: Spot check of command line arguments failed" HOT 1
- SpecAugment is not used due to a typo in the prefetch_called assignment HOT 1
- Getting OOM error at the middle of the training in asr_swbd recipe on lstm encoder decoder model HOT 4
- How can I find Kaldi directory? HOT 37
- Google Colab research creation HOT 1
- token_text as outputs HOT 2
- Which is the difference between stage 5 and stage 7 HOT 1
- Error in training stage of run_chain_e2e_bichar.sh: 'odict_items' object is not an iterator HOT 4
- Transformer LM in ASR HOT 2
- 'frame_subsampling_factor' in chain e2e setup HOT 2
- hydra.errors.ConfigCompositionException: Could not override 'task.data'. HOT 1
- Android Espresso not able to test fragement
- SHA hashes in 'main' branch are different from those in the 'origin/main' HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from espresso.