I have run all the procedures in run.sh for several days and finally got 'train_960_cleaned' for training the deep fsmn. But when I start training deep fsmn by running 'local/nnet/run_fsmn.sh DFSMN_S', it gives error:

`steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3_cleaned/ivectors_dev_other_hires using the extractor in exp/nnet3_cleaned/extractor.
steps/make_fbank.sh --nj 30 --cmd run.pl --fbank-config conf/fbank.conf data_fbank/train_960_cleaned exp/make_fbank/train_960_cleaned fbank/train_960_cleaned
steps/make_fbank.sh: moving data_fbank/train_960_cleaned/feats.scp to data_fbank/train_960_cleaned/.backup
utils/validate_data_dir.sh: Error: in data_fbank/train_960_cleaned, recording-ids extracted from wav.scp and reco2dur file
utils/validate_data_dir.sh: differ, partial diff is:
1,301545c1,281081
< 100-121669-0000-1
< 100-121669-0001-1
< 100-121669-0002-1
< 100-121669-0003-1
< 100-121669-0004-1
...

986-129388-0107
986-129388-0108
986-129388-0109
986-129388-0110
986-129388-0111
986-129388-0112
[Lengths are /tmp/kaldi.rudy/utts=301545 versus /tmp/kaldi.rudy/recordings.reco2dur=281081]`

It seems the number of records in file utts and file recordings.reco2dur is not the same, but validate_data_dir.sh expects them to be same. Does anyone know how to fix this? Any advice would be appreciated. Thanks!

hi, when running the run_fsmn_ivector.sh , the log/iter00.initial.log show "steps/nnet/train_faster_scheduler.sh: line 89: nnet-train-fsmn- streams: command not found" .how can I solve it?

前台打印是这样的：
5777
gmm-info ./exp/tri6b_cleaned/final.mdl
5776
run.pl: job failed, log is in exp/tri7b_DFSMN_L/_train_nnet.log

log文件最后是这样的：

RUNNING THE NN-TRAINING SCHEDULER

steps/nnet/train_faster_scheduler.sh --train-tool nnet-train-fsmn-streams --train-tool-opts --minibatch-size=4096 --feature-transform exp/tri7b_DFSMN_L/final.feature_transform --learn-rate 0.00001 --momentum 0.9
--start_half_lr 5 exp/tri7b_DFSMN_L/nnet.init ark:copy-feats scp:exp/tri7b_DFSMN_L/train.scp ark:- | apply-cmvn --norm-means=true --norm-vars=false
--utt2spk=ark:data_fbank/train_960_cleaned/utt2spk scp:data_fbank/train_960_cleaned/cmvn.scp ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- | ark:copy-feats scp:exp/tri7b_DFSMN_L/cv.scp ark:- | apply-cmvn --norm-means=true --norm-vars=false --utt2spk=ark:data_fbank/dev_clean/utt2spk scp:data_fbank/dev_clean/cmvn.scp ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- | ark:ali-to-pdf exp/tri6b_cleaned_ali_train_960_cleaned/final.mdl "ark:gunzip -c exp/tri6b_cleaned_ali_train_960_cleaned/ali..gz |" ark:- | ali-to-post ark:- ark:- | ark:ali-to-pdf exp/tri6b_cleaned_ali_train_960_cleaned/final.mdl "ark:gunzip -c exp/tri6b_cleaned_ali_dev_clean/ali..gz |" ark:- | ali-to-post ark:- ark:- | exp/tri7b_DFSMN_L
CROSSVAL PRERUN AVG.LOSS 8.6614 (Xent),
ITERATION 01: TRAIN AVG.LOSS 1.2125, (lrate1e-05), CROSSVAL AVG.LOSS 0.7093, nnet accepted (nnet_iter01_learnrate0.00001_tr1.2125_cv0.7093)
ITERATION 02: steps/nnet/train_faster_scheduler.sh: line 104: 37799 Aborted $train_tool --cross-validate=false --randomize=true --verbose=$verbose $train_tool_opts --learn-rate=$learn_rate --momentum=$momentum --l1-penalty=$l1_penalty --l2-penalty=$l2_penalty ${feature_transform:+ --feature-transform=$feature_transform} ${frame_weights:+ "--frame-weights=$frame_weights"} ${utt_weights:+ "--utt-weights=$utt_weights"} "$feats_tr_portion" "$labels_tr" $mlp_best $mlp_next 2>> $log

Accounting: time=47244 threads=1

Ended (code 1) at Thu Dec 5 05:13:12 CST 2019, elapsed time 47244 seconds

请问有人遇到过这个问题么？应该怎么解决，谢谢

When to extract the fbank feature, the fbank.cfg is not in conf dirs, so how can I get it?

run.sh error:
...
local/chain/run_tdnn.sh
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
utils/data/perturb_data_dir_speed_3way.sh: data/train_960_cleaned_sp/feats.scp already exists: refusing to run this (please delete data/train_960_cleaned_sp/feats.scp if you want this to run)

After I deleted this file, this error happen again.
Could anybody help us?

now DFSMN is use single GPU ,when ali privode muti gpu version
I use BMUF method for muti GPU ,but wer is not better than single GPU.

This script is intended to be used with GPUs but you have not compiled Kaldi with CUDA
If you want to use GPUs (and have them), go to src/, and configure and make on a machine
where "nvcc" is installed.

i see the warning, maybe it will not block the trainning, but could i know how to shorten the training period if there is no gpu. i think my machine is well configured, it has 256G memory and 26 processor, but after two weeks training, it only complet half of the run.sh script. anybody could provide help?

It seems that DFSMN does not support CUDA 10.

ASSERTION_FAILED (nnet-train-fsmn-streams[5.4.155~1-fdd8]:Eval():nnet-loss.cc:72) : 'KALDI_ISFINITE(net_out.Sum())'

有没有准确率较高的中文模型提供试用下啊，
或者有标记的语料让我们训练下

hi, there's no such file named train_faster.sh in steps/nnet/

When I run the command, I get the log:
Applying: add DFSMN related codes
/data/glusterfs_speech_04/11085090/kaldi/.git/rebase-apply/patch:300: trailing whitespace.

/data/glusterfs_speech_04/11085090/kaldi/.git/rebase-apply/patch:328: space before tab in indent.
steps/nnet/train_faster.sh --learn-rate $lrate --nnet-proto $proto
/data/glusterfs_speech_04/11085090/kaldi/.git/rebase-apply/patch:331: space before tab in indent.
--feat-type plain --splice 1
/data/glusterfs_speech_04/11085090/kaldi/.git/rebase-apply/patch:336: space before tab in indent.
$data_fbk/train_960_cleaned $data_fbk/dev_clean data/lang exp/tri6b_cleaned_ali_train_960_cleaned exp/tri6b_cleaned_ali_dev_clean $dir
/data/glusterfs_speech_04/11085090/kaldi/.git/rebase-apply/patch:343: space before tab in indent.
for set in $dataset
warning: squelched 115 whitespace errors
warning: 120 lines add whitespace errors.
There are many warnings, does it matter ?

My System info:
Ubuntu: 16.04
g++: 4.8.5
with no GPU
Quest:
after typing "local/nnet/run_fsmn_ivector.sh DFSMN_S", it indeed ran for a while, but then "local/nnet/run_fsmn_ivector.sh: line 27: syntax error near unexpected token `then' " came out.
why was that?

No file nnet-train-fsmn-streams in src/nnetbin

i realized that the training will cost much time, is there any place we can download the trained model, thanks in advance, looking forward your response.

阿里大帝,中文模型有没有?

where is the source code?

Errors 1: then
Errors 2: train_set=train_960_cleaned,I have no the train_960_cleaned dir,so use train dir
Errors 3: no feats.scp file,but can find it in dae dir and mfcc dir,which is ok for me
I simulate thchs30 data temple to build my data,run.sh get errors in dfsmn step,please help me!

I have a question that whether DFSMN is based on nnet3 or the older version.

运行命令：local/nnet/run_fsmn_ivector.sh DFSMN_S
之后提示：
cp: cannot stat 'data/train_960_cleaned': No such file or directory

i have complete the training, but not sure how to check the model is produced, if i want to export the model, what files should be included.

thanks in advance.

Error when updating the patch error: Failed to patch: src/cudamatrix/cu-matrix.h:693
Error: src/cudamatrix/cu-matrix.h: Patch not applied
What are the reasons for this?thanks

What should $cuda_cmd be? The same value with that of $train_cmd?