wenet-e2e / wespeaker

Research and Production Oriented Speaker Verification, Recognition and Diarization Toolkit

WeSpeaker

Roadmap | Docs | Paper | Runtime | Pretrained Models | Huggingface Demo | Modelscope Demo

WeSpeaker mainly focuses on speaker embedding learning, with application to the speaker verification task. We support online feature extraction or loading pre-extracted features in kaldi-format.

Installation

Install python package

pip install git+https://github.com/wenet-e2e/wespeaker.git

Command-line usage (use -h for parameters):

$ wespeaker --task embedding --audio_file audio.wav --output_file embedding.txt
$ wespeaker --task embedding_kaldi --wav_scp wav.scp --output_file /path/to/embedding
$ wespeaker --task similarity --audio_file audio.wav --audio_file2 audio2.wav
$ wespeaker --task diarization --audio_file audio.wav

Python programming usage:

import wespeaker

model = wespeaker.load_model('chinese')
embedding = model.extract_embedding('audio.wav')
utt_names, embeddings = model.extract_embedding_list('wav.scp')
similarity = model.compute_similarity('audio1.wav', 'audio2.wav')
diar_result = model.diarize('audio.wav')

Please refer to python usage for more command line and python programming usage.

Install for development & deployment

• Clone this repo

git clone https://github.com/wenet-e2e/wespeaker.git

• Create conda env: pytorch version >= 1.12.1 is recommended !!!

conda create -n wespeaker python=3.9
conda activate wespeaker
conda install pytorch=1.12.1 torchaudio=0.12.1 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install -r requirements.txt
pre-commit install  # for clean and tidy code

🔥 News

• 2024.05.15: Add support for quality-aware score calibration, see #320.
• 2024.04.25: Add support for the gemini-dfresnet model, see #291.
• 2024.04.23: Support MNN inference engine in runtime, see #310.
• 2024.04.02: Release Wespeaker document with detailed model-training tutorials, introduction of various runtime platforms, etc.
• 2024.03.04: Support the eres2net-cn-common-200k and campplus-cn-common-200k of damo #281, check python usage for details.
• 2024.02.05: Support the ERes2Net #272 and Res2Net #273 models.
• 2023.11.13: Support CLI usage of wespeaker, check python usage for details.
• 2023.07.18: Support the kaldi-compatible PLDA and unsupervised adaptation, see #186.
• 2023.07.14: Support the NIST SRE16 recipe, see #177.

Recipes

• VoxCeleb: Speaker Verification recipe on the VoxCeleb dataset
  • 🔥 UPDATE 2024.05.15: We support score calibration for Voxceleb and achieve better performance!
  • 🔥 UPDATE 2023.07.10: We support self-supervised learning recipe on Voxceleb! Achieving 2.627% (ECAPA_TDNN_GLOB_c1024) EER on vox1-O-clean test set without any labels.
  • 🔥 UPDATE 2022.10.31: We support deep r-vector up to the 293-layer version! Achieving 0.447%/0.043 EER/mindcf on vox1-O-clean test set
  • 🔥 UPDATE 2022.07.19: We apply the same setups as the CNCeleb recipe, and obtain SOTA performance considering the open-source systems
    • EER/minDCF on vox1-O-clean test set are 0.723%/0.069 (ResNet34) and 0.728%/0.099 (ECAPA_TDNN_GLOB_c1024), after LM fine-tuning and AS-Norm
• CNCeleb: Speaker Verification recipe on the CnCeleb dataset
  • 🔥 UPDATE 2024.05.16: We support score calibration for Cnceleb and achieve better EER.
  • 🔥 UPDATE 2022.10.31: 221-layer ResNet achieves 5.655%/0.330 EER/minDCF
  • 🔥 UPDATE 2022.07.12: We migrate the winner system of CNSRC 2022 report slides
    • EER/minDCF reduction from 8.426%/0.487 to 6.492%/0.354 after large margin fine-tuning and AS-Norm
• NIST SRE16: Speaker Verification recipe for the 2016 NIST Speaker Recognition Evaluation Plan. Similar recipe can be found in Kaldi.
  • 🔥 UPDATE 2023.07.14: We support NIST SRE16 recipe. After PLDA adaptation, we achieved 6.608%, 10.01%, and 2.974% EER on trial Pooled, Tagalog, and Cantonese, respectively.
• VoxConverse: Diarization recipe on the VoxConverse dataset

Discussion

For Chinese users, you can scan the QR code on the left to follow our offical account of WeNet Community. We also created a WeChat group for better discussion and quicker response. Please scan the QR code on the right to join the chat group.

Citations

If you find wespeaker useful, please cite it as

@inproceedings{wang2023wespeaker,
  title={Wespeaker: A research and production oriented speaker embedding learning toolkit},
  author={Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1--5},
  year={2023},
  organization={IEEE}
}

Looking for contributors

If you are interested to contribute, feel free to contact @wsstriving or @robin1001

wespeaker's Issues

DDP error

When I trained the VoxCeleb dataset, the following problem occurred that prevented the training. Is there a way to fix it? I am sure the environment was installed as prompted.

CRC error on VoxCeleb-ResNet34 pretrained checkpoint

Hi, thanks for sharing the pretrained checkpoints!
I found that a CRC error occurred when I unzip the checkpoint of VoxCeleb-ResNet34 (from https://wespeaker-1256283475.cos.ap-shanghai.myqcloud.com/models/voxceleb/voxceleb_resnet34_LM.zip) and the corrupted file cannot be loaded in torch.
Since the downloaded checkpoint of CNCeleb-ResNet3 works just fine, I'm wondering if there's something wrong with the VoxCeleb-ResNet34 one. Thanks.

Question about num_class used in pretrained model

Hi, I download the checkpoint of pre-trained ResNet-34 for voxceleb and find the num_class of projections is 17982. I think you use this num_class during training as well because the shape of projection.weight is [256,17982]. I am just curious why 17982 instead of 5994?

Finetuning on other dataset

Hi. How I can understand it's impossible to use a model which was trained on some larger dataset and then use it checkpoint to train on another small dataset, right?
When I'm trying this, I'm getting error:

wespeaker/wespeaker/utils/checkpoint.py", line 21, in load_checkpoint   
    model.load_state_dict(checkpoint, strict=False)                                                                                            
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1483, in load_state_dict                                      
    self.__class__.__name__, "\n\t".join(error_msgs)))                                                                                         
RuntimeError: Error(s) in loading state_dict for ResNet:                                                                                       
        size mismatch for projection.weight: copying a param with shape torch.Size([26685, 256]) from checkpoint, the shape in current model is
 torch.Size([7548, 256]).                                                                                                                      

How I understand its output shape, but I don't understand, where these numbers are to come from (26685, 7548). My first guess was that, it's a number of speakers, but I don't have so many speakers in my datasets.

怎么没有 eval 过程？

你好
发现训练过程没有验证集验证的过程？是这种任务不需要吗？能不能从训练集中每个speaker 取10%得数据，然后再自己生成一个trials， 这样每个epoch 后，计算这训练集 trials的eer？
谢谢！

如果想更快的训练，可以调哪些参数

你好
我想训练更快，可以调试哪些参数。 我简单的增加训练显卡,batch size 和 num_workers，发现并没有把速度加快？我的显卡是RTX3090， 内存大约有110G， CPU 使用htop 看 有80个运行核心线程。当训练时，这些资源其实都没有跑满。
下面是2块显卡训练，batch_size =128，num_workers = 16

[ INFO : 2023-01-18 14:57:38,151 ] - <========== Training process ==========>
[ INFO : 2023-01-18 14:57:38,152 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-18 14:57:38,152 ] - |     Epoch|     Batch|        Lr|    Margin|      Loss|       Acc|
[ INFO : 2023-01-18 14:57:38,152 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-18 14:58:31,274 ] - |         1|       100|  0.011281|         0|    8.7511|    0.1875|
[ INFO : 2023-01-18 14:58:31,274 ] - |         1|       100|  0.011281|         0|    8.7692|   0.14844|
[ INFO : 2023-01-18 14:58:59,495 ] - |         1|       200|  0.022479|         0|    8.7614|   0.21094|
[ INFO : 2023-01-18 14:58:59,498 ] - |         1|       200|  0.022479|         0|    8.7471|   0.21875|
[ INFO : 2023-01-18 14:59:26,335 ] - |         1|       300|  0.033482|         0|    8.7254|   0.26302|
[ INFO : 2023-01-18 14:59:26,335 ] - |         1|       300|  0.033482|         0|    8.7233|   0.26302|
[ INFO : 2023-01-18 14:59:58,885 ] - |         1|       400|  0.044291|         0|    8.6589|   0.35156|
[ INFO : 2023-01-18 14:59:58,887 ] - |         1|       400|  0.044291|         0|    8.6674|   0.35156|
[ INFO : 2023-01-18 15:00:25,761 ] - |         1|       500|   0.05491|         0|     8.494|   0.54844|
[ INFO : 2023-01-18 15:00:25,761 ] - |         1|       500|   0.05491|         0|    8.5094|   0.53125|
[ INFO : 2023-01-18 15:00:47,811 ] - |         1|       576|  0.062854|         0|    8.3458|   0.79753|
[ INFO : 2023-01-18 15:00:47,814 ] - |         1|       581|  0.063373|         0|    8.3217|   0.82293|
[ INFO : 2023-01-18 15:01:30,632 ] - |         2|       100|  0.073551|         0|    6.7583|    4.0859|
[ INFO : 2023-01-18 15:01:30,633 ] - |         2|       100|  0.073551|         0|    6.7547|    4.1094|
[ INFO : 2023-01-18 15:01:58,688 ] - |         2|       200|  0.083649|         0|    6.4056|    5.6602|
[ INFO : 2023-01-18 15:01:58,690 ] - |         2|       200|  0.083649|         0|    6.3935|    5.5039|
[ INFO : 2023-01-18 15:02:26,501 ] - |         2|       300|  0.093566|         0|    6.1018|    7.1901|
[ INFO : 2023-01-18 15:02:26,502 ] - |         2|       300|  0.093566|         0|    6.0818|    7.0052|
[ INFO : 2023-01-18 15:02:59,211 ] - |         2|       400|    0.1033|         0|     5.843|    8.7246|
[ INFO : 2023-01-18 15:02:59,212 ] - |         2|       400|    0.1033|         0|     5.816|    8.6348|
[ INFO : 2023-01-18 15:03:26,125 ] - |         2|       500|   0.11287|         0|    5.6205|    10.272|
[ INFO : 2023-01-18 15:03:26,126 ] - |         2|       500|   0.11287|         0|    5.5996|    10.189|
[ INFO : 2023-01-18 15:03:48,382 ] - |         2|       576|   0.12002|         0|    5.4428|    11.347|
[ INFO : 2023-01-18 15:03:48,385 ] - |         2|       581|   0.12048|         0|    5.4588|    11.576|
[ INFO : 2023-01-18 15:04:33,767 ] - |         3|       100|   0.12964|         0|    4.4194|    19.453|
[ INFO : 2023-01-18 15:04:33,768 ] - |         3|       100|   0.12964|         0|    4.3239|    19.703|
[ INFO : 2023-01-18 15:05:03,629 ] - |         3|       200|   0.13872|         0|    4.1974|    21.406|
[ INFO : 2023-01-18 15:05:03,631 ] - |         3|       200|   0.13872|         0|    4.2324|    21.465|
[ INFO : 2023-01-18 15:05:30,726 ] - |         3|       300|   0.14763|         0|    4.0818|    23.013|
[ INFO : 2023-01-18 15:05:30,728 ] - |         3|       300|   0.14763|         0|     4.118|    22.904|
[ INFO : 2023-01-18 15:06:01,690 ] - |         3|       400|   0.15638|         0|    3.9665|     24.59|
[ INFO : 2023-01-18 15:06:01,692 ] - |         3|       400|   0.15638|         0|    3.9983|    24.354|
[ INFO : 2023-01-18 15:06:28,560 ] - |         3|       500|   0.16496|         0|    3.8888|    25.794|
[ INFO : 2023-01-18 15:06:28,560 ] - |         3|       500|   0.16496|         0|    3.8642|    25.897|
[ INFO : 2023-01-18 15:06:50,724 ] - |         3|       576|   0.17138|         0|    3.8156|    26.794|
[ INFO : 2023-01-18 15:06:50,728 ] - |         3|       581|   0.17179|         0|     3.793|    26.877|
[ INFO : 2023-01-18 15:07:33,772 ] - |         4|       100|   0.18001|         0|    3.3482|    33.266|
[ INFO : 2023-01-18 15:07:33,772 ] - |         4|       100|   0.18001|         0|    3.3716|    32.461|
[ INFO : 2023-01-18 15:08:04,396 ] - |         4|       200|   0.18814|         0|    3.2329|    34.617|
[ INFO : 2023-01-18 15:08:04,398 ] - |         4|       200|   0.18814|         0|    3.2565|    34.195|
[ INFO : 2023-01-18 15:08:31,456 ] - |         4|       300|   0.19612|         0|    3.1699|    35.719|
[ INFO : 2023-01-18 15:08:31,467 ] - |         4|       300|   0.19612|         0|     3.193|    35.362|
[ INFO : 2023-01-18 15:09:01,868 ] - |         4|       400|   0.20395|         0|    3.0966|     37.02|
[ INFO : 2023-01-18 15:09:01,870 ] - |         4|       400|   0.20395|         0|    3.1204|    36.666|
[ INFO : 2023-01-18 15:09:28,683 ] - |         4|       500|   0.21163|         0|    3.0387|    37.911|
[ INFO : 2023-01-18 15:09:28,687 ] - |         4|       500|   0.21163|         0|    3.0469|    37.825|
[ INFO : 2023-01-18 15:09:51,764 ] - |         4|       573|   0.21714|         0|    3.0001|    38.572|
[ INFO : 2023-01-18 15:09:51,767 ] - |         4|       584|   0.21797|         0|    2.9838|    38.783|

这个是4块显卡训练，batch_size=128，num_workers = 16：

[ INFO : 2023-01-28 11:12:05,139 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-28 11:12:05,139 ] - |     Epoch|     Batch|        Lr|    Margin|      Loss|       Acc|
[ INFO : 2023-01-28 11:12:05,139 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-28 11:12:55,194 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:12:55,194 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:12:55,195 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:12:55,196 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:13:46,474 ] - |         1|       100|  0.044737|         0|     8.835|   0.14844|
[ INFO : 2023-01-28 11:13:46,488 ] - |         1|       100|  0.044737|         0|      8.83|   0.21875|
[ INFO : 2023-01-28 11:13:46,506 ] - |         1|       100|  0.044737|         0|    8.8506|   0.16406|
[ INFO : 2023-01-28 11:13:46,547 ] - |         1|       100|  0.044737|         0|    8.8426|   0.17187|
[ INFO : 2023-01-28 11:14:28,078 ] - |         1|       200|  0.088367|         0|    8.8855|   0.26953|
[ INFO : 2023-01-28 11:14:28,079 ] - |         1|       200|  0.088367|         0|    8.8919|   0.25391|
[ INFO : 2023-01-28 11:14:28,080 ] - |         1|       200|  0.088367|         0|    8.8931|   0.25781|
[ INFO : 2023-01-28 11:14:28,085 ] - |         1|       200|  0.088367|         0|    8.8911|   0.27344|
[ INFO : 2023-01-28 11:15:04,185 ] - |         1|       280|   0.12217|         0|    8.6288|    0.5692|
[ INFO : 2023-01-28 11:15:04,189 ] - |         1|       288|    0.1255|         0|    8.6059|   0.61849|
[ INFO : 2023-01-28 11:15:04,189 ] - |         1|       280|   0.12217|         0|    8.6433|   0.54129|
[ INFO : 2023-01-28 11:15:04,197 ] - |         1|       277|   0.12092|         0|    8.6579|   0.54998|
[ INFO : 2023-01-28 11:17:02,958 ] - |         2|       100|    0.1671|         0|    6.8822|    3.7422|
[ INFO : 2023-01-28 11:17:02,963 ] - |         2|       100|    0.1671|         0|    6.9869|    3.3203|
[ INFO : 2023-01-28 11:17:02,960 ] - |         2|       100|    0.1671|         0|     6.982|    3.3594|
[ INFO : 2023-01-28 11:17:02,978 ] - |         2|       100|    0.1671|         0|    6.8386|    3.6797|
[ INFO : 2023-01-28 11:17:47,129 ] - |         2|       200|   0.20642|         0|    6.3126|    6.0391|
[ INFO : 2023-01-28 11:17:47,131 ] - |         2|       200|   0.20642|         0|    6.3824|    5.7109|
[ INFO : 2023-01-28 11:17:47,137 ] - |         2|       200|   0.20642|         0|    6.2994|    6.1641|
[ INFO : 2023-01-28 11:17:47,140 ] - |         2|       200|   0.20642|         0|    6.3859|     5.625|
[ INFO : 2023-01-28 11:18:23,677 ] - |         2|       280|   0.23685|         0|     6.018|    7.8711|
[ INFO : 2023-01-28 11:18:23,678 ] - |         2|       280|   0.23685|         0|    5.9233|     8.365|
[ INFO : 2023-01-28 11:18:23,678 ] - |         2|       280|   0.23685|         0|    5.9489|    8.1613|
[ INFO : 2023-01-28 11:18:23,680 ] - |         2|       285|   0.23872|         0|    5.9775|    7.9989|
[ INFO : 2023-01-28 11:20:09,038 ] - |         3|       100|   0.27726|         0|    4.5193|    18.359|
[ INFO : 2023-01-28 11:20:09,039 ] - |         3|       100|   0.27726|         0|    4.5032|    18.867|
[ INFO : 2023-01-28 11:20:09,056 ] - |         3|       100|   0.27726|         0|    4.7157|     16.25|
[ INFO : 2023-01-28 11:20:09,088 ] - |         3|       100|   0.27726|         0|    4.4967|     19.07|
[ INFO : 2023-01-28 11:20:54,522 ] - |         3|       200|   0.31258|         0|    4.2442|    21.676|
[ INFO : 2023-01-28 11:20:54,523 ] - |         3|       200|   0.31258|         0|    4.2583|    21.707|
[ INFO : 2023-01-28 11:20:54,525 ] - |         3|       200|   0.31258|         0|    4.4396|    19.262|
[ INFO : 2023-01-28 11:20:54,528 ] - |         3|       200|   0.31258|         0|    4.2747|    21.465|
[ INFO : 2023-01-28 11:21:31,715 ] - |         3|       280|    0.3399|         0|    4.0777|    23.641|
[ INFO : 2023-01-28 11:21:31,715 ] - |         3|       285|   0.34158|         0|    4.2243|    21.812|
[ INFO : 2023-01-28 11:21:31,716 ] - |         3|       280|    0.3399|         0|    4.0897|    23.783|
[ INFO : 2023-01-28 11:21:31,716 ] - |         3|       280|    0.3399|         0|    4.0841|     23.77|

这个是2块显卡训练，batch_size=256，num_workers = 16：

[ INFO : 2023-01-28 11:33:22,513 ] - <========== Training process ==========>
[ INFO : 2023-01-28 11:33:22,514 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-28 11:33:22,514 ] - |     Epoch|     Batch|        Lr|    Margin|      Loss|       Acc|
[ INFO : 2023-01-28 11:33:22,514 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-28 11:34:02,143 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:34:02,143 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:34:53,235 ] - |         1|       100|  0.044737|         0|    8.8315|   0.19141|
[ INFO : 2023-01-28 11:34:53,241 ] - |         1|       100|  0.044737|         0|    8.8492|   0.17969|
[ INFO : 2023-01-28 11:35:47,455 ] - |         1|       200|  0.088367|         0|    8.8502|   0.24414|
[ INFO : 2023-01-28 11:35:47,456 ] - |         1|       200|  0.088367|         0|     8.865|   0.25781|
[ INFO : 2023-01-28 11:36:31,365 ] - |         1|       280|   0.12217|         0|    8.7101|   0.43666|
[ INFO : 2023-01-28 11:36:31,367 ] - |         1|       283|   0.12342|         0|    8.7135|   0.46516|
[ INFO : 2023-01-28 11:37:41,691 ] - |         2|       100|    0.1671|         0|    7.3378|    2.4961|
[ INFO : 2023-01-28 11:37:41,692 ] - |         2|       100|    0.1671|         0|    7.3062|    2.5664|
[ INFO : 2023-01-28 11:38:35,321 ] - |         2|       200|   0.20642|         0|    6.6031|    5.1016|
[ INFO : 2023-01-28 11:38:35,324 ] - |         2|       200|   0.20642|         0|    6.6145|    5.0293|
[ INFO : 2023-01-28 11:39:19,234 ] - |         2|       280|   0.23685|         0|    6.1876|    7.2349|
[ INFO : 2023-01-28 11:39:19,238 ] - |         2|       283|   0.23797|         0|    6.1842|    7.1803|
[ INFO : 2023-01-28 11:40:31,343 ] - |         3|       100|   0.27726|         0|    4.8013|    15.473|
[ INFO : 2023-01-28 11:40:31,343 ] - |         3|       100|   0.27726|         0|    4.8056|    15.891|
[ INFO : 2023-01-28 11:41:23,851 ] - |         3|       200|   0.31258|         0|    4.4971|    18.879|
[ INFO : 2023-01-28 11:41:23,854 ] - |         3|       200|   0.31258|         0|    4.4887|     18.77|
[ INFO : 2023-01-28 11:42:07,840 ] - |         3|       280|    0.3399|         0|    4.2963|    21.109|
[ INFO : 2023-01-28 11:42:07,843 ] - |         3|       283|   0.34091|         0|    4.2871|    20.974|

这个是2块显卡训练，batch_size=128，num_workers = 24：

[ INFO : 2023-01-28 11:45:43,429 ] - <========== Training process ==========>
[ INFO : 2023-01-28 11:45:43,429 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-28 11:45:43,429 ] - |     Epoch|     Batch|        Lr|    Margin|      Loss|       Acc|
[ INFO : 2023-01-28 11:45:43,429 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-28 11:46:09,936 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:46:09,936 ] - Reducer buckets have been rebuilt in this iteration.
[ INFO : 2023-01-28 11:46:37,103 ] - |         1|       100|  0.011281|         0|    8.7407|   0.14063|
[ INFO : 2023-01-28 11:46:37,109 ] - |         1|       100|  0.011281|         0|    8.7853|   0.17187|
[ INFO : 2023-01-28 11:47:04,761 ] - |         1|       200|  0.022479|         0|    8.8746|   0.19141|
[ INFO : 2023-01-28 11:47:04,762 ] - |         1|       200|  0.022479|         0|    8.8379|   0.19531|
[ INFO : 2023-01-28 11:47:32,274 ] - |         1|       300|  0.033482|         0|    8.8437|   0.29427|
[ INFO : 2023-01-28 11:47:32,278 ] - |         1|       300|  0.033482|         0|    8.8078|   0.26823|
[ INFO : 2023-01-28 11:47:59,533 ] - |         1|       400|  0.044291|         0|    8.6562|   0.49219|
[ INFO : 2023-01-28 11:47:59,534 ] - |         1|       400|  0.044291|         0|    8.6269|   0.45117|
[ INFO : 2023-01-28 11:48:26,122 ] - |         1|       500|   0.05491|         0|    8.3694|   0.79375|
[ INFO : 2023-01-28 11:48:26,123 ] - |         1|       500|   0.05491|         0|    8.3984|   0.81719|
[ INFO : 2023-01-28 11:48:46,078 ] - |         1|       568|  0.062023|         0|    8.1982|    1.2035|
[ INFO : 2023-01-28 11:48:46,081 ] - |         1|       573|  0.062542|         0|    8.1674|    1.1807|
[ INFO : 2023-01-28 11:49:44,856 ] - |         2|       100|  0.073551|         0|    6.3352|     5.875|
[ INFO : 2023-01-28 11:49:44,863 ] - |         2|       100|  0.073551|         0|    6.3675|    5.6797|
[ INFO : 2023-01-28 11:50:12,738 ] - |         2|       200|  0.083649|         0|    6.0153|    7.4336|
[ INFO : 2023-01-28 11:50:12,739 ] - |         2|       200|  0.083649|         0|    6.0508|    7.1484|
[ INFO : 2023-01-28 11:50:41,223 ] - |         2|       300|  0.093566|         0|    5.7454|    9.0313|
[ INFO : 2023-01-28 11:50:41,235 ] - |         2|       300|  0.093566|         0|    5.7974|    8.5911|
[ INFO : 2023-01-28 11:51:08,609 ] - |         2|       400|    0.1033|         0|     5.559|    10.252|
[ INFO : 2023-01-28 11:51:08,611 ] - |         2|       400|    0.1033|         0|    5.5118|    10.562|
[ INFO : 2023-01-28 11:51:35,370 ] - |         2|       500|   0.11287|         0|    5.3533|    11.811|
[ INFO : 2023-01-28 11:51:35,372 ] - |         2|       500|   0.11287|         0|     5.319|    12.017|
[ INFO : 2023-01-28 11:51:55,559 ] - |         2|       568|   0.11927|         0|    5.1925|    13.123|
[ INFO : 2023-01-28 11:51:55,563 ] - |         2|       573|   0.11974|         0|     5.218|    12.931|
[ INFO : 2023-01-28 11:52:53,000 ] - |         3|       100|   0.12964|         0|    4.2709|    21.172|
[ INFO : 2023-01-28 11:52:53,008 ] - |         3|       100|   0.12964|         0|     4.215|    21.242|
[ INFO : 2023-01-28 11:53:21,961 ] - |         3|       200|   0.13872|         0|     4.063|    23.609|
[ INFO : 2023-01-28 11:53:21,967 ] - |         3|       200|   0.13872|         0|    4.0894|     22.93|
[ INFO : 2023-01-28 11:53:49,669 ] - |         3|       300|   0.14763|         0|    3.9379|    24.917|
[ INFO : 2023-01-28 11:53:49,670 ] - |         3|       300|   0.14763|         0|    3.9662|    24.753|
[ INFO : 2023-01-28 11:54:16,975 ] - |         3|       400|   0.15638|         0|    3.8292|    26.402|
[ INFO : 2023-01-28 11:54:16,978 ] - |         3|       400|   0.15638|         0|    3.8593|    26.064|
[ INFO : 2023-01-28 11:54:43,775 ] - |         3|       500|   0.16496|         0|    3.7385|    27.598|
[ INFO : 2023-01-28 11:54:43,777 ] - |         3|       500|   0.16496|         0|    3.7602|    27.333|
[ INFO : 2023-01-28 11:55:04,015 ] - |         3|       568|    0.1707|         0|     3.693|    28.239|
[ INFO : 2023-01-28 11:55:04,018 ] - |         3|       573|   0.17112|         0|    3.6736|    28.576|

上面看基本都是3分钟一个epoch。

local/download_data.sh: line 43: syntax error near unexpected token `fi'

In exmples/cnceleb/v2.
It seems that a fi is missing in that loop.

The error in gpu training

Err info：/opt/conda/conda-bld/pytorch_1659484809662/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:365: operator(): block: [0,0,0], thread: [24,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.

怎么用vscode 调试train.py

你好
我对这个项目非常感兴趣，想用vscode调试一下trian的过程。仿照run.sh的写法，修改train.py的main 函数，如下：

    if __name__ == '__main__':
    fire.Fire(train(config ='/home/yangjie/wespeaker/examples/cnceleb/v2/conf/resnet.yaml', exp_dir = '/home/yangjie/wespeaker/examples/cnceleb/v2/exp/test',
                    gpus = '[0]', data_type ='share',train_data ='/home/yangjie/wespeaker_train_data/cnceleb_train/shard_test.list',
                    train_label = '/home/yangjie/wespeaker_train_data/cnceleb_train/utt2spk',
                    reverb_data = '/home/yangjie/wespeaker_train_data/rirs/lmdb',
                    noise_data = '/home/yangjie/wespeaker_train_data/musan/lmdb'))

但是调试的时候还是在 https://github.com/wenet-e2e/wespeaker/blob/master/wespeaker/bin/train.py#L47 这里出错，感觉需要设置一些环境变量，但是我不知道怎么设置，因为以前都是使用 pytorch lightning 训练。
能指导一下应该怎么修改trian.py main 函数，才能单步调试吗？
谢谢！

train with voxceleb1&2 dev set

Hi,

Thank you for sharing your amazing work. I am trying to use your work in my research. However, the current Voxceleb recipe just allows training on Voxceleb 2 dev set and testing on Voxceleb 1 eval set. Could I ask how to perform model training on the Voxceleb1&2 dev set under the shard input format?

Thank you for your support!

怎么创建自己的cn-celeb数据集呢？我想利用现有的run.sh进行训练。有没有生成数据集的脚本呢？

使用进行多语种识别出错

在使用四语种分类时正常，用同样的代码进行多语种分类时出错，模型不收敛，这有可能是什么原因导致的呢

add reverberation and noise related probability issues

If aug_prob=0.6, does that mean there is only a 0.4 chance of adding reverb and noise. Is this something that should be used less than the sign in order to add noise with a probability of 0.6.

if aug_prob > random.random():

infer_onnx.py的waveform * (1<<15)的操作

infer_onnx.py的waveform * (1<<15)的操作是为什么呢？

How to train ResNet293

Hi, how can I train ResNet293? In config files, maximum size is 152. Should I write a special config for Resnet293?

about the dino logger

非常感谢您可以开源这个项目，我最近再根据您的项目复现dino，您可以提供一下eacpa-tdnn在dino的训练日志吗，我前几轮的eer我发现都有14.多。 非常感谢

Issue with generating nan values ​​in sphereface2 loss function

I am a speech processing researcher who studies speaker recognition using the wespeaker tool.

A new loss function, the sphereface2 module, was introduced in the wespeaker tool, so I tested it using voxceleb DB.

As shown in the log below, it was observed that a Nan value was generated in the loss value.

Experimenting with the same settings, in some cases no nan values ​​occur and training ends.

However, when experimenting with the same settings, nan values ​​are generated.

In the v2/conf/resnet.yaml file, the model name was "ResNet152", and the project_type was set to "sphereface2" to experiment.

Can you comment on what causes the nan values?

[train log]
[ INFO : 2023-07-06 07:55:30,898 ] - | 3| 200| 0.52112| 0| 5.2225| 9.7266|
[ INFO : 2023-07-06 07:55:30,898 ] - | 3| 200| 0.52112| 0| 3.8506| 18.965|
[ INFO : 2023-07-06 07:55:30,898 ] - | 3| 200| 0.52112| 0| 3.9819| 17.094|
[ INFO : 2023-07-06 07:55:30,898 ] - | 3| 200| 0.52112| 0| 3.8734| 18.602|
[ INFO : 2023-07-06 07:55:30,898 ] - | 3| 200| 0.52112| 0| 4.0083| 16.66|
[ INFO : 2023-07-06 07:55:30,899 ] - | 3| 200| 0.52112| 0| 3.8383| 18.98|
[ INFO : 2023-07-06 07:55:30,903 ] - | 3| 200| 0.52112| 0| 3.9939| 17.066|
[ INFO : 2023-07-06 07:56:53,859 ] - | 3| 300| 0.54052| 0| nan| 17.698|
[ INFO : 2023-07-06 07:56:53,860 ] - | 3| 300| 0.54052| 0| nan| 16.5|
[ INFO : 2023-07-06 07:56:53,860 ] - | 3| 300| 0.54052| 0| nan| 10.859|
[ INFO : 2023-07-06 07:56:53,860 ] - | 3| 300| 0.54052| 0| nan| 18.258|
[ INFO : 2023-07-06 07:56:53,860 ] - | 3| 300| 0.54052| 0| nan| 16.273|
[ INFO : 2023-07-06 07:56:53,861 ] - | 3| 300| 0.54052| 0| nan| 18.099|
[ INFO : 2023-07-06 07:56:53,861 ] - | 3| 300| 0.54052| 0| nan| 17.799|
[ INFO : 2023-07-06 07:56:53,864 ] - | 3| 300| 0.54052| 0| nan| 16.539|
[ INFO : 2023-07-06 07:58:16,756 ] - | 3| 400| 0.55973| 0| nan| 8.1445|
[ INFO : 2023-07-06 07:58:16,756 ] - | 3| 400| 0.55973| 0| nan| 12.375|
[ INFO : 2023-07-06 07:58:16,756 ] - | 3| 400| 0.55973| 0| nan| 13.574|
[ INFO : 2023-07-06 07:58:16,756 ] - | 3| 400| 0.55973| 0| nan| 13.35|
[ INFO : 2023-07-06 07:58:16,756 ] - | 3| 400| 0.55973| 0| nan| 12.205|
[ INFO : 2023-07-06 07:58:16,756 ] - | 3| 400| 0.55973| 0| nan| 13.693|
[ INFO : 2023-07-06 07:58:16,757 ] - | 3| 400| 0.55973| 0| nan| 13.273|
[ INFO : 2023-07-06 07:58:16,760 ] - | 3| 400| 0.55973| 0| nan| 12.404|
[ INFO : 2023-07-06 07:59:40,291 ] - | 3| 500| 0.57875| 0| nan| 10.619|
[ INFO : 2023-07-06 07:59:40,292 ] - | 3| 500| 0.57875| 0| nan| 10.68|
[ INFO : 2023-07-06 07:59:40,292 ] - | 3| 500| 0.57875| 0| nan| 9.9|
[ INFO : 2023-07-06 07:59:40,292 ] - | 3| 500| 0.57875| 0| nan| 6.5156|
[ INFO : 2023-07-06 07:59:40,292 ] - | 3| 500| 0.57875| 0| nan| 10.859|
[ INFO : 2023-07-06 07:59:40,292 ] - | 3| 500| 0.57875| 0| nan| 9.7641|
[ INFO : 2023-07-06 07:59:40,293 ] - | 3| 500| 0.57875| 0| nan| 10.955|
[ INFO : 2023-07-06 07:59:40,296 ] - | 3| 500| 0.57875| 0| nan| 9.9234|
[ INFO : 2023-07-06 08:01:06,822 ] - | 3| 600| 0.59758| 0| nan| 8.849|
[ INFO : 2023-07-06 08:01:06,822 ] - | 3| 600| 0.59758| 0| nan| 5.4297|
[ INFO : 2023-07-06 08:01:06,823 ] - | 3| 600| 0.59758| 0| nan| 9.1289|
[ INFO : 2023-07-06 08:01:06,823 ] - | 3| 600| 0.59758| 0| nan| 8.8997|
[ INFO : 2023-07-06 08:01:06,823 ] - | 3| 600| 0.59758| 0| nan| 8.25|
[ INFO : 2023-07-06 08:01:06,823 ] - | 3| 600| 0.59758| 0| nan| 8.1367|
[ INFO : 2023-07-06 08:01:06,823 ] - | 3| 600| 0.59758| 0| nan| 9.0495|
[ INFO : 2023-07-06 08:01:06,827 ] - | 3| 600| 0.59758| 0| nan| 8.2695|
[ INFO : 2023-07-06 08:02:29,752 ] - | 3| 700| 0.61623| 0| nan| 7.0714|
[ INFO : 2023-07-06 08:02:29,752 ] - | 3| 700| 0.61623| 0| nan| 4.654|
[ INFO : 2023-07-06 08:02:29,752 ] - | 3| 700| 0.61623| 0| nan| 7.6283|
[ INFO : 2023-07-06 08:02:29,752 ] - | 3| 700| 0.61623| 0| nan| 7.7567|
[ INFO : 2023-07-06 08:02:29,752 ] - | 3| 700| 0.61623| 0| nan| 7.5848|
[ INFO : 2023-07-06 08:02:29,753 ] - | 3| 700| 0.61623| 0| nan| 7.8248|
[ INFO : 2023-07-06 08:02:29,753 ] - | 3| 700| 0.61623| 0| nan| 6.9743|
[ INFO : 2023-07-06 08:02:29,756 ] - | 3| 700| 0.61623| 0| nan| 7.0882|
[ INFO : 2023-07-06 08:03:52,705 ] - | 3| 800| 0.63469| 0| nan| 6.6367|
[ INFO : 2023-07-06 08:03:52,706 ] - | 3| 800| 0.63469| 0| nan| 6.1875|
[ INFO : 2023-07-06 08:03:52,706 ] - | 3| 800| 0.63469| 0| nan| 4.0723|
[ INFO : 2023-07-06 08:03:52,706 ] - | 3| 800| 0.63469| 0| nan| 6.8467|
[ INFO : 2023-07-06 08:03:52,706 ] - | 3| 800| 0.63469| 0| nan| 6.7871|
[ INFO : 2023-07-06 08:03:52,706 ] - | 3| 800| 0.63469| 0| nan| 6.6748|
[ INFO : 2023-07-06 08:03:52,706 ] - | 3| 800| 0.63469| 0| nan| 6.1025|
[ INFO : 2023-07-06 08:03:52,710 ] - | 3| 800| 0.63469| 0| nan| 6.2021|
[ INFO : 2023-07-06 08:05:18,838 ] - | 3| 900| 0.65296| 0| nan| 5.5|
[ INFO : 2023-07-06 08:05:18,839 ] - | 3| 900| 0.65296| 0| nan| 3.6198|
[ INFO : 2023-07-06 08:05:18,839 ] - | 3| 900| 0.65296| 0| nan| 6.033|
[ INFO : 2023-07-06 08:05:18,839 ] - | 3| 900| 0.65296| 0| nan| 6.0859|
[ INFO : 2023-07-06 08:05:18,839 ] - | 3| 900| 0.65296| 0| nan| 5.8993|
[ INFO : 2023-07-06 08:05:18,839 ] - | 3| 900| 0.65296| 0| nan| 5.4245|
[ INFO : 2023-07-06 08:05:18,839 ] - | 3| 900| 0.65296| 0| nan| 5.9332|
[ INFO : 2023-07-06 08:05:18,842 ] - | 3| 900| 0.65296| 0| nan| 5.513|
[ INFO : 2023-07-06 08:06:41,560 ] - | 3| 1000| 0.67105| 0| nan| 5.3094|
[ INFO : 2023-07-06 08:06:41,561 ] - | 3| 1000| 0.67105| 0| nan| 5.4297|
[ INFO : 2023-07-06 08:06:41,561 ] - | 3| 1000| 0.67105| 0| nan| 5.4773|
[ INFO : 2023-07-06 08:06:41,561 ] - | 3| 1000| 0.67105| 0| nan| 4.95|
[ INFO : 2023-07-06 08:06:41,561 ] - | 3| 1000| 0.67105| 0| nan| 5.3398|
[ INFO : 2023-07-06 08:06:41,561 ] - | 3| 1000| 0.67105| 0| nan| 4.882|
[ INFO : 2023-07-06 08:06:41,562 ] - | 3| 1000| 0.67105| 0| nan| 3.2578|
[ INFO : 2023-07-06 08:06:41,564 ] - | 3| 1000| 0.67105| 0| nan| 4.9617|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1082| 0.68575| 0| nan| 5.0622|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1088| 0.68682| 0| nan| 4.8799|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1080| 0.68539| 0| nan| 4.5833|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1088| 0.68682| 0| nan| 4.9079|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1080| 0.68539| 0| nan| 3.0165|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1080| 0.68539| 0| nan| 4.5942|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1080| 0.68539| 0| nan| 4.5204|
[ INFO : 2023-07-06 08:07:55,262 ] - | 3| 1088| 0.68682| 0| nan| 4.9905|
[ INFO : 2023-07-06 08:09:32,093 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:09:32,093 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:09:32,093 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:09:32,093 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:09:32,093 ] - | 4| 100| 0.70475| 0| nan| 0.0078125|
[ INFO : 2023-07-06 08:09:32,094 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:09:32,093 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:09:32,096 ] - | 4| 100| 0.70475| 0| nan| 0|
[ INFO : 2023-07-06 08:11:00,766 ] - | 4| 200| 0.72232| 0| nan| 0|
[ INFO : 2023-07-06 08:11:00,766 ] - | 4| 200| 0.72232| 0| nan| 0|

Resnet 152 model cuda memory error (large-margin finetuning)

Thanks for sharing your very effective training code and pretrained model.

In order to reproduce the results of the 'wespeaker', we are experimenting with resnet152/221/293 while changing the batch size and chunk size, but an error occurred.

The training GPU environment is A100 (GPU: 0~7, 80GB memory).

However, when batch size (=64) and num_frms (=600) are applied in resnet_lm.yaml for large-margine finetuning, a cuda memory error occurs as shown in the figure below.

Could you please comment on what caused the error?

[ INFO : 2023-05-09 08:51:31,879 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-05-09 08:51:31,879 ] - | Epoch| Batch| Lr| Margin| Loss| Acc|
[ INFO : 2023-05-09 08:51:31,879 ] - +----------+----------+----------+----------+----------+----------+
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 4; 79.17 GiB total capacity; 76.47 GiB already allocated; 113.81 MiB free; 76.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 5; 79.17 GiB total capacity; 76.47 GiB already allocated; 113.81 MiB free; 76.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 2; 79.17 GiB total capacity; 76.47 GiB already allocated; 113.81 MiB free; 76.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 3; 79.17 GiB total capacity; 76.47 GiB already allocated; 113.81 MiB free; 76.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 0; 79.17 GiB total capacity; 76.47 GiB already allocated; 135.81 MiB free; 76.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 7; 79.17 GiB total capacity; 76.47 GiB already allocated; 135.81 MiB free; 76.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 6; 79.17 GiB total capacity; 76.47 GiB already allocated; 113.81 MiB free; 76.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 238, in
fire.Fire(train)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/work_speaker_recognition/wespeaker/examples/voxceleb/v2/wespeaker/bin/train.py", line 210, in train
run_epoch(train_dataloader,
File "/data/work_speaker_recognition/wespeaker/wespeaker/utils/executor.py", line 61, in run_epoch
outputs = model(features) # (embed_a,embed_b) in most cases
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 971, in _run_ddp_forward
return module_to_run(*inputs, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 179, in forward
out = self.layer4(out)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/work_speaker_recognition/wespeaker/wespeaker/models/resnet.py", line 106, in forward
out += self.shortcut(x)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
return F.batch_norm(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
return torch.batch_norm(
RuntimeError: CUDA out of memory. Tried to allocate 188.00 MiB (GPU 1; 79.17 GiB total capacity; 76.47 GiB already allocated; 113.81 MiB free; 76.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 216843) of binary: /home/azureuser/miniconda3/envs/wespeaker/bin/python
Traceback (most recent call last):
File "/home/azureuser/miniconda3/envs/wespeaker/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==1.12.1', 'console_scripts', 'torchrun')())
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 345, in wrapper
return f(*args, **kwargs)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/run.py", line 761, in main
run(args)
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/azureuser/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

关于wespeaker提取说话人嵌入

请问该工具是否可以使用mel谱图，或者语谱图作为输入进行speake embedding，因为好多语音合成任务中都是使用mel谱图作为模型的输入和输出

Get stuck in load a batch

Hi, dear.

I am following the settings of the original script, using the shards data type, I want to know, is your experiment using SSD? Is it because of the fuck mechanical hard disk that it has been stuck in load a batch? It is not very slow, but it is always stuck there forever.

I have a record about it

`

    print("before get a batch")
    for i, batch in enumerate(dataloader):
        print("get a batch")
        utts = batch['key']
        targets = batch['label']
        features = batch['feat']

`

after, ... of, there is no after.

voxcelab/v2/local/prepare_data.sh 运行错误

你好
voxcelab/v2/local/prepare_data.sh line 83 -105行的 $(pwd) 是不是应该去掉，rawdata_dir 就是完整的路径了？对比cnceleb的prepare_data.sh 就没有这个参数？

How to evaluate

Hi,

Could you tell me which script I need to run in order to evaluate the pretrained model for voxceleb? I would like to test it with a different dataset.

Thank you

mini-batch with no duplicate speaker_id

Hi Wespeaker team,

Thank you for sharing your amazing work. I am trying to use your work in my research. However, I am not familiar with the data sampler using your toolkit. Could I ask how to modify your data sampler so that in a mini-batch, each training sample has a different speaker_id (no duplicate speaker_id in a mini-batch)?

Thank you for your support!

在框架中引入新的模型出现：RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.

[bug] Multi jobs running on one node

Hi,

Thank you for developing this amazing toolkit. I am currently running my experiments with your toolkit. However, if I run multiple experiments on one computing node, I notice that if one job finish first, it will cause the following errors for the remaining running jobs:

WARNING:torch.distributed.elastic.rendezvous.dynamic_rendezvous:The node 'node04.localdomain_3882442_0' has failed to shutdown the rendezvous '9987b766-350c-48bf-aa45-cfc2da182f33' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
RuntimeError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tuantruong001/miniconda3/envs/wespeaker/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.12.1', 'console_scripts', 'torchrun')())
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/run.py", line 761, in main
    run(args)
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 236, in launch_agent
    result = agent.run()
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper
    result = f(*args, **kwargs)
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 709, in run
    result = self._invoke_run(role)
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 881, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1079, in num_nodes_waiting
    self._state_holder.sync()
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 408, in sync
    get_response = self._backend.get_state()
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/home/tuantruong001/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.

I am asking whether you encountered this issue. If yes, could you guide me how to fix this bug?
Once again, thank you for sharing this toolkit and helping us.

windows 编译 onnxruntime 后，执行提取部分音频embedding会出现异常

执行命令以及输出日志：

extract_emb_main.exe --wav_list wav_scp.txt                        --result result.txt                        --speaker_model_path models/cnceleb_resnet34_LM.onnx                        --embedding_size 256                        --SamplesPerChunk  80000
I1008 10:29:00.920819  6684 extract_emb_main.cc:39] Init model ...
I1008 10:29:00.922801  6684 speaker_engine.cc:37] Reading model models/cnceleb_resnet34_LM.onnx
I1008 10:29:00.922801  6684 speaker_engine.cc:39] Embedding size: 256
I1008 10:29:00.922801  6684 speaker_engine.cc:41] per_chunk_samples: 80000
I1008 10:29:00.922801  6684 speaker_engine.cc:43] Sample rate: 16000
I1008 10:29:02.459808  6684 onnx_speaker_model.cc:60] Ouput name: feats
I1008 10:29:02.465803  6684 onnx_speaker_model.cc:68] Output name: embs
I1008 10:29:02.468802  6684 extract_emb_main.cc:44] embedding size: 256
I1008 10:29:03.410821  6684 extract_emb_main.cc:84] process: aa-01 RTF: 0.171875
I1008 10:29:03.469822  6684 extract_emb_main.cc:84] process: aa-02 RTF: 0.0070864

运行过程中，出现弹窗错误，错误信息如下：

Microsoft Visual C++ Runtime Library
Debug Assertion Failed!
Program:
...aker runtime onnxruntime build bin\Debug extract emb main.exeFile: G: Program Files Microsoft VisualStudio 2022 Community VC Tool MSVC 14.34.31933 include vectorLine: 112
Expression: cannot seek vector iterator after end
For information on how your program can cause an assertionfailure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)

音频文件：
wav.zip

wespeaker的预训练模型测试声纹对比报错

使用wespeaker的预训练模型测试声纹对比报错，显示没有extract_embedding_wav该方法，你们遇到过吗

外部训练集上，同一模型，不同训练轮次的模型推理速度差异很大

你好，我在使用外部数据集进行训练的时候，使用的默认的ecapa的config。训练完成后，所保存的不同轮次的模型，会出现推理速度上的差异，最大的差异会到10倍左右。请问这里该如何解决？

Setting dither = 0 during inference

Hi,

During training, features are extracted with dither=1.0. However, during inference, features are forced to be extracted with dither=0.0. Is it a bug?

For training speed

I try to train the ResNet model on voxceleb2 with 3 2080Ti and find that the training is slow (about 6 hours for one epoch). Is it normal and is there any function to reduce the training time?

[Feature Wanted] EEND: speaker diraization

QMF and fusion in VoxCeleb Speaker Recognition Challenge 2023 papers

greatly thanks to the project of speaker verification. in VoxSRC 2023, many team use QMF and ASnorm to improve the score, seems QMF is as good as ASnorm, if it is possible to add the QMF in this project? and they also use Fusion to fuse different model result, seems this can improve performance too. sorry to ask these requirement.

Training instruction

Is there any guide to launch training? I tried to run run.sh, but nothing is printed(

For pretrain model

Thanks for your amazing open source code, very great work and I learnt a lot from that, really appreciate!

So if you have trained it, could you please share the pretrained model (.pt file) on the VoxCeleb2 for the large-scale ResNet? Such as ResNet152 or ResNet101 model? Thanks a lot!

windows 编译有错误

你好，我这边Windows编译出现如下错误：

       行 523: ..\speaker\speaker_engine.cc(163): error C2039: "inner_product": 不是 "std" 的成员
	行 525: ..\speaker\speaker_engine.cc(163): error C3861: “inner_product”: 找不到标识符
	行 526: ..\speaker\speaker_engine.cc(164): error C2039: "inner_product": 不是 "std" 的成员
	行 528: ..\speaker\speaker_engine.cc(164): error C3861: “inner_product”: 找不到标识符
	行 529: ..\speaker\speaker_engine.cc(166): error C2039: "inner_product": 不是 "std" 的成员
	行 531: ..\speaker\speaker_engine.cc(166): error C3861: “inner_product”: 找不到标识符
	行 558: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\xutility(149): error C2664: “Ort::Session::Session(Ort::Env &,const wchar_t *,const Ort::SessionOptions &)”: 无法将参数 2 从“_Ty”转换为“const wchar_t *”

前面的错误可以通过 #include <numeric> 这个头文件解决，
但是后面的错误应该是这行代码导致的：

  speaker_session_ = std::make_shared<Ort::Session>(env_, model_path.c_str(),
                                                      session_options_);

看样子是api 不对？
谢谢！

参数保存问题

在训练时，projection_args中可以参数类型，然后这部分的参数是不是并没有保存到最终模型中呢？那这部分参数我在训练完成后想调用是不是不能用了呀？

resnet

非常感谢您的代码，我想问一下resnet model中，为什么会有一个two_emb_layer的参数，为什么会返回两个值。我印象中resnet返回embedding就OK了 这样做的意义是什么。期待您的回答

自监督训练速度过慢

用dino是不是会比非自监督训练慢很多呢？我用了一张A100训练voxceleb和cnceleb的混合数据集，基本上2个多小时一个epoch，我有点不明白是本身就这么慢还是我的训练产生了问题？

KeyError: 'test/id00807-speech-01-001.wav'

Hi, sorry to bother you, It seem apply cos score encounter erro，but I didn't know how to solve it. File format is made by .sh file，It seems that there should be no file mismatch,,,,,,

关于使用python预训练模型

我通过python binding的方式使用预训练模型，进行说话人识别推理，但是推理的过程中没有使用gpu，速度很慢，请问怎么样才可以打开gpu加速推理过程，谢谢你！

如果我想识别机器异常的声音，可以直接替换数据集吗？

8000 hz data

Hi,

Do you have any advice or experience for training 8000 hz data? Is it enough to change sampling rate in config, or should I decrease mel-size from 80 to 40 ?

When calculating DER, is overlapping taken into account?

When I use the process provided by WeSpeaker(example of VoxConverse) on the AMI dataset, I end up with a result of 27.54(DER), so I wonder if overlap is taken into account when calculating DER? If so, what parameters should be changed so that overlap is not taken into account when calculating DER?

Loading pretrained weights into torch model

Hello! I was trying to find how to inference ResNet293 pretrained model (this one) in pytorch, however I haven't found how to use it directly to load weights like:

import torch
from wespeaker.models.resnet import ResNet293
state_dict = torch.load('voxceleb_resnet293_LM.zip')
rn293 = ResNet293()
rn293.load_state_dict(state_dict)

Could you please tell, how to do it?

能否上传一下train log

尝试使用ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch150的示例，发现训练到120个epoch，ACC只有77.16，loss=1.22；想知道这个进展是正常的嘛？
训练数据：vox2_dev
训练耗时：1小时/epoch，V100，4 GPUs

what is the "assert num == len(path_list)" mean in average_model.py?

wespeaker/wespeaker/bin/average_model.py

Lines 54 to 58 in 7a7c3b1

	path_list = path_list[-args.num:]
	print(path_list)
	avg = None
	num = args.num
	assert num == len(path_list)

Hi, The purpose here is not to integrate all trained models, if I only train arg.nums models? What if the number of trained models is less than arg.nums, although this may not necessarily happen, but I think it should be considered in the code?

Any number about the performance (i.e. EER) that WeSpeaker can achieve?

Hi,

WeSpeaker implements many useful training and test tools in speaker verfication. It's awesome! However, in the repo, it seems that there is no information about how is the performance (EER, minDCF, etc.) on VoxCeleb and VoxConverse.

It would be very helpful if this information can be provided. The developers can use this information to check whether they use WeSpeaker in the correct way. The researchers can use this number to compare with the SOTA system.

Please inform me if I miss something. 😄

compute_cosine_score 方法调用报错

import wespeakerruntime as wespeaker
speaker = wespeaker.Speaker(lang='chs')
wav1_path = "./1.wav"
wav2_path = "./2.wav"
emb1 = speaker.extract_embedding(wav1_path)
emb2 = speaker.extract_embedding(wav2_path)
score = speaker.compute_cosine_score(emb1, emb2)
print(score)

错误提示：

Traceback (most recent call last):
File "/home/score2.py", line 8, in
score = speaker.compute_cosine_score(emb1, emb2)
File "/usr/local/lib/python3.9/site-packages/wespeakerruntime/speaker.py", line 173, in compute_cosine_score
return np.dot(emb1,
ValueError: shapes (1,256) and (1,256) not aligned: 256 (dim 1) != 1 (dim 0)

Is it normal that training loss degrades when margin updating?

As the logs shown below, the training loss increases with the margin when training.
Is it normal and will it converge in the following epochs? If not, how should I change the configs to avoid this?

[ INFO : 2023-01-29 01:24:00,298 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-29 01:24:00,298 ] - |     Epoch|     Batch|        Lr|    Margin|      Loss|       Acc|
[ INFO : 2023-01-29 01:24:00,298 ] - +----------+----------+----------+----------+----------+----------+
[ INFO : 2023-01-29 04:29:07,153 ] - |        19|      1600|   0.15384|         0|   0.80848|    84.598|
[ INFO : 2023-01-29 04:29:07,155 ] - |        19|      1600|   0.15384|         0|   0.81305|    84.564|
[ INFO : 2023-01-29 04:30:01,387 ] - |        19|      1700|   0.15342|         0|   0.80819|    84.619|
[ INFO : 2023-01-29 04:30:01,387 ] - |        19|      1700|   0.15342|         0|   0.81384|    84.558|
[ INFO : 2023-01-29 04:30:55,370 ] - |        19|      1800|     0.153|         0|   0.80901|     84.61|
[ INFO : 2023-01-29 04:30:55,370 ] - |        19|      1800|     0.153|         0|   0.81248|    84.578|
[ INFO : 2023-01-29 04:31:26,231 ] - |        19|      1856|   0.15277|         0|   0.81266|    84.564|
[ INFO : 2023-01-29 04:31:26,239 ] - |        19|      1856|   0.15277|         0|   0.80908|    84.604|
[ INFO : 2023-01-29 04:32:34,994 ] - |        20|       100|   0.15232| 0.0036354|   0.79256|     85.07|
[ INFO : 2023-01-29 04:32:34,995 ] - |        20|       100|   0.15232| 0.0036354|   0.78737|    84.875|
[ INFO : 2023-01-29 04:33:29,630 ] - |        20|       200|   0.15191| 0.0072404|    0.8009|    84.621|
[ INFO : 2023-01-29 04:33:29,630 ] - |        20|       200|   0.15191| 0.0072404|    0.8114|    84.301|
[ INFO : 2023-01-29 04:34:23,817 ] - |        20|       300|    0.1515|  0.010779|   0.82122|    84.161|
[ INFO : 2023-01-29 04:34:23,818 ] - |        20|       300|    0.1515|  0.010779|    0.8245|    84.089|
[ INFO : 2023-01-29 04:35:17,964 ] - |        20|       400|   0.15108|  0.014253|   0.84229|    83.695|
[ INFO : 2023-01-29 04:35:17,965 ] - |        20|       400|   0.15108|  0.014253|   0.83855|    83.779|
[ INFO : 2023-01-29 04:36:12,081 ] - |        20|       500|   0.15067|  0.017663|   0.85528|    83.375|
[ INFO : 2023-01-29 04:36:12,083 ] - |        20|       500|   0.15067|  0.017663|   0.85417|    83.405|
[ INFO : 2023-01-29 04:37:06,410 ] - |        20|       600|   0.15026|  0.021011|   0.87648|     82.93|
[ INFO : 2023-01-29 04:37:06,411 ] - |        20|       600|   0.15026|  0.021011|   0.87643|    82.975|
[ INFO : 2023-01-29 04:38:00,344 ] - |        20|       700|   0.14986|  0.024297|   0.89987|     82.44|
[ INFO : 2023-01-29 04:38:00,344 ] - |        20|       700|   0.14986|  0.024297|   0.89862|    82.439|
[ INFO : 2023-01-29 04:38:54,487 ] - |        20|       800|   0.14945|  0.027522|   0.92245|    81.938|
[ INFO : 2023-01-29 04:38:54,487 ] - |        20|       800|   0.14945|  0.027522|   0.91954|    81.979|
[ INFO : 2023-01-29 04:39:48,606 ] - |        20|       900|   0.14904|  0.030689|   0.94458|    81.431|
[ INFO : 2023-01-29 04:39:48,607 ] - |        20|       900|   0.14904|  0.030689|   0.94449|    81.456|
[ INFO : 2023-01-29 04:40:42,756 ] - |        20|      1000|   0.14864|  0.033797|   0.96551|    80.988|
[ INFO : 2023-01-29 04:40:42,756 ] - |        20|      1000|   0.14864|  0.033797|   0.96677|    81.007|
[ INFO : 2023-01-29 04:41:36,928 ] - |        20|      1100|   0.14824|  0.036848|   0.98557|    80.548|
[ INFO : 2023-01-29 04:41:36,930 ] - |        20|      1100|   0.14824|  0.036848|   0.98618|    80.554|
[ INFO : 2023-01-29 04:42:31,358 ] - |        20|      1200|   0.14783|  0.039844|    1.0029|    80.172|
[ INFO : 2023-01-29 04:42:31,361 ] - |        20|      1200|   0.14783|  0.039844|    1.0056|     80.13|
[ INFO : 2023-01-29 04:43:26,015 ] - |        20|      1300|   0.14743|  0.042784|    1.0271|    79.651|
[ INFO : 2023-01-29 04:43:26,015 ] - |        20|      1300|   0.14743|  0.042784|    1.0218|    79.749|
[ INFO : 2023-01-29 04:44:20,381 ] - |        20|      1400|   0.14703|   0.04567|    1.0474|    79.214|
[ INFO : 2023-01-29 04:44:20,383 ] - |        20|      1400|   0.14703|   0.04567|    1.0431|    79.259|
[ INFO : 2023-01-29 04:45:14,764 ] - |        20|      1500|   0.14663|  0.048504|    1.0639|    78.821|
[ INFO : 2023-01-29 04:45:14,766 ] - |        20|      1500|   0.14663|  0.048504|    1.0644|    78.844|
[ INFO : 2023-01-29 04:46:09,084 ] - |        20|      1600|   0.14623|  0.051285|    1.0828|    78.431|
[ INFO : 2023-01-29 04:46:09,087 ] - |        20|      1600|   0.14623|  0.051285|    1.0818|    78.489|
[ INFO : 2023-01-29 04:47:03,841 ] - |        20|      1700|   0.14584|  0.054015|     1.103|    78.008|
[ INFO : 2023-01-29 04:47:03,842 ] - |        20|      1700|   0.14584|  0.054015|    1.1002|    78.114|
[ INFO : 2023-01-29 04:47:58,349 ] - |        20|      1800|   0.14544|  0.056695|    1.1188|    77.724|
[ INFO : 2023-01-29 04:47:58,351 ] - |        20|      1800|   0.14544|  0.056695|    1.1212|    77.591|
[ INFO : 2023-01-29 04:48:29,242 ] - |        20|      1856|   0.14522|  0.058175|    1.1301|    77.496|
[ INFO : 2023-01-29 04:48:29,253 ] - |        20|      1856|   0.14522|  0.058175|    1.1303|    77.388|
[ INFO : 2023-01-29 04:49:41,690 ] - |        21|       100|    0.1448|  0.060984|    1.4157|    71.047|
[ INFO : 2023-01-29 04:49:41,691 ] - |        21|       100|    0.1448|  0.060984|    1.4362|    71.195|
[ INFO : 2023-01-29 04:50:36,524 ] - |        21|       200|    0.1444|  0.063537|    1.4283|    71.004|
[ INFO : 2023-01-29 04:50:36,525 ] - |        21|       200|    0.1444|  0.063537|     1.441|    70.438|
[ INFO : 2023-01-29 04:51:31,489 ] - |        21|       300|   0.14401|  0.066042|    1.4579|    70.469|
[ INFO : 2023-01-29 04:51:31,490 ] - |        21|       300|   0.14401|  0.066042|    1.4763|    69.698|
[ INFO : 2023-01-29 04:52:26,456 ] - |        21|       400|   0.14362|  0.068501|    1.4945|    69.498|
[ INFO : 2023-01-29 04:52:26,458 ] - |        21|       400|   0.14362|  0.068501|     1.471|    70.191|
[ INFO : 2023-01-29 04:53:20,965 ] - |        21|       500|   0.14323|  0.070915|    1.5203|    69.103|
[ INFO : 2023-01-29 04:53:20,969 ] - |        21|       500|   0.14323|  0.070915|    1.4889|    69.759|
[ INFO : 2023-01-29 04:54:15,555 ] - |        21|       600|   0.14284|  0.073285|    1.5317|    68.842|
[ INFO : 2023-01-29 04:54:15,555 ] - |        21|       600|   0.14284|  0.073285|    1.5124|    69.322|
[ INFO : 2023-01-29 04:55:10,008 ] - |        21|       700|   0.14245|  0.075612|    1.5276|    69.044|
[ INFO : 2023-01-29 04:55:10,009 ] - |        21|       700|   0.14245|  0.075612|    1.5454|    68.496|
[ INFO : 2023-01-29 04:56:04,298 ] - |        21|       800|   0.14207|  0.077895|    1.5538|    68.302|
[ INFO : 2023-01-29 04:56:04,300 ] - |        21|       800|   0.14207|  0.077895|    1.5388|     68.76|
[ INFO : 2023-01-29 04:56:58,865 ] - |        21|       900|   0.14168|  0.080137|    1.5685|    68.024|
[ INFO : 2023-01-29 04:56:58,866 ] - |        21|       900|   0.14168|  0.080137|    1.5571|    68.409|
[ INFO : 2023-01-29 04:57:53,207 ] - |        21|      1000|    0.1413|  0.082337|    1.5764|    68.077|
[ INFO : 2023-01-29 04:57:53,207 ] - |        21|      1000|    0.1413|  0.082337|    1.5795|    67.808|
[ INFO : 2023-01-29 04:58:47,680 ] - |        21|      1100|   0.14091|  0.084498|     1.593|    67.587|
[ INFO : 2023-01-29 04:58:47,681 ] - |        21|      1100|   0.14091|  0.084498|     1.593|    67.751|
[ INFO : 2023-01-29 04:59:42,159 ] - |        21|      1200|   0.14053|  0.086618|    1.6082|    67.334|
[ INFO : 2023-01-29 04:59:42,161 ] - |        21|      1200|   0.14053|  0.086618|    1.6096|    67.482|
[ INFO : 2023-01-29 05:00:36,535 ] - |        21|      1300|   0.14015|    0.0887|    1.6245|    67.215|
[ INFO : 2023-01-29 05:00:36,536 ] - |        21|      1300|   0.14015|    0.0887|    1.6241|    67.044|
[ INFO : 2023-01-29 05:01:31,089 ] - |        21|      1400|   0.13977|  0.090743|    1.6398|    66.897|
[ INFO : 2023-01-29 05:01:31,089 ] - |        21|      1400|   0.13977|  0.090743|    1.6423|    66.761|
[ INFO : 2023-01-29 05:02:26,021 ] - |        21|      1500|   0.13939|  0.092749|     1.656|    66.518|
[ INFO : 2023-01-29 05:02:26,021 ] - |        21|      1500|   0.13939|  0.092749|    1.6564|    66.635|
[ INFO : 2023-01-29 05:03:20,579 ] - |        21|      1600|   0.13901|  0.094718|     1.671|    66.285|
[ INFO : 2023-01-29 05:03:20,580 ] - |        21|      1600|   0.13901|  0.094718|    1.6705|    66.374|
[ INFO : 2023-01-29 05:04:15,176 ] - |        21|      1700|   0.13863|  0.096651|    1.6869|    65.983|
[ INFO : 2023-01-29 05:04:15,177 ] - |        21|      1700|   0.13863|  0.096651|    1.6842|    66.128|
[ INFO : 2023-01-29 05:05:09,682 ] - |        21|      1800|   0.13826|  0.098548|     1.705|    65.699|
[ INFO : 2023-01-29 05:05:09,683 ] - |        21|      1800|   0.13826|  0.098548|    1.6977|     65.88|
[ INFO : 2023-01-29 05:05:40,750 ] - |        21|      1856|   0.13805|  0.099595|    1.7063|    65.742|
[ INFO : 2023-01-29 05:05:40,759 ] - |        21|      1856|   0.13805|  0.099595|    1.7144|    65.534|
[ INFO : 2023-01-29 05:06:52,529 ] - |        22|       100|   0.13764|   0.10158|    1.8749|    62.453|
[ INFO : 2023-01-29 05:06:52,529 ] - |        22|       100|   0.13764|   0.10158|    1.8852|    62.328|
[ INFO : 2023-01-29 05:07:47,611 ] - |        22|       200|   0.13727|   0.10339|    1.9199|    62.102|
[ INFO : 2023-01-29 05:07:47,614 ] - |        22|       200|   0.13727|   0.10339|    1.9199|     61.93|
[ INFO : 2023-01-29 05:08:42,832 ] - |        22|       300|   0.13689|   0.10516|    1.9241|    61.831|
[ INFO : 2023-01-29 05:08:42,834 ] - |        22|       300|   0.13689|   0.10516|    1.9386|    61.544|
[ INFO : 2023-01-29 05:09:37,676 ] - |        22|       400|   0.13652|   0.10691|     1.954|    61.336|
[ INFO : 2023-01-29 05:09:37,676 ] - |        22|       400|   0.13652|   0.10691|    1.9551|    61.217|
[ INFO : 2023-01-29 05:10:32,377 ] - |        22|       500|   0.13615|   0.10862|    1.9711|    61.041|
[ INFO : 2023-01-29 05:10:32,378 ] - |        22|       500|   0.13615|   0.10862|    1.9664|        61|
[ INFO : 2023-01-29 05:11:27,275 ] - |        22|       600|   0.13578|   0.11029|    1.9839|    60.822|
[ INFO : 2023-01-29 05:11:27,276 ] - |        22|       600|   0.13578|   0.11029|    1.9817|     60.69|
[ INFO : 2023-01-29 05:12:22,047 ] - |        22|       700|   0.13541|   0.11194|    1.9986|    60.325|
[ INFO : 2023-01-29 05:12:22,048 ] - |        22|       700|   0.13541|   0.11194|    1.9983|    60.506|
[ INFO : 2023-01-29 05:13:16,659 ] - |        22|       800|   0.13505|   0.11356|    2.0139|    60.273|
[ INFO : 2023-01-29 05:13:16,660 ] - |        22|       800|   0.13505|   0.11356|    2.0111|    60.187|
[ INFO : 2023-01-29 05:14:11,286 ] - |        22|       900|   0.13468|   0.11514|    2.0237|     59.99|
[ INFO : 2023-01-29 05:14:11,286 ] - |        22|       900|   0.13468|   0.11514|    2.0307|    59.984|
[ INFO : 2023-01-29 05:15:05,794 ] - |        22|      1000|   0.13431|    0.1167|    2.0436|    59.753|
[ INFO : 2023-01-29 05:15:05,796 ] - |        22|      1000|   0.13431|    0.1167|    2.0369|    59.799|
[ INFO : 2023-01-29 05:16:00,662 ] - |        22|      1100|   0.13395|   0.11823|    2.0554|    59.582|
[ INFO : 2023-01-29 05:16:00,662 ] - |        22|      1100|   0.13395|   0.11823|    2.0472|     59.58|
[ INFO : 2023-01-29 05:16:55,399 ] - |        22|      1200|   0.13359|   0.11973|    2.0703|    59.396|
[ INFO : 2023-01-29 05:16:55,399 ] - |        22|      1200|   0.13359|   0.11973|    2.0605|     59.38|
[ INFO : 2023-01-29 05:17:50,036 ] - |        22|      1300|   0.13322|   0.12121|    2.0841|    59.173|
[ INFO : 2023-01-29 05:17:50,037 ] - |        22|      1300|   0.13322|   0.12121|     2.075|    59.135|
[ INFO : 2023-01-29 05:18:44,705 ] - |        22|      1400|   0.13286|   0.12265|    2.0905|    58.906|
[ INFO : 2023-01-29 05:18:44,707 ] - |        22|      1400|   0.13286|   0.12265|    2.0982|     58.94|
[ INFO : 2023-01-29 05:19:39,525 ] - |        22|      1500|    0.1325|   0.12407|    2.1088|    58.774|
[ INFO : 2023-01-29 05:19:39,526 ] - |        22|      1500|    0.1325|   0.12407|    2.1014|    58.735|
[ INFO : 2023-01-29 05:20:34,116 ] - |        22|      1600|   0.13214|   0.12547|    2.1197|     58.63|
[ INFO : 2023-01-29 05:20:34,117 ] - |        22|      1600|   0.13214|   0.12547|    2.1122|    58.575|
[ INFO : 2023-01-29 05:21:28,752 ] - |        22|      1700|   0.13178|   0.12683|    2.1234|    58.391|
[ INFO : 2023-01-29 05:21:28,754 ] - |        22|      1700|   0.13178|   0.12683|     2.135|    58.397|
[ INFO : 2023-01-29 05:22:23,357 ] - |        22|      1800|   0.13142|   0.12818|    2.1342|    58.214|
[ INFO : 2023-01-29 05:22:23,359 ] - |        22|      1800|   0.13142|   0.12818|    2.1457|     58.27|
[ INFO : 2023-01-29 05:22:54,387 ] - |        22|      1856|   0.13122|   0.12892|    2.1406|    58.106|
[ INFO : 2023-01-29 05:22:54,397 ] - |        22|      1856|   0.13122|   0.12892|    2.1495|    58.193|
[ INFO : 2023-01-29 05:24:03,348 ] - |        23|       100|   0.13084|   0.13033|    2.2634|    56.031|
[ INFO : 2023-01-29 05:24:03,349 ] - |        23|       100|   0.13084|   0.13033|    2.2784|    56.172|
[ INFO : 2023-01-29 05:24:57,665 ] - |        23|       200|   0.13048|   0.13161|    2.3075|    55.465|
[ INFO : 2023-01-29 05:24:57,666 ] - |        23|       200|   0.13048|   0.13161|    2.3063|    55.371|
[ INFO : 2023-01-29 05:25:51,934 ] - |        23|       300|   0.13013|   0.13286|    2.3099|     55.37|
[ INFO : 2023-01-29 05:25:51,935 ] - |        23|       300|   0.13013|   0.13286|    2.3306|    55.094|
[ INFO : 2023-01-29 05:26:45,998 ] - |        23|       400|   0.12978|   0.13409|    2.3343|    55.092|
[ INFO : 2023-01-29 05:26:45,998 ] - |        23|       400|   0.12978|   0.13409|    2.3066|    55.473|
[ INFO : 2023-01-29 05:27:40,059 ] - |        23|       500|   0.12942|    0.1353|    2.3241|    55.272|
[ INFO : 2023-01-29 05:27:40,059 ] - |        23|       500|   0.12942|    0.1353|    2.3461|    54.936|
[ INFO : 2023-01-29 05:28:34,219 ] - |        23|       600|   0.12907|   0.13649|    2.3481|     54.94|
[ INFO : 2023-01-29 05:28:34,219 ] - |        23|       600|   0.12907|   0.13649|    2.3594|    54.747|
[ INFO : 2023-01-29 05:29:28,482 ] - |        23|       700|   0.12872|   0.13766|    2.3673|    54.737|
[ INFO : 2023-01-29 05:29:28,483 ] - |        23|       700|   0.12872|   0.13766|    2.3759|    54.522|
[ INFO : 2023-01-29 05:30:22,585 ] - |        23|       800|   0.12837|    0.1388|    2.3752|     54.65|
[ INFO : 2023-01-29 05:30:22,586 ] - |        23|       800|   0.12837|    0.1388|    2.3841|    54.455|
[ INFO : 2023-01-29 05:31:16,910 ] - |        23|       900|   0.12803|   0.13993|    2.3974|    54.261|
[ INFO : 2023-01-29 05:31:16,910 ] - |        23|       900|   0.12803|   0.13993|    2.3828|    54.549|
[ INFO : 2023-01-29 05:32:11,252 ] - |        23|      1000|   0.12768|   0.14103|    2.4068|    54.131|
[ INFO : 2023-01-29 05:32:11,253 ] - |        23|      1000|   0.12768|   0.14103|    2.3941|    54.398|
[ INFO : 2023-01-29 05:33:05,590 ] - |        23|      1100|   0.12733|   0.14211|    2.3992|    54.347|
[ INFO : 2023-01-29 05:33:05,592 ] - |        23|      1100|   0.12733|   0.14211|    2.4136|    54.006|
[ INFO : 2023-01-29 05:33:59,902 ] - |        23|      1200|   0.12699|   0.14317|    2.4028|    54.283|
[ INFO : 2023-01-29 05:33:59,902 ] - |        23|      1200|   0.12699|   0.14317|    2.4216|    53.928|
[ INFO : 2023-01-29 05:34:54,078 ] - |        23|      1300|   0.12664|   0.14422|    2.4102|    54.221|
[ INFO : 2023-01-29 05:34:54,078 ] - |        23|      1300|   0.12664|   0.14422|    2.4336|    53.802|
[ INFO : 2023-01-29 05:35:48,403 ] - |        23|      1400|    0.1263|   0.14524|    2.4196|    54.109|
[ INFO : 2023-01-29 05:35:48,404 ] - |        23|      1400|    0.1263|   0.14524|    2.4425|    53.682|
[ INFO : 2023-01-29 05:36:42,880 ] - |        23|      1500|   0.12595|   0.14625|    2.4306|    53.951|
[ INFO : 2023-01-29 05:36:42,881 ] - |        23|      1500|   0.12595|   0.14625|    2.4515|    53.546|
[ INFO : 2023-01-29 05:37:37,511 ] - |        23|      1600|   0.12561|   0.14723|    2.4399|    53.835|
[ INFO : 2023-01-29 05:37:37,515 ] - |        23|      1600|   0.12561|   0.14723|    2.4595|    53.463|
[ INFO : 2023-01-29 05:38:32,029 ] - |        23|      1700|   0.12527|    0.1482|    2.4471|    53.767|
[ INFO : 2023-01-29 05:38:32,031 ] - |        23|      1700|   0.12527|    0.1482|    2.4676|    53.412|
[ INFO : 2023-01-29 05:39:26,419 ] - |        23|      1800|   0.12493|   0.14915|    2.4535|    53.646|
[ INFO : 2023-01-29 05:39:26,420 ] - |        23|      1800|   0.12493|   0.14915|    2.4738|    53.353|
[ INFO : 2023-01-29 05:39:57,393 ] - |        23|      1856|   0.12474|   0.14968|    2.4585|    53.587|
[ INFO : 2023-01-29 05:39:57,394 ] - |        23|      1856|   0.12474|   0.14968|     2.478|    53.288|
[ INFO : 2023-01-29 05:41:07,032 ] - |        24|       100|   0.12438|   0.15068|    2.5563|    51.914|
[ INFO : 2023-01-29 05:41:07,039 ] - |        24|       100|   0.12438|   0.15068|     2.563|     52.32|
[ INFO : 2023-01-29 05:42:01,723 ] - |        24|       200|   0.12404|   0.15158|    2.5665|    51.895|
[ INFO : 2023-01-29 05:42:01,724 ] - |        24|       200|   0.12404|   0.15158|    2.5628|    52.223|
[ INFO : 2023-01-29 05:42:56,303 ] - |        24|       300|    0.1237|   0.15247|    2.5734|    52.065|
[ INFO : 2023-01-29 05:42:56,303 ] - |        24|       300|    0.1237|   0.15247|    2.5771|     51.76|
[ INFO : 2023-01-29 05:43:50,909 ] - |        24|       400|   0.12336|   0.15334|    2.5842|     51.91|
[ INFO : 2023-01-29 05:43:50,910 ] - |        24|       400|   0.12336|   0.15334|    2.5741|    51.883|
[ INFO : 2023-01-29 05:44:45,155 ] - |        24|       500|   0.12303|    0.1542|    2.5872|      51.8|
[ INFO : 2023-01-29 05:44:45,158 ] - |        24|       500|   0.12303|    0.1542|    2.5799|    52.016|
[ INFO : 2023-01-29 05:45:39,563 ] - |        24|       600|    0.1227|   0.15504|    2.5862|    51.897|
[ INFO : 2023-01-29 05:45:39,564 ] - |        24|       600|    0.1227|   0.15504|    2.5979|    51.719|
[ INFO : 2023-01-29 05:46:33,810 ] - |        24|       700|   0.12236|   0.15587|    2.5894|    51.888|
[ INFO : 2023-01-29 05:46:33,811 ] - |        24|       700|   0.12236|   0.15587|    2.6018|    51.662|
[ INFO : 2023-01-29 05:47:28,234 ] - |        24|       800|   0.12203|   0.15668|    2.6075|    51.592|
[ INFO : 2023-01-29 05:47:28,234 ] - |        24|       800|   0.12203|   0.15668|    2.5956|    51.695|
[ INFO : 2023-01-29 05:48:22,799 ] - |        24|       900|    0.1217|   0.15747|    2.6014|     51.69|
[ INFO : 2023-01-29 05:48:22,800 ] - |        24|       900|    0.1217|   0.15747|    2.6129|    51.542|
[ INFO : 2023-01-29 05:49:17,135 ] - |        24|      1000|   0.12137|   0.15825|    2.6211|    51.434|
[ INFO : 2023-01-29 05:49:17,135 ] - |        24|      1000|   0.12137|   0.15825|    2.6105|      51.6|
[ INFO : 2023-01-29 05:50:11,478 ] - |        24|      1100|   0.12104|   0.15902|    2.6323|    51.331|
[ INFO : 2023-01-29 05:50:11,478 ] - |        24|      1100|   0.12104|   0.15902|    2.6185|    51.529|
[ INFO : 2023-01-29 05:51:05,996 ] - |        24|      1200|   0.12071|   0.15977|    2.6408|    51.259|
[ INFO : 2023-01-29 05:51:05,997 ] - |        24|      1200|   0.12071|   0.15977|    2.6245|     51.43|
[ INFO : 2023-01-29 05:52:00,556 ] - |        24|      1300|   0.12038|   0.16051|    2.6482|    51.171|
[ INFO : 2023-01-29 05:52:00,558 ] - |        24|      1300|   0.12038|   0.16051|     2.634|    51.329|

sphereface2 loss with AAM pretrained

Hi,

Dou you think that it will be a good option to finetune pretrained Resnet models (with AAM loss) with sphereface2 ? Do you have any experience like that?