[WaveRNN]RuntimeError: cuda runtime error (59) : device-side assert triggered

TTS (Text-to-Speech)
#1

When i tried to train WaveRNN based on 9 bits, this error info shown up.

 > Training
 | > Epoch: 1/10000 -- Batch: 10/123 -- Loss: 6.242 -- Speed: 2.01 steps/sec -- Step: 10 -- lr: 9e-06 -- GradNorm: 0.1077281360050306
/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [22,0,0], thread: [457,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [22,0,0], thread: [302,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/generic/THCStorage.cpp line=36 error=59 : device-side assert triggered
 ! Run is removed from /home/jovyan/data1/peter/indonesia_tts_experiments/test_1/wavernn_test/WaveRNN/wavernn_output/indonesia_wavernn_4241_finetune_gaussian-May-12-2020_06+39AM-7c52874
Traceback (most recent call last):
  File "train.py", line 352, in <module>
    main(args)
  File "train.py", line 269, in main
    args=args,
  File "train.py", line 95, in train
    if loss.item() is None:
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/generic/THCStorage.cpp:36
#2

Now i guess the length and duration of datasets might affect this problem.

#3

I tried to replace the ‘quant’ dataset output which is generated based on notebook ExtractTTSpectrogram.ipynb with fatchord wavernn data preprocess quant output, based on preprocess.py. It stills causes this errors.

But when i use fatchord repo to train tacotron and wavernn, everything is ok. It’s really worth to digging out what causes the problem. @erogol

And i would like to point out that many people are really interested in using Mozilla TTS with WaveRNN. I think that they highly anticipated that. Thanks for the great work.@erogol.

#4

I don’t think I will have time to dig into wavernn anymore. There are a lot of new techniques that I plan to pursue.

#5

Thanks erogol. Good to know that :pray: