Hello guys , I m training on 0.9.1 deepspeech model on common voice single word dataset on Google colab. After training the data set for 8-10 epochs validation loss gets inf and if I train further the training loss also gets inf can anyone tell me what should I do?
I tried to lower the learning rate from 0.0001 to 0.00001 but still same result.
Wer I get is around 0.88 which is not good at all please tell how can I improve the model
Deepspeech version 0.9.1
Training
Python version 3.6
Tensorflow-gpu-1.15.4
Windows 10
Google colab
Ерoch 0 | Training | Elapsed Time: 0:38:02 | steps: 12706 | Loss: 4.281445
Epoch 0| validat ion | Elapsed Tine: 0:05:05 | Steps: 7198 | Loss: 4.998548 | Dataset: /content/en
I Saved new best validating model with loss 4.998548 to: /content/drive/MyDrive/common_voice_eng/cl
Ероch 1 | Training | Elapsed Time: 0:37:52 Steps: 12706| Loss: 4.097944
Epoch 1| validation | Elapsed Time: 0:05:03 Steps: 7198 | Loss: 5.146969 | Dataset: /content/en/clips
Ерoch 2 |Training | Elapsed Time: 0:37:53 steps: 12706 | Loss: 4.106288
Epoch 2 | validation | Elapsed Time: 0:05:03 Steps: 7198 | Loss: 5.551651 | Dataset: /content/en/clips
Epoch 3 |Training | Elapsed Time: 0:37:53 | Steps: 12706 | Loss: 4.052530
Epoch 3| validation Elapsed Time: 0:05:40 | Steps: 7198 | Loss: 4.784578 | Dataset: /cantent/en/
I Saved new best validat ing model with loss 4.784578 to: /content/drive/comon_voice_eng/checkpoints
Epoch 4 |Training | Elapsed Time: 0:37:55 | Steps: 12706 Loss: 4.000706
Epoch 4 |validation | Elapsed Time: 0:02:28 Steps: 3287| loss: 6.391232 | Dataset: /cantent/en/clips
Epoch 4 | validation | Elapsed Time: 0:03:43 Steps: 4881 | Loss: inf | Dataset: /content/en/clips
Epoch 4 | Validation | Elapsed Tine: 0:04:46 Steps: 6195 | Loss: inf | Dataset /content/en/clips
Epoch 4 | validation | Elapsed Tine: 0:05:51 Steps: 7198I Loss: inf | Datas.: /content/en/clips