Hi, am trying to train new model of Deep Speech, it run for 7 steps and crash, error logs below:
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:06 | Steps: 7 | Loss: 467.228032
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | DatasEpoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/hangtg/Desktop/DeepSpeech/data_processing/vn/clips/dev.csv
Traceback (most recent call last):
File “DeepSpeech.py”, line 965, in
absl.app.run(main)
File “/home/hangtg/Desktop/deepspech-env/lib/python3.6/site-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/home/hangtg/Desktop/deepspech-env/lib/python3.6/site-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “DeepSpeech.py”, line 938, in main
train()
File “DeepSpeech.py”, line 645, in train
dev_loss = dev_loss / total_steps
ZeroDivisionError: float division by zero
as I dig in the Deepspeech.py file, i found out that the total_loss, batch_loss, step_count, total_steps didnt increase as it should be and I dont know why, the CSV files loaded in correctly.
So I tried to change step_count to 1 instead of 0 at the beginning, and it “worked”, kind of, the batch_loss still is 0, so the validation loss alway is 0.000
After training completed, the testing get the same error. This time in evaluate_tools.py file:
wer = sum(s.word_distance for s in samples) / sum(s.word_length for s in samples)
what is wrong with my files?