Hi! I got stuck in the training process.
I am on macOS Big Sur.
I use this command: python3 DeepSpeech.py –train_files …/en/clips/train.csv --dev_files …/en/clips/dev.csv --test_files …/en/clips/test.csv
And get this log:
I Couldn’t find the best validating checkpoint.
I Loading now recent checkpoint from /Users/doldlevente/.local/share/deepspeech/checkpoints/train-0
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm / rnn / multi_rnn_cell / cell_0 / cudnn_compatible_lstm_cell / bias
I Loading variable from checkpoint: cudnn_lstm / rnn / multi_rnn_cell / cell_0 / cudnn_compatible_lstm_cell / bias / Adam
I Loading variable from checkpoint: cudnn_lstm / rnn / multi_rnn_cell / cell_0 / cudnn_compatible_lstm_cell / bias / Adam_1
I Loading variable from checkpoint: cudnn_lstm / rnn / multi_rnn_cell / cell_0 / cudnn_compatible_lstm_cell / kernel
I Loading variable from checkpoint: cudnn_lstm / rnn / multi_rnn_cell / cell_0 / cudnn_compatible_lstm_cell / kernel / Adam
I Loading variable from checkpoint: cudnn_lstm / rnn / multi_rnn_cell / cell_0 / cudnn_compatible_lstm_cell / kernel / Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1 / bias
I Loading variable from checkpoint: layer_1 / bias / Adam
I Loading variable from checkpoint: layer_1 / bias / Adam_1
I Loading variable from checkpoint: layer_1 / weights
I Loading variable from checkpoint: layer_1 / weights / Adam
I Loading variable from checkpoint: layer_1 / weights / Adam_1
I Loading variable from checkpoint: layer_2 / bias
I Loading variable from checkpoint: layer_2 / bias / Adam
I Loading variable from checkpoint: layer_2 / bias / Adam_1
I Loading variable from checkpoint: layer_2 / weights
I Loading variable from checkpoint: layer_2 / weights / Adam
I Loading variable from checkpoint: layer_2 / weights / Adam_1
I Loading variable from checkpoint: layer_3 / bias
I Loading variable from checkpoint: layer_3 / bias / Adam
I Loading variable from checkpoint: layer_3 / bias / Adam_1
I Loading variable from checkpoint: layer_3 / weights
I Loading variable from checkpoint: layer_3 / weights / Adam
I Loading variable from checkpoint: layer_3 / weights / Adam_1
I Loading variable from checkpoint: layer_5 / bias
I Loading variable from checkpoint: layer_5 / bias / Adam
I Loading variable from checkpoint: layer_5 / bias / Adam_1
I Loading variable from checkpoint: layer_5 / weights
I Loading variable from checkpoint: layer_5 / weights / Adam
I Loading variable from checkpoint: layer_5 / weights / Adam_1
I Loading variable from checkpoint: layer_6 / bias
I Loading variable from checkpoint: layer_6 / bias / Adam
I Loading variable from checkpoint: layer_6 / bias / Adam_1
I Loading variable from checkpoint: layer_6 / weights
I Loading variable from checkpoint: layer_6 / weights / Adam
I Loading variable from checkpoint: layer_6 / weights / Adam_1
I Loading variable from checkpoint: learning_rate
I STARTING Optimization
| Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000
| Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script ()
File “/Users/doldlevente/Desktop/Training/DeepSpeech/training/deepspeech_training/train.py”, line 982, in run_script
absl.app.run (main)
File “/Users/doldlevente/opt/anaconda3/envs/deepspeech/lib/python3.7/site-packages/absl/app.py”, line 312, in run
_run_main (main, args)
File “/Users/doldlevente/opt/anaconda3/envs/deepspeech/lib/python3.7/site-packages/absl/app.py”, line 258, in _run_main
sys.exit (main (argv))
File “/Users/doldlevente/Desktop/Training/DeepSpeech/training/deepspeech_training/train.py”, line 954, in main
train ()
File “/Users/doldlevente/Desktop/Training/DeepSpeech/training/deepspeech_training/train.py”, line 622, in train
dev_loss = dev_loss / total_steps
ZeroDivisionError: float division by zero
Thanks, ItzMeLevi