Whenever I am trying to run the deepspeech.py script with the checkpoint as the pre-trained checkpoint, I am getting the following error -
python DeepSpeech.py --train_files ./20140421/scripts/Ib/clips/train.csv --dev_files ./20140421/scripts/Ib/clips/dev.csv --test_files ./20140421/scripts/Ib/clips/test.csv --beam_width 500 --checkpoint_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/ --epochs 3 --export_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/ --export_author_id ritish_20140421 -export_file_name epoch_3 --export_model_name model_3_epochs --learning_rate 0.005 --lm_alpha 0.4 --lm_beta 1.5 --train_batch_size 5
I am getting the following error output:
I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/pretrained_checkpoints/deepspeech-0.6.1-checkpoint/best_dev-233784
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/train.py”, line 942, in run_script
absl.app.run(main)
File “/home/ritish/.local/lib/python3.7/site-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/home/ritish/.local/lib/python3.7/site-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/train.py”, line 914, in main
train()
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/train.py”, line 515, in train
load_or_init_graph(session, method_order)
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 103, in load_or_init_graph
return _load_checkpoint(session, ckpt_path)
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 70, in _load_checkpoint
v.load(ckpt.get_tensor(v.op.name), session=session)
File “/home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py”, line 915, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam not found in checkpoint
The following files are located in 20140421/pretrained_checkpoints/deepspeech-0.6.1-checkpoint/
drwxr-xr-x 2 ritish ritish 4096 Apr 17 11:45 .
drwxr-xr-x 4 ritish ritish 4096 Apr 17 13:41 …
-rw-r–r-- 1 ritish ritish 329 Dec 13 18:52 alphabet.txt
-rw-r–r-- 1 ritish ritish 701047148 Dec 3 11:31 best_dev-233784.data-00000-of-00001
-rw-r–r-- 1 ritish ritish 1475 Dec 3 11:34 best_dev-233784.index
-rw-r–r-- 1 ritish ritish 8688210 Dec 3 11:35 best_dev-233784.meta
-rw-r–r-- 1 ritish ritish 87 Dec 3 11:34 best_dev_checkpoint
-rw-r–r-- 1 ritish ritish 87 Dec 13 18:52 checkpoint
-rw-r–r-- 1 ritish ritish 4775 Apr 17 13:43 flags.txt
But once I am running the same script through my own checkpoint directory, I am able to train the script without any problem.
ython DeepSpeech.py --train_files ./20140421/scripts/Ib/clips/train.csv --dev_files ./20140421/scripts/Ib/clips/dev.csv --test_files ./20140421/scripts/Ib/clips/test.csv --beam_width 500 --checkpoint_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/ --epochs 3 --export_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/ --export_author_id ritish_20140421 -export_file_name epoch_3 --export_model_name model_3_epochs --learning_rate 0.005 --lm_alpha 0.4 --lm_beta 1.5 --train_batch_size 5
I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-11
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam_1
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/bias/Adam
I Loading variable from checkpoint: layer_6/bias/Adam_1
I Loading variable from checkpoint: layer_6/weights
I Loading variable from checkpoint: layer_6/weights/Adam
I Loading variable from checkpoint: layer_6/weights/Adam_1
I Loading variable from checkpoint: learning_rate
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:03:11 | Steps: 11 | Loss: 100.631165
Epoch 0 | Validation | Elapsed Time: 0:01:05 | Steps: 60 | Loss: 93.185731 | Dataset: ./20140421/scripts/Ib/clips/dev.csv
I Saved new best validating model with loss 93.185731 to: /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-22
Epoch 1 | Training | Elapsed Time: 0:03:06 | Steps: 11 | Loss: 86.365255
Epoch 1 | Validation | Elapsed Time: 0:01:03 | Steps: 60 | Loss: 86.035273 | Dataset: ./20140421/scripts/Ib/clips/dev.csv
I Saved new best validating model with loss 86.035273 to: /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-33
Epoch 2 | Training | Elapsed Time: 0:03:05 | Steps: 11 | Loss: 84.992410
Epoch 2 | Validation | Elapsed Time: 0:01:02 | Steps: 60 | Loss: 84.860918 | Dataset: ./20140421/scripts/Ib/clips/dev.csv
I Saved new best validating model with loss 84.860918 to: /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-44
I FINISHED optimization in 0:12:53.370869
I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-44
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on ./20140421/scripts/Ib/clips/test.csv
Test epoch | Steps: 55 | Elapsed Time: 0:02:24
Test on ./20140421/scripts/Ib/clips/test.csv - WER: 1.000000, CER: 1.000000, loss: 89.356972
Best WER:
WER: 2.272727, CER: 0.685185, loss: 214.728745
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18930114.wav
- src: “an ceann comhairle a bheidh ag dul as oifig a atoghadh”
- res: “a a a a a a a a a a a a a a a a a a a a a a a a a a a”
WER: 2.400000, CER: 0.615385, loss: 100.148514
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18921387.wav
- src: “an bhfuil aon chapall agat”
- res: “a a a a a a a a a a a a”
WER: 2.500000, CER: 0.736842, loss: 146.114197
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182562.wav
- src: “an bhfuil an leabhar dearg ar an mbord”
- res: “a a a a a a a a a a a a a a a a a a a a”
WER: 2.500000, CER: 0.760000, loss: 92.700058
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18921726.wav
- src: “an frog agus an madra rua”
- res: “a a a a a a a a a a a a a a a”
WER: 2.500000, CER: 1.062500, loss: 69.561539
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182115.wav
- src: “a seacht a chlog”
- res: “a a a a a a a a a a a a”
Median WER:
WER: 4.000000, CER: 1.583333, loss: 59.964828
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182545.wav
- src: “ar an mbinse”
- res: “a a a a a a a a a a a a”
WER: 4.250000, CER: 1.285714, loss: 95.893059
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17776000.wav
- src: “an bhfuil hata uirthi”
- res: “a a a a a a a a a a a a a a a a a”
WER: 4.250000, CER: 1.190476, loss: 90.645256
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18629958.wav
- src: “ex machina gan amhras”
- res: “a a a a a a a a a a a a a a a a a”
WER: 4.250000, CER: 1.473684, loss: 85.036110
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17570054.wav
- src: “agus a chairde gael”
- res: “a a a a a a a a a a a a a a a a a a”
WER: 4.333333, CER: 1.333333, loss: 65.942833
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182490.wav
- src: “an iomad airgid”
- res: “a a a a a a a a a a a a a”
Worst WER:
WER: 7.000000, CER: 2.785714, loss: 83.998901
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18115882.wav
- src: “ceacht a deich”
- res: “a a a a a a a a a a a a a a a a a a a a a a”
WER: 7.500000, CER: 1.315789, loss: 88.528778
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18363521.wav
- src: “an cheardchomhairle”
- res: “a a a a a a a a a a a a a a a”
WER: 7.500000, CER: 1.857143, loss: 71.964149
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17570046.wav
- src: “ceithre bliana”
- res: “a a a a a a a a a a a a a a a”
WER: 9.000000, CER: 2.818182, loss: 57.004471
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17569834.wav
- src: “ar bhealach”
- res: “a a a a a a a a a a a a a a a a a a”
WER: 12.000000, CER: 5.500000, loss: 33.971874
- wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17686714.wav
- src: “ainm”
- res: “a a a a a a a a a a a a”
I Exporting the model…
I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-44
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
I Models exported at /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/
I Model metadata file saved to /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/ritish_20140421_model_3_epochs_0.0.1.md. Before submitting the exported model for publishing make sure all information in the metadata file is correct, and complete the URL fields.
The following files are placed in my own created checkpoint directory:
ls -al 20140421/scripts/Ib/checkpoints_dir/
drwxr-xr-x 2 ritish ritish 4096 Apr 17 12:27 .
drwxr-xr-x 6 ritish ritish 4096 Apr 16 15:59 …
-rw-r–r-- 1 ritish ritish 566698352 Apr 17 12:04 best_dev-11.data-00000-of-00001
-rw-r–r-- 1 ritish ritish 1514 Apr 17 12:04 best_dev-11.index
-rw-r–r-- 1 ritish ritish 1265057 Apr 17 12:04 best_dev-11.meta
-rw-r–r-- 1 ritish ritish 221 Apr 17 12:04 best_dev_checkpoint
-rw-r–r-- 1 ritish ritish 325 Apr 17 12:08 checkpoint
-rw-r–r-- 1 ritish ritish 4703 Apr 17 11:56 flags.txt
-rw-r–r-- 1 ritish ritish 566698352 Apr 17 12:02 train-11.data-00000-of-00001
-rw-r–r-- 1 ritish ritish 1514 Apr 17 12:02 train-11.index
-rw-r–r-- 1 ritish ritish 1265051 Apr 17 12:02 train-11.meta
Hence Kindly let me know, where am I going wrong, when I am trying to train with pre-trained checkpoint directory