Want to train on new dataset over the Pretrained model provided by Deepspeech

DeepSpeech
#1

Hi I want to train on new wav files over the existing DeepSpeech’s pretrained model which is

models/output_graph.pbmm

.

Is there an available checkpoint directory for the existing pre-trained model, over which I may train on my existing wav files and its associated transcripts?

#2

Do you read release pages ? It’s all explained.

1 Like
#3
#4

Thank You very Much :slight_smile:

#5

Whenever I am trying to run the deepspeech.py script with the checkpoint as the pre-trained checkpoint, I am getting the following error -

python DeepSpeech.py --train_files ./20140421/scripts/Ib/clips/train.csv --dev_files ./20140421/scripts/Ib/clips/dev.csv --test_files ./20140421/scripts/Ib/clips/test.csv --beam_width 500 --checkpoint_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/ --epochs 3 --export_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/ --export_author_id ritish_20140421 -export_file_name epoch_3 --export_model_name model_3_epochs --learning_rate 0.005 --lm_alpha 0.4 --lm_beta 1.5 --train_batch_size 5

I am getting the following error output:

I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/pretrained_checkpoints/deepspeech-0.6.1-checkpoint/best_dev-233784
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/train.py”, line 942, in run_script
absl.app.run(main)
File “/home/ritish/.local/lib/python3.7/site-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/home/ritish/.local/lib/python3.7/site-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/train.py”, line 914, in main
train()
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/train.py”, line 515, in train
load_or_init_graph(session, method_order)
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 103, in load_or_init_graph
return _load_checkpoint(session, ckpt_path)
File “/home/ritish/DeepSpeech/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 70, in _load_checkpoint
v.load(ckpt.get_tensor(v.op.name), session=session)
File “/home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py”, line 915, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam not found in checkpoint

The following files are located in 20140421/pretrained_checkpoints/deepspeech-0.6.1-checkpoint/

drwxr-xr-x 2 ritish ritish 4096 Apr 17 11:45 .
drwxr-xr-x 4 ritish ritish 4096 Apr 17 13:41 …
-rw-r–r-- 1 ritish ritish 329 Dec 13 18:52 alphabet.txt
-rw-r–r-- 1 ritish ritish 701047148 Dec 3 11:31 best_dev-233784.data-00000-of-00001
-rw-r–r-- 1 ritish ritish 1475 Dec 3 11:34 best_dev-233784.index
-rw-r–r-- 1 ritish ritish 8688210 Dec 3 11:35 best_dev-233784.meta
-rw-r–r-- 1 ritish ritish 87 Dec 3 11:34 best_dev_checkpoint
-rw-r–r-- 1 ritish ritish 87 Dec 13 18:52 checkpoint
-rw-r–r-- 1 ritish ritish 4775 Apr 17 13:43 flags.txt

But once I am running the same script through my own checkpoint directory, I am able to train the script without any problem.

ython DeepSpeech.py --train_files ./20140421/scripts/Ib/clips/train.csv --dev_files ./20140421/scripts/Ib/clips/dev.csv --test_files ./20140421/scripts/Ib/clips/test.csv --beam_width 500 --checkpoint_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/ --epochs 3 --export_dir ~/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/ --export_author_id ritish_20140421 -export_file_name epoch_3 --export_model_name model_3_epochs --learning_rate 0.005 --lm_alpha 0.4 --lm_beta 1.5 --train_batch_size 5

I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-11
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam_1
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/bias/Adam
I Loading variable from checkpoint: layer_6/bias/Adam_1
I Loading variable from checkpoint: layer_6/weights
I Loading variable from checkpoint: layer_6/weights/Adam
I Loading variable from checkpoint: layer_6/weights/Adam_1
I Loading variable from checkpoint: learning_rate
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:03:11 | Steps: 11 | Loss: 100.631165
Epoch 0 | Validation | Elapsed Time: 0:01:05 | Steps: 60 | Loss: 93.185731 | Dataset: ./20140421/scripts/Ib/clips/dev.csv
I Saved new best validating model with loss 93.185731 to: /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-22
Epoch 1 | Training | Elapsed Time: 0:03:06 | Steps: 11 | Loss: 86.365255
Epoch 1 | Validation | Elapsed Time: 0:01:03 | Steps: 60 | Loss: 86.035273 | Dataset: ./20140421/scripts/Ib/clips/dev.csv
I Saved new best validating model with loss 86.035273 to: /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-33
Epoch 2 | Training | Elapsed Time: 0:03:05 | Steps: 11 | Loss: 84.992410
Epoch 2 | Validation | Elapsed Time: 0:01:02 | Steps: 60 | Loss: 84.860918 | Dataset: ./20140421/scripts/Ib/clips/dev.csv
I Saved new best validating model with loss 84.860918 to: /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-44
I FINISHED optimization in 0:12:53.370869
I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-44
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on ./20140421/scripts/Ib/clips/test.csv
Test epoch | Steps: 55 | Elapsed Time: 0:02:24
Test on ./20140421/scripts/Ib/clips/test.csv - WER: 1.000000, CER: 1.000000, loss: 89.356972

Best WER:

WER: 2.272727, CER: 0.685185, loss: 214.728745

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18930114.wav
  • src: “an ceann comhairle a bheidh ag dul as oifig a atoghadh”
  • res: “a a a a a a a a a a a a a a a a a a a a a a a a a a a”

WER: 2.400000, CER: 0.615385, loss: 100.148514

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18921387.wav
  • src: “an bhfuil aon chapall agat”
  • res: “a a a a a a a a a a a a”

WER: 2.500000, CER: 0.736842, loss: 146.114197

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182562.wav
  • src: “an bhfuil an leabhar dearg ar an mbord”
  • res: “a a a a a a a a a a a a a a a a a a a a”

WER: 2.500000, CER: 0.760000, loss: 92.700058

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18921726.wav
  • src: “an frog agus an madra rua”
  • res: “a a a a a a a a a a a a a a a”

WER: 2.500000, CER: 1.062500, loss: 69.561539

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182115.wav
  • src: “a seacht a chlog”
  • res: “a a a a a a a a a a a a”

Median WER:

WER: 4.000000, CER: 1.583333, loss: 59.964828

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182545.wav
  • src: “ar an mbinse”
  • res: “a a a a a a a a a a a a”

WER: 4.250000, CER: 1.285714, loss: 95.893059

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17776000.wav
  • src: “an bhfuil hata uirthi”
  • res: “a a a a a a a a a a a a a a a a a”

WER: 4.250000, CER: 1.190476, loss: 90.645256

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18629958.wav
  • src: “ex machina gan amhras”
  • res: “a a a a a a a a a a a a a a a a a”

WER: 4.250000, CER: 1.473684, loss: 85.036110

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17570054.wav
  • src: “agus a chairde gael”
  • res: “a a a a a a a a a a a a a a a a a a”

WER: 4.333333, CER: 1.333333, loss: 65.942833

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18182490.wav
  • src: “an iomad airgid”
  • res: “a a a a a a a a a a a a a”

Worst WER:

WER: 7.000000, CER: 2.785714, loss: 83.998901

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18115882.wav
  • src: “ceacht a deich”
  • res: “a a a a a a a a a a a a a a a a a a a a a a”

WER: 7.500000, CER: 1.315789, loss: 88.528778

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_18363521.wav
  • src: “an cheardchomhairle”
  • res: “a a a a a a a a a a a a a a a”

WER: 7.500000, CER: 1.857143, loss: 71.964149

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17570046.wav
  • src: “ceithre bliana”
  • res: “a a a a a a a a a a a a a a a”

WER: 9.000000, CER: 2.818182, loss: 57.004471

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17569834.wav
  • src: “ar bhealach”
  • res: “a a a a a a a a a a a a a a a a a a”

WER: 12.000000, CER: 5.500000, loss: 33.971874

  • wav: file://20140421/scripts/Ib/clips/common_voice_ga-IE_17686714.wav
  • src: “ainm”
  • res: “a a a a a a a a a a a a”

I Exporting the model…
I Loading best validating checkpoint from /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/checkpoints_dir/best_dev-44
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
I Models exported at /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/
I Model metadata file saved to /home/ritish/DeepSpeech/DeepSpeech/20140421/scripts/Ib/models_dir/ritish_20140421_model_3_epochs_0.0.1.md. Before submitting the exported model for publishing make sure all information in the metadata file is correct, and complete the URL fields.

The following files are placed in my own created checkpoint directory:
ls -al 20140421/scripts/Ib/checkpoints_dir/

drwxr-xr-x 2 ritish ritish 4096 Apr 17 12:27 .
drwxr-xr-x 6 ritish ritish 4096 Apr 16 15:59 …
-rw-r–r-- 1 ritish ritish 566698352 Apr 17 12:04 best_dev-11.data-00000-of-00001
-rw-r–r-- 1 ritish ritish 1514 Apr 17 12:04 best_dev-11.index
-rw-r–r-- 1 ritish ritish 1265057 Apr 17 12:04 best_dev-11.meta
-rw-r–r-- 1 ritish ritish 221 Apr 17 12:04 best_dev_checkpoint
-rw-r–r-- 1 ritish ritish 325 Apr 17 12:08 checkpoint
-rw-r–r-- 1 ritish ritish 4703 Apr 17 11:56 flags.txt
-rw-r–r-- 1 ritish ritish 566698352 Apr 17 12:02 train-11.data-00000-of-00001
-rw-r–r-- 1 ritish ritish 1514 Apr 17 12:02 train-11.index
-rw-r–r-- 1 ritish ritish 1265051 Apr 17 12:02 train-11.meta

Hence Kindly let me know, where am I going wrong, when I am trying to train with pre-trained checkpoint directory

#6

This is totally unreadable. Please properly format your console output, we can’t read between what is consoel and what are your questions.

#7

Please make sure that:

  • you are on the correct tag v0.6.1
  • you have read the training documentation, especially about loading cudnn model
  • you read extensive other forum thread about fine-tuning.
#8

My question is I am not able to load the checkpoint from the pre-trained checkpoint and I am getting this error :slightly_smiling_face:

return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str)) tensorflow.python.framework.errors_impl.NotFoundError: Key cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam not found in checkpoint

But when I am using my own checkpoint, I am able to train on my checkpoints. Please can you help me resolve the error?

#9

If I can use the checkpoint from the pre-trained checkpoint doc, then I can use that checkpoint to start training my own wav files and csv transcripts. Your help will allow me to train my own datasets by utilizing the pre-trained checkpoint.

#10

As I said, have you read the documentation about training, and especially around cudnn loading?

#11

Sir, I will be grateful if you can please share the link. It will be of immense help to me.

#12

It’s linked from the Github front page …

#13

Ok, it looks like we added the v0.6.1 doc about CUDNN checkpoint after the release. My bad: --cudnn_checkpoint is what you need to check for in python DeepSpeech.py --helpfull to properly load the checkpoint.

#14

Sir the --load_cudnn true did the work for me. Thank you for your support.

#15

So you are not working on 0.6.1. You can’t use 0.6.1 checkpoints with non 0.6.1 version. Please respect instructions …

#16

Sir, I had followed your instructions and have since used the following link

https://deepspeech.readthedocs.io/en/master/TRAINING.html#installing-python-dependencies

to set up dependencies for the trainable parameters.

If possible please can you share the link for the 0.6.1 version? I shall surely follow the link post that.

#17

Everything at readthedocs ist for the 0.61 branch. I know this can be a bit confusing, but DeepSpeeach is still quite young, so read all the docs and search a bit. To install the 0.61 branch run

git clone --branch v0.6.1 https://github.com/mozilla/DeepSpeech.git

You probably cloned the current master branch which has different parameters and incompatible checkpoints.

1 Like
#18

You link explicitely the master doc. The v0.6.1 docs are on the same link, and they are the default ones when you follow the doc link from Github …

1 Like
#19

Importantly in this case, if you use latest master with the right incantations, the checkpoint will load fine, but the model will perform badly, because the feature computation code was changed to fix a bug. So you can waste lots of GPU hours by not using the correct version of the code.

#20

Sir, just confirming whether the correct documentation for this branch you have mentioned resides here: