I am resuming training on my own dataset using checkpoint 0.6.1. Specification; tensorflow - 1.14, Ubuntu 16.04, Cuda 10, CuDNN - 7.5.
I have downloaded the realised checkpoint and using the following command
CUDA_VISIBLE_DEVICES=2,3 python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir checkcheck/deepspeech-0.6.1-checkpoint/ --epochs 3 --train_files extracted/language/archive/clips/train.csv --dev_files extracted/language/archive/clips/dev.csv --test_files extracted/language/archive/clips/dev.csv --learning_rate 0.0001 --use_cudnn_rnn true --use_allow_growth true
However, i am getting the following error
Instructions for updating: Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from checkcheck/deepspeech-0.6.1-checkpoint/best_dev-233784 I0118 11:53:46.365865 140480301135616 saver.py:1280] Restoring parameters from checkcheck/deepspeech-0.6.1-checkpoint/best_dev-233784 E Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: E E No OpKernel was registered to support Op 'CudnnRNNCanonicalToParams' used by node tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams (defined at DeepSpeech.py:119) with these attrs: [dropout=0, seed=4568, num_params=8, input_mode="linear_input", T=DT_FLOAT, direction="unidirectional", rnn_mode="lstm", seed2=240] E Registered devices: [CPU, XLA_CPU] E Registered kernels: E device='GPU'; T in [DT_DOUBLE] E device='GPU'; T in [DT_FLOAT] E device='GPU'; T in [DT_HALF] E E [[tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams]] E **The checkpoint in checkcheck/deepspeech-0.6.1-checkpoint/best_dev-233784 does not match the shapes of the model. Did you change alphabet.txt or the --n_hidden parameter between train runs using the same checkpoint dir? Try moving or removing the contents of checkcheck/deepspeech-0.6.1-checkpoint/best_dev-233784.**
I have checked the alphabet file and it seems fine. Can someone guide if i am missing out something?