Hello!
I started several times to run fine-tuning of your model from checkpoint gathered from here: https://github.com/mozilla/DeepSpeech/releases/tag/v0.6.1
And code gathered from the v0.6.1.tar.gz archive.
I’ve installed all of the requirements from the requirements.txt file.
- additional requirements.
pip3 install $(python3 util/taskcluster.py --decoder)
I have CUDA Version 10.0.130, CuDNN 7.4.2.24, tensorflow-gpu==1.14.0.
I use this command to run my fine-tuning process:
python DeepSpeech.py --n_hidden 2048 --checkpoint_dir deepspeech-0.6.1-checkpoint/ --epochs 3 --train_files train_full.csv --dev_files val_full.csv --test_files test_full.csv --learning_rate 0.0001 --use_cudnn_rnn --use_allow_growth
And I get this error:
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0513 10:13:31.574302 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0513 10:13:31.661218 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0513 10:13:31.661349 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0513 10:13:31.661422 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:13:31.762649 140563218061120 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:13:31.762798 140563218061120 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:13:31.762930 140563218061120 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:13:31.804666 140563218061120 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0513 10:13:31.835743 140563218061120 deprecation.py:323] From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:13:31.964800 140563218061120 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0513 10:13:32.247514 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0513 10:13:32.384267 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I0513 10:13:32.385055 140563218061120 saver.py:1280] Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
Traceback (most recent call last):
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
[[{{node save/CudnnRNNCanonicalToParams}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "DeepSpeech.py", line 974, in <module>
absl.app.run(main)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "DeepSpeech.py", line 947, in main
train()
File "DeepSpeech.py", line 555, in train
loaded = try_loading(session, checkpoint_saver, 'checkpoint', 'most recent')
File "DeepSpeech.py", line 405, in try_loading
saver.restore(session, checkpoint_path)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
[[node save/CudnnRNNCanonicalToParams (defined at DeepSpeech.py:495) ]]
Original stack trace for 'save/CudnnRNNCanonicalToParams':
File "DeepSpeech.py", line 974, in <module>
absl.app.run(main)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "DeepSpeech.py", line 947, in main
train()
File "DeepSpeech.py", line 495, in train
checkpoint_saver = tfv1.train.Saver(max_to_keep=FLAGS.max_to_keep)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
self.build()
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 744, in restore
restored_tensors)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 221, in tf_canonical_to_opaque
opaque_params = self._cu_canonical_to_opaque(cu_weights, cu_biases)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 271, in _cu_canonical_to_opaque
direction=self._direction)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 917, in cudnn_rnn_canonical_to_params
seed=seed, seed2=seed2, name=name)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
I already tried to use docker with cuda to prevent mismatching and broken dependencies on my machine.
But then I face different type of error, I double-checked and I don’t have characters in my transcripts which are not available in alphabet.txt in the checkpoing folder.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0513 10:19:02.836340 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0513 10:19:02.937894 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0513 10:19:02.938112 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0513 10:19:02.938226 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:19:03.067150 140199107434304 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:19:03.067367 140199107434304 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:19:03.067694 140199107434304 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:19:03.125893 140199107434304 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0513 10:19:03.163637 140199107434304 deprecation.py:323] From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0513 10:19:03.472953 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0513 10:19:03.634698 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I0513 10:19:03.635579 140199107434304 saver.py:1280] Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
E Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
E
E No OpKernel was registered to support Op 'CudnnRNNCanonicalToParams' used by node tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams (defined at DeepSpeech.py:119) with these attrs: [seed=4568, dropout=0, num_params=8, input_mode="linear_input", T=DT_FLOAT, direction="unidirectional", rnn_mode="lstm", seed2=240]
E Registered devices: [CPU, XLA_CPU, XLA_GPU]
E Registered kernels:
E device='GPU'; T in [DT_DOUBLE]
E device='GPU'; T in [DT_FLOAT]
E device='GPU'; T in [DT_HALF]
E
E [[tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams]]
E The checkpoint in deepspeech-0.6.1-checkpoint/best_dev-233784 does not match the shapes of the model. Did you change alphabet.txt or the --n_hidden parameter between train runs using the same checkpoint dir? Try moving or removing the contents of deepspeech-0.6.1-checkpoint/best_dev-233784.
If you have any suggestions how to fix it, please, help. 
