Cannot start fine-tuning with DeepSpeech 0.6.1

Hello!
I started several times to run fine-tuning of your model from checkpoint gathered from here: https://github.com/mozilla/DeepSpeech/releases/tag/v0.6.1
And code gathered from the v0.6.1.tar.gz archive.
I’ve installed all of the requirements from the requirements.txt file.

  • additional requirements. pip3 install $(python3 util/taskcluster.py --decoder)

I have CUDA Version 10.0.130, CuDNN 7.4.2.24, tensorflow-gpu==1.14.0.

I use this command to run my fine-tuning process:

python DeepSpeech.py --n_hidden 2048 --checkpoint_dir deepspeech-0.6.1-checkpoint/ --epochs 3 --train_files train_full.csv --dev_files val_full.csv --test_files test_full.csv --learning_rate 0.0001 --use_cudnn_rnn --use_allow_growth

And I get this error:

WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
W0513 10:13:31.574302 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0513 10:13:31.661218 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0513 10:13:31.661349 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0513 10:13:31.661422 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:13:31.762649 140563218061120 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:13:31.762798 140563218061120 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:13:31.762930 140563218061120 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:13:31.804666 140563218061120 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0513 10:13:31.835743 140563218061120 deprecation.py:323] From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:13:31.964800 140563218061120 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fd75596c748>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0513 10:13:32.247514 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0513 10:13:32.384267 140563218061120 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I0513 10:13:32.385055 140563218061120 saver.py:1280] Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
Traceback (most recent call last):
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
	 [[{{node save/CudnnRNNCanonicalToParams}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 555, in train
    loaded = try_loading(session, checkpoint_saver, 'checkpoint', 'most recent')
  File "DeepSpeech.py", line 405, in try_loading
    saver.restore(session, checkpoint_path)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
	 [[node save/CudnnRNNCanonicalToParams (defined at DeepSpeech.py:495) ]]

Original stack trace for 'save/CudnnRNNCanonicalToParams':
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 495, in train
    checkpoint_saver = tfv1.train.Saver(max_to_keep=FLAGS.max_to_keep)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
    self.build()
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 837, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 875, in _build
    build_restore=build_restore)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 744, in restore
    restored_tensors)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 221, in tf_canonical_to_opaque
    opaque_params = self._cu_canonical_to_opaque(cu_weights, cu_biases)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 271, in _cu_canonical_to_opaque
    direction=self._direction)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 917, in cudnn_rnn_canonical_to_params
    seed=seed, seed2=seed2, name=name)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

I already tried to use docker with cuda to prevent mismatching and broken dependencies on my machine.
But then I face different type of error, I double-checked and I don’t have characters in my transcripts which are not available in alphabet.txt in the checkpoing folder.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
W0513 10:19:02.836340 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0513 10:19:02.937894 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0513 10:19:02.938112 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0513 10:19:02.938226 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:19:03.067150 140199107434304 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:19:03.067367 140199107434304 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:19:03.067694 140199107434304 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:19:03.125893 140199107434304 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f8243d8a6a0>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0513 10:19:03.163637 140199107434304 deprecation.py:323] From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0513 10:19:03.472953 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0513 10:19:03.634698 140199107434304 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I0513 10:19:03.635579 140199107434304 saver.py:1280] Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
E Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
E 
E No OpKernel was registered to support Op 'CudnnRNNCanonicalToParams' used by node tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams (defined at DeepSpeech.py:119) with these attrs: [seed=4568, dropout=0, num_params=8, input_mode="linear_input", T=DT_FLOAT, direction="unidirectional", rnn_mode="lstm", seed2=240]
E Registered devices: [CPU, XLA_CPU, XLA_GPU]
E Registered kernels:
E   device='GPU'; T in [DT_DOUBLE]
E   device='GPU'; T in [DT_FLOAT]
E   device='GPU'; T in [DT_HALF]
E 
E 	 [[tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams]]
E The checkpoint in deepspeech-0.6.1-checkpoint/best_dev-233784 does not match the shapes of the model. Did you change alphabet.txt or the --n_hidden parameter between train runs using the same checkpoint dir? Try moving or removing the contents of deepspeech-0.6.1-checkpoint/best_dev-233784.

If you have any suggestions how to fix it, please, help. :slight_smile:

the error is right here … your cudnn setup is wrong. Try 7.6

Or cuDNN 7.5, which is what the v0.6.1 documentation says: https://deepspeech.readthedocs.io/en/v0.6.1/USING.html#cuda-dependency

The same error with libcudnn7_7.6.2.24-1+cuda10.0_amd64.deb :

WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
W0513 10:51:09.133306 139684631988032 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0513 10:51:09.220713 139684631988032 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0513 10:51:09.220843 139684631988032 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0513 10:51:09.220914 139684631988032 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:51:09.321336 139684631988032 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:51:09.321515 139684631988032 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 10:51:09.321645 139684631988032 deprecation.py:506] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:51:09.363153 139684631988032 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0513 10:51:09.394001 139684631988032 deprecation.py:323] From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 10:51:09.522720 139684631988032 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7f0ac5c73748>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0513 10:51:09.803131 139684631988032 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0513 10:51:09.939387 139684631988032 deprecation.py:323] From /home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I0513 10:51:09.940295 139684631988032 saver.py:1280] Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
Traceback (most recent call last):
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
	 [[{{node save/CudnnRNNCanonicalToParams}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 555, in train
    loaded = try_loading(session, checkpoint_saver, 'checkpoint', 'most recent')
  File "DeepSpeech.py", line 405, in try_loading
    saver.restore(session, checkpoint_path)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
	 [[node save/CudnnRNNCanonicalToParams (defined at DeepSpeech.py:495) ]]

Original stack trace for 'save/CudnnRNNCanonicalToParams':
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 495, in train
    checkpoint_saver = tfv1.train.Saver(max_to_keep=FLAGS.max_to_keep)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
    self.build()
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 837, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 875, in _build
    build_restore=build_restore)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 744, in restore
    restored_tensors)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 221, in tf_canonical_to_opaque
    opaque_params = self._cu_canonical_to_opaque(cu_weights, cu_biases)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 271, in _cu_canonical_to_opaque
    direction=self._direction)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 917, in cudnn_rnn_canonical_to_params
    seed=seed, seed2=seed2, name=name)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/home/dash/projects/DeepSpeech/DeepSpeech-0.6.1/env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Should I somehow specify directly which cudnn to use? I have CUDA 10.1 installed on my machine as default and I specify in ~/.bashrc directly to use CUDA 10.0 with this line:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64

My CuDNN is located here after installation:
/usr/local/cuda-10.0/lib64/

There are 4 files:
-rwxr-xr-x 1 root root 270M Jan 11 17:29 libcudnn.so*
-rwxr-xr-x 1 root root 270M Jan 11 17:29 libcudnn.so.7*
-rwxr-xr-x 1 root root 270M Jan 11 17:29 libcudnn.so.7.0.5*
-rw-r–r-- 1 root root 263M Jan 11 17:29 libcudnn_static.a

Is everything is setted up properly now?

This can’t work if you have 10.1 installed by default, it’s going to be picked by your linker since it’s the first one. You need 10.0 to be first / only.

IS this CuDNN 7.0 ? That’s wrong.

Thank you for the answer!
I used this image https://hub.docker.com/layers/nvidia/cuda/10.0-cudnn7-devel-ubuntu18.04/images/sha256-e277b9eef79d6995b10d07e30228daa9e7d42f49bcfc29d512c1534b42d91841 to create a proper set of dependencies.
Then I installed all of the previously mentioned dependencies: tensorflow-gpu==1.14.0 + all dependencies from the requirements.txt + additionally $(python3 util/taskcluster.py --decoder)

Then I use the same command to run my fine-tuning process:
python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir deepspeech-0.6.1-checkpoint/ --epochs 3 --train_files train_full.csv --dev_files val_full.csv --test_files test_full.csv --learning_rate 0.0001 --use_cudnn_rnn --use_allow_growth

And eventually, I get the same type of error again:

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
W0513 13:50:08.969976 140590735099712 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0513 13:50:09.070896 140590735099712 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0513 13:50:09.071090 140590735099712 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0513 13:50:09.071287 140590735099712 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 13:50:09.201271 140590735099712 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:342: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 13:50:09.201562 140590735099712 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0513 13:50:09.201940 140590735099712 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py:345: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fdd72acb668>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fdd72acb668>>: AttributeError: module 'gast' has no attribute 'Num'
W0513 13:50:09.258405 140590735099712 ag_logging.py:145] Entity <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fdd72acb668>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _CudnnRNN.call of <tensorflow.contrib.cudnn_rnn.python.layers.cudnn_rnn.CudnnLSTM object at 0x7fdd72acb668>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0513 13:50:09.296621 140590735099712 deprecation.py:323] From DeepSpeech.py:236: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0513 13:50:09.608414 140590735099712 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/slot_creator.py:193: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0513 13:50:09.775830 140590735099712 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I0513 13:50:09.776924 140590735099712 saver.py:1280] Restoring parameters from deepspeech-0.6.1-checkpoint/best_dev-233784
I Restored variables from most recent checkpoint at deepspeech-0.6.1-checkpoint/best_dev-233784, step 233784
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000                            Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size]: [1, 2048, 2048, 1, 89999, 1] 
	 [[{{node tower_0/cudnn_lstm/CudnnRNNV3}}]]
	 [[tower_0/gradients/tower_0/BiasAdd_4_grad/BiasAddGrad/_73]]
  (1) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size]: [1, 2048, 2048, 1, 89999, 1] 
	 [[{{node tower_0/cudnn_lstm/CudnnRNNV3}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 640, in train
    train_loss, _ = run_set('train', epoch, train_init_op)
  File "DeepSpeech.py", line 602, in run_set
    feed_dict=feed_dict)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size]: [1, 2048, 2048, 1, 89999, 1] 
	 [[node tower_0/cudnn_lstm/CudnnRNNV3 (defined at DeepSpeech.py:119) ]]
	 [[tower_0/gradients/tower_0/BiasAdd_4_grad/BiasAddGrad/_73]]
  (1) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size]: [1, 2048, 2048, 1, 89999, 1] 
	 [[node tower_0/cudnn_lstm/CudnnRNNV3 (defined at DeepSpeech.py:119) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node tower_0/cudnn_lstm/CudnnRNNV3:
 tower_0/Reshape_2 (defined at DeepSpeech.py:177)	
 tower_0/IteratorGetNext (defined at DeepSpeech.py:222)

Input Source operations connected to node tower_0/cudnn_lstm/CudnnRNNV3:
 tower_0/Reshape_2 (defined at DeepSpeech.py:177)	
 tower_0/IteratorGetNext (defined at DeepSpeech.py:222)

Original stack trace for 'tower_0/cudnn_lstm/CudnnRNNV3':
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 477, in train
    gradients, loss, non_finite_files = get_tower_results(iterator, optimizer, dropout_rates)
  File "DeepSpeech.py", line 303, in get_tower_results
    avg_loss, non_finite_files = calculate_mean_edit_distance_and_loss(iterator, dropout_rates, reuse=i > 0)
  File "DeepSpeech.py", line 230, in calculate_mean_edit_distance_and_loss
    logits, _ = create_model(batch_x, batch_seq_len, dropout, reuse=reuse, rnn_impl=rnn_impl)
  File "DeepSpeech.py", line 181, in create_model
    output, output_state = rnn_impl(layer_3, seq_length, previous_state, reuse)
  File "DeepSpeech.py", line 119, in rnn_impl_cudnn_rnn
    sequence_lengths=seq_length)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 537, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 634, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 146, in wrapper
    ), args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 446, in converted_call
    return _call_unconverted(f, args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 253, in _call_unconverted
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py", line 440, in call
    training)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_rnn.py", line 518, in _forward
    seed=self._seed)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 1038, in _cudnn_rnn
    outputs, output_h, output_c, _, _ = gen_cudnn_rnn_ops.cudnn_rnnv3(**args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 1627, in cudnn_rnnv3
    time_major=time_major, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Am I doing something wrong again?
This image is used directly in repository’s Dockerfile, so I thought that all of the CUDA + CuDNN dependencies are working correctly.

Also, I noticed that something is processing on my GPU, because memory starts allocating, gets to almost 10.5Gb and then I see an error.
Could it be somehow related to the lack of GPU memory?

(Experimented with --train_batch_size 1 --test_batch_size 1 --dev_batch_size 1 - same behaviour )