Hello ! I would like some help with this problem I have been having for a while now and haven’t been able to solve.
I have CUDA 10.0 and CudNN 7.5.0 installed with driver 430.34.
My system has a Nvidia RTX 2080.
I installed the requirements from the text file provided, then I uninstalled tensorflow
and installed tensorflow-gpu==1.14.0
However when I try to run DeepSpeech.py, it returns the following
(speech) roge@Alien:~/AI/DeepSpeech$ python3 ./DeepSpeech.py --train_files '/home/roge/AI/es/clips/train.csv' --dev_files '/home/roge/AI/es/clips/dev.csv' --test_files '/home/roge/AI/es/clips/test.csv'
W0715 04:21:45.741396 140378568243008 deprecation.py:323] From /home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0715 04:21:45.789885 140378568243008 deprecation.py:323] From /home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0715 04:21:45.790004 140378568243008 deprecation.py:323] From /home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0715 04:21:45.790069 140378568243008 deprecation.py:323] From /home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0715 04:21:45.911934 140378568243008 deprecation.py:506] From /home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0715 04:21:46.388525 140378568243008 deprecation.py:323] From /home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I Initializing variables...
Traceback (most recent call last):
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1339, in _run_fn
self._extend_graph()
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1374, in _extend_graph
tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation tower_0/DeserializeSparse: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='' supported_device_types_=[CPU] possible_devices_=[]
DeserializeSparse: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
tower_0/DeserializeSparse (DeserializeSparse) /device:GPU:0
Op: DeserializeSparse
Node attrs: dtype=DT_INT32, Tserialized=DT_VARIANT
Registered kernels:
device='CPU'; Tserialized in [DT_VARIANT]
device='CPU'; Tserialized in [DT_STRING]
[[{{node tower_0/DeserializeSparse}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./DeepSpeech.py", line 817, in <module>
tfv1.app.run(main)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "./DeepSpeech.py", line 801, in main
train()
File "./DeepSpeech.py", line 443, in train
session.run(initializer)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation tower_0/DeserializeSparse: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='' supported_device_types_=[CPU] possible_devices_=[]
DeserializeSparse: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
tower_0/DeserializeSparse (DeserializeSparse) /device:GPU:0
Op: DeserializeSparse
Node attrs: dtype=DT_INT32, Tserialized=DT_VARIANT
Registered kernels:
device='CPU'; Tserialized in [DT_VARIANT]
device='CPU'; Tserialized in [DT_STRING]
[[node tower_0/DeserializeSparse (defined at ./DeepSpeech.py:184) ]]
Original stack trace for 'tower_0/DeserializeSparse':
File "./DeepSpeech.py", line 817, in <module>
tfv1.app.run(main)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "./DeepSpeech.py", line 801, in main
train()
File "./DeepSpeech.py", line 401, in train
gradients, loss = get_tower_results(iterator, optimizer, dropout_rates)
File "./DeepSpeech.py", line 254, in get_tower_results
avg_loss = calculate_mean_edit_distance_and_loss(iterator, dropout_rates, reuse=i > 0)
File "./DeepSpeech.py", line 184, in calculate_mean_edit_distance_and_loss
(batch_x, batch_seq_len), batch_y = iterator.get_next()
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 427, in get_next
return self._structure._from_tensor_list(flat_ret)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/util/structure.py", line 471, in _from_tensor_list
flat_ret.append(structure._from_tensor_list(sub_value))
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/util/structure.py", line 645, in _from_tensor_list
return self._from_compatible_tensor_list(flat_value)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/data/util/structure.py", line 649, in _from_compatible_tensor_list
flat_value[0], dtype=self._dtype, rank=self._dense_shape.ndims)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/ops/sparse_ops.py", line 2065, in deserialize_sparse
gen_sparse_ops.deserialize_sparse(serialized_sparse, dtype, name=name))
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/ops/gen_sparse_ops.py", line 474, in deserialize_sparse
name=name)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/roge/AI/DeepSpeech/speech/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
When I install tensorflow this error goes away and I can run the script normally, but DeepSpeech does not use the GPU obviously. Any help would be appreciated. Thanks!
