Error deepspeech-gpu

Archives DeepSpeech
1

DeepSpeech works well.
But deepspeech-gpu gives below mistake

deepspeech -h
Traceback (most recent call last):
File “/home/farruh/tmp/deepspeech-gpu-venv/lib/python3.6/site-packages/deepspeech/impl.py”, line 14, in swig_import_helper
return importlib.import_module(mname)
File “/home/farruh/tmp/deepspeech-gpu-venv/lib/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 658, in _load_unlocked
File “”, line 571, in module_from_spec
File “”, line 922, in create_module
File “”, line 219, in _call_with_frames_removed
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/farruh/tmp/deepspeech-gpu-venv/bin/deepspeech”, line 5, in
from deepspeech.client import main
File “/home/farruh/tmp/deepspeech-gpu-venv/lib/python3.6/site-packages/deepspeech/init.py”, line 23, in
from deepspeech.impl import PrintVersions as printVersions
File “/home/farruh/tmp/deepspeech-gpu-venv/lib/python3.6/site-packages/deepspeech/impl.py”, line 17, in
_impl = swig_import_helper()
File “/home/farruh/tmp/deepspeech-gpu-venv/lib/python3.6/site-packages/deepspeech/impl.py”, line 16, in swig_import_helper
return importlib.import_module(’_impl’)
File “/home/farruh/tmp/deepspeech-gpu-venv/lib/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named ‘_impl’

Version on deep speech is:
pip list | grep deepspeech
deepspeech-gpu 0.6.0

Cuda version:
nvcc --version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

Tensorflow version:
pip3 list | grep tensorflow
tensorflow-estimator 1.14.0
tensorflow-gpu 1.14.0

2

As documented, you need CUDA 10.0 accessible by the GPU runtime. You can just download and install in your home directory, and then use LD_LIBRARY_PATH to make it accessible.

3

Got you, I will try with CUDA 10. When update will come that can works with cuda 10.2.89?

4

Never, this depends on TensorFlow, and even r1.15 is CUDA 10.0.

5

@farruh As I said, you can always do a local installation of CUDA.

6

@lissyx Do you have a tutorial of how to do a local installation of CUDA?

7

No, but it’s really simple, and i think it’s documented by NVIDIA. Just download the tarball, extract it to e.g., $HOME/CUDA/ and then set LD_LIBRARY_PATH accordingly.

You might also need to install cudnn in this place as well.

1 Like
8

hi @lissyx:
I am trying to train deepspeech-gpu==0.6.1,with GPU nvidia rtx 2060, and my nvidia driver is 440.64, cuda10.2, cudnn 7.6.5 . and tensorflow-gpu==1.15。 i red the deepspeech-gpu==0.6.1 require cuda10,cudnn 7.6, so i install another cuda10.0, and then successly switch to cudnn 10.0。and then run deepspeech.py ,but i got the error :

tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node tower_0/conv1d (defined at /home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node tower_0/conv1d (defined at /home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
	 [[tower_0/gradients/tower_0/MatMul_2_grad/tuple/control_dependency_1/_109]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'tower_0/conv1d':
  File "DeepSpeech.py", line 974, in <module>
    absl.app.run(main)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 947, in main
    train()
  File "DeepSpeech.py", line 477, in train
    gradients, loss, non_finite_files = get_tower_results(iterator, optimizer, dropout_rates)
  File "DeepSpeech.py", line 303, in get_tower_results
    avg_loss, non_finite_files = calculate_mean_edit_distance_and_loss(iterator, dropout_rates, reuse=i > 0)
  File "DeepSpeech.py", line 230, in calculate_mean_edit_distance_and_loss
    logits, _ = create_model(batch_x, batch_seq_len, dropout, reuse=reuse, rnn_impl=rnn_impl)
  File "DeepSpeech.py", line 158, in create_model
    batch_x = create_overlapping_windows(batch_x)
  File "DeepSpeech.py", line 60, in create_overlapping_windows
    batch_x = tf.nn.conv1d(input=batch_x, filters=eye_filter, stride=1, padding='SAME')
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 574, in new_func
    return func(*args, **kwargs)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 574, in new_func
    return func(*args, **kwargs)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/ops/nn_ops.py", line 1681, in conv1d
    name=name)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_nn_ops.py", line 1071, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/zhangp/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

could you give me some advice?

9

seem i have get the answer :just set the flags : – use_allow_growth True if with deepspeech-gpu 0.6.1。 thanks!

10

You are wrong, deepspeech-gpu is only for inference, not for training.

11

I want to speed up training with GPU, just with deepspeech , can i do that?

12

Again, deepspeech-gpu IS NOT FOR TRAINING. If you have properly setup tensorflow-gpu then it will use your GPU.

13

ok, i have got it ,thank you!