How to train Deepspeech model on google colab?

Archives DeepSpeech
#1

Hi, guys I don’t own a gpu. So I’m trying to infering it on google colab. I get this error I think it is because of cuda
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/deepspeech/impl.py", line 14, in swig_import_helper return importlib.import_module(mname) File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 658, in _load_unlocked File "<frozen importlib._bootstrap>", line 571, in module_from_spec File "<frozen importlib._bootstrap_external>", line 922, in create_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed ImportError: libcusolver.so.9.0: cannot open shared object file: No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/bin/deepspeech", line 7, in <module> from deepspeech.client import main File "/usr/local/lib/python3.6/dist-packages/deepspeech/__init__.py", line 4, in <module> from deepspeech.impl import AudioToInputVector as audioToInputVector File "/usr/local/lib/python3.6/dist-packages/deepspeech/impl.py", line 17, in <module> _impl = swig_import_helper() File "/usr/local/lib/python3.6/dist-packages/deepspeech/impl.py", line 16, in swig_import_helper return importlib.import_module('_impl') File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ModuleNotFoundError: No module named '_impl'

Colab has following version of cuda and tensorflow:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148
mesh-tensorflow          0.0.4                

tensorflow               1.12.0               
tensorflow-hub           0.2.0                
tensorflow-metadata      0.9.0                
tensorflow-probability   0.5.0

How to solve this issue ? Is it not possible to run it on 9.2 instead of 9.0. ?? Waiting for reply

#2

TensorFlow upstream r1.12 is using CUDA 9.0, so we are building against it as well. You can rebuild against CUDA 9.2.

#3

I use the following code snippets:

# Download and Install CUDA-9.0
!wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run
!sh cuda_9.0.176_384.81_linux-run --silent --toolkit --toolkitpath=/usr/local/cuda-9.0 --override
# Set environment variables
import os
os.environ['LD_LIBRARY_PATH'] = "/usr/local/cuda-9.0/lib64:/usr/local/cuda-9.0/extras/CUPTI/lib64:" + os.environ['LD_LIBRARY_PATH']
os.environ['PATH'] = "/usr/local/cuda-9.0/bin:" + os.environ['PATH']
#4

You need to rebuild everything, sadly.

#5

Is rebuilding needed for the version 0.4? Cause I successfully trained on Colab with DeepSpeech v0.3 using these methods

#6

If you’re training on v0.4, uninstall the deepspeech package and remove the old libctc_decoder_with_kenlm.so, then you should have no binary dependencies on CUDA anymore.

#7

Assuming the TensorFlow that’s preinstalled in Colab is working fine.

#8

Have you completed your training deepspeech on google colab. Will you please your experience and issues you might had faced. It will be a help and guidance for newcomers like me to train Deepspeech engine on google colab.

1 Like
#9

Please don’t hijack 2 year old posts. Search the forum and start a new post with what you learned and what you want more details for. There were a lot of posts about colab in the last 2 years …