It seems like it is not running on GPU which I get a prompt after I run nvidia-smi
±----------------------------------------------------------------------------+
| NVIDIA-SMI 418.43 Driver Version: 418.43 CUDA Version: 10.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro P4000 Off | 00000000:AF:00.0 Off | N/A |
| 61% 76C P0 76W / 105W | 1488MiB / 8119MiB | 100% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 10523 C python3 77MiB |
| 0 13468 C colmap 1401MiB |
±----------------------------------------------------------------------------+
initially did not use mixed precision the process in even slower
lissyx
((slow to reply) [NOT PROVIDING SUPPORT])
4
So you are using default batch size of 1. Please read python DeepSpeech.py --helpfull and set batch size. We can’t suggest value since it depends on your data and your GPU, so you need to find the best compromise between training speech, learning and not getting GPU OOM.
libcudart.so.10.0’; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2020-04-14 02:06:58.877930: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcublas.so.10.0’; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2020-04-14 02:06:58.877979: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcufft.so.10.0’; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2020-04-14 02:06:58.878030: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcurand.so.10.0’; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-04-14 02:06:58.878078: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcusolver.so.10.0’; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2020-04-14 02:06:58.878126: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcusparse.so.10.0’; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
seems like it require cuda10.0 but current version is 10.1?