Hi, I am trying to train DeepSpeech-0.8.1 on my data set using transfer learning. I have 5000 training examples, 2000 Dev and 1000 test set. I am getting Fatal Python error can any one help me with this?
I Loading best validating checkpoint from deepspeech-0.8.1-checkpoint/best_dev-732522
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/bias/Adam
I Loading variable from checkpoint: layer_6/bias/Adam_1
I Loading variable from checkpoint: layer_6/weights
I Loading variable from checkpoint: layer_6/weights/Adam
I Loading variable from checkpoint: layer_6/weights/Adam_1
I Initializing variable: learning_rate
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:08:19 | Steps: 1377 | Loss: 138.792073 2020-08-24 10:34:08.297475: F tensorflow/stream_executor/cuda/cuda_dnn.cc:1403] Check failed: max_seq_length > 0 (0 vs. 0)
Fatal Python error: Aborted
Thread 0x00007fde9cffd700 (most recent call first):
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379 in _recv
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407 in _recv_bytes
File "/usr/lib/python3.6/multiprocessing/connection.py", line 250 in recv
File "/usr/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fde9d7fe700 (most recent call first):
File "/storage/Hassaan/DeepSpeech-0.8.1/training/deepspeech_training/util/helpers.py", line 97 in _limit
File "/usr/lib/python3.6/multiprocessing/pool.py", line 290 in _guarded_task_generation
File "/usr/lib/python3.6/multiprocessing/pool.py", line 419 in _handle_tasks
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fde9dfff700 (most recent call first):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fdefb35c700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fdefbb5d700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fdfe0ff9700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fe05eb21740 (most recent call first):
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443 in _call_tf_sessionrun
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350 in _run_fn
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365 in _do_call
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359 in _do_run
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180 in _run
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956 in run
File "/storage/Hassaan/DeepSpeech-0.8.1/training/deepspeech_training/train.py", line 566 in run_set
File "/storage/Hassaan/DeepSpeech-0.8.1/training/deepspeech_training/train.py", line 601 in train
File "/storage/Hassaan/DeepSpeech-0.8.1/training/deepspeech_training/train.py", line 933 in main
File "/home/ubuntu/.local/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
File "/home/ubuntu/.local/lib/python3.6/site-packages/absl/app.py", line 299 in run
File "/storage/Hassaan/DeepSpeech-0.8.1/training/deepspeech_training/train.py", line 961 in run_script
File "Dpy.py", line 12 in <module>
I am using following command to run deepspeech
python DeepSpeech.py --n_hidden 2048 --checkpoint_dir deepspeech-0.8.1-checkpoint --epochs 3 --train_files ../Audios/Data/train/train.csv --dev_files ../Audios/Data/dev/dev.csv --test_files ../Audios/Data/test/test.csv --learning_rate 0.0001 --use_allow_growth --train_cudnn
Package Version
----------------------- -------------------
decorator 4.4.2
deepspeech-gpu 0.8.1
ds-ctcdecoder 0.9.0a5
future 0.18.2
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.2
numpy 1.18.5
pandas 1.1.0
pandocfilters 1.4.2
pip 20.2
progressbar2 3.47.0
prompt-toolkit 3.0.5
pycrypto 2.6.1
pydub 0.24.1
tensorflow-estimator 1.15.1
tensorflow-gpu 1.15.0