Error with Deepspeech 0.5.1

sayantangangs.91 · July 14, 2019, 1:41pm

I’ve started updating to Deepspeech 0.5.1. To do the same, I’m using the checkpoint released for transer learning.

It seems there is an error that propped up as the trainng was undergoing. Not sure what was it. I’m attaching a screenshot of it.

Following is the code I used:

python DeepSpeech.py --n_hidden 2048 --checkpoint_dir /home/sayantan/Desktop/ai_learning/new_deepspeech/deepspeech-0.5.1-checkpoint --export_dir /home/sayantan/Desktop/ai_learning/new_deepspeech/trained_model/ --validation_step 1  --train_batch_size 4 --dev_batch_size 4 --test_batch_size 16 --epoch -2 --train_files my-train.csv --dev_files my-dev.csv --test_files my-test.csv --learning_rate 0.00002 --dropout_rate 0.15

lissyx · July 14, 2019, 2:23pm

Which makes it hard to read and search for information. We won’t be able to efficiently help you until you share complete text output.

sayantangangs.91 · July 14, 2019, 3:41pm

I understand. I’m pasting the error log

I Restored variables from most recent checkpoint at /home/sayantan/Desktop/ai_learning/new_deepspeech/deepspeech-0.5.1-checkpoint/train-469905, step 469905
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:35:44 | Steps: 2366 | Loss: 7.414255                                                                                                                                    
Epoch 0 |   Training | Elapsed Time: 0:36:07 | Steps: 2387 | Loss: 7.434547                                                                                                                                   
Epoch 0 |   Training | Elapsed Time: 1:00:03 | Steps: 3669 | Loss: 7.863337                                                                                                                                   
W0714 12:50:50.478254 140394913916736 deprecation.py:323] From /home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
Epoch 0 |   Training | Elapsed Time: 1:01:16 | Steps: 3731 | Loss: 7.924674                                                                                                                                   
Epoch 0 |   Training | Elapsed Time: 2:58:50 | Steps: 9101 | Loss: 10.700255                                                                                                                                  
Epoch 0 |   Training | Elapsed Time: 3:01:42 | Steps: 9220 | Loss: 10.774141                                                                                                                                  
Epoch 0 |   Training | Elapsed Time: 3:29:46 | Steps: 10367 | Loss: 11.485737                                                                                                                                 
Traceback (most recent call last):
 File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
 File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
 File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Header mismatch: Expected fmt  but found JUNK
 [[{{node DecodeWav}}]]
 [[tower_0/IteratorGetNext]]
 [[layer_2/bias/read/_41]]
(1) Invalid argument: Header mismatch: Expected fmt  but found JUNK
 [[{{node DecodeWav}}]]
 [[tower_0/IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "DeepSpeech.py", line 836, in <module>
tf.app.run(main)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "DeepSpeech.py", line 820, in main
train()
File "DeepSpeech.py", line 510, in train
train_loss, _ = run_set('train', epoch, train_init_op)
File "DeepSpeech.py", line 483, in run_set
feed_dict=feed_dict)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/sayantan/anaconda3/envs/deepspeech_0_5_train/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Header mismatch: Expected fmt  but found JUNK
 [[{{node DecodeWav}}]]
 [[tower_0/IteratorGetNext]]
 [[layer_2/bias/read/_41]]
(1) Invalid argument: Header mismatch: Expected fmt  but found JUNK
 [[{{node DecodeWav}}]]
 [[tower_0/IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

This is the complete log (the starting part is the epochs running followed by the errors). Thanks a lot again for your quick reply.

sayantangangs.91 · July 14, 2019, 6:51pm

Hey @lissyx , I’ve pasted the error log. In case you need anythong more that I can help with, I shall do it. You know, it was working grea with Deepspeech 0.4.1 and then I updated the environment, and things are working great, but still this error.

lissyx · July 15, 2019, 7:29am

Well, it means nothing, we upgraded the TensorFlow version.

That and the stack of the graph would suggest a bogus WAV file.

sayantangangs.91 · July 15, 2019, 7:35am

Exactly. That’s what I thought.

Ohh. Sure. Need to figure the neddle in the haystack.

lissyx · July 15, 2019, 7:46am

Binary search on your .csv files that’s the best.

sayantangangs.91 · July 15, 2019, 8:15am

Well sure. Shall try that. Good idea. Thanks a lot.