Hi I’m trying to finetune the DeepSpeech 0.9.3 model using Google Colab. Currently I’m getting the following error. For now, I’m just trying to get it to run so I haven’t added in the 0.9.3 checkpoints yet.
I0331 18:49:15.880641 139792172775296 utils.py:157] NumExpr defaulting to 2 threads.
I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
I Initializing all variables.
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 Traceback (most recent call last):
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 1365, in _do_call
return fn(*args)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 1350, in _run_fn
target_list, run_metadata)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[{{node tower_0/IteratorGetNext}}]]During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 572, in run_set
feed_dict=feed_dict)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 1359, in _do_run
run_metadata)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py”, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[node tower_0/IteratorGetNext (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]Original stack trace for ‘tower_0/IteratorGetNext’:
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 982, in run_script
absl.app.run(main)
File “/usr/local/lib/python3.7/dist-packages/absl/app.py”, line 303, in run
_run_main(main, args)
File “/usr/local/lib/python3.7/dist-packages/absl/app.py”, line 251, in _run_main
sys.exit(main(argv))
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 954, in main
train()
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 484, in train
gradients, loss, non_finite_files = get_tower_results(iterator, optimizer, dropout_rates)
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 317, in get_tower_results
avg_loss, non_finite_files = calculate_mean_edit_distance_and_loss(iterator, dropout_rates, reuse=i > 0)
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 236, in calculate_mean_edit_distance_and_loss
batch_filenames, (batch_x, batch_seq_len), batch_y = iterator.get_next()
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/data/ops/iterator_ops.py”, line 426, in get_next
name=name)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_dataset_ops.py”, line 2518, in iterator_get_next
output_shapes=output_shapes, name=name)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/op_def_library.py”, line 794, in _apply_op_helper
op_def=op_def)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py”, line 3357, in create_op
attrs, op_def, compute_device)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py”, line 3426, in _create_op_internal
op_def=op_def)
File “/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py”, line 1748, in init
self._traceback = tf_stack.extract_stack()During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 982, in run_script
absl.app.run(main)
File “/usr/local/lib/python3.7/dist-packages/absl/app.py”, line 303, in run
_run_main(main, args)
File “/usr/local/lib/python3.7/dist-packages/absl/app.py”, line 251, in _run_main
sys.exit(main(argv))
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 954, in main
train()
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 607, in train
train_loss, _ = run_set(‘train’, epoch, train_init_op)
File “/content/DeepSpeech/training/deepspeech_training/train.py”, line 575, in run_set
exception_box.raise_if_set()
File “/content/DeepSpeech/training/deepspeech_training/util/helpers.py”, line 149, in raise_if_set
raise exception # pylint: disable = raising-bad-type
File “/content/DeepSpeech/training/deepspeech_training/util/helpers.py”, line 157, in do_iterate
yield from iterable()
File “/content/DeepSpeech/training/deepspeech_training/util/feeding.py”, line 104, in generate_values
samples = samples_from_sources(sources, buffering=buffering, labeled=True, reverse=reverse)
File “/content/DeepSpeech/training/deepspeech_training/util/sample_collections.py”, line 622, in samples_from_sources
return samples_from_source(sample_sources[0], buffering=buffering, labeled=labeled, reverse=reverse)
File “/content/DeepSpeech/training/deepspeech_training/util/sample_collections.py”, line 584, in samples_from_source
return CSV(sample_source, labeled=labeled, reverse=reverse)
File “/content/DeepSpeech/training/deepspeech_training/util/sample_collections.py”, line 543, in init
wav_filename = Path(row[‘wav_filename’])
KeyError: 'wav_filename
I looked through the forum and read through the posts that talked about this issue but there weren’t any successful resolutions discussed.
I have successfully trained using my current notebook setup before so I’m unsure about what is different this time. It is able to initialize the training but fails during the first step.
My command is as follows:
!python3 DeepSpeech.py
–n_hidden 2048
–checkpoint_dir fine_tuning_checkpoints/
–epochs 3
–train_files train.csv
–dev_files dev.csv
–test_files test.csv
–learning_rate 0.0001
–export_dir output_models/
–use_allow_growth true
–export_tflite
A couple of lines from train.csv are:
wav_filename,wav_filesize,transcript
/path/to/file/wav (1).wav,183716, alice was beginning to get very tired
/path/to/file/wav (2523).wav, 163244, are you hoping some other kind of monster will come along
My dev.csv are test.csv are formatted the same way.
Apologies if this isn’t clear I’m using Deepspeech for my senior undergrad project so I’m still learning. Please let me know what other information I can provide.