EOFError while validation

Archives DeepSpeech
,
#1

I am using DeepSpeech 0.7.4 for transfer learning from the default English model to Urdu. I amtrying to make a subtitle generating platform for Urdu videos. Following is the code I am using to train:

%cd /content/DeepSpeech
!python3 DeepSpeech.py --train_cudnn True --early_stop True \
    --es_epochs 10 --drop_source_layers 2 \
    --n_hidden 2048 --epochs 50 \
    --learning_rate 0.0001 \
    --alphabet_config_path /content/drive/MyDrive/alphabet-urdu.txt \
    --save_checkpoint_dir /content/model_checkpoints/deepspeech-0.7.4-checkpoint/ \
    --load_checkpoint_dir /content/model_checkpoints/deepspeech-0.7.4-checkpoint/ \
    --export_dir /content/drive/MyDrive/models/ --export_file_name 'ft_model' \
    --train_files /content/train.csv \
    --dev_files /content/dev.csv \
    --test_files /content/test.csv \
    --scorer_path /content/drive/MyDrive/deepspeech-0.7.4-models.scorer

I am getting an EOF Error just when it starts validating in first Epoch while the whole training Epoch runs smoothly, here is the complete run:

/content/DeepSpeech
I0424 07:34:31.799855 140052807796608 utils.py:157] NumExpr defaulting to 2 threads.
I Loading best validating checkpoint from /content/model_checkpoints/deepspeech-0.7.4-checkpoint/best_dev-732522
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Initializing variable: layer_5/bias
I Initializing variable: layer_5/bias/Adam
I Initializing variable: layer_5/bias/Adam_1
I Initializing variable: layer_5/weights
I Initializing variable: layer_5/weights/Adam
I Initializing variable: layer_5/weights/Adam_1
I Initializing variable: layer_6/bias
I Initializing variable: layer_6/bias/Adam
I Initializing variable: layer_6/bias/Adam_1
I Initializing variable: layer_6/weights
I Initializing variable: layer_6/weights/Adam
I Initializing variable: layer_6/weights/Adam_1
I Initializing variable: learning_rate
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 1:09:48 | Steps: 9734 | Loss: 183.179909   
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /content/dev.csvTraceback (most recent call last):
  File "DeepSpeech.py", line 12, in <module>
    ds_train.run_script()
  File "/content/DeepSpeech/training/deepspeech_training/train.py", line 955, in run_script
    absl.app.run(main)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/content/DeepSpeech/training/deepspeech_training/train.py", line 927, in main
    train()
  File "/content/DeepSpeech/training/deepspeech_training/train.py", line 605, in train
    set_loss, steps = run_set('dev', epoch, init_op, dataset=source)
  File "/content/DeepSpeech/training/deepspeech_training/train.py", line 563, in run_set
    exception_box.raise_if_set()
  File "/content/DeepSpeech/training/deepspeech_training/util/helpers.py", line 124, in raise_if_set
    raise exception  # pylint: disable = raising-bad-type
  File "/content/DeepSpeech/training/deepspeech_training/util/helpers.py", line 132, in do_iterate
    yield from iterable()
  File "/content/DeepSpeech/training/deepspeech_training/util/feeding.py", line 110, in generate_values
    for sample_index, sample in enumerate(samples):
  File "/content/DeepSpeech/training/deepspeech_training/util/augmentations.py", line 221, in apply_sample_augmentations
    yield from pool.imap(_augment_sample, timed_samples())
  File "/content/DeepSpeech/training/deepspeech_training/util/helpers.py", line 103, in imap
    for obj in self.pool.imap(fun, self._limit(it)):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 140, in _helper_reraises_exception
    raise ex
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 292, in _guarded_task_generation
    for i, x in enumerate(iterable):
  File "/content/DeepSpeech/training/deepspeech_training/util/helpers.py", line 96, in _limit
    for obj in it:
  File "/content/DeepSpeech/training/deepspeech_training/util/augmentations.py", line 201, in timed_samples
    for sample_index, sample in enumerate(samples):
  File "/content/DeepSpeech/training/deepspeech_training/util/sample_collections.py", line 318, in __getitem__
    return load_sample(sample_spec[0], label=sample_spec[2] if self.labeled else None)
  File "/content/DeepSpeech/training/deepspeech_training/util/sample_collections.py", line 75, in load_sample
    return LabeledSample(audio_type, audio_file.read(), label, sample_id=filename)
  File "/content/DeepSpeech/training/deepspeech_training/util/sample_collections.py", line 47, in __init__
    super().__init__(audio_type, raw_data, audio_format=audio_format, sample_id=sample_id)
  File "/content/DeepSpeech/training/deepspeech_training/util/audio.py", line 74, in __init__
    self.duration = read_duration(audio_type, self.audio)
  File "/content/DeepSpeech/training/deepspeech_training/util/audio.py", line 367, in read_duration
    return read_wav_duration(audio_file)
  File "/content/DeepSpeech/training/deepspeech_training/util/audio.py", line 356, in read_wav_duration
    with wave.open(wav_file, 'rb') as wav_file_reader:
  File "/usr/lib/python3.7/wave.py", line 510, in open
    return Wave_read(f)
  File "/usr/lib/python3.7/wave.py", line 164, in __init__
    self.initfp(f)
  File "/usr/lib/python3.7/wave.py", line 129, in initfp
    self._file = Chunk(file, bigendian = 0)
  File "/usr/lib/python3.7/chunk.py", line 63, in __init__
    raise EOFError
EOFError

I have checked all WAV files for integrity using ffmpeg so I don’t understand why this is here. Is this WAV issue or CSV file issue? Also, Is there a way that DeepSpeech can ignore the faulty WAV files and continue training? so the progress isn’t lost because of one file.

The specifications for environment are:
Google Colab (Linux Ubuntu 18.04)
Python 3.6.9
tensorflow-gpu 1.15.2
CUDA 10.0

Any help would be much appreciated, thank you so much!

#2

Please use 0.9.3

From your stack it’s obviously a WAV file that is broken.

No, please fix your dataset.

1 Like
#3

It’s best to use 0.9.3 as lissyx points out. You should also put your data into Common Voice format and then take advantage of the bin/import_cv2.py script which will do some basic checks for wav file sanity etc.

#4

@lissyx @ftyers Thank you for your replies, I will be checking my WAV files.

I am afraid the dependencies will be different or of a different version for 0.9.3 or will the same code work for that? Also is there a performance difference between 0.7.4 and 0.9.3?

#5

You should be able to use it on CoLab.

#6

Okay, that sounds cool, I’m gonna try that. Thanks!

#7

Perfs should be better with 0.9

1 Like
#8

okay, thank you! let me try that.

#9

Also, a little off-topic but I have a confusion. Transfer learning section in the documentation does not mention making your own language model or scorer file but online help and tutorials say that you have to generate one yourself if you are transfer learning to a different language (in my case, Urdu). Can you kindly clarify if it needs to be generated?

#10

This is a common question. I have a script to generate the scorers. I realise it is a bit complex, but I’m happy to explain any parts you might find difficult. You can also find us on Matrix.

1 Like
#11

So that means it is necessary for different language? Then why does the documentation not mention that?

And thanks for the script, I’ll try it and get back to you.

#12

As for why the documentation does not mention it, I don’t know. But I’m sure the team would welcome suggestions for making the documentation better! Have you checked out the Playbook ? Particularly the part about scorers.

1 Like
#13

Thanks for the resources ftyers, that helped me with generating the scorer.

1 Like