I am trying to train mozilla TTS tacotron2 on my private dataset.
Training starts fine but around the end of 1st epoch, I get following error.
Also, changed number of worker in config for data loading to 2 from 4 as suggested at discourse posts, but getting the same error again.
Please no screenshots, people can’t search in them.
As always, check the error msg itself, it tells you exactly what the problem is.
BTW, best practice is to name the wavs with a hash or very long number.
4 Likes
below is the error msg:
Traceback (most recent call last):
File “/home/ubuntu/drive_a/mayank/TTS/TTS/bin/train_tts.py”, line 715, in
main(args)
File “/home/ubuntu/drive_a/mayank/TTS/TTS/bin/train_tts.py”, line 627, in main
global_step, epoch, amp, speaker_mapping)
File “/home/ubuntu/drive_a/mayank/TTS/TTS/bin/train_tts.py”, line 144, in train
for num_iter, data in enumerate(data_loader):
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 345, in next
data = self._next_data()
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 856, in _next_data
return self._process_data(data)
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 881, in _process_data
data.reraise()
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/_utils.py”, line 395, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 1.
Original Traceback (most recent call last):
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py”, line 178, in _worker_loop
data = fetcher.fetch(index)
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File “/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File “/home/ubuntu/drive_a/mayank/TTS/TTS/tts/datasets/TTSDataset.py”, line 185, in getitem
return self.load_data(idx)
File “/home/ubuntu/drive_a/mayank/TTS/TTS/tts/datasets/TTSDataset.py”, line 124, in load_data
wav = np.asarray(self.load_wav(wav_file), dtype=np.float32)
File “/home/ubuntu/drive_a/mayank/TTS/TTS/tts/datasets/TTSDataset.py”, line 75, in load_wav
audio = self.ap.load_wav(filename)
File “/home/ubuntu/drive_a/mayank/TTS/TTS/utils/audio.py”, line 325, in load_wav
x, sr = sf.read(filename)
File “/home/ubuntu/.local/lib/python3.6/site-packages/soundfile.py”, line 257, in read
subtype, endian, format, closefd) as f:
File “/home/ubuntu/.local/lib/python3.6/site-packages/soundfile.py”, line 629, in init
self._file = self._open(file, mode_int, closefd)
File “/home/ubuntu/.local/lib/python3.6/site-packages/soundfile.py”, line 1184, in _open
"Error opening {0!r}: “.format(self.name))
File “/home/ubuntu/.local/lib/python3.6/site-packages/soundfile.py”, line 1357, in _error_check
raise RuntimeError(prefix + _ffi.string(err_str).decode(‘utf-8’, ‘replace’))
RuntimeError: Error opening '/home/ubuntu/drive_a/tanmay/tts/s3_tts_trainingWaveFiles/mono/normalised/chunks/Dataset/wavs/et thee hence, wretch!”.wav’: System error.
since the error says 'RuntimeError: Error opening ‘/home/ubuntu/drive_a/tanmay/tts/s3_tts_trainingWaveFiles/mono/normalised/chunks/Dataset/wavs/et thee hence, wretch!”.wav’: System error.’
I have no file named ‘et thee hence, wretch!".wav’
since at the end it says ‘system error’, is it related to device or machine on which training is being performed?
As @othiele already mentioned i’d highly recommend using guid-generated filenames (or something machine readable). A filename like:
et thee hence, wretch!”.wav
isn’t good at all - independend from machine learing/tts.
1 Like