--continue__path on vocoder training seems to bump up all loss values

TTS (Text-to-Speech)
#1

Hello,

I’m training Multiband MelGAN vocoder by following instructions given here: https://github.com/mozilla/TTS/tree/master/TTS/vocoder

Things went well until around 475K steps and the output audio was reasonably good and getting better. On my machine, it took about 14 days for this. Today morning I checked and the training process had stopped. So, I tried to continue training with the --continue_path option.

I was expecting the training to pick up from where it left off, with the loss values being the same and the quality of audio output being what it was at 475K. However, there is a big jump in loss and the audio quality has gone down… almost like starting from scratch :-(.

Here is how tensorboard looks. Notice the jump at around 475K when I continued training.

vocoder-continue-problem
vocoder-continue-problem1920×1080 221 KB

Any pointers to what I might be doing wrong? How to continue training properly for vocoder? Thanks.

#2

Hi! What is the current learning rate when you restore using --continue?

#3

Sounds strange, we continued a vocoder training and didn’t have any problems. Maybe @georroussos’s answer and we always used the dev branch. Don’t know whether that could be a source for trouble?

#4

I remember that at some point when I was continuing a previous run, the LR was reset to the initial one and I had to dig in the code and fix it. It was 3 months ago though, so it might have been fixed :slight_smile: I thought that may be the reason because otherwise I cannot really think of anything.

#5

Hi @georroussos,

When I continue, I’m seeing:

lrG = 0.0001
lrD = 0.0001

Is this ok? Or am I running into the same problem you encountered?
Thanks.

#6

@georroussos,

Looks like LR is reset (0.0001 is the initial value in config.json). Any idea what code fix you did to get over this? Thanks.

#7

Thank you @georroussos and @othiele for pointing me in the right direction.
In TTS/bin/train_vocoder.py, function main(), I fixed the part where LR was getting reset even though it was continuing.

Now, it looks like training is continuing well. Thanks again!