Most of my recent training run with the Vocoder was done with --continue_path but last night I switched to try fine-tuning with --restore_path and varying the LR. I was surprised as the LR seems to be following the original settings.
Q. What’s the expected behaviour with --restore_path?
I’m using dev (72a6ac5). The lrD and lrG values it shows at the bottom of this quote are not what I’ve put in my config file, attached (which has 1e-4 now, although with the initial run one was 0.0001 and one was 1e-5, which was what had worked for me with the original external MelGAN). This seems to continue too (ie it’s not just the first few evaluations that it’s like that).
Maybe it’s some effect from the LR optimiser that I’m misunderstanding or not using correctly? I see below it mentions several “Restoring…” lines which I believe indicate that it’s using the original optimiser settings rather than reading from the config file even though I’m trying to fine-tune now.
CUDA_VISIBLE_DEVICES=‘0’ python bin/train_vocoder.py --restore_path …/…/models/neil17/multiband-melgan-July-27-2020_02+28PM-72a6ac5/checkpoint_1000000.pth.tar --config_path …/…/models/neil17/multiband-melgan-July-27-2020_02+28PM-72a6ac5/config.json |& tee …/…/models/neil17/training_vocoder_n17_D11_02aug2020.log
Using CUDA: True
Number of GPUs: 1
Git Hash: 72a6ac5
Experiment folder: /home/neil/main/Projects/TTSJul2020/models/neil17/multiband-melgan-August-02-2020_07+38PM-72a6ac5
Loading wavs from: /home/neil/data/Projects/NeilTTS/neil16/wavs/
Setting up Audio Processor…
| > sample_rate:22050
| > num_mels:80
| > min_level_db:-10
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:0
| > fft_size:1024
| > power:None
| > preemphasis:0.99
| > griffin_lim_iters:None
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:40.0
| > mel_fmax:8000.0
| > spec_gain:1.0
| > stft_pad_mode:reflect
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:60
| > do_sound_norm:False
| > stats_path:None
| > hop_length:256
| > win_length:1024
Generator Model: multiband_melgan_generator
Discriminator Model: melgan_multiscale_discriminator
Restoring Generator Model…
Restoring Generator Optimizer…
Restoring Discriminator Model…
Restoring Discriminator Optimizer…
Restoring Generator LR Scheduler…
Restoring Discriminator LR Scheduler…
Model restored from step 1000000
Generator has 2539448 parameters
Discriminator has 4354998 parametersEPOCH: 0/10000
TRAINING (2020-08-02 19:38:21)
–> STEP: 199/493 – GLOBAL_STEP: 1000200
| > G_stft_loss_mg: 0.77097 (0.77146)
| > G_stft_loss_sc: 0.48925 (0.51256)
| > G_subband_stft_loss_mg: 0.64009 (0.64357)
| > G_subband_stft_loss_sc: 0.49436 (0.52802)
| > G_mse_fake_loss: 0.51116 (0.42568)
| > G_loss: 2.47523 (2.29200)
| > G_gen_loss: 1.19733 (1.22781)
| > G_adv_loss: 1.27790 (1.06419)
| > D_mse_gan_loss: 0.36095 (0.38322)
| > D_mse_gan_real_loss: 0.15346 (0.13951)
| > D_mse_gan_fake_loss: 0.11444 (0.12729)
| > D_loss: 0.36095 (0.38322)
| > step_time: 0.68
| > loader_time: 0.00
| > lrG: 0.0001
| > lrD: 0.0001
config.zip (2.8 KB)