The quality of the model is constantly changing

I am trying to train my model but after 20k steps qulity of model start jumping. I tried to change every setting but it got even worst. Is there any way to fix this?

My best attempt:
Config: https://pastebin.com/Yw6yAAiZ
Tensorboard: https://i.imgur.com/Tj5Q5AD.png
Test Audio (30k, 90k, 100k, 130k): https://drive.google.com/drive/folders/12mrxWlsy_t51kHiPfKqIf70qmw4t01fH

Thank you in advance for your help.