Whilst training the integrated Vocoder from dev, I ran into a problem v late last night where the computer eventually ran out of memory and the task was killed - I’ve got 32Gb RAM on the machine so it was a bit of a surprise!
It had reached a global step of just past 283062. There was no error message, it simply stopped with “Killed” (which came from the OS I believe). RAM usage remained high and there were no other significant tasks running.
I’ll need to investigate more this evening, but I did see this message in the logs, from early on:
/home/neil/main/Projects/TTSJul2020/TTS20Jul2020dev/TTS/tts/utils/visual.py:37: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). > fig = plt.figure(figsize=fig_size)
It appeared right after one of the early evaluations. It might be a red-herring, but given that it mentions memory, perhaps it is connected.
I saw this issue but that doesn’t appear to be with the vocoder (output suggests regular TTS model training run)
Q. Has anyone else run into issues with memory usage whilst training the vocoder?