Fixed the wav-files and sorted out the data-cleaner/phoneme issue with help of @erogol. Fresh training session just started - I will report back when we see first results (or problems)…
5 Likes
Short update for statistic fans:
- Phrases recorded: 18.036
- Recorded audio length: 17 hours
- Average sentence length: 47 chars
- Chars per second (avg): 13,5
- Sentences with question mark: 1.893
- Sentences with exclamation mark: 1.462
The recordings are slower than my every-day speech, but therefor they are clear and without any characters swallowed (hopefully).
3.700 phrases remain for recording. After that i will finish my recordings and upload/update the complete dataset for community use.
2 Likes
We reached Epoch 340 /step 112.000 - time for an update.
Diagrams:
You can clearly see where the gradual training r=3 kicked in at at 50k steps. Since that the loss values slowly increase again.
The Eval alignment was a (more or less) straight diagonal in between now again has that gap in the upper right.
The Train alignment looks better though:
Here are some audio examples:Audio-samples-step112k.zip (806,4 KB)
Overall it starts sounding good, there are some problems with the intonation of the german Umlaut vowels, probably we need to enhance the dataset with more examples for that.
Any comments, any reasons to worry?
3 Likes
My German is v rusty but it sounds like it’s coming along well. Would be interesting to hear it after the r=2 stage which isn’t that far (assuming you’ve left the gradual training stages as per those in config.json)
2 Likes
When my understanding of config.json is correct we’ll be switching to r=2 on step 130k.
@dkreutz Should i modify the sentence order in corpus to record all sentences containing german umlauts next (so we have all umlaut sentences on next (appended) training run?
BTW I don’t want to divert you if this is something that was already fixed but I see that problems with handling umlauts were mentioned here:
@mrthorstenm I see you’d commented on that issue but not sure how far up it you’d read, plus it was a little while back.
Thanks @nmstoker for the link.
Looking forward to take a closer look.
Currently i’m trying to start a server with the “best_model.tar” and config.json provided by @dkreutz for our speaker-embedded dataset.
Based on https://github.com/mozilla/TTS/tree/master/server i was able to create and install a TTS-0.0.1+df42a4a-py3-none-any.whl in ./dist directory.
But running python -m TTS.server.server failed with following output:
(venv) thorsten@thorsten-desktop:~/___dev/tts/mozilla/branch-dev/TTS$ python -m TTS.server.server
> Loading TTS model ...
| > model config: /tmp/venv/lib/python3.6/site-packages/TTS/server/model/config.json
| > checkpoint file: /tmp/venv/lib/python3.6/site-packages/TTS/server/model/checkpoint.pth.tar
> Setting up Audio Processor...
| > sample_rate:22050
| > num_mels:80
| > min_level_db:-100
| > frame_shift_ms:12.5
| > frame_length_ms:50
| > ref_level_db:20
| > num_freq:1025
| > power:1.5
| > preemphasis:0.98
| > griffin_lim_iters:60
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:50.0
| > mel_fmax:8000.0
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:False
| > sound_norm:False
| > n_fft:2048
| > hop_length:275
| > win_length:1100
> Using model: Tacotron2
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/venv/lib/python3.6/site-packages/TTS/server/server.py", line 39, in <module>
synthesizer = Synthesizer(config)
File "/tmp/venv/lib/python3.6/site-packages/TTS/server/synthesizer.py", line 31, in __init__
self.config.use_cuda)
File "/tmp/venv/lib/python3.6/site-packages/TTS/server/synthesizer.py", line 58, in load_tts
self.tts_model.load_state_dict(cp['model'])
File "/tmp/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 830, in load_state_dict
self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Tacotron2:
Unexpected key(s) in state_dict: "speaker_embedding.weight".
(venv) thorsten@thorsten-desktop:~/___dev/tts/mozilla/branch-dev/TTS$
I tried running python -m TTS.server.server using following branches without success:
- master
- dev
- fix_server
- fix_db7f3d3
I found the following “issue” on that but i’m unsure if that addresses my problem.
What could be the problem on that?
@nmstoker Thanks for the pointer - sounds like I made the wrong decision by setting use_phonemes = true 
I will keep on training with the current setting for now and give it another try with use_phonemes = false or when @mrthorstenm has finished recording his dataset.
I could “fix” the not running tts-server issue quick’n dirty by commenting out lines 819 - 822 in file
/tmp/venv/lib/python3.6/site-packages/torch/nn/modules/module.py
After that the server starts without problems.
Your examples sound quite good (and up to date :)).
Actually you can already reduce the reduction rate r to 2 I guess since the model looks saturated in r=3
1 Like
“r” has already been reduced to “2” on reaching step 130k. Next reducation would be on step 290k to “1”.
These are the newest graphs and sample wavs @dkreutz provided me yesterday on step 167k.
Three audio samples:
Samples_167k.zip (358,2 KB)
1 Like
Today Epoch 500 / step 185k was reached and there is no more improvement visible/audible. As there are still the problems with Umlaut pronounciation I stopped training for now.
I want to start a new training with “use_phonemes=false”.
Any suggestion to modify some of the other parameters, e.g. “use_forward_attn” or “bidirectional_decoder”?
2 Likes
Here a sample from tts-server loaded with 185k model.
Vielen Dank an die Community von Mozilla und Meicroft.
Many thanks to the community of Mozilla and MyCroft and of course @dkreutz for amazing support on this.
5 Likes
For those interested - here is a link to the 185k model
(Note to visitors from a far and distant future - the download link might have disappeared)
3 Likes
What is your opinion on the following non-technical aspect of this topic.
I have decided to donate my voice to the community so that (OpenSource) projects have the opportunity to offer a qualitatively usable German voice in their project free of charge (and offline), regardless of the major cloud providers such as Amazon or Google.
I also see that several people record their voice for TTS but keep it private. Of course, I can understand this for data protection reasons. However, I ask myself the following questions:
- Am I careless to donate my voice (incl. dataset) to the general public?
- Do I lose control over part of my identity?
- Should I be worried that my voice will be used for illegal activities?
- Am I legally in a problematic situation by donating my voice?
- Why don’t more people contribute their voice?
That doesn’t change the fact that I want to donate my voice to give part back to the community.
However, I would be interested in your opinion on these points.
2 Likes
Hi Thorsten,
I am grateful that you recorded your voice, it takes up a lot of time and energy. Thanks.
Looking at advances in transfer learning, it will most likely be possible to clone a voice with just some samples in the future, so we all have to worry about losing our identity, same as with images 
I don’t think you should be legally worried, your CCO-1 license should suffice. In German courts it is the intention that counts.
In time we’ll have a couple of different speaker sets for German and it is wonderful that we will be able to find differences between them and learn through that experience.
Thanks for your efforts
Olaf
1 Like
Hi Oliver.
Thanks for your feedback.
You’re right on time and energy. I started recording six month ago and record since that on a regular basis. Probably when i’m finished with recording it’s enough to record 10 phrases to clone your voice .
Whenever we are finished (i with recording and @dkreutz with tacotron2 training) i’m excited on the results and if my model is used by anyone else.
1 Like
@dkreutz Wanted to test it on my Corona holidays, what config.json did you use to train the 185k model? Standard Tacotron?
Thanks in advance
Oh, and what revision did you train it on? I get the following error for the current master branch indicating that you were using a different one
RuntimeError: Error(s) in loading state_dict for Tacotron2:
Missing key(s) in state_dict: "decoder.prenet.layers.0.bn.weight", "decoder.prenet.layers.0.bn.bias", "decoder.prenet.layers.0.bn.running_mean", "decoder.prenet.layers.0.bn.running_var", "decoder.prenet.layers.1.bn.weight", "decoder.prenet.layers.1.bn.bias", "decoder.prenet.layers.1.bn.running_mean", "decoder.prenet.layers.1.bn.running_var".
Unexpected key(s) in state_dict: "speaker_embedding.weight".My config is based on standard Tacotron2 model with some modification for german phonemes and speaker embedding (see “crazy idea” above).
185k-config.zip (4,0 KB)
I am using a fork of TTS master from end of January. Some minor changes were applied (dataset/preprocess.py, utils/text/symbols.py) but I didn’t find time to commit them to my repo, yet.
2 Likes