Hi, I have been training some models with not too good results for now, WER in test ~0.45.
I launch it doing the following:
./DeepSpeech.py --train_files ../spanish_data/train2.csv --dev_files ../val2.csv --test_files ../test2.csv --lm_binary_path models/lm.binary --lm_trie_path models/trie --checkpoint_dir checkpoint --alphabet_config_path models/alphabet.txt --epoch -1 --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1 --validation_step 1 --summary_dir tensorboar_summaries/ --summary_secs 60 --report_count 100
My audios are mono, 16Khz, 16 bits and .wav
I am training a Spanish model from scratch.
At first, training seems to go well
The loss goes down both in training and validation.
However, after a few epochs loss starts to go up. That is what I would have expected from the val loss, the problem is that it also happens in the training loss.
I deactivated the early stop on purpose to try to overfit the NN, I expected to see the training loss go down or at least being stuck in a value, but as the validation loss goes up so does the training loss.
If I keep training for more epochs loss go to inf (train, val and test). Has this any sense? I thought it might be a problem with the learning rate, but in that case, it should not have started learning.
I do not really understand what is going on.
Thanks a lot !!