Training of Epoch x - loss: inf. again decrease LR, stil same issue repeating

DeepSpeech
#1

Training of Epoch x - loss: inf. again decrease LR, stil same issue repeating. can’t guess what is happened?
help me how to rectify this? thank you sir

#2

We need more context, but generally speaking, it’s because of inconsistent hyperparameters and the network is not able to learn properly.

#3

then if i decreasing my learning rate like 0.0001 to 0.00005 or low and increase epoch as well as, it will learn properly or not. i have 25000 audiofiles like 40 hours of audio files sir. thank you for your reply sir,

#4

@muruganrajenthirean Why don’t you use the documented hyper-parameters we have ? 40 hours, are you fine-tuning on top of English model ?

#5

ofcourse sir i was followed same documented hyper-parameters value. and i was fine tuned on DeepSpeechv0.2.0 pretrained model with my data.

sir if i have to customize deepspeech structure (change GD, Loss fun or other weight initialization st.) is it resolve or this is inconvenient.

thank you sir.

#6

If you change that, then you won’t be able to fine-tune.

#7

Ok, how many epochs have you tested ?

#8

`

python3 DeepSpeech.py --n_hidden 2048 --initialize_from_frozen_model ownaudiomodel_e10/output_graph.pb --checkpoint_dir ckpt_comboownaudiofile_aug_e10 --epoch 3 --train_files trainaudiofile_aug.csv --dev_files devaudiofile_aug.csv --test_files testaudiofile_aug.csv --learning_rate 0.0001 --decoder_library_path new_native_client/libctc_decoder_with_kenlm.so --alphabet_config_path data/alphabet.txt --lm_binary_path data/lm/lm.binary --lm_trie_path data/lm/trie --train_batch_size 3 --dev_batch_size 2 --test_batch_size 2 --early_stop True --earlystop_nsteps 6 --estop_mean_thresh 0.1 --estop_std_thresh 0.1 --dropout_rate 0.12 --export_dir ownaudiomodel_e10_with_aug/

`

#9

sir Epoch 3
LR -0.0001
batch_size 3-2-2

thank you sir

#10

You should try to wait a bit more, it’s possible it will get better after. In my case, with ~100h, only the first two epochs shows this behavior, then it does converge properly

#11

then if i will fine tune Epoch 5 or 10 ,it may working properly sir.