Hello, I’m trying to transfer learning for Korean.
I have 2 questions for Korean transfer learning.
My setting
DeepSpeech version : 0.9.0-alpha.3
GPU : 7G
training data : 500h+
validation data : almost 9h
test data : almost 9h
training parameter :
python3 DeepSpeech.py
–drop_source_layers 1
–alphabet_config_path deepspeech_alphabet.txt
–load_checkpoint_dir deepspeech-0.8.0-checkpoint
–save_checkpoint_dir my_checkpoint_dir
–export_dir scorer_model
–train_files corpora/ko/clips/train.csv
–dev_files corpora/ko/clips/dev.csv
–test_files corpora/ko/clips/test.csv
–train_cudnn
–learning_rate 0.00001
–scorer my_kor_scorer.scorer
–summary_dir summary_dir &> log/0914_transfer_learning.log &
alphabet size: 2200
scorer :
scorer is created from sentences in train.csv, dev.csv, test.csv
Question 1.
My training loss and validation loss reduces in 0~6 epoch,
after that validation loss is not changed from previous value but training loss is still reduced.
Epoch 0 | Training | Elapsed Time: 16:41:11 | Steps: 481008 | Loss: 28.460519
Epoch 0 | Validation | Elapsed Time: 0:01:02 | Steps: 1070 | Loss: 22.371003 | Dataset: corpora/ko/clips/dev.csv
Epoch 1 | Training | Elapsed Time: 16:33:00 | Steps: 481008 | Loss: 19.757104
Epoch 1 | Validation | Elapsed Time: 0:01:02 | Steps: 1070 | Loss: 19.216509 | Dataset: corpora/ko/clips/dev.csv
Epoch 2 | Training | Elapsed Time: 16:32:23 | Steps: 481008 | Loss: 17.215991
Epoch 2 | Validation | Elapsed Time: 0:00:58 | Steps: 1070 | Loss: 18.027218 | Dataset: corpora/ko/clips/dev.csv
Epoch 3 | Training | Elapsed Time: 16:33:35 | Steps: 481008 | Loss: 15.617158
Epoch 3 | Validation | Elapsed Time: 0:01:01 | Steps: 1070 | Loss: 17.366926 | Dataset: corpora/ko/clips/dev.csv
Epoch 6 | Training | Elapsed Time: 1 day, 0:33:51 | Steps: 481008 | Loss: 12.790936
Epoch 6 | Validation | Elapsed Time: 0:01:34 | Steps: 1070 | Loss: 16.471936 | Dataset: corpora/ko/clips/dev.csv
After Epoch 6, validation loss is not changed so much.
Epoch 7 | Training | Elapsed Time: 1 day, 1:59:56 | Steps: 481008 | Loss: 12.143217
Epoch 7 | Validation | Elapsed Time: 0:01:35 | Steps: 1070 | Loss: 16.556080 | Dataset: corpora/ko/clips/dev.csv
Epoch 9 | Training | Elapsed Time: 16:33:11 | Steps: 481008 | Loss: 11.009611
Epoch 9 | Validation | Elapsed Time: 0:01:02 | Steps: 1070 | Loss: 16.671552 | Dataset: corpora/ko/clips/dev.csv
Epoch 11 | Training | Elapsed Time: 16:32:19 | Steps: 481008 | Loss: 10.096103
Epoch 11 | Validation | Elapsed Time: 0:01:11 | Steps: 1070 | Loss: 16.909459 | Dataset: corpora/ko/clips/dev.csv
Is it means overfitting? If so, what should I do?
Question 2.
I create output.pb from checkpoint (from Epoch 6) with my scorer (from csv files) and my alphabet (size 2200)
and tried to inference with scorer, but it’s not so correct and it takes 9s.
deepspeech \
--model 0914_1672loss_model/output_graph.pb \
--scorer my_kor_scorer.scorer \
--audio my_data/blob.wav
둘 제자리다
Inference took 9.047s for 5.952s audio file.
Inference without scorer is much correct but it takes time too much.
Why inference takes too much time?
deepspeech \
--model 0914_1672loss_model/output_graph.pb \
--audio my_data/blob.wav
둘 제 회이로 작성부탁내리이다
Inference took 2231.380s for 5.952s audio file.
English inference with scorer or not, takes less than a second.
deepspeech \
--model deepspeech-0.8.0-models.pbmm \
--scorer deepspeech-0.8.0-models.scorer \
--audio english_voice.wav
english test
Inference took 0.648s for 1.272s audio file.
##########
deepspeech
–model deepspeech-0.8.0-models.pbmm
–audio english_voice.wav
english test
Inference took 0.792s for 1.272s audio file.
What makes the different? alphabet size?
I want to inference rapidly without scorer or inference with scorer correctly.
Please give me some advice.