Hello. Im trying to train Russian model on Deepspeech 0.6.1 and face with really high loss.
I get corpus from here , its about 72 hours of speech. Then i convert mp3 to wav with import_cv2.py, in result i have:
train.csv - 9743 wavs
dev.csv - 5714 wavs
test.csv - 5892 wavs
Then i start training by this arguments:
python -u DeepSpeech.py
–train_files /media/djo/DS/work/rumodel/clips/train.csv
–dev_files /media/djo/DS/work/rumodel/clips/dev.csv
–test_files /media/djo/DS/work/rumodel/clips/test.csv
–n_hidden 375
–epochs 10
–learning_rate 0.00095
–export_dir data/export
–checkpoint_dir data/cp_ru_full
In result i have:
I Early stop triggered as (for last 4 steps) validation loss: 131.891235 with standard deviation: 0.253853 and mean: 127.696702
Why so huge loss?
othiele
(Olaf Thiele)
March 19, 2020, 8:05pm
2
Looks like your wavs are quite long, what is the mean?
Use a 70,20,10 split for train,dev test
Batch size >= 8
usecudnn_rnn True
How good is your material? for 70 hours this could be ok
Now i train that corpus:
train.csv - 9743 wavs
dev.csv - 2000 wavs
test.csv - 1000 wavs
New parameters:
python -u DeepSpeech.py
–train_files /media/djo/DS/work/rumodel/clips/train.csv
–dev_files /media/djo/DS/work/rumodel/clips/dev2k.csv
–test_files /media/djo/DS/work/rumodel/clips/test1k.csv
–n_hidden 375
–epochs 10
–learning_rate 0.00095
–export_dir data/export
–checkpoint_dir data/cp_rumodel2
–train_batch_size 8
–dev_batch_size 8
–test_batch_size 8
and get result:
I Saved new best validating model with loss 95.456321 to: data/cp_rumodel2/best_dev-12170
othiele:
what is the mean
i dont know what is the mean, i just copy paste log from terminal.
othiele:
usecudnn_rnn True
im using cpu, so i cant
othiele
(Olaf Thiele)
March 20, 2020, 9:12am
4
Ok, do you have a question? This result could be just right, but without more information it is hard to tell. Russian has a lot of cases and many pronouns, you might just need more audio input.
Yes, i have a question: how to make loss lower than 20%?
Also im not specify language model, can it affect on training process?
A little bit more logs:
Epoch 9 | Training | Elapsed Time: 0:19:07 | Steps: 1217 | Loss: 53.498155
Epoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | DatasEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 24.001438 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 2 | Loss: 26.139920 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 29.984301 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 4 | Loss: 33.154804 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 5 | Loss: 35.215176 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 6 | Loss: 38.371660 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 7 | Loss: 38.652054 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 8 | Loss: 38.825130 | DataEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 9 | Loss: 39.280251 | DataEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 10 | Loss: 40.134796 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 11 | Loss: 42.435915 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 12 | Loss: 42.122548 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 13 | Loss: 42.439535 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 14 | Loss: 43.937102 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 15 | Loss: 44.491772 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 16 | Loss: 44.213127 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 17 | Loss: 44.542344 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 18 | Loss: 44.711684 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 19 | Loss: 45.564220 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 20 | Loss: 45.202640 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 21 | Loss: 45.491929 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 22 | Loss: 46.109399 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 23 | Loss: 46.741205 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 24 | Loss: 46.471454 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 25 | Loss: 45.891328 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 26 | Loss: 46.225850 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 27 | Loss: 47.317082 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 28 | Loss: 47.156155 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 29 | Loss: 47.622437 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 30 | Loss: 47.424197 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 31 | Loss: 48.216693 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 32 | Loss: 48.476222 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 33 | Loss: 48.676489 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 34 | Loss: 48.684854 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 35 | Loss: 48.969126 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 36 | Loss: 49.080404 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 37 | Loss: 49.778108 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 38 | Loss: 49.909148 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 39 | Loss: 50.261603 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 40 | Loss: 50.344049 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 41 | Loss: 50.647617 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 42 | Loss: 50.924914 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 43 | Loss: 51.235628 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 44 | Loss: 52.299264 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 45 | Loss: 52.982479 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 46 | Loss: 53.631909 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 47 | Loss: 54.205429 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 48 | Loss: 54.306188 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 49 | Loss: 54.451473 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 50 | Loss: 54.526620 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 51 | Loss: 55.211812 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 52 | Loss: 55.197487 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 53 | Loss: 55.696617 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 54 | Loss: 56.570952 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 55 | Loss: 57.052983 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 56 | Loss: 57.392976 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 57 | Loss: 57.701493 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 58 | Loss: 57.884207 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 59 | Loss: 58.080325 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 60 | Loss: 58.293715 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 61 | Loss: 58.931091 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 62 | Loss: 59.011049 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 63 | Loss: 59.257370 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 64 | Loss: 60.076057 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 65 | Loss: 60.483996 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 66 | Loss: 60.755636 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 67 | Loss: 61.100232 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 68 | Loss: 61.485705 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 69 | Loss: 62.023526 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 70 | Loss: 62.229157 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 71 | Loss: 62.479189 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 72 | Loss: 63.117895 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 73 | Loss: 63.364378 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 74 | Loss: 63.593335 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 75 | Loss: 64.020090 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 76 | Loss: 64.333973 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 77 | Loss: 64.479004 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 78 | Loss: 64.752624 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 79 | Loss: 64.965228 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 80 | Loss: 65.150344 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 81 | Loss: 65.561469 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 82 | Loss: 65.624622 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 83 | Loss: 65.664437 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 84 | Loss: 65.883643 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 85 | Loss: 66.108031 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 86 | Loss: 66.310900 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 87 | Loss: 66.288677 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 88 | Loss: 66.511101 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 89 | Loss: 66.445487 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 90 | Loss: 66.417586 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 91 | Loss: 66.486743 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 92 | Loss: 66.654646 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 93 | Loss: 66.910886 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 94 | Loss: 67.146643 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 95 | Loss: 67.259888 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 96 | Loss: 67.745446 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 97 | Loss: 68.105616 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 98 | Loss: 68.725401 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 99 | Loss: 69.229914 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 100 | Loss: 69.518019 | DaEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 101 | Loss: 69.933013 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 102 | Loss: 70.141830 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 103 | Loss: 70.237128 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 104 | Loss: 70.324611 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 105 | Loss: 70.590663 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 106 | Loss: 70.764109 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 107 | Loss: 70.891378 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 108 | Loss: 71.559604 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 109 | Loss: 71.869520 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 110 | Loss: 72.144179 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 111 | Loss: 72.432612 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 112 | Loss: 72.789465 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 113 | Loss: 72.855290 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 114 | Loss: 72.888665 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 115 | Loss: 73.414483 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 116 | Loss: 73.691735 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 117 | Loss: 73.921992 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 118 | Loss: 73.931742 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 119 | Loss: 74.343568 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 120 | Loss: 74.704309 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 121 | Loss: 74.908236 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 122 | Loss: 74.942185 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 123 | Loss: 75.049731 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 124 | Loss: 75.161850 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 125 | Loss: 75.568701 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 126 | Loss: 75.811799 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 127 | Loss: 75.918382 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 128 | Loss: 76.199509 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 129 | Loss: 76.237561 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 130 | Loss: 76.679952 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 131 | Loss: 76.941891 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 132 | Loss: 77.120592 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 133 | Loss: 77.459202 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 134 | Loss: 77.563689 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 135 | Loss: 77.801109 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 136 | Loss: 77.957253 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 137 | Loss: 78.096835 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 138 | Loss: 78.330076 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 139 | Loss: 78.616373 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 140 | Loss: 78.939457 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 141 | Loss: 78.892939 | DaEpoch 9 | Validation | Elapsed Time: 0:00:29 | Steps: 142 | Loss: 79.041815 | DaEpoch 9 | Validation | Elapsed Time: 0:00:29 | Steps: 143 | Loss: 79.246707 | DaEpoch 9 | Validation | Elapsed Time: 0:00:29 | Steps: 144 | Loss: 79.587958 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 145 | Loss: 79.864611 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 146 | Loss: 80.264299 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 147 | Loss: 80.545905 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 148 | Loss: 80.653374 | DaEpoch 9 | Validation | Elapsed Time: 0:00:31 | Steps: 149 | Loss: 80.810095 | DaEpoch 9 | Validation | Elapsed Time: 0:00:31 | Steps: 150 | Loss: 80.830239 | DaEpoch 9 | Validation | Elapsed Time: 0:00:31 | Steps: 151 | Loss: 81.297724 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 152 | Loss: 81.435584 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 153 | Loss: 81.806836 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 154 | Loss: 81.794360 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 155 | Loss: 81.953380 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 156 | Loss: 82.078743 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 157 | Loss: 82.307890 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 158 | Loss: 82.387073 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 159 | Loss: 82.565338 | DaEpoch 9 | Validation | Elapsed Time: 0:00:34 | Steps: 160 | Loss: 82.817048 | DaEpoch 9 | Validation | Elapsed Time: 0:00:34 | Steps: 161 | Loss: 83.044786 | DaEpoch 9 | Validation | Elapsed Time: 0:00:34 | Steps: 162 | Loss: 83.145943 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 163 | Loss: 83.576915 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 164 | Loss: 83.759789 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 165 | Loss: 83.874057 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 166 | Loss: 83.972620 | DaEpoch 9 | Validation | Elapsed Time: 0:00:36 | Steps: 167 | Loss: 83.993627 | DaEpoch 9 | Validation | Elapsed Time: 0:00:36 | Steps: 168 | Loss: 84.208651 | DaEpoch 9 | Validation | Elapsed Time: 0:00:36 | Steps: 169 | Loss: 84.412908 | DaEpoch 9 | Validation | Elapsed Time: 0:00:37 | Steps: 170 | Loss: 84.589244 | DaEpoch 9 | Validation | Elapsed Time: 0:00:37 | Steps: 171 | Loss: 84.805894 | DaEpoch 9 | Validation | Elapsed Time: 0:00:37 | Steps: 172 | Loss: 85.011773 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 173 | Loss: 85.017073 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 174 | Loss: 85.096596 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 175 | Loss: 85.170380 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 176 | Loss: 85.357464 | DaEpoch 9 | Validation | Elapsed Time: 0:00:39 | Steps: 177 | Loss: 85.414403 | DaEpoch 9 | Validation | Elapsed Time: 0:00:39 | Steps: 178 | Loss: 85.501928 | DaEpoch 9 | Validation | Elapsed Time: 0:00:39 | Steps: 179 | Loss: 85.570174 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 180 | Loss: 85.655280 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 181 | Loss: 85.704665 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 182 | Loss: 85.872735 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 183 | Loss: 86.054915 | DaEpoch 9 | Validation | Elapsed Time: 0:00:41 | Steps: 184 | Loss: 86.274824 | DaEpoch 9 | Validation | Elapsed Time: 0:00:41 | Steps: 185 | Loss: 86.384777 | DaEpoch 9 | Validation | Elapsed Time: 0:00:41 | Steps: 186 | Loss: 86.673783 | DaEpoch 9 | Validation | Elapsed Time: 0:00:42 | Steps: 187 | Loss: 86.962934 | DaEpoch 9 | Validation | Elapsed Time: 0:00:42 | Steps: 188 | Loss: 87.025896 | DaEpoch 9 | Validation | Elapsed Time: 0:00:42 | Steps: 189 | Loss: 87.168968 | DaEpoch 9 | Validation | Elapsed Time: 0:00:43 | Steps: 190 | Loss: 87.619509 | DaEpoch 9 | Validation | Elapsed Time: 0:00:43 | Steps: 191 | Loss: 87.850966 | DaEpoch 9 | Validation | Elapsed Time: 0:00:43 | Steps: 192 | Loss: 87.980060 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 193 | Loss: 88.184805 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 194 | Loss: 88.459959 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 195 | Loss: 88.638910 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 196 | Loss: 88.730926 | DaEpoch 9 | Validation | Elapsed Time: 0:00:45 | Steps: 197 | Loss: 88.997776 | DaEpoch 9 | Validation | Elapsed Time: 0:00:45 | Steps: 198 | Loss: 89.163274 | DaEpoch 9 | Validation | Elapsed Time: 0:00:45 | Steps: 199 | Loss: 89.264240 | DaEpoch 9 | Validation | Elapsed Time: 0:00:46 | Steps: 200 | Loss: 89.444111 | DaEpoch 9 | Validation | Elapsed Time: 0:00:46 | Steps: 201 | Loss: 89.557194 | DaEpoch 9 | Validation | Elapsed Time: 0:00:46 | Steps: 202 | Loss: 89.654985 | DaEpoch 9 | Validation | Elapsed Time: 0:00:47 | Steps: 203 | Loss: 89.816107 | DaEpoch 9 | Validation | Elapsed Time: 0:00:47 | Steps: 204 | Loss: 89.891788 | DaEpoch 9 | Validation | Elapsed Time: 0:00:47 | Steps: 205 | Loss: 90.064587 | DaEpoch 9 | Validation | Elapsed Time: 0:00:48 | Steps: 206 | Loss: 90.122345 | DaEpoch 9 | Validation | Elapsed Time: 0:00:48 | Steps: 207 | Loss: 90.127684 | DaEpoch 9 | Validation | Elapsed Time: 0:00:48 | Steps: 208 | Loss: 90.261094 | DaEpoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 209 | Loss: 90.598459 | DaEpoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 210 | Loss: 90.717597 | DaEpoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 211 | Loss: 90.853893 | DaEpoch 9 | Validation | Elapsed Time: 0:00:50 | Steps: 212 | Loss: 90.980856 | DaEpoch 9 | Validation | Elapsed Time: 0:00:50 | Steps: 213 | Loss: 91.058446 | DaEpoch 9 | Validation | Elapsed Time: 0:00:50 | Steps: 214 | Loss: 91.125155 | DaEpoch 9 | Validation | Elapsed Time: 0:00:51 | Steps: 215 | Loss: 91.356190 | DaEpoch 9 | Validation | Elapsed Time: 0:00:51 | Steps: 216 | Loss: 91.687895 | DaEpoch 9 | Validation | Elapsed Time: 0:00:51 | Steps: 217 | Loss: 91.754220 | DaEpoch 9 | Validation | Elapsed Time: 0:00:52 | Steps: 218 | Loss: 91.879785 | DaEpoch 9 | Validation | Elapsed Time: 0:00:52 | Steps: 219 | Loss: 92.022793 | DaEpoch 9 | Validation | Elapsed Time: 0:00:52 | Steps: 220 | Loss: 92.177591 | DaEpoch 9 | Validation | Elapsed Time: 0:00:53 | Steps: 221 | Loss: 92.299894 | DaEpoch 9 | Validation | Elapsed Time: 0:00:53 | Steps: 222 | Loss: 92.559568 | DaEpoch 9 | Validation | Elapsed Time: 0:00:53 | Steps: 223 | Loss: 92.566857 | DaEpoch 9 | Validation | Elapsed Time: 0:00:54 | Steps: 224 | Loss: 92.716912 | DaEpoch 9 | Validation | Elapsed Time: 0:00:54 | Steps: 225 | Loss: 92.667962 | DaEpoch 9 | Validation | Elapsed Time: 0:00:55 | Steps: 226 | Loss: 92.845427 | DaEpoch 9 | Validation | Elapsed Time: 0:00:55 | Steps: 227 | Loss: 93.016710 | DaEpoch 9 | Validation | Elapsed Time: 0:00:55 | Steps: 228 | Loss: 93.274283 | DaEpoch 9 | Validation | Elapsed Time: 0:00:56 | Steps: 229 | Loss: 93.460927 | DaEpoch 9 | Validation | Elapsed Time: 0:00:56 | Steps: 230 | Loss: 93.443831 | DaEpoch 9 | Validation | Elapsed Time: 0:00:56 | Steps: 231 | Loss: 93.556214 | DaEpoch 9 | Validation | Elapsed Time: 0:00:57 | Steps: 232 | Loss: 93.674318 | DaEpoch 9 | Validation | Elapsed Time: 0:00:57 | Steps: 233 | Loss: 93.822391 | DaEpoch 9 | Validation | Elapsed Time: 0:00:57 | Steps: 234 | Loss: 93.986279 | DaEpoch 9 | Validation | Elapsed Time: 0:00:58 | Steps: 235 | Loss: 94.218827 | DaEpoch 9 | Validation | Elapsed Time: 0:00:58 | Steps: 236 | Loss: 94.266340 | DaEpoch 9 | Validation | Elapsed Time: 0:00:59 | Steps: 237 | Loss: 94.336616 | DaEpoch 9 | Validation | Elapsed Time: 0:00:59 | Steps: 238 | Loss: 94.387358 | DaEpoch 9 | Validation | Elapsed Time: 0:00:59 | Steps: 239 | Loss: 94.437320 | DaEpoch 9 | Validation | Elapsed Time: 0:01:00 | Steps: 240 | Loss: 94.576774 | DaEpoch 9 | Validation | Elapsed Time: 0:01:00 | Steps: 241 | Loss: 94.631176 | DaEpoch 9 | Validation | Elapsed Time: 0:01:01 | Steps: 242 | Loss: 94.727343 | DaEpoch 9 | Validation | Elapsed Time: 0:01:01 | Steps: 243 | Loss: 94.832889 | DaEpoch 9 | Validation | Elapsed Time: 0:01:01 | Steps: 244 | Loss: 95.004763 | DaEpoch 9 | Validation | Elapsed Time: 0:01:02 | Steps: 245 | Loss: 95.130512 | DaEpoch 9 | Validation | Elapsed Time: 0:01:02 | Steps: 246 | Loss: 95.136806 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 247 | Loss: 95.219078 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 248 | Loss: 95.351087 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 249 | Loss: 95.456321 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 249 | Loss: 95.456321 | Dataset: /media/djo/DS/work/rumodel/clips/dev2k.csv
I Saved new best validating model with loss 95.456321 to: data/cp_rumodel2/best_dev-12170
I FINISHED optimization in 3:23:23.062624
[scorer.cpp:77] FATAL: "(access(filename, 4)) == (0)" check failed. Invalid language model path
othiele
(Olaf Thiele)
March 20, 2020, 9:28pm
6
You need more and better data. I guess that the data is not as good yet and you will probably need 250 hours more. Just contribute to common voice and you’ll eventually get there, sorry, there is not much else to do but gather more data.
my main target is make small model like that guy TUTORIAL : How I trained a specific french model to control my robot He make working model with just 1000 sentences, and i also want create small model for my robot. In result i want to recognize just 100-200 commands. Can you give me advise to achieve it for Russian model?
nshmyrev
(Nshmyrev)
March 22, 2020, 2:32pm
8
You can actually get a big Russian dataset, the pretrained models and scripts here:
Russian Speech Recognition system based on Mozilla's DeepSpeech TensorFlow implementation.
2 Likes
Hi There, I am trying to train deepspeech model with russian language (Common voice + voxforge). So should I edit something else besides the alphabet.txt file? Using colab, I run the following script:
!python3 DeepSpeech.py
–train_files “…/train.csv” \ (about 100 hours)
–dev_files “…/dev.csv”
–test_files “…test.csv”
–train_batch_size 16
–dev_batch_size 16
–test_batch_size 16
–learning_rate 0.0001
–epoch 15
–dropout_rate 0.3
–export_dir …/exports \
After all I have pretty low CTC loss (about 30-40) during training, but then test phrases is recognized incorrectly, like:
-src:“женя достала аккордеон и перекинула ремень через плечо”
-res:“оноокдвнур”
What did I do wrong?