High loss on Russian model

Evgenij_Gerasimov · March 19, 2020, 7:58pm

Hello. Im trying to train Russian model on Deepspeech 0.6.1 and face with really high loss.
I get corpus from here, its about 72 hours of speech. Then i convert mp3 to wav with import_cv2.py, in result i have:
train.csv - 9743 wavs
dev.csv - 5714 wavs
test.csv - 5892 wavs

Then i start training by this arguments:
python -u DeepSpeech.py
–train_files /media/djo/DS/work/rumodel/clips/train.csv
–dev_files /media/djo/DS/work/rumodel/clips/dev.csv
–test_files /media/djo/DS/work/rumodel/clips/test.csv
–n_hidden 375
–epochs 10
–learning_rate 0.00095
–export_dir data/export
–checkpoint_dir data/cp_ru_full

In result i have:
I Early stop triggered as (for last 4 steps) validation loss: 131.891235 with standard deviation: 0.253853 and mean: 127.696702

Why so huge loss?

othiele · March 19, 2020, 8:05pm

Looks like your wavs are quite long, what is the mean?

Use a 70,20,10 split for train,dev test

Batch size >= 8

usecudnn_rnn True

How good is your material? for 70 hours this could be ok

Evgenij_Gerasimov · March 20, 2020, 4:50am

Now i train that corpus:
train.csv - 9743 wavs
dev.csv - 2000 wavs
test.csv - 1000 wavs

New parameters:
python -u DeepSpeech.py
–train_files /media/djo/DS/work/rumodel/clips/train.csv
–dev_files /media/djo/DS/work/rumodel/clips/dev2k.csv
–test_files /media/djo/DS/work/rumodel/clips/test1k.csv
–n_hidden 375
–epochs 10
–learning_rate 0.00095
–export_dir data/export
–checkpoint_dir data/cp_rumodel2
–train_batch_size 8
–dev_batch_size 8
–test_batch_size 8

and get result:
I Saved new best validating model with loss 95.456321 to: data/cp_rumodel2/best_dev-12170

i dont know what is the mean, i just copy paste log from terminal.

im using cpu, so i cant

othiele · March 20, 2020, 9:12am

Ok, do you have a question? This result could be just right, but without more information it is hard to tell. Russian has a lot of cases and many pronouns, you might just need more audio input.

Evgenij_Gerasimov · March 20, 2020, 9:14pm

Yes, i have a question: how to make loss lower than 20%?
Also im not specify language model, can it affect on training process?

A little bit more logs:

Epoch 9 |   Training | Elapsed Time: 0:19:07 | Steps: 1217 | Loss: 53.498155   
Epoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | DatasEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 24.001438 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 2 | Loss: 26.139920 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 29.984301 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 4 | Loss: 33.154804 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 5 | Loss: 35.215176 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 6 | Loss: 38.371660 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 7 | Loss: 38.652054 | DataEpoch 9 | Validation | Elapsed Time: 0:00:00 | Steps: 8 | Loss: 38.825130 | DataEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 9 | Loss: 39.280251 | DataEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 10 | Loss: 40.134796 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 11 | Loss: 42.435915 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 12 | Loss: 42.122548 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 13 | Loss: 42.439535 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 14 | Loss: 43.937102 | DatEpoch 9 | Validation | Elapsed Time: 0:00:01 | Steps: 15 | Loss: 44.491772 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 16 | Loss: 44.213127 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 17 | Loss: 44.542344 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 18 | Loss: 44.711684 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 19 | Loss: 45.564220 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 20 | Loss: 45.202640 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 21 | Loss: 45.491929 | DatEpoch 9 | Validation | Elapsed Time: 0:00:02 | Steps: 22 | Loss: 46.109399 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 23 | Loss: 46.741205 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 24 | Loss: 46.471454 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 25 | Loss: 45.891328 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 26 | Loss: 46.225850 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 27 | Loss: 47.317082 | DatEpoch 9 | Validation | Elapsed Time: 0:00:03 | Steps: 28 | Loss: 47.156155 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 29 | Loss: 47.622437 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 30 | Loss: 47.424197 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 31 | Loss: 48.216693 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 32 | Loss: 48.476222 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 33 | Loss: 48.676489 | DatEpoch 9 | Validation | Elapsed Time: 0:00:04 | Steps: 34 | Loss: 48.684854 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 35 | Loss: 48.969126 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 36 | Loss: 49.080404 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 37 | Loss: 49.778108 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 38 | Loss: 49.909148 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 39 | Loss: 50.261603 | DatEpoch 9 | Validation | Elapsed Time: 0:00:05 | Steps: 40 | Loss: 50.344049 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 41 | Loss: 50.647617 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 42 | Loss: 50.924914 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 43 | Loss: 51.235628 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 44 | Loss: 52.299264 | DatEpoch 9 | Validation | Elapsed Time: 0:00:06 | Steps: 45 | Loss: 52.982479 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 46 | Loss: 53.631909 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 47 | Loss: 54.205429 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 48 | Loss: 54.306188 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 49 | Loss: 54.451473 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 50 | Loss: 54.526620 | DatEpoch 9 | Validation | Elapsed Time: 0:00:07 | Steps: 51 | Loss: 55.211812 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 52 | Loss: 55.197487 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 53 | Loss: 55.696617 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 54 | Loss: 56.570952 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 55 | Loss: 57.052983 | DatEpoch 9 | Validation | Elapsed Time: 0:00:08 | Steps: 56 | Loss: 57.392976 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 57 | Loss: 57.701493 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 58 | Loss: 57.884207 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 59 | Loss: 58.080325 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 60 | Loss: 58.293715 | DatEpoch 9 | Validation | Elapsed Time: 0:00:09 | Steps: 61 | Loss: 58.931091 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 62 | Loss: 59.011049 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 63 | Loss: 59.257370 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 64 | Loss: 60.076057 | DatEpoch 9 | Validation | Elapsed Time: 0:00:10 | Steps: 65 | Loss: 60.483996 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 66 | Loss: 60.755636 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 67 | Loss: 61.100232 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 68 | Loss: 61.485705 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 69 | Loss: 62.023526 | DatEpoch 9 | Validation | Elapsed Time: 0:00:11 | Steps: 70 | Loss: 62.229157 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 71 | Loss: 62.479189 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 72 | Loss: 63.117895 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 73 | Loss: 63.364378 | DatEpoch 9 | Validation | Elapsed Time: 0:00:12 | Steps: 74 | Loss: 63.593335 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 75 | Loss: 64.020090 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 76 | Loss: 64.333973 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 77 | Loss: 64.479004 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 78 | Loss: 64.752624 | DatEpoch 9 | Validation | Elapsed Time: 0:00:13 | Steps: 79 | Loss: 64.965228 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 80 | Loss: 65.150344 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 81 | Loss: 65.561469 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 82 | Loss: 65.624622 | DatEpoch 9 | Validation | Elapsed Time: 0:00:14 | Steps: 83 | Loss: 65.664437 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 84 | Loss: 65.883643 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 85 | Loss: 66.108031 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 86 | Loss: 66.310900 | DatEpoch 9 | Validation | Elapsed Time: 0:00:15 | Steps: 87 | Loss: 66.288677 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 88 | Loss: 66.511101 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 89 | Loss: 66.445487 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 90 | Loss: 66.417586 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 91 | Loss: 66.486743 | DatEpoch 9 | Validation | Elapsed Time: 0:00:16 | Steps: 92 | Loss: 66.654646 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 93 | Loss: 66.910886 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 94 | Loss: 67.146643 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 95 | Loss: 67.259888 | DatEpoch 9 | Validation | Elapsed Time: 0:00:17 | Steps: 96 | Loss: 67.745446 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 97 | Loss: 68.105616 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 98 | Loss: 68.725401 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 99 | Loss: 69.229914 | DatEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 100 | Loss: 69.518019 | DaEpoch 9 | Validation | Elapsed Time: 0:00:18 | Steps: 101 | Loss: 69.933013 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 102 | Loss: 70.141830 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 103 | Loss: 70.237128 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 104 | Loss: 70.324611 | DaEpoch 9 | Validation | Elapsed Time: 0:00:19 | Steps: 105 | Loss: 70.590663 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 106 | Loss: 70.764109 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 107 | Loss: 70.891378 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 108 | Loss: 71.559604 | DaEpoch 9 | Validation | Elapsed Time: 0:00:20 | Steps: 109 | Loss: 71.869520 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 110 | Loss: 72.144179 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 111 | Loss: 72.432612 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 112 | Loss: 72.789465 | DaEpoch 9 | Validation | Elapsed Time: 0:00:21 | Steps: 113 | Loss: 72.855290 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 114 | Loss: 72.888665 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 115 | Loss: 73.414483 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 116 | Loss: 73.691735 | DaEpoch 9 | Validation | Elapsed Time: 0:00:22 | Steps: 117 | Loss: 73.921992 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 118 | Loss: 73.931742 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 119 | Loss: 74.343568 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 120 | Loss: 74.704309 | DaEpoch 9 | Validation | Elapsed Time: 0:00:23 | Steps: 121 | Loss: 74.908236 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 122 | Loss: 74.942185 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 123 | Loss: 75.049731 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 124 | Loss: 75.161850 | DaEpoch 9 | Validation | Elapsed Time: 0:00:24 | Steps: 125 | Loss: 75.568701 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 126 | Loss: 75.811799 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 127 | Loss: 75.918382 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 128 | Loss: 76.199509 | DaEpoch 9 | Validation | Elapsed Time: 0:00:25 | Steps: 129 | Loss: 76.237561 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 130 | Loss: 76.679952 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 131 | Loss: 76.941891 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 132 | Loss: 77.120592 | DaEpoch 9 | Validation | Elapsed Time: 0:00:26 | Steps: 133 | Loss: 77.459202 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 134 | Loss: 77.563689 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 135 | Loss: 77.801109 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 136 | Loss: 77.957253 | DaEpoch 9 | Validation | Elapsed Time: 0:00:27 | Steps: 137 | Loss: 78.096835 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 138 | Loss: 78.330076 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 139 | Loss: 78.616373 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 140 | Loss: 78.939457 | DaEpoch 9 | Validation | Elapsed Time: 0:00:28 | Steps: 141 | Loss: 78.892939 | DaEpoch 9 | Validation | Elapsed Time: 0:00:29 | Steps: 142 | Loss: 79.041815 | DaEpoch 9 | Validation | Elapsed Time: 0:00:29 | Steps: 143 | Loss: 79.246707 | DaEpoch 9 | Validation | Elapsed Time: 0:00:29 | Steps: 144 | Loss: 79.587958 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 145 | Loss: 79.864611 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 146 | Loss: 80.264299 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 147 | Loss: 80.545905 | DaEpoch 9 | Validation | Elapsed Time: 0:00:30 | Steps: 148 | Loss: 80.653374 | DaEpoch 9 | Validation | Elapsed Time: 0:00:31 | Steps: 149 | Loss: 80.810095 | DaEpoch 9 | Validation | Elapsed Time: 0:00:31 | Steps: 150 | Loss: 80.830239 | DaEpoch 9 | Validation | Elapsed Time: 0:00:31 | Steps: 151 | Loss: 81.297724 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 152 | Loss: 81.435584 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 153 | Loss: 81.806836 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 154 | Loss: 81.794360 | DaEpoch 9 | Validation | Elapsed Time: 0:00:32 | Steps: 155 | Loss: 81.953380 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 156 | Loss: 82.078743 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 157 | Loss: 82.307890 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 158 | Loss: 82.387073 | DaEpoch 9 | Validation | Elapsed Time: 0:00:33 | Steps: 159 | Loss: 82.565338 | DaEpoch 9 | Validation | Elapsed Time: 0:00:34 | Steps: 160 | Loss: 82.817048 | DaEpoch 9 | Validation | Elapsed Time: 0:00:34 | Steps: 161 | Loss: 83.044786 | DaEpoch 9 | Validation | Elapsed Time: 0:00:34 | Steps: 162 | Loss: 83.145943 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 163 | Loss: 83.576915 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 164 | Loss: 83.759789 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 165 | Loss: 83.874057 | DaEpoch 9 | Validation | Elapsed Time: 0:00:35 | Steps: 166 | Loss: 83.972620 | DaEpoch 9 | Validation | Elapsed Time: 0:00:36 | Steps: 167 | Loss: 83.993627 | DaEpoch 9 | Validation | Elapsed Time: 0:00:36 | Steps: 168 | Loss: 84.208651 | DaEpoch 9 | Validation | Elapsed Time: 0:00:36 | Steps: 169 | Loss: 84.412908 | DaEpoch 9 | Validation | Elapsed Time: 0:00:37 | Steps: 170 | Loss: 84.589244 | DaEpoch 9 | Validation | Elapsed Time: 0:00:37 | Steps: 171 | Loss: 84.805894 | DaEpoch 9 | Validation | Elapsed Time: 0:00:37 | Steps: 172 | Loss: 85.011773 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 173 | Loss: 85.017073 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 174 | Loss: 85.096596 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 175 | Loss: 85.170380 | DaEpoch 9 | Validation | Elapsed Time: 0:00:38 | Steps: 176 | Loss: 85.357464 | DaEpoch 9 | Validation | Elapsed Time: 0:00:39 | Steps: 177 | Loss: 85.414403 | DaEpoch 9 | Validation | Elapsed Time: 0:00:39 | Steps: 178 | Loss: 85.501928 | DaEpoch 9 | Validation | Elapsed Time: 0:00:39 | Steps: 179 | Loss: 85.570174 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 180 | Loss: 85.655280 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 181 | Loss: 85.704665 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 182 | Loss: 85.872735 | DaEpoch 9 | Validation | Elapsed Time: 0:00:40 | Steps: 183 | Loss: 86.054915 | DaEpoch 9 | Validation | Elapsed Time: 0:00:41 | Steps: 184 | Loss: 86.274824 | DaEpoch 9 | Validation | Elapsed Time: 0:00:41 | Steps: 185 | Loss: 86.384777 | DaEpoch 9 | Validation | Elapsed Time: 0:00:41 | Steps: 186 | Loss: 86.673783 | DaEpoch 9 | Validation | Elapsed Time: 0:00:42 | Steps: 187 | Loss: 86.962934 | DaEpoch 9 | Validation | Elapsed Time: 0:00:42 | Steps: 188 | Loss: 87.025896 | DaEpoch 9 | Validation | Elapsed Time: 0:00:42 | Steps: 189 | Loss: 87.168968 | DaEpoch 9 | Validation | Elapsed Time: 0:00:43 | Steps: 190 | Loss: 87.619509 | DaEpoch 9 | Validation | Elapsed Time: 0:00:43 | Steps: 191 | Loss: 87.850966 | DaEpoch 9 | Validation | Elapsed Time: 0:00:43 | Steps: 192 | Loss: 87.980060 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 193 | Loss: 88.184805 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 194 | Loss: 88.459959 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 195 | Loss: 88.638910 | DaEpoch 9 | Validation | Elapsed Time: 0:00:44 | Steps: 196 | Loss: 88.730926 | DaEpoch 9 | Validation | Elapsed Time: 0:00:45 | Steps: 197 | Loss: 88.997776 | DaEpoch 9 | Validation | Elapsed Time: 0:00:45 | Steps: 198 | Loss: 89.163274 | DaEpoch 9 | Validation | Elapsed Time: 0:00:45 | Steps: 199 | Loss: 89.264240 | DaEpoch 9 | Validation | Elapsed Time: 0:00:46 | Steps: 200 | Loss: 89.444111 | DaEpoch 9 | Validation | Elapsed Time: 0:00:46 | Steps: 201 | Loss: 89.557194 | DaEpoch 9 | Validation | Elapsed Time: 0:00:46 | Steps: 202 | Loss: 89.654985 | DaEpoch 9 | Validation | Elapsed Time: 0:00:47 | Steps: 203 | Loss: 89.816107 | DaEpoch 9 | Validation | Elapsed Time: 0:00:47 | Steps: 204 | Loss: 89.891788 | DaEpoch 9 | Validation | Elapsed Time: 0:00:47 | Steps: 205 | Loss: 90.064587 | DaEpoch 9 | Validation | Elapsed Time: 0:00:48 | Steps: 206 | Loss: 90.122345 | DaEpoch 9 | Validation | Elapsed Time: 0:00:48 | Steps: 207 | Loss: 90.127684 | DaEpoch 9 | Validation | Elapsed Time: 0:00:48 | Steps: 208 | Loss: 90.261094 | DaEpoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 209 | Loss: 90.598459 | DaEpoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 210 | Loss: 90.717597 | DaEpoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 211 | Loss: 90.853893 | DaEpoch 9 | Validation | Elapsed Time: 0:00:50 | Steps: 212 | Loss: 90.980856 | DaEpoch 9 | Validation | Elapsed Time: 0:00:50 | Steps: 213 | Loss: 91.058446 | DaEpoch 9 | Validation | Elapsed Time: 0:00:50 | Steps: 214 | Loss: 91.125155 | DaEpoch 9 | Validation | Elapsed Time: 0:00:51 | Steps: 215 | Loss: 91.356190 | DaEpoch 9 | Validation | Elapsed Time: 0:00:51 | Steps: 216 | Loss: 91.687895 | DaEpoch 9 | Validation | Elapsed Time: 0:00:51 | Steps: 217 | Loss: 91.754220 | DaEpoch 9 | Validation | Elapsed Time: 0:00:52 | Steps: 218 | Loss: 91.879785 | DaEpoch 9 | Validation | Elapsed Time: 0:00:52 | Steps: 219 | Loss: 92.022793 | DaEpoch 9 | Validation | Elapsed Time: 0:00:52 | Steps: 220 | Loss: 92.177591 | DaEpoch 9 | Validation | Elapsed Time: 0:00:53 | Steps: 221 | Loss: 92.299894 | DaEpoch 9 | Validation | Elapsed Time: 0:00:53 | Steps: 222 | Loss: 92.559568 | DaEpoch 9 | Validation | Elapsed Time: 0:00:53 | Steps: 223 | Loss: 92.566857 | DaEpoch 9 | Validation | Elapsed Time: 0:00:54 | Steps: 224 | Loss: 92.716912 | DaEpoch 9 | Validation | Elapsed Time: 0:00:54 | Steps: 225 | Loss: 92.667962 | DaEpoch 9 | Validation | Elapsed Time: 0:00:55 | Steps: 226 | Loss: 92.845427 | DaEpoch 9 | Validation | Elapsed Time: 0:00:55 | Steps: 227 | Loss: 93.016710 | DaEpoch 9 | Validation | Elapsed Time: 0:00:55 | Steps: 228 | Loss: 93.274283 | DaEpoch 9 | Validation | Elapsed Time: 0:00:56 | Steps: 229 | Loss: 93.460927 | DaEpoch 9 | Validation | Elapsed Time: 0:00:56 | Steps: 230 | Loss: 93.443831 | DaEpoch 9 | Validation | Elapsed Time: 0:00:56 | Steps: 231 | Loss: 93.556214 | DaEpoch 9 | Validation | Elapsed Time: 0:00:57 | Steps: 232 | Loss: 93.674318 | DaEpoch 9 | Validation | Elapsed Time: 0:00:57 | Steps: 233 | Loss: 93.822391 | DaEpoch 9 | Validation | Elapsed Time: 0:00:57 | Steps: 234 | Loss: 93.986279 | DaEpoch 9 | Validation | Elapsed Time: 0:00:58 | Steps: 235 | Loss: 94.218827 | DaEpoch 9 | Validation | Elapsed Time: 0:00:58 | Steps: 236 | Loss: 94.266340 | DaEpoch 9 | Validation | Elapsed Time: 0:00:59 | Steps: 237 | Loss: 94.336616 | DaEpoch 9 | Validation | Elapsed Time: 0:00:59 | Steps: 238 | Loss: 94.387358 | DaEpoch 9 | Validation | Elapsed Time: 0:00:59 | Steps: 239 | Loss: 94.437320 | DaEpoch 9 | Validation | Elapsed Time: 0:01:00 | Steps: 240 | Loss: 94.576774 | DaEpoch 9 | Validation | Elapsed Time: 0:01:00 | Steps: 241 | Loss: 94.631176 | DaEpoch 9 | Validation | Elapsed Time: 0:01:01 | Steps: 242 | Loss: 94.727343 | DaEpoch 9 | Validation | Elapsed Time: 0:01:01 | Steps: 243 | Loss: 94.832889 | DaEpoch 9 | Validation | Elapsed Time: 0:01:01 | Steps: 244 | Loss: 95.004763 | DaEpoch 9 | Validation | Elapsed Time: 0:01:02 | Steps: 245 | Loss: 95.130512 | DaEpoch 9 | Validation | Elapsed Time: 0:01:02 | Steps: 246 | Loss: 95.136806 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 247 | Loss: 95.219078 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 248 | Loss: 95.351087 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 249 | Loss: 95.456321 | DaEpoch 9 | Validation | Elapsed Time: 0:01:03 | Steps: 249 | Loss: 95.456321 | Dataset: /media/djo/DS/work/rumodel/clips/dev2k.csv
I Saved new best validating model with loss 95.456321 to: data/cp_rumodel2/best_dev-12170
I FINISHED optimization in 3:23:23.062624
[scorer.cpp:77] FATAL: "(access(filename, 4)) == (0)" check failed. Invalid language model path

othiele · March 20, 2020, 9:28pm

You need more and better data. I guess that the data is not as good yet and you will probably need 250 hours more. Just contribute to common voice and you’ll eventually get there, sorry, there is not much else to do but gather more data.

Evgenij_Gerasimov · March 22, 2020, 2:27pm

my main target is make small model like that guy TUTORIAL : How I trained a specific french model to control my robot He make working model with just 1000 sentences, and i also want create small model for my robot. In result i want to recognize just 100-200 commands. Can you give me advise to achieve it for Russian model?

nshmyrev · March 22, 2020, 2:32pm

You can actually get a big Russian dataset, the pretrained models and scripts here:

Ilia_Ilich · March 24, 2020, 9:57pm

Hi There, I am trying to train deepspeech model with russian language (Common voice + voxforge). So should I edit something else besides the alphabet.txt file? Using colab, I run the following script:

!python3 DeepSpeech.py
–train_files “…/train.csv” \ (about 100 hours)
–dev_files “…/dev.csv”
–test_files “…test.csv”
–train_batch_size 16
–dev_batch_size 16
–test_batch_size 16
–learning_rate 0.0001
–epoch 15
–dropout_rate 0.3
–export_dir …/exports \

After all I have pretty low CTC loss (about 30-40) during training, but then test phrases is recognized incorrectly, like:
-src:“женя достала аккордеон и перекинула ремень через плечо”
-res:“оноокдвнур”

What did I do wrong?