Hi,
Thank you replying.
- Please see the below logs of Epoch 27 training and validation:
I have posted few lines from logs
Epoch 26 | Validation | Elapsed Time: 0:27:56 | Steps: 1498 | Loss: 928.760906 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:28:04 | Steps: 1499 | Loss: 929.050569 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:28:14 | Steps: 1500 | Loss: 930.103331 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:28:24 | Steps: 1501 | Loss: 930.834689 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:28:39 | Steps: 1502 | Loss: 933.888949 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:28:53 | Steps: 1503 | Loss: 935.579138 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:29:08 | Steps: 1504 | Loss: 937.148359 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:29:46 | Steps: 1505 | Loss: 951.052634 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:30:04 | Steps: 1506 | Loss: 952.619517 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:30:38 | Steps: 1507 | Loss: 962.583601 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:31:16 | Steps: 1508 | Loss: 969.871313 | Dataset: ../clips/dev.csv
Epoch 26 | Validation | Elapsed Time: 0:31:16 | Steps: 1508 | Loss: 969.871313 | Dataset: ../clips/dev.csv
--------------------------------------------------------------------------------
Epoch 27 | Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000
Epoch 27 | Training | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 16.012011
Epoch 27 | Training | Elapsed Time: 0:00:00 | Steps: 2 | Loss: 40.323937
Epoch 27 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 42.284379
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 4 | Loss: 43.334262
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 5 | Loss: 50.498967
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 6 | Loss: 52.865224
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 7 | Loss: 58.218975
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 8 | Loss: 53.031822
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 9 | Loss: 55.255576
Epoch 27 | Training | Elapsed Time: 0:00:01 | Steps: 10 | Loss: 55.001126
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 11 | Loss: 70.587121
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 12 | Loss: 74.051502
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 13 | Loss: 78.278034
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 14 | Loss: 74.092985
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 15 | Loss: 77.536041
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 16 | Loss: 78.462801
Epoch 27 | Training | Elapsed Time: 0:00:02 | Steps: 17 | Loss: 81.968311
Epoch 27 | Training | Elapsed Time: 0:00:03 | Steps: 18 | Loss: 80.938078
Epoch 27 | Training | Elapsed Time: 0:00:03 | Steps: 19 | Loss: 79.977340
Epoch 27 | Training | Elapsed Time: 0:00:03 | Steps: 20 | Loss: 86.060360
Epoch 27 | Training | Elapsed Time: 0:00:03 | Steps: 21 | Loss: 88.086165
Epoch 27 | Training | Elapsed Time: 0:00:03 | Steps: 22 | Loss: 90.268335
Epoch 27 | Training | Elapsed Time: 0:00:04 | Steps: 23 | Loss: 96.002163
Epoch 27 | Training | Elapsed Time: 0:00:04 | Steps: 24 | Loss: 97.024613
Epoch 27 | Training | Elapsed Time: 0:00:04 | Steps: 25 | Loss: 96.164787
Epoch 27 | Training | Elapsed Time: 0:00:04 | Steps: 26 | Loss: 95.270884
Epoch 27 | Training | Elapsed Time: 0:00:04 | Steps: 27 | Loss: 102.807339
Epoch 27 | Training | Elapsed Time: 0:00:05 | Steps: 28 | Loss: 107.701536
Epoch 27 | Training | Elapsed Time: 0:00:05 | Steps: 29 | Loss: 106.914660
Epoch 27 | Training | Elapsed Time: 0:00:05 | Steps: 30 | Loss: 111.810944
Epoch 27 | Training | Elapsed Time: 0:00:05 | Steps: 31 | Loss: 108.951781
Epoch 27 | Training | Elapsed Time: 0:00:06 | Steps: 32 | Loss: 111.481779
Epoch 27 | Training | Elapsed Time: 0:00:06 | Steps: 33 | Loss: 113.189505
Epoch 27 | Training | Elapsed Time: 0:00:06 | Steps: 34 | Loss: 113.131423
Epoch 27 | Training | Elapsed Time: 0:00:06 | Steps: 35 | Loss: 112.271983
Epoch 27 | Training | Elapsed Time: 0:00:06 | Steps: 36 | Loss: 114.633477
Epoch 27 | Training | Elapsed Time: 0:00:07 | Steps: 37 | Loss: 113.646733
Epoch 27 | Training | Elapsed Time: 0:00:07 | Steps: 38 | Loss: 112.647386
Epoch 27 | Training | Elapsed Time: 0:00:07 | Steps: 39 | Loss: 115.887965
Epoch 27 | Training | Elapsed Time: 0:00:07 | Steps: 40 | Loss: 114.054972
Epoch 27 | Training | Elapsed Time: 0:00:08 | Steps: 41 | Loss: 116.054757
Epoch 27 | Training | Elapsed Time: 0:00:08 | Steps: 42 | Loss: 115.993475
Epoch 27 | Training | Elapsed Time: 0:00:08 | Steps: 43 | Loss: 117.596279
Epoch 27 | Training | Elapsed Time: 0:00:08 | Steps: 44 | Loss: 115.539979
Epoch 27 | Training | Elapsed Time: 6:54:29 | Steps: 12179 | Loss: 860.764300
Epoch 27 | Training | Elapsed Time: 6:55:06 | Steps: 12180 | Loss: 862.080741
Epoch 27 | Training | Elapsed Time: 6:55:29 | Steps: 12181 | Loss: 862.154552
Epoch 27 | Training | Elapsed Time: 6:55:50 | Steps: 12182 | Loss: 862.176993
Epoch 27 | Training | Elapsed Time: 6:56:12 | Steps: 12183 | Loss: 862.218486
Epoch 27 | Training | Elapsed Time: 6:56:34 | Steps: 12184 | Loss: 862.244087
Epoch 27 | Training | Elapsed Time: 6:56:55 | Steps: 12185 | Loss: 862.256227
Epoch 27 | Training | Elapsed Time: 6:57:37 | Steps: 12186 | Loss: 864.360452
Epoch 27 | Training | Elapsed Time: 6:58:03 | Steps: 12187 | Loss: 864.612945
Epoch 27 | Training | Elapsed Time: 6:58:31 | Steps: 12188 | Loss: 864.942953
Epoch 27 | Training | Elapsed Time: 6:59:10 | Steps: 12189 | Loss: 866.288809
Epoch 27 | Training | Elapsed Time: 6:59:39 | Steps: 12190 | Loss: 866.691137
Epoch 27 | Training | Elapsed Time: 7:00:07 | Steps: 12191 | Loss: 866.903732
Epoch 27 | Training | Elapsed Time: 7:00:53 | Steps: 12192 | Loss: 868.637313
Epoch 27 | Training | Elapsed Time: 7:01:40 | Steps: 12193 | Loss: 869.857616
Epoch 27 | Training | Elapsed Time: 7:02:32 | Steps: 12194 | Loss: 871.582724
Epoch 27 | Training | Elapsed Time: 7:03:10 | Steps: 12195 | Loss: 871.903624
Epoch 27 | Training | Elapsed Time: 7:03:54 | Steps: 12196 | Loss: 872.549494
Epoch 27 | Training | Elapsed Time: 7:04:26 | Steps: 12197 | Loss: 872.609729
Epoch 27 | Training | Elapsed Time: 7:04:59 | Steps: 12198 | Loss: 872.681917
Epoch 27 | Training | Elapsed Time: 7:05:45 | Steps: 12199 | Loss: 873.124481
Epoch 27 | Training | Elapsed Time: 7:06:30 | Steps: 12200 | Loss: 873.234334
Epoch 27 | Training | Elapsed Time: 7:06:30 | Steps: 12200 | Loss: 873.234334
Epoch 27 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 80.903816 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 93.584595 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:00 | Steps: 5 | Loss: 87.633374 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:00 | Steps: 6 | Loss: 117.426702 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:00 | Steps: 7 | Loss: 128.203088 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 8 | Loss: 150.572115 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 9 | Loss: 160.595259 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 10 | Loss: 160.264433 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 11 | Loss: 165.206120 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 12 | Loss: 173.128124 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 13 | Loss: 169.424146 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 14 | Loss: 167.738200 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:01 | Steps: 15 | Loss: 178.499871 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:00:02 | Steps: 16 | Loss: 190.872425 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:16:52 | Steps: 1343 | Loss: 793.615168 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:16:55 | Steps: 1344 | Loss: 795.010740 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:16:57 | Steps: 1345 | Loss: 794.881033 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:00 | Steps: 1346 | Loss: 795.183453 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:02 | Steps: 1347 | Loss: 796.830010 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:05 | Steps: 1348 | Loss: 798.038055 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:07 | Steps: 1349 | Loss: 797.732362 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:10 | Steps: 1350 | Loss: 797.699558 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:12 | Steps: 1351 | Loss: 797.269964 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:14 | Steps: 1352 | Loss: 797.968867 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:17 | Steps: 1353 | Loss: 798.801794 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:20 | Steps: 1354 | Loss: 799.961696 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:23 | Steps: 1355 | Loss: 803.273815 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:26 | Steps: 1356 | Loss: 804.786562 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:29 | Steps: 1357 | Loss: 807.159425 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:31 | Steps: 1358 | Loss: 808.853647 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:34 | Steps: 1359 | Loss: 808.520369 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:36 | Steps: 1360 | Loss: 808.759987 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:39 | Steps: 1361 | Loss: 810.536654 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:42 | Steps: 1362 | Loss: 810.537405 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:45 | Steps: 1363 | Loss: 812.788877 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:47 | Steps: 1364 | Loss: 813.746725 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:50 | Steps: 1365 | Loss: 814.942635 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:53 | Steps: 1366 | Loss: 815.926070 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:56 | Steps: 1367 | Loss: 817.258293 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:17:59 | Steps: 1368 | Loss: 818.337001 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:01 | Steps: 1369 | Loss: 818.693495 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:04 | Steps: 1370 | Loss: 819.338378 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:07 | Steps: 1371 | Loss: 820.987847 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:10 | Steps: 1372 | Loss: 822.994076 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:13 | Steps: 1373 | Loss: 823.755432 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:16 | Steps: 1374 | Loss: 824.632283 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:19 | Steps: 1375 | Loss: 826.120103 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:21 | Steps: 1376 | Loss: 826.865366 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:24 | Steps: 1377 | Loss: 828.347706 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:27 | Steps: 1378 | Loss: 829.133034 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:30 | Steps: 1379 | Loss: 830.704378 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:33 | Steps: 1380 | Loss: 830.779444 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:36 | Steps: 1381 | Loss: 832.427738 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:39 | Steps: 1382 | Loss: 834.554614 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:42 | Steps: 1383 | Loss: 835.129614 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:45 | Steps: 1384 | Loss: 836.792431 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:48 | Steps: 1385 | Loss: 837.699977 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:51 | Steps: 1386 | Loss: 837.926158 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:54 | Steps: 1387 | Loss: 838.783328 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:56 | Steps: 1388 | Loss: 838.467737 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:18:59 | Steps: 1389 | Loss: 839.295391 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:02 | Steps: 1390 | Loss: 839.949800 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:05 | Steps: 1391 | Loss: 839.548893 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:08 | Steps: 1392 | Loss: 840.358808 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:11 | Steps: 1393 | Loss: 841.728153 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:14 | Steps: 1394 | Loss: 844.044022 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:17 | Steps: 1395 | Loss: 845.332092 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:20 | Steps: 1396 | Loss: 846.231919 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:24 | Steps: 1397 | Loss: 848.490235 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:27 | Steps: 1398 | Loss: 848.223876 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:29 | Steps: 1399 | Loss: 847.893346 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:33 | Steps: 1400 | Loss: 850.723191 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:35 | Steps: 1401 | Loss: 850.549955 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:39 | Steps: 1402 | Loss: 853.738204 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:43 | Steps: 1403 | Loss: 855.902811 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:46 | Steps: 1404 | Loss: 857.191115 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:50 | Steps: 1405 | Loss: 863.227975 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:53 | Steps: 1406 | Loss: 863.443730 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:56 | Steps: 1407 | Loss: 865.461796 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:19:59 | Steps: 1408 | Loss: 865.455532 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:03 | Steps: 1409 | Loss: 867.688529 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:06 | Steps: 1410 | Loss: 868.592965 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:09 | Steps: 1411 | Loss: 868.622914 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:12 | Steps: 1412 | Loss: 869.686508 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:16 | Steps: 1413 | Loss: 871.034898 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:20 | Steps: 1414 | Loss: 873.534031 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:23 | Steps: 1415 | Loss: 875.361313 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:27 | Steps: 1416 | Loss: 876.908869 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:31 | Steps: 1417 | Loss: 878.584480 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:34 | Steps: 1418 | Loss: 879.382126 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:37 | Steps: 1419 | Loss: 880.418144 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:41 | Steps: 1420 | Loss: 881.659437 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:45 | Steps: 1421 | Loss: 883.414596 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:48 | Steps: 1422 | Loss: 884.148551 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:52 | Steps: 1423 | Loss: 887.322311 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:20:56 | Steps: 1424 | Loss: 888.454441 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:00 | Steps: 1425 | Loss: 889.784765 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:04 | Steps: 1426 | Loss: 890.893559 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:07 | Steps: 1427 | Loss: 891.702160 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:11 | Steps: 1428 | Loss: 893.009839 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:15 | Steps: 1429 | Loss: 895.353628 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:19 | Steps: 1430 | Loss: 896.263222 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:22 | Steps: 1431 | Loss: 897.374290 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:27 | Steps: 1432 | Loss: 900.152110 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:31 | Steps: 1433 | Loss: 901.776398 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:35 | Steps: 1434 | Loss: 903.093278 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:39 | Steps: 1435 | Loss: 905.595848 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:43 | Steps: 1436 | Loss: 907.800903 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:48 | Steps: 1437 | Loss: 909.420871 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:51 | Steps: 1438 | Loss: 909.482330 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:21:55 | Steps: 1439 | Loss: 909.743433 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:00 | Steps: 1440 | Loss: 914.024669 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:03 | Steps: 1441 | Loss: 914.596138 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:08 | Steps: 1442 | Loss: 917.180580 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:13 | Steps: 1443 | Loss: 919.655434 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:17 | Steps: 1444 | Loss: 921.280003 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:21 | Steps: 1445 | Loss: 922.620245 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:27 | Steps: 1446 | Loss: 925.851441 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:31 | Steps: 1447 | Loss: 926.727849 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:35 | Steps: 1448 | Loss: 926.880099 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:39 | Steps: 1449 | Loss: 928.102048 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:44 | Steps: 1450 | Loss: 929.907238 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:48 | Steps: 1451 | Loss: 929.841633 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:53 | Steps: 1452 | Loss: 932.190388 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:22:58 | Steps: 1453 | Loss: 932.535442 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:03 | Steps: 1454 | Loss: 933.865215 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:08 | Steps: 1455 | Loss: 937.271984 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:14 | Steps: 1456 | Loss: 939.500949 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:19 | Steps: 1457 | Loss: 940.536606 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:24 | Steps: 1458 | Loss: 943.492790 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:30 | Steps: 1459 | Loss: 945.415508 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:36 | Steps: 1460 | Loss: 949.264211 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:42 | Steps: 1461 | Loss: 953.206602 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:48 | Steps: 1462 | Loss: 955.411060 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:53 | Steps: 1463 | Loss: 957.947126 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:23:58 | Steps: 1464 | Loss: 958.226640 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:04 | Steps: 1465 | Loss: 960.473449 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:09 | Steps: 1466 | Loss: 961.644230 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:15 | Steps: 1467 | Loss: 962.994007 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:21 | Steps: 1468 | Loss: 965.554322 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:26 | Steps: 1469 | Loss: 966.367580 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:31 | Steps: 1470 | Loss: 967.182017 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:39 | Steps: 1471 | Loss: 973.170356 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:44 | Steps: 1472 | Loss: 973.492197 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:50 | Steps: 1473 | Loss: 976.403724 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:24:56 | Steps: 1474 | Loss: 978.280402 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:03 | Steps: 1475 | Loss: 980.894186 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:10 | Steps: 1476 | Loss: 984.094085 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:17 | Steps: 1477 | Loss: 986.639380 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:22 | Steps: 1478 | Loss: 986.501973 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:28 | Steps: 1479 | Loss: 988.203432 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:34 | Steps: 1480 | Loss: 988.158931 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:39 | Steps: 1481 | Loss: 987.921383 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:45 | Steps: 1482 | Loss: 989.142558 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:52 | Steps: 1483 | Loss: 991.321306 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:25:59 | Steps: 1484 | Loss: 994.284802 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:06 | Steps: 1485 | Loss: 995.222084 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:13 | Steps: 1486 | Loss: 998.174079 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:19 | Steps: 1487 | Loss: 998.327108 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:24 | Steps: 1488 | Loss: 998.138159 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:34 | Steps: 1489 | Loss: 1003.222412 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:43 | Steps: 1490 | Loss: 1008.583870 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:51 | Steps: 1491 | Loss: 1010.855054 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:26:58 | Steps: 1492 | Loss: 1010.652691 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:27:08 | Steps: 1493 | Loss: 1014.717950 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:27:18 | Steps: 1494 | Loss: 1018.666777 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:27:25 | Steps: 1495 | Loss: 1019.157669 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:27:37 | Steps: 1496 | Loss: 1024.630466 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:27:44 | Steps: 1497 | Loss: 1024.446256 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:27:57 | Steps: 1498 | Loss: 1029.979388 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:28:05 | Steps: 1499 | Loss: 1029.911999 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:28:15 | Steps: 1500 | Loss: 1030.822440 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:28:24 | Steps: 1501 | Loss: 1031.266270 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:28:40 | Steps: 1502 | Loss: 1034.604185 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:28:54 | Steps: 1503 | Loss: 1036.173080 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:29:09 | Steps: 1504 | Loss: 1037.541424 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:29:47 | Steps: 1505 | Loss: 1053.331423 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:30:05 | Steps: 1506 | Loss: 1054.584528 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:30:39 | Steps: 1507 | Loss: 1065.840281 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:31:16 | Steps: 1508 | Loss: 1073.914524 | Dataset: ../clips/dev.csv
Epoch 27 | Validation | Elapsed Time: 0:31:16 | Steps: 1508 | Loss: 1073.914524 | Dataset: ../clips/dev.csv
--------------------------------------------------------------------------------
-
I would be using --dropout_rate parameter in future . Could you please help me understand what exactly this parameter mean and what would be perfect 0.3 or 0.4 value for it?
-
we are using 12200 steps or files for training and 1508 files for validation
-
We are using English language ALPHABET file for training. Pasting ALPHABET.txt file below:
# Each line in this file represents the Unicode codepoint (UTF-8 encoded)
# associated with a numeric label.
# A line that starts with # is a comment. You can escape it with \# if you wish
# to use '#' as a label.
,
!
"
$
%
'
+
-
.
0
1
2
3
4
5
6
7
8
9
:
?
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
z
# The last (non-comment) line needs to end with a newline.