Hi,
Usually I used to store the Training Outputs in Text file. When I compare those Training outputs with the Tensorboard Graphs. Its Mismatching. . .
Tensorboard is showing graphs which is irrelavant to the Training Output.
See in my Training Output. . .
Loading best validating checkpoint from /home/bala/Speech_Recognition/Training_Pappa/v1-Jan5_No_Augment/deepspeech-0.9.2-checkpoint/best_dev-1564701
learning_rate : 0.000005
Epoch 0 | Training | Elapsed Time: 0:07:48 | Steps: 963 | Loss: 33.665629
Epoch 0 | Validation | Elapsed Time: 0:01:14 | Steps: 482 | Loss: 32.672577 | Dataset: /home/bala/Speech_Recognition/Training_Data_AWS/Dec-30_v1_Files/awsgroupdev.csv
I Saved new best validating model with loss 32.672577 to: /home/bala/Speech_Recognition/Training_Pappa/v1-Jan5_No_Augment/deepspeech-0.9.2-checkpoint/best_dev-1565664
Epoch 1 | Training | Elapsed Time: 0:07:59 | Steps: 963 | Loss: 33.600953
Epoch 1 | Validation | Elapsed Time: 0:01:13 | Steps: 482 | Loss: 32.527303 |
Epoch 2 | Training | Elapsed Time: 0:07:58 | Steps: 963 | Loss: 33.350906
Epoch 2 | Validation | Elapsed Time: 0:01:12 | Steps: 482 | Loss: 32.453275 |
Epoch 3 | Training | Elapsed Time: 0:07:58 | Steps: 963 | Loss: 33.195642
Epoch 3 | Validation | Elapsed Time: 0:01:13 | Steps: 482 | Loss: 32.487144 |
Epoch 4 | Training | Elapsed Time: 0:07:58 | Steps: 963 | Loss: 33.144140
Epoch 4 | Validation | Elapsed Time: 0:01:12 | Steps: 482 | Loss: 32.416610
Epoch 5 | Training | Elapsed Time: 0:07:59 | Steps: 963 | Loss: 32.796107
Epoch 5 | Validation | Elapsed Time: 0:01:13 | Steps: 482| Loss: 32.416092
Epoch 6 | Training | Elapsed Time: 0:08:00 | Steps: 963 | Loss: 32.730357
Epoch 6 | Validation | Elapsed Time: 0:01:12 | Steps: 482 | Loss: 32.320890
Epoch 7 | Training | Elapsed Time: 0:08:00 | Steps: 963 | Loss: 32.626243
Epoch 7 | Validation | Elapsed Time: 0:01:13 | Steps: 482 | Loss: 32.322866
Epoch 8 | Training | Elapsed Time: 0:08:00 | Steps: 963 | Loss: 32.543887
Epoch 8 | Validation | Elapsed Time: 0:01:12 | Steps: 482 | Loss: 32.240156 |
Epoch 9 | Training | Elapsed Time: 0:07:59 | Steps: 963 | Loss: 32.429364
Epoch 9 | Validation | Elapsed Time: 0:01:13 | Steps: 482 | Loss: 32.168196
Epoch 10 | Training | Elapsed Time: 0:07:58 | Steps: 963 | Loss: 32.202116
Epoch 10 | Validation | Elapsed Time: 0:01:12 | Steps: 482 | Loss: 32.198536
.
.
.
Epoch 44 | Training | Elapsed Time: 0:08:06 | Steps: 963 | Loss: 28.093275
Epoch 44 | Validation | Elapsed Time: 0:01:13 | Steps: 482 | Loss: 31.266112
See My Training and Validation loss is Keep on decreasing, but the tensorgraph showing irrelevant outputs.