What does the "Test of Epoch [number]" mean?

Franck_Dernoncourt · June 30, 2018, 8:00pm

What does the “Test of Epoch [number]” mean in Mozilla DeepSpeech?

In the following example, it says Test of Epoch 77263, even though there should be just 1 epoch from my understanding, since I gave --display_step 1 --limit_train 1 --limit_dev 1 --limit_test 1 --early_stop False --epoch 1 as arguments:

dernoncourt@ilcomp:~/asr/DeepSpeech$ ./DeepSpeech.py --train_files data/common-voice-v1/cv-valid-train.csv,data/common-voice-v1/cv-other-train.csv --dev_files data/common-voice-v1/cv-valid-dev.csv --test_files data/common-voice-v1/cv-valid-test.csv --decoder_library_path /asr/DeepSpeech/libctc_decoder_with_kenlm.so --fulltrace True --display_step 1  --limit_train 1  --limit_dev 1  --limit_test 1 --early_stop False --epoch 1
W Parameter --validation_step needs to be >0 for early stopping to work
I Test of Epoch 77263 - WER: 1.000000, loss: 60.50202560424805, mean edit distance: 0.894737
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 58.900837, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 60.517113, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 60.668221, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 61.921925, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------

lissyx · July 2, 2018, 5:09am

That looks like a bug, since it should be the number of the epoch it’s running test for :-(.

Is this the whole console output?

Franck_Dernoncourt · July 2, 2018, 6:22am

Yes. Here is the whole console output, with --log_level 0:

dernonco@ilcomp:~/asr/mozilla-deepspeech/DeepSpeech$  ./DeepSpeech.py --train_files data/common-voice-v1/cv-valid-train.csv,data/common-voice-v1/cv-other-train.csv --dev_files data/common-voice-v1/cv-valid-dev.csv --test_files data/common-voice-v1/cv-valid-test.csv --decoder_library_path ./libctc_decoder_with_kenlm.so --fulltrace True --display_step 1  --limit_train 1  --limit_dev 1  --limit_test 1 --early_stop False --epoch 1 --log_level 0
D Starting coordinator...
D Coordinator started.
2018-07-01 23:16:54.691759: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-07-01 23:16:55.393652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:02:00.0
totalMemory: 7.92GiB freeMemory: 7.77GiB
2018-07-01 23:16:55.943243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 1 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:03:00.0
totalMemory: 7.93GiB freeMemory: 7.81GiB
2018-07-01 23:16:56.532443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 2 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:81:00.0
totalMemory: 7.93GiB freeMemory: 7.81GiB
2018-07-01 23:16:57.187047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 3 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:82:00.0
totalMemory: 7.93GiB freeMemory: 7.81GiB
2018-07-01 23:16:57.190133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1227] Device peer to peer matrix
2018-07-01 23:16:57.190309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1233] DMA: 0 1 2 3
2018-07-01 23:16:57.190327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1243] 0:   Y Y N N
2018-07-01 23:16:57.190340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1243] 1:   Y Y N N
2018-07-01 23:16:57.190351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1243] 2:   N N Y Y
2018-07-01 23:16:57.190362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1243] 3:   N N Y Y
2018-07-01 23:16:57.190382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0, 1, 2, 3
2018-07-01 23:16:58.402244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/device:GPU:0 with 7511 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
2018-07-01 23:16:58.518643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/device:GPU:1 with 7543 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080, pci bus id: 0000:03:00.0, compute capability: 6.1)
2018-07-01 23:16:58.711720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/device:GPU:2 with 7543 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080, pci bus id: 0000:81:00.0, compute capability: 6.1)
2018-07-01 23:16:58.907316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/device:GPU:3 with 7543 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080, pci bus id: 0000:82:00.0, compute capability: 6.1)
W Parameter --validation_step needs to be >0 for early stopping to work
2018-07-01 23:17:05.517874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0, 1, 2, 3
2018-07-01 23:17:05.518236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 169 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
2018-07-01 23:17:05.518496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 171 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080, pci bus id: 0000:03:00.0, compute capability: 6.1)
2018-07-01 23:17:05.518995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 171 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080, pci bus id: 0000:81:00.0, compute capability: 6.1)
2018-07-01 23:17:05.519336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 171 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080, pci bus id: 0000:82:00.0, compute capability: 6.1)

D Starting queue runners...
D Queue runners started.
D step: 93750
D epoch: 93750
D target epoch: 1
D steps per epoch: 1
D number of batches in train set: 1
D batches per job: 4
D batches per step: 4
D number of jobs in train set: 1
D number of jobs already trained in first epoch: 0
D Computing Job (ID: 2, worker: 0, epoch: 93750, set_name: test)...
D Starting batch...
D Finished batch step 93750.
D Sending Job (ID: 2, worker: 0, epoch: 93750, set_name: test)...
I Test of Epoch 93750 - WER: 1.000000, loss: 57.80370330810547, mean edit distance: 0.894737
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 57.285061, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 57.502094, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 58.014362, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 58.413303, mean edit distance: 0.894737
I  - src: "how do you like her"
I  - res: "i "
I --------------------------------------------------------------------------------
D Epochs - running: 0, done: 1
D Closing queues...
D Queues closed.
D Session closed.
D Done.
D Stopping coordinator...
D Coordinator stopped.

I opened a GitHub issue: https://github.com/mozilla/DeepSpeech/issues/1425