How can I estimate training time of deepspeech

rao_jjn · January 11, 2018, 10:45am

I am running deepspeech training on Tedlium-2 and Mozzila open voice.
in approximate 12 hours it reached

Training of Epoch 0 - loss - 141.758459
I used batch size 8 for training, validation and test data with -use_warpctc option. Except these I am using default options.
Approximately how much time my training should take.
I am using 1 GeForce GTX 1080 GPU.

lissyx · January 11, 2018, 11:22am

How big is your overall dataset? TED-LIUM2 + CommonVoice ? That would be ~1000 hours of audio ?

rao_jjn · January 11, 2018, 12:15pm

It should be between 300 - 500 hours. I have used valid sets only

lissyx · January 11, 2018, 1:28pm

I think that ballpark for ~500 hours on our cluster with 16 TITAN X GPUs is around 2h for one epoch, so 12 hours for one epoch on your 1080 seems consistent.

rao_jjn · January 12, 2018, 7:22am

What total training time I can expect ? Any idea ?

rao_jjn · January 15, 2018, 5:33am

Hello,

Can you provide some hints about the total training time in my case ? Thanks a lot.

lissyx · January 15, 2018, 8:21am

I think I just did …

rao_jjn · January 16, 2018, 7:34am

Does it mean that on my hardware, to complete 75 epochs it will take ~38 days ?

Tilman_Kamp · January 16, 2018, 10:12am

Yes. Running one full epoch will give you a better estimate.

adryan · February 5, 2018, 11:51am

How much time an epoch will last on TED-LIUM2 if using newly NVIDIA Titan V ? Or what would be the speed up over Titan X Pascal? Will the tensor cores help ?