How long does is take time to train the model with Librispeech

(Branden Stark) #1

I use three GPUs (GTX 1080 , Memory 8G) to train model with librispeech ( I use the script ./bin/ ) and have for 4 days ,but it just displayed “I STARTING Optimization” , why so low? Then , I found my GPU Utilization is always 0% ,just a few moments is 100%. How to use the GPU?

(Lissyx) #2

You might need to tweak the script to change batch size to improve GPU usage?

(Branden Stark) #3

now, my GPU’s memory usage is 7797Mib( total is 8113Mib), if I increase the batch size , is it OK ? or will over memory?

(Branden Stark) #4

my batch size is the script’s default : train_batch_size = 12, dev_batch_size=12 test_batch_size=12

(Lissyx) #5

I’m not an oracle, you’ll only know if you try. It might also be that your CPUs is not fast enough to feed the GPUs ?

(Branden Stark) #6

ok,I will try it ,thank you.

(Vincent Foucault) #7

Lissyx, you’re an oracle…sometimes LOL :joy:

(Branden Stark) #8

Yeah,you are right.Thank you.