Query regarding speed of training and issues with convergence

[carlfm01]

@alchemi5t

I am training a model on my own marathi dataset but i’ve noticed that the training is faster in the initial steps of the epoch and it keeps getting slower to the end.

That’s expected:

Loss function appears to slowly climb over batches during epoch, reset every epoch

DeepSpeech uses curriculum learning, an epoch starts with easier, short sentences and ends with harder, longer ones. Thus, the loss is lowest on the easy, short examples and higher on the hard, long examples.

Generally I see that the Titan V is faster on this kind of task, you should try only training first with the V’s, the weight sync can be a bottleneck limiting the true power of the V’s, I mean the V’s waiting for the XP’s to complete the epoch to sync is not really a good thing.

Try using the nvidia optimized container for auto mixed precision training (only with both Titan V)?

Guide: Using Automatic Mixed Precision for NVIDIA Tensor Cores

Hello, this will be a quick guide to deploy Nvidia Docker container and take advantage of the Nvidia Tensor Cores without changing the code. Before starting the deployment of the optimized container should read: The requirements are almost the same as normal DeepSpeech deployment for training, we will require 3 extra things: Requirements : A GPU that contains Tensorcores, you should research if your GPU has TensorCores. The NVIDIA NGC TensorFlow 19.03 container Install docker, it d…