Best Validation Loss Calculation for Overfitting Training

My training parameter epoch is 150, I stopped training at 100 using ctrl+c and then exported the model. Is the exported model returns the best validation loss or the last checkpoint saved one??? @lissyx

Another one, i need a clarification. I trained with 30 Hrs of data, the model is overfitted. Does the Overfitted one gives best validation loss . . .

Don’t tag specific individuals in general question threads, it’s rude. By default the exported model is the one with the best validation loss. As for your second question it’s impossible for anyone but yourself to answer.

I don’t know, if the model is overfit, how it export the best validation loss.

If the model is identified as overfitted one using tensorboard graph. Can i proceed to stop training using ctrl+c. Because in docs, they said best validation loss is calculated at the end of the 200 epochs.

Because i trained my data using deepspeech 0.9.2 checkpoint. First I created the base model and then done augmentation.

When I completed the base model, I used the base model checkpoint and done the augmentation(volume) with learning rate 0.0001. It starts with overfitting. . .

Can I augment a single parameter such as volume alone , pitch, tempo, framerate alone for training . or i include the multiple parameter as training

Fine tuning for 150 epochs with 30 hours and a high learning rate doesn’t sound like a winning strategy. Maybe 30 hours is not enough to get good results? And check the docs for the augmentation.

@kreid, as augmentation is quite new it might not be at the top of your list, but there are more and more questions on how to use it or what realistic parameters are.

1 Like

If the model is identified as overfitted one using tensorboard graph. Can i proceed to stop training using ctrl+c. Because in docs, they said best validation loss is calculated at the end of the 200 epochs.

Search this forum here for fine tuning and transfer learning. You are referring to training 2000 hours of material, this is not what you are doing.

And set the flags to keep many checkpoints. That way you can test different ones in a longer training.

Dev loss is a good indicator, but in Speech you should have a good test set and try that with different checkpoints around the overfitting checkpoint. Sometimes a couple more epochs make the training better, sometimes not. Not an exact science :frowning:

@reuben can you be more specific

Can any one reply this

“If the model is identified as overfitted one using tensorboard graph. Can i proceed to stop training using ctrl+c”

“Can I export the Overfitted Model”

Please read my comment, it looks like you didn’t read or didn’t understand it. Read the docs and come up with some questions that show that you made an effort to understand what is going on.

how did u use the tensorboard to generate this loss graphic?