When I’m training and using the early stopping, is the output pb model generated from the epoch with best validation result or from the last one?
I noticed that in the release notes, you state “The weights with the best validation loss were selected at the end of the 30 epochs.”.
Is this done automatically by the training script or do you store each checkpoint during the training and pick it manually for the released model somehow?
Hi, can someone from the team please react to this?
If there’s a relevant documentation or publicly accessible code that does the best model selection it would be very helpful as I assume almost everyone who tries to train a model is interested in the best trained model from the run.
I have a patch here that keeps the best validation loss checkpoint during training: https://gist.github.com/reuben/dcc2deaf85568591e34ce363bc3bac2a (previously discussed and shared in this issue: https://github.com/mozilla/DeepSpeech/issues/1475)
It hasn’t been merged because it’s super hacky, TensorFlow’s MonitoredTrainingSession makes it hard to do it cleanly.
Thanks for the link, is this the solution that you use internally to generate “The weights with the best validation loss were selected at the end of the 30 epochs.” ?
Yes, this is the solution we use internally.
ok, thanks for the clarification
suppose my training parameter epoch is 150, I stopped training at 100 using ctrl+c and then exported the model. The exported model is the best validation loss or the last checkpoint saved one??? @lissyx
Another one, i need a clarification. I trained with 30 Hrs of data, the model is overfitted. Does the Overfitted one gives best validation loss . . .
or the “Training and Dev” intercepting point gives the Best validation loss.Please don’t hijack old threads. Delete this post and start a new one.
