Hi,
I’m trying to infering deepspeech model on google’s colaboratory as I don’t own a GPU. Currently, due to the GPU limitations the service can be used only for 2 hours within the day after that it gets exhausted. I’m using the epoch as ‘100’, how I can run the model so it gets save after every 10 epochs. For example, if today i’m executing the model at epoch value of 100 and GPU gets exhausted after 10 epochs so in next run should I start the model with epoch value as 90??? what would be the best way to do it?? Please, help me out in this.
You want to use checkpoints, see the documentation on how to configure checkpoints interval and path.
https://deepspeech.readthedocs.io/en/r0.9/TRAINING.html#checkpointing
1 Like