I was wondering how to the use the best checkpoint model on new dataset [let’s say 5 hours of new data] quickly without doing a full epoch training. Freezing inner layers and just using FC doesn’t help in learning. Is there any other way - just like Kaldi adapts
In that case, you’re probably stuck with fine tuning the whole network. An epoch should be pretty fast on just 5 hours of data though.
My model has been trained on 2000hours of data. If I add more 30 hrs and run epoch from checkpoint , the loss shoots up which is obvious. Is there any intuitive paper on how to fine tune the whole network?
@reuben with any sort of fine tuning like changing lr , the model overfits to the new dataset while performing worse on the overall test set . Any suggestions?