Unfreeze only the last FC layer abd RNN layer for transfer learning

Archives DeepSpeech
1

As I understand, deepspeech 0.4.1 version in 3 FC layer, 1 RNN layer followed by one more FC layer and then the CTC loss is calculated.

I am using the following command over the pretrained model:

python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir /home/sayantan/Desktop/ai_learning/deepspeech_check/deepspeech-0.4.1-checkpoint --export_dir /home/sayantan/Desktop/ai_learning/deepspeech_check/trained_model/ --validation_step 1  --train_batch_size 32 --dev_batch_size 8 --test_batch_size 8 --epoch -3 --train_files my-train.csv --dev_files my-dev.csv --test_files my_test.csv --learning_rate 0.0001

However, I’m planning on training only on the last FC layer and the RNN layer after unfreezing them for a few epoch and then train on the complete layers for the next few epochs.

Is there a possibility to do the same, i.e. to unfreeze the last layers and then the remaining layers?

If so, how could it be done? Thank You.

@tuttlebr @kdavis @lissyx can you suggest?

2 
python3 DeepSpeech.py \
--n_hidden 2048 \
--checkpoint_dir /home/sayantan/Desktop/ai_learning/deepspeech_check/deepspeech-0.4.1-checkpoint \
--export_dir /home/sayantan/Desktop/ai_learning/deepspeech_check/trained_model/  \ 
--validation_step 1  \
--train_batch_size 32 \
--dev_batch_size 8 \
--test_batch_size 8 \
--epoch -3 \
--train_files my-train.csv \
--dev_files my-dev.csv \
--test_files my_test.csv \
--learning_rate 0.0001

Assuming the data feeding your model is mono, 16-bit PCM @ 16kHZ and represents appropriate audio/transcription pairs. Up-sampling may cause issues.

If you run this, it should create three files / epoch and three logging files in your checkpoint_dir directory.
at each epoch:
-model.ckpt-#####.data-#####-of-#####
-model.ckpt-#####.index
-model.ckpt-#####.meta

for logging:
-checkpoint
-events.out.tfevents…
-graph.pbtxt

after you run your 3 epochs, you’ll see that the checkpoint file lists all the unfrozen graph file names you can use to restart training. use the one where loss was lowest. You should then be able to run your script again, with a lower learning rate, to fine tune your most recent checkpoint. (That worked for me anyway)

I believe there is a warning for it but make sure you empty the folder of checkpoint files you don’t want to use.

1 Like
3

Thanks @tuttlebr for sharing your tips.

4

Thanks a lot @tuttlebr for your detailed reply. Will take this up.