Non native english with transfer learning from V0.5.1 Model, right branch, method and discussion

DeepSpeech
#21

Got it ! Very helpful thanks !
And what were your machine features ? GPU ?

You need 500h because you had to transfer from english to spanish but for my accent problematic, I think I don’t need so much data as I stay in english.

meaning only the output layer ? or the output layer and the last layer ?

I just re-trained my model with CV dataset (14 epochs, --drop_source_layers 2, lr 0.00001) and the result is worse than before retraining (goes from 48% WER before to 56% after). That shows that indeed more retraining make it worse…

Next try, 1 epoch with same parameters

#22

Azure NC instance with K80 GPU, took 1 day to complete 1 epoch.

Only the output

1 Like
#23

What are the advantages of specifically using the transfer-learning2 branch? I ran the following command successfully from the master branch, to retrain the pretrained english model some differently-accented English voice data:

    (voice_to_text) mepstein@pop-os:~/voice_to_text/vendor/DeepSpeech$ python DeepSpeech.py \
        --n_hidden 2048 \
        --checkpoint_dir ../../deepspeech-0.5.1-checkpoint/ \
        --epochs 50 \
        --train_files ../../data/ds_csvs/train.csv \
        --dev_files ../../data/ds_csvs/val.csv \
        --test_files ../../data/ds_csvs/test.csv \
        --learning_rate 0.0001 \
        --train_batch_size 4 \
        --dev_batch_size 4 \
        --test_batch_size 4 \
        --es_steps 15 \
        --lm_binary_path models-0.5.1/lm.binary \
        --lm_trie_path models-0.5.1/trie \
        --export_dir ../../models_retrained_1
    """

Is the transfer learning branch for dropping or freezing certain layers, can that not be done from master? If not, is there a timetable for merging the transfer learning branch into master?

#24

Yes

@josh_meyer might be able to answer that, I don’t know how much work that is though. Do you think we should merge it? @kdavis

#25

As a user, it seems to me like it would be helpful to merge tf2 into master because it makes it less likely that over time tf2 will fall far behind master.

If there’s a set of issues you are aware of that the tf2 branch needs resolved before it can be merged into master, I’d be happy to take a shot at a PR to help get it there.

#26

@josh_meyer might know …

#27

Josh will know better, but my understanding is that there aren’t really any outstanding problems to be solved, it’s just that we didn’t want to give the impression that transfer learning is supposed to be a simple thing, just set this flag and it works. We already get people who think training DeepSpeech is supposed to be like this… But other than that, I’m not opposed to merging the code into master.

1 Like
#28

Perhaps a big warning in documentation ? I know people might still be wrong about training and transfer learning but It can help :slight_smile:

#29

Honestly, we already have a lot of things clearly documented and straightforward, and still people not reading it :slight_smile:

#30

Yeah I know but there is no other solution. I hope with time, people may read documentation before going straight into training or use

#31

I’d love to share your hope, but I don’t think it will change over time. That being said, a doc PR is always welcome and is always useful to show people “hey we documented that.”, so if you have the need it’s worth sending a PR :slight_smile:

#32

Thanks for all the comments everyone.

Re:

I don’t know whether I’d call it “simple” or not, but in my experience transfer learning is a “standard” application of a deep-learning library. It would be great to have it in master! Or rather, I’ve already done transfer learning from master (retraining from your pretrained model checkout), but it would be great to be able to freeze layers as part of my transfer learning model development / experiments, without relying on a non-master branch.

Just my two cents. Much appreciated if you decide to merge in tf2, but also, thanks regardless for all the great work on DeepSpeech already!

2 Likes