Questions on fine-tuning 0.6.1 model

[Aleksei_Smoliarchuk]

Hello, guys!
First, many thanks for doing great job for the community. I have several questions for you to ask:

1)Which layers should be frozen and which ones should be trained when I want to do transfer learning on non-native telephone English data set, what are the recommended hyperparameters in that case and the number of hours required in a dataset to infer with WER close to original model?

2. How do we teach DeepSpeech to split utterances by sentences having proper punctuation?

3. How can I convert my train set into a format feedable to the model?

Thanks for your further answers.

[nmstoker (Neil Stoker)]

Hi @Aleksei_Smoliarchuk - I’ll have to leave the other questions but for Q2, there was some prior discussion on punctuation here (which also links to an earlier discussion):

Punctuation Model

So if one were to use DeepSpeech for live transcription (like Google’s Live Transcribe, for instance), they would need to implement a punctuation model as well. Has anyone looked into how this might work? Or are there already Python libraries that do this? So you would run your DeepSpeech output through those models and hopefully get well-punctuated and more readable sentences. I know that this issue may be out of scope for the DeepSpeech project, but is relevant to the overall context in whic…

I didn’t see any further updates on how people got on. Hope that’s at least a start. I’m sure this would be really useful in a range of scenarios, so would be great if you could share any progress / findings if you get anywhere with this.

[othiele (Olaf Thiele)]

As for transfer learning, why don’t you search the forum first, there are some ideas and code.

Take as much training data as you can get in the same format as you will feed data later for inferencing. Depending on the language, transferring might not work. What language are working on?

As for converting, just look at the documentation and try a run with a small dataset.

[Aleksei_Smoliarchuk]

I am working on non-native English language recognition, skype calls, quality is far worse than the original model was trained on.

[othiele (Olaf Thiele)]

Search through the posts and you’ll find that non-native and bad audio quality are two main factors for high WERs. You can try, but don’t expect great results.

[Aleksei_Smoliarchuk]

Nice idea! Does training with augmentations really help to generalise to bad speech quality?

[othiele (Olaf Thiele)]

In theory yes, but you take the pre-trained model on clean data and you have noisy transfer data, so you would have to annotate the clean data first and then train it.

Ideally you train lots of your own data and see how that works out. Test and let us know how it went.