Including punctuation (and capitalization?) in the training text / language model?

Hey @lissyx you wrote :

We’ve been chatting about this a bit over here at Te Hiku, contemplating giving it a go here. Biggest question, of course, is how to handle the kenlm/language model side of things? I am thinking you probably kept everything lower case and just added “?”, “.”, “!” etc as tokens in their own right? Is that right?

Or did you try to include capitalization as well? I know that character level language models work well in the RNN/LSTM space for text generation - has anyone thought of trying to integrate such models into DeepSpeech? Maybe even an encoder/decoder based language model?

Note for others: Per documentation it is currently recommended to lower case and remove punctuation from training text and then perhaps use a different technique to ‘add it back in’, after DeepSpeech, based on context…

//cc @tippy_top

1 Like

I’m sorry, but this was like a 5 minutes tests, and I have no memory of how I did it.

1 Like

OK cool no worries. Thanks though. If we try it we’ll report back.