Any resources on beam search decoding using NN based langauge models instead of n-gram based language models?
We currently don’t use a neural network based language model, but there should be lots of literature on neural network language models out there, e.g. Exploring the Limits of Language Modeling.
If you integrate a neural network based language model, we’d love to hear back about you results.
In our team, the current plan is to build a generative text model using an RNN and feed the output into Ken-LM. We haven’t finished this yet (just starting, in fact), but we’re optimistic that it’ll help in our case where there isn’t much text available.
Hi @mathematiguy
Can u explain feed the output into Ken-LM?
I am planning to get sentence probabilities from a RNNLM and use it instaed of probabilities fron ngram based KenLM model. Can u suggest if thats the way?
You train Ken-LM from a corpus of text files. Our plan is to train an RNN to generate lots of text and then save the text as a bunch of text files (to inflate our corpus).
Then we can use the generated corpus to train Ken-LM. The result is still Ken-LM, but the idea is that the RNN will help fill in the gaps in the language model, so there are fewer situations when the language model comes across a sequence that it’s unfamiliar with.