I’m currently experimenting with new language models with a limited vocabulary, the 10k, 20k, 30k, 40k, or 50k most common words from librispeech-lm-norm.txt.
Using this limited vocabulary should throw out rare words in librispeech-lm-norm.txt that appear only once or twice and thus address this problem. But we have to run the benchmarks to be sure.