Using in-tree KenLM for building LM?

[dvz]

Should the KenLM inside the DeepSpeech repo be used for building the lm files or is the official KenLM compatible as well?

[lissyx ((slow to reply) [NOT PROVIDING SUPPORT])]

Like we have it documented in data/lm/README.md ?

[dvz]

I don’t think that recipe implies where the lmplz and build_binary executables are searched for. I have used ones built from the official KenLM repo, but was wondering whether DS depends on the in-tree version.

[kdavis]

In the past I’ve used the official repo and had no problems with the so created language models.

[lissyx ((slow to reply) [NOT PROVIDING SUPPORT])]

I have used ones built from the official KenLM repo, but was wondering whether DS depends on the in-tree version.

Yeah we have no requirement here, maybe you could file an issue and/or a PR to augment the doc to make it clear? It was clear in our mind, but obviously it’d be better to state it

[javi.rahman]

Hi @kdavis @lissyx ,

I am stuck with creating a lm.binary, where should I run the python file which is in data/lm/README.md
I have saved as python file and try running and I am getting a syntax error

File “lmbinary.py”, line 21
!lmplz --order 5
^
SyntaxError: invalid syntax

I have tried running inside the kenlm/build/bin folder there only I can see lmplz and build_binary.

Please clarify me.

[nmstoker (Neil Stoker)]

In the README where it says:

following this recipe (Jupyter notebook code):

… that means it’s Jupyter notebook code and as a result you would need to run it in a Jupyter notebook (it doesn’t say to save it as a python script and run that, as you appear to have done).

The reason it’s going wrong is that “!” is a special Jupyter feature to allow you to run command line commands (which lmplz is) and that won’t work in a regular python script. Alternatively you could just use os subprocess or os run to make it work from within your script (but I’d advise the Jupyter option). Running via subprocess or run is a general python thing unrelated to anything here, so I’ll leave you to Google that if you want to go that route.

Hope that helps!

[swarajbadhei (swarajbadhei6@gmail.com)]

I am using 0.7.1 version. I could not really understand what to do for the kenlm file. It would be really helpful if someone explains.

[othiele (Olaf Thiele)]

Check that you are using 0.7.1 and not current master as it changes stuff with trie …

Then this should get you going:

Generating own scorer file

How I fixed my problem: path/to/lmplz --order 5 --temp_prefix tmp --memory 50% --text vocab.txt --arpa words.arpa --prune 0 0 1 Then: path/to/build_binary -a 255 -q 8 -v trie words.arpa lm.binary Now make sure you are in lm directory and: python generate_package.py --alphabet path/alphabet.txt --lm path/lm.binary --vocab path/vocab.txt --default_alpha 0.75 --default_beta 1.85 --package kenlm.scorer

[swarajbadhei (swarajbadhei6@gmail.com)]

it worked sir. Thanks a lot.