Generating own scorer file

How I fixed my problem:

path/to/lmplz --order 5 --temp_prefix tmp --memory 50% --text vocab.txt --arpa words.arpa --prune 0 0 1
Then:
path/to/build_binary -a 255 -q 8 -v trie words.arpa lm.binary

Now make sure you are in lm directory and:

python generate_package.py --alphabet path/alphabet.txt --lm path/lm.binary --vocab path/vocab.txt --default_alpha 0.75 --default_beta 1.85 --package kenlm.scorer

3 Likes