Have some problem with evaluate.py

I want to see the performance without lm and using greedy decoding. So here is the test of pretrained 0.6.1.

My command
./evaluate.py --n_hidden 2048 --checkpoint_dir ./deepspeech-0.6.1-checkpoint --test_files ../../librispeech/librivox-test-clean.csv --lm --beam_width 1 --n_steps 1

Error
[scorer.cpp:77] FATAL: “(access(filename, 4)) == (0)” check failed. Invalid language model path

Also, evaluate.py doesn’t show the overall WER. It just showing the WER for each utterance (see the img)


Edit: It shows in previous instead of in the end. This is my own screen issue.

Any advice would be appreciated!

You gave no LM filename, --lm expects one.

Your statement is unclear and you don’t show the full console output. Also, please share as text and not as image.
The code in evaluate.py is the same that is run for test set when training. It prints general WER on dataset and then shows 10 worst examples.

Unless you can prove bad WER with the LM, this is not a bug, this is expected.

The code is here: DeepSpeech/evaluate.py at v0.6.1 · mozilla/DeepSpeech · GitHub

So sorry, the latter seems to be my own issue. Will edit that out later. About the LM, I don’t need any language model when decoding, what filename should I give?

Decoding without a LM will always produce poor performances.

Also, --lm is not in 0.6.1: https://github.com/mozilla/DeepSpeech/blob/v0.6.1/util/flags.py

I know it, and this is what I want to test. Could you tell me how to set it please? Thanks!

Hi, @lissyx

this is the new command
./evaluate.py --n_hidden 2048 --checkpoint_dir ./deepspeech-0.6.1-models --test_files ../../librispeech/librivox-test-clean.csv --lm_binary_path --lm_trie_path --beam_width 1
still get same error

You have to specify empty strings: --lm_binary_path "" --lm_trie_path ""

1 Like

It works. Much appreciate!