Incorrect results using evaluate.py

I’m using evaluate.py to get a report on a no. of test files which I’ve put in test.csv(wav_name,size,transcript).
However, when I run evaluate.py using a custom LM and trie file, the transcriptions are incorrect.
Whereas if i run these transcriptions individually, I get a (almost) correct transcription…
Can you tell me what I must be doing wrong??

Command used to run evaluate.py:

python evaluate.py --test_files ../Test_Files/Test.csv --checkpoint_dir ../original_checkpoints/ --alphabet_config_path ./data/alphabet.txt --lm_binary_path ./working/language_models/lm.binary --trie ./working/language_models/trie --test_output_file ../Output/output.txt

Example:

Original transcript: i am good and what about you.
Individual inference: i am good and what about you
evaluate.py inference: do you

How do you run the individual inference ?

deepspeech --model deepspeech-0.6.1-models/output_graph.pbmm --lm working/language_models/lm.binary --trie working/language_models/trie --audio ../Test_Files/Test1.wav

The original_checkpoints directory contains the original checkpoint files downloaded from the 0.6.1 release page. i.e pt 233784

Are you sure about your files? About the LM alpha and beta values?

I haven’t given them any value as such so i assumed they would be the default values. I passed them again along with evaluate.py and i got the same results…

Well, i’d suspect how you exported your model.

I loaded the relevant checkpoint and i thought that would be it.
evaluate.py doesn’t seem to take --model as an input parameter…

Which one ?

Produced how ?

The same point in the deepspeech-0.6.1-checkpoint directory:

  • best_dev-233784.data,
  • best_dev-233784.index,
  • best_dev-233784.meta

Used the following steps to make the lm.binary and trie:

  • build/bin/lmplz --text …/cmlm/file/comcast_updated_3.txt --arpa …/cmlm/lm/words.arpa --order 5 --temp_prefix …/tmp --prune 0 0 0 1
  • build/bin/build_binary -T -s trie …/cmlm/lm/words.arpa …/cmlm/lm/lm.binary
  • ./generate_trie data/alphabet.txt ./cmlm/lm/lm.binary ./cmlm/lm/trie

As i mentioned above, my custom lm.binary and trie file work perfectly fine with individual transcriptions…

Can you please share more logs from your evaluate.py ?

Hey so I got the problem :sweat_smile:
Just had to convert the .wav files to 16k.
This is usually not a problem when you transcribe files individually and I should’ve read the logs better!
Thank you for your help

1 Like