Running Evaluation on Model with Custom Kernel

DeepSpeech
#1

Hello,

Is there an existing flow for running just an evaluation dataset and getting mean_edit_distance/WER from the frozen_graph provided?

I am attempting to replace the LSTMBlockFusedCell operation with my own custom LSTMCell implementation for an experiment. I would like to not have to implement a gradient calculation for my custom kernel since I am using the pre-trained model provided and only care about the evaluation WER. I am currently using --initialize_from_frozen_model and some string manipulation to shove in the pre-trained weights into my custom kernel.

However, the in-memory graph still contains all of the training nodes / the optimizer, which is causing errors when the gradients are being calculated for my custom kernel. Ideally I could run my experiment by using tensorflow’s graph_editor to insert my custom kernel without having any additional gradient calculations.

Command I am currently running:

python DeepSpeech.py --notrain --test --initialize_from_frozen_model=models/output_graph.pb -display_step=1 --epoch 1 --test_files=~/test.csv --dev_files=~/dev.csv --train_files=~/train.csv --limit_test=5 --log_level 0

#2

We have DeepSpeech.py flags to neutralize training (--notrain as much as I remember and as you exposed), so you can likely play. We also have evaluate.py, and the kind of LSTM change that you want to do is already what we do in https://github.com/mozilla/DeepSpeech/commit/57da51659cebfb98fba5d82b5c3674ea51a1e2cc#diff-79c8372f97f7f5a6d8660bd265f836d7R452 so you can try the same kind of workaround with fixup :slight_smile: