Upgrading pre-trained model to version 1

DeepSpeech
#1

Here’s what the readme says about converting a model:

Model compatibility

DeepSpeech models are versioned to keep you from trying to use an incompatible graph with a newer client after a breaking change was made to the code. If you get an error saying your model file version is too old for the client, you should either upgrade to a newer model release, re-export your model from the checkpoint using a newer version of the code, or downgrade your client if you need to use the old model and can’t re-export it.

But what is the exact process of re-exporting from the checkpoint? The documentation on training seems to assume I am training the checkpoint with additional data.

It wasn’t clear exactly which arguments are needed to re-export the model - can I do so with only --checkpoint_dir and --export_dir?

When I try to do so from the 0.4.1 model I get the following error:

Key layer_1/bias not found in checkpoint

In another thread this issue was solved by checking out from master but I am already using master.

1 Like
#2

After further digging into DeepSpeech.py, it seems like this may be something that’s actually not possible to do. Filed an issue here: https://github.com/mozilla/DeepSpeech/issues/2050

#3

The process is on a case-by-case basis, because it depends on how backwards compatibility was broken. In this case, we had two breaking changes:

  1. Variable names were changed
  2. Audio feature computation changed

The first one can be fixed by passing a custom mapping to tf.train.Saver or loading the checkpoint manually using tf.train.load_checkpoint. The second is more complicated, because it means the old checkpoint is not very useful with the new code. I guess you could try to use it to initialize weights in a new training run.

Here’s some code that does the conversion if you’re interested in experimenting with the old checkpoint: https://github.com/reuben/DeepSpeech/commits/upgrade-v0.4.1

#4

Thanks Reuben. I eventually managed to get it to update the model format but I got no useful output from it.

So I gave up at that point and am just using LDC93S1 for development purposes.

#5

Hi Reuben,

I can export the model from 0.4.1 checkpoint with alpha4, but fail with alpha5.

Can I assume the model is still compatible until alpha4?

Or, has the model become incompatible even before alpha4 (but the code to check model version hasn’t checked in so it just doesn’t complain)?

Thanks!

Add version info to exported graphs

@reuben|20x20

reuben committed 16 days ago

#6

This is the last commit that should work with the v0.4.1 checkpoint without any changes: https://github.com/mozilla/DeepSpeech/commit/bd7358d94e17e61f841d0507f0003b732b69d226

1 Like