Coredump when I use my trained model

I have trained deepspeech 0.6.0a and it produced the output_graph.pb.
During generation I see this warning in the log:
WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f09320becf8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. … : AttributeError: module ‘gast’ has no attribute ‘Num’

When I try to use the downloaded client get this error:

deepspeech --model /home/rik/deepspeech-cleaner/languages/it/training/standard/model_export_1024/output_graph.pbmm --alphabet /home/rik/deepspeech-cleaner/languages/it/training/standard/alphabet.txt --audio /home/rik/prova2.wav

TensorFlow: v1.14.0-18-g351a98a
DeepSpeech: v0.6.0-alpha.10-2-g469ddd2
2019-10-20 22:43:32.623466: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-20 22:43:32.625721: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-10-20 22:43:32.651985: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-20 22:43:32.652511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2070 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.785
pciBusID: 0000:08:00.0
2019-10-20 22:43:32.652518: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-10-20 22:43:32.652535: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-20 22:43:32.652925: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-20 22:43:32.653304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-10-20 22:43:33.072711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-20 22:43:33.072731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-10-20 22:43:33.072734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-10-20 22:43:33.072805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-20 22:43:33.073190: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-20 22:43:33.073554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-20 22:43:33.073905: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2019-10-20 22:43:33.073922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7123 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:08:00.0, compute capability: 7.5)
2019-10-20 22:43:33.083423: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] model_pruner failed: Invalid argument: Invalid input graph.
Unable to fetch metadata: Invalid argument: Tensor metadata_feature_win_len:0, specified in either feed_devices or fetch_devices was not found in the Graph
Could not create model.

what am I doing wrong?

Your graph is not compatible.

Thanks Lissix,
I am not an expert in the field, I installed cuda 10.0 and followed the instructions of the deepspeech site and after several attempts I was able to train a model.
So I thought I had all the right versions of the software (ubuntu 18, cuda 10.0, tensorflow-gpu 14.0, deepspeech 0.6.0a).

Now you tell me that graph is not compatible, a nightmare :slight_smile:

Can you tell me which software graph belongs to?
tensorflow/core/grappler/optimizers/meta_optimizer.cc
seems to belong to tensorflow, is it possible that a system trained with a tensoflow installation fails to use a component of it?

In any case, thanks for your help

We can’t help if you don’t provide exact versions. As I said, you are hitting this because you trained with a version of the code that is incompatible with your runtime.

And what is your git head for training ? Likely before v0.6.0-alpha.10.