Fatal Python error when run DeepSpeech

nthanhha26 · February 11, 2020, 8:43am

Hello, am new to DeepSpeech, I got a trained custom DS model from my friend and I am trying to make some predictions.
But I got this error when run “python DeepSpeech.py”

WARNING:tensorflow:From DeepSpeech.py:704: The name tf.nn.rnn_cell.LSTMStateTuple is deprecated. Please use tf.compat.v1.nn.rnn_cell.LSTMStateTuple instead.

W0211 11:04:11.854850 139979131139904 deprecation_wrapper.py:119] From DeepSpeech.py:704: The name tf.nn.rnn_cell.LSTMStateTuple is deprecated. Please use tf.compat.v1.nn.rnn_cell.LSTMStateTuple instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0211 11:04:12.282489 139979131139904 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0211 11:04:12.284732 139979131139904 deprecation.py:506] From /home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f4f62c5a908>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f4f62c5a908>>: AttributeError: module 'gast' has no attribute 'Num'
W0211 11:04:12.344106 139979131139904 ag_logging.py:145] Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f4f62c5a908>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f4f62c5a908>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From /home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0211 11:04:12.395726 139979131139904 deprecation.py:323] From /home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from checkpointVN/best_dev-14352
I0211 11:04:12.396571 139979131139904 saver.py:1280] Restoring parameters from checkpointVN/best_dev-14352
I Restored variables from best validation checkpoint at checkpointVN/best_dev-14352
Fatal Python error: Segmentation fault

Current thread 0x00007f4f6e633740 (most recent call first):
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429 in _call_tf_sessionrun
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341 in _run_fn
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356 in _do_call
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350 in _do_run
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173 in _run
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950 in run
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5579 in _eval_using_default_session
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 731 in eval
  File "DeepSpeech.py", line 907 in do_single_file_inference
  File "DeepSpeech.py", line 961 in main
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
  File "/home/vnc/DeepSpeech_hangtg/envs/deepspeech-train-venv/lib/python3.6/site-packages/absl/app.py", line 299 in run
  File "DeepSpeech.py", line 965 in <module>
Segmentation fault (core dumped)

I tried to debug but dont have any clue where to start. I can provide more information if needed.

lissyx · February 11, 2020, 8:44am

Obviously, you are giving more parameters than that. Please share you exact context, there’s nothing clearly actionable here and we can’t even check how you call the code to help you.

lissyx · February 11, 2020, 8:59am

You need to share more knowledge on that.

othiele · February 11, 2020, 9:27am

Are you trying to train or running inference? Looks like you are loading checkpoints which would mean you are continuing to train. @lissyx is right, post all parameters. If you just want to do inferencing, install

pip install deepspeech

in a virtual environment and put the pbmm (rather than the pb) in a dir close by. This is a lot easier than cloning the git.

And, of course, ask your friend what version and parameters (e.g. n_hidden if changed) it is trained on.

nthanhha26 · February 12, 2020, 1:57am

yes, I have another file to pass parameters, have a look at this:

def create_flags():
# Importer
# ========

f = absl.flags

f.DEFINE_string('train_files', '/media/clips/train.csv', 'comma separated list of files specifying the dataset used for training. Multiple files will get merged. If empty, training will not be run.')
f.DEFINE_string('dev_files', '/media/clips/dev.csv', 'comma separated list of files specifying the dataset used for validation. Multiple files will get merged. If empty, validation will not be run.')
f.DEFINE_string('test_files', '/media/clips/test.csv', 'comma separated list of files specifying the dataset used for testing. Multiple files will get merged. If empty, the model will not be tested.')

f.DEFINE_string('feature_cache', '', 'path where cached features extracted from --train_files will be saved. If empty, caching will be done in memory and no files will be written.')

f.DEFINE_integer('feature_win_len', 32, 'feature extraction audio window length in milliseconds')
f.DEFINE_integer('feature_win_step', 20, 'feature extraction window step length in milliseconds')
f.DEFINE_integer('audio_sample_rate', 16000, 'sample rate value expected by model')

# Data Augmentation
# ================

f.DEFINE_float('data_aug_features_additive', 0, 'std of the Gaussian additive noise')
f.DEFINE_float('data_aug_features_multiplicative', 0, 'std of normal distribution around 1 for multiplicative noise')

f.DEFINE_float('augmentation_spec_dropout_keeprate', 1, 'keep rate of dropout augmentation on spectrogram (if 1, no dropout will be performed on spectrogram)')

f.DEFINE_boolean('augmentation_freq_and_time_masking', False, 'whether to use frequency and time masking augmentation')
f.DEFINE_integer('augmentation_freq_and_time_masking_freq_mask_range', 5, 'max range of masks in the frequency domain when performing freqtime-mask augmentation')
f.DEFINE_integer('augmentation_freq_and_time_masking_number_freq_masks', 3, 'number of masks in the frequency domain when performing freqtime-mask augmentation')
f.DEFINE_integer('augmentation_freq_and_time_masking_time_mask_range', 2, 'max range of masks in the time domain when performing freqtime-mask augmentation')
f.DEFINE_integer('augmentation_freq_and_time_masking_number_time_masks', 3, 'number of masks in the time domain when performing freqtime-mask augmentation')

f.DEFINE_float('augmentation_speed_up_std', 0, 'std for speeding-up tempo. If std is 0, this augmentation is not performed')

f.DEFINE_boolean('augmentation_pitch_and_tempo_scaling', False, 'whether to use spectrogram speed and tempo scaling')
f.DEFINE_float('augmentation_pitch_and_tempo_scaling_min_pitch', 0.95, 'min value of pitch scaling')
f.DEFINE_float('augmentation_pitch_and_tempo_scaling_max_pitch', 1.2, 'max value of pitch scaling')
f.DEFINE_float('augmentation_pitch_and_tempo_scaling_max_tempo', 1.2, 'max vlaue of tempo scaling')


# Global Constants
# ================

f.DEFINE_integer('epochs', 75, 'how many epochs (complete runs through the train files) to train for')

f.DEFINE_float('dropout_rate', 0.05, 'dropout rate for feedforward layers')
f.DEFINE_float('dropout_rate2', -1.0, 'dropout rate for layer 2 - defaults to dropout_rate')
f.DEFINE_float('dropout_rate3', -1.0, 'dropout rate for layer 3 - defaults to dropout_rate')
f.DEFINE_float('dropout_rate4', 0.0, 'dropout rate for layer 4 - defaults to 0.0')
f.DEFINE_float('dropout_rate5', 0.0, 'dropout rate for layer 5 - defaults to 0.0')
f.DEFINE_float('dropout_rate6', -1.0, 'dropout rate for layer 6 - defaults to dropout_rate')

f.DEFINE_float('relu_clip', 20.0, 'ReLU clipping value for non-recurrent layers')

# Adam optimizer(http://arxiv.org/abs/1412.6980) parameters

f.DEFINE_float('beta1', 0.9, 'beta 1 parameter of Adam optimizer')
f.DEFINE_float('beta2', 0.999, 'beta 2 parameter of Adam optimizer')
f.DEFINE_float('epsilon', 1e-8, 'epsilon parameter of Adam optimizer')
f.DEFINE_float('learning_rate', 0.001, 'learning rate of Adam optimizer')

# Batch sizes

f.DEFINE_integer('train_batch_size', 64, 'number of elements in a training batch')
f.DEFINE_integer('dev_batch_size', 64, 'number of elements in a validation batch')
f.DEFINE_integer('test_batch_size', 64, 'number of elements in a test batch')

f.DEFINE_integer('export_batch_size', 1, 'number of elements per batch on the exported graph')

# Performance

f.DEFINE_integer('inter_op_parallelism_threads', 0, 'number of inter-op parallelism threads - see tf.ConfigProto for more details. USE OF THIS FLAG IS UNSUPPORTED')
f.DEFINE_integer('intra_op_parallelism_threads', 0, 'number of intra-op parallelism threads - see tf.ConfigProto for more details. USE OF THIS FLAG IS UNSUPPORTED')
f.DEFINE_boolean('use_allow_growth', False, 'use Allow Growth flag which will allocate only required amount of GPU memory and prevent full allocation of available GPU memory')
f.DEFINE_boolean('use_cudnn_rnn', True, 'use CuDNN RNN backend for training on GPU. Note that checkpoints created with this flag can only be used with CuDNN RNN, i.e. fine tuning on a CPU device will not work')
f.DEFINE_string('cudnn_checkpoint', 'checkpointCudnn', 'path to a checkpoint created using --use_cudnn_rnn. Specifying this flag allows one to convert a CuDNN RNN checkpoint to a checkpoint capable of running on a CPU graph.')

f.DEFINE_boolean('automatic_mixed_precision', False, 'whether to allow automatic mixed precision training. USE OF THIS FLAG IS UNSUPPORTED. Checkpoints created with automatic mixed precision training will not be usable without mixed precision.')

# Sample limits

f.DEFINE_integer('limit_train', 0, 'maximum number of elements to use from train set - 0 means no limit')
f.DEFINE_integer('limit_dev', 0, 'maximum number of elements to use from validation set- 0 means no limit')
f.DEFINE_integer('limit_test', 0, 'maximum number of elements to use from test set- 0 means no limit')

# Checkpointing

f.DEFINE_string('checkpoint_dir', 'checkpoint', 'directory in which checkpoints are stored - defaults to directory "deepspeech/checkpoints" within user\'s data home specified by the XDG Base Directory Specification')
f.DEFINE_integer('checkpoint_secs', 600, 'checkpoint saving interval in seconds')
f.DEFINE_integer('max_to_keep', 5, 'number of checkpoint files to keep - default value is 5')
f.DEFINE_string('load', 'best', '"last" for loading most recent epoch checkpoint, "best" for loading best validated checkpoint, "init" for initializing a fresh model, "auto" for trying the other options in order last > best > init')

# Exporting

f.DEFINE_string('export_dir', 'model', 'directory in which exported models are stored - if omitted, the model won\'t get exported')
f.DEFINE_boolean('remove_export', False, 'whether to remove old exported models')
f.DEFINE_boolean('export_tflite', False, 'export a graph ready for TF Lite engine')
f.DEFINE_integer('n_steps', 16, 'how many timesteps to process at once by the export graph, higher values mean more latency')
f.DEFINE_string('export_language', '', 'language the model was trained on e.g. "en" or "English". Gets embedded into exported model.')
f.DEFINE_boolean('export_zip', False, 'export a TFLite model and package with LM and info.json')

# Reporting

f.DEFINE_integer('log_level', 1, 'log level for console logs - 0: INFO, 1: WARN, 2: ERROR, 3: FATAL')
f.DEFINE_boolean('show_progressbar', True, 'Show progress for training, validation and testing processes. Log level should be > 0.')

f.DEFINE_boolean('log_placement', False, 'whether to log device placement of the operators to the console')
f.DEFINE_integer('report_count', 10, 'number of phrases with lowest WER(best matching) to print out during a WER report')

f.DEFINE_string('summary_dir', 'summary', 'target directory for TensorBoard summaries - defaults to directory "deepspeech/summaries" within user\'s data home specified by the XDG Base Directory Specification')

f.DEFINE_string('test_output_file', 'summary/out', 'path to a file to save all src/decoded/distance/loss tuples generated during a test epoch')

# Geometry

f.DEFINE_integer('n_hidden', 2048, 'layer width to use when initialising layers')

# Initialization

f.DEFINE_integer('random_seed', 4568, 'default random seed that is used to initialize variables')

# Early Stopping

f.DEFINE_boolean('early_stop', False, 'enable early stopping mechanism over validation dataset. If validation is not being run, early stopping is disabled.')
f.DEFINE_integer('es_steps', 4, 'number of validations to consider for early stopping. Loss is not stored in the checkpoint so when checkpoint is revived it starts the loss calculation from start at that point')
f.DEFINE_float('es_mean_th', 0.5, 'mean threshold for loss to determine the condition if early stopping is required')
f.DEFINE_float('es_std_th', 0.5, 'standard deviation threshold for loss to determine the condition if early stopping is required')

# Decoder

f.DEFINE_boolean('utf8', False, 'enable UTF-8 mode. When this is used the model outputs UTF-8 sequences directly rather than using an alphabet mapping.')
f.DEFINE_string('alphabet_config_path', 'data/alphabet.txt', 'path to the configuration file specifying the alphabet used by the network. See the comment in data/alphabet.txt for a description of the format.')
f.DEFINE_string('lm_binary_path', 'data/lm/lm.binary', 'path to the language model binary file created with KenLM')
f.DEFINE_alias('lm', 'lm_binary_path')
f.DEFINE_string('lm_trie_path', 'data/lm/trie', 'path to the language model trie file created with native_client/generate_trie')
f.DEFINE_alias('trie', 'lm_trie_path')
f.DEFINE_integer('beam_width', 1024, 'beam width used in the CTC decoder when building candidate transcriptions')

# f.DEFINE_float('lm_alpha', 0, 'the alpha hyperparameter of the CTC decoder. Language Model weight.')
# f.DEFINE_float('lm_beta', 0, 'the beta hyperparameter of the CTC decoder. Word insertion weight.')

f.DEFINE_float('lm_alpha', 0.75, 'the alpha hyperparameter of the CTC decoder. Language Model weight.')
f.DEFINE_float('lm_beta', 1.85, 'the beta hyperparameter of the CTC decoder. Word insertion weight.')

f.DEFINE_float('cutoff_prob', 1.0, 'only consider characters until this probability mass is reached. 1.0 = disabled.')
f.DEFINE_integer('cutoff_top_n', 300, 'only process this number of characters sorted by probability mass for each time step. If bigger than alphabet size, disabled.')

# Inference mode

f.DEFINE_string('one_shot_infer', '/home/Desktop/voice_00149164.wav', 'one-shot inference mode: specify a wav file and the script will load the checkpoint and perform inference on it.')

# Register validators for paths which require a file to be specified

f.register_validator('alphabet_config_path',
                     os.path.isfile,
                     message='The file pointed to by --alphabet_config_path must exist and be readable.')

f.register_validator('one_shot_infer',
                     lambda value: not value or os.path.isfile(value),
                     message='The file pointed to by --one_shot_infer must exist and be readable.')

nthanhha26 · February 12, 2020, 2:35am

Please specify what information do you need.
This model I got is for my native language, I believe I have enough train data, source code is from DS git respo 0.6.0. Basically, I just follow the steps in training doc to create my own Kenlm language model and DS model.
My friend have finished training progress, now I just trying to run inference to test its performance

nthanhha26 · February 12, 2020, 2:04am

If you just want to do inferencing, install

pip install deepspeech

no, I will not use that, since I am trying to create a DS model for my native language

lissyx · February 12, 2020, 8:42am

So you should use evaluate.py. Now, since your friend ran the training, why are you the one performing the evaluation ? There’s already a test set evaluation at the end of the training.

I already said that: how do you call deepspeech, what command line parameters do you give ?

I don’t understand why you copy/paste our code here.

Yeah, well, obviously, from the crash you get, something is wrong in your setup. We can’t help you if you don’t share all the steps you performed exactly to figure out what is wrong.

Since the crash seems to involve absl, can you try pip uninstall absl-py && pip install --upgrade absl-py==0.22 ?

nthanhha26 · February 14, 2020, 2:17am

I finally figured it out what is wrong. I put .mp3 files directly to the DeepSpeech run file without converting into .wav 16kHz
Thanks for your attention!