DeepSpeech Version: v0.9.3
pre-trained modell Version: v0.9.3
OS System: Ubuntu 20.04
Env: Container by nvidia. (tensorlow:20.11-tf1-py3) with tf version is 1.15.4
p.s. this container suitable for training a model without checkpoint.
G-card: Nvidia 3090
I already set the env var TF_FORCE_GPU_ALLOW_GROWTH=1 and TF_CUDNN_RESET_RND_GEN_STATE=1
Command line:
python3 DeepSpeech.py --train_files /workspace/de/clips/train.csv \
–dev_files /workspace/de/clips/dev.csv \
–train_batch_size=56 \
–dev_batch_size=56
–epochs=200
–dropout_rate 0.4
–learning_rate 0.0001
–load_checkpoint_dir /workspace/DeepSpeech/checkpoint/deepspeech-0.9.3-checkpoint
–save_checkpoint_dir /workspace/DeepSpeech/checkpoint/deepspeech-0.9.3-checkpoint
–drop_source_layers 1
–export_dir /workspace/DeepSpeech/data/model
–export_file_name output_706
–export_author_id sun
–export_model_name 706
–export_model_version 1
–summary_dir /workspace/DeepSpeech/data/summaries
–n_hidden 2048
–early_stop True
–es_epochs 10
–es_min_delta 0.05
–reduce_lr_on_plateau True
–plateau_epochs 5
–plateau_reduction 0.1
–alphabet_config_path /workspace/DeepSpeech/data/alphabet.txt
–train_cudnn\
Process and Error logs:
I Loading best validating checkpoint from /workspace/DeepSpeech/checkpoint/deepspeech-0.9.3-checkpoint/best_dev-1466475
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: learning_rate
I Initializing variable: layer_6/bias
I Initializing variable: layer_6/bias/Adam
I Initializing variable: layer_6/bias/Adam_1
I Initializing variable: layer_6/weights
I Initializing variable: layer_6/weights/Adam
I Initializing variable: layer_6/weights/Adam_1
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:15:15 | Steps: 4235 | Loss: 136.162398
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /worEpoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 35.049599 | Dataset: /woEpoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 4 | Loss: 36.623719 | Dataset: /woEpoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 7 | Loss: 38.231940 | Dataset: /woEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 9 | Loss: 39.575112 | Dataset: /woEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 11 | Loss: 39.760513 | Dataset: /wEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 13 | Loss: 40.870941 | Dataset: /wEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 15 | Loss: 41.908309 | Dataset: /wEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 17 | Loss: 42.542558 | Dataset: /wEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 19 | Loss: 43.154726 | Dataset: /wEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 21 | Loss: 43.870366 | Dataset: /wEpoch 0 | Validation | Elapsed Time: 0:00:01 | Steps: 23 | Loss: 44.380074 | Dataset: /workspace/de/clips/dev.csvTraceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1365, in _do_call
return fn(*args)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1350, in _run_fn
target_list, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 2048, 2048, 1, 175, 56, 2048]
[[{{node tower_0/cudnn_lstm/CudnnRNNV3}}]]
[[tower_0/Where/_171]]
(1) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 2048, 2048, 1, 175, 56, 2048]
[[{{node tower_0/cudnn_lstm/CudnnRNNV3}}]]
0 successful operations.
0 derived errors ignored.During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 982, in run_script
absl.app.run(main)
File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 303, in run
_run_main(main, args)
File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 251, in _run_main
sys.exit(main(argv))
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 954, in main
train()
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 617, in train
set_loss, steps = run_set(‘dev’, epoch, init_op, dataset=source)
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 572, in run_set
feed_dict=feed_dict)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1359, in _do_run
run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 2048, 2048, 1, 175, 56, 2048]
[[node tower_0/cudnn_lstm/CudnnRNNV3 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[tower_0/Where/_171]]
(1) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 2048, 2048, 1, 175, 56, 2048]
[[node tower_0/cudnn_lstm/CudnnRNNV3 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.Original stack trace for ‘tower_0/cudnn_lstm/CudnnRNNV3’:
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 982, in run_script
absl.app.run(main)
File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 303, in run
_run_main(main, args)
File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 251, in _run_main
sys.exit(main(argv))
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 954, in main
train()
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 484, in train
gradients, loss, non_finite_files = get_tower_results(iterator, optimizer, dropout_rates)
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 317, in get_tower_results
avg_loss, non_finite_files = calculate_mean_edit_distance_and_loss(iterator, dropout_rates, reuse=i > 0)
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 244, in calculate_mean_edit_distance_and_loss
logits, _ = create_model(batch_x, batch_seq_len, dropout, reuse=reuse, rnn_impl=rnn_impl)
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 195, in create_model
output, output_state = rnn_impl(layer_3, seq_length, previous_state, reuse)
File “/workspace/DeepSpeech/deepspeech_training/train.py”, line 133, in rnn_impl_cudnn_rnn
sequence_lengths=seq_length)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/layers/base.py”, line 548, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 854, in call
outputs = call_fn(cast_inputs, *args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py”, line 234, in wrapper
return converted_call(f, options, args, kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py”, line 439, in converted_call
return _call_unconverted(f, args, kwargs, options)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py”, line 330, in _call_unconverted
return f(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/cudnn_rnn/python/layers/cudnn_rnn.py”, line 440, in call
training)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/cudnn_rnn/python/layers/cudnn_rnn.py”, line 518, in _forward
seed=self._seed)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py”, line 1132, in _cudnn_rnn
outputs, output_h, output_c, _, _ = gen_cudnn_rnn_ops.cudnn_rnnv3(**args)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_cudnn_rnn_ops.py”, line 2051, in cudnn_rnnv3
time_major=time_major, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py”, line 794, in _apply_op_helper
op_def=op_def)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py”, line 513, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3357, in create_op
attrs, op_def, compute_device)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3426, in _create_op_internal
op_def=op_def)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 1748, in init
self._traceback = tf_stack.extract_stack()
Has someone ever met this Error and know how to solve it?
Thx a lot!