Error at the stage of test epoch when replaced dataset

Dear supporting friends.

Test epoch | Steps: 23 | Elapsed Time: 0:00:06
Test epoch | Steps: 24 | Elapsed Time: 0:00:07
Test epoch | Steps: 25 | Elapsed Time: 0:00:07
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Not enough time for target transition sequence (required: 95, available: 76)10You can turn this error into a warning by using the flag ignore_longer_outputs_than_inputs
	 [[{{node CTCLoss}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "DeepSpeech.py", line 976, in <module>
    absl.app.run(main)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 953, in main
    test()
  File "DeepSpeech.py", line 686, in test
    samples = evaluate(FLAGS.test_files.split(','), create_model, try_loading)
  File "/run/media/user/zukhruf_anas_ssd/acoustic_lab/speech_lab/DeepSpeech/evaluate.py", line 155, in evaluate
    samples.extend(run_test(init_op, dataset=csv))
  File "/run/media/user/zukhruf_anas_ssd/acoustic_lab/speech_lab/DeepSpeech/evaluate.py", line 116, in run_test
    session.run([batch_wav_filename, transposed, loss, batch_x_len, batch_y])
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Not enough time for target transition sequence (required: 95, available: 76)10You can turn this error into a warning by using the flag ignore_longer_outputs_than_inputs
	 [[node CTCLoss (defined at /home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'CTCLoss':
  File "DeepSpeech.py", line 976, in <module>
    absl.app.run(main)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "DeepSpeech.py", line 953, in main
    test()
  File "DeepSpeech.py", line 686, in test
    samples = evaluate(FLAGS.test_files.split(','), create_model, try_loading)
  File "/run/media/user/zukhruf_anas_ssd/acoustic_lab/speech_lab/DeepSpeech/evaluate.py", line 73, in evaluate
    sequence_length=batch_x_len)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/ops/ctc_ops.py", line 176, in ctc_loss
    ignore_longer_outputs_than_inputs=ignore_longer_outputs_than_inputs)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_ctc_ops.py", line 336, in ctc_loss
    name=name)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

I am using
deepspeech 0.6.1.
Tensorflow 1.14 gpu.
batch_size is 12.

I have earlier no issue and successful runs. Since I have replaced the old common-voice data with new common-voice released, This is the error at the stage of test epoch and process killing.

Due to long words, I have added
at Total_loss = ignore_longer_outputs_than_inputs=True

which worked fairly well with old dataset.
Is this due to anything I am making mistake / negligence? or due to any other reason.
If you can hint me.

1 Like

I remember there are 2 places to change that at, check DeepSpeech.py at the given lines and run just a test set, not train/dev to save time.

In general, don’t use this for real training, it will lead to bad results.

1 Like

OK. Thank you. !
I removed this from total_loss and still the error is same.
by the way, where is the second place to add this comment? one is line 235 where I tried to add. and now removed. Still same.

Pleeeeeeeaaaaaase learn how to use the search function, this is not the first, second, third time this error occurs …

1 Like

This must be a broken file. As i assume. ? OK. Because now I have just removed the test folder path to previous and it is well behaving until now.
But, thanks to you. i learnt that I can go with only a test run without train and dev.

total_loss = tfv1.nn.ctc_loss(labels=batch_y, inputs=logits, sequence_length=batch_seq_len,ignore_longer_outputs_than_inputs=True)

On DeepSpeech.py and evaluate.py? Better to find the short files, try using a sort on the csv files and listen to the short ones?

2 Likes