Error when try to train chinese model use --utf-8 flag

DeepSpeech
#1

tag: 0.8.2

####train command

python3 DeepSpeech.py --train_files data/zh_cn/test_02/train.csv \
--dev_files data/zh_cn/test_02/train.csv \
--test_files data/zh_cn/test_02/train.csv \
--export_tflite \
--export_dir export/test_02/ \
--checkpoint_dir /Users/ruiliping/.local/share/deepspeech/test_02 \
--train_batch_size 1 --test_batch_size 1 --n_hidden 100 --epochs 60 \
--utf8

get error

I Loading best validating checkpoint from /Users/ruiliping/.local/share/deepspeech/test_02/best_dev-60
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam_1
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/bias/Adam
I Loading variable from checkpoint: layer_6/bias/Adam_1
I Loading variable from checkpoint: layer_6/weights
I Loading variable from checkpoint: layer_6/weights/Adam
I Loading variable from checkpoint: layer_6/weights/Adam_1
I Loading variable from checkpoint: learning_rate
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000                                                                                              Traceback (most recent call last):
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[{{node tower_0/IteratorGetNext}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 570, in run_set
    feed_dict=feed_dict)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[node tower_0/IteratorGetNext (defined at /Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'tower_0/IteratorGetNext':
  File "DeepSpeech.py", line 12, in <module>
    ds_train.run_script()
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 976, in run_script
    absl.app.run(main)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 948, in main
    train()
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 483, in train
    gradients, loss, non_finite_files = get_tower_results(iterator, optimizer, dropout_rates)
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 316, in get_tower_results
    avg_loss, non_finite_files = calculate_mean_edit_distance_and_loss(iterator, dropout_rates, reuse=i > 0)
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 235, in calculate_mean_edit_distance_and_loss
    batch_filenames, (batch_x, batch_seq_len), batch_y = iterator.get_next()
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
    name=name)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "DeepSpeech.py", line 12, in <module>
    ds_train.run_script()
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 976, in run_script
    absl.app.run(main)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 948, in main
    train()
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 605, in train
    train_loss, _ = run_set('train', epoch, train_init_op)
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/train.py", line 573, in run_set
    exception_box.raise_if_set()
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/util/helpers.py", line 123, in raise_if_set
    raise exception  # pylint: disable = raising-bad-type
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/util/helpers.py", line 131, in do_iterate
    yield from iterable()
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/util/feeding.py", line 118, in generate_values
    transcript = text_to_char_array(sample.transcript, Config.alphabet, context=sample.sample_id)
  File "/Users/ruiliping/program/DeepSpeech/training/deepspeech_training/util/text.py", line 12, in text_to_char_array
    if not alphabet.CanEncode(transcript):
  File "/Users/ruiliping/program/DeepSpeech/venv/lib/python3.7/site-packages/ds_ctcdecoder/swigwrapper.py", line 612, in CanEncode
    return _swigwrapper.UTF8Alphabet_CanEncode(self, string)
TypeError: in method 'UTF8Alphabet_CanEncode', argument 2 of type 'std::string const &'

now I get the type error, so I try to fix it on line 13 in test.py:

#FIXME
transcript = transcript.encode('utf-8')
if not alphabet.CanEncode(transcript):

then, when I train again, i got a model named ‘output_graph.tflite’

but I got an error again when transcribe model

command

deepspeech --model export/test_02/output_graph.tflite --audio data/zh_cn/test_02/common_voice_zh-CN_20712534.wav

get error

Loading model from file export/test_02/output_graph.tflite
TensorFlow: v2.2.0-24-g1c1b2b9dd9
DeepSpeech: v0.8.2-0-g02e4c762
Loaded model in 0.00112s.
Running inference.
Traceback (most recent call last):
  File "/usr/local/bin/deepspeech", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/deepspeech/client.py", line 156, in main
    print(ds.stt(audio))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-5: surrogates not allowed

sos…

#2

Do you really need this? Do you understand what this flag does?

Typical symptom of a broken transcription file, likely too short

Unrelated, maybe a symptom of broken --utf8 flag?

This makes no sense given the meaning of --utf8 and/or it advocates for the fact that your dataset is broken. Please fix your dataset / importer code and do not hack like that. Also, we have no test.py file, so I don’t know what you are fixing here.

Fallout of the previous things …

1 Like
#4

I get it, maybe. :rofl:thanks, I only fllow the document, I will remove the utf-8 flag, and create a big alphabet file

#5

What document?

Using the flag is fine if you understand (you are working on Chinese, and this is designed for those cases) what it does and its requirements. I am not convinced yours errors are related to that, except the TypeError.

Please fix your dataset first.

#6

So indeed this is training/deepspeech_training/util/text.py and not test.py.

I know @reuben still has some pending fixes for Chinese model. I’m unsure this is supposed to work properly on 0.8.2. Can you @linggao work out of current master / 0.9.0a9 tag?

#7

I made a mistake, it is text.py certaintly.I will try master/0.9.0a9 tag tomorrow, and a new dataset is a good idea. Thanks for your reply.

#8

@linggao Did working out of the current master version address your problem?