Train your own Chinese model, no content output for inference

DeepSpeech
#1

Training and just running inference
Python：3.6.9
OS Platform and Distribution：18.04.1-Ubuntu
TensorFlow: v2.3.0-6-g23ad988
DeepSpeech: v0.9.3-0-gf2e9c85

(deepspeech-train-venv) root@ip-10-0-1-86:DeepSpeech# python3 DeepSpeech.py --epochs 10 --alphabet_config_path data/alphabet_zh.txt --test_files /data/CV/zh/clips/test.csv --train_files /data/CV/zh/clips/train.csv
I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
I Initializing all variables.
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 47.253460

Epoch 1 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 7.184163

Epoch 2 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.898287

Epoch 3 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 5.426113

Epoch 4 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.878400

Epoch 5 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.356960

Epoch 6 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.224484

Epoch 7 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.225007

Epoch 8 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.006283

Epoch 9 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.806337

I FINISHED optimization in 0:02:14.722819
I Could not find best validating checkpoint.
I Loading most recent checkpoint from /root/.local/share/deepspeech/checkpoints/train-30
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on /data/CV/zh/clips/test.csv
Test epoch | Steps: 3 | Elapsed Time: 0:00:01
Test on /data/CV/zh/clips/test.csv - WER: 1.000000, CER: 1.000000, loss: 3.772484

Best WER:

WER: 1.000000, CER: 1.000000, loss: 4.271754

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 4.006919

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 3.038779

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “”

Median WER:

WER: 1.000000, CER: 1.000000, loss: 3.038779

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “”

Worst WER:

WER: 1.000000, CER: 1.000000, loss: 4.271754

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 4.006919

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 3.038779

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “”

(deepspeech-train-venv) root@ip-10-0-1-86:DeepSpeech# cat data/alphabet_zh.txt
猫
兔
子
你
好
(deepspeech-train-venv) root@ip-10-0-1-86:DeepSpeech#

Please help me to see what is the problem? thank you very much

1 Like
#3

You’re probably better using bytes-only mode for Chinese.

#4

use bytes_output_mode It doesn’t work either！Please advise

#5

(deepspeech-train-venv) root@ip-10-0-1-86:DeepSpeech# python3 DeepSpeech.py --epochs 5 --bytes_output_mode --train_files /data/CV/zh/clips/train.csv --test_files /data/CV/zh/clips/test.csv
I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
I Initializing all variables.
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 129.249555

Epoch 1 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 45.828649

Epoch 2 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 21.102582

Epoch 3 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 21.228093

Epoch 4 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 20.650595

I FINISHED optimization in 0:01:07.084870
I Could not find best validating checkpoint.
I Loading most recent checkpoint from /root/.local/share/deepspeech/checkpoints/train-15
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on /data/CV/zh/clips/test.csv
Test epoch | Steps: 3 | Elapsed Time: 0:00:16
Test on /data/CV/zh/clips/test.csv - WER: 1.000000, CER: 1.000000, loss: 21.577051

Best WER:

WER: 1.000000, CER: 1.000000, loss: 32.662514

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 16.510719

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 15.557926

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “”

Median WER:

WER: 1.000000, CER: 1.000000, loss: 15.557926

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “”

Worst WER:

WER: 1.000000, CER: 1.000000, loss: 32.662514

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 16.510719

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “”

WER: 1.000000, CER: 1.000000, loss: 15.557926

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “”
#6

5 epochs is very few, you should start with 20 or so.

#7

It is indeed related to the number of epochs, the default 75 times should be the value obtained after multiple training. I almost gave up, thank you very much!

(deepspeech-train-venv) root@ip-10-0-1-86:DeepSpeech# python3 DeepSpeech.py --alphabet_config_path data/alphabet_zh.txt --test_files /data/CV/zh/clips/test.csv --train_files /data/CV/zh/clips/train.csv --n_hidden 128
I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
I Initializing all variables.
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 52.149993

Epoch 1 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 15.762320

Epoch 2 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 4.754972

Epoch 3 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 4.512008

Epoch 4 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 5.370374

Epoch 5 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 5.044640

Epoch 6 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 4.709360

Epoch 7 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 4.141421

Epoch 8 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.809529

Epoch 9 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.825165

Epoch 10 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.817950

Epoch 11 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.783805

Epoch 12 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.706200

Epoch 13 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.560291

Epoch 14 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.525983

Epoch 15 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.540818

Epoch 16 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.431262

Epoch 17 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.278736

Epoch 18 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.183220

Epoch 19 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 3.031345

Epoch 20 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.969184

Epoch 21 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.771214

Epoch 22 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.617507

Epoch 23 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.519028

Epoch 24 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.418153

Epoch 25 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.177320

Epoch 26 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 2.100820

Epoch 27 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.903169

Epoch 28 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.818427

Epoch 29 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.607687

Epoch 30 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.477969

Epoch 31 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.451099

Epoch 32 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.246622

Epoch 33 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.192600

Epoch 34 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 1.091892

Epoch 35 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.994714

Epoch 36 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.913258

Epoch 37 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.856929

Epoch 38 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.793805

Epoch 39 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.744543

Epoch 40 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.712810

Epoch 41 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.709571

Epoch 42 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.661562

Epoch 43 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.586857

Epoch 44 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.585346

Epoch 45 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.527048

Epoch 46 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.501100

Epoch 47 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.438777

Epoch 48 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.453286

Epoch 49 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.409024

Epoch 50 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.413580

Epoch 51 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.423006

Epoch 52 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.391483

Epoch 53 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.383238

Epoch 54 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.356670

Epoch 55 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.361572

Epoch 56 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.338862

Epoch 57 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.358271

Epoch 58 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.344788

Epoch 59 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.348979

Epoch 60 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.344279

Epoch 61 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.317950

Epoch 62 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.361206

Epoch 63 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.327678

Epoch 64 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.351748

Epoch 65 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.330640

Epoch 66 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.319666

Epoch 67 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.320402

Epoch 68 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.330999

Epoch 69 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.316899

Epoch 70 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.322423

Epoch 71 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.337318

Epoch 72 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.319121

Epoch 73 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.465888

Epoch 74 | Training | Elapsed Time: 0:00:00 | Steps: 3 | Loss: 0.342708

I FINISHED optimization in 0:00:31.627415
I Could not find best validating checkpoint.
I Loading most recent checkpoint from /root/.local/share/deepspeech/checkpoints/train-225
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on /data/CV/zh/clips/test.csv
Test epoch | Steps: 3 | Elapsed Time: 0:00:00
Test on /data/CV/zh/clips/test.csv - WER: 0.000000, CER: 0.000000, loss: 0.352873

Best WER:

WER: 0.000000, CER: 0.000000, loss: 1.012258

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “兔子”

WER: 0.000000, CER: 0.000000, loss: 0.025613

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “你好”

WER: 0.000000, CER: 0.000000, loss: 0.020749

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “猫”

Median WER:

WER: 0.000000, CER: 0.000000, loss: 0.020749

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “猫”

Worst WER:

WER: 0.000000, CER: 0.000000, loss: 1.012258

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “兔子”

WER: 0.000000, CER: 0.000000, loss: 0.025613

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “你好”

WER: 0.000000, CER: 0.000000, loss: 0.020749

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “猫”
1 Like
#8

n_hidden can affect the results

(deepspeech-train-venv) root@ip-10-0-1-86:DeepSpeech# python3 DeepSpeech.py --alphabet_config_path data/alphabet_zh.txt --test_files /data/CV/zh/clips/test.csv --train_files /data/CV/zh/clips/train.csv
I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
I Initializing all variables.
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 47.253460

Epoch 1 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 7.184163

Epoch 2 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.898287

Epoch 3 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 5.426113

Epoch 4 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.878400

Epoch 5 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.356960

Epoch 6 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.224484

Epoch 7 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.225007

Epoch 8 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.006283

Epoch 9 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.806337

Epoch 10 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.893083

Epoch 11 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.886233

Epoch 12 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.891705

Epoch 13 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.852670

Epoch 14 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.782030

Epoch 15 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.714365

Epoch 16 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.572957

Epoch 17 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.539987

Epoch 18 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.512839

Epoch 19 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.438812

Epoch 20 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.300516

Epoch 21 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.179128

Epoch 22 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.060064

Epoch 23 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.120039

Epoch 24 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.868900

Epoch 25 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.859094

Epoch 26 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.773279

Epoch 27 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.789344

Epoch 28 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.779395

Epoch 29 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.707343

Epoch 30 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.556233

Epoch 31 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.789206

Epoch 32 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.722127

Epoch 33 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.573869

Epoch 34 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.589694

Epoch 35 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.609893

Epoch 36 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.294645

Epoch 37 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.150214

Epoch 38 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.077179

Epoch 39 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.112425

Epoch 40 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.102474

Epoch 41 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.042811

Epoch 42 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 1.924266

Epoch 43 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 1.847015

Epoch 44 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 1.737032

Epoch 45 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 1.891092

Epoch 46 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.522384

Epoch 47 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.208287

Epoch 48 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 6.848302

Epoch 49 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 4.863593

Epoch 50 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.935957

Epoch 51 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.274645

Epoch 52 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 3.136940

Epoch 53 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.958249

Epoch 54 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.874161

Epoch 55 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.827601

Epoch 56 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.664444

Epoch 57 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.772871

Epoch 58 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.558177

Epoch 59 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.605640

Epoch 60 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.536995

Epoch 61 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.220146

Epoch 62 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.352709

Epoch 63 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.372688

Epoch 64 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.315384

Epoch 65 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.168832

Epoch 66 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.174972

Epoch 67 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.244471

Epoch 68 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.079391

Epoch 69 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.036106

Epoch 70 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.168854

Epoch 71 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.080075

Epoch 72 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 2.011232

Epoch 73 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 1.966749

Epoch 74 | Training | Elapsed Time: 0:00:12 | Steps: 3 | Loss: 1.908187

I FINISHED optimization in 0:16:51.236346
I Could not find best validating checkpoint.
I Loading most recent checkpoint from /root/.local/share/deepspeech/checkpoints/train-225
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on /data/CV/zh/clips/test.csv
Test epoch | Steps: 3 | Elapsed Time: 0:00:01
Test on /data/CV/zh/clips/test.csv - WER: 0.666667, CER: 0.600000, loss: 1.832030

Best WER:

WER: 0.000000, CER: 0.000000, loss: 1.403480

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “猫”

WER: 1.000000, CER: 1.000000, loss: 2.414233

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “猫”

WER: 1.000000, CER: 0.500000, loss: 1.678377

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “你”

Median WER:

WER: 1.000000, CER: 0.500000, loss: 1.678377

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “你”

Worst WER:

WER: 0.000000, CER: 0.000000, loss: 1.403480

  • wav: file:///data/CV/zh/clips/mao.wav
  • src: “猫”
  • res: “猫”

WER: 1.000000, CER: 1.000000, loss: 2.414233

  • wav: file:///data/CV/zh/clips/tuzi.wav
  • src: “兔子”
  • res: “猫”

WER: 1.000000, CER: 0.500000, loss: 1.678377

  • wav: file:///data/CV/zh/clips/nihao.wav
  • src: “你好”
  • res: “你”