Two fatal issues encountered while testing self-trained model

zcwang7-c · November 13, 2020, 8:51am

To whom it may concern,

Hi. Thanks for your efforts first.

I am training a Cantonese Model with self-collected audio data (around 250 hours) and external decoder (around 700M uncompressed). Even though the overfitting is observed due to the data size, the curve of the training loss is quit beautiful.

However, while I am conducting some tests, two fatal issues pop up.

Here is my project setting:

Have I written custom code (as opposed to running examples on an unmodified clone of the repository): : For the training, no change to code.
OS Platform and Distribution : Ubuntu 18.04
TensorFlow installed from : Using the docker file
TensorFlow version : v1.15.0.92.g5d80e1e 1.15.2
Python version : 3.6.9
Bazel version (if compiling from source) : 0.24.1
GCC/Compiler version (if compiling from source) : 7.5.0
CUDA/cuDNN version : 10.0/7
GPU model and memory : RTX 2080TI, 11G
Deepspeech Version : 0.7.4

Issue One :
"Segmentation fault (core dumped)" is raised when an empty audio clip with nobody speaking is given to the ds.stt . It may happens as well when the model can’t provide any prediction (no confirmed), since “core dumped” is also raised when I run the lm_optimizer.py with proper test audios.

In the above figure, a proper audio is used and a recognized result is provided (Shown in the red box). However, once an empty audio is fed, core dumped is given and the whole server crashes (Shown in the green box).

This is a fatal issue because the whole service is shut down. Thus, is there an approach to tackle the root cause or stop it from shutting the service?

Issue Two :
Unproperly decode of the recognized result when using ds.sttWithMetaData . Shown in the following figure, the external decoder works properly with ds.stt. However, once I switch to the ds.WithMetaData, the decoding becomes a mess. Is it caused by that I didn’t use the --utf8 for training?

The content in the blue box is produced by the following code.

def metadata_to_string(metadata):
    for token in metadata.tokens:
        try:
            print(token.text)
        except:
            print(repr(token.text)[1:-1])

    return ''.join(token.text for token in metadata.tokens)

More supported info produced using the origin client.py of deepspeech :

Issue One :

root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio empty_audio.wav 
2020-11-13 07:30:41.801501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:30:41.912246: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:30:41.913273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:30:41.945059: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:41.945704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:30:41.945718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:30:41.946654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:30:41.947472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:30:41.947661: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:30:41.948748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:30:41.949540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:30:41.952087: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:30:41.952187: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:41.952822: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:41.953342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:30:42.342786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:30:42.342813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-11-13 07:30:42.342818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-11-13 07:30:42.342932: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:42.343459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:42.343972: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:42.344458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9767 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.591s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000225s.
Running inference.
2020-11-13 07:30:43.248752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Segmentation fault (core dumped)
root@cd69049a871f:/DeepSpeech/asr_test_server#

Issue Two :
Success Case :

root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio audio_collect_zh-HK_19be47fa770a4a12826735b395f7e8ad.wav 
2020-11-13 07:38:53.148996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:38:53.261789: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:38:53.262647: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:38:53.291063: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.291840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:38:53.291854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:38:53.292750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:38:53.293539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:38:53.293701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:38:53.294676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:38:53.295394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:38:53.297740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:38:53.297831: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.298394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.298898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:38:53.703120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:38:53.703142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-11-13 07:38:53.703147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-11-13 07:38:53.703255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.703766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.704260: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.704746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.598s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000179s.
Running inference.
2020-11-13 07:38:54.636989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
將軍澳居民可價車從跨灣大橋
Inference took 1.467s for 3.840s audio file.
root@cd69049a871f:/DeepSpeech/asr_test_server#

Using “Extended” mode :

root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio audio_collect_zh-HK_19be47fa770a4a12826735b395f7e8ad.wav --extended
2020-11-13 07:39:02.605366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:39:02.715122: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:39:02.716027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:39:02.744854: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:02.745402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:39:02.745416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:39:02.746300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:39:02.747077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:39:02.747246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:39:02.748233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:39:02.748973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:39:02.751328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:39:02.751420: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:02.752022: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:02.752536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:39:03.249879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:39:03.249904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-11-13 07:39:03.249909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-11-13 07:39:03.250025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:03.250557: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:03.251069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:03.251559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9798 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.699s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000212s.
Running inference.
2020-11-13 07:39:04.193455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Traceback (most recent call last):
  File "/usr/local/bin/deepspeech", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/deepspeech/client.py", line 152, in main
    print(metadata_to_string(ds.sttWithMetadata(audio, 1).transcripts[0]))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-38: surrogates not allowed
root@cd69049a871f:/DeepSpeech/asr_test_server#

Using “Json” mode :

root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio audio_collect_zh-HK_19be47fa770a4a12826735b395f7e8ad.wav --json
2020-11-13 07:39:17.533590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:39:17.642701: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:39:17.643614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:39:17.672544: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:17.673301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:39:17.673321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:39:17.674439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:39:17.675416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:39:17.675613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:39:17.676623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:39:17.677360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:39:17.679744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:39:17.679845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:17.680468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:17.681029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:39:18.073182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:39:18.073203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-11-13 07:39:18.073208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-11-13 07:39:18.073316: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:18.073853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:18.074362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:18.074851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.586s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000177s.
Running inference.
2020-11-13 07:39:18.927302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
{
  "transcripts": [
    {
      "confidence": -251.61355590820312,
      "words": [
        {
          "word": "\udce5\udcb0\udc87\udce8\udcbb\udc8d\udce6\udcbe\udcb3\udce5\udcb1\udc85\udce6\udcb0\udc91\udce5\udc8f\udcaf\udce5\udc83\udcb9\udce8\udcbb\udc8a\udce5\udcbe\udc9e\udce8\udcb7\udca8\udce7\udc81\udca3\udce5\udca4\udca7\udce6\udca9\udc8b",
          "start_time ": 0.0,
          "duration": 3.8
        }
      ]
    },
    {
      "confidence": -252.30313110351562,
      "words": [
        {
          "word": "\udce5\udcb0\udc87\udce8\udcbb\udc8d\udce6\udcbe\udcb3\udce5\udcb1\udc85\udce6\udcb0\udc91\udce5\udc8f\udcaf\udce5\udc81\udc87\udce8\udcbb\udc8a\udce5\udcbe\udc9e\udce8\udcb7\udca8\udce7\udc81\udca3\udce5\udca4\udca7\udce6\udca9\udc8b",
          "start_time ": 0.0,
          "duration": 3.8
        }
      ]
    },
    {
      "confidence": -253.93341064453125,
      "words": [
        {
          "word": "\udce5\udcb0\udc87\udce8\udcbb\udc8d\udce6\udcbe\udcb3\udce5\udcb1\udc85\udce6\udcb0\udc91\udce5\udc8f\udcaf\udce9\udc81\udc87\udce8\udcbb\udc8a\udce5\udcbe\udc9e\udce8\udcb7\udca8\udce7\udc81\udca3\udce5\udca4\udca7\udce6\udca9\udc8b",
          "start_time ": 0.0,
          "duration": 3.8
        }
      ]
    }
  ]
}
Inference took 1.365s for 3.840s audio file.
root@cd69049a871f:/DeepSpeech/asr_test_server#

Thanks in advanced and best regards.
Dow

othiele · November 13, 2020, 9:25am

Thanks

What do you mean by empty? Length or no one speaking? It should work for the latter.

Just get JSON output and write your own metadta function, you’ll have changes for yourself eventually as you use DS more. Otherwise correct it for Unicodes and post a PR on github.

And you seem to have a 0.7 model, have you tried the 0.9.x code for it? Maybe that solves your problem directly. Models should be compatible.

zcwang7-c · November 13, 2020, 9:39am

Empty means no one speaking. It is quit weird that the same empty clip works on my Windows deepspeech==0.7.4 (without GPU support), but it raises error on Ubuntu 18.04 deepspeech-gpu==0.7.4 (with GPU).

Anyway, I will try the new version first. Hope it will work smoothly.

Thanks.

othiele · November 13, 2020, 9:47am

Have you tried different empty clips or is it just this clip? This sounds really strange.

And please let us know if there is anything different with the new DS version. Use a different virtual environment. Maybe it is some 3rd party lib that is causing the error.

lissyx · November 13, 2020, 9:59am

Please avoid screenshots

we can’t investigate if you don’t repro with:

our models, lm
share your code
share your stack

This is your error

I think this is expected behavior when using wrongfully encoded utf8 in json

As said above, please try and repro with our released models, and using 0.9.1. ALso verify with and without GPU.