Hi, everyone
I am trying train a Deepspeech model to Brazilian portuguese language
I generated the kenlm file with my dataset and start a train with deepspeech
The Deepspeech version: 0.7.4
I use the follow command:
python3 DeepSpeech.py \
--train_files ../dataset/train.csv \
--dev_files ../dataset/dev.csv \
--test_files ../dataset/test.csv \
--train_batch_size 64 \
--dev_batch_size 64 \
--test_batch_size 64 \
--audio_sample_rate 22050 \
--feature_win_len 40 \
--n_hidden 256 \
--epochs 100 \
--early_stop True \
--early_stop 6 \
--dropout_rate 0.20 \
--learning_rate 0.0001 \
--report_count 100 \
--export_dir ../results/model_export/ \
--checkpoint_dir ../results/checkout/ \
--alphabet_config_path data/alphabet.txt \
--utf8=true
my pip list is
Package Version Location
-------------------- ------------ -------------------------------------------------
absl-py 0.9.0
alembic 1.4.2
asn1crypto 0.24.0
astor 0.8.1
attrdict 2.0.1
attrs 19.3.0
audioread 2.1.8
beautifulsoup4 4.9.1
bs4 0.0.1
certifi 2020.6.20
cffi 1.11.5
chardet 3.0.4
cliff 3.3.0
cmaes 0.5.1
cmd2 1.1.0
colorama 0.4.3
colorlog 4.1.0
configobj 5.0.6
cryptography 2.3
dbus-python 1.2.4
decorator 4.2.1
deepspeech 0.7.4
deepspeech-training 0.7.4 /home/usr-ws-2/deepspeech-env/DeepSpeech/training
ds-ctcdecoder 0.7.4
ethtool 0.14
gast 0.2.2
google-pasta 0.2.0
gpg 1.10.0
grpcio 1.30.0
h5py 2.10.0
idna 2.5
importlib-metadata 1.6.1
iniparse 0.4
isc 2.0
joblib 0.15.1
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.2
librosa 0.7.2
llvmlite 0.31.0
Mako 1.1.3
Markdown 3.2.2
MarkupSafe 1.1.1
netifaces 0.10.6
nftables 0.1
numba 0.47.0
numpy 1.19.0
opt-einsum 3.2.1
optuna 1.5.0
opuslib 2.0.0
pandas 1.0.5
pbr 5.4.5
pciutils 2.3.6
perf 0.1
pip 20.0.2
ply 3.9
prettytable 0.7.2
progressbar2 3.51.3
protobuf 3.12.2
pycairo 1.16.3
pycparser 2.14
pygobject 3.28.3
pyinotify 0.9.6
pyOpenSSL 18.0.0
pyparsing 2.4.7
pyperclip 1.8.0
python-dateutil 2.6.1
python-dmidecode 3.12.2
python-editor 1.0.4
python-linux-procfs 0.6
python-utils 2.4.0
pytz 2020.1
pyudev 0.21.0
pyxdg 0.26
PyYAML 3.12
requests 2.24.0
resampy 0.2.2
rhnlib 2.8.6
rpm 4.14.2
schedutils 0.6
scikit-learn 0.23.1
scipy 1.5.0
selinux 2.9
semver 2.10.2
sepolicy 1.1
setools 4.2.2
setroubleshoot 1.1
setuptools 46.1.3
six 1.11.0
slip 0.6.4
slip.dbus 0.6.4
sos 3.8
SoundFile 0.10.3.post1
soupsieve 2.0.1
sox 1.3.7
SQLAlchemy 1.3.17
SSSDConfig 2.2.3
stevedore 2.0.1
subscription-manager 1.26.17
syspurpose 1.26.17
systemd-python 234
tensorboard 1.15.0
tensorflow-estimator 1.15.1
tensorflow-gpu 1.15.2
termcolor 1.1.0
threadpoolctl 2.1.0
tqdm 4.46.1
urllib3 1.25.9
wcwidth 0.2.5
Werkzeug 1.0.1
wheel 0.34.2
wrapt 1.12.1
zipp 3.1.0
The error
I FINISHED optimization in 0:10:26.618456
I Loading best validating checkpoint from ../results/checkout/best_dev-17544
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on ../dataset/test.csv
Test epoch | Steps: 0 | Elapsed Time: 0:00:00 Fatal Python error: Segmentation fault
Thread 0x00007f478cb2c700 (most recent call first):
Thread 0x00007f477633d700 (most recent call first):
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 383 in _recv
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 411 in _recv_bytes
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 254 in recv
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 463 in _handle_results
File "/usr/lib64/python3.6/threading.py", line 864 in run
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f4893777700 (most recent call first):
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/util/sample_collections.py", line 72 in load_sample
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/util/sample_collections.py", line 318 in __getitem__
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/util/augmentations.py", line 201 in timed_samples
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/util/helpers.py", line 96 in _limit
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 290 in _guarded_task_generation
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 419 in _handle_tasks
File "/usr/lib64/python3.6/threading.py", line 864 in run
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f4893f78700 (most recent call first):
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
File "/usr/lib64/python3.6/threading.py", line 864 in run
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f478fb72700 (most recent call first):
File "/usr/lib64/python3.6/threading.py", line 295 in wait
File "/usr/lib64/python3.6/queue.py", line 164 in get
File "/usr/local/lib64/python3.6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f47f0ffd700 (most recent call first):
File "/usr/lib64/python3.6/threading.py", line 295 in wait
File "/usr/lib64/python3.6/queue.py", line 164 in get
File "/usr/local/lib64/python3.6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f4829654700 (most recent call first):
File "/usr/lib64/python3.6/threading.py", line 295 in wait
File "/usr/lib64/python3.6/queue.py", line 164 in get
File "/usr/local/lib64/python3.6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f489a632740 (most recent call first):
File "/usr/local/lib64/python3.6/site-packages/ds_ctcdecoder/swigwrapper.py", line 364 in ctc_beam_search_decoder_batch
File "/usr/local/lib64/python3.6/site-packages/ds_ctcdecoder/__init__.py", line 134 in ctc_beam_search_decoder_batch
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/evaluate.py", line 111 in run_test
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/evaluate.py", line 129 in evaluate
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/train.py", line 659 in test
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/train.py", line 931 in main
File "/usr/local/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
File "/usr/local/lib/python3.6/site-packages/absl/app.py", line 299 in run
File "/home/usr-ws-2/deepspeech-env/DeepSpeech/training/deepspeech_training/train.py", line 955 in run_script
File "DeepSpeech.py", line 12 in <module>
Segmentation fault (core dumped)