Illegal Instruction when running sample training


(Rishabh Shah) #1

Hi,

I have compiled DeepSpeech and Tensorflow from source and was able to run the smoke_test for deepspeech. However, when I run the run-ldc93s1.sh I get the following error,

image

Not sure which part of installation would be a problem here. I have downloaded the Common Voice dataset and want to start training with that, but since I am stuck here not able to make sense of this first.

Appreciate any help. Thank you


(Lissyx) #2

Is there any reason you rebuilt everything ? Why not using prebuilt binaries with upstream TensorFlow ?


(Rishabh Shah) #3

Every time I create a virtual environment, I follow the instructions on https://github.com/mozilla/DeepSpeech#using-the-model

image

But when I try to use the deepspeech command, it gives me “Illegal Instruction”. When I looked up online I realized this could be due to difference in the architecture where the binaries are compiled on, which is why I compile both DeepSpeech and Tensorflow from source.

Thank you


(Lissyx) #4

For training you don’t need our TensorFlow nor our DeepSpeech binaries. Please retry with upstream Tensorflow.


(Rishabh Shah) #5

To get this right, if I use pip install tensorflow in a new venv, it will install upstream Tensorflow?

And then simply run the following?

run-ldc93s1.sh


(Rishabh Shah) #6

Following is after performing simple pip install tensorflow and then installing the dependent modules to run DeepSpeech.py (like numpy, pandas, python_speech_features and xdg).


(Lissyx) #7

Your output is partly missing. Please provide all the steps you follow. You should just:

$ virtualenv ds_venv
$ source ds_venv/bin/activate
$ pip install -r requirements.txt
$ python -u bin/run-ldc93s1.sh

Also, in general it is a bad idea to run code as root, you should avoid.


(Rishabh Shah) #8

Will keep in mind regarding the root user suggestion. This is merely a test run on virtualbox to get things working asap.

so I created a new virtualenv. Then performed the above mentioned steps. Initially, I was getting Syntax Error when running run-ld93s1.sh which was as follows,

image

So, since I knew I have all the required files, I commented out these lines to check for script and dataset. After executing the script it still thows the following error,

1

I am using Python 2.7.5.

Thank you!


(Rishabh Shah) #9

Attaching cpuinfo from the vm. might give some more insight.


(Lissyx) #10

I cannot help you more if you keep not documenting all your steps. I still cannot assert which version of tensorflow you are actually using in your virtual env. Please copy/paste instead of screenshotting.

And even if it’s just for quick test, running as root is still a bad idea: gives you bad habit, and sometimes can lead to tricky issues. Please try with a fresh user, fresh virtualenv, and document exactly all of your steps, from scratch.

Also, you might want to check your virtualization stack. Your virtual CPU has very few flags exposed, while the docs states much more: https://ark.intel.com/products/65719/Intel-Core-i7-3770-Processor-8M-Cache-up-to-3_90-GHz ; we should at least see SSE4.1, SSE4.2 and AVX. Even upstream TensorFlow requires at least SSE4.1/SSE4.2 to be available, as much as I can remember.