ImportError: DLL load failed while importing _impl: The specified module could not be found

ZachVorhies · May 19, 2021, 7:12am

Trying to debug why deep speech is failing. I’ve made a bug report here:

ZachVorhies · May 19, 2021, 7:51pm

In order to continue investigating I created the minimal test case that I believe is triggering the problem: the failure to initialize libdeepspeech.so after load.

Example:

github.com

zackees/pydeepspeech/blob/main/tests/test_win_loadlib.py

import ctypes
import os
import unittest
import sys

import faulthandler
faulthandler.enable()

class WinLibdeepspeechTester(unittest.TestCase):
    @unittest.skipIf(sys.platform != 'win32', 'Win32 only test skipped.')
    def test_loading_win_dlls(self):
        """Gives the user a clue on any dlls that aren't installed yet."""
        # These dlls were found via DUMPBIN /import libdeepspeech.so
        # You'll need to install the VC build tools and enter the build
        # console in order to find all the files.
        dlls = [
            "USERENV.dll",
            "VERSION.dll",
            "ADVAPI32.dll",
            "WS2_32.dll",

This file has been truncated. show original

Notice here that all the dependent DLLs from windows are loading correctly. But when I try and load the libdeepspeech.so file the system core dumps on one of my machines. Looking at the dump file I see that there is an exception:

FAILED_INSTRUCTION_ADDRESS: 
libdeepspeech+148cc
00007ff8`873448cc c5f9efc0        vpxor   xmm0,xmm0,xmm0

Here’s the full stack trace:


Microsoft (R) Windows Debugger Version 10.0.21349.1004 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\niteris\AppData\Local\CrashDumps\python.exe.14588.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available


************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
Windows 10 Version 19043 MP (12 procs) Free x64
Product: WinNt, suite: SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Debug session time: Tue May 18 22:39:58.000 2021 (UTC - 7:00)
System Uptime: not available
Process Uptime: not available
..............................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(38fc.145c): Illegal instruction - code c000001d (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ff9`0d4ed974 c3              ret
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 1327

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 2898

    Key  : Analysis.Init.CPU.mSec
    Value: 359

    Key  : Analysis.Init.Elapsed.mSec
    Value: 3349

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 97

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 3.8.150.1013


NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
rax=00000234563cd7b0 rbx=0000000000000039 rcx=000000007ffe0380
rdx=00007ff8886f7638 rsi=00000234563cd7b0 rdi=0000000000000000
rip=00007ff8873448cc rsp=000000cfce9e9e60 rbp=00007ff8886f75f8
 r8=0000000000000024  r9=0000000000000019 r10=0000000000000000
r11=0a0000000001a1f0 r12=00007ff88869799c r13=0000000000000001
r14=00007ff887330000 r15=0000000000000000
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
libdeepspeech+0x148cc:
00007ff8`873448cc c5f9efc0        vpxor   xmm0,xmm0,xmm0
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff8873448cc (libdeepspeech+0x00000000000148cc)
   ExceptionCode: c000001d (Illegal instruction)
  ExceptionFlags: 00000000
NumberParameters: 0

PROCESS_NAME:  python.exe

ERROR_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

EXCEPTION_CODE_STR:  c000001d

STACK_TEXT:  
000000cf`ce9e9e60 00007ff9`0b07e473     : 00007ff8`886f1f60 00000000`00000000 00000000`0000000e 00007ff8`d8f321fb : libdeepspeech+0x148cc
000000cf`ce9e9f30 00007ff8`88697766     : 00000000`7ffe0300 000000cf`ce9ea338 00000000`00000000 000000cf`ce9ea338 : ucrtbase!initterm+0x43
000000cf`ce9e9f60 00007ff8`886978dc     : 00000000`7ffe0385 00000000`00000000 00000000`00000000 00000000`00000001 : libdeepspeech!stream_executor::Stream::ThenBlasGemm+0x2dcf96
000000cf`ce9e9f90 00007ff9`0d469a1d     : 00007ff8`87330000 00000000`00000001 00000000`00000000 00000000`7ffe0385 : libdeepspeech!stream_executor::Stream::ThenBlasGemm+0x2dd10c
000000cf`ce9e9ff0 00007ff9`0d4bc1e7     : 00000234`563d54d0 00007ff8`87330000 00007ff8`00000001 00007ff8`88697540 : ntdll!LdrpCallInitRoutine+0x61
000000cf`ce9ea060 00007ff9`0d4bbf7a     : 00000234`563d9bb0 00000234`563d9b00 000000cf`ce9ea301 00000234`00000001 : ntdll!LdrpInitializeNode+0x1d3
000000cf`ce9ea1b0 00007ff9`0d4bc000     : 000000cf`ce9ea300 00000234`563d9d30 000000cf`ce9ea338 00000234`563d2580 : ntdll!LdrpInitializeGraphRecurse+0x42
000000cf`ce9ea1f0 00007ff9`0d48d937     : 00000000`00000000 00000000`00000000 000000cf`ce9ea3f0 000000cf`ce9ea338 : ntdll!LdrpInitializeGraphRecurse+0xc8
000000cf`ce9ea230 00007ff9`0d46fbae     : 000000cf`ce9ea338 000000cf`ce9ea340 000000cf`ce9ea300 000000cf`ce9ea340 : ntdll!LdrpPrepareModuleForExecution+0xbf
000000cf`ce9ea270 00007ff9`0d4673e4     : 000000cf`ce9ea340 000000cf`ce9ea4e0 000000cf`ce9ea5d0 000000cf`ce9ea4d0 : ntdll!LdrpLoadDllInternal+0x19a
000000cf`ce9ea2f0 00007ff9`0d466af4     : 00000000`00000000 00000000`00001101 00000234`566bd1c0 00000234`566b80a0 : ntdll!LdrpLoadDll+0xa8
000000cf`ce9ea4a0 00007ff9`0abbaa02     : 00000234`563d9520 00000000`00001100 00000234`566ea92a 00000000`00000000 : ntdll!LdrLoadDll+0xe4
000000cf`ce9ea590 00007ff8`d84b9bda     : 00007ff8`00000000 00000234`56757b90 00007ff8`d8591260 00000000`00000000 : KERNELBASE!LoadLibraryExW+0x162
000000cf`ce9ea600 00007ff8`d84b98cf     : 00000000`00000000 00007ff8`d858b988 00000234`56fa75d0 00000234`56f87030 : python38!PyRun_SimpleFileExFlags+0x5d2
000000cf`ce9ea9b0 00007ff8`d84b9adf     : 00000234`56fa75d0 00007ff8`d85912b8 00000000`00000001 00000234`56fa5e80 : python38!PyRun_SimpleFileExFlags+0x2c7
000000cf`ce9eaa20 00007ff8`d84b9a3d     : 00000000`00000000 00000234`566c8810 00000000`00000001 00000234`56fa5ef8 : python38!PyRun_SimpleFileExFlags+0x4d7
000000cf`ce9eaa50 00007ff8`d83fd6e8     : 00000000`00000000 00000234`566b9310 00000234`566727f0 00000000`000000a7 : python38!PyRun_SimpleFileExFlags+0x435
000000cf`ce9eaa80 00007ff8`d83c60d6     : 00000000`00000000 00000234`566c8810 00000234`563d9520 00000234`56fa5ef8 : python38!PyObject_CallFunctionObjArgs+0x6d8
000000cf`ce9eaac0 00007ff8`d83c5fa7     : 00000000`00000004 00000000`00000000 00000234`56580000 00000234`00000000 : python38!PyVectorcall_Call+0x6e
000000cf`ce9eab40 00007ff8`d83dc2c0     : 00000000`00000000 000000cf`ce9eac59 00000234`566c8810 00000234`566727f0 : python38!PyNumber_Long+0x617
000000cf`ce9eab90 00007ff8`d83d69e8     : 00000234`56f73c10 00000000`0000004f 00000234`56f73c10 00000234`566ba9d0 : python38!PyEval_EvalFrameDefault+0x1940
000000cf`ce9eacc0 00007ff8`d83d99c4     : 00000000`00000001 00000000`00000001 00000000`00000002 00000234`56f4ba90 : python38!PyEval_EvalCodeWithName+0x228
000000cf`ce9ead80 00007ff8`d83db43a     : 00000234`566b9310 00000000`00000000 00000234`566b9310 00000234`563ed0d8 : python38!Py_CheckFunctionResult+0xde4
000000cf`ce9eae60 00007ff8`d83d9770     : 00000234`566af9c0 00000000`00000002 00000234`56679a28 000000cf`ce9eb110 : python38!PyEval_EvalFrameDefault+0xaba
000000cf`ce9eaf90 00007ff8`d83db1a2     : 00000234`563d9520 00000000`00000000 000000cf`ce9eb128 67ec4627`61538d4d : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9eb070 00007ff8`d83d9770     : 00000000`00000002 00000000`00000001 00000234`56725978 000000cf`ce9eb340 : python38!PyEval_EvalFrameDefault+0x822
000000cf`ce9eb1a0 00007ff8`d83db00f     : 00000234`563d9520 00000000`00000000 00000234`566cae50 00000234`56403268 : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9eb280 00007ff8`d83d9770     : 00000000`00000002 00000000`00000001 00000234`56f55be8 000000cf`ce9eb550 : python38!PyEval_EvalFrameDefault+0x68f
000000cf`ce9eb3b0 00007ff8`d83db00f     : 00000234`563d9520 00000000`00000000 00000234`566cb0d0 00000234`56403268 : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9eb490 00007ff8`d83d9770     : 80000000`00000003 00000000`00000002 00000234`56c15b80 000000cf`ce9eb760 : python38!PyEval_EvalFrameDefault+0x68f
000000cf`ce9eb5c0 00007ff8`d83db00f     : 00000234`563d9520 00000000`00000000 00000234`566ce310 00000234`56403268 : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9eb6a0 00007ff8`d83d845d     : 00007ff8`d858b99a 00000000`00000002 00000000`00000002 000000cf`ce9eb8b0 : python38!PyEval_EvalFrameDefault+0x68f
000000cf`ce9eb7d0 00007ff8`d83ec7db     : 00000000`00000000 00000234`566a2e80 00000234`566b4870 00000000`000000ff : python38!PyFunction_Vectorcall+0xdd
000000cf`ce9eb880 00007ff8`d83ec6b4     : 00000234`566ce3a0 00000234`566a2e80 00000234`563d6360 00007ff8`d8731880 : python38!PyObject_CallMethodIdObjArgs+0x193
000000cf`ce9eb8f0 00007ff8`d83eba7d     : 00000234`566b4780 00007ff8`d87459e0 00000234`56fa75d0 00000234`566b03b0 : python38!PyObject_CallMethodIdObjArgs+0x6c
000000cf`ce9eb940 00007ff8`d83bcffc     : 00000000`00000000 00000000`00000000 00000000`00000000 00000234`56fa75d0 : python38!PyEval_EvalCodeEx+0x3d9
000000cf`ce9eb9b0 00007ff8`d83c7a59     : 00000234`563d6360 00000234`566b03b0 000000cf`ce9eba90 00000234`56f7db40 : python38!PyImport_ImportModuleLevelObject+0x3ac
000000cf`ce9eba20 00007ff8`d83cb064     : 000000cf`ce9ebbe0 00000000`0000000a 00000000`00000000 000000cf`00000000 : python38!PyUnicode_FromEncodedObject+0x179
000000cf`ce9ebaa0 00007ff8`d83c6025     : 00000000`00000000 00000234`566b03b0 00000234`56fa5c70 00000234`563d9520 : python38!PyCFunction_Call+0x7c
000000cf`ce9ebad0 00007ff8`d83dc2c0     : 00000000`00000000 000000cf`ce9ebbe9 00000234`566b03b0 00007ff8`d83fb75c : python38!PyNumber_Long+0x695
000000cf`ce9ebb20 00007ff8`d83d69e8     : 00000234`56f738b0 00000000`0000004f 00000234`56f738b0 00000234`566ba9d0 : python38!PyEval_EvalFrameDefault+0x1940
000000cf`ce9ebc50 00007ff8`d83d99c4     : 00000000`00000001 00000000`00000001 00000000`00000002 00000234`5642c6c8 : python38!PyEval_EvalCodeWithName+0x228
000000cf`ce9ebd10 00007ff8`d83db00f     : 00000000`00000000 00000000`00000000 00000234`566ca4c0 00000234`56403268 : python38!Py_CheckFunctionResult+0xde4
000000cf`ce9ebdf0 00007ff8`d83d69e8     : 00000234`5642c690 00000000`00000043 00000234`5642c510 00000234`566c53a0 : python38!PyEval_EvalFrameDefault+0x68f
000000cf`ce9ebf20 00007ff8`d83d853f     : 00000000`00000000 00000000`00000004 00000000`00000000 000000cf`ce9ec0b0 : python38!PyEval_EvalCodeWithName+0x228
000000cf`ce9ebfe0 00007ff8`d83ec7db     : 00000000`00000000 00000234`56f96a90 00000234`56f7bb70 00000234`56f7bb70 : python38!PyFunction_Vectorcall+0x1bf
000000cf`ce9ec090 00007ff8`d83ec6b4     : 00000234`566ce4c0 000000cf`ce9ec188 00000000`00000000 00007ff8`d842b9e8 : python38!PyObject_CallMethodIdObjArgs+0x193
000000cf`ce9ec100 00007ff8`d83bcfd7     : 00000234`56f47f40 00007ff8`d8745ad0 00000234`56f47f40 00000234`56f96a90 : python38!PyObject_CallMethodIdObjArgs+0x6c
000000cf`ce9ec150 00007ff8`d849f846     : 00000234`563d6360 00000234`563d9520 00007ff8`d87696a0 00000234`56fa8040 : python38!PyImport_ImportModuleLevelObject+0x387
000000cf`ce9ec1c0 00007ff8`d83dcede     : 00000234`56f7bb70 00000234`566a2eb0 000000cf`ce9ec329 00000234`566a9630 : python38!PyObject_GenericGetDict+0x77a
000000cf`ce9ec260 00007ff8`d83d69e8     : 00000234`566af9c0 00000000`00000040 00000234`56fa8040 00000234`56fa6be0 : python38!PyEval_EvalFrameDefault+0x255e
000000cf`ce9ec390 00007ff8`d83eb73f     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : python38!PyEval_EvalCodeWithName+0x228
000000cf`ce9ec450 00007ff8`d83eb69d     : 00000234`56fa0b80 00000234`56fa6be0 00000000`00000000 00000234`56fa1c58 : python38!PyEval_EvalCodeEx+0x9b
000000cf`ce9ec4e0 00007ff8`d83eb8a1     : 00000234`56fa0b80 00007ff8`d858b9cc 00000234`56fa6be0 00000234`56f56cc8 : python38!PyEval_EvalCode+0x2d
000000cf`ce9ec550 00007ff8`d83eb7ab     : 00000234`566b0310 00000234`56fa1c58 00000000`00000002 00000000`000003ff : python38!PyEval_EvalCodeEx+0x1fd
000000cf`ce9ec5a0 00007ff8`d83fd6e8     : 00000000`00000000 00000234`566b0810 00000234`566b0310 00000000`00000002 : python38!PyEval_EvalCodeEx+0x107
000000cf`ce9ec5d0 00007ff8`d83c60d6     : 00000000`00000000 00000234`566b0810 00000234`56724578 00000234`56fa1c58 : python38!PyObject_CallFunctionObjArgs+0x6d8
000000cf`ce9ec610 00007ff8`d83c5fa7     : 00000000`00000000 00000000`00000000 00000234`566ebee0 00000000`00000000 : python38!PyVectorcall_Call+0x6e
000000cf`ce9ec690 00007ff8`d83dc2c0     : 00000000`00000000 000000cf`ce9ec7a9 00000234`566b0810 00000000`00000003 : python38!PyNumber_Long+0x617
000000cf`ce9ec6e0 00007ff8`d83d69e8     : 00000234`56f951f0 00000000`0000004f 00000234`56f951f0 00000234`566ba9d0 : python38!PyEval_EvalFrameDefault+0x1940
000000cf`ce9ec810 00007ff8`d83d99c4     : 00000000`00000001 00000000`00000001 00000000`00000003 00000234`56f4bfd0 : python38!PyEval_EvalCodeWithName+0x228
000000cf`ce9ec8d0 00007ff8`d83db43a     : 00000234`56fa3540 00000000`00000000 00000234`566b0810 4fe50118`e77cc8ad : python38!Py_CheckFunctionResult+0xde4
000000cf`ce9ec9b0 00007ff8`d83d9770     : 00000234`566af9c0 00000000`00000002 00000234`56f75788 000000cf`ce9ecc60 : python38!PyEval_EvalFrameDefault+0xaba
000000cf`ce9ecae0 00007ff8`d83db1a2     : 00000234`563d9520 00000000`00000000 000000cf`ce9ecc78 60db0423`68966a08 : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9ecbc0 00007ff8`d83d9770     : 00000000`00000002 00000000`00000001 00000234`56f553e8 000000cf`ce9ece90 : python38!PyEval_EvalFrameDefault+0x822
000000cf`ce9eccf0 00007ff8`d83db00f     : 00000234`563d9520 00000000`00000000 00000234`566cb0d0 00000234`56403268 : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9ecdd0 00007ff8`d83d9770     : 80000000`00000003 00000000`00000002 00000234`56f755b0 000000cf`ce9ed0a0 : python38!PyEval_EvalFrameDefault+0x68f
000000cf`ce9ecf00 00007ff8`d83db00f     : 00000234`563d9520 00000000`00000000 00000234`566ce310 00000234`56403268 : python38!Py_CheckFunctionResult+0xb90
000000cf`ce9ecfe0 00007ff8`d83d845d     : 00000234`566af9c0 00000000`00000002 00000000`00000002 000000cf`ce9ed1f0 : python38!PyEval_EvalFrameDefault+0x68f
000000cf`ce9ed110 00007ff8`d83ec7db     : 00000000`00000000 00000234`566a2e80 00000234`566b4870 00000000`000000ff : python38!PyFunction_Vectorcall+0xdd
000000cf`ce9ed1c0 00007ff8`d83ec6b4     : 00000234`566ce3a0 00000234`566a2e80 00000234`563d6360 00007ff8`d8731880 : python38!PyObject_CallMethodIdObjArgs+0x193
000000cf`ce9ed230 00007ff8`d83eba7d     : 00000234`566b4780 00007ff8`d87459e0 00000234`56f638f0 00000234`566b03b0 : python38!PyObject_CallMethodIdObjArgs+0x6c
000000cf`ce9ed280 00007ff8`d83bcffc     : 00000000`00000000 00000000`00000000 00000000`00000000 00000234`56f638f0 : python38!PyEval_EvalCodeEx+0x3d9
000000cf`ce9ed2f0 00007ff8`d849f846     : 00000234`563d6360 00000234`563d9520 00007ff8`d8769680 00000234`56efc040 : python38!PyImport_ImportModuleLevelObject+0x3ac
000000cf`ce9ed360 00007ff8`d83dcede     : 00000234`56efc1a8 00000234`56efc040 000000cf`ce9ed4b8 00007ff9`0b06f05b : python38!PyObject_GenericGetDict+0x77a
000000cf`ce9ed400 00007ff8`d83d69e8     : 00000234`566af9c0 00000000`00000040 00000234`56efc040 00000234`56f64450 : python38!PyEval_EvalFrameDefault+0x255e
000000cf`ce9ed530 00007ff8`d83eb73f     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : python38!PyEval_EvalCodeWithName+0x228
000000cf`ce9ed5f0 00007ff8`d83eb69d     : 00000234`56f21940 00000234`56f64450 00000000`00000000 00000234`56f63998 : python38!PyEval_EvalCodeEx+0x9b
000000cf`ce9ed680 00007ff8`d83eb8a1     : 00000234`56f21940 00007ff8`d858b9cc 00000234`56f64450 00000234`56738100 : python38!PyEval_EvalCode+0x2d

FAILED_INSTRUCTION_ADDRESS: 
libdeepspeech+148cc
00007ff8`873448cc c5f9efc0        vpxor   xmm0,xmm0,xmm0

SYMBOL_NAME:  libdeepspeech+148cc

MODULE_NAME: libdeepspeech

IMAGE_NAME:  libdeepspeech.so

STACK_COMMAND:  ~0s ; .ecxr ; kb

FAILURE_BUCKET_ID:  ILLEGAL_INSTRUCTION_c000001d_libdeepspeech.so!Unknown

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {7ebad748-7f9a-037f-ec25-ba02f12eca74}

Followup:     MachineOwner
---------

ZachVorhies · May 20, 2021, 4:58pm

I’m pretty sure this is because of the product requiring the AVX instruction set, which the failing machine using an older 32nm Intel processor does not have.

Others seem to be running into this problem as well.

I may make a pull request that catches the import error at the python level and outputs an appropriate message.

sanjaesc · May 21, 2021, 4:18pm

Wrong section ^^ should be in https://discourse.mozilla.org/c/deepspeech/247