Conflicting numpy dependency pins for ds_ctcdecoder in osx, linux wheels

DeepSpeech
#1

Hi,

I noticed that in the python3.7 OSX wheel for ds_ctcdecoder 0.8.2, numpy has the requirement: (<=1.17.0,>=1.14.5), whereas in linux it’s simply (>=1.14.5).

I’m curious if anyone could give any background on the upper bound in the OSX build, as the differing dependencies across wheels are causing some problems for us. In particular, would be glad to know if anyone has a sense of the lift involved in removing this in, whether a PR to that effect would be welcome, or whether it’s more advisable to simply try to rebuild the wheel ourselves under our own build constraints.

Thanks in advance!

#2

this is simply constrainted by ensuring binary compatibility and availability of numpy depending on python version.

#3

Specifically, 1.17.0 is a side-effect from https://github.com/mozilla/DeepSpeech/commit/dfe8be30b43d3a4f2157dd7c61db676c435ddbbe#diff-2ab07bff3114ac69f3134f2321eb6156f24b064ee40ba248ad26255e58569bacR1040

we don’t want to remove that per se

if you don’t articulate those constraints, it’s hard to give a definitive answer on that matter.

#4

@Patrick_ONeill it’s handled here: https://github.com/mozilla/DeepSpeech/blob/master/taskcluster/tc-py-utils.sh#L204-L261

#5

Thanks for this. We have an application with another dependency that requires numpy >=1.18.2, meaning that our dependencies are currently satisfiable on linux but not on mac. AFAICT the latest numpy version for python3.7 on mac is 1.19. Could you say more about “ensuring binary compatibility and availability of numpy depending on python version.”?

thanks again.

#6

We forced 1.17.0 at a time we did not want to have higher versions pulled.

If 1.19.0 is binary compatible, it would be fine to push the version.

We discovered that some interna structure of numpy would change from version to version, and so if we build against X and run Y, they may not have the same infos (e.g. structure of different size) leading to warnings, errors or crashes.

Also, we needed to ensure that numpy wheels exists and we dont have to pull on full numpy builds.

But what is your setup that requires you to have ds_ctcdecoder in something else ? Its really only useful for deepspeech training, and this is nicer when isolated in its own virtualenv. What is it that you need to have numpy >= 1.18.2 on this same virtualenv ?

1 Like
#7

We are actually using ds_ctcdecoder in production along with Nemo 0.11.0, which introduces the numpy conflict [0].

I’ll see if I’m able to relax the pin on OSX, according to the constraints you mentioned, and can prepare a PR if all goes well.