I get an error 404 downloading 'convert graphdef memmapped format' with util/taskcluster.py

DeepSpeech
#1

Hi, when i try to make a mmap-able model for inference, i use this command:
util/taskcluster.py --source tensorflow --artifact convert_graphdef_memmapped_format --target //tensorflow/contrib/util:convert_graphdef_memmapped_format
Docs

And get this error
Downloading https://index.taskcluster.net/v1/task/project.deepspeech.tensorflow.pip.master.cpu/artifacts/public/convert_graphdef_memmapped_format …
Traceback (most recent call last):
File “util/taskcluster.py”, line 157, in
main()
File “util/taskcluster.py”, line 151, in main
maybe_download_tc(target_dir=args.target, tc_url=get_tc_url(args.arch, args.artifact, args.branch))
File “util/taskcluster.py”, line 57, in maybe_download_tc
urllib.request.urlretrieve(tc_url, target_file, reporthook=(report_progress if progress else None))
File “/usr/lib/python3.5/urllib/request.py”, line 188, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File “/usr/lib/python3.5/urllib/request.py”, line 163, in urlopen
return opener.open(url, data, timeout)
File “/usr/lib/python3.5/urllib/request.py”, line 472, in open
response = meth(req, response)
File “/usr/lib/python3.5/urllib/request.py”, line 582, in http_response
‘http’, request, response, code, msg, hdrs)
File “/usr/lib/python3.5/urllib/request.py”, line 510, in error
return self._call_chain(*args)
File “/usr/lib/python3.5/urllib/request.py”, line 444, in _call_chain
result = func(*args)
File “/usr/lib/python3.5/urllib/request.py”, line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

I open the first URL on my chrome, and gets web response like json:
{

  • code: “ResourceNotFound”,

  • message: “Indexed task not found — * method: findArtifactFromTask * errorCode: ResourceNotFound * statusCode: 404 * time: 2019-09-26T10:39:09.951Z”,

  • requestInfo: {

    • method: “findArtifactFromTask”,

    • params: {

      • 0: “public/convert_graphdef_memmapped_format”,

      • indexPath: “project.deepspeech.tensorflow.pip.master.cpu”,

      • name: “public/convert_graphdef_memmapped_format”},

    • payload: { },

    • time: “2019-09-26T10:39:09.951Z”}

}

#2

You should use --branch with the matching branch name, i.e., r1.14.

#3

Hello sir I am also getting this error can you provide exact command for download convert-graphdef-memmapped-format

#4

Please be more descriptive @Sudarshan.gurav14 … A lot has moved since, so without more context on what you do, it’s hard to know for sure.

#5

It is hard to get the right call, this works for tf 1.13 for cpu

wget https://index.taskcluster.net/v1/task/project.deepspeech.tensorflow.pip.r1.13.cpu/artifacts/public/convert_graphdef_memmapped_format
2 Likes
#6

@othiele thank you it’s working for me

#7

@lissyx I have question if i have 4000 sample ( each have 10 sec) so, what will be my dev and test size please help. Basically what is ratio for train,dev,test

#8

Typical are 80/10/10 or 70/20/10

Please start a new topic if you want to follow up

#9

@othiele thank you sir

#10

Hi @othiele

I’m getting the same error while trying to export the model file. I ran training on 0.7.4 release checkpoint.

When I run:

python3 util/taskcluster.py --source tensorflow --artifact convert_graphdef_memmapped_format --branch r1.15 --target .

Error stack trace:

Downloading https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.tensorflow.pip.r1.15.cpu/artifacts/public/convert_graphdef_memmapped_format ...
Traceback (most recent call last):
  File "util/taskcluster.py", line 12, in <module>
    dsu_taskcluster.main()
  File "/content/DeepSpeech/training/deepspeech_training/util/taskcluster.py", line 128, in main
    maybe_download_tc(target_dir=args.target, tc_url=get_tc_url(args.arch, args.artifact, args.branch))
  File "/content/DeepSpeech/training/deepspeech_training/util/taskcluster.py", line 58, in maybe_download_tc
    _, headers = urllib.request.urlretrieve(tc_url, target_file, reporthook=(report_progress if progress else None))
  File "/usr/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 564, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
#11

The artifact expired after six months, as expected. I’m going to try and republish it, but you can build it yourself in the meantime.

1 Like
#12

So, because of infra issues, we can’t produce those artifacts anymore on r1.15 branch for windows and osx, but the linux builds are done and indexed, I’ve pushed the expiration to 3 years.

1 Like
#13

Hi @lissyx,

I am facing a HTTP Error 404: Not Found error too for r1.14 branch for Linux, when i run the following command:

python3 util/taskcluster.py --source tensorflow --artifact convert_graphdef_memmapped_format --branch r1.14 --target .

When ran for r1.15 it downloads the artifact, so is it that this artifact expired too?

#14

Yes, and unfortunately, CI changes makes it very hard to re-push them.

#15

@lissyx is there a way to produce the binary? Or another way to make a mmap-able model?

#16

yes, just git clone the tensorflow repo at branch r1.14 (upstream repo for that should be fine) and build the binary using bazel build -c opt "//tensorflow/contrib/util:convert_graphdef_memmapped_format" (maybe you’ll have to adjust a bit the build command args, but that should be fairly easy).

#17

Actually the exact same command worked for me, thank you @lissyx!