Network Out Traffic

DeepSpeech
#1

Hi everyone,

I’ve been leveraging Deep Speech for a few months now and I noticed a large amount Network Out traffic when training a model.

I’m currently training a model starting with the deepspeech-0.5.1-checkpoint and an additional 50 hours of speech data. The training is running on an AWS EC2 and all audio and transcript csvs are on a mounted EFS.

During the training I observe about 100MB of Network Out traffic every 5 minutes. Intuitively I don’t understand why there would be data leaving my network.

Can anybody help me understand this traffic? Any insights would be greatly appreciated.

Mike

#2

We don’t have any outbound networking in our training code since we removed the distributed training support. Can you somehow get more insight into the traffic? What hosts is it connecting to? Are you running tensorboard in the same machine maybe? Is the checkpoints dir inside a networked folder?

#3

Thanks for the quick response!

AWS doesn’t make it easy to see where the traffic is going, I’m digging into this and what the traffic is a bit more. The amount of data makes me think checkpointing is likely the traffic, as I checkpoint to an EFS. The EFS is mounted and is in the same region so it shouldn’t be leaving the Network. As far as tensorboard, I’m not doing anything with it at the moment so I don’t believe that is the culprit.

Here is are graphs of the network traffic:

image
image.jpg1778×950 217 KB

image
image.jpg1756×944 193 KB

I’ll be sure to update this thread as I learn more.

Mike

#4

It looks closer to every 10 minutes than 5 to me from the graph (6 peaks in one hour), which does point to checkpointing, which is done every 600 seconds by default.

#5

It did turn out to be the checkpointing that was creating Network Out traffic. Despite the EFS being mounted and in the same AZ as the instance, the Network Out metric appears to include traffic to that EFS. I proved this by simply moving the checkpoint to the local EBS and saw the Network Out disappear. Turns out it’s an AWS metric vagueness issue, not a deep speech issue. Thanks for your input Reuben!

1 Like