Guide: Using Automatic Mixed Precision for NVIDIA Tensor Cores

[carlfm01]

Alternatively, would a simple alternative be to load the existing pre-trained weights and then using this mode do a few passes of the data just to adjust the weights with the new lower precision fp16 and then export that new model

It should work, I’m doing the opposite, training first on fp16 and then fine tuning on fp32. When you train on fp16 the entire saved model will be fp32.

Do you have any idea on potential benchmark numbers for inference?

No, sorry

It’s certainly not using the Tensor Cores.

You can ask here, maybe there are new uptades :

Deepspeech with TensorRT

I am actually attempting this right now. So, from what I have read online, you need to freeze the graph after running the checkpoints and then run TensorRT? Have you had an any luck?