RAdam instead of adam

just a topic to discuss the use of Rectifier Adam (RAdam) instead of Adam in DeepSpeech.

If I trust this link using Radam might improve results.

Did this method will be implemented or is it not in the current reflexion ?

As far as I can tell, it’s not on our radar.

Is an RAdam implementation in TensorFlow?

Looks like it’s not there yet, issue 422

This looks testable: https://github.com/CyberZHG/keras-radam#tensorflow-without-keras

It might be interesting to test it, don’t know if there will be improvement