at first I want to thank you for doing this project. This is just great.
I wonder whether you could talk a little bit more about the hardware you used to train the model? For example which mainboard did you use for the 8 Titan XP cards? Did you experience some heat/power issues? Could it be beneficial to use GPUs that have stronger half precision performance or is single precision mandatory? In general: What kind of hardware would you buy if you had to build a system like this today (and yes, I’m thinking about building a machine like that, that’s why I’m asking )?