Abstract: Distributed training is the most common way to scale out and accelerate Deep Neural Network (DNN) training. Distributed DNN training requires synchronization of gradient aggregation among ...