Full Stack GPU Training and An Architecture of RL Policy with Optimization-Based Control

We provide a full-stack GPU training pipeline that reduces the training time by mitigating the data transfer between GPU and CPU. The capability of massive parallelization suites well for the environment includes an optimization-based controller since the optimization is typically slow. For this setup, the training speed is a few magnitudes faster than the CPU-based RL. In our task, the policy for stepping stones with various speeds and contact sequences converges in around 30 minutes. Moreover, we show an instant QP + RL policy has a similar performance to NMPC but runs in a fraction of time.  

Dr. Zhaoming Xie