Reinforcement Learning in the Task of Spherical Robot Motion Control

This article discusses one of the DDPG (Deep Deterministic Policy Gradient) reinforcement learning algorithms applied to the problem of motion control of a spherical robot. Inside the spherical robot shell there is a platform with a wheel, and the robot is simulated in the MuJoCo physical simulation environment.
The goal is to teach the robot to move along an arbitrary closed curve with minimal error.
The output control algorithm is a pair of trained neural networks — actor and critic, where the actor-network is used to obtain the control torques applied to the robot wheel and the criticnetwork is only involved in the learning process. The results of the training are shown below, namely how the robot performs the motion along ten arbitrary trajectories, where the main quality functional is the average error magnitude over the trajectory length scale. The algorithm is implemented using the PyTorch machine learning library.