Oxford Robotics Institute - From Primates to Robots: Emerging Oscillatory Latent-Space Dynamics for Sensorimotor Control

From Primates to Robots: Emerging Oscillatory Latent-Space Dynamics for Sensorimotor Control

Project Background

Understanding the intricacies of locomotion and manipulation is a key driver for both neuroscientists and roboticists - recent work in neuroscience has revealed that locomotion in primates is controlled by oscillations in monkey’s motor cortex [1]. These oscillations are observed in low-dimensional projections of neural populations in the motor cortex while the monkey is performing rhythmic movement such as swimming and walking and are also present with linear limb movement such as reaching. The utilisation of generative models in the field of robotics has revealed that robot locomotion can also be represented as oscillatory signals in low-dimensional representations. The ORI research team had previously created a structured latent-space for locomotion utilising a deep generative model in order to control a real quadruped robot [2,3]. The team wanted to explore the structure of the learnt latent-space for locomotion and show that key gait-specific properties emerge automatically.

Project Results

The ORI team found that the learnt latent-spaces were straightforward to inspect compared to the motor cortex. Therefore, they probed the latent-space structures to investigate the structure and compare the resulting trajectories to those observed in motor cortex. Additionally, they modulated the signals injected into the latent-space to discover how these affected the robot’s movement.

As seen in the diagram below, the locomotion trajectories in latent-space form a periodic cycle where the time period of the red signal in (c) controls the cadence of the robot and the amplitude governs the footstep height. The blue signal is inferred from the robot states and controls the footstep length of the robot, and is π/2 out of phase with the red signal.

They repeated this experiment for manipulation, shown above in (d) and found that two sine wave oscillations, where the latter is also π/2 out of phase with the first, can solve reaching tasks. This resembled what was found in previous research [1], shown above in (b), where a cyclical pattern can be seen, constructed from two slightly out of phase periodic signals for both reaching and locomotion. At this stage the team could be sure whether this was a curious resemblance or something more fundamental.

The team then extended the analogy between the motor cortex and the latent-space by artificially impairing the locomotion decoder - something that cannot be done on a live motor cortex on both practical and ethical grounds, but was possible within the confines of their models. They utilised the same oscillatory drive signals and found that two pairs of legs are dragged along the ground and no longer take swing steps. The same sets of legs failed together since the locomotion is conditioned on the trot gait where the left front and right hind move together. They then encoded the resulting motion and noticed how the oscillations in latent-space changed shape and were no longer symmetrical, as shown in the diagram below.

Project Wins