The new system allows a human operator to remotely control a humanoid robot using only a webcam

Scarlett Evans, Assistant Editor, IoT World Today

March 13, 2024

2 Min Read
The system is based on reinforcement learning
The system is based on reinforcement learningCMU

Researchers have developed a new system that allows a human operator to remotely control a humanoid robot, in what the team says is the first known demonstration of whole-body humanoid teleoperation.

The team, from Carnegie Mellon University (CMU) based the system on reinforcement learning. Utilizing only an RGB camera, similar to those found in webcams, the platform translates an operator's actions into the humanoids’ movements. 

A video demonstration shows the system being used to make a humanoid robot from Unitree perform simple tasks including picking and placing items, as well as more dynamic actions such as boxing, kicking a football and pushing a pram.

To create a large-scale dataset of human movements for humanoid robots to copy, the team deployed a scalable "sim-to-data" process to filter and identify feasible motions using a privileged motion imitator. 

“Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner,” the team wrote. “We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc.

“To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.”

Related:Humanoid Robot Shows off Boxing Skills

The team said the design marks a breakthrough in teleoperation and opens the door to more streamlined human-to-robot interactions and control. 

Writing on X, formerly Twitter, lead researcher Tairan He said the framework “holds the key for collecting large-scale state-action pairs to train sensorimotor models for autonomous agents.”

“In the future, we can remotely teleoperate a humanoid to perform chores, dangerous missions and complex tasks that require a human-like level of dexterity and adaptability,” he wrote. 

View post on X

About the Author(s)

Scarlett Evans

Assistant Editor, IoT World Today

Scarlett Evans is the assistant editor for IoT World Today, with a particular focus on robotics and smart city technologies. Scarlett has previous experience in minerals and resources with Mine Australia, Mine Technology and Power Technology. She joined Informa in April 2022.

Sign Up for the Newsletter
The most up-to-date news and insights into the latest emerging technologies ... delivered right to your inbox!

You May Also Like