Imitation Learning with Franka Emika Panda
How to transfer the imitation learning techniques to a more complex robotic arm like Franka Emika Panda.
Overview
In our previous work, we demonstrated how robotic arms can learn from human demonstrations through Action Chunking Transformer (ACT). In this project, we demonstrate how we transfer that approach to the Franka Emika Panda.
Key Results
My Contribution
My Contribution — Stanley Chueh 闕楷宸
- Collected teleoperated demonstrations and trained/evaluated the ACT policy as part of the City Science Lab robotics team
- Set up leader-follower teleoperation control for the Franka Emika Panda, transferring the ACT data-collection pipeline first built on the Koch arm
Collaborators
City Science Lab robotics team (VLA/imitation-learning project)
External / Open-Source Methods Used
ACT (Action Chunking Transformer) · GELLO · Franka ROS · Avp_teleoperate (Vision Pro teleop reference)
1. Data Collection
The first step is collecting high-quality demonstration data from human teleoperation.
Dataset Format
We use the same LeRobot dataset format, which stores episodes as Parquet files and mp4 for front and wrist camera observation.
# Robot Joint States (7-DOF)
- "eef_x"
- "eef_y"
- "eef_z"
- "eef_quat_x"
- "eef_quat_y"
- "eef_quat_z"
- "eef_quat_w"
- "gripper_width"
# Camera Observations
- "observation.images.front"
- "observation.images.wrist_front"
- "observation.images.wrist_rear"
LeRobot dataset structure with end effector positions, quaternions and camera observations
Control Method
Human demonstrations are collected via teleoperation using a leader-follower setup, where the operator controls a leader arm and the follower arm mimics the movements.
Detail: We use the GELLO project to mirror the same leader-follower method used for the Koch robot's data collection. We use joint impedance control for the Franka Emika Panda via Franka ROS, mapping the leader arm's joint states to command the follower arm.
Leader-follower teleoperation for data collection
Previous Control Method
Previously, we employed Vision Pro to control the Franka Emika Panda. Vision Pro tracks the operator's hand and maps its relative movement to relative movement of the robot's end effector.
However, we found that Vision Pro control was not intuitive for this Franka Emika Panda setup; it may be more useful for bimanual robots.
Detail: we were inspired by Unitree Robotics' Avp_teleoperate, which also uses Vision Pro for data collection. We used the same method in the early stage of data collection, before switching to the leader-follower teleoperation method.
Vision Pro controlled pick-and-place demonstration
2. Model Training
After collecting demonstration data, we train imitation learning models to predict robot actions from visual observations.
ACT (Action Chunking Transformer)
A pure imitation learning approach that predicts action sequences("chunks") rather than single actions . It uses a transformer encoder-decoder architecture with a CVAE (Conditional Variational Autoencoder) for modeling action distributions.
ACT Architecture (Source: ACT Paper)
✅ Strengths
- • Fast training (no VLM backbone)
- • Lightweight (~25M parameters)
- • Good for single-task learning
⚠️ Limitations
- • No language understanding
- • Requires task-specific training
- • Limited generalization
3. Deployment
The trained model is deployed on the robot for real-time inference and autonomous task execution.
Real-time Inference
The model runs at ~10Hz, predicting action chunks that are executed by the robot controller in real-time.