Imitation Learning with low-cost robotic arm

This project demonstrates how low-cost robotic arms (Koch) can learn from human demonstrations using imitation learning by Action Chunking Transformer (ACT).

LeRobot Pipeline: Training imitation learning models by Koch robot

The first step is collecting high-quality demonstration data from human teleoperation.

Dataset Format

We use the LeRobot dataset format, which stores episodes as Parquet files. Since the leader and follower arms share the same joint names, observation and action use the identical joint-name schema below, plus mp4 for top and front camera observation.

LeRobot dataset structure with joint positions and camera observations

Control Method

Human demonstrations are collected via teleoperation using a leader-follower setup, where the operator controls a leader arm and the follower arm mimics the movements.

Leader-follower teleoperation for data collection

After collecting demonstration data, we train imitation learning models to predict robot actions from visual observations.

ACT (Action Chunking Transformer)

A pure imitation learning approach that predicts action sequences("chunks") rather than single actions . It uses a transformer encoder-decoder architecture with a CVAE (Conditional Variational Autoencoder) for modeling action distributions.

ACT Architecture (Source: ACT Paper)

✅ Strengths

• • Fast training (no VLM backbone)
• • Lightweight (~25M parameters)
• • Good for single-task learning

⚠️ Limitations

• • No language understanding
• • Requires task-specific training
• • Limited generalization

The trained model is deployed on the robot for real-time inference and autonomous task execution.

Real-time Inference

The model runs at ~30Hz, predicting action chunks that are executed by the robot controller in real-time.