VLAs with low-cost robotic arm
Imagine a robotic arm that can learn to perform tasks just by watching human demonstrations and reading instruction!
Overview
In our previous work, we demonstrated how Action Chunking Transformer (ACT) enables robotic arms to learn from human demonstrations, but language could not be directly incorporated.
This project demonstrates how robotic arms can learn using Visual Language Action (VLA) models like SmolVLA, that we can use language instructions with.
1. Data Collection
In data collection, we use the same setup as the previous projects, but we also collect language instructions along with the demonstrations.
Dataset Format
We use the LeRobot dataset format, which stores episodes as Parquet files. Since the leader and follower arms share the same joint names, observation and action use the identical joint-name schema below, plus mp4 for top and front camera observation.
# Robot Joint States (6-DOF, shared by observation & action)
- "shoulder_pan.pos"
- "shoulder_lift.pos"
- "elbow_flex.pos"
- "wrist_flex.pos"
- "wrist_roll.pos"
- "gripper.pos"
# Camera Observations
- "observation.images.front"
- "observation.images.top"
LeRobot dataset structure with joint positions and camera observations
Control Method
Human demonstrations are collected via teleoperation using a leader-follower setup, where the operator controls a leader arm and the follower arm mimics the movements.
Leader-follower teleoperation for data collection
2. Model Training
After collecting demonstration data, we train VLA model(SmolVLA) to predict robot actions from visual observations and language instructions.
SmolVLA (Small Vision-Language Action Model)
Unlike common VLA models that use large vision-language backbones, SmolVLA is a lightweight model designed for fast training and inference on robotic arms.
SmolVLA Architecture (Source: SmolVLA Paper)
✅ Strengths
- • Fast training (Lightweight architecture)
- • Resource-efficient (lightweight inference)
- • Smooth trajectory generation(SA interleaves CA)
⚠️ Limitations
- • Poor zero-shot generalization
- • Not optimized for cross-embodiment transfer
- • Limited generalization
3. Deployment
The trained model is deployed on the robot for real-time inference and autonomous task execution.
Real-time Inference
The model runs at ~30Hz, predicting actions that are executed by the robot controller in real-time.
Autonomous real-time task switching execution after 100 episodes for each task after training Dataset
Autonomous real-time task switching execution after 60 episodes for each task after training Dataset