VLAs with bimanual robotic arm(OpenArm)
Can we train VLAs on fully simulated bimanual robotic arms and transfer to real-world tasks?
Overview
In our previous work, we demonstrated how Action Chunking Transformer (ACT) and SmolVLA enable robotic arms to learn from human demonstrations and language instructions, but we only used low-cost, single-arm robots, and all models are trained on real-world data. In this project, we explore how to train VLAs on fully simulated bimanual robotic arms and transfer to real-world tasks.
Real2Sim2Real pipeline for training bimanual robot policies
Key Results
My Contribution
My Contribution — Stanley Chueh 闕楷宸
- Co-leading the OpenArm VLA project at City Science Lab
- Built and maintain the OpenArm Isaac Lab simulation environment for teleoperation and Isaac Lab Mimic data generation
- Built and maintain the VR teleoperation data-collection pipeline (Meta Quest 3 Pro → Isaac Sim)
- Built and maintain the real-world OpenArm deployment code (SmolVLA policy → LeRobot → real robot)
Collaborators
City Science Lab VLA project team
External / Open-Source Methods Used
SmolVLA (Hugging Face) · Isaac Lab Mimic (NVIDIA) · Isaac Sim (NVIDIA)
1. Teleoperation - Isaac Sim
In data collection, we use the same setup as the previous projects, but we also collect language instructions along with the demonstrations.
Dataset Format
We use the LeRobot dataset format, which stores episodes as Parquet files containing observation(Robot joint states) , and action(Joint states commands) and mp4 for body camera and two wrist cameras observation.
# Robot Joint States (16-DOF)
- "LJ1.pos"
- "LJ2.pos"
- "LJ3.pos"
- "LJ4.pos"
- "LJ5.pos"
- "LJ6.pos"
- "LJ7.pos"
- "LJ8.pos"
- "RJ1.pos"
- "RJ2.pos"
- "RJ3.pos"
- "RJ4.pos"
- "RJ5.pos"
- "RJ6.pos"
- "RJ7.pos"
- "RJ8.pos"
# Camera Observations
- "observation.images.body_cam"
- "observation.images.wrist_cam"
- "observation.images.right_wrist_cam"
LeRobot dataset structure with joint positions and camera observations
Control Method
Demonstrations are collected via teleoperation using Meta Quest 3 Pro in Isaac Sim.
Leader-follower teleoperation for data collection, reference: [Github]
Source dataset
The collected dataset is available on Hugging Face.
10 source demos by human demos
2. Dataset generation - Isaac Sim
After collecting small amount of demonstration data(ex: 10 demos), we generate a large-scale dataset(~400 demos) in Isaac Sim by randomizing pringles positions, size, textures, lighting and background.
Dataset Annotation
For dataset generation(Isaac Lab Mimic), we have to annotate each episode into several subtasks.
right arm
Segment 1: Reach → grasp
Segment 2: Handover → release → return
left arm
Segment 3: Wait → handover → return
Dataset Generation
After annotating the source dataset, we can use Isaac Lab Mimic to generate a large-scale dataset by randomizing pringles positions, size, textures, lighting and background.
Domain randomization in Isaac Lab Mimic introduces varied lighting and backgrounds to improve dataset diversity. Dataset
2. Model Training
After generating enough diverse data, we train VLA model(SmolVLA) to predict robot actions from visual observations and language instructions.
SmolVLA (Small Vision-Language Action Model)
Unlike common VLA models that use large vision-language backbones, SmolVLA is a lightweight model designed for fast training and inference on robotic arms.
SmolVLA Architecture (Source: SmolVLA Paper)
✅ Strengths
- • Fast training (Lightweight architecture)
- • Resource-efficient (lightweight inference)
- • Smooth trajectory generation(SA interleaves CA)
⚠️ Limitations
- • Poor zero-shot generalization
- • Not optimized for cross-embodiment transfer
- • Limited generalization
3. Deployment
We deploy model on simulated OpenArm and real-world OpenArm for real-time inference and autonomous task execution.
Deployment in Isaac Sim
The success rate in Isaac Sim is 80% for 100 rollouts, and the model runs at ~30Hz, predicting actions that are executed by the robot controller in real-time.
Autonomous execution after training Dataset
Deployment in Real-world
The success rate in real-world is around 40% for 10 rollouts, and the model runs at ~30Hz, predicting actions that are executed by the robot controller in real-time.
Autonomous execution in real-world scenarios model
4. Limitations and Future Work
While the model performs well in simulation, there are still challenges in transferring to real-world scenarios. Future work will focus on improving the model's robustness and generalization capabilities.
- Fixing simulation-to-real-world domain gaps
- Switch to another VLAs as GR00T N1.7, PI0.7...
- Fixing jerky trajectories in real-world deployment