VLAs with bimanual robotic arm(OpenArm)

In previous work, we demonstrate how we use Action Chunking Transformer(ACT) and SmolVLA to enable robotic arms to learn from human demonstrations and language instructions, but we only use low-cost single-arm robots, and all models are trained on real-world data. In this project, we explore how to train VLAs on fully simulated bimanual robotic arms and transfer to real-world tasks.

Real2Sim2Real pipeline for training bimanual robot policies

In data collection, we use the same setup as the previous projects, but we also collect language instructions along with the demonstrations.

Dataset Format

We use the LeRobot dataset format, which stores episodes as Parquet files containing observation(Robot joint states) , and action(Joint states commands) and mp4 for body camera and two wrist cameras observation.

LeRobot dataset structure with joint positions and camera observations

Control Method

Demonstrations are collected via teleoperation using Meta Quest3 Pro in Isaac Sim.

Leader-follower teleoperation for data collection, reference: [Github]

Source dataset

The collected dataset is available on Hugging Face.

10 source demos by human demos

After collecting small amount of demonstration data(ex: 10 demos), we generate a large-scale dataset(~400 demos) in Isaac Sim by randomizing pringles positions, size, textures, lighting and background.

Dataset Annotation

For dataset generation(Isaac Lab Mimic), we have to annotate each episode into several subtasks.

Dataset Generation

After annotating the source dataset, we can use Isaac Lab Mimic to generate a large-scale dataset by randomizing pringles positions, size, textures, lighting and background.

Domain randomization in Isaac Lab Mimic introduces varied lighting and backgrounds to improve dataset diversity. Dataset

After generating enough diverse data, we train VLA model(SmolVLA) to predict robot actions from visual observations and language instructions.

SmolVLA (Small Vision-Language Action Model)

Unlike common VLA models that use large vision-language backbones, SmolVLA is a lightweight model designed for fast training and inference on robotic arms.

SmolVLA Architecture (Source: SmolVLA Paper)

✅ Strengths

• • Fast training (Lightweight architecture)
• • Resource constraint (efficient inference)
• • Smooth trajectory generation(SA interleaves CA)

⚠️ Limitations

• • Poor zero-shot generalization
• • Not optimized for cross-embodiment transfer
• • Limited generalization

We deploy model on simulated OpenArm and real-world OpenArm for real-time inference and autonomous task execution.

Deployment in Isaac Sim

The success rate in Isaac Sim is 80% for 100 rollouts, and the model runs at ~30Hz, predicting actions that are executed by the robot controller in real-time.

Deployment in Real-world

The success rate in real-world is around 40% for 10 rollouts, and the model runs at ~30Hz, predicting actions that are executed by the robot controller in real-time.

While the model performs well in simulation, there are still challenges in transferring to real-world scenarios. Future work will focus on improving the model's robustness and generalization capabilities.

• Fixing simulation-to-real-world domain gaps
• Switch to another VLAs as GR00T N1.7, PI0.7...
• Fixing jerky trajectories in real-world deployment