Vision-Only USV Fleet Formation in Rain and Darkness
Follower boats hold formation using only a camera and a Kalman filter, with no leader telemetry, speed sensor or GPS. Tested in a Unity simulation.
Key Results
Can a boat hold formation from what its camera sees, with nothing else to rely on? In simulated rain, night and storms, yes: the Kalman filter holds followers within a few metres and recovers from a lost detection in about a second.
All results are from a Unity simulation at a fixed 10 Hz. Nothing here has been run on a real boat yet.
How It Works
Camera
One forward view from each follower
YOLOv8
Finds the leader's bounding box
Kalman filter
Estimates range and leader speed, and predicts when the box is missing
Controller
Sets throttle and steering to hold the slot
Throttle commands feed back into the filter, so it can tell the boat's own motion from the leader's
My Contribution
My Contribution — Stanley Chueh 闕楷宸
- Built the Unity three-boat fleet simulation: hull response to waves, stern wakes, rain, fog and night conditions
- Developed the vision-based follower controller and the YOLOv8 detector, including an auto-labelled dataset from the simulator
- Designed the vision-only Kalman filter that estimates range and leader speed from the camera, and the experiments testing it
External / Open-Source Methods Used
Unity 2022.3 · Suimono water system · Ultralytics YOLOv8
Reading Speed from the Image
When the leader's box shrinks, either the leader sped up or the follower slowed down. The filter separates the two using the follower's own commands, and keeps the leader's speed as a state it estimates from images alone. It also rejects boxes that disagree with its prediction.
Leader speed recovered from images. It lags by about 1 to 2 s and overshoots during speed changes (RMS error 2.6 m/s in this run).
Results
Kalman about halves the error in rain and darkness. Huge waves are unsolved.
Formation RMSE per run in metres, on a log scale. Lower is better; the vertical line marks 10 m. Both methods are vision-only. The detector was trained and tested on the same simulated weather.
View the numbers
| Condition | Kalman ON (m) | Kalman OFF (m) |
|---|---|---|
| Heavy rain | 2.08 / 2.59 | 4.19 / 4.29 |
| Night | 3.52 / 4.25 | 7.67 / 8.26 |
| Night + rain | 3.27 / 4.04 | 6.38 / 7.68 |
| Dark storm | 3.18 / 3.88 | 6.55 / 7.6 |
| Clear, speed changes | 8.46 / 9.12 / 7.44 | 18.43 / 10.39 / 7.82 |
| Huge waves | 891.13 / 3.55 | 6.92 / 7.6 |
| Huge waves, speed changes | 14.73 / 100.66 / 14.13 | 105.56 / 15.53 / 80.91 |
After a 2 s detection loss on these laps, Kalman is back in formation in 0.9 to 1.2 s, and Kalman OFF takes 1.8 to 5.2 s. In heavy rain and at night, the original detector almost never found the leader, so I generated an auto-labelled rain, fog and night dataset from the simulator and fine-tuned YOLOv8 on it. Detection rose to about 88% of frames, close to the maximum here because the scheduled dropouts remove about 11%.
How Error Is Measured
Each follower has an assigned slot 22 m behind and 13 m to the side of the leader, and the slots move with it. The error is how far the followers are from their slots:
error = √((e₁² + e₂²) / 2)
RMSE is that error averaged over the run after start-up. The formation counts as broken above 10 m. Positions come from the simulator and are used for scoring only; the boats never see them.
What Still Fails
- Huge waves. On a full lap, one of two Kalman runs lost the leader for good after a dropout, and one follower drifted away (891 m error). The other run held at 3.6 m, and Kalman OFF held both at about 7 m. With speed changes added, both struggle, with 1 of 3 Kalman runs and 2 of 3 Kalman OFF runs ending far out of formation.
- Speed changes. The speed estimate lags, so error grows each time the leader changes speed. In clear water with S-turns, median error is 8.5 m for Kalman and 10.4 m for Kalman OFF, and Kalman is more than 10 m out of formation about 26% of the time.
- Tight turns. Long turns of about 164 m radius broke formation for every method in earlier tests, and have not been re-tested in this vision-only setup.
- Simulation limits. The loop has zero processing delay. The hull model is simplified (thrust, steering torque, drag, wave-driven heave, pitch and roll), with no wind or currents, and the own-speed model closely matches this simulated hull. Box size is calibrated at a known starting distance, and the followers start on a shared signal.
- These are 2 or 3 seeds per condition, so the results are descriptive rather than statistically confirmed.
The Simulator
I built the simulator in Unity so the camera sees realistic motion and conditions: hulls pitch and roll on waves, boats leave wakes, and rain, fog and night lighting can be set for each run.
Water and wake rendering from above, in a scripted rendering test on a tighter turn