Broaden the choices
Combine high-confidence action chunks with exploratory candidates, retaining pretrained knowledge while searching beyond familiar behavior.
ROBOT LEARNING · RESEARCH PROJECT
Never Stop Learning
Test-Time Reinforcement Learning
for Vision-Language-Action Models
MoE Key Lab of Artificial Intelligence, Institute of AI
Shanghai Jiao Tong University, Shanghai, China
* Corresponding author
A robot policy can do more than act.
It can explore, evaluate, and learn as it goes.
SIMULATED BIMANUAL MANIPULATION
“Assemble the three pieces to form a block.”
01 / OVERVIEW
Learning does not have to end
when training does.
Pretrained robot policies often repeat familiar actions. When those actions fail, a final success-or-failure signal offers little guidance on what to change.
EXPLORE is a simulator-based framework that improves vision-language-action models during task execution. At every decision, it combines confident action sequences with exploratory alternatives, evaluates their physical outcomes from the same simulator state, and uses that feedback both to choose an action and to update the policy.
Read the paper abstract +
Test-time compute scaling offers a promising path for improving foundation models beyond training-time data and parameters. For Vision-Language-Action (VLA) models, however, applying this paradigm through online reinforcement learning remains challenging due to two coupled limitations. Candidate chunks are often biased toward the pretrained policy’s high-likelihood regions, while sparse outcome-level feedback lacks process grounding, making reward signals weakly discriminative and exploration insufficiently guided. To address these challenges, we propose EXPLORE, a simulator-based test-time reinforcement learning framework that coordinates exploitation and exploration for action-chunk generation and drives policy optimization with physics-grounded process rewards. Through this joint design, EXPLORE broadens the effective action space while preserving the pretrained prior and derives faithful process rewards from dense physical interaction feedback to guide online policy improvement. Experiments on SimplerEnv-WidowX, DexMG, and RoboCasa validate the effectiveness of EXPLORE, with consistent improvements across autoregressive and diffusion-based VLA backbones.
02 / METHOD
Explore. Evaluate. Improve.
One loop at every decision step.
Two complementary action groups.
Ground the feedback
Use simulated object poses, contacts, grasp states, and spatial relations to measure progress throughout each candidate rollout.
Learn from the group
Choose the best chunk and use architecture-adaptive group relative policy optimization (A²-GRPO) to improve future actions. A reward-margin gate filters weak update signals.
03 / IN ACTION
Better alignment.
More coordinated manipulation.
The assembly example above compares EXPLORE with ACG: physical feedback supports coordinated bimanual actions. These are rollout visualizations from simulation, not real-robot demonstrations.
04 / EXPERIMENTS
Three benchmarks.
Two model architectures.
Success rate (%) across simulated
robotic manipulation tasks.
69.8% re-evaluated UniVLA baseline
36.27% re-evaluated ACG baseline
41.2% re-evaluated ACG baseline
| Benchmark | Backbone | Baseline | EXPLORE |
|---|---|---|---|
| SimplerEnv-WidowX | UniVLA · Autoregressive | 69.8 | 81.3 |
| DexMG | ACG · Diffusion | 36.27 ± 1.15 | 46.19 ± 1.86 |
| RoboCasa | ACG · Diffusion | 41.2 | 43.8 |
Values follow the paper’s main result tables. DexMG reports mean ± standard deviation over three random seeds. SimplerEnv results are rounded to one decimal place.
05 / CITATION
BibTeX
@misc{yi2026never,
title={Never Stop Learning: Test-Time Reinforcement
Learning for Vision-Language-Action Models},
author={Yi, Xinying and Jiang, Jingjing and Ma, Chao},
year={2026},
note={Project page; manuscript}
}The citation will be updated when an arXiv identifier or publication record is available.