NVIDIA releases AlpaGym, closed-loop RL for driving policies

NVIDIA has released AlpaGym, a system that post-trains autonomous driving policies on the consequences of their own actions inside a simulator rather than on recorded expert trajectories. It ships as part of Alpamayo, NVIDIA's open platform of AV models, simulation tools and datasets, alongside the AlpaSim simulator. The target is a structural weakness in vision-language-action driving models: they are trained open-loop, with their outputs scored against reference behaviour, and never see how their own braking or steering changes the next state of the road.

That gap is not academic. A deployed driving policy operates in a closed loop, where every navigation decision alters the scene it will next perceive, and small errors compound. AlpaGym turns AlpaSim rollouts into training experience, so the simulator stops being a final checkpoint gate and becomes a feedback source inside the learning loop itself. Reinforcement learning then improves a policy that was originally trained open-loop, optimising it against what its own actions produce.

The failure modes this is meant to catch are the ones static datasets cannot show. If a prediction or planning error shifts the environment into a state the expert data never contained, open-loop evaluation records a small trajectory deviation and moves on. Closed-loop training records what happens three seconds later.

The cost is orchestration. Model inference, simulation, training, synchronisation of updated weights, data exchange and communication between instances all have to run in parallel, which means both component coordination and efficient use of the hardware.

AlpaGym is assembled from three existing pieces: AlpaSim simulator microservices, NVIDIA's open physical AI datasets, and the Cosmos-RL distributed training system. AlpaSim and Cosmos-RL act as the runtime and the orchestration layer; the default algorithm is GRPO; reference reward functions come validated against Alpamayo models and the Physical AI AV NuRec dataset. NVIDIA describes the pipeline as asynchronous and stable, scaling from a single GPU to multi-node clusters without changes to user code.

The entry requirements set the real audience. NVIDIA asks for a CUDA-capable GPU with at least 40 GB of video memory and roughly 100–150 GB of free disk for the environment, container images and about 21 GB of weights. Each NuRec scene costs around 1.5 GB, and the full public_2601 set runs to roughly 1.5 TB. Installation comes from the Alpamayo repository: CUDA and Redis dependencies on the host, then a uv workspace sync, with cuDNN, NCCL and the redis-server binary treated as host-level dependencies for the CUDA component set and Cosmos-RL. A Dockerfile is offered as an alternative, and downloading scene files requires a Hugging Face login.

A run is described by a Hydra configuration naming the policy checkpoint, the AlpaSim scene set, the degree of rollout parallelism, the reward function and the Cosmos-RL training parameters. The reward is where the actual work sits. Candidate components for trajectory quality include progress, lane keeping, collision avoidance, off-road rate, comfort and distance to a reference trajectory. NVIDIA's suggested starting point is deliberately crude — reward progress, penalise critical safety violations — expressed as a small sum of terms drawn where possible from AlpaSim metrics, with finer components added once the pipeline is stable and the video and metrics have shown which error types are worth naming.

The worked example is Alpamayo 1.5, a 10-billion-parameter model. The checkpoint is downloaded and converted into an AlpaGym-compatible format, then AlpaGym runs alongside AlpaSim on two GPUs; NVIDIA validated the setup on two RTX 6000 Ada cards with 50 GB each. The 10B model does not fit on a single GPU, and a distillation script producing a single-GPU checkpoint is described as planned. Multi-GPU and multi-node scaling examples live in the NVlabs/alpagym repository.

Which is where the "single GPU to multi-node clusters" line starts to read as aspirational rather than descriptive. The lower bound holds for the framework, not for anything you can actually train with it today: the only supported model needs two cards, and the tool that would change that does not exist yet. The 40 GB minimum is similarly a floor for the software, not for the job. Anyone reading the requirements as a shopping list will buy the wrong machine.

During training, AlpaGym requests scene rollouts from AlpaSim, collects per-episode data, computes rewards and updates the policy. The signals to watch are mean reward and its variance, error rate, policy loss, rollout throughput, and the lag between when rollouts were generated and the latest policy weights. Run artifacts land in `tmp/alpagym-runs/` and Hydra outputs in `outputs/`.

Validation after training is a conversion problem. Cosmos-RL exports safetensors into `tmp/alpagym-runs/`, which the AlpaSim driver cannot load directly, so the checkpoint has to be converted back into an inference format — the inverse of the conversion done before training. Running it on a representative scenario confirms that policy, driver and simulation loop are wired together correctly and shows how the model's own actions move the environment. That step requires access to a gated vision-language model, `nvidia/Cosmos-Reason2-8B`, which the driver uses to restore the original tokenizer and preprocessor.

Notably absent from all of this is a number. The premise is that closed-loop post-training produces better driving policies than open-loop imitation, and the material offers reward curves, episode metrics, rollout videos and collected failure cases — instruments, not results. Not one benchmark delta appears anywhere. For a release aimed at engineers who will spend two GPUs and a terabyte and a half of scenes to try it, the absence of a single before-and-after figure is the most informative thing in the announcement. The second most informative is the gated dependency: an open platform whose evaluation path routes through a model you have to be granted access to is open in a narrower sense than the word usually implies.

There is also the question the framing sidesteps. Every reward in this loop is computed inside a reconstruction of the world, against scenes NVIDIA assembled, using metrics NVIDIA defined. Closing the gap between training and deployment is the stated goal, but what AlpaGym closes is the gap between training and simulation. Whether a policy that has learned to maximise AlpaSim's notion of progress and comfort drives better on a real road is not a claim the release makes, and nothing here would establish it.

NVIDIA has meanwhile launched two open autonomous driving competitions at CVPR 2026, with a public leaderboard for model evaluation, and published further material in NVlabs/alpamayo-recipes and NVlabs/alpagym. That completes a stack in which one company supplies the model, the simulator, the dataset, the training system, the reference reward functions and the scoreboard. Whoever writes the reward function decides what good driving is, and right now the default one comes in the box.