NVIDIA Mission Control 2.5 extends autonomous recovery to Kubernetes
NVIDIA said on October 26, 2026, that Mission Control 2.5 extends its autonomous recovery engine to Kubernetes-based workloads managed with NVIDIA Run:ai, moving a resiliency framework that was previously limited to Slurm-based high-performance computing jobs into Kubernetes environments. The company framed the update as part of a broader push toward self-healing AI factory infrastructure, where large training runs can continue with less manual intervention when hardware or software faults arise.
Evidence and context
According to NVIDIA’s technical blog published October 26, Mission Control’s autonomous recovery engine combines two services: autonomous job recovery and autonomous hardware recovery. In the Kubernetes setup, Run:ai acts as the default scheduler, while the job-recovery service uses workload context, failure history and job topology to decide whether a workload should restart, retry, re-queue or be placed elsewhere. The hardware-recovery layer monitors nodes and GPUs, isolates anomalies from scheduling and runs diagnostics or remediation so suspect equipment does not keep circulating into new jobs.
The company described three recovery paths. In a hardware-fault scenario during training, the system can restart or reschedule workloads onto healthy nodes while the faulty node is isolated and diagnosed. For transient software faults, it can retry according to scheduler policy without quarantining healthy hardware. If degradation is detected early, it can remove a node from scheduling before a job lands on it. NVIDIA said the separation of scheduling, job context and hardware-health functions is intended to make the model portable across orchestrators.
Release notes in the verified background material add technical detail beyond the blog post, including Run:ai 2.26, Base Command Manager 11.34.0, recovery integration with the KAI scheduler using Kubeflow PyTorch training operators, and NVRx 0.6.0 asynchronous checkpointing support. NVIDIA also highlights centralized Keycloak authentication improvements and security and Kubernetes hardening updates in this release.
The same release notes also set limits on the update. Support for Vera Rubin NVL72 is described as early access for essential Base Command Manager bring-up only, and Run:ai and the Autonomous Resiliency Engine are not supported on that platform in this release. Minerva network-telemetry and AIOps, along with NVIDIA Dynamic Power Software, are scoped as pre-release betas rather than standard components.