NVIDIA details edge AI workflow for DGX Spark and TensorRT Edge-LLM

NVIDIA said developers can now prototype and serve edge AI models on DGX Spark with a few commands, using its TensorRT Edge-LLM runtime to prepare and run Nemotron-3.5 Lightning for embedded workloads. The company paired the tutorial with benchmark data showing faster token generation when DSpark speculative decoding is enabled, while keeping time to first token broadly unchanged.

Evidence and context

In a blog post published on 30 October 2026, NVIDIA said DGX Spark is available in a 64GB configuration and can be used to develop agents and physical AI applications that must operate within tight memory, power and latency budgets. The company said TensorRT Edge-LLM is a lightweight C++ inference runtime intended for embedded deployments and is designed to carry the same inference framework from Spark to platforms including Jetson Thor, DRIVE AGX Thor and IGX Thor.

The tutorial walks through starting from a virtual environment, installing tensorrt-edgellm[server]==0.11.1, verifying TensorRT compatibility, and then launching a chat session with nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4. NVIDIA said the first launch requires downloading about 21GB of model files and building engines, with cached engines reused on later runs. The company also showed how to expose the model through a local OpenAI-compatible API on 127.0.0.1:8000 for application prototyping.

NVIDIA said the examples were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11, and that the environment requires NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10 to 3.12. The post did not present independent verification of those setup claims, but framed them as the supported prerequisites for the walkthrough.

The performance table in the post compared DSpark off and on for three workloads on DGX Spark 64GB, measured with NVFP4, thinking enabled, batch size one, one concurrent streaming request and a 128-token output limit. With DSpark off, median time to first token was 201.0 ms and mean generation speed was 73.3 tokens per second for SpeedBench 1K; 329.5 ms and 72.8 tokens per second for SpeedBench 2K; and 98.1 ms and 73.4 tokens per second for HumanEval prompts. With DSpark on, the corresponding figures were 203.0 ms and 108.5 tokens per second; 331.6 ms and 105.3 tokens per second; and 98.7 ms and 126.9 tokens per second. NVIDIA said DSpark used three draft tokens in the tested configuration.

The company also reported peak total system memory use during inference with DSpark of 27.9GB, including the operating system, on DGX Spark 64GB running DGX OS 7.6.0 based on Ubuntu 24.04.4 LTS. Because the data comes from NVIDIA’s own benchmarking on its own hardware and software stack, it should be read as vendor-reported performance rather than an independent production benchmark.

The post closes by pointing developers to other supported models, including vision-language, speech recognition and text-to-speech systems, but does not provide a dated follow-up catalyst. Timing for any next step is therefore unknown.