NAVER Enhances Vertical Services with NVIDIA TensorRT-LLM
NAVER, a prominent South Korean search engine company, has integrated NVIDIA's TensorRT-LLM technology to enhance the performance and efficiency of its SLM-based vertical services, according to NVIDIA's official blog. NAVER Place, a geo-based platform, offers comprehensive information about businesses and points of interest in South Korea, enabling users to search for locations, leave reviews, and make bookings or orders in real-time.
Optimizing SLM-Based Vertical Services
To improve the usability of NAVER Place's vertical services, which are specialized for Place, Map, and Travel, the company employs small language models (SLMs). These models are optimized for specific domain tasks and require less memory and computational power compared to large language models (LLMs). The integration of NVIDIA TensorRT-LLM allows NAVER to enhance the inference performance of these SLMs on the NVIDIA Triton Inference Server.
The tailored SLMs, utilizing NAVER's proprietary dataset, generate summaries from user reviews and provide detailed microreviews about each location. This helps users understand what each place is like through concise, AI-generated insights.
NVIDIA TensorRT-LLM: A Game-Changer
NVIDIA TensorRT-LLM is a powerful tool that accelerates and optimizes the inference performance of language models on NVIDIA GPUs. It supports in-flight batching to maximize throughput and employs memory optimization techniques to enhance efficiency. NAVER's adoption of TensorRT-LLM has led to superior performance in throughput, time to first token (TTFT), and time per output token (TPOT) compared to other inference solutions.
The technology consistently delivers strong performance across various input lengths and output token scenarios, significantly boosting the capabilities of NAVER Place's services.
Balancing Throughput and Latency
In optimizing inference performance, NAVER has explored strategies to balance throughput and latency, such as adjusting batch sizes and employing memory optimization techniques like paged KV cache and in-flight batching. These strategies ensure that the system can handle both real-time requests requiring low latency and bulk-processing scenarios demanding higher throughput.
For instance, in real-time services like POI matching, NAVER uses a small model size and older GPU architectures to achieve minimal latency, demonstrating the flexibility and robustness of their approach.
Enhancing Usability with Triton Inference Server
NAVER Place's SLM engine, built with TensorRT-LLM, is served on the Triton Inference Server. This server provides features like ensemble models and Business Logic Scripting (BLS) for creating efficient pipelines. NAVER has leveraged these features to maximize usability and optimize the Triton-based LLM serving development.
By standardizing input/output schemas and modularizing the business logic, NAVER has improved data validation, code maintainability, and development productivity, leading to a more robust and efficient system.
In summary, NAVER's collaboration with NVIDIA has significantly enhanced the performance and efficiency of its SLM-based vertical services, making NAVER Place a more user-friendly platform. The integration of NVIDIA TensorRT-LLM has allowed NAVER to maximize GPU utilization and optimize its services for better user experience.
For further details, you can visit the NVIDIA blog.