NVIDIA NeMo Curator Enhances Non-English Dataset Preparation for LLM Training

Data curation is critical for developing effective and fair large language models (LLMs). High-quality, diverse training data directly impacts LLM performance by addressing issues like bias, inconsistencies, and redundancy. NVIDIA has recently announced the open-source release of the NVIDIA NeMo Curator, a data curation library designed to enhance LLM training accuracy through scalable and efficient dataset preparation.

Importance of Data Curation

When training localized multilingual LLMs, particularly for low-resourced languages, web-crawled data such as OSCAR is vital. However, this data often contains noise, irrelevant content, duplicates, and formatting issues. Effective data curation is essential to mitigate these problems and ensure high-quality LLM performance. The NeMo Curator offers a customizable and modular interface that simplifies pipeline expansion and accelerates model convergence by preparing high-quality tokens.

NeMo Curator Overview

The NeMo Curator leverages GPU-accelerated data curation using Dask and RAPIDS, enabling users to mine high-quality text at scale from massive uncurated web corpora as well as custom datasets. For instance, a data curation pipeline can be constructed using the Thai Wikipedia dataset, a smaller subset of the Wikipedia dataset, which can be processed on a single GPU. Wikipedia is considered high-quality for LLM pretraining due to its accurate, well-structured content. NeMo Curator enhances this by detecting and filtering low-quality documents, ensuring only the best data is used for training.

Data Curation Pipeline Example

Using the Thai Wikipedia as an example, the data curation pipeline involves several steps:

1. Download and extract the dataset to a JSONL file.
2. Perform preliminary data cleaning, including language separation and Unicode text fixes.
3. Advanced cleaning, such as GPU-accelerated exact and fuzzy deduplication, and heuristic filtering.

For the complete code sample for this tutorial, see the NVIDIA NeMo Curator GitHub repo.

Prerequisites and Setup

To use GPU-accelerated deduplication, the following hardware setup is recommended:

• NVIDIA GPU: This tutorial uses the NVIDIA A10 24GB GPU.
• CUDA and NVIDIA Drivers: CUDA 12.2 with Driver 535.154.05.
• Ubuntu 22.04.
• NVIDIA-container-toolkit version 1.14.6.

To install the NeMo Curator library, run the following commands:

git clone https://github.com/NVIDIA/NeMo-Curator.git
cd NeMo-Curator
pip install --extra-index-url https://pypi.nvidia.com "[cuda12x]"

Advanced Data Cleaning

Advanced data curation techniques such as deduplication and heuristic filtering are applied to yield better data quality. For example, the ExactDuplicates class removes identical documents using GPU-accelerated implementations from the RAPIDS cuDF library. Similarly, the FuzzyDuplicates class removes near-identical documents using the MinhashLSH algorithm, which is computationally efficient.

Heuristic Filtering

Heuristic filtering helps remove low-quality content from the dataset using simple, efficient-to-compute rules. At the time of publication, NeMo Curator provides 24 heuristics for natural languages and eight for coding languages. These filters can be applied using a YAML config file to define the filters for heuristic filtering.

Next Steps

The tutorial demonstrated how to construct a sample data curation pipeline for Thai Wikipedia data. For more information and examples, see the collection of data curation examples on GitHub. Enterprises can also request access to the NVIDIA NeMo Curator microservice, which provides streamlined performance and scalability.