Found me on Upwork? If you found me through Upwork and want to work together, please contact me through Upwork. Messages from Upwork clients sent outside Upwork will not be answered.
Localization notes🌏 Localization

Building a Vietnamese LLM Dataset: The 6 Quality Pitfalls

Six pitfalls corrupt Vietnamese LLM datasets: encoding mojibake, Unicode normalization, web noise, tokenization, dialect gaps, annotation. Fix checklist inside.

Dao Huy (Lucas)5 min read8 sections
Building a Vietnamese LLM Dataset: The 6 Quality Pitfalls

💡 Six pitfalls silently corrupt Vietnamese LLM training data before a single training step: legacy encoding mojibake, Unicode NFC/NFD mismatches, web crawl noise, syllable tokenization errors, dialect imbalance, and LLM annotation failures. Fixing them costs far less than retraining after deployment. This checklist lets your data team identify and patch each one.

Key takeaways

  • Normalize all text to UTF-8 NFC on ingest. Legacy VISCII, Windows-1258, and NFD-encoded Vietnamese characters look identical on screen but break deduplication and tokenization.
  • Expect to discard roughly 90% of raw CommonCrawl Vietnamese text. Unfiltered documents can be 99%+ symbols; classifier-based filtering is required, not optional.
  • Run a Vietnamese word segmenter (RDRSegmenter or VnCoreNLP) before subword tokenization. Whitespace splitting alone fragments multi-syllable words and degrades downstream performance.
  • Sample all three dialect regions: Northern (Hanoi), Central (Hue), and Southern (Ho Chi Minh City). Northern-only corpora bias the model against 40% of Vietnam's population.
  • LLM-based annotation underperforms on low-resource Vietnamese. Budget for native human reviewers to validate at minimum 10% of each annotation batch.

What makes Vietnamese uniquely hard to build a clean LLM dataset for?

Vietnamese LLM dataset quality is uniquely challenging because five problems rarely found together in European languages all converge here. Vietnamese has six tones marked by stacked diacritics - up to three Unicode combining marks per character. It was written in three different byte encodings before UTF-8 became universal. Its syllable-level spacing looks like word segmentation but is not. It has three regional dialects with distinct phonology and vocabulary. And it remains under-resourced relative to its 97 million speakers.

Each challenge maps to a concrete failure mode in a data pipeline. The good news: every failure mode is preventable. The six pitfalls below each come with a fix your engineering team can implement without outside help.

What encoding and normalization pitfalls corrupt Vietnamese training data?

Before UTF-8 became the web standard, Vietnamese used several single-byte encodings: VISCII (Vietnamese Standard Code for Information Interchange), Windows-1258 (CP1258), TCVN 5712, and VNI. Legacy Vietnamese websites, government databases, and digitized documents still serve content in these encodings. A pipeline that assumes UTF-8 will silently corrupt diacritics into mojibake (garbled characters) on every affected document.

The encoding fix: use a library like chardet (Python) to detect encoding on each document, then convert immediately to UTF-8 before any processing. Apply this at ingest, not as an afterthought during cleaning.

The second, subtler problem is Unicode normalization. Vietnamese diacritics can be stored two ways. NFC (precomposed) stores the character ề as a single code point (U+1EC1). NFD (decomposed) stores it as three separate code points: e + combining circumflex + combining grave accent. Both render identically in every browser. But string hashing and deduplication treat them as different strings, keeping both copies.

The normalization fix: normalize all text to NFC before deduplication, tokenization, and any hashing step. In Python: import unicodedata; text = unicodedata.normalize("NFC", text). The Unicode Normalization FAQ documents which code points are affected and why NFC is the recommended form for text interchange and storage.

How noisy is Vietnamese web data from CommonCrawl and mC4?

Extremely noisy, more so than most teams expect. NVIDIA's NeMo Curator team built a Vietnamese pretraining corpus from CommonCrawl, mC4, OSCAR, Wikipedia, and a news corpus. After Unicode reformatting, heuristic filtering, and classifier-based filtering, they retained roughly 10% of the raw data. Unfiltered documents included content where symbols made up 99.44% of characters and numbers made up 91.53%.

VietVault, a publicly available Vietnamese corpus, started from 6TB of raw CommonCrawl text and produced 27.5GB of curated data after filtering - a reduction of over 99.5%.

A practical three-step noise filter for Vietnamese web data:

  1. Heuristic filtering: remove documents with extreme symbol ratios (>30%), very short length (<50 words), or high whitespace density. This covers the worst noise quickly.
  2. Classifier-based filtering: train a lightweight fastText or BERT-style classifier on clean Vietnamese seed examples vs. noisy content. In the NVIDIA pipeline this step removed 45% of data and is the primary quality lever.
  3. Exact deduplication: hash each document after normalization and remove duplicates. CommonCrawl-based Vietnamese corpora typically have around 8% exact duplicates.

What is the tokenization trap in Vietnamese?

Vietnamese is a monosyllabic isolating language. Spaces separate syllables, not words. The phrase "học sinh" (student) appears as two whitespace-separated tokens: "học" and "sinh". Neither carries its full meaning independently. A tokenizer that splits on whitespace treats each syllable as an independent token, destroying multi-syllable word semantics across the entire corpus.

The standard fix is pre-segmentation. PhoBERT, the leading Vietnamese BERT model, requires RDRSegmenter as a mandatory preprocessing step before BPE encoding. VnCoreNLP is a widely used alternative. Both tools annotate multi-syllable word boundaries, producing "học_sinh" (with underscore) as a single token for the subword tokenizer to handle correctly.

For training corpus pipelines: run word segmentation on a held-out 5% sample first. Compare perplexity with and without pre-segmentation. Most teams see 10-15% improvement in downstream task performance from this single fix, with no change to training infrastructure.

Dialect gaps your model will inherit from Northern-dominant corpora

Standard vi-VN is based on Northern Vietnamese (Hanoi). Northern dialect has six distinct tones. Southern Vietnamese (Ho Chi Minh City) effectively has five: the hỏi and ngã tones merge into a single falling-rising contour. Central Vietnamese (Hue) has its own distinct vowel qualities.

In text data, dialect differences appear in vocabulary (Northern "bố/mẹ" vs. Southern "ba/má" for father and mother), certain orthographic conventions, and in informal online writing from each region. A corpus built almost entirely from Northern news outlets produces a model that generates more natural text for Northern users and may mishandle Southern slang, idioms, and informal register.

For a production Vietnamese model serving national audiences, aim for intentional dialect sampling: Northern news and government documents, Southern social media and e-commerce text, Central regional sources. If budget is limited, Northern plus Southern coverage is the minimum viable balance.

What can you fix in-house, and when is a native Vietnamese specialist worth it?

In-house is entirely viable for pipeline infrastructure. Encoding detection and UTF-8 conversion, Unicode NFC normalization, deduplication hashing, heuristic noise filters, and basic language identification are engineering tasks with no linguistic judgment required. Your data engineers can implement all of them from the specifications in this article.

A native Vietnamese specialist earns the investment at three specific points:

  • Seed annotation for quality classifiers: the classifier that filters low-quality documents needs clean Vietnamese seed examples. An annotator without native Vietnamese fluency cannot reliably judge register, tone-mark correctness, or domain appropriateness in context.
  • Annotation validation: LLM-based annotation shows higher error rates on low-resource Vietnamese than on English. A 10% native-review budget per annotation batch catches systematic failures before they compound across the full dataset.
  • Dialect and domain coverage audit: verifying that your corpus represents Southern informal writing, Northern formal register, and technical domain vocabulary requires a native reviewer who reads all three. A data engineer cannot catch systematic Southern underrepresentation by scanning the text.

For projects requiring Vietnamese AI training data annotation and quality review, the Vietnamese AI training data services page covers annotation, LQA, and corpus review options for English-source teams.

FAQ

What is the minimum corpus size for a clean Vietnamese pretraining dataset?

There is no universal minimum. Teams have successfully trained domain-specific Vietnamese models on as little as 1-5 billion tokens of clean text. Quality matters more than raw size: one billion properly normalized, deduplicated, dialect-balanced Vietnamese tokens outperforms 10 billion tokens of noisy CommonCrawl data with encoding errors. Start with a clean smaller corpus rather than a large noisy dump.

Which language detection library works best for Vietnamese text?

FastText's lid.176.bin model and langdetect are the most commonly used. FastText is faster for batch processing. For web-scraped data with heavy English technical terms mixed in, CLD3 (used in mC4) sometimes misclassifies content; a fastText classifier fine-tuned on Vietnamese web samples handles those edge cases better. Always run language detection after encoding normalization, not before.

Should I use PhoBERT tokenization or SentencePiece for a Vietnamese generative LLM?

For encoder-only models (classification, NER, QA), PhoBERT with RDRSegmenter pre-processing is the established standard. For generative LLMs, SentencePiece trained on pre-segmented Vietnamese text is more practical because it handles out-of-vocabulary tokens gracefully. The critical step: train the SentencePiece vocabulary on pre-segmented text (word-boundary annotated), not raw Vietnamese whitespace-tokenized input. This gives the model meaningful subword units rather than arbitrary syllable fragments.

How do I handle the Northern vs. Southern tone merge in annotation tasks?

In written Vietnamese, the hỏi and ngã tones are always spelled with distinct diacritics, regardless of how they sound in the Southern dialect. Standard written vi-VN follows Northern orthography. Annotation guidelines should specify standard written orthography, not phonetic transcription. For ASR-related annotation, build a consistency check against the standard written form to catch transcription variance between Northern and Southern annotators.

Can I use GPT-4 or Claude to annotate a Vietnamese LLM dataset?

LLMs are useful for bulk pre-annotation in Vietnamese, but error rates are higher on Vietnamese than on English - especially for domain-specific terminology, tone-sensitive sentiment, and dialect recognition. Use LLMs to generate initial labels, then validate with native Vietnamese speakers on at minimum 10% of each batch. For tasks involving formal Hán Việt register or Southern informal register, human validation is essential rather than a cost-cutting consideration.

Official Sources

Dao Huy (Lucas)
Dao Huy (Lucas)
Graduate, University of Foreign Language Studies, University of Da Nang. Signs a certificate of accuracy on every certified translation (USCIS, IRCC).

Written by Dao Huy (Lucas), Vietnamese translator and localization specialist (English, Chinese and French into Vietnamese). See translation services →

Get QuoteWhatsApp