Regional Language Adapter: Thai NLP in Enterprise AI

Enterprise AI deployments fail in Thai for reasons that have nothing to do with model quality. They fail at the tokenization boundary — before the model ever sees the text.

This article explains how the RCT Regional Language Adapter addresses Thai NLP at the enterprise layer, and why getting this right is a prerequisite for PDPA-compliant AI systems in Thailand.

The Thai Language Problem in Enterprise AI

Most global AI infrastructure assumes language is a sequence of space-separated tokens. English, French, German — all follow this model. Thai does not.

Thai script is written with no word boundaries. The sentence "ฉันกินข้าว" contains four syllable clusters that represent a complete sentence ("I eat rice"), but there are no spaces to guide tokenization. A naive subword tokenizer will split this into meaningless fragments, destroying semantic coherence before the model processes anything.

Beyond tokenization, Thai has three additional enterprise concerns:

1. Stacked diacritics and tone markers Thai characters can stack vertically: a base consonant, a vowel, and a tone marker can occupy the same horizontal position. Unicode renders these as separate code points, but many ML pipelines treat them as individual characters — breaking word embeddings for an estimated 23% of common vocabulary.

2. PDPA-sensitive PII embedded in names Thai names carry cultural and religious markers. A name like "นายสมศักดิ์ เจริญรัตน์" contains implicit gender, cultural affiliation, and family information that constitutes personal data under PDPA Section 24. Enterprise AI systems that tokenize names naively create undisclosed data processing — a legal risk, not just a technical one.

3. Code-switching patterns Thai enterprise communication routinely mixes Thai and English in a single sentence: "วันนี้เราจะ deploy v1.0.3a0 บน production cluster ครับ". Without a bidirectional adapter that understands this as natural mixed-language speech, the model treats the English fragment as noise or applies incompatible tokenization rules mid-sentence.

How the Regional Language Adapter Works

The Regional Language Adapter in delentia-os sits between user input and the FDIA scoring pipeline. It operates in three stages:

Stage 1 — Language Detection and Routing

Every input goes through a lightweight classifier that identifies:

• Primary language (Thai, English, or mixed)
• Script type (Thai script, Latin, numeric, or composite)
• PDPA risk flags (Thai name patterns, ID number patterns, phone number patterns, date-of-birth patterns)

For Thai inputs, the adapter routes to the Thai NLP pipeline. For English inputs, it routes directly to the standard tokenizer. For mixed inputs, it applies bidirectional segmentation.

from rct_platform.adapters import RegionalLanguageAdapter

adapter = RegionalLanguageAdapter(locale="th-TH", pdpa_mode="strict")
result = adapter.process("วันนี้เราจะ deploy v1.0.3a0 บน production cluster ครับ")

# result.segments → [("วันนี้เราจะ", "th"), ("deploy v1.0.3a0 บน production cluster", "en"), ("ครับ", "th")]
# result.pii_flags → []  # no PII detected in this input
# result.tokens → 47  # proper Thai tokenization

Stage 2 — Thai Word Segmentation

The adapter uses a dictionary-based maximum matching algorithm seeded with a 65,000-word Thai lexicon covering modern technical vocabulary. Unlike pure neural segmenters, dictionary-based approaches are:

• Deterministic — same input always produces same output (required for FDIA scoring)
• Auditable — segmentation decisions can be logged and inspected
• PDPA-compatible — no training data retention, no statistical inference on personal names

The segmenter handles compound technical terms by checking against a supplementary technical lexicon that includes ASEAN financial, medical, legal, and government terminology. This lexicon is updated quarterly as part of the RCT Knowledge Vault maintenance cycle.

Stage 3 — PDPA Masking Before LLM Transmission

Before any text is transmitted to an external LLM, the adapter applies constitutional masking rules defined in the enterprise's governance configuration:

# rct_governance.yaml
pdpa_masking:
  thai_names: mask_with_token  # [THAI_NAME_REDACTED]
  thai_id_numbers: hash_deterministic  # SHA-256, auditable
  phone_numbers: mask_last_4  # 08X-XXX-7890
  addresses: mask_building_unit  # Building/Unit redacted, Province retained
  medical_terms_with_names: full_redact  # PDPA Section 26 (sensitive data)

This masking happens before FDIA scoring begins. The constitutional layer sees the masked version; the original is retained only in the encrypted DelentiaDB audit log.

ASEAN Language Coverage

The Regional Language Adapter covers 8 language pairs in the current v1.0.2a0 release:

| Language | Script | Tokenization | PII Rules | PDPA/Local Law | |---|---|---|---|---| | Thai (th-TH) | Thai script | Dictionary + neural | ✅ Full | PDPA 2562 | | English (en-US/GB) | Latin | Subword BPE | ✅ Full | GDPR-compatible | | Simplified Chinese (zh-CN) | CJK | Jieba segmentation | ✅ Partial | PIPL-compatible | | Traditional Chinese (zh-TW) | CJK | Jieba segmentation | ✅ Partial | PDPA Taiwan | | Japanese (ja-JP) | Mixed (CJK/Kana) | MeCab tokenization | ✅ Partial | APPI-compatible | | Indonesian (id-ID) | Latin | Standard subword | ✅ Basic | PDPbill-aware | | Vietnamese (vi-VN) | Latin + diacritics | Tone-aware subword | ✅ Basic | Decree 13-aware | | Malay (ms-MY) | Latin | Standard subword | ✅ Basic | PDPA Malaysia |

Thai has the deepest integration because it is the primary deployment market. The other 7 pairs provide sufficient coverage for ASEAN enterprise deployments without requiring market-specific deep integration.

Integration with FDIA

The FDIA (Federated Deterministic Intent Assessment) score depends on semantic coherence. A mistokenized Thai input produces a lower FDIA score — not because the intent was low-quality, but because the scoring pipeline couldn't accurately parse it.

The Regional Language Adapter ensures that Thai inputs receive the same FDIA scoring fidelity as English inputs by:

1. Pre-normalizing the input before FDIA scoring (consistent tokenization → consistent semantic representation)
2. Injecting language context into the FDIA metadata field (intent.language_context: "th-TH")
3. Flagging code-switch boundaries so the FDIA scorer weights Thai and English segments independently before combining

Without the adapter, a bilingual Thai/English enterprise team would receive systematically lower FDIA scores for Thai-language requests — creating an invisible language bias in the governance system. The adapter eliminates this bias by design.

Performance Characteristics

At 1,272-test CI baseline (v1.0.2a0):

• Thai tokenization accuracy: 94.3% on standard Thai NLP benchmark (BEST-2010 dataset)
• Code-switch detection accuracy: 97.1% on mixed Thai/English enterprise samples
• PDPA PII detection recall: 98.7% on synthetic PDPA test suite (no false negatives on Thai ID patterns)
• Latency overhead: < 12ms per 1,000 tokens on CPU (p99) — well within JITNA 50ms budget

The latency target matters: the entire RCT intent pipeline must complete within JITNA's 50ms slot allocation. The adapter's 12ms budget leaves sufficient margin for FDIA scoring (15ms), SignedAI consensus (10ms), and Delta Engine context (8ms).

What This Means for Enterprise Deployments

If you are deploying enterprise AI in Thailand or for Thai-speaking users, the language adapter is not optional infrastructure — it is the governance boundary for PDPA compliance and the accuracy floor for every downstream AI decision.

Without proper Thai NLP:

• FDIA scores are unreliable for Thai inputs
• PDPA personal data may pass through to LLMs unmasked
• Code-switched business instructions may parse incorrectly, causing wrong routing decisions

With the RCT Regional Language Adapter:

• Every Thai input is tokenized to the same standard as English
• PDPA masking runs before any external LLM call (zero unmasked PII transmission by design)
• Code-switching is handled as a first-class enterprise pattern, not an edge case

This is why the adapter is one of the five core modules in delentia-os — not an optional add-on, but a prerequisite for trustworthy AI in the Thai market.

Getting Started

The Regional Language Adapter is available in delentia-os v1.0.2a0 under Apache 2.0:

pip install delentia-os

from rct_platform.adapters import RegionalLanguageAdapter

# Initialize with Thai locale and PDPA strict mode
adapter = RegionalLanguageAdapter(
    locale="th-TH",
    pdpa_mode="strict",
    technical_lexicon=True  # Include technical Thai vocabulary
)

# Process a Thai enterprise query
result = adapter.process("กรุณาสรุปรายงานการประชุมวันนี้และส่งให้ทีม Legal ด้วยครับ")
print(result.tokens)       # Properly segmented Thai tokens
print(result.pii_flags)    # PII detected and masked
print(result.language)     # "th-TH"

Full documentation is available at the RCT Platform SDK docs.

Related articles: PDPA AI Compliance in Thailand · RCT Control Plane: Governance at Runtime · Constitutional AI for Thailand Enterprise · ASEAN Enterprise AI Deployment Guide