Tokenizers (pg_tokenizer)
pg_tokenizer (maintained by Pigsty at pgsty/pg_tokenizer) is the companion PostgreSQL extension for vchord_bm25 (pgsty/vchord_bm25) that provides full-text analyzers and vocabulary models to convert raw text into integer token ID arrays (INT[]) and sparse bm25vector embeddings.
How Text Analyzers and Models combine to turn raw text into token IDs.
Configure character filters, pre-tokenizers (Unicode, Regex, Jieba), stemmers (30 languages), stopwords, synonyms, and n-grams.
Use built-in models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), train custom corpus models, or import HuggingFace & Lindera models.
Ready-to-use configurations for English, Chinese (Jieba), Japanese (Lindera), and custom domain corpora.
Understand the two-stage pipeline of pg_tokenizer: Text Analyzer and Tokenization Model.
Complete configuration reference for character filters, pre-tokenizers, and token filters in pg_tokenizer.
Pretrained models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), custom corpus-trained models, HuggingFace tokenizers, and Lindera models.
Complete tokenization configurations for English, Chinese (Jieba), Japanese (Lindera), and known limitations.