# Tokenizers (pg_tokenizer)

> Complete guide to pg_tokenizer: text analyzers, character/token filters, pretrained and custom models, and multilingual tokenization.

---

LLMS index: [llms.txt](/llms.txt)

---

**`pg_tokenizer`** (maintained by Pigsty at [`pgsty/pg_tokenizer`](https://github.com/pgsty/pg_tokenizer)) is the companion PostgreSQL extension for [`vchord_bm25` (`pgsty/vchord_bm25`)](https://github.com/pgsty/vchord_bm25) that provides full-text analyzers and vocabulary models to convert raw text into integer token ID arrays (`INT[]`) and sparse `bm25vector` embeddings.


  - [Tokenizer Architecture \& Usage](/bm25/tokenizer/overview/) — How Text Analyzers and Models combine to turn raw text into token IDs.
  - [Text Analyzers \& Filters](/bm25/tokenizer/text-analyzer/) — Configure character filters, pre-tokenizers (Unicode, Regex, Jieba), stemmers (30 languages), stopwords, synonyms, and n-grams.
  - [Tokenization Models](/bm25/tokenizer/models/) — Use built-in models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), train custom corpus models, or import HuggingFace & Lindera models.
  - [Multilingual Recipes \& Limitations](/bm25/tokenizer/multilingual/) — Ready-to-use configurations for English, Chinese (Jieba), Japanese (Lindera), and custom domain corpora.

---

Section pages:

- [Tokenizer Architecture & Usage](/bm25/tokenizer/overview/): Understand the two-stage pipeline of pg_tokenizer: Text Analyzer and Tokenization Model.
- [Text Analyzers & Filters](/bm25/tokenizer/text-analyzer/): Complete configuration reference for character filters, pre-tokenizers, and token filters in pg_tokenizer.
- [Tokenization Models](/bm25/tokenizer/models/): Pretrained models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), custom corpus-trained models, HuggingFace tokenizers, and Lindera models.
- [Multilingual Recipes & Limitations](/bm25/tokenizer/multilingual/): Complete tokenization configurations for English, Chinese (Jieba), Japanese (Lindera), and known limitations.

---

Backlinks:

- [Optimize RAG with Contextual Retrieval in PostgreSQL](/blog/post/optimize-rag-with-contextual-retrieval-in-postgresql/)
- [VectorChord-BM25: Introducing pg_tokenizer—A Standalone, Multilingual Tokenizer for Advanced Search](/blog/post/vectorchord-bm25-introducing-pgtokenizera-standalone-multilingual-tokenizer-for-advanced-search/)
- [VectorChord-BM25: Revolutionize PostgreSQL Search with BM25 Ranking — 3x Faster Than ElasticSearch](/blog/post/vectorchord-bm25-revolutionize-postgresql-search-with-bm25-ranking-3x-faster-than-elasticsearch/)
- [BM25](/bm25/)
- [Overview](/bm25/getting-started/overview/)
- [Quick Start](/bm25/getting-started/quick-start/)