# pg_tokenizer Documentation

> Complete guide to pg_tokenizer: installation, text analyzers, character/token filters, pretrained and custom models, multilingual tokenization, and SQL reference.

---

LLMS index: [llms.txt](/llms.txt)

---

**`pg_tokenizer`** (maintained by Pigsty at [`pgsty/pg_tokenizer`](https://github.com/pgsty/pg_tokenizer)) is a standalone multilingual tokenization extension for PostgreSQL built in Rust (`pgrx`). It provides customizable text analyzers, character/token filters, and vocabulary models to convert raw text into integer token ID arrays (`INT[]`) and sparse `bm25vector` embeddings for [`vchord_bm25`](/docs/bm25/).


  - [Installation](/docs/tokenizer/installation/) — Install pg_tokenizer via Pigsty native Linux packages, Docker, Debian/ZIP archives, or cargo-pgrx.
  - [Architecture \& Usage](/docs/tokenizer/overview/) — How Text Analyzers and Models combine to turn raw text into token IDs.
  - [Text Analyzers \& Filters](/docs/tokenizer/text-analyzer/) — Configure character filters, pre-tokenizers (Unicode, Regex, Jieba), stemmers (30 languages), stopwords, synonyms, and n-grams.
  - [Tokenization Models](/docs/tokenizer/models/) — Use built-in models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), train custom corpus models, or import HuggingFace & Lindera models.
  - [Multilingual Recipes \& Limitations](/docs/tokenizer/multilingual/) — Ready-to-use configurations for English, Chinese (Jieba), Japanese (Lindera), and custom domain corpora.
  - [SQL \& TOML Reference](/docs/tokenizer/reference/) — Complete reference for all tokenizer_catalog SQL functions and TOML configuration options.

---

Section pages:

- [Installation](/docs/tokenizer/installation/): Install pg_tokenizer via Pigsty native Linux packages, Docker, Debian/ZIP archives, or cargo-pgrx.
- [Tokenizer Architecture & Usage](/docs/tokenizer/overview/): Understand the two-stage pipeline of pg_tokenizer: Text Analyzer and Tokenization Model.
- [Text Analyzers & Filters](/docs/tokenizer/text-analyzer/): Complete configuration reference for character filters, pre-tokenizers, and token filters in pg_tokenizer.
- [Tokenization Models](/docs/tokenizer/models/): Pretrained models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), custom corpus-trained models, HuggingFace tokenizers, and Lindera models.
- [Multilingual Recipes & Limitations](/docs/tokenizer/multilingual/): Complete tokenization configurations for English, Chinese (Jieba), Japanese (Lindera), and known limitations.
- [SQL & TOML Reference](/docs/tokenizer/reference/): Complete reference for all tokenizer_catalog SQL functions and TOML configuration options in pg_tokenizer.

---

Backlinks:

- [All-in-one VectorChord Suite: Building Production-Ready RAG Solutions Directly in PostgreSQL](/blog/post/all-in-one-vectorchord-suite-building-production-ready-rag-solutions-directly-in-postgresql/)
- [Optimize RAG with Contextual Retrieval in PostgreSQL](/blog/post/optimize-rag-with-contextual-retrieval-in-postgresql/)
- [VectorChord-BM25: Introducing pg_tokenizer—A Standalone, Multilingual Tokenizer for Advanced Search](/blog/post/vectorchord-bm25-introducing-pgtokenizera-standalone-multilingual-tokenizer-for-advanced-search/)
- [VectorChord-BM25: Revolutionize PostgreSQL Search with BM25 Ranking — 3x Faster Than ElasticSearch](/blog/post/vectorchord-bm25-revolutionize-postgresql-search-with-bm25-ranking-3x-faster-than-elasticsearch/)
- [Documentation](/docs/)
- [About](/docs/about/)
- [pg_bestmatch](/docs/bestmatch/)
- [SQL Reference](/docs/bestmatch/reference/)
- [vchord_bm25](/docs/bm25/)
- [Overview](/docs/bm25/getting-started/overview/)
- [Quick Start](/docs/bm25/getting-started/quick-start/)
- [vchord](/docs/vchord/)
- [Getting Started](/docs/vector/getting-started/)
- [Querying & Filtering](/docs/vector/querying/)