Tokenizers (pg_tokenizer) on Vchord

Tokenizer Architecture & Usage

Mon, 01 Jan 0001 00:00:00 +0000

Two-Stage Tokenization Pipeline

A tokenizer in pg_tokenizer consists of up to two stages that transform a raw TEXT string into an INT[] array of integer token IDs:

flowchart LR
    A["Raw Text\\n(TEXT)"] --> B["Stage 1: Text Analyzer\\n1. character_filters\\n2. pre_tokenizer\\n3. token_filters"]
    B --> C["Normalized Tokens\\n(TEXT[])"]
    C --> D["Stage 2: Model\\nVocabulary Mapping / Subword BPE"]
    D --> E["Token ID Array\\n(INT[] → bm25vector)"]

Text Analyzer (text_analyzer): Splits raw text into a TEXT[] array of normalized string tokens by applying a sequential pipeline of:
- Character Filters (character_filters): Transform the raw string before splitting (e.g., to_lowercase, unicode_normalization).
- Pre-tokenizer (pre_tokenizer): Splits the string into initial substrings (unicode_segmentation, regex, or Chinese jieba).
- Token Filters (token_filters): Filter or transform individual tokens (skip_non_alphanumeric, stemmer, stopwords, synonym, pg_dict, ngram).
Model (model): Maps the string tokens (or raw text, for standalone subword models like llmlingua2, bert_base_uncased, gemma2b, HuggingFace, or lindera) into integer token IDs (INT[]).

Note

Pretrained subword models (llmlingua2, bert_base_uncased, gemma2b), HuggingFace models, and Lindera models include their own built-in tokenization rules and do not require a separate text_analyzer stage.
Dictionary/corpus models (wiki_tocken and custom models created via create_custom_model) require a text_analyzer stage to split and normalize text before looking up token IDs in the model’s vocabulary table.

Creating and Using a Tokenizer

All configuration in pg_tokenizer uses TOML format passed as a dollar-quoted string ($$ ... $$).

Text Analyzers & Filters

Mon, 01 Jan 0001 00:00:00 +0000

A Text Analyzer processes raw text through three sequential stages:

Character Filters (character_filters) — Zero or more character-level transformations on the full text string.
Pre-tokenizer (pre_tokenizer) — Exactly one splitter that breaks the string into initial tokens.
Token Filters (token_filters) — Zero or more token-level transformations, expansions, or filters.

SELECT create_text_analyzer('analyzer1', $$
character_filters = [
    "to_lowercase",
    { unicode_normalization = "nfkd" },
]
pre_tokenizer = "unicode_segmentation"
token_filters = [
    "skip_non_alphanumeric",
    { stopwords = "nltk_english" },
    { stemmer = "english_porter2" },
]
$$);

Tip

You can test any text analyzer directly using SELECT apply_text_analyzer('Sample text here', 'analyzer1'); to inspect the resulting TEXT[] array before wiring it to a model.

Tokenization Models

Mon, 01 Jan 0001 00:00:00 +0000

In pg_tokenizer, a Model maps string tokens (or raw text) to integer token IDs (INT[]). You can choose from four built-in pretrained models, train a custom vocabulary model directly on your PostgreSQL table, import any HuggingFace tokenizer.json, or use a Lindera morphological model.

1. Built-in Pretrained Models

Model Name	Source / Architecture	Requires `text_analyzer`?	Memory Footprint	Best For
`llmlingua2`	`microsoft/llmlingua-2-xlm-roberta-large-meetingbank` (XLM-RoBERTa multilingual subword)	No (forbidden)	~200 MB (default preloaded)	Multilingual general text out of the box without training a custom vocabulary.
`bert_base_uncased`	`google-bert/bert-base-uncased` (WordPiece, 30,522 vocab)	No (forbidden)	~15 MB	General English text; fast startup and compact vocabulary.
`wiki_tocken`	`Snowolf/wiki-tocken` (Built-in English Wikipedia token dictionary)	Yes (required)	~30 MB	English full-text search combined with custom stemmers, stopwords, and synonyms.
`gemma2b`	`google/gemma-2b` (SentencePiece BPE, 256k vocab)	No (forbidden)	~100 MB	Code, technical identifiers, and multilingual LLM workloads.

Example: Using `bert_base_uncased` or `llmlingua2`

SELECT create_tokenizer('bert_tok', $$
model = "bert_base_uncased"
$$);

SELECT tokenize('VectorChord-BM25 brings Block-WeakAnd to PostgreSQL.', 'bert_tok');

2. Custom Corpus Models (`create_custom_model`)

For maximum BM25 accuracy and minimal memory footprint, you can build a custom vocabulary model directly from your own table column. Unlike subword LLM tokenizers (which split rare domain words into multiple sub-tokens), a custom model assigns a dedicated 1:1 integer ID to every unique normalized token produced by your text_analyzer—and weighs only a few megabytes in memory.

Multilingual Recipes & Limitations

Mon, 01 Jan 0001 00:00:00 +0000

This page provides production-ready tokenization recipes for English, Chinese, and Japanese full-text BM25 search, followed by important operational limitations to keep in mind.

Recipe 1: English with Stemming & Stopwords (`wiki_tocken`)

Combines Unicode word segmentation, NLTK stopwords removal, and Porter2 stemming with the built-in Wikipedia vocabulary (wiki_tocken):

SELECT create_tokenizer('eng_wiki', $$
model = "wiki_tocken"

[text_analyzer.character_filters.to_lowercase]
[text_analyzer.pre_tokenizer.unicode_segmentation]
[[text_analyzer.token_filters.skip_non_alphanumeric]]
[[text_analyzer.token_filters.stopwords]]
words = "nltk_english"
[[text_analyzer.token_filters.stemmer]]
kind = "english_porter2"
$$);

SELECT tokenize('PostgreSQL databases are running analytical and vector workloads.', 'eng_wiki');

Recipe 2: English Custom Corpus Model with Trigger

From the upstream pg_tokenizer examples, this workflow builds a domain-specific English vocabulary from your documents table and keeps embeddings updated automatically via trigger:

Tokenizers (pg_tokenizer) on Vchord

Tokenizer Architecture & Usage

Two-Stage Tokenization Pipeline

Creating and Using a Tokenizer

Text Analyzers & Filters

Tokenization Models

1. Built-in Pretrained Models

Example: Using bert_base_uncased or llmlingua2

2. Custom Corpus Models (create_custom_model)

Multilingual Recipes & Limitations

Recipe 1: English with Stemming & Stopwords (wiki_tocken)

Recipe 2: English Custom Corpus Model with Trigger

Example: Using `bert_base_uncased` or `llmlingua2`

2. Custom Corpus Models (`create_custom_model`)

Recipe 1: English with Stemming & Stopwords (`wiki_tocken`)