pg_tokenizer 1.1.0: Consistent Caches and Analysis Limits

2026-11-08 By Ruohang Feng In Release Notes 712 words 4 min

Ruohang FengCreator of Pigsty & founder of PGSTY, PostgreSQL contributor & database architect (@Vonng).

pg_tokenizer 1.1.0 adds WAL-replicated definition invalidation, fixes prepared-transaction cache cleanup, separates runtime vocabulary privileges, and introduces configurable analysis budgets.

pg_tokenizer 1.1.0 improves definition-cache consistency, text-analysis correctness, and configuration privileges. It supports PostgreSQL 14–18 and retains the existing INT[] tokenization and native tsvector interfaces.

Release scope

This release is accepted for fresh installations. Upgrade compatibility, per-schema default-privilege migration, snapshots spanning an upgrade, and staging manifests left by failed build preflight are outside this release’s acceptance scope. Existing upgrade scripts remain available for separate evaluation.

Extension & Release Links

GitHub Release: pg_tokenizer v1.1.0 · Full changelog
Documentation: Installation · SQL reference · Models
Source: Repository · Versioned release notes

The GitHub release is available as a draft, which requires repository access. Its download assets become publicly available when the draft is published.

Definition caches and prepared transactions

Definition caches now validate against a transactional catalog version replicated through WAL. Analyzer, tokenizer, model, stopword, and synonym lookups read the version and definitions under the same SQL snapshot. A standby observes committed changes after replay reaches them and its query takes a suitable snapshot; promotion preserves this behavior.

The preparing backend now clears derived objects and validation state at PREPARE TRANSACTION. This fixes a specific failure sequence: direct catalog DML loads configuration X into a session’s cache, another connection rolls back the prepared transaction, and a later transaction commits configuration Y with the same version number. The original connection now loads Y. The regression verifies that Hello World returns {"H","W"} for the replacement regex configuration instead of retaining {"hello","world"}.

Read-only transactions can still be prepared. Definition-changing APIs continue to reject PREPARE. Ordinary transaction commits retain the normal cache-reuse path, and direct catalog DML remains an administrative operation because it bypasses API configuration validation.

Other changes

Area	Change
Runtime schema	Vocabulary tables and their sequences live in `tokenizer_data`; database owners do not need CREATE on `tokenizer_catalog`.
Parallel queries	Catalog-writing functions are `PARALLEL UNSAFE`; read-only analysis remains `PARALLEL SAFE`.
PostgreSQL dictionaries	New `pg_dict` definitions store schema-qualified names, so later callers’ search paths cannot select a different dictionary.
Lindera privileges	Loading dictionary files requires superuser or `pg_read_server_files` privileges. Matching models already loaded or preloaded remain reusable.
Custom vocabularies	Long tokens split at UTF-8 boundaries; trigger output follows the target column’s assignment semantics.
Native full-text search	Repeated lexemes are aggregated before conversion to `tsvector`.
Jieba	`full` and `precise` select the intended algorithms; the default `search` mode is unchanged.
Packaging	Safe shared-library installation examples, a PGXN-compatible source archive, and tests of the actual DEB and ZIP contents.

Analysis budgets

The following session-settable limits reject oversized input or output with SQLSTATE 54000:

Setting	Default
`pg_tokenizer.max_input_bytes`	16 MiB
`pg_tokenizer.max_output_tokens`	1,000,000
`pg_tokenizer.max_output_bytes`	64 MiB
`pg_tokenizer.max_piece_bytes`	1 MiB

These budgets do not impose a hard memory limit or a bounded cancellation latency inside third-party libraries. Load trusted model configurations and bound documents in the application. Issue #13 remains open for this limitation; see the resource-budget reference.

Installation and packages

GitHub DEB and ZIP packages target PostgreSQL 14–18 on x86_64/amd64 and aarch64/arm64 and are built on Ubuntu 24.04 with embedded IPADIC. Use a matching native package or a source build on other distributions. Native Pigsty RPM/DEB packages do not embed Lindera dictionaries; supply a dictionary compiled with Lindera 6.

The GitHub release also includes pg_tokenizer-1.1.0.zip, its SHA-256 file, and SHA256SUMS covering the binary packages and source ZIP. PGXN publication is separate from preparing this source archive.

Install the matching files, add pg_tokenizer.so to shared_preload_libraries while preserving existing entries, and restart PostgreSQL. Then run:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer;
SET search_path TO "$user", public, tokenizer_catalog;

SELECT create_text_analyzer(
    'words',
    'character_filters = ["to_lowercase"]
pre_tokenizer = "unicode_segmentation"'
);
SELECT apply_text_analyzer('Hello World', 'words');

The result is {hello,world}. Neither vchord nor vchord_bm25 is required. Follow the installation guide for package selection and server configuration.

If reusing persisted output from older versions, note that Jieba’s full and precise modes were swapped in 1.0.0 and earlier. Regenerate output produced with either mode and rebuild dependent indexes.

Validation

Draft creation is gated on a successful Check run for the tagged commit across PostgreSQL 14–18 and both Linux architectures. Each release build installs its DEB, compares the installed files with its ZIP, and runs SQLLogicTest, integration, preload, and physical-standby tests against the packaged library.

The prepared-transaction regression was also verified on macOS arm64/PostgreSQL 18.6: the same test fails against the unpatched baseline and passes after the fix, including version reuse and normal COMMIT PREPARED.