pg_tokenizer 1.0.0: First Release Under PGSTY

Tue, 03 Nov 2026 00:00:00 +0000

pg_tokenizer 1.0.0 is the first release maintained by PGSTY in pgsty/pg_tokenizer under the Apache-2.0 license. It connects the multilingual analyzers and vocabulary models to PostgreSQL’s native full-text search, makes corpus-trained custom models safe under concurrent writes, and lets non-superuser database owners manage their own tokenizer definitions.

The existing tokenize(text, tokenizer_name) interface still returns INT[]. Applications using the historical vchord_bm25 0.3.x bm25vector workflow can retain that interface; applications using GIN/GiST or the vchord_bm25 1.0.0 API can use the new tsvector functions. pg_tokenizer does not require either BM25 or vector-search extensions.

vchord_bm25 1.0.0: First Release Under PGSTY

Tue, 03 Nov 2026 00:00:00 +0000

vchord_bm25 1.0.0 is the first release maintained by PGSTY in pgsty/vchord_bm25 under the AGPL-3.0 license. It replaces the custom bm25vector data type and fixed bm25_catalog schema from 0.3.x with native PostgreSQL tsvector indexing, reworks the Block-WeakAnd (WAND) scan and storage engine for continuing scans and concurrent VACUUM, and enforces table privileges and Row-Level Security (RLS) across both indexed scans and scalar scoring.

Extension & Release Links

GitHub Release: pgsty/vchord_bm25 1.0.0 · Source Repository
Extension Catalog: pgext.cloud (vchord_bm25) · Pigsty Extension Page (pigsty.io)
Binary Packages: RPM / DEB Repository Setup · SBOM & License Inventory

Changes in 1.0.0

Area	Change
Native `tsvector` API	Indexes standard PostgreSQL `tsvector` columns with `USING bm25 (... bm25_ops)`. `to_bm25query(tsvector, regclass)` builds queries, and `<&>` returns negative `double precision` BM25 scores (`+0.0` for zero scores).
Relocatable schema	Extension objects install into the target schema (`public` by default) and support `ALTER EXTENSION vchord_bm25 SET SCHEMA`. The fixed `bm25_catalog` schema is no longer required.
Continuing WAND scans	`bm25.limit` and the index `limit` reloption (`0..=65535`, default `100`, `0` selects `100`) set the initial WAND top-K batch size rather than a hard result cap; scans continue through matching, zero-score, and `NULL` rows as needed.
Shared statistics cache	Prepared corpus statistics and growing-segment scores are cached per `(index, query)` within `work_mem` and shared between indexed scans and scalar `<&>` evaluations (`PARALLEL RESTRICTED`). `NULL` rows are excluded from document count, and WAND upper bounds adjust for `avgdl` drift.
Concurrent maintenance	Format 4 storage uses generation stamps, a growing segment, and one crash-resumable retired generation. `VACUUM` and `bm25_maintain(regclass)` merge growing tuples under `ShareUpdateExclusiveLock` using PostgreSQL’s `GlobalVis` horizon without taking `AccessExclusiveLock` or allocating an XID. `ANALYZE` is a no-op.
Standby replay	`VACUUM` tombstones only already-pruned heap TIDs, raising no false standby conflicts; only page recycling raises SQLSTATE `40001` (`serialization_failure`) if an older standby query is still reading a retired generation.
Corpus permissions & RLS	Both indexed scans and scalar `<&>` check current-role table `SELECT` (or column `SELECT` for simple single-column non-partial indexes) and reject queries when RLS policies apply (`0A000`), preventing corpus-statistic leaks. Scalar scoring locks the heap before the index.
Build & platform support	Upgraded to `pgrx 0.19.3` for PostgreSQL 14–18 on Linux x86_64 and aarch64, with runtime SIMD dispatch (`AVX512` / `AVX2` / `NEON`) and responsive query cancellation outside page locks.

Native `tsvector` indexing and relocatable schema

In vchord_bm25 1.0.0, documents are stored as standard PostgreSQL tsvector values with lexeme positions (1..=16383, up to 256 positions per lexeme). Each position entry counts as one occurrence of that lexeme when computing term frequency (tf) and document length (dl); lexemes without positions are ignored by BM25 scoring.

full-text-search on Vchord

pg_tokenizer 1.0.0: First Release Under PGSTY

vchord_bm25 1.0.0: First Release Under PGSTY

Changes in 1.0.0

Native tsvector indexing and relocatable schema

Native `tsvector` indexing and relocatable schema