Reference & Benchmarks

• 1: SQL Reference
• 2: GUC & Index Parameters
• 3: Benchmarks
• 4: Migrating to 0.4.0
• 5: Release History & API Evolution

1 - SQL Reference

vchord_bm25 Data Types

bm25vector Literal Syntax (v0.1 – v0.3)

-- Empty bm25vector
SELECT '{}'::bm25vector;

-- Explicit token_id:frequency pairs
SELECT '{1:1, 2:3, 10:2}'::bm25vector;

-- Implicit cast from int[] (automatically aggregates duplicate token IDs into counts)
SELECT ARRAY[1, 2, 2, 2, 10, 10]::bm25vector;
-- Output: {1:1, 2:3, 10:2}

vchord_bm25 Operators

vchord_bm25 Functions

vchord_bm25 Access Method & Operator Class

pg_tokenizer Functions (tokenizer_catalog)

All pg_tokenizer functions reside in the tokenizer_catalog schema:

Tokenizer Management & Execution

Text Analyzer Management

Model Management

Dictionary & Preload Management

Internal Trigger & Helper Functions (tokenizer_catalog)

pg_tokenizer Catalog Tables (tokenizer_catalog)

pg_tokenizer TOML Configuration Reference

Tokenizer Options (create_tokenizer)

Text Analyzer Options (create_text_analyzer)

Custom Model Options (create_custom_model)

2 - GUC & Index Parameters

Stable Releases (v0.1.0 – v0.3.0): bm25_catalog.* GUCs

In versions v0.1.0 through v0.3.0, all session GUC parameters use the bm25_catalog. prefix and can be changed per session or transaction by any user (GucContext::Userset):

Examples (v0.1 – v0.3)

-- Increase Top-K window for deep pagination (e.g., LIMIT 500)
SET bm25_catalog.bm25_limit = 500;

-- Force brute-force scan over all index postings
SET bm25_catalog.bm25_limit = -1;

-- Enable prefiltering for queries with selective WHERE clauses
SET bm25_catalog.enable_prefilter = true;

-- Disable index scan to benchmark sequential scan baseline
SET bm25_catalog.enable_index = false;

Next-Gen main Branch: bm25.* GUCs & Index Reloptions

In the main branch rewrite, session GUCs use the bm25. prefix and work together with per-index storage parameters (WITH (...)):

Session GUC Parameters (main)

Settings from postgresql.conf, ALTER DATABASE, ALTER ROLE, or client startup options also override index reloptions. RESET restores that configured default; it falls back to the index only when the GUC source returns to the built-in default.

Index Storage Parameters (WITH (...) in main)

3 - Benchmarks

Throughput Benchmark (QPS vs. Elasticsearch, Lucene, ParadeDB, PG GIN)

VectorChord-BM25’s Block-WeakAnd implementation with 128-document delta-bitpacked blocks and 1-byte fieldnorm quantization delivers up to 3x higher QPS than Tantivy-based PostgreSQL extensions (ParadeDB / pg_search) and an order of magnitude higher QPS than native PostgreSQL GIN indexes on Top-10 ranked queries:

Why vchord_bm25 Achieves High QPS

1. Zero Cross-Engine Serialization: Unlike foreign-data-wrapper or external Tantivy/Lucene bridges, vchord_bm25 stores its segments directly inside PostgreSQL buffer pages (PostgresRelation), avoiding tuple marshaling overhead.
2. Aggressive Block-Max Pruning: On multi-term queries, Block-WeakAnd skips over 80%–95% of 128-document posting blocks without decompressing them once the Top-10 threshold stabilizes.
3. 256-Entry Precomputed BM25 Lookup Table: Quantizing document length into a 1-byte fieldnorm replaces expensive per-document floating-point math with a single array lookup.

Retrieval Accuracy Benchmark (NDCG@10 on BEIR & LoTTE)

The table below (from our BEIR / LoTTE Hybrid Search & ColBERT Reranking evaluation) compares retrieval accuracy (NDCG@10) across:

• PostgreSQL vchord Dense Vector Search (Vector)
• PostgreSQL vchord_bm25 Lexical Search (BM25)
• Hybrid Search (vchord + vchord_bm25 with ColBERT Rerank)

4 - Migrating to 0.4.0

The current pgsty/vchord_bm25 source prepares 0.4.0 (unreleased). It uses PostgreSQL’s native tsvector and a new index storage engine. Historical 0.3.x packages and suite images use bm25vector; the two APIs and index formats are incompatible.

The current build ships vchord_bm25--0.4.0.sql, generated from the Rust library. Packaging compares the checked-in schema with the compiled API and rejects mismatches. The old install SQL remains available in Git for old binaries; changing the control version to 0.3.0 does not make the current engine compatible with it.

Install and query

Build current source using PG_CONFIG=/path/to/pg_config make build and PG_CONFIG=/path/to/pg_config make install. Add vchord_bm25 to shared_preload_libraries and restart PostgreSQL. Standard text search needs no external tokenizer:

CREATE EXTENSION vchord_bm25;
CREATE TABLE documents (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    passage text NOT NULL,
    vector tsvector GENERATED ALWAYS AS (to_tsvector('english', passage)) STORED
);
INSERT INTO documents (passage) VALUES
    ('PostgreSQL supports full text search.'),
    ('BM25 ranks documents by relevance.');
CREATE INDEX documents_bm25 ON documents USING bm25 (vector bm25_ops)
WITH (limit = 100);

SELECT id, vector <&> to_bm25query(to_tsvector('english', 'PostgreSQL'), 'documents_bm25') AS rank
FROM documents ORDER BY rank LIMIT 10;

The index needs document positions to recover term frequencies. strip(tsvector) and array_to_tsvector cannot supply them. Use the same analyzer for documents and queries. An integer token ID is not a lexeme; no automatic bm25vector-to-tsvector cast is provided.

An explicit SET "bm25.limit" = 100 overrides the index limit. RESET "bm25.limit" restores the index setting. Explicit zero makes the scan error; it does not select the index default or unlimited results. Both defaults are zero, so a positive setting is required. The same source-based precedence applies to bm25.prefilter, including explicitly setting false. SQL LIMIT does not set either GUC.

Settings from postgresql.conf, ALTER DATABASE, ALTER ROLE, or client startup options also override index reloptions. RESET restores that configured default; it falls back to the index only when the GUC source returns to the built-in default.

Schema access

A DBA can install into a schema chosen for the application:

CREATE SCHEMA search;
CREATE EXTENSION vchord_bm25 WITH SCHEMA search;
GRANT USAGE ON SCHEMA search TO app_user;
ALTER ROLE app_user SET search_path = public, search;

This is an alternative to installing into public; do not run both installations in one database. Grant table access separately. Function execution is granted to PUBLIC by PostgreSQL’s default privileges; if your deployment revokes it, grant the required functions explicitly. Schema-qualified calls to search.to_bm25query(...) work without adding search to search_path, including after relocating the extension.

Move existing data

ALTER EXTENSION vchord_bm25 UPDATE TO '0.4.0' from 0.3.0 deliberately raises an error before changing objects. It neither rewrites old indexes nor drops dependent application data.

1. Keep the old database and old binary available. Inventory bm25vector columns, indexes, views, functions, and tokenizer triggers; export source text and application data while the old datatype can still be read.
2. Install 0.4.0 into a separate PostgreSQL installation and new database. Databases in the same installation share the extension library, so replacing the binary while any database still uses 0.3.x is unsafe.
3. Restore the source text, generate tsvector values, and rebuild all BM25 indexes. Set a positive search limit. A binary copy of an old index or an ordinary REINDEX on a bm25vector column cannot migrate the API.
4. Update the constructor argument order, types, GUC names, and tokenizer triggers. Compare ranking quality and query results before switching the application.

The pg_tokenizer integer-array workflow remains documented for 0.3.x. For 0.4.0, use to_tsvector until the tokenizer version you deploy explicitly supports compatible tsvector output.

5 - Release History & API Evolution

This page tracks the release history and SQL schema evolution of vchord_bm25 (pgsty/vchord_bm25 / supervc-stack/VectorChord-bm25).

Version Summary Matrix

Detailed Upgrade Changes

v0.3.0 (sql/vchord_bm25--0.2.2--0.3.0.sql)

• Added page diagnostic function:

  CREATE FUNCTION bm25_page_inspect(index regclass, blkno INT) RETURNS TEXT;

• Added to_bm25query(regclass, int[]) convenience overload in main/late 0.3.x.

v0.2.2 (sql/vchord_bm25--0.2.1--0.2.2.sql)

• Added prefiltering support in Block-WeakAnd (bm25_catalog.enable_prefilter GUC, enabled by default in 0.3.0).
• Improved MVCC visibility checking during index scans to prevent dead/filtered tuples from polluting the Top-K threshold.

v0.2.0 (sql/vchord_bm25--0.1.1--0.2.0.sql)

• Added = and <> operators for bm25vector.
• Added implicit cast from int[] to bm25vector (_vchord_bm25_cast_array_to_bm25vector), allowing tokenize(text, name) output (INT[]) to be assigned or passed directly wherever bm25vector is expected.

v0.1.1 (sql/vchord_bm25--0.1.0--0.1.1.sql)

• Extracted built-in create_tokenizer / tokenize functions into the dedicated pg_tokenizer extension (tokenizer_catalog), enabling rich text analyzers, Snowball stemmers, stopwords, synonyms, HuggingFace models, jieba, and lindera.
• Simplified to_bm25query(index_oid regclass, query_vector bm25vector) to accept a bm25vector directly.

Next-Generation Rewrite (main Branch)

• Replaced custom bm25vector with PostgreSQL’s native tsvector (CREATE OPERATOR CLASS bm25_ops FOR TYPE tsvector USING bm25), allowing users to index to_tsvector(...) columns directly with USING bm25.
• Added parallel index build (bm25_parallel_build_main) with external merge sort and constant maintenance memory consumption.
• Added configurable BM25 hyperparameters ($k_1 \in [1.2, 2.0]$, $b \in [0.0, 1.0]$) via WITH (options = $$ k1 = 1.2, b = 0.75 $$).