Reference & Benchmarks

• 1: SQL Reference
• 2: GUC & Index Parameters
• 3: Benchmarks
• 4: Release History & API Evolution

1 - SQL Reference

vchord_bm25 Data Types

bm25vector Literal Syntax (v0.1 – v0.3)

-- Empty bm25vector
SELECT '{}'::bm25vector;

-- Explicit token_id:frequency pairs
SELECT '{1:1, 2:3, 10:2}'::bm25vector;

-- Implicit cast from int[] (automatically aggregates duplicate token IDs into counts)
SELECT ARRAY[1, 2, 2, 2, 10, 10]::bm25vector;
-- Output: {1:1, 2:3, 10:2}

vchord_bm25 Operators

vchord_bm25 Functions

vchord_bm25 Access Method & Operator Class

pg_tokenizer Functions (tokenizer_catalog)

All pg_tokenizer functions reside in the tokenizer_catalog schema:

Tokenizer Management & Execution

Text Analyzer Management

Model Management

Dictionary & Preload Management

Internal Trigger & Helper Functions (tokenizer_catalog)

pg_tokenizer Catalog Tables (tokenizer_catalog)

pg_tokenizer TOML Configuration Reference

Tokenizer Options (create_tokenizer)

Text Analyzer Options (create_text_analyzer)

Custom Model Options (create_custom_model)

2 - GUC & Index Parameters

Stable Releases (v0.1.0 – v0.3.0): bm25_catalog.* GUCs

In versions v0.1.0 through v0.3.0, all session GUC parameters use the bm25_catalog. prefix and can be changed per session or transaction by any user (GucContext::Userset):

Examples (v0.1 – v0.3)

-- Increase Top-K window for deep pagination (e.g., LIMIT 500)
SET bm25_catalog.bm25_limit = 500;

-- Force brute-force scan over all index postings
SET bm25_catalog.bm25_limit = -1;

-- Enable prefiltering for queries with selective WHERE clauses
SET bm25_catalog.enable_prefilter = true;

-- Disable index scan to benchmark sequential scan baseline
SET bm25_catalog.enable_index = false;

Next-Gen main Branch: bm25.* GUCs & Index Reloptions

In the main branch rewrite, session GUCs use the bm25. prefix and work together with per-index storage parameters (WITH (...)):

Session GUC Parameters (main)

Index Storage Parameters (WITH (...) in main)

3 - Benchmarks

Throughput Benchmark (QPS vs. Elasticsearch, Lucene, ParadeDB, PG GIN)

VectorChord-BM25’s Block-WeakAnd implementation with 128-document delta-bitpacked blocks and 1-byte fieldnorm quantization delivers up to 3x higher QPS than Tantivy-based PostgreSQL extensions (ParadeDB / pg_search) and an order of magnitude higher QPS than native PostgreSQL GIN indexes on Top-10 ranked queries:

Why vchord_bm25 Achieves High QPS

1. Zero Cross-Engine Serialization: Unlike foreign-data-wrapper or external Tantivy/Lucene bridges, vchord_bm25 stores its segments directly inside PostgreSQL buffer pages (PostgresRelation), avoiding tuple marshaling overhead.
2. Aggressive Block-Max Pruning: On multi-term queries, Block-WeakAnd skips over 80%–95% of 128-document posting blocks without decompressing them once the Top-10 threshold stabilizes.
3. 256-Entry Precomputed BM25 Lookup Table: Quantizing document length into a 1-byte fieldnorm replaces expensive per-document floating-point math with a single array lookup.

Retrieval Accuracy Benchmark (NDCG@10 on BEIR & LoTTE)

The table below (from our BEIR / LoTTE Hybrid Search & ColBERT Reranking evaluation) compares retrieval accuracy (NDCG@10) across:

• PostgreSQL vchord Dense Vector Search (Vector)
• PostgreSQL vchord_bm25 Lexical Search (BM25)
• Hybrid Search (vchord + vchord_bm25 with ColBERT Rerank)

4 - Release History & API Evolution

This page tracks the release history and SQL schema evolution of vchord_bm25 (pgsty/vchord_bm25 / supervc-stack/VectorChord-bm25).

Version Summary Matrix

Detailed Upgrade Changes

v0.3.0 (sql/vchord_bm25--0.2.2--0.3.0.sql)

• Added page diagnostic function:

  CREATE FUNCTION bm25_page_inspect(index regclass, blkno INT) RETURNS TEXT;

• Added to_bm25query(regclass, int[]) convenience overload in main/late 0.3.x.

v0.2.2 (sql/vchord_bm25--0.2.1--0.2.2.sql)

• Added prefiltering support in Block-WeakAnd (bm25_catalog.enable_prefilter GUC, enabled by default in 0.3.0).
• Improved MVCC visibility checking during index scans to prevent dead/filtered tuples from polluting the Top-K threshold.

v0.2.0 (sql/vchord_bm25--0.1.1--0.2.0.sql)

• Added = and <> operators for bm25vector.
• Added implicit cast from int[] to bm25vector (_vchord_bm25_cast_array_to_bm25vector), allowing tokenize(text, name) output (INT[]) to be assigned or passed directly wherever bm25vector is expected.

v0.1.1 (sql/vchord_bm25--0.1.0--0.1.1.sql)

• Extracted built-in create_tokenizer / tokenize functions into the dedicated pg_tokenizer extension (tokenizer_catalog), enabling rich text analyzers, Snowball stemmers, stopwords, synonyms, HuggingFace models, jieba, and lindera.
• Simplified to_bm25query(index_oid regclass, query_vector bm25vector) to accept a bm25vector directly.

Next-Generation Rewrite (main Branch)

• Replaced custom bm25vector with PostgreSQL’s native tsvector (CREATE OPERATOR CLASS bm25_ops FOR TYPE tsvector USING bm25), allowing users to index to_tsvector(...) columns directly with USING bm25.
• Added parallel index build (bm25_parallel_build_main) with external merge sort and constant maintenance memory consumption.
• Added configurable BM25 hyperparameters ($k_1 \in [1.2, 2.0]$, $b \in [0.0, 1.0]$) via WITH (options = $$ k1 = 1.2, b = 0.75 $$).