Reference & Benchmarks on Vchord

SQL Reference

Mon, 01 Jan 0001 00:00:00 +0000

`vchord_bm25` Data Types

Data Type	Schema	Versions	Text Representation / Structure	Description
`bm25vector`	`bm25_catalog`	`v0.1.0` – `v0.3.0`	`{token_id:freq, ...}` (e.g., `{1:1, 2:2, 15:1}`)	Sparse vector mapping 32-bit unsigned token IDs to 32-bit term frequencies. Stored with `STORAGE = EXTERNAL`.
`bm25query`	`bm25_catalog`	`v0.1.0` – `v0.3.0`	`(index_oid regclass, query_vector bm25vector)`	Composite type pairing a target `bm25` index OID with a query `bm25vector`.
`bm25query`	`public` / ext schema	`main`	`(vector tsvector, index regclass)`	Composite type pairing a query `tsvector` with a target `bm25` index OID.

`bm25vector` Literal Syntax (`v0.1` – `v0.3`)

-- Empty bm25vector
SELECT '{}'::bm25vector;

-- Explicit token_id:frequency pairs
SELECT '{1:1, 2:3, 10:2}'::bm25vector;

-- Implicit cast from int[] (automatically aggregates duplicate token IDs into counts)
SELECT ARRAY[1, 2, 2, 2, 10, 10]::bm25vector;
-- Output: {1:1, 2:3, 10:2}

`vchord_bm25` Operators

Operator	Left Arg	Right Arg	Return Type	Versions	Description
`<&>`	`bm25vector`	`bm25query`	`real` (`f32`)	`v0.1.0` – `v0.3.0`	Computes the negative BM25 relevance score ($-\text{BM25}(D, Q)$). Supports index-accelerated `ORDER BY ... ASC` scans using `bm25_ops`.
`<&>`	`tsvector`	`bm25query`	`float8` (`f64`)	`main`	Computes the negative BM25 relevance score on native `tsvector`. Supports index-accelerated `ORDER BY ... ASC` scans using `bm25_ops`.
`=`	`bm25vector`	`bm25vector`	`bool`	`v0.2.0` – `v0.3.0`	Exact equality comparison between two `bm25vector` values.
`<>`	`bm25vector`	`bm25vector`	`bool`	`v0.2.0` – `v0.3.0`	Inequality comparison between two `bm25vector` values.

`vchord_bm25` Functions

Function Signature	Return Type	Versions	Description
`to_bm25query(index_oid regclass, query_vector bm25vector)`	`bm25query`	`v0.1.1` – `v0.3.0`	Constructs a `bm25query` from a `bm25` index name/OID and a query `bm25vector` (or `INT[]` via implicit cast).
`to_bm25query(index_oid regclass, query_vector int[])`	`bm25query`	`v0.3.0+`	Explicit overload accepting an `INT[]` token ID array directly.
`to_bm25query(vector tsvector, index regclass)`	`bm25query`	`main`	Constructs a `bm25query` from a native `tsvector` and target `bm25` index name/OID.
`search_bm25query(target_vector bm25vector, query bm25query)`	`real`	`v0.1.0` – `v0.3.0`	Underlying implementation function for `bm25vector <&> bm25query`.
`_bm25_evaluate(left tsvector, right bm25query)`	`float8`	`main`	Underlying implementation function for `tsvector <&> bm25query`.
`bm25_page_inspect(index regclass, blkno int)`	`text`	`v0.3.0`	Returns human-readable diagnostic metadata for block `blkno` of a `bm25` index.

`vchord_bm25` Access Method & Operator Class

Object	Name	Target Type	Ordering Operator
Access Method	`bm25`	Index	Handler `_bm25_amhandler`
Operator Family	`bm25_ops`	`USING bm25`	`OPERATOR 1 pg_catalog.<&> FOR ORDER BY float_ops`
Operator Class	`bm25_ops`	`bm25vector` (`v0.1`–`v0.3`) / `tsvector` (`main`)	Default operator class for `USING bm25`

`pg_tokenizer` Functions (`tokenizer_catalog`)

All pg_tokenizer functions reside in the tokenizer_catalog schema:

GUC & Index Parameters

Mon, 01 Jan 0001 00:00:00 +0000

Stable Releases (`v0.1.0` – `v0.3.0`): `bm25_catalog.*` GUCs

In versions v0.1.0 through v0.3.0, all session GUC parameters use the bm25_catalog. prefix and can be changed per session or transaction by any user (GucContext::Userset):

Parameter Name	Type	Default	Valid Range	Description
`bm25_catalog.bm25_limit`	`integer`	`100`	`-1` .. `65535`	Maximum number of top-scoring documents to return from a `bm25` index scan. Set to `-1` to disable Block-WeakAnd pruning and perform an exhaustive brute-force scan; set to `0` to return no rows. Must be $\ge$ your SQL `LIMIT`.
`bm25_catalog.enable_index`	`boolean`	`true`	`true` / `false`	Enables or disables the `bm25` index scan in the PostgreSQL query planner. Set to `false` to force a sequential scan with exact `search_bm25query` evaluation.
`bm25_catalog.enable_prefilter`	`boolean`	`true`	`true` / `false`	Introduced in `v0.2.2`. When `true`, filters out documents that do not satisfy the query/visibility check before inserting them into the Block-WeakAnd Top-K heap.
`bm25_catalog.segment_growing_max_page_size`	`integer`	`4096`	`1` .. `1000000`	Maximum size (in 8 KB pages, default `4096` = 32 MB) of the append-only growing segment before it is sealed into a compressed, delta-bitpacked read-only segment.

Examples (`v0.1` – `v0.3`)

-- Increase Top-K window for deep pagination (e.g., LIMIT 500)
SET bm25_catalog.bm25_limit = 500;

-- Force brute-force scan over all index postings
SET bm25_catalog.bm25_limit = -1;

-- Enable prefiltering for queries with selective WHERE clauses
SET bm25_catalog.enable_prefilter = true;

-- Disable index scan to benchmark sequential scan baseline
SET bm25_catalog.enable_index = false;

Next-Gen `main` Branch: `bm25.*` GUCs & Index Reloptions

In the main branch rewrite, session GUCs use the bm25. prefix and work together with per-index storage parameters (WITH (...)):

Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Throughput Benchmark (QPS vs. Elasticsearch, Lucene, ParadeDB, PG GIN)

VectorChord-BM25’s Block-WeakAnd implementation with 128-document delta-bitpacked blocks and 1-byte fieldnorm quantization delivers up to 3x higher QPS than Tantivy-based PostgreSQL extensions (ParadeDB / pg_search) and an order of magnitude higher QPS than native PostgreSQL GIN indexes on Top-10 ranked queries:

Why `vchord_bm25` Achieves High QPS

Zero Cross-Engine Serialization: Unlike foreign-data-wrapper or external Tantivy/Lucene bridges, vchord_bm25 stores its segments directly inside PostgreSQL buffer pages (PostgresRelation), avoiding tuple marshaling overhead.
Aggressive Block-Max Pruning: On multi-term queries, Block-WeakAnd skips over 80%–95% of 128-document posting blocks without decompressing them once the Top-10 threshold stabilizes.
256-Entry Precomputed BM25 Lookup Table: Quantizing document length into a 1-byte fieldnorm replaces expensive per-document floating-point math with a single array lookup.

Retrieval Accuracy Benchmark (`NDCG@10` on BEIR & LoTTE)

The table below (from our BEIR / LoTTE Hybrid Search & ColBERT Reranking evaluation) compares retrieval accuracy (NDCG@10) across:

Release History & API Evolution

Mon, 01 Jan 0001 00:00:00 +0000

This page tracks the release history and SQL schema evolution of vchord_bm25 (pgsty/vchord_bm25 / supervc-stack/VectorChord-bm25).

Version Summary Matrix

Version	Schema	Indexed Type	Tokenizer	Key Additions
`main` (Next-Gen)	Extension schema	`tsvector`	Native `to_tsvector` (or custom)	Complete storage & index engine rewrite on native `tsvector`; external sort parallel `ambuild`; index reloptions (`options`, `limit`, `prefilter`); `bm25.*` GUCs; PostgreSQL 18 support.
`v0.3.0`	`bm25_catalog`	`bm25vector`	`pg_tokenizer`	Added `bm25_page_inspect(regclass, int)`, `to_bm25query(regclass, int[])` overload, and PostgreSQL 18 compatibility.
`v0.2.2`	`bm25_catalog`	`bm25vector`	`pg_tokenizer`	Added `bm25_catalog.enable_prefilter` GUC for accurate filtered Top-K BM25 search with `WHERE` clauses.
`v0.2.1`	`bm25_catalog`	`bm25vector`	`pg_tokenizer`	Bug fixes for growing/sealed segment transitions and vacuum cleanup.
`v0.2.0`	`bm25_catalog`	`bm25vector`	`pg_tokenizer`	Added `=` and `<>` equality operators on `bm25vector` and implicit cast `CREATE CAST (int[] AS bm25vector) AS IMPLICIT`.
`v0.1.1`	`bm25_catalog`	`bm25vector`	`pg_tokenizer`	Decoupled tokenization into standalone `pg_tokenizer` extension; changed `to_bm25query(regclass, bm25vector)` signature.
`v0.1.0`	`bm25_catalog`	`bm25vector`	Built-in `unicode`	Initial release with `bm25vector`, `bm25query`, `USING bm25`, and Block-WeakAnd Top-K scan.

Detailed Upgrade Changes

`v0.3.0` (`sql/vchord_bm25--0.2.2--0.3.0.sql`)

Added page diagnostic function:

CREATE FUNCTION bm25_page_inspect(index regclass, blkno INT) RETURNS TEXT;

Added to_bm25query(regclass, int[]) convenience overload in main/late 0.3.x.

`v0.2.2` (`sql/vchord_bm25--0.2.1--0.2.2.sql`)

Added prefiltering support in Block-WeakAnd (bm25_catalog.enable_prefilter GUC, enabled by default in 0.3.0).
Improved MVCC visibility checking during index scans to prevent dead/filtered tuples from polluting the Top-K threshold.

`v0.2.0` (`sql/vchord_bm25--0.1.1--0.2.0.sql`)

Added = and <> operators for bm25vector.
Added implicit cast from int[] to bm25vector (_vchord_bm25_cast_array_to_bm25vector), allowing tokenize(text, name) output (INT[]) to be assigned or passed directly wherever bm25vector is expected.

`v0.1.1` (`sql/vchord_bm25--0.1.0--0.1.1.sql`)

Extracted built-in create_tokenizer / tokenize functions into the dedicated pg_tokenizer extension (tokenizer_catalog), enabling rich text analyzers, Snowball stemmers, stopwords, synonyms, HuggingFace models, jieba, and lindera.
Simplified to_bm25query(index_oid regclass, query_vector bm25vector) to accept a bm25vector directly.

Next-Generation Rewrite (`main` Branch)

Replaced custom bm25vector with PostgreSQL’s native tsvector (CREATE OPERATOR CLASS bm25_ops FOR TYPE tsvector USING bm25), allowing users to index to_tsvector(...) columns directly with USING bm25.
Added parallel index build (bm25_parallel_build_main) with external merge sort and constant maintenance memory consumption.
Added configurable BM25 hyperparameters ( $k_1 \in [1.2, 2.0]$ , $b \in [0.0, 1.0]$ ) via WITH (options = $$ k1 = 1.2, b = 0.75 $$).

Reference & Benchmarks on Vchord

SQL Reference

vchord_bm25 Data Types

bm25vector Literal Syntax (v0.1 – v0.3)

vchord_bm25 Operators

vchord_bm25 Functions

vchord_bm25 Access Method & Operator Class

pg_tokenizer Functions (tokenizer_catalog)

GUC & Index Parameters

Stable Releases (v0.1.0 – v0.3.0): bm25_catalog.* GUCs

Examples (v0.1 – v0.3)

Next-Gen main Branch: bm25.* GUCs & Index Reloptions

Benchmarks

Throughput Benchmark (QPS vs. Elasticsearch, Lucene, ParadeDB, PG GIN)

Why vchord_bm25 Achieves High QPS

Retrieval Accuracy Benchmark (NDCG@10 on BEIR & LoTTE)

Release History & API Evolution

Version Summary Matrix

Detailed Upgrade Changes

v0.3.0 (sql/vchord_bm25--0.2.2--0.3.0.sql)

v0.2.2 (sql/vchord_bm25--0.2.1--0.2.2.sql)

v0.2.0 (sql/vchord_bm25--0.1.1--0.2.0.sql)

v0.1.1 (sql/vchord_bm25--0.1.0--0.1.1.sql)

Next-Generation Rewrite (main Branch)

`vchord_bm25` Data Types

`bm25vector` Literal Syntax (`v0.1` – `v0.3`)

`vchord_bm25` Operators

`vchord_bm25` Functions

`vchord_bm25` Access Method & Operator Class

`pg_tokenizer` Functions (`tokenizer_catalog`)

Stable Releases (`v0.1.0` – `v0.3.0`): `bm25_catalog.*` GUCs

Examples (`v0.1` – `v0.3`)

Next-Gen `main` Branch: `bm25.*` GUCs & Index Reloptions

Why `vchord_bm25` Achieves High QPS

Retrieval Accuracy Benchmark (`NDCG@10` on BEIR & LoTTE)

`v0.3.0` (`sql/vchord_bm25--0.2.2--0.3.0.sql`)

`v0.2.2` (`sql/vchord_bm25--0.2.1--0.2.2.sql`)

`v0.2.0` (`sql/vchord_bm25--0.1.1--0.2.0.sql`)

`v0.1.1` (`sql/vchord_bm25--0.1.0--0.1.1.sql`)

Next-Generation Rewrite (`main` Branch)