Usage & Hybrid Search on Vchord

Indexing & Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

vchord_bm25 provides the bm25 index access method and the explicit bm25_ops operator class. The SQL handler is bm25_amhandler in 0.4.0 and _bm25_amhandler in 0.3.x.

Creating a `bm25` Index

Stable Releases (`v0.1` – `v0.3`, `bm25vector`)

CREATE INDEX documents_embedding_bm25
ON documents
USING bm25 (embedding bm25_ops);

Next-Gen `main` Branch (`tsvector`)

In the main branch, bm25_ops operates on tsvector and supports index-level storage parameters (options, limit, and prefilter):

Querying & Scoring

Mon, 01 Jan 0001 00:00:00 +0000

Basic Top-K Query Syntax

To run an index-accelerated Top-K BM25 search, use the <&> operator in the ORDER BY clause together with to_bm25query:

-- v0.1 – v0.3 (bm25vector + pg_tokenizer)
SELECT
    id,
    passage,
    embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL vector search', 'my_tokenizer')) AS raw_score,
    -(embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL vector search', 'my_tokenizer'))) AS bm25_score
FROM documents
ORDER BY raw_score ASC
LIMIT 10;

-- main branch (native tsvector)
SELECT
    id,
    passage,
    -(vector <&> to_bm25query(to_tsvector('english', 'PostgreSQL vector search'), 'documents_bm25_idx')) AS bm25_score
FROM documents
ORDER BY vector <&> to_bm25query(to_tsvector('english', 'PostgreSQL vector search'), 'documents_bm25_idx') ASC
LIMIT 10;

Why `to_bm25query` Requires the Index Name

Unlike vector distance operators (where the distance between two vectors depends only on the two vectors themselves), the BM25 score of a document depends on corpus-wide statistics stored inside the BM25 index:

Filtered Search & Prefiltering

Mon, 01 Jan 0001 00:00:00 +0000

Real-world applications frequently combine full-text BM25 ranking with structured relational filters (such as category_id = 5, tenant_id = 'acme', or date ranges). vchord_bm25 supports three strategies for filtered BM25 search.

1. Post-Filtering vs. Pre-Filtering

When you execute a query with a WHERE clause and ORDER BY embedding <&> to_bm25query(...) LIMIT K:

SELECT id, passage
FROM documents
WHERE category = 'database'
ORDER BY embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL BM25', 'my_tokenizer'))
LIMIT 10;

There are two ways vchord_bm25 can evaluate the WHERE predicate:

Hybrid Search (BM25 + VectorChord)

Mon, 01 Jan 0001 00:00:00 +0000

Modern retrieval pipelines combine dense vector search (vchord, which captures semantic similarity) with sparse lexical BM25 search (vchord_bm25, which captures exact keyword, identifier, and rare-term matches). Because both extensions run natively inside PostgreSQL, you can execute end-to-end hybrid search in a single SQL query with zero external orchestration.

Schema Setup for Hybrid Search

CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;

SET search_path TO "$user", public, bm25_catalog, tokenizer_catalog;

-- 1. Create a BM25 tokenizer
SELECT create_tokenizer('bert', $$
model = "bert_base_uncased"
$$);

-- 2. Create a table storing both dense vectors and sparse bm25vectors
CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    dense_emb vector(768),
    bm25_emb bm25vector
);

-- 3. Create VectorChord dense index (vchordrq) and BM25 lexical index (bm25)
CREATE INDEX chunks_dense_idx ON chunks USING vchordrq (dense_emb vector_l2_ops);
CREATE INDEX chunks_bm25_idx  ON chunks USING bm25 (bm25_emb bm25_ops);

Method 1: Reciprocal Rank Fusion (RRF) in Pure SQL

Reciprocal Rank Fusion (RRF) combines rankings from multiple retrieval systems without requiring score calibration between L2/cosine distance and BM25 scores:

BM25 & Block-WeakAnd Internals

Mon, 01 Jan 0001 00:00:00 +0000

This page details the exact scoring equations and data structures implemented in vchord_bm25 (crates/bm25/src/bm25.rs and src/algorithm/block_wand.rs).

1. Okapi BM25 Scoring Formula

Given a document $D$ and a query $Q = \\{q_1, q_2, \\dots, q_m\\}$ with query term frequencies $c(q_i, Q)$ and document term frequencies $f(q_i, D)$ , vchord_bm25 computes:

Usage & Hybrid Search on Vchord

Indexing & Maintenance

Creating a bm25 Index

Stable Releases (v0.1 – v0.3, bm25vector)

Next-Gen main Branch (tsvector)

Querying & Scoring

Basic Top-K Query Syntax

Why to_bm25query Requires the Index Name

Filtered Search & Prefiltering

1. Post-Filtering vs. Pre-Filtering

Hybrid Search (BM25 + VectorChord)

Schema Setup for Hybrid Search

Method 1: Reciprocal Rank Fusion (RRF) in Pure SQL

BM25 & Block-WeakAnd Internals

1. Okapi BM25 Scoring Formula

Creating a `bm25` Index

Stable Releases (`v0.1` – `v0.3`, `bm25vector`)

Next-Gen `main` Branch (`tsvector`)

Why `to_bm25query` Requires the Index Name