vchord_bm25 Documentation

Native Okapi BM25 ranking index and Block-WeakAnd full-text search for PostgreSQL (vchord_bm25).

Important

Current source targets 0.4.0 (unreleased), with native tsvector and a new index format. Examples labeled 0.3.x use the historical bm25vector API. In-place upgrades are blocked; read the 0.4.0 migration guide before changing binaries.

vchord_bm25 (maintained by Pigsty at pgsty/vchord_bm25) is a native PostgreSQL extension built in Rust (pgrx) that implements the Okapi BM25 ranking algorithm and the Block-WeakAnd (Block-Max WAND) Top-K index scan access method inside PostgreSQL.

Paired with pg_tokenizer (maintained by Pigsty at pgsty/pg_tokenizer for customizable multilingual tokenization and vocabulary models) or PostgreSQL’s built-in tsvector analyzer, vchord_bm25 brings Elasticsearch/Lucene-class lexical full-text ranking directly into your relational database—and combines seamlessly with vchord and pgvector for hybrid semantic + lexical search.

Getting Started

Understand why BM25 matters, install vchord_bm25 and pg_tokenizer, and run your first ranked full-text query.

Usage & Indexing

Build bm25 indexes, run Top-K Block-WeakAnd queries, apply WHERE prefilters, and inspect index pages.

Hybrid Search

Fuse sparse BM25 lexical search with vchord dense and ColBERT multi-vector search via RRF in SQL.

Reference & Benchmarks

Complete SQL types, operators, functions, GUC parameters, version evolution, and BEIR benchmark results.

0.3.x to 0.4.0 Migration

Migrate from bm25vector + pg_tokenizer (0.3.x) to the native tsvector BM25 index format in 0.4.0.

pg_tokenizer Documentation

Configure text analyzers, stemmers, stopwords, synonyms, HuggingFace models, Chinese Jieba, and Japanese Lindera.

Highlights

True Okapi BM25 Scoring: Supports inverse document frequency (IDF), term frequency saturation ( $k_1$ ), and quantized 1-byte document length normalization ( $b$ and average document length $\text{avgdl}$ )—solving fundamental relevance limitations of PostgreSQL’s native ts_rank.
Block-WeakAnd Index Acceleration: Stores compressed posting lists with per-block maximum score upper bounds using delta-bitpacking, skipping non-competitive document blocks during Top-K queries without scanning entire inverted lists.
Multilingual Tokenization: Works with pg_tokenizer (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b, custom corpus-trained models, Chinese jieba, Japanese lindera) in v0.1–v0.3, and supports PostgreSQL’s native tsvector directly in the next-generation main architecture.
Native Hybrid Search with Vchord: Run lexical BM25 (vchord_bm25) and dense vector search (vchordrq) inside a single SQL query using Reciprocal Rank Fusion (RRF) or ColBERT MaxSim reranking.