# vchord_bm25 Documentation

> Native Okapi BM25 ranking index and Block-WeakAnd full-text search for PostgreSQL (`vchord_bm25`).

---

LLMS index: [llms.txt](/llms.txt)

---

> [!IMPORTANT]
> Current source targets **0.4.0 (unreleased)**, with native `tsvector` and a new index format. Examples labeled 0.3.x use the historical `bm25vector` API. In-place upgrades are blocked; read the [0.4.0 migration guide](/docs/bm25/reference/migration/) before changing binaries.


**`vchord_bm25`** (maintained by Pigsty at [`pgsty/vchord_bm25`](https://github.com/pgsty/vchord_bm25)) is a native PostgreSQL extension built in Rust (`pgrx`) that implements the **Okapi BM25** ranking algorithm and the **Block-WeakAnd (Block-Max WAND)** Top-K index scan access method inside PostgreSQL.

Paired with **[`pg_tokenizer`](/docs/tokenizer/)** (maintained by Pigsty at [`pgsty/pg_tokenizer`](https://github.com/pgsty/pg_tokenizer) for customizable multilingual tokenization and vocabulary models) or PostgreSQL's built-in `tsvector` analyzer, `vchord_bm25` brings Elasticsearch/Lucene-class lexical full-text ranking directly into your relational database—and combines seamlessly with **[`vchord`](/docs/vchord/)** and **[`pgvector`](/docs/vector/)** for hybrid semantic + lexical search.


  - [Getting Started](/docs/bm25/getting-started/) — Understand why BM25 matters, install vchord_bm25 and pg_tokenizer, and run your first ranked full-text query.
  - [Usage \& Indexing](/docs/bm25/usage/) — Build bm25 indexes, run Top-K Block-WeakAnd queries, apply WHERE prefilters, and inspect index pages.
  - [Hybrid Search](/docs/bm25/usage/hybrid-search/) — Fuse sparse BM25 lexical search with [`vchord`](/docs/vchord/) dense and ColBERT multi-vector search via RRF in SQL.
  - [Reference \& Benchmarks](/docs/bm25/reference/) — Complete SQL types, operators, functions, GUC parameters, version evolution, and BEIR benchmark results.
  - [0\.3\.x to 0\.4\.0 Migration](/docs/bm25/reference/migration/) — Migrate from `bm25vector` + `pg_tokenizer` (0.3.x) to the native `tsvector` BM25 index format in 0.4.0.
  - [pg\_tokenizer Documentation](/docs/tokenizer/) — Configure text analyzers, stemmers, stopwords, synonyms, HuggingFace models, Chinese Jieba, and Japanese Lindera.


## Highlights

- **True Okapi BM25 Scoring**: Supports inverse document frequency (IDF), term frequency saturation (\(k_1\)), and quantized 1-byte document length normalization (\(b\) and average document length \(\text{avgdl}\))—solving fundamental relevance limitations of PostgreSQL's native `ts_rank`.
- **Block-WeakAnd Index Acceleration**: Stores compressed posting lists with per-block maximum score upper bounds using delta-bitpacking, skipping non-competitive document blocks during Top-K queries without scanning entire inverted lists.
- **Multilingual Tokenization**: Works with [`pg_tokenizer`](/docs/tokenizer/) (`llmlingua2`, `bert_base_uncased`, `wiki_tocken`, `gemma2b`, custom corpus-trained models, Chinese `jieba`, Japanese `lindera`) in `v0.1`–`v0.3`, and supports PostgreSQL's native `tsvector` directly in the next-generation `main` architecture.
- **Native Hybrid Search with Vchord**: Run lexical BM25 (`vchord_bm25`) and dense vector search (`vchordrq`) inside a single SQL query using Reciprocal Rank Fusion (RRF) or ColBERT MaxSim reranking.

---

Section pages:

- [Getting Started](/docs/bm25/getting-started/): Overview, installation, and quick start guide for vchord_bm25 and pg_tokenizer.
- [Usage & Hybrid Search](/docs/bm25/usage/): Index management, Top-K querying, prefiltering, hybrid search with vchord, and Block-WeakAnd internals.
- [Reference & Benchmarks](/docs/bm25/reference/): Complete SQL reference, GUC parameters, benchmarks, and version history for vchord_bm25 and pg_tokenizer.

---

Backlinks:

- [All-in-one VectorChord Suite: Building Production-Ready RAG Solutions Directly in PostgreSQL](/blog/post/all-in-one-vectorchord-suite-building-production-ready-rag-solutions-directly-in-postgresql/)
- [Hybrid search with Postgres Native BM25 and VectorChord](/blog/post/hybrid-search-with-postgres-native-bm25-and-vectorchord/)
- [Introducing Vechord: Turn PostgreSQL into your search engine in a Pythonic way.](/blog/post/introducing-vechord-turn-postgresql-into-your-search-engine-in-a-pythonic-way/)
- [PostgreSQL Full-Text Search: Fast When Done Right (Debunking the Slow Myth)](/blog/post/postgresql-full-text-search-fast-when-done-right-debunking-the-slow-myth/)
- [Supercharge vector search with ColBERT rerank in PostgreSQL](/blog/post/supercharge-vector-search-with-colbert-rerank-in-postgresql/)
- [Documentation](/docs/)
- [About](/docs/about/)
- [pg_bestmatch](/docs/bestmatch/)
- [SQL Reference](/docs/bestmatch/reference/)
- [pg_tokenizer](/docs/tokenizer/)
- [vchord](/docs/vchord/)
- [Getting Started](/docs/vector/getting-started/)
- [Querying & Filtering](/docs/vector/querying/)