pg_bestmatch Documentation on Vchord

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

How Does `pg_bestmatch` Work?

Given a document $D$ and a query $Q$ tokenized into vocabulary terms $q_1, q_2, \dots, q_n$ , the Okapi BM25 relevance score is:

\text{BM25}(D, Q) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}

Notice that this sum is the inner product (dot product) of two sparse vectors over the vocabulary space $V$ (dimension $|V| = 30{,}522$ for bert-base-uncased):

Usage Guide

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through a complete BM25 text retrieval pipeline using pg_bestmatch and pgvector (sparsevec), followed by the equivalent configuration for pgvecto.rs (svector).

1. Enable Extensions & Set Search Path

CREATE EXTENSION IF NOT EXISTS pg_bestmatch;
CREATE EXTENSION IF NOT EXISTS vector;

SET search_path TO "$user", public, bm25_catalog;

2. Create the Documents Table & Load Data

CREATE TABLE documents (
    id bigint PRIMARY KEY,
    passage text
);

INSERT INTO documents (id, passage) VALUES
  (1, 'PostgreSQL is an advanced open-source relational database system.'),
  (2, 'VectorChord brings disk-friendly RaBitQ vector search to PostgreSQL.'),
  (3, 'BM25 is a bag-of-words ranking function used in information retrieval.');

3. Create BM25 Corpus Statistics

Call bm25_create(table_name, column_name, statistic_name, b, k) to register corpus statistics (b = 0.75 and k = 1.2 are standard defaults):

SQL Reference & Comparison

Mon, 01 Jan 0001 00:00:00 +0000

SQL Function Reference (`bm25_catalog`)

All pg_bestmatch functions reside in the bm25_catalog schema.

`bm25_create`

Creates BM25 corpus statistics for a specified table and text column.

bm25_create(
    table_name     regclass,
    column_name    text,
    statistic_name text,
    b              real DEFAULT 0.75,
    k              real DEFAULT 1.2
) RETURNS void

table_name: Target table containing the text documents.
column_name: Name of the text column to analyze.
statistic_name: Unique identifier for this BM25 statistic set (must be a valid SQL identifier).
b: BM25 document-length normalization factor (default 0.75).
k: BM25 term-frequency saturation parameter $k_1$ (default 1.2).

`bm25_refresh`

Refreshes the materialized corpus statistics for statistic_name after document insertions, updates, or deletions.

pg_bestmatch Documentation on Vchord

Getting Started

How Does pg_bestmatch Work?

Usage Guide

1. Enable Extensions & Set Search Path

2. Create the Documents Table & Load Data

3. Create BM25 Corpus Statistics

SQL Reference & Comparison

SQL Function Reference (bm25_catalog)

bm25_create

bm25_refresh

How Does `pg_bestmatch` Work?

SQL Function Reference (`bm25_catalog`)

`bm25_create`

`bm25_refresh`