Getting Started

• 1: Overview
• 2: Installation
• 3: Quick Start

1 - Overview

PostgreSQL includes built-in full-text search via tsvector, tsquery, ts_rank, and GIN indexes. While effective for boolean keyword matching (@@), PostgreSQL’s built-in ranking has two structural limitations for modern search and Retrieval-Augmented Generation (RAG) workloads:

1. Suboptimal Ranking Formula (ts_rank): PostgreSQL’s ts_rank and ts_rank_cd do not track corpus-wide Inverse Document Frequency (IDF) or Average Document Length ($\text{avgdl}$) across the index. Common words are weighted the same as rare, high-signal terms, and term frequency is not properly saturated.
2. Exhaustive Index Scans on ORDER BY ... LIMIT K: A PostgreSQL GIN index only accelerates boolean matching (WHERE tsv @@ tsquery). To return the Top-10 highest-ranked documents, PostgreSQL must fetch every matching row from the heap, compute ts_rank for all of them, and sort the entire result set.

How VectorChord-BM25 Solves Both Problems

VectorChord-BM25 (vchord_bm25) introduces a dedicated PostgreSQL index access method (USING bm25) and sparse BM25 scoring operator (<&>) specifically engineered for Top-K lexical retrieval:

Architecture Overview

flowchart LR
    subgraph Tokenization["1. Tokenization Layer"]
        T1["Raw Text"] --> T2["pg_tokenizer (v0.1–v0.3)\\nor to_tsvector (main)"]
        T2 --> T3["bm25vector / tsvector\\n(Token IDs + Term Frequencies)"]
    end
    subgraph Index["2. BM25 Index (USING bm25)"]
        I1["Growing Segment\\n(Append-only pages)"] --> I2["Sealed Segment\\n(Delta-Bitpacked 128-doc blocks)"]
        I2 --> I3["Term Statistics (DF)\\n+ 1-Byte Fieldnorm Array"]
    end
    subgraph Query["3. Top-K Query (ORDER BY vec <&> bm25query LIMIT k)"]
        Q1["Block-Max Score Upper Bounds"] --> Q2["Block-WeakAnd Pivot Pruning"]
        Q2 --> Q3["Top-K Heap (-BM25 Score)"]
    end
    T3 --> Index
    Index --> Query

Two API Generations

When working with vchord_bm25, be aware of the two API generations across releases:

1. Stable Releases (v0.1.0 – v0.3.0, schema bm25_catalog):

   • Uses custom data types bm25vector (sparse token_id:frequency map, e.g., {1:1, 2:2}) and bm25query.
   • Pairs with the pg_tokenizer extension (tokenizer_catalog), which converts text into INT[] token ID arrays that cast implicitly to bm25vector.
   • Index creation: CREATE INDEX idx ON documents USING bm25 (embedding bm25_ops);
   • Query construction: to_bm25query('idx', tokenize('query text', 'tokenizer_name'))

2. Next-Gen Native tsvector Architecture (main branch):

   • Operates directly on PostgreSQL’s built-in tsvector data type (CREATE OPERATOR CLASS bm25_ops FOR TYPE tsvector USING bm25), removing the requirement for a separate bm25vector column.
   • Index creation: CREATE INDEX idx ON documents USING bm25 (vector bm25_ops) WITH (options = $$ k1 = 1.2, b = 0.75 $$);
   • Query construction: to_bm25query(to_tsvector('english', 'query text'), 'idx')

2 - Installation

vchord_bm25 supports PostgreSQL 14, 15, 16, 17, and 18 on Linux (x86_64 and aarch64) and macOS.

Method 1: Pigsty / Native OS Packages (Recommended)

Pigsty provides pre-built native RPM and DEB packages for vchord, vchord_bm25, and pg_tokenizer across EL 8/9/10, Debian 12/13, and Ubuntu 22.04/24.04 for PostgreSQL 14–18.

# Install via Pigsty CLI (automatically resolves PG version and OS distro)
pig ext install vchord_bm25
pig ext install pg_tokenizer

Or install directly via apt / dnf once the Pigsty repository is configured:

# Debian / Ubuntu (replace 17 with your PostgreSQL major version)
sudo apt-get install postgresql-17-vchord-bm25 postgresql-17-pg-tokenizer

# RHEL / Rocky / AlmaLinux (replace 17 with your PostgreSQL major version)
sudo dnf install vchord_bm25_17 pg_tokenizer_17

Method 2: Docker (tensorchord/vchord-suite)

The official tensorchord/vchord-suite Docker image bundles PostgreSQL, VectorChord (vchord), VectorChord-BM25 (vchord_bm25), and pg_tokenizer out of the box:

docker run \\
  --name vchord-suite \\
  -e POSTGRES_PASSWORD=postgres \\
  -p 5432:5432 \\
  -d tensorchord/vchord-suite:pg18-latest

Connect with psql:

psql -h localhost -p 5432 -U postgres

Method 3: GitHub Release Packages (.deb & RHEL .zip)

Download prebuilt .deb or RHEL 9+ .zip packages for your PostgreSQL major version (14–18) and CPU architecture (amd64 / x86_64 or arm64 / aarch64) from the GitHub Releases pages of pgsty/vchord_bm25 (upstream supervc-stack/VectorChord-bm25) and pgsty/pg_tokenizer (upstream supervc-stack/pg_tokenizer.rs):

Debian / Ubuntu (.deb)

# Install vchord_bm25 (example for PostgreSQL 16 on x86_64)
curl -LO https://github.com/supervc-stack/VectorChord-bm25/releases/download/0.3.0/postgresql-16-vchord-bm25_0.3.0-1_amd64.deb
sudo apt-get install ./postgresql-16-vchord-bm25_0.3.0-1_amd64.deb

# Install pg_tokenizer (example for PostgreSQL 16 on x86_64)
curl -LO https://github.com/supervc-stack/pg_tokenizer.rs/releases/download/0.1.1/postgresql-16-pg-tokenizer_0.1.1-1_amd64.deb
sudo apt-get install ./postgresql-16-pg-tokenizer_0.1.1-1_amd64.deb

RHEL 9+ / Enterprise Linux (.zip Archive)

For RHEL 9+ systems when installing manually from the release .zip archive (requires glibc >= 2.34):

# 1. Locate PostgreSQL library and extension directories
pg_config --pkglibdir
# Example: /usr/pgsql-16/lib
pg_config --sharedir
# Example: /usr/pgsql-16/share

# 2. Download and extract the release archive
wget https://github.com/supervc-stack/pg_tokenizer.rs/releases/download/0.1.1/pg16-pg_tokenizer-0.1.1-x86_64-unknown-linux-gnu.zip
unzip pg16-pg_tokenizer-0.1.1-x86_64-unknown-linux-gnu.zip

# 3. Copy the shared library (.so) and extension control/SQL files
sudo cp ./usr/lib/postgresql/16/lib/pg_tokenizer.so $(pg_config --pkglibdir)
sudo cp ./usr/share/postgresql/16/extension/* $(pg_config --sharedir)/extension

Method 4: Build from Source (cargo-pgrx)

To compile pgsty/vchord_bm25 and pgsty/pg_tokenizer from source:

# 1. Install Rust toolchain and system dependencies (clang, libclang, pkg-config, PostgreSQL dev headers)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# 2. Build and install vchord_bm25
git clone https://github.com/pgsty/vchord_bm25.git
cd vchord_bm25
cargo install cargo-pgrx --version $(grep -o 'pgrx = "=[^"]*' Cargo.toml | cut -d = -f 3)
cargo pgrx init --pg17=$(which pg_config)
cargo pgrx install --release --pg-config $(which pg_config)

# 3. Build and install pg_tokenizer
git clone https://github.com/pgsty/pg_tokenizer.git
cd pg_tokenizer
cargo pgrx install --release --pg-config $(which pg_config)

cargo pgrx install --release --pg-config $(which pg_config) \
  --features "lindera-ipadic lindera-ipadic-neologd lindera-unidic lindera-ko-dic lindera-cc-cedict"

Development & Debugging (cargo-pgrx)

When developing or debugging vchord_bm25 or pg_tokenizer locally:

# Initialize pgrx managed PostgreSQL instances (PG 14–18)
cargo pgrx init

# Use the `dev-opt` profile to enable optimizations alongside debug symbols
RUST_BACKTRACE=1 cargo pgrx run --profile dev-opt

# Inside the interactive psql session, inspect the backend PID to attach gdb if needed:
SELECT pg_backend_pid();

Configure PostgreSQL & Enable Extensions

1. Shared Preload Libraries (Optional / Recommended for pg_tokenizer)

If you use pg_tokenizer and want preloaded vocabulary models initialized at server startup, add pg_tokenizer (and vchord if used) to shared_preload_libraries in postgresql.conf:

shared_preload_libraries = 'vchord.so, pg_tokenizer.so'

Restart PostgreSQL after modifying shared_preload_libraries. (vchord_bm25 itself does not require shared_preload_libraries and can be loaded dynamically per session via CREATE EXTENSION.)

2. Create Extensions and Set search_path

Connect to your target database and enable the extensions:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;

In v0.1.0–v0.3.0, vchord_bm25 installs its objects into the bm25_catalog schema and pg_tokenizer installs into tokenizer_catalog. Add both schemas to your search_path:

-- Per-session
SET search_path TO "$user", public, bm25_catalog, tokenizer_catalog;

-- Or persist for your database / role
ALTER DATABASE postgres SET search_path TO "$user", public, bm25_catalog, tokenizer_catalog;

Verify the installed version:

SELECT extname, extversion FROM pg_extension WHERE extname IN ('vchord_bm25', 'pg_tokenizer');

3 - Quick Start

This guide walks through creating a table, tokenizing documents, building a bm25 index, and executing Top-K BM25 queries.

Workflow A: bm25vector + pg_tokenizer (v0.1 – v0.3)

In released versions v0.1.0 through v0.3.0, vchord_bm25 pairs with pg_tokenizer to tokenize text into a bm25vector column.

1. Enable Extensions and Configure search_path

CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;

SET search_path TO "$user", public, bm25_catalog, tokenizer_catalog;

2. Create a Tokenizer

Use create_tokenizer with the built-in llmlingua2 (or bert_base_uncased) model:

SELECT create_tokenizer('my_tokenizer', $$
model = "llmlingua2"
$$);

3. Create a Table and Populate Documents

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    passage TEXT NOT NULL,
    embedding bm25vector
);

INSERT INTO documents (passage) VALUES
  ('PostgreSQL is a powerful, open-source object-relational database system. It has over 35 years of active development.'),
  ('Full-text search is a technique for searching a single computer-stored document or a collection in a full-text database.'),
  ('BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document.'),
  ('VectorChord-bm25 is a PostgreSQL extension that provides native BM25 ranking index using the Block-WeakAnd algorithm.');

4. Tokenize Documents into bm25vector

tokenize(text, tokenizer_name) returns an INT[] array of token IDs, which implicitly casts to bm25vector (aggregating duplicate token IDs into term frequencies):

UPDATE documents
SET embedding = tokenize(passage, 'my_tokenizer');

-- Inspect the sparse bm25vector representation ({token_id:term_frequency, ...})
SELECT id, embedding FROM documents LIMIT 2;

5. Build the bm25 Index

CREATE INDEX documents_embedding_bm25
ON documents
USING bm25 (embedding bm25_ops);

6. Run Top-K BM25 Queries

Use the <&> operator with to_bm25query(index_name, query_vector) in an ascending ORDER BY clause:

SELECT
    id,
    passage,
    -(embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL BM25 index', 'my_tokenizer'))) AS bm25_score
FROM documents
ORDER BY embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL BM25 index', 'my_tokenizer'))
LIMIT 10;

Workflow B: Native tsvector API (main Branch)

In the next-generation main branch rewrite, vchord_bm25 indexes PostgreSQL’s native tsvector type directly—no external tokenizer extension is required for standard languages.

1. Create Extension and Table

CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    passage TEXT NOT NULL,
    vector tsvector GENERATED ALWAYS AS (to_tsvector('english', passage)) STORED
);

INSERT INTO documents (passage) VALUES
  ('PostgreSQL is a powerful, open-source object-relational database system.'),
  ('Full-text search is a technique for searching documents in a full-text database.'),
  ('BM25 is a bag-of-words retrieval function that ranks documents based on query terms.'),
  ('VectorChord-bm25 provides a native BM25 ranking index in PostgreSQL.');

2. Build the bm25 Index on tsvector

CREATE INDEX documents_bm25_idx
ON documents
USING bm25 (vector bm25_ops)
WITH (options = $$
k1 = 1.2
b = 0.75
$$);

3. Query with to_bm25query(tsvector, regclass)

SELECT
    id,
    passage,
    -(vector <&> to_bm25query(to_tsvector('english', 'PostgreSQL BM25 index'), 'documents_bm25_idx')) AS bm25_score
FROM documents
ORDER BY vector <&> to_bm25query(to_tsvector('english', 'PostgreSQL BM25 index'), 'documents_bm25_idx')
LIMIT 10;