Getting Started on Vchord

Overview

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL includes built-in full-text search via tsvector, tsquery, ts_rank, and GIN indexes. While effective for boolean keyword matching (@@), PostgreSQL’s built-in ranking has two structural limitations for modern search and Retrieval-Augmented Generation (RAG) workloads:

Suboptimal Ranking Formula (ts_rank): PostgreSQL’s ts_rank and ts_rank_cd do not track corpus-wide Inverse Document Frequency (IDF) or Average Document Length ( $\text{avgdl}$ ) across the index. Common words are weighted the same as rare, high-signal terms, and term frequency is not properly saturated.
Exhaustive Index Scans on ORDER BY ... LIMIT K: A PostgreSQL GIN index only accelerates boolean matching (WHERE tsv @@ tsquery). To return the Top-10 highest-ranked documents, PostgreSQL must fetch every matching row from the heap, compute ts_rank for all of them, and sort the entire result set.

How VectorChord-BM25 Solves Both Problems

VectorChord-BM25 (vchord_bm25) introduces a dedicated PostgreSQL index access method (USING bm25) and sparse BM25 scoring operator (<&>) specifically engineered for Top-K lexical retrieval:

Installation

Mon, 01 Jan 0001 00:00:00 +0000

vchord_bm25 supports PostgreSQL 14, 15, 16, 17, and 18 on Linux (x86_64 and aarch64) and macOS.

Method 1: Pigsty / Native OS Packages (Recommended)

Pigsty provides pre-built native RPM and DEB packages for vchord, vchord_bm25, and pg_tokenizer across EL 8/9/10, Debian 12/13, and Ubuntu 22.04/24.04 for PostgreSQL 14–18.

# Install via Pigsty CLI (automatically resolves PG version and OS distro)
pig ext install vchord_bm25
pig ext install pg_tokenizer

Or install directly via apt / dnf once the Pigsty repository is configured:

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through creating a table, tokenizing documents, building a bm25 index, and executing Top-K BM25 queries.

Workflow A: `bm25vector` + `pg_tokenizer` (`v0.1` – `v0.3`)

In released versions v0.1.0 through v0.3.0, vchord_bm25 pairs with pg_tokenizer to tokenize text into a bm25vector column.

1. Enable Extensions and Configure `search_path`

CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;

SET search_path TO "$user", public, bm25_catalog, tokenizer_catalog;

2. Create a Tokenizer

Use create_tokenizer with the built-in llmlingua2 (or bert_base_uncased) model:

Getting Started on Vchord

Overview

How VectorChord-BM25 Solves Both Problems

Installation

Method 1: Pigsty / Native OS Packages (Recommended)

Quick Start

Workflow A: bm25vector + pg_tokenizer (v0.1 – v0.3)

1. Enable Extensions and Configure search_path

2. Create a Tokenizer

Workflow A: `bm25vector` + `pg_tokenizer` (`v0.1` – `v0.3`)

1. Enable Extensions and Configure `search_path`