Installation

Linux Packages via pig (Recommended)

On EL 8/9/10, Debian 12/13, and Ubuntu 22.04/24.04 (PostgreSQL 14–18), you can install pre-built native packages from the Pigsty repository (see package matrix on pgext.cloud):

curl -fsSL https://repo.pigsty.io/pig | bash
pig repo add pgsql -u
pig ext add pg_tokenizer

Docker (vchord-suite)

The tensorchord/vchord-suite Docker image comes pre-configured with the Vchord extension suite:

• pg_tokenizer — Standalone multilingual tokenizer (pgsty/pg_tokenizer)
• vchord_bm25 — Native BM25 ranking index (pgsty/vchord_bm25)
• vchord — Scalable, disk-friendly vector search (pgsty/vchord)
• pgvector — Standard PostgreSQL vector data types

docker run \
  --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg18-latest

Connect with psql and enable the extension:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;

Configuration (shared_preload_libraries & search_path)

When installing via native packages or archives, add pg_tokenizer.so to shared_preload_libraries (required for model preloading) and include tokenizer_catalog in search_path:

psql -U postgres -c 'ALTER SYSTEM SET shared_preload_libraries = "pg_tokenizer.so"'
psql -U postgres -c 'ALTER SYSTEM SET search_path TO "$user", public, tokenizer_catalog'
sudo systemctl restart postgresql.service

Then enable the extension in your target database:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;

From Debian Package or ZIP Archive

Pre-built .deb and .zip release archives (GLIBC >= 2.35, x86_64 and aarch64) are available on the pgsty/pg_tokenizer Releases page:

# Debian / Ubuntu
sudo apt install ./postgresql-17-pg-tokenizer_*.deb

From Source (cargo-pgrx)

git clone https://github.com/pgsty/pg_tokenizer.git
cd pg_tokenizer
cargo pgrx install --sudo --release

Upgrading to 0.1.2

The 0.1.2 source changes are not yet a published release. After installing the matching library and SQL files, restart PostgreSQL to load the new preloaded library, then update each database as the extension owner/administrator:

ALTER EXTENSION pg_tokenizer UPDATE TO '0.1.2';

Both 0.1.0 and 0.1.1 have upgrade paths. The upgrade replaces trigger/helper definitions and grants catalog access without rebuilding custom vocabularies or changing existing token IDs. Six-argument helper dependencies remain valid. Standard cargo pgrx install/package and tools/package.sh include the migration scripts. Lindera’s separate dictionary-ID migration still requires re-tokenization as described in the model notes.

Database-owner permissions

Installation and upgrade grant pg_database_owner USAGE/CREATE on tokenizer_catalog, EXECUTE on its functions, SELECT/INSERT/UPDATE/DELETE on catalog and model tables, and USAGE/SELECT on their sequences. Future objects created in that schema by the installing role receive the same table/sequence/function defaults; newly created custom models also explicitly grant access. The grants follow a database ownership change.

A non-superuser database owner can create/drop its own custom models, analyzers, tokenizers, stopwords, and synonyms after an administrator installs the extension. Grants do not transfer table ownership: dropping a model created by a different role still needs that object’s owner or an administrator. Other application roles require explicit grants; the extension does not give PUBLIC schema access or catalog write privileges. Functions continue to execute with the caller’s privileges.