Use Cases

• 1: Hybrid search with Postgres Native BM25 and VectorChord
• 2: Supercharge vector search with ColBERT rerank in PostgreSQL
• 3: Beyond Text: Unlock OCR-Free RAG in PostgreSQL with Modal & VectorChord

1 - Hybrid search with Postgres Native BM25 and VectorChord

In the era of RAG (Retrieval-Augmented Generation), efficiently retrieving relevant data from vast datasets is crucial for businesses and developers. Traditional keyword-based retrieval methods, such as those utilizing BM25 as a scoring mechanism, excel in ranking documents based on term frequency and exact keyword matches. This makes them highly effective for structured queries and keyword-heavy content. However, they struggle with understanding synonyms, contextual nuances, and semantic intent. Conversely, vector-based retrieval captures deep semantic meaning, enabling better generalization across varied queries but sometimes sacrificing precision in exact keyword matches. Hybrid Search bridges this gap by combining BM25’s precision with vector search’s contextual understanding, delivering faster, more accurate, and semantically aware results.

This article explores how to implement Hybrid Search in Postgres using VectorChord-bm25 for keyword-based retrieval and VectorChord for semantic search. By leveraging VectorChord-bm25’s BM25 ranking with the Block-WeakAnd algorithm and VectorChord’s advanced vector similarity capabilities, you can build a robust search system that seamlessly integrates keyword precision with semantic understanding. Whether you’re building a recommendation engine, a document retrieval system, or an enterprise search solution, this guide will walk you through the steps to unlock the full potential of hybrid search.

All related benchmark codes can be found here.

Hybrid Search Explained

Hybrid Search merges the results from vector search and keyword search. Vector search is based on the semantic similarity between the query and the documents, while keyword search relies on the BM25 ranking algorithm. However, it is important to note that BM25 itself is only responsible for scoring documents, while the broader keyword search process also includes steps such as tokenization, indexing, and query parsing. Hybrid Search utilizes reranker or fuses the results from both methods. Fuses include RRF (Reciprocal Rank Fusion) or weighted scoring for multiple recall strategies. Model-based rerank includes cross encoder models like bge-reranker-v2-m3 and multi-vector representation model ColBERT.

What is the VectorChord?

You may be familiar with pgvector, an open-source vector similarity search extension for Postgres. However, scalable or performance-critical scenarios may require a more advanced vector search solution. VectorChord is an excellent choice.

Check out the blog post “VectorChord: Store 400k Vectors for $1 in PostgreSQL” to learn more about its motivation and design.

What is VectorChord-BM25

VectorChord-BM25 is a PostgreSQL extension for keyword search. It not only implements BM25 ranking but also includes a tokenizer and a Block-WeakAnd index to improve speed.

What is BM25?

BM25 (Best Matching 25) is a probabilistic ranking function used in information retrieval to assess how well a document matches a query. It calculates relevance scores based on term frequency (TF) and inverse document frequency (IDF), while also applying document length normalization. The formula ensures that terms appearing frequently within a document (TF) and those rare across the corpus (IDF) are appropriately weighted, improving search accuracy and relevance. IDF measures how often a word appears across the document collection. The fewer the occurrences, the higher its value. TF represents the frequency of a specific word from the query appearing in the given document. A higher TF indicates a stronger relevance between the query and the document.

Why a Postgres native BM25 Ranking implementation is needed?

In our previous blog post, VectorChord-BM25: Revolutionizing PostgreSQL Search with BM25 Ranking — 3x Faster Than Elasticsearch, my colleague Allen has already provided a detailed and sufficient explanation, which can be referred to.

Tutorial: Use VectorChord-BM25 and VectorChord implement Hybrid Search

In this tutorial, we will guide you through the steps to implement Hybrid Search using VectorChord-BM25 and VectorChord in PostgreSQL. We will cover the following topics:

• Semantic Search with VectorChord

• Keyword Search with VectorChord-BM25

• Rerank

Prerequisites

1. Postgres with VectorChord-BM25 and VectorChord

If you want to reproduce the tutorial quickly, you can use the tensorchord/vchord-suite image to run multiple extensions which are provided by TensorChord.

You can run the following command to build and start the Postgres with VectorChord-BM25 and VectorChord.

docker run \
  --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg17-latest

CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;
\dx
pg_tokenizer | 0.1.0   | tokenizer_catalog | pg_tokenizer
vchord       | 0.3.0   | public            | vchord: Vector database plugin for Postgres, written in Rust, specifically designed for LLM
vchord_bm25  | 0.2.0   | bm25_catalog      | vchord_bm25: A postgresql extension for bm25 ranking algorithm
vector       | 0.8.0   | public            | vector data type and ivfflat and hnsw access methods

2. Prepare the Embedding Model and Data

For embedding, we can use a pre-trained embedding model like BGE-M3 to generate embeddings for the documents. BGE-M3 is a high-quality embedding model known for its versatility in multi-functionality, multi-linguality, and multi-granularity.

For validation, we use the BEIR dataset, a heterogeneous benchmark for information retrieval. It is easy to use and allows you to evaluate your models across 15+ diverse IR datasets.

First, you need to load your data into PostgreSQL. Then you can use the following SQL query to generate embeddings for the documents.

with self.conn.cursor() as cursor:
    cursor.execute(
        f"CREATE TABLE IF NOT EXISTS {self.dataset}_corpus (id TEXT, text TEXT, emb vector({self.vector_dim}), bm25 bm25vector);"
    )

    for did, doc in tqdm(zip(doc_ids, docs), desc="insert corpus"):
        emb = self.sentence_encoder.encode_doc(doc)
        cursor.execute(
            f"INSERT INTO {self.dataset}_corpus (id, text, emb) VALUES (%s, %s, %s)",
            (did, doc, emb),
        )

For embedding:

from FlagEmbedding import BGEM3FlagModel
class SentenceEmbedding:
    def __init__(self, model_name: str = "BAAI/bge-m3"):
        self.model = BGEM3FlagModel(
            model_name,
            use_fp16=True,
        )
    def encode_docs(self, documents: list[str]):
        return self.model.encode(
            documents,
            batch_size=32,
            max_length=8192,
        )['dense_vecs']

The data process and embedding generation code can be found here.

Semantic Search with VectorChord

For semantic search, we utilize the RaBitQ algorithm, which is highly optimized in the VectorChord PostgreSQL extension. RaBitQ is a quantization algorithm for high-dimensional spaces, designed to improve the storage and retrieval efficiency of high-dimensional data, such as embedded vectors.

It achieves this by mapping high-dimensional vectors to a low-dimensional discrete space while preserving the similarity information of the original vectors. This process reduces storage requirements and computational costs while maintaining high retrieval accuracy.

centroids = min(4 * int(self.num**0.5), self.num // 40)
ivf_config = f"""
residual_quantization = true
[build.internal]
lists = [{centroids}]
build_threads = {workers}
spherical_centroids = false
"""
with self.conn.cursor() as cursor:
    cursor.execute(f"SET max_parallel_maintenance_workers TO {workers}")
    cursor.execute(f"SET max_parallel_workers TO {workers}")
    cursor.execute(
        f"CREATE INDEX {self.dataset}_rabitq ON {self.dataset}_corpus USING vchordrq (emb vector_l2_ops) WITH (options = $${ivf_config}$$)"
    )

If you find that building the index on your own dataset is too slow, you can utilize external build to accelerate the process. For more details, please refer to our previous blog: Benefits and Steps of External Centroids Building in VectorChord.

Once the index is built, you can use the following SQL query to search for similar documents:

probe = int(0.1 * min(4 * int(self.num**0.5), self.num // 40))
with self.conn.cursor() as cursor:
    cursor.execute(f"SET vchordrq.probes = {probe}")
    cursor.execute(
        f"select q.id as qid, c.id, c.score from {self.dataset}_query q, lateral ("
        f"select id, {self.dataset}_corpus.emb <-> q.emb as score from "
        f"{self.dataset}_corpus order by score limit {topk}) c;"
    )

Key-Word Search with VectorChord-BM25

For keyword search, we need to use a tokenizer to convert the text into a BM25 vector, which is similar to a sparse vector that stores the vocabulary ID and frequency.

cursor.execute(
    f"SELECT create_tokenizer('{self.dataset}_token', $$",
    f"tokenizer = 'unicode'",
    f"stopwords = 'nltk'",
    f"table = '{self.dataset}_corpus'",
    f"column = 'text'",
    f"$$);"
)
cursor.execute(
    f"UPDATE {self.dataset}_corpus SET bm25 = tokenize(text, '{self.dataset}_token')"
)

Let me explain this operation in detail. The tokenize function is used to tokenize the text with the specified tokenizer. In this case, we used the Unicode tokenizer. The output of the tokenize function is a BM25 vector, which is a sparse vector that stores the vocabulary ID and frequency of each word in the text. For example, 1035:7 means the word with vocabulary ID 1035 appears 7 times in the text.

postgres=# select bm25 from fiqa_corpus limit 1;
-- Output: {1035:7, 1041:1, 1996:1, 1997:1, 1999:3, 2010:3, 2015:7, 2019:1, 2022:1, 2028:4, 2036:2, 2041:1, 2051:2, 2054...

After creating the index, you can calculate the BM25 score between the query and the vectors. Note that the BM25 score is negative, meaning that the higher the score, the more relevant the document is. We intentionally make it negative so that you can use the default ORDER BY to retrieve the most relevant documents first.

with self.conn.cursor() as cursor:
    cursor.execute(
        f"SELECT q.id AS qid, c.id, c.score FROM {self.dataset}_query q, LATERAL ("
        f"SELECT id, {self.dataset}_corpus.bm25 <&> to_bm25query('{self.dataset}_text_bm25', q.text , '{self.dataset}_token') AS score "
        f"FROM {self.dataset}_corpus "
        f"ORDER BY score "
        f"LIMIT {topk}) c;"
    )

Rerank/Fuse

Once you obtain the results from both semantic search and keyword search, you can use a fuse or rerank process to merge the results.

1. Fuse (Reciprocal Rank Fusion (RRF))

The advantage of RRF lies in its independence from specific score units; it is based on ranking for fusion, making it suitable for sorting systems with different scoring criteria.

Among them:

• d is the document.

• n is the number of ranking systems.

• rank is the ranking in the sorting system.

• k is the adjustment parameter, the value is usually 60 (empirical value), used to control the impact of ranking on scores.

for rank, (query_id, doc_id, _) in enumerate(result, start=1):
    if query_id not in rrf_scores:
        rrf_scores[query_id] = {}
    if doc_id not in rrf_scores[query_id]:
        rrf_scores[query_id][doc_id] = 0
    # Calculate and accumulate RRF scores
    rrf_scores[query_id][doc_id] += 1 / (k + rank)

2. Cross-Encoder model Rerank

In semantic search, we already use Bi-Encoder vectorized the documents and queries separately. But this independent encoding leads to a lack of interaction between the query and the document. Cross-Encoder model will input the query and document as a whole into the model, the model will see the content of both at the same time, capturing the fine-grained semantic relationship between them through the Transformer layer. Compared with RRF, Cross-Encoder model can capture the fine-grained semantic relationship between the query and the document, and it can be more accurate but slower.

reranker = FlagReranker(
    'BAAI/bge-reranker-v2-m3',
    query_max_length=256,
    passage_max_length=512,
    use_fp16=True,
    devices=["cuda:0"] # change ["cpu"] if you do not have gpu, but it will be very slow
) # Setting use_fp16 to True speeds up computation with a slight performance degradation

for query_id, docs in tqdm(results.items()):
    scores = reranker.compute_score(pairs, normalize=True)
    for i, doc_id in enumerate(docs):
        bge_scores[query_id][doc_id] = scores[i]

Evaluation

We have tested the NDCG@10 with those methods on several BEIR datasets. Here are the top10 results:

The table above demonstrates several conclusions:

• Compared to BM25 and semantic search, the cross-encoder model rerank can significantly improve search performance across different datasets.

• On some datasets, RRF may lead the performance decrease. Please conduct verification tests before deciding whether to use RRF. If effective, choosing RRF will be a very economical choice, as it is much, much faster than reranking with the cross-encoder model, and almost no resource consumption.

All related benchmark codes can be found here. If you have any questions, please feel free to contact us on Discord or email us at [email protected].

Future Work

The results highlight the great potential of the hybrid search method. In the future, we will focus on the following aspects to advance RAG:

• Improve the reranking method: Explore the other model-based rerank, such as ColBERT or ColPali, to enhance reranking performance.
• Integrate graph search: Investigate the use of the Apache AGE extension for graph search and integrate it with the hybrid search method to further improve search performance.

References

• https://github.com/pgsty/vchord
• https://github.com/pgsty/vchord_bm25
• https://techcommunity.microsoft.com/blog/adforpostgresql/introducing-the-graphrag-solution-for-azure-database-for-postgresql/4299871
• https://www.pinecone.io/learn/hybrid-search-intro/
• https://www.pinecone.io/learn/series/rag/rerankers/
• https://blog.vespa.ai/improving-zero-shot-ranking-with-vespa-part-two/

2 - Supercharge vector search with ColBERT rerank in PostgreSQL

Traditional vector search methods typically employ sentence embeddings to locate similar content. However, generating sentence embeddings through pooling token embeddings can potentially sacrifice fine-grained details present at the token level. ColBERT overcomes this by representing text as token-level multi-vectors rather than a single, aggregated vector. This approach, leveraging contextual late interaction at the token level, allows ColBERT to retain more nuanced information and improve search accuracy compared to methods relying solely on sentence embeddings.

As illustrated in the above image, ColBERT encodes each document/query into a list of token vectors and computes the MaxSim during the query time.

Token-level late interaction requires more computing power and storage. This makes using ColBERT search in large datasets challenging, especially when low latency is important.

One possible solution is to combine sentence-level vector search with token-level late interaction rerank, which leverages the efficiency of approximate vector search and the high quality of multi-vector similarity search.

The multi-vector approach is not limited to pure text retrieval tasks; it can also be used in visual document understanding. For multimodal retrieval models, state-of-the-art models like ColPali and ColQwen directly encode document images into multi-vectors and demonstrate stronger performance compared to OCR-to-text approaches.

This blog will demonstrate using the PostgreSQL extension VectorChord and pgvector with ColBERT rerank.

Tutorial

Assume we already have the documents, let’s create a table to store all of them:

import psycopg
from pgvector.psycopg import register_vector

class PgClient:
    def __init__(self, url: str, dataset: str, sentence_emb_dim: int, token_emb_dim: int):
        self.dataset = dataset
        self.sentence_emb_dim = sentence_emb_dim
        self.token_emb_dim = token_emb_dim
        self.conn = psycopg.connect(url, autocommit=True)
        with self.conn.cursor() as cursor:
            cursor.execute("CREATE EXTENSION IF NOT EXISTS vchord CASCADE;")
        register_vector(self.conn)

    def create(self):
        with self.conn.cursor() as cursor:
            cursor.execute(
                f"CREATE TABLE IF NOT EXISTS {self.dataset}_corpus "
                "(id INT BY DEFAULT AS IDENTITY PRIMARY KEY, text TEXT, "
                f"emb vector({self.sentence_emb_dim}), embs vector({self.token_emb_dim})[]);"
            )

Here we created a table with sentence-level embedding and token-level embeddings.

There are numerous embedding APIs and open-source models. You can choose the one that fits your use case.

For token-level embedding:

from colbert.infra import ColBERTConfig
from colbert.modeling.checkpoint import Checkpoint

class TokenEncoder:
    def __init__(self):
        self.config = ColBERTConfig(doc_maxlen=220, query_maxlen=32)
        self.checkpoint = Checkpoint(
            "colbert-ir/colbertv2.0", colbert_config=self.config, verbose=0
        )

    def encode_doc(self, doc: str):
        return self.checkpoint.docFromText([doc], keep_dims=False)[0].numpy()

    def encode_query(self, query: str):
        return self.checkpoint.queryFromText([query])[0].numpy()

ColBERT model produces 128-dim vectors by default.

To insert the data:

class PgClient:
    ...
    def insert(self, documents: list[str]):
        with self.conn.cursor() as cursor:
            for doc in tqdm(documents):
                sentence_emb = sentence_encoder.encode_doc(doc)
                token_embs = [emb for emb in token_encoder.encode(doc)]
                cursor.execute(
                    f"INSERT INTO {self.dataset}_corpus (text, emb, embs) VALUES (%s, %s, %s)"
                    (doc, sentence_emb, token_embs)
                )

For the vector search part, we can use VectorChord to build a high-performance RaBitQ index:

class PgClient:
    ...
    def index(self, num_doc: int, workers: int):
        n_cluster = 1 << math.ceil(math.log2(num_doc ** 0.5 * 4))
        config = f"""
        residual_quantization = true
        [build.internal]
        lists = [{n_cluster}]
        build_threads = {workers}
        spherical_centroids = false
        """
        with self.conn.cursor() as cursor:
            cursor.execute(f"SET max_parallel_maintenance_workers TO {workers}")
            cursor.execute(f"SET max_parallel_workers TO {workers}")
            cursor.execute(
                f"CREATE INDEX {self.dataset}_rabitq ON {self.dataset}_corpus USING "
                f"vchordrq (emb vector_l2_ops) WITH (options = $${config}$$)"
            )

To speed up the indexing building process, we can utilize the external centroids build. Check “Benefits and Steps of External Centroids Building in VectorChord” for more details.

Now, we can query the PostgreSQL:

class PgClient:
    ...
    def query(self, doc: str, topk: int):
        sentence_emb = sentence_encoder.encode_query(doc)
        with self.conn.cursor() as cursor:
            cursor.execute(
                f"SELECT id, text FROM {self.dataset}_corpus ORDER BY emb <-> %s LIMIT {topk}"
            )
            res = cursor.fetchall()
        return res

To support MaxSim rerank, we’ll need to create a function:

class PgClient:
    def __init__(self, url: str, dataset: str, sentence_emb_dim: int, token_emb_dim: int):
        ...
        self.conn.execute("""
            CREATE OR REPLACE FUNCTION max_sim(document vector[], query vector[]) RETURNS double precision AS $$
            WITH queries AS (
                SELECT row_number() OVER () AS query_number, * FROM (SELECT unnest(query) AS query)
            ),
            documents AS (
                SELECT unnest(document) AS document
            ),
            similarities AS (
                SELECT query_number, document <=> query AS similarity FROM queries CROSS JOIN documents
            ),
            max_similarities AS (
                SELECT MAX(similarity) AS max_similarity FROM similarities GROUP BY query_number
            )
            SELECT SUM(max_similarity) FROM max_similarities
            $$ LANGUAGE SQL
        """)

Now, we can rerank the documents retrieved by vector search:

class PgClient:
    def rerank(self, query: str, ids: list[int], topk: int):
        token_embs = [emb for emb in token_encoder.encode_query(query)]
        with self.conn.cursor() as cursor:
            cursor.execute(
                f"SELECT id, text FROM {self.dataset}_corpus WHERE id = ANY(%s) ORDER BY "
                f"max_sim(embs, %s) DESC LIMIT {topk}"
                (ids, token_embs)
            )
            res = cursor.fetchall()
        return res

Evaluation

We have tested this method on several BEIR datasets. Here are the results:

This shows that ColBERT reranking can significantly enhance the results of vector searches.

All the related benchmark codes can be found here.

Future work

Vector search and full-text search with ColBERT rerank can further improve performance. We’re also working on the PostgreSQL BM25 extensions. Stay tuned.

References

• https://www.answer.ai/posts/colbert-pooling.html

• https://github.com/stanford-futuredata/ColBERT

• https://github.com/pgsty/vchord

• https://github.com/pgsty/vchord_bm25

• https://huggingface.co/blog/manu/colpali

3 - Beyond Text: Unlock OCR-Free RAG in PostgreSQL with Modal & VectorChord

Building effective Retrieval-Augmented Generation (RAG) systems for documents often feels like wrestling with messy, complex pipelines. Especially when dealing with PDFs or scanned images, traditional methods rely heavily on Optical Character Recognition (OCR) and layout analysis. These steps can be slow, error-prone, and often lose crucial visual context like tables, figures, and formatting. But what if you could query documents based on how they look, not just the extracted text?

This post is your guide to building exactly that: an OCR-free RAG system directly within your familiar PostgreSQL database. We’ll leverage the power of the ColQwen2 Vision Language Model, the efficiency of VectorChord for multi-vector search in Postgres, and the scalability of Modal for GPU-powered embedding generation. Get ready to simplify your RAG stack and potentially boost your retrieval accuracy, all without complex pre-processing.

We’ll cover:

• What ColQwen2 is and why it’s a game-changer.

• How VectorChord makes advanced vector search possible in Postgres.

• A step-by-step tutorial to build and evaluate the system.

What is ColQwen2? The Power of Visual Understanding

To grasp ColQwen2, let’s first look at its foundation: ColPali. As introduced in the paper “ColPali: Efficient Document Retrieval with Vision Language Models”, ColPali represents a novel approach using Vision Language Models (VLMs). Instead of relying on imperfect OCR, it directly indexes documents using their rich visual features – text, images, tables, layout, everything the eye can see.

Think about the limitations of traditional OCR: complex layouts get mangled, tables become gibberish, and images are often ignored entirely. It’s like trying to understand a book by only reading a flawed transcript. ColPali avoids this by using a powerful VLM (originally PaliGemma) to create embeddings that capture the document’s holistic visual nature. Two key concepts make it shine:

1. Contextualized Vision Embeddings: Generating rich embeddings directly from the document image using a VLM.

2. Late Interaction: This clever technique allows the query’s textual meaning to directly interact with the document’s detailed visual features at search time. It’s not just matching text summaries; it’s comparing the query concept against the visual evidence within the document page.

The ColPali architecture (Image from the ColPali paper)

ColQwen2 builds upon this powerful ColPali architecture but swaps the underlying VLM for the more recent Qwen2-VL-2B. It generates ColBERT-style multi-vector representations, capturing fine-grained details from both text and images. As seen on the vidore-leaderboard, ColQwen2 delivers impressive performance with practical model size.

How Does VectorChord Enable ColQwen2 in Postgres?

This is where VectorChord becomes the crucial piece of the puzzle, bringing this cutting-edge VLM capability into your PostgreSQL database. ColQwen2 (and ColPali) relies heavily on those multi-vector representations and the Late Interaction mechanism, specifically requiring an efficient MaxSim (Maximum Similarity) operation.

Calculating MaxSim – finding the highest similarity score between any vector in the query set and any vector in the document set – can be computationally brutal, especially across millions of document vectors. VectorChord tackles this head-on:

• Native Multi-Vector Support: It’s designed from the ground up to handle multi-vector data efficiently within Postgres.

• Optimized MaxSim: Drawing inspiration from the WARP paper, VectorChord uses techniques like dynamic similarity imputation to dramatically speed up MaxSim calculations, making large-scale visual document retrieval feasible.

• Hybrid Search Ready: Beyond multi-vector, it also supports dense, sparse, and hybrid search (check out our previous post!).

• Scalable & Disk-Friendly: Designed for performance without demanding excessive resources.

In short, VectorChord transforms PostgreSQL into a powerhouse capable of handling the advanced vector search techniques required by models like ColQwen2 or ColPali.

Tutorial: Building Your OCR-Free RAG System

Alright, theory’s great, but let’s roll up our sleeves and build this thing! We’ll walk through setting up the environment, processing data using Modal for scalable embedding generation, indexing into VectorChord within Postgres, and finally, evaluating our shiny new OCR-free RAG system.

Prerequisites

Before we start, ensure you have:

• A PostgreSQL instance (Docker recommended) with the VectorChord extension installed OR a VectorChord Cloud cluster.

• A Modal account (free tier available). Modal’s fast GPU provisioning and scaling are perfect for the embedding generation step. To efficiently process a large volume of documents using the ColQwen2 model, it’s crucial to leverage Modal’s rapid startup and GPU expansion features. This approach will significantly reduce the time required for local processing.

If you want to reproduce the tutorial quickly, you can use the tensorchord/vchord-suite image to run multiple extensions that TensorChord provides.

You can run the following command to build and start Postgres with VectorChord-BM25 and VectorChord.

docker run \
  --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg17-latest

CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;
\dx
pg_tokenizer | 0.1.0   | tokenizer_catalog | pg_tokenizer
vchord       | 0.3.0   | public            | vchord: Vector database plugin for Postgres, written in Rust, specifically designed for LLM
vchord_bm25  | 0.2.0   | bm25_catalog      | vchord_bm25: A postgresql extension for bm25 ranking algorithm
vector       | 0.8.0   | public            | vector data type and ivfflat and hnsw access methods

Set up Modal:

$ pip install modal
$ python3 -m modal setup  # click the link to authorize

Step 1: Load the Data (Using Modal Volumes)

We’ll use the ViDoRe Benchmark dataset. To handle this data efficiently across potentially distributed Modal functions, we’ll download it to a Modal Volume. Volumes provide persistent, shared storage ideal for this ‘download once, process many times’ scenario.

image = modal.Image.debian_slim().pip_install("datasets","huggingface_hub","Pillow")
DATASET_DIR = "/data"
DATASET_VOLUME  = modal.Volume.from_name(
    "colpali-dataset", create_if_missing=True
)
app = modal.App(image=image)

@app.function(volumes={DATASET_DIR: DATASET_VOLUME}, timeout=3000)
def download_dataset(cache=False) -> None:
    from datasets import load_dataset
    from tqdm import tqdm

    collection_dataset_names = get_collection_dataset_names("vidore/vidore-benchmark-667173f98e70a1c0fa4db00d")
    for dataset_name in tqdm(collection_dataset_names, desc="vidore benchmark dataset(s)"):
        dataset = load_dataset(dataset_name, split="test",num_proc=10)
        unique_indices = dataset.to_pandas().drop_duplicates(subset="image_filename", keep="first").index #to remove repeating PDF pages with different queries
        dataset = dataset.select(unique_indices)
        dataset.save_to_disk(f"{DATASET_DIR}/{dataset_name}")

Modal offers a straightforward Python API for interacting with its platform. With modal.Image, you can set the base image for the Modal App, and the function decorator helps define the function to be executed within the Modal application. To download the dataset to the Modal Volume, execute the following command. Modal will then automatically build the Docker image and promptly launch the container to run your function.

Run the download function:

$ modal run dataset.py::download_dataset

Modal handles building the environment and running the script in the cloud.

Step 2: Process Data & Generate Embeddings (With Modal & ColQwen2)

This is the heavy lifting: converting document images into ColQwen2 multi-vector embeddings. Doing this locally for many documents would be slow. Modal shines here:

• Easy GPU Access & Autoscaling: Launch a ColQwen2 model service on Modal and use it to generate the image embeddings and query embeddings for the dataset. The decision to implement an HTTP service instead of using the SDK directly was made to ensure seamless auto-scaling during large-scale embedding operations. If you want to deploy the embedding service as a persistent web endpoint, you can directly change modal.cls to modal.web_server for ColPaliServer and running modal deploy.

• Recovery: We checkpoint progress to a separate Modal Volume, allowing us to resume embedding generation if interrupted.

# embedding.py (Illustrative - keep original code)
# ... imports, volumes setup ...
modal_app = modal.App() # Define app

# Function to coordinate embedding generation
@modal_app.function(...)
def embed_dataset(down_scale: float = 1.0, batch_size: int = BATCH_SIZE):
    # ... (logic for loading dataset names, handling checkpoints) ...
    colpali_server = ColPaliServer() # Get handle to our GPU class
    # ... (loop through datasets, batch items, call server for embeddings) ...
    # ... (save embeddings and update checkpoint) ...
    print("Embedding generation complete.")

# server.py (Illustrative - keep original code)
# Class running on GPU to serve embedding requests
@modal_app.cls(gpu=GPU_CONFIG, ...)
class ColPaliServer:
    @modal.enter()
    def load_model_and_start_server(self):
        # ... (Load ColQwen2 model, start internal FastAPI server) ...
        self.client = httpx.AsyncClient(...) # Client to talk to internal server

    @modal.exit()
    def shutdown_server(self):
        # ... (Cleanup) ...

    # Method called by embed_dataset function
    @modal.method()
    async def embed_images(self, images: List[str]) -> np.ndarray:
        # ... (Prepare batch, send request to internal server via self.client) ...
        # ... (Decode response using msgspec for speed) ...
        return embeddings_numpy_array

    # ... (Potentially add embed_queries method too) ...

# colpali.py
class ColPaliModel:
    def __init__(self, model_name: str = "vidore/colqwen2-v1.0", cache_dir: str="/model"):
        # ...
        if self.model_name == "vidore/colqwen2-v1.0":
            from colpali_engine.models import ColQwen2, ColQwen2Processor
            from transformers.utils.import_utils import is_flash_attn_2_available

            # load model
            model = ColQwen2.from_pretrained(
                self.model_name,
                torch_dtype=torch.bfloat16,
                device_map="cuda:0",
                attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,
                cache_dir=self.cache_dir,
            ).eval()

            colpali_processor = ColQwen2Processor.from_pretrained(
                self.model_name,
                cache_dir=self.cache_dir,
            )
    # ... functions for embedding images and queries
    # @modal.method()
    # async def batch_embed_images(self, images_base64: List[str]) -> np.ndarray:
    # @modal.method()
    # async def batch_embed_queries(self, queries: List[str]) -> np.ndarray:

Generate the embeddings:

$ modal run embedding.py::embed_dataset

Step 3: Create Index in VectorChord (Using Vechord SDK)

With embeddings generated and saved in a Modal Volume, we first need to bring them local.

$ modal volume get colpali-embedding-checkpoint /path/to/local/vidore_embeddings

Now, we’ll use the vechord SDK to load these embeddings into our PostgreSQL database and create the necessary multi-vector index. vechord provides a Pythonic, ORM-like way to interact with VectorChord in Postgres.

%[https://github.com/tensorchord/vechord]

MultiVector = List[Vector[128]] # Assuming 128 dimensions for ColQwen2

lists = 2500 # lists is the number of the cluster
# Define the database table schema using vechord
class Image(Table, kw_only=True):
    uid: Optional[PrimaryKeyAutoIncrease] = None
    image_embedding: Annotated[MultiVector, MultiVectorIndex(lists=lists)] # Stores the document's visual embeddings
    query_embedding: Annotated[MultiVector, MultiVectorIndex(lists=lists)]  # Stores the query's embeddings
    query: str = None
    dataset: Optional[str] = None
    dataset_id: Optional[int] = None

# Connect to your PostgreSQL database
# Ensure the URL points to your local Docker or VectorChord Cloud instance
DB_URL = "postgresql://postgres:[email protected]:5432/postgres" # Default DB
vr = VechordRegistry("colpali", DB_URL)
vr.register([Image]) # Creates the table and multi-vector index if they don't exist

# Function to load embeddings from disk and yield Image objects
@vr.inject(output=Image)
def load_image_embeddings(path: str) -> Iterator[Image]:
    # ... (logic to load numpy arrays from disk, convert to List[Vector]) ...
    # ... (yield Image(...) instances) ...
    print(f"Loaded embeddings from {path}")

if __name__ == "__main__":
    # Path where you downloaded the embeddings from Modal
    embedding_dir = "/path/to/local/vidore_embeddings"
    load_image_embeddings(embedding_dir) # This triggers vechord to insert data
    print("Data loaded and indexed into VectorChord.")

Run the indexer script, vechord handles table creation, index creation, and data insertion.

Step 4: Evaluation - Does it Work?

The final step is crucial: evaluating retrieval performance. Does this OCR-free system deliver accurate results, and can it be fast? We’ll use vechord to run queries against our indexed data and look at NDCG@10 and Recall@10. We’ll also test the impact of VectorChord’s WARP optimization, which accelerates MaxSim calculations. In this tutorial, we will use vidore/arxivqa_test_subsampled dataset for evaluation queries, the data are shown below.

TOP_K = 10

# Define structure for results (optional but good practice)
class Evaluation(msgspec.Struct):
    map: float
    ndcg: float
    recall: float

TOP_K = 10
def evaluate(queries: list[Image], probes: int, max_maxsim_tuples: int) -> list[Evaluation]:
    result  = []
    for query in queries:
        vector = query.query_embedding
        docs: list[Image] = vr.search_by_multivec(
            Image, vector, topk=TOP_K, probe=probes, max_maxsim_tuples=max_maxsim_tuples
        )
        score = BaseEvaluator.evaluate_one(query.uid, [doc.uid for doc in docs])
        result.append(Evaluation(
            map=score.get("map"),
            ndcg=score.get("ndcg"),
            recall=score.get(f"recall_{TOP_K}"),
        ))
    return result

if __name__ == "__main__":
    # Select some queries from the benchmark dataset stored in the DB
    # Example: Get 100 queries from a specific subset
    test_queries: list[Image] = vr.select_by(
        Image.partial_init(dataset="vidore/arxivqa_test_subsampled"), limit=100
    )

    # ... optimize the param ...
    res: list[Evaluation] = evaluate(queries, probes=probes, maxsim_refine=maxsim_refine)
    print("ndcg@10", sum(r.ndcg for r in res) / len(res))
    print("recall@10", sum(r.recall for r in res) / len(res))
    print(f"Total execution time: {total_time:.4f} seconds")

Running the evaluation script yields impressive results:

# Disable WARP
ndcg@10 0.8615
recall@10 0.92
Total execution time: 810 seconds # Baseline accuracy and time

# Enable WARP
# There is no need to focus on specific times,
# only relative times need to be taken into account,
# as the tests were performed on a local model with poor performance.
ndcg@10 0.8353
recall@10 0.90
Total execution time: 41 seconds # Dramatic WARP Speed Boost

Analysis:

• High Baseline Accuracy: First, let’s look at the baseline performance without WARP optimization. The system achieves an excellent NDCG@10 of 0.8615 and Recall@10 of 0.92. This confirms the fundamental effectiveness of ColQwen2 embeddings paired with VectorChord’s precise MaxSim search, delivering state-of-the-art results for visual document retrieval even without speed optimizations.

• Dramatic WARP Speed Boost: Now, observe the impact of enabling VectorChord’s WARP optimization. The total execution time plummets from a substantial 810 seconds down to just 41 seconds! This represents a massive ~18.7x speedup for the evaluation query set. This clearly demonstrates WARP’s power in dramatically accelerating the computationally intensive MaxSim operations required for late-interaction models like ColQwen2.

• Minimal Accuracy Trade-off: Impressively, this significant speed gain comes at the cost of only a very minor dip in retrieval accuracy. The NDCG@10 slightly decreases to 0.8353 (a difference of less than 0.3), and Recall@10 reduces minimally to 0.9 (a difference of only 0.02).

Conclusion: Visual RAG Made Simpler in Postgres

In this tutorial, we successfully built a high-performance, OCR-free RAG system by bringing together the visual understanding capabilities of ColQwen2, the scalable multi-vector search of VectorChord within PostgreSQL, and the efficient GPU processing of Modal. We saw how this stack allows us to directly query documents based on their visual content, bypassing the pitfalls of traditional OCR pipelines.

The ability to seamlessly integrate this visual dimension into your RAG system directly within Postgres opens up exciting possibilities for applications dealing with visually rich documents like scientific papers, invoices, product manuals, historical archives, and much more.

Ready to try it yourself?

• Explore the Code: https://github.com/xieydd/vectorchord-colqwen2

• Dive Deeper into VectorChord: Check out the VectorChord documentation or try the hassle-free VectorChord Cloud.

• Experiment: Try different VLMs or datasets.

• Share Your Thoughts: Let us know your experiences or questions in the comments below!

This approach represents a significant step towards simpler, more robust, and visually-aware document retrieval systems.

References

• https://huggingface.co/vidore/colqwen2-v1.0

• https://huggingface.co/blog/manu/colpali

• https://blog.vespa.ai/scaling-colpali-to-billions/