Benchmarks

• 1: Price Benchmark
• 2: Performance Benchmarking
• 3: Benchmark with pgvector
• 4: Benchmark with Elasticsearch
• 5: Benchmark with pgvectorscale

1 - Price Benchmark

Using VectorChord, you can achieve a QPS of 131 with 0.95 precision on 100 million 768-dimensional vectors for the top 10 queries. This setup costs only $250 monthly and can be hosted on a single machine. If you are interested in the details of our implementation, please refer to our blog VectorChord: Store 400k Vectors for $1 in PostgreSQL.

This means you can store 400k vectors for only $1, allowing you to save significantly: 6x more vectors compared to Pinecone (storage optimized instance) and 26x more than pgvector/pgvecto.rs for the same price.

In the monthly cost comparison for storing vectors, based on MyScale benchmark data, the chart highlights how VectorChord emerges as an affordable option, priced at just $247 for storing 100 million vectors. In contrast, Pinecone, despite its optimized storage, costs $1,600 per month, while Qdrant is priced at $4,374. pgvector/pgvecto.rs has a considerably higher cost of $6,580.

2 - Performance Benchmarking

Here we present a comprehensive benchmark comparing our implementation of VectorChord with other vector databases, focusing on QPS and scalability. If you are interested in the details of our implementation, please refer to our blog VectorChord: Store 400k Vectors for $1 in PostgreSQL.

QPS

LAION 5M

In LAION 5M dataset, VectorChord achieved more than 3x QPS at 0.95 recall compared to different vector databases, including MyScale, pgvector, pinecone, weaviate, qdrant and zilliz(milvus).

LAION 100M on i4i.xlarge

This setup utilizes 4 CPUs and 32 GB of memory, with 937 GB of SSD storage for the 100 million vectors. We achieved a QPS of 16.2 at a recall rate of 0.95 for the top 10 results and 4.3 at 0.95 for the top 100, using a single-threaded query. The results are truly impressive.

Scalability

If you are interested VectorChord scalable capability, you can refer our blog Vector Search at 10,000 QPS in PostgreSQL with VectorChord.

LAION 100M && LAION 5M

3 - Benchmark with pgvector

In this section, we compare VectorChord with pgvector, a PostgreSQL extension that provides vector similarity search capabilities. We will focus on four key metrics: QPS (Queries Per Second), Recall@100, Index Building Time, and Insertion Time. If you are interested in the details of our implementation, please refer to our blog Vector Search Over PostgreSQL: A Comparative Analysis of Memory and Disk Solutions.

QPS & Recall

Here are some initial benchmark results for the GIST dataset, which consists of 1 million vectors in 960 dimensions. With equivalent recall, VectorChord’s QPS could be twice that of pgvector. More details will be provided in the Benchmark section.

Index Build Time & Insert Time & Query Completion Time

VectorChord takes 186 seconds to build the index by utilizing a separate machine for KMeans clustering, making it 16 times faster than pgvector. Additionally, the insert time is also 14 times faster than that of pgvector. As we know, indexing is the most resource-intensive part of vector databases, requiring significant computation and increasing the demand for CPUs and memory. By utilizing a more capable machine to build the index and then importing it to a smaller machine for querying, it becomes possible to support billions of vectors on a single machine.

4 - Benchmark with Elasticsearch

Here we present a comprehensive benchmark comparing our implementation of VectorChord-BM25 with ElasticSearch, focusing on two key metrics: QPS (Queries Per Second) and NDCG@10 (Normalized Discounted Cumulative Gain at rank 10). If you are interested in the details of our implementation, please refer to our blog VectorChord-BM25: Revolutionize PostgreSQL Search with BM25 Ranking — 3x Faster Than ElasticSearch.

RPS

We compare the Top1000 Queries Per Second (QPS)—a metric that measures how many queries a system can process per second while retrieving the top 1000 results for each query—between our implementation and ElasticSearch across various datasets from bm25-benchmarks. On average, our implementation achieves 3 times higher QPS compared to ElasticSearch across the tested datasets, showcasing its efficiency and scalability.

NDCG@10

We have dedicated substantial effort to align VectorChord-BM25 with ElasticSearch’s behavior, ensuring a fair and precise comparison. As demonstrated in the table, our implementation achieves NDCG@10 scores that are comparable across most datasets, with certain cases even surpassing ElasticSearch (e.g., trec-covid and scifact).

5 - Benchmark with pgvectorscale

In this section, we compare VectorChord with pgvectorscale, a PostgreSQL extension that provides vector similarity search capabilities. We will focus on four key metrics: QPS (Queries Per Second), Recall@10/100, Index Building Time, and Insertion Time. Memory and disk performance will also be discussed. If you are interested in the details of our implementation, please refer to our blog Vector Search Over PostgreSQL: A Comparative Analysis of Memory and Disk Solutions.

QPS & Recall

Memory

The graphs illustrate the trade-off between query speed (QPS) and search quality (Recall@10/100). Recall@10 measures the percentage of true nearest neighbors found within the top 10 results. In general, higher QPS can be achieved at the cost of lower recall, and vice versa. A better system may achieve higher recall at the same QPS, or higher QPS at the same recall level. Our goal is typically to maximize QPS while maintaining a high recall target (e.g., 95%).

Disk

As always, VectorChord consistently achieves higher QPS at high recall levels (e.g., >95% Recall@10) compared to the other extensions, followed by pgvector and pgvectorscale with similar QPS.

Index Building

VectorChord maximizes CPU usage, reduces memory consumption, and shortens index-building time compared to pgvectorscale.

Insertion Time

As you can see, VectorChord (1565 Insert/Sec) performs much better than pgvector (246 Insert/Sec) and pgvectorscale (107 Insert/Sec) when inserting data. For most workloads, 100 Insert/Sec is more than enough. However, if you find that a lot of vectors are continuously inserted in your production (for example, 500 Insert/Sec), this could be a problem.