Usage on Vchord

Indexing

Mon, 01 Jan 0001 00:00:00 +0000

Indexing is the process of building a data structure that allows for efficient search. pgvecto.rs supports three indexing algorithms: brute force (flat), IVF, and HNSW. The default algorithm is HNSW.

To construct an index for vectors, first create a table named items with a column named embedding of type vector(n). Then, populate the table with generated data.

CREATE TABLE items (embedding vector(3));
INSERT INTO items (embedding) SELECT ARRAY[random(), random(), random()]::real[] FROM generate_series(1, 1000);

Tip

vector is the suitable data type for most situations. pgvecto.rs also supports other specific vector types, including:

Half-Precision Vector
8-Bit Integer Vector since v0.3.0
Sparse Vector since v0.3.0
Binary Vector since v0.3.0

Search

Mon, 01 Jan 0001 00:00:00 +0000

Get the nearest 5 neighbors to a vector

SET vectors.hnsw_ef_search = 64;
SELECT * FROM items ORDER BY embedding <-> '[3,2,1]' LIMIT 5;

Operators

These operators are used for distance metrics:

Name	Description
<->	squared Euclidean distance
<#>	negative dot product
<=>	cosine distance

For their definitions, see overview.

Filter

For a given category, get the nearest 10 neighbors to a vector

Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

We provide a view pg_vector_index_stat to monitor the progress of indexing.

Column	Type	Description
tablerelid	oid	The oid of the table.
indexrelid	oid	The oid of the index.
tablename	name	The name of the table.
indexname	name	The name of the index.
idx_status	text	Its value is `NORMAL` or `UPGRADE`. Whether this index is normal or needs upgrade.
idx_indexing	bool	Not null if `idx_status` is `NORMAL`. Whether the background thread is indexing.
idx_tuples	int8	Not null if `idx_status` is `NORMAL`. The number of tuples.
idx_sealed	int8[]	Not null if `idx_status` is `NORMAL`. The number of tuples in each sealed segment.
idx_growing	int8[]	Not null if `idx_status` is `NORMAL`. The number of tuples in each growing segment.
idx_write	int8	Not null if `idx_status` is `NORMAL`. The number of tuples in write buffer.
idx_size	int8	Not null if `idx_status` is `NORMAL`. The byte size for all the segments.
idx_options	text	Not null if `idx_status` is `NORMAL`. The configuration of the index.

idx_options Details

The idx_options is a JSON string that includes vector, segment, optimization, and indexing options. Examples below:

{
    "vector": {
        "dimensions": 256,
        "distance": "L2",
        "kind": "F32"
    },
    "segment": {
        "max_growing_segment_size": 20000,
        "max_sealed_segment_size": 1000000
    },
    "optimizing": {
        "sealing_secs": 60,
        "sealing_size": 1,
        "delete_threshold": 0.2,
        "optimizing_threads": 2
    },
    "indexing": {
        "hnsw": {
            "m": 12,
            "ef_construction": 300,
            "quantization": {
                "trivial": {}
            }
        }
    }
}

Quantization

Mon, 01 Jan 0001 00:00:00 +0000

Quantization is a technique to reduce the memory consumption and computation complexity with the affordable decrease in precision. In the context of vector search, there are mainly two types of quantization: production quantization and scalar quantization.

Production Quantization

Production quantization is a method that trains several clustering models on the dataset and maps each vector to a list of cluster center indexes. It is commonly used address the problem of excessive memory usage with high-dimensional data.

`pgvector` compatibility

Mon, 01 Jan 0001 00:00:00 +0000

pgvecto.rs is natively compatible with pgvector at:

CREATE TABLE commands, e.g. CREATE TABLE t (val vector(3))
INSERT INTO commands, e.g. INSERT INTO t (val) VALUES ('[0.6,0.6,0.6]')

pgvecto.rs can be configured to be compatible with pgvector at:

Index options, which allows you to create index by USING hnsw (val vector_ip_ops)
Query options, which can be set by SET ivfflat.probes = 10

This feature is named pgvector compatibility. It can be enabled by SET vectors.pgvector_compatibility = on.