https://vchord.pgsty.com/tags/pg_tokenizer/ Recent content in pg_tokenizer on Vchord Hugo en-US Thu, 05 Nov 2026 00:32:10 +0800 https://vchord.pgsty.com/blog/release/pg-tokenizer-1.0.0/ Tue, 03 Nov 2026 00:00:00 +0000 https://vchord.pgsty.com/blog/release/pg-tokenizer-1.0.0/ <p><strong>pg_tokenizer 1.0.0</strong> is the first release maintained by PGSTY in <a href="https://github.com/pgsty/pg_tokenizer"><code>pgsty/pg_tokenizer</code></a> under the Apache-2.0 license. It connects the multilingual analyzers and vocabulary models to PostgreSQL&rsquo;s native full-text search, makes corpus-trained custom models safe under concurrent writes, and lets non-superuser database owners manage their own tokenizer definitions.</p> <p>The existing <code>tokenize(text, tokenizer_name)</code> interface still returns <code>INT[]</code>. Applications using the historical <code>vchord_bm25</code> 0.3.x <code>bm25vector</code> workflow can retain that interface; applications using GIN/GiST or the <a href="https://vchord.pgsty.com/blog/release/vchord-bm25-1.0.0/"><code>vchord_bm25</code> 1.0.0</a> API can use the new <code>tsvector</code> functions. <code>pg_tokenizer</code> does not require either BM25 or vector-search extensions.</p>