Star 历史趋势
数据来源: GitHub API · 生成自 Stargazers.cn
README.md

VexDB

English | 中文

VexDB-Lite extends PostgreSQL, DuckDB, and SQLite with a shared graph index, SIMD distance dispatch, and PQ/RaBitQ quantization kernels. The SQLite edition also includes VexFS, so one database can hold regular tables, vector indexes, and database-managed files.

See vexdb_duckdb/README.md for the DuckDB extension docs.
This root README is a project-level overview and build guide.


1. Components

1.1 PostgreSQL: vexdb_lite

Current functionality:

  • floatvector(N) type
  • Distance functions and operators:
    • l2_distance (<->)
    • cosine_distance (<=>)
    • inner_product (use <~> for negative inner product / MIPS)
  • Scalar helpers: vector_dims(), vector_norm(), l2_normalize(), vexdb_index_info()
  • CREATE INDEX ... USING vexdb_graph
  • Index options: m, ef_construction, parallel_workers, quantizer (pq / rabitq), pq_m, and memory_mode
  • PQ and RaBitQ code storage; memory_mode='compact' requires quantizer activation rather than silently storing raw index vectors
  • Optimizer rewrite into an ANN Index Scan
  • Shared-memory vector buffer cache and parallel index build
  • Runtime settings: vexdb.ef_search, vexdb.vec_architecture

1.2 DuckDB: vexdb_lite

Current functionality:

  • GRAPH_INDEX on FLOAT[N] vector columns
  • Distance functions and operators:
    • l2_distance (<->)
    • cosine_distance (<=>)
    • inner_product (use <~> for negative inner product / MIPS)

    Note: <#> is unavailable in DuckDB (# clashes with its comment syntax); use <~> for negative inner product — same meaning as in PG.

  • Scalar helpers: vector_dims(), l2_normalize(), vexdb_version(), vexdb_index_info()
  • CREATE INDEX ... USING GRAPH_INDEX (vec) with WHERE filtering during queries
  • Index options: m, ef_construction, parallel_workers, quantizer (pq / rabitq), and pq_m
  • PQ and RaBitQ support memory_mode='compact' for index-side storage
  • Optimizer rewrite into VEXDB_INDEX_SCAN
  • Vector buffer cache and parallel index build
  • Runtime settings: vexdb_ef_search, vexdb_brute_force_threshold, vexdb_pq_search_mode, vexdb_pq_refine_k_factor

1.3 SQLite: unified VexDB-Lite

The unified macOS package installs:

  • vexdb, with SQLite 3.45.3 and automatic extension registration;
  • GRAPH_INDEX and vector distance functions;
  • VexFS files, directories, history, restore, SHA-256, and read-only integrity checks;
  • vexdb fs ..., including database-batched grep and an optional FTS5 trigram index, with vexfs ... kept as a compatible shortcut;
  • a VexFS FSKit extension for native ls, cat, grep, cp, and mv after enablement.
vexdb agent.db
vexdb agent.db "SELECT vexdb_l2_distance('[1,2]', '[4,6]');"
vexdb fs --db agent.db write /notes/hello.txt
vexdb fs --db agent.db grep -n hello /notes
vexdb fs --db agent.db check
vexdb fs --db agent.db index enable
vexdb fs --db agent.db mount ~/VexDB

The text index is disabled by default, so ordinary writes have no index overhead. Without the index, or when the SQLite host has no FTS5 support, grep falls back to a single-connection database scan.

Build the unified macOS preview with bash agent_files/macos/package_preview.sh.


2. Capability Matrix

2.1 PG Extension Comparison (PGVector vs VexDB-Lite vs VexDB)

CategoryFeatureDescriptionPGVectorVexDB-Lite (open-source)VexDB (commercial)
Graph Indexgraph_indexA fully self-developed high-performance graph index that merges the advantages of various graph indexes and works seamlessly across all scenarios.❌✅✅
DistanceDistance function dispatchInlined distance functions, compile-time optimized❌✅✅
Cachevector bufferGeneral vector cache, all scenarios❌✅✅
Cachebulk bufferFull in-memory vector cache for max throughput❌❌✅
CacheAsync I/O cacheAccelerated disk-to-cache reads under memory pressure❌❌✅
Data typesfloatvectorStandard float32 vector type✅✅✅
Data typeshalfvectorFloat16 vector type✅🟡✅
Data typesint8vectorInt8 vector type❌🟡✅
QuantizationPQ quantizationCode compression and optional refinement; speed depends on data, filters, and configuration❌✅✅
QuantizationRaBitQ quantizationQuantized distance estimation and graph search with compact index-side storage❌✅✅
QuantizationAuto quantizationBackground auto-enable, supports empty-table index build❌❌✅
Graph index enhancementAsync insertFast ingestion for write-heavy workloads❌❌✅
Graph index enhancementGraph shardingLarge-scale vectors on small-memory machines❌❌✅
Graph index enhancementSubgraph index buildContinue using memory for index building even in low-memory scenarios to accelerate build speed❌❌✅
HAPrimary-replica HASynchronous replication and backup restore✅❌✅
MaintenanceParallel vacuumParallel index cleanup and reclaim❌❌✅

2.2 DuckDB Extension Comparison (DuckDB VSS vs VexDB-Lite)

CategoryFeatureDescriptionDuckDB VSSVexDB-Lite (vexdb_lite)
IndexGraph indexVSS: HNSW; VexDB: graph_index (self-developed hybrid)✅✅
DistanceSIMD dispatchInlined distance functions, compile-time optimized❌✅
QuantizationPQVector compression for memory-constrained scenarios❌✅
QuantizationRaBitQQuantized graph search with optional compact index-side storage❌✅
CacheBuffer managementDisk-to-memory vector caching❌✅
MaintenanceIndex compactionReclaim space from soft-deleted entries✅❌
SearchFiltered ANN searchOpt-in scalar/graph/PQ integration; exact filtering remains the default❌🟡
PersistenceDisk-backed indexIndex survives database restart without rebuild✅†✅

† VSS persistence is experimental — WAL recovery is not implemented, unexpected shutdowns may cause index corruption. VexDB-Lite persists via DuckDB's standard serialization.


✅ Supported · 🟡 Coming soon · ❌ Not included in open-source edition

3. PostgreSQL Syntax Examples

3.1 Install and Create Table

CREATE EXTENSION vexdb_lite;

CREATE TABLE items (
    id  BIGSERIAL PRIMARY KEY,
    vec floatvector(128)
);

INSERT INTO items (vec) VALUES
    ('[0.10, 0.20, 0.30]'),
    ('[0.40, 0.50, 0.60]');

3.2 Build Index

CREATE INDEX idx_items_vec
ON items
USING vexdb_graph (vec floatvector_l2_ops)
WITH (
    m = 16,
    ef_construction = 64
);

3.2.1 PQ Quantized Index (v1)

PQ reduces vector storage by encoding indexed vectors with a trained codebook:

SET maintenance_work_mem = '2GB';   -- required; low memory falls back to a plain graph index

CREATE INDEX idx_pq
ON items
USING vexdb_graph (vec floatvector_l2_ops)
WITH (quantizer = 'pq', pq_m = 4);

Current v1 behavior:

  • If there are fewer than 256 training samples, or the memory build budget is too low, PQ falls back to a plain graph index with a NOTICE.
  • PQ indexes support post-build INSERT / UPDATE / DELETE; new vectors are encoded with the trained codebook.
  • For parallel_workers > 0, PQ training still runs in the leader process; PG parallel workers only participate in the disk-build phase.

Check the active index state:

SELECT indexname, use_pq, pq_m
FROM vexdb_index_info()
WHERE indexname = 'idx_pq';

3.3 ANN Query

SET vexdb.ef_search = 100;
SET enable_seqscan = off;

SELECT id, vec <-> '[0.15, 0.25, 0.35]' AS dist
FROM items
ORDER BY vec <-> '[0.15, 0.25, 0.35]'
LIMIT 10;

3.4 Other Metrics

SELECT id
FROM items
ORDER BY vec <~> '[0.15, 0.25, 0.35]'
LIMIT 10;

SELECT id
FROM items
ORDER BY vec <=> '[0.15, 0.25, 0.35]'
LIMIT 10;

4. DuckDB Syntax Examples

4.1 Load Extension

LOAD '/path/to/vexdb_lite.duckdb_extension';
SELECT vexdb_version();

Typical Python usage:

import duckdb

con = duckdb.connect(config={"allow_unsigned_extensions": "true"})
con.execute("LOAD '/path/to/vexdb_lite.duckdb_extension'")

4.2 Create Table and Index

CREATE TABLE items (
    id       INTEGER,
    category VARCHAR,
    vec      FLOAT[128]
);

CREATE INDEX idx_items_vec
ON items
USING GRAPH_INDEX (vec)
WITH (
    metric = 'l2',
    m = 16,
    ef_construction = 64
);

4.3 ANN Query

SET vexdb_ef_search = 100;

SELECT id
FROM items
ORDER BY l2_distance(vec, [0.15, 0.25, 0.35]::FLOAT[3])
LIMIT 10;

4.4 Filtered Index Example

CREATE INDEX idx_items_vec_meta
ON items
USING GRAPH_INDEX (vec);

SELECT id
FROM items
WHERE category = 'book'
ORDER BY l2_distance(vec, [0.15, 0.25, 0.35]::FLOAT[3])
LIMIT 10;

4.5 Other Functions

SELECT inner_product([1.0, 0.0]::FLOAT[2], [0.5, 0.5]::FLOAT[2]);
SELECT cosine_distance([1.0, 0.0]::FLOAT[2], [0.5, 0.5]::FLOAT[2]);
SELECT vector_dims([1.0, 2.0, 3.0]::FLOAT[3]);
SELECT l2_normalize([3.0, 4.0]::FLOAT[2]);
SELECT * FROM vexdb_index_info();

5. Build

5.1 Build the PostgreSQL Variant

Dependencies

  • PostgreSQL 16 ~ 19 (PG 16/17/18/19 supported; primary validation target is 19devel)
  • CMake ≥ 3.14
  • C++17 compiler (GCC 9+ or Clang 10+)

Build PostgreSQL (release example)

cd /path/to/postgresql-19-source
./configure \
  --prefix=/opt/postgresql-19rel-install \
  --without-icu \
  --without-readline \
  --without-zlib \
  CFLAGS="-O3 -DNDEBUG"
make -j$(nproc)
make install

Build vexdb_lite

cd /path/to/VexDB-Lite
export PG_CONFIG=/opt/postgresql-19rel-install/bin/pg_config
cmake -S vexdb_pg -B build/pg -DCMAKE_BUILD_TYPE=Release
cmake --build build/pg -j$(nproc)
cmake --install build/pg

PostgreSQL Configuration

At minimum:

shared_preload_libraries = 'vexdb_lite'

Then restart PostgreSQL and run:

CREATE EXTENSION vexdb_lite;

5.2 Build the DuckDB Variant

Recommended: use build_duck.sh — it handles DuckDB clone, cmake configuration, compilation, and metadata processing in one command.

bash build_duck.sh setup   # First time: clone DuckDB v1.5.2 and cmake configure
bash build_duck.sh build   # Compile the extension (incremental)

Output: build/duck/v1.5.2/build/extension/vexdb_lite/vexdb_lite.duckdb_extension

Dependencies

  • CMake ≥ 3.28, < 4.x
  • C++17 compiler (GCC 9+ or Clang 10+)
  • Git

Why a shell script and not plain cmake?

DuckDB extensions must be compiled inside DuckDB's source tree — you cannot run cmake -B build vexdb_duckdb/ standalone. build_duck.sh automates:

  1. Cloning DuckDB v1.5.2
  2. Writing extension_config_local.cmake to register the vexdb_lite extension
  3. Running cmake + cmake --build
  4. Appending the extension metadata footer (required by DuckDB's release format)

6. Running Tests

DuckDB Extension Tests

bash build_duck.sh build          # Build the extension
bash tests/spec/_lib/docker/run_duckdb.sh test  # Run full spec tests locally; Docker is not needed

PostgreSQL Plugin Tests

python3 tests/spec/_lib/render.py --engine pg --out build/spec
BUILDX_EXTRA_ARGS='--build-arg PG_GIT_TAG=REL_19_STABLE' bash tests/spec/_lib/docker/run_pg.sh build
bash tests/spec/_lib/docker/run_pg.sh test      # Requires Docker

Install renderer dependencies with python3 -m pip install -r tests/spec/requirements.txt. SQLite build and tests use bash build_sqlite.sh test. Test files live under tests/spec/. See current status and CI.


7. Benchmark Results

Dataset: SIFT-1M 128-dim, m=16, ef_construction=128. Columns: QPS (reads=1) / QPS (reads=16) / Recall@10.

Test environment: Intel Core Ultra 7-265K (20c/20t, 3.9 GHz) / 16 GB DDR5 / x86_64 Linux

7.1 Comparison with pgvector / VSS (x86_64)

ef_search = 50

SystemQPS (r=1)QPS (r=16)Recall@10
pgvector507.97153.596.22%
vexdb_lite (PostgreSQL)994.712084.695.97%
vexdb_lite (DuckDB)717.58667.895.06%
duckdb-vss496.15360.994.07%

ef_search = 100

SystemQPS (r=1)QPS (r=16)Recall@10
pgvector313.44272.598.82%
vexdb_lite (PostgreSQL)618.57883.198.62%
vexdb_lite (DuckDB)547.25379.198.40%
duckdb-vss405.24433.398.04%

ef_search = 200

SystemQPS (r=1)QPS (r=16)Recall@10
pgvector193.12694.199.66%
vexdb_lite (PostgreSQL)421.35038.099.58%
vexdb_lite (DuckDB)383.64298.899.53%
duckdb-vss321.93809.399.42%

8. Known Limitations

PostgreSQL

  • Supports PostgreSQL 16 ~ 19; primary validation target is PostgreSQL 19

DuckDB

  • The compiled extension must match the exact DuckDB runtime version.
  • CPU distance dispatch is selected at runtime; inspect vexdb_simd_arch() for the active architecture.

7. Repository Structure

DirectoryDescription
common/Shared core: graph index algorithm, SIMD distance dispatch, quantizer (PQ/RaBitQ), template containers
vexdb_pg/PostgreSQL extension: index AM, build, search, DML, WAL, distance entry
vexdb_duckdb/DuckDB extension: index lifecycle, optimizer rewrite, distance functions → README
vexdb_sqlite/SQLite GRAPH_INDEX, distance functions and VexFS SQL
agent_files/File commands, versions, snapshots and mount adapters
documentation/Feature docs, build guide and current CI scope
tests/spec/YAML-based spec tests (shared / pg / duckdb / sqlite / opengauss)
scripts/Build, release, and packaging scripts
thirdparties/Vendored dependencies (patched Boost)

Community

ChannelDescription
GitHub IssuesBug reports and feature requests
GitHub DiscussionsQuestions, proposals and general discussion
DiscordReal-time chat and Q&A
WeChat GroupScan QR code at vexdb.com/community · Chinese community

License

MIT License. See LICENSE.

关于 About

A cross-platform vector database, which can be integrated into existing databases as a plugin.

语言 Languages

C++61.2%
Python12.4%
PLpgSQL12.0%
Shell6.7%
C4.7%
Rust1.1%
Swift1.0%
CMake0.6%
Ruby0.1%
templ0.1%
Dockerfile0.1%
Objective-C++0.0%
Makefile0.0%

提交活跃度 Commit Activity

代码提交热力图
过去 52 周的开发活跃度
133
Total Commits
峰值: 36次/周
Less
More

核心贡献者 Contributors