Index Storage

mdvdb keeps project configuration and generated local state in .markdownvdb/. The primary binary index stores vectors and metadata, Tantivy owns the lexical index, and disposable sidecars hold Shard-local analysis. The primary index is designed for fast memory-mapped loading and safe replacement through atomic writes.

Directory Structure

.markdownvdb/
  config.yaml          # Project settings (YAML; keep this)
  .env                 # Optional project secrets (keep this private)
  index                # Binary index file (vectors + metadata)
  fts/                 # Full-text search directory (Tantivy segments)
    meta.json          # Tantivy meta file
    *.managed.json     # Segment metadata
    *.store            # Stored fields
    *.idx              # Inverted index segments
    *.pos              # Position data
    *.term             # Term dictionary
    *.fast             # Fast fields (columnar data)
  cache/
    shards/*.json      # Disposable Shard-local communities and Topics
  modules.lock         # Cross-process computed-module coordination

index (Binary Index File)

The main index file contains document and semantic-edge vectors plus indexed document metadata. It uses a custom format with three regions: a fixed header, an rkyv-serialized metadata region, and a usearch HNSW graph. The lexical index and Shard sidecars remain separate.

fts/ (Full-Text Search Directory)

The FTS directory contains a Tantivy full-text search index. Tantivy uses a segment-based architecture similar to Lucene, with BM25 scoring for lexical search. This directory is managed entirely by Tantivy and is rebuilt during ingestion.

config.yaml and .env

config.yaml stores non-secret project settings. .env stores optional project credentials. Computed-field declarations live separately in .markdownvdb.schema.yml at the collection root. See Configuration and Computed Fields.

Binary Index Format

The index file uses a custom binary format optimized for memory-mapped loading:

block-beta
    columns 1
    block:HEADER["Header (64 bytes)"]
        columns 6
        A["Magic<br/>6B"] B["Version<br/>4B"] C["Meta Offset<br/>8B"] D["Meta Size<br/>8B"] E["HNSW Offset<br/>8B"] F["HNSW Size<br/>8B"]
    end
    block:EXT["Extension Fields (within header bytes 42-63)"]
        columns 4
        G["Quant Type<br/>1B"] H["Compression<br/>1B"] I["Uncomp. Size<br/>4B"] J["Reserved<br/>16B"]
    end
    block:META["rkyv Metadata Region (variable size)"]
        columns 1
        K["Files, chunks, embeddings config, schema,<br/>cluster state, link graph, file mtimes"]
    end
    block:HNSW["usearch HNSW Region (variable size)"]
        columns 1
        L["HNSW graph for approximate<br/>nearest-neighbor vector search"]
    end

Header Layout (64 Bytes)

rkyv Metadata Region

The metadata region contains document and chunk data serialized with rkyv. mdvdb validates and deserializes that region when opening the index.

The metadata includes:

When MDVDB_INDEX_COMPRESSION is true (the default), the metadata region is compressed with zstd at level 3 before writing. This typically reduces the metadata size by 60-80%. The uncompressed size is stored in the header so the decompressor knows how much memory to allocate.

usearch HNSW Region

The HNSW (Hierarchical Navigable Small World) region contains the vector index for approximate nearest-neighbor search. It is serialized and loaded by usearch.

Key HNSW parameters (set at index creation):

Vector Quantization

Vectors can be stored in two precisions, controlled by MDVDB_VECTOR_QUANTIZATION:

F16 quantization halves memory usage with negligible impact on search quality for most use cases.

Memory Mapping

mdvdb opens the binary index through memory mapping (memmap2) so its regions can be validated and handed to the metadata and HNSW decoders without a separate whole-file read.

Benefits

• Efficient file access — the OS can page the mapped index regions as they are read.
• OS-managed caching — the operating system manages the page cache, automatically evicting pages under memory pressure.
• Shared file cache — multiple processes can benefit from the operating system’s cached pages.

How It Works

flowchart LR
    PROCESS["mdvdb Process"] -->|"mmap()"| VM["Virtual Memory<br/>(address space)"]
    VM -->|"page fault"| CACHE["OS Page Cache"]
    CACHE -->|"read from disk"| DISK["index file<br/>on disk"]
    CACHE -->|"cached pages"| VM

    style PROCESS fill:#e3f2fd,color:#111827
    style DISK fill:#fff9c4,color:#111827

1. memmap2::Mmap::map() creates a memory-mapped view of the index file.
2. The header is read to locate the metadata and HNSW regions.
3. The metadata and HNSW regions are validated and decoded by their respective loaders.
4. Subsequent accesses to the same pages hit the cache without disk I/O.

Atomic Writes

Index writes are atomic to prevent corruption from crashes or interrupts:

1. Write to .tmp — all data is written to index.tmp (the index path with a .tmp extension).
2. fsync — file.sync_all() ensures all data is flushed to the storage device.
3. Rename — fs::rename("index.tmp", "index") atomically replaces the old index with the new one.

This pattern ensures that the index file is always either the complete old version or the complete new version — never a partially written state. If the process crashes during step 1 or 2, the .tmp file is left behind (and overwritten on the next ingest), but the existing index file remains intact.

Why This Matters

• Crash safety — a power failure or process kill during ingestion does not corrupt the index.
• Read safety — other processes reading the index via memory mapping see a consistent snapshot.
• Coordinated mutations — atomic replacement prevents partial generations; mdvdb also uses project lock files to coordinate cross-process index and computed-module writes.

Concurrency

At runtime, the index is protected by a parking_lot::RwLock:

parking_lot::RwLock is preferred over std::sync::RwLock for its superior performance in read-heavy workloads (no syscalls for uncontended read locks).

Configuration

Index Lifecycle

Creation

An index is created on the first mdvdb ingest run. The .markdownvdb/ directory is created automatically if it does not exist. Running mdvdb init creates .markdownvdb/config.yaml but does not create the index itself.

Updates

Incremental ingestion (mdvdb ingest) updates the index:

1. Discovers all markdown files (applying ignore rules).
2. Compares SHA-256 content hashes against the existing index.
3. Re-embeds only changed files.
4. Writes the updated index atomically.

Full Rebuild

mdvdb ingest --reindex forces a complete rebuild:

1. Discards all existing embeddings.
2. Re-chunks and re-embeds every file.
3. Writes a fresh index.

This is necessary when changing embedding providers, models, dimensions, or chunk settings.

Safe rebuild

Rebuild generated vector, metadata, lexical, community, and Topic state without deleting project configuration or secrets:

mdvdb ingest --reindex

Diagnostics

Use mdvdb doctor to check the health of your index:

mdvdb doctor

Doctor returns nine named checks:

Relations and Shards report repairable content/configuration problems as warnings. Some corrupt or incompatible storage failures can prevent the project from opening before a DoctorResult is produced; they are not separate doctor checks.

Use mdvdb status to see index statistics:

mdvdb status

This shows document, chunk, total-vector, and semantic-edge counts; file size and last-update time; the stored embedding configuration; and whether the runtime embedding space is compatible. JSON also exposes reindex_required and an actionable compatibility reason when present.

See Also

• mdvdb ingest — Build or update the index
• mdvdb status — View index statistics
• mdvdb doctor — Diagnose index issues
• Chunking — How files are chunked before indexing
• Embedding Providers — How chunks are embedded
• Search Modes — How the index is queried
• Configuration — All environment variables