Content-defined identities
Versioned hashes separate canonical chunk content from source offsets and embedding-model identity.
Open source. Apache-2.0. Python 3.10+.
Incremental indexing for RAG corpora that change.
Steadlith combines content-defined chunk identities, cache-aware planning, and transactional updates so every revision becomes an explicit, inspectable index operation.
$
python -m pip install steadlithInstall the current release from PyPI.
$ steadlith plan
INDEX PLAN
add2
keep187
move1
delete1
Embeddings2 needed
Tokens421
Provider priceunknown
$ steadlith index --allow-delete
Applied 190 active, embedded 2, tombstoned 1
$ steadlith verify
Active index and manifest agree.
Rabin CDCDry-run planningTransactional SQLiteOffline by default
WHY STEADLITH
A small edit should be visible as a small change plan.
Offset-based chunking can shift downstream boundaries after an early insertion, changing hashes for content that is otherwise unchanged. Steadlith places candidate boundaries from a rolling fingerprint over normalized words, then compares versioned manifests to classify what changed.
A focused indexing layer, not another RAG framework. Steadlith sits below orchestration libraries and above embedding and vector providers.
Cache-aware planning
Preview add, keep, move, and delete operations without writing state or calling a provider.
Transactional indexing
Apply one validated SQLite snapshot. Removed occurrences become inactive tombstones immediately.
Verifiable state
Manifests, Merkle roots, generation checks, and record digests make committed state explicit.
ARCHITECTURE
A pure core. Effects at the edge.
Chunking and content identity stay deterministic. Files, credentials, providers, and databases enter only through explicit application boundaries.
InputSource text
Pure coreChunk + identify
State diffManifest plan
EffectsCache + index
Chunk
Normalize words and select rolling Rabin fingerprint boundaries.
Identify
Hash canonical content with versioned normalization and chunking parameters.
Plan
Compare manifests and price only known embedding cache misses.
Apply
Reuse vectors, embed misses, tombstone removals, and commit atomically.
CLI WORKFLOW
Start offline. Inspect every transition.
The starter config uses deterministic local lexical embeddings, so indexing and keyword retrieval work without credentials or network access.
Select a learned provider when queries require semantic similarity or synonyms.
quick-start.shoffline
# create a local configuration
steadlith init
# inspect before any write
steadlith plan
steadlith index
steadlith status
steadlith query "release policy"
steadlith verifyPositional paths describe the complete desired corpus, not additions. Prefer committed source globs and inspect the plan before applying changes.
OPERATIONAL SAFETY
Destructive and networked work stays explicit.
plan is the safe entry point: it makes no writes and sends no content to an embedding provider.
Network access
Optional providers require --allow-network.
Deletion
Deleting plans require --allow-delete; emptying an index also requires --allow-empty.
Source scope
Paths and resolved symlinks must stay inside the configuration directory.
Imported state
Unsigned cache imports require --trust-source; compaction supports dry run.
CURRENT SUPPORT
A deliberately narrow supported core.
The reference path is implemented end to end. Additional backends are not presented as supported until they pass the shared adapter conformance suite.
- Index
- Transactional SQLite, one logical index per database
- Embeddings
- Offline lexical, optional OpenAI and sentence-transformers
- Chunking
- Rabin CDC, opt-in snapping, comparison strategies
- State
- Cache, manifests, Merkle roots, tombstones, verification
- Measurement
- Published five-corpus churn and retrieval regressions
- Output
- Human-readable terminal tables and machine-readable JSON
HONEST LIMITATIONS
Limits stay explicit.
TTTD v1 is regression-tested for churn, but it does not claim a universal fixed-distance locality proof.
The stateful fallback can remain out of phase until a common primary boundary. Steadlith keeps the counterexample as a regression and reports measured churn instead of turning an empirical result into a theorem.
Across the bundled five-corpus benchmark, default CDC re-embeds 32.7% of revised chunks versus 53.3% for fixed chunking, with recall@5 of 1.0 under both offline scorers. Structural snapping remains opt-in and requires project-specific review.
APACHE-2.0 OPEN SOURCE
Evaluate Steadlith on your corpus.
Start offline, inspect the delta, then measure churn and retrieval quality before connecting a paid provider.