lance-format

Lance v13 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x faster than Parquet for random access." It is not one format but a stack of interoperating specs: a file format, a table format, index formats, catalog specs, and a namespace client spec. The Rust workspace at lance-format/lance implements all of them plus Python (pylance) and Java bindings.

This skill tracks v13.0.0-beta.4 (the lance-format/lance git tag), the current development frontier; v12.0.0 is the stable pin, released 2026-09-17. Pin against tags, not main - Lance ships beta tags every few days and next-format encodings can change. Version landscape below.

Three layers of reference, load what the task needs:

• The deep reference - any concrete schema, parameter, proto, or constraint. Split by topic:

  File in references/	Covers	Sections
  format-file.md	What Lance is, the 26 crates, file format, data types	1-4
  format-table.md	Dataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL	5-10
  indexes.md	Vector / scalar / FTS / geo indexes, distributed builds	11-12
  ops.md	Object store, capability matrix, source map	13, 15, 16
  changelog-v7-v13.md	The full v7 -> v13 delta	14

  Cross-references written as "section N" resolve through references/lance-reference.md.

• references/performance.md - ALL performance guidance. Part A routes to the official text and adds the source-derived changes upstream has not documented; Part B is field-verified remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this slow" question.

• references/docs/ - a verbatim mirror of the official docs (docs/src at the tracked tag): every guide, quickstart, and format spec, unedited. Load when you need the full official text. Directory map below.

references/maintenance.md covers refreshing this skill against a new upstream tag.

Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

• Lance - the format and engine. The lance-format/lance repo; the lance /lance-* Rust crates; pylance. It gives you datasets, the file/table format, indexes, commits, scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own Rust/Python code. This skill is about Lance.
• LanceDB - a separate database product (lancedb/lancedb) built on top of Lance. It adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK parity, and managed Cloud / Enterprise tiers. Not covered here.

The wider ecosystem (separate repos, own version lines, none covered here): Flink streaming writes (lance-flink), PostgreSQL reads via pglance, a Cypher graph engine (lance-graph), a dataset browser (lance-data-viewer), agentic context management (lance-context), and namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, and AWS Glue. The canonical docs site is lance.org. Generated per-language SDK docs live at lance-format.github.io/lance-python-doc for Python and javadoc.io for Java - the matching lance-format.github.io/lance-java-doc path 404s.

Linking the lance crate in Cargo.toml means you are using Lance directly - use this skill. For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the authority for the format itself.

The wrapper can hide format features. LanceDB's create_table cannot enable stable row IDs; only pylance's write_dataset(enable_stable_row_ids=True) can. If a format-level capability matters to your design, check whether the wrapper exposes it before assuming the underlying format settles the question - and reach for pylance directly when it does not.

The crate workspace

26 crate directories under rust/. lance is the public entry point - Dataset, scanner, indexes, commits; everything else (lance-table, lance-file, lance-encoding, lance-index, lance-io, lance-core, lance-datafusion, lance-linalg, lance-namespace*, ...) is a layer beneath it. Edition 2024, MSRV 1.91.0, arrow 58, datafusion 54; Python bindings need 3.10+. Full table with roles, versions, and every workspace dep in references/format-file.md section 2.

If you depend on anything below lance, v11 will break you - PRs #8020-#8026 deleted lance-encoding::version with no re-export (LanceFileVersion and ConcreteFileVersion both live in lance-file::version now), removed lance_io::encodings and the previous namespaces, and gave each current format its own versions/v2_{0,1,2,3} module. Section 2.1.

The transaction code moved too (#8053/#8054/#8056): rust/lance/src/dataset/transaction.rs is deleted, replaced by a rust/lance-table/src/transaction/ module tree (builder, conflicts, operation, proto, manifest_build, validate, index_maintenance, row_version, update_map). A lance::dataset::transaction shim still re-exports Operation, Transaction, TransactionBuilder, RewriteGroup, UpdateMap and friends, so the common surface is unbroken - but a symbol the shim omits, or a citation of the old path, needs retargeting.

File format versions

The file format carries a single major.minor version. data_storage_version is set per dataset at creation - but as of v12.0.0 it is no longer fixed once the dataset exists. It is the default for writes that omit a target, not a summary of what the dataset holds: "Create and overwrite establish this default; append, update, merge-insert, and compaction do not change it." An existing V2 dataset can take "2.0", "2.1", "2.2" or "2.3" per operation without rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2 versions. V1 and V2 still cannot be mixed. Section 3.

Version	Status	Notes
0.1 (legacy)	read-only	Original format; no longer writable
2.0	stable	Removed row groups; null support for lists/FSL/primitives
2.1	previous default	Adaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access. Was the default from Lance 5.0.0 until v12.0.0-beta.15
2.2	current default (stable)	Map type, Blob v2, VariablePackedStruct, larger mini-blocks. Required for Map and Blob v2
2.3	unstable (next)	The current next alias target (V2_3 in the enum). Ships sparse structural pages, which the 2.3 writer now auto-selects under a rep/def budget heuristic

stable now resolves to 2.2, not 2.1 (#8657, beta.15), and 2.2 is the enum #[default], so a dataset created without an explicit data_storage_version is written as 2.2. The change reaches new-dataset creation through DataStorageFormat::default() -> stable_file_version(), and Python's write_dataset inherits it because its default routes through stable. The docs were not updated with it - format/file/versioning.md still only says stable is an "alias for the default version", so the code is the authority here. next resolves to 2.3. Pin an explicit number for deterministic behavior across builds.

2.3 is the only version the code flags unstable; 2.2 never was, and is now what you get by default. The release selectors (LanceFileVersion) are a type distinct from the persisted identity (ConcreteFileVersion). Details, plus the sparse auto-selection rules, in references/format-file.md sections 3.1 and 3.6.

Version landscape

The major is bumped by a bot, not a human: ci/publish_beta.sh re-roots at MAJOR+1 whenever any PR since the release root carries the GitHub breaking-change label - the marker is the label, not a conventional-commit !. A major bump therefore means "some labeled breaking change landed", not a redesign, and a ! without the label bumps nothing. It has now fired on four consecutive lines, which is why none of v9.1.0, v10.1.0, v11.1.0, or v12.1.0 was ever released. The 12.1 line is the clearest case: main took a chore: bump main to 12.1.0-beta.0 commit, and four commits later the bot re-rooted to 13, so release-root/12.1.0-beta.N and release-root/13.0.0-beta.N are the same base commit (c3c9632a2) and no v12.1.0-beta.* tag exists.

Three recent lines did ship a final: v10.0.0 (2026-08-08), v11.0.0 (2026-08-30) and v12.0.0 (2026-09-17). Each sits on a stabilization branch that is not an ancestor of main

• normal for a Lance final, not a sign the release is unofficial.

Major	Its breaking theme
v13 (current, v13.0.0-beta.4)	WriteParams gained file_writer_options (#9192 - the one labeled PR that re-rooted the major); lazy page-metadata init changed the StructuralFieldScheduler signature and the metadata cache key shape (#7465); json_extract/json_get no longer route to JSON indices (#9101). Delta below
v12 (v12.0.0, 2026-09-17)	WrappingObjectStore implementors must add wrap_paginated (no default); MemWAL ShardManifestStore renamed and narrowed; lance-namespace returns response objects; external stores gained predecessor-conditioned publication; namespace merge-insert keys became a list; the caller-provided Writer / open_part flow was removed (#9072). Unlabeled but bigger: stable -> 2.2 and the IVF_RQ 5-bit default. Net-new format capability: mixed data-file versions. Delta below
v11 (v11.0.0, 2026-08-30)	Fragment ids became a dataset-lifetime high-water mark; large internal reorganization of lance-file / lance-encoding; the first new manifest feature flag since v7 - which was then reallocated before the final. Net-new: covering indexes, merge_insert write_mode, row-address prefilter. Delta below
v10	Blob APIs preserve null selections; cache keys became opaque BLAKE3 digests (every warm or persisted cache cold-misses, no legacy fallback); async create_remapper; MemWAL renamed generation -> SSTable, merge -> compaction (wire-compatible, symbol-breaking)
v9.1 (never released; renamed into v10)	FTS/inverted creation took a block_size param. Net-new: Data Overlay Files (cell-level updates without base-file rewrite, unstable + env-gated), sparse structural pages, lance-index-core
v9	Python 3.9 dropped; alter_columns fails fast when casting an indexed column; FM-Index proto rename made existing FM indexes unreadable; FTS/inverted defaults to on-disk format v2
v8	All index builds unified onto one segment-based lifecycle. Net-new: lance-derive, FM-Index, multi-bit IVF_RQ, public approx_mode, TOS + GooseFS object stores
v7	MemWAL, branches, the geo/RTree index, the lance-select crate, ICU FTS

v12.0.0 is the stable pin and what GitHub Releases marks Latest. crates.io carries finals only (newest lance 12.0.0, no 13.x); PyPI pylance is likewise at 12.0.0. So a beta pin means a git dependency - beta wheels publish to fury.io instead, under the renamed org (https://pypi.fury.io/lance-format), which currently carries pylance-13.0.0b1 through b4.

Full per-tag deltas with every PR citation: references/changelog-v7-v13.md.

The v11 delta

357 commits from v10.0.0-beta.7 to the v11.0.0 final, with 16 breaking-change-labeled PRs (14 through beta.16, plus #8407 and #8535 in the final). Most structural invariants held: 26 crates, 16 transaction ops, CommitConfig.num_retries 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024, Python 3.10+ - and all of them still hold at v13.0.0-beta.4.

references/changelog-v7-v13.md has the full delta - every PR citation, the per-tag breakdown from v7 forward, the Python/Java surface, and each correctness fix with its trigger condition. Load it for any "what changed / will this break me" question. What follows is only what bites hardest.

Five things that break you at v11:

• Fragment ids are a dataset-lifetime high-water mark (#8206) - a format invariant, not just an API. Overwrite no longer restarts ids at 0, an overwrite fragment carrying a deletion file is rejected, and any commit producing duplicate ids is rejected - so datasets written by Lance 0.16 and earlier may still read but no longer commit. dataset.get_fragment(0) after an overwrite must read ids from the manifest. Section 5 - which also covers a resolution hazard on pre-0.10 unsorted manifests that can make a fragment-filtered index cover the wrong fragments.
• The file-version types and reader/writer composition moved (#8020-#8026) - lance-encoding::version deleted with no re-export; LanceFileVersion lost PartialOrd/Ord (#8027, #8028), so v >= LanceFileVersion::Next no longer compiles. FileWriter is now an enum with all constructors removed. Most of these break silently at compile time. Section 3.6.
• Transaction code moved to lance-table (#8053/#8054/#8056) - see the crate-workspace note above; the lance::dataset::transaction shim covers the common surface.
• Operation::Project / Merge gained preserves_nullability (#8347) - a nullability tightening must not set it, and such a projection now conflicts with any concurrent value-write. This closed a real hole where alter_columns could let a racing write land nulls unreadable under the tightened schema. Section 9.2.
• The external-manifest protocol changed (#8499) - object storage is authoritative, the external store's put-if-not-exists is a reservation, and a stored ETag must be ignored; a retained one makes readers reject a good manifest with Manifest e_tag mismatch. Section 9.

The manifest feature flags changed - and bit 128 was reallocated before the final. v11 added the first new bit since v7 and moved FLAG_UNKNOWN 128 -> 256. But the bit it added, FLAG_MEM_WAL_INDEX_CATCHUP, was retired again (#8680) and the reclaimed bit handed to FLAG_COVERED_INDEX_METADATA = 128 (#8535) before v11.0.0 shipped. At the final and at v12 there is no index-catchup flag and no require_index_catchup proto field; a shard absent from index_catchup now unconditionally means unknown. Both reader and writer must hold bit 128 or refuse the table. Section 7.

Do not pin anywhere in v11.0.0-beta.4 through beta.17. Those builds treat bit 128 as a MemWAL flag they support, so they open a covering-index dataset instead of refusing it - wrong neighbours, no error. The exposure is inherited by whichever flag takes the bit.

Covering indexes are the v11 net-new format feature (#8535), redefined at v13 (#8856). IndexMetadata.covering_fields (proto field 11) names the columns an index carries values for, so a query projecting only those columns is answered without a base-table take. It is no longer a trailing suffix of fields: it "must be a subset of fields, in the order the index emits them. A column is carried if and only if it is named here", including a column the index is also keyed on - and fields[0] remains a keyed column. Index invalidation stays wide: any index whose fields include the updated column, "whether the index is keyed on it or merely carries it".

The old "no index builder writes carried values yet" no longer holds. V3 IVF auxiliary files can physically carry columns, and "a reader discovers carried columns by exclusion, not by position: any column in the auxiliary file's schema that is not one of the quantizer's internal columns is a carried column", bound to source fields by a new covering_field_ids metadata key. Coverage is now per-segment, not per-index: "one logical index may hold values for some of its segments and not others". VectorQueryProto.covering_projection (field 15) reserves the query-side tag, where absent / present-and-empty / present-and-non-empty are three distinct meanings. Section 11.

Bit 8 was spent in the v12.0.0 final. FLAG_MIXED_DATA_FILE_VERSIONS = 1 << 8 (256) is no longer a reservation pinned equal to FLAG_UNKNOWN: the assert relaxed to FLAG_MIXED_DATA_FILE_VERSIONS < FLAG_UNKNOWN, FLAG_UNKNOWN moved 1 << 8 -> 1 << 9 (512), and the build now both reads and writes mixed-version datasets. It is still carried by STICKY_PAIRED_FLAGS, and a half-set manifest is now a hard error: "Manifest has only one of the mixed data-file-version reader and writer feature bits set, so its semantics are undefined". Section 7.

Bit 1024 is where the docs and the code disagree - trust the code. FLAG_FRAGMENT_REUSE_INDEX = 1 << 10 is declared at rust/lance-table/src/feature_flags.rs:69 and, at v13.0.0-beta.4, that declaration is its only occurrence in the entire tree. It sits above FLAG_UNKNOWN (512), and the supported set is computed as FLAG_UNKNOWN - 1, so a manifest setting it is refused. The spec page meanwhile lists it as reader Yes / writer Yes and puts the unknown boundary at 2048. The docs describe the intended end state; the code has only reserved the constant. Anything you build against tagged FRI today is building against prose, not behavior.

Two LANCE_* env vars landed (from the AMX work, #8540): LANCE_DISABLE_AMX (runtime kill switch) and LANCE_AMX_FP16_CC (build-time compiler override). Grep trap: LANCE_AMX_CFG_* and LANCE_AMX_TILE_COUNT are C macros in amx_fp16.c, not env vars, and LANCE_FACTOR is a substring of BALANCE_FACTOR - a plain LANCE_* grep reports all four as if they were real.

Worth knowing without reading the full delta: FTS gained a document-boundary axis (DocumentGranularity, #7788) whose list_element mode is a third trigger requiring FTS on-disk format v3; transactions above 20 MiB spill out of the manifest entirely (#7881); MemWAL catch-up became derived rather than declared (#8481); transaction proto field 9 is deprecated for field 10 (#7432); compaction gained row/byte budgets plus fragment exclusion (#8235, #8532); merge_insert gained write_mode (#8423); and Python commit conflicts became lance.commit.CommitConflictError, a subclass of OSError, so existing handlers keep working (#8563). Full list with citations in references/changelog-v7-v13.md.

Address-domain indexes stopped falsely claiming compacted fragments (v11, beta.16 or earlier). A rewrite used to advance every index's fragment_bitmap onto the new fragment ids - including ZoneMap, whose stored addresses point into the fragments the rewrite dropped. The Rewrite path now branches on results_are_row_addrs(): an address-domain index gets drop_rewritten_fragments and a full-scan fallback, correct-but-slower instead of stale addresses. Heals only for new compactions: an index already damaged under v10 or earlier must be recreated, and the damage does not self-heal through routine maintenance, because the refreshed fragment_bitmap also makes incremental folds a no-op. Section 11.

Correctness fixes split by whether upgrading is enough. Most are read-path only and heal on upgrade. These do not - they need data rewritten or repaired: #8382, #8669, #8509, #7703, #8539, #8459, #8378, #8482, #8834 (rebuild HNSW - a persisted graph can hold edges to ids it does not contain; lost recall stays lost), #8101 (nullable primary keys silently duplicated rows on every repeat merge_insert; existing duplicates must be removed by hand), #8511, #8427, #8513, #8839, #8904. Conditions for each in references/changelog-v7-v13.md.

The v12 delta

225 commits from release-root/12.0.0-beta.N to the v12.0.0 final, with 7 breaking-change-labeled PRs - the five visible at beta.15 plus #9072 and #9101 in the run-up to the final. No new index types and no new crates; every structural invariant above still holds. The label is a floor, not a ceiling - the two biggest behavior changes in the line carry a conventional-commit ! but no label, so the bot never counted them: the stable -> 2.2 move (#8657, above) and the IVF_RQ 5-bit default (below).

• WrappingObjectStore implementors must add wrap_paginated (#8606) - "There is deliberately no default: getting this wrong is either a silent loss of speed or a silent loss of the wrapper, and neither announces itself." Return Some to keep listing pushdown through the wrapper, None to give it up and fall back through inner. One wrapper giving it up gives it up for the whole chain. Anything wrapping the object store fails to compile until updated.
• New paged listing: ObjectStore::read_dir_page (#8606) - one page of a prefix's immediate children plus an opaque resume token. The trap: "One page is one request, so a page can hold fewer children than limit asked for and still be followed by more" - walk until the token is None, never until a page comes back short.
• MemWAL ShardManifestStore renamed and narrowed (#8640) - read_latest -> latest, read_latest_uncached -> refresh_latest, and write is now crate-private (reach it through commit_update, claim_epoch, or initialize_shard). Existing commit_update closures need no change. Section 10.
• lance-namespace 0.8.5 -> 0.11.1 (#8903) - four LanceNamespace methods now return response objects instead of bare values: count_table_rows -> CountTableRowsResponse, query_table -> QueryTableResponse, namespace_exists / table_exists -> their own response types. Callers unwrap; anyone implementing the trait needs the same signature updates.
• External manifest stores gained predecessor-conditioned publication (#8800) - put_if_predecessor reserves a version only while the predecessor still carries the identity the writer observed, and commit_after refuses with PrerequisiteFailed, "never a conflict". The hard compile break is the new ManifestLocation.identity field, not the trait methods (all default-implemented). No built-in store implements it. Section 9.
• Namespace merge-insert keys became a list (#8915) - on moves from Option<String> to Option<Vec<String>>, with arity-dependent NULL semantics: a single-column key treats NULL as equal to NULL, a composite key uses SQL equality, "under which a NULL key matches nothing - not even a byte-identical NULL".

The lance-namespace pin is no longer one number. #8915 moved the Rust client to 0.12.0 and #8979 moved Java to 0.12.0 as well; only Python still holds >=0.11.1,<0.12, because its generated models still send on as a bare string. Quote a language-specific pin, never one number for all three - and note this split moved once already, so re-check it rather than carrying the pairing forward.

IVF_RQ now defaults to 5 bits per dimension, not 1 (#8936) - roughly a 4.4x index-size increase at the default (upstream's 100M x 768d example: ~10.8 GiB -> ~47.3 GiB). Fast search mode "uses only the 1-bit sign code even when the index stores additional bits", so it pays the storage without using it; set num_bits=1 explicitly to opt out, at the cost of the multi-bit distance estimate and some recall. Sizing formulas in references/indexes.md.

Column slice stitching (#8660) was reverted at beta.9 (#8926) - it "should not ship while the caller-managed replacement in #8923 is being developed". rust/lance-file/src/concat.rs exists again at beta.15, but holds #8923's caller-managed data file parts, not the reverted stitching.

Two proto additions. MemWAL SsTable gained in_memory_bytes, physical_rows and primary_key_bytes (fields 3-5, #8981); all optional, and absence must not be read as zero. FilteredReadOptions gained materialization_readahead_bytes and batch_size_bytes (13, 14) - not a format change under a new rule in protos/AGENTS.md: execution-plan schemas "are wire contracts, not persisted Lance formats". transaction.proto / ann.proto / index.proto are untouched.

Net-new, non-breaking: provider-native bulk copy and a deep-clone concurrency bound (section 13); Python ObjectStoreProvider registration (#8522); BinaryView in the packed blob writer (#8700); caller-managed data file parts (#8923); cleanup of specific versions (#8617); LanceDataset.slice() (#8059) and six more LanceFragment.scanner options (#8429); restored Python index retraining (#8786); namespace-managed clone deprecated to a shim (#8964). Namespace latest-version resolution no longer lists the whole _versions/ prefix (#8679) - on a ~340k-version table that was ~344 list pages, "~25s of pure I/O wait", paid by every open.

Fixes needing a rebuild or rewrite, not just an upgrade: #8779 (rebuild NGRAM indexes), #8510 (rewrite data compacted from uniformly reordered fragments), #8984 (re-drop a resurrected index), #8837 (repair a MemWAL shard below ~2.7KB/row - it cannot be reopened). Full per-PR conditions, plus the much longer list that does heal on upgrade, in references/changelog-v7-v13.md.

Mixed data-file versions LANDED - it is no longer "1 of 6". #8581-#8584 shipped in v12.0.0 (validation, per-operation V2 write targets, propagation across dataset operations, compaction targeting) and #8585 exposed it in the bindings in the v13 line. The proto changed with it: DataStorageFormat.version is now "the default format version used when writing data files", and "each DataFile's version is authoritative for decoding" once the capability is set.

In flight, not landed - do not treat as shipped: generic block v5 compression is still 1 of 10 PRs merged (#8324; #8325-#8333 all remain open). Next big dependency break in the queue: #8997, "upgrade to arrow 59, DataFusion 55, and pyo3 0.29" - still open at v13.0.0-beta.4, so arrow 58 / datafusion 54 still hold. It also gates two outstanding PyO3 advisories (RUSTSEC-2026-0176/0177); rustls was separately patched to 0.23.45 for RUSTSEC-2026-0285 (#9212).

The v13 delta

66 commits from release-root/13.0.0-beta.N to v13.0.0-beta.4, with 2 breaking-change-labeled PRs. No new crates and no new index types; 26 crates, 16 transaction ops, CommitConfig.num_retries 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024 and Python 3.10+ all still hold.

The !-vs-label rule inverted in this window. All three conventional-commit ! commits (#7465, #9192, #9101) do carry the breaking-change label. Keep treating the label as a floor rather than a ceiling - but this window is the counter-example, not more evidence for the gap.

• WriteParams gained file_writer_options (#9192) - the single labeled PR that re-rooted the major. FileWriterOptions { data_cache_bytes, max_page_bytes, keep_original_array } is now reachable from the dataset write APIs in Rust, Python and Java. A zero max_page_bytes is rejected before encoder construction rather than misbehaving later. Anything constructing WriteParams by struct literal fails to compile.
• Page metadata is initialized lazily, and the metadata cache key changed shape (#7465). StructuralFieldScheduler::initialize now takes requested_ranges, and the page-scheduler initialize splits into init_ranges() and init_from_buffers(buffers, io) - any external implementor fails to compile. The public DecodeBatchScheduler::try_new kept its signature; the range-aware entry point is the crate-private try_new_with_ranges. The part that bites without a compile error: caching moved from a per-column FieldDataCacheKey to a per-page PageDataCacheKey { column_index, page_index, view_tag }, so every warm or persisted metadata cache cold-misses across this upgrade. The payoff is real - "a cold point/range read's metadata IO is invariant to the column's total page count".
• json_extract and json_get no longer route to JSON indices (#9101). Only the four typed accessors (json_get_int / _float / _bool / _string) reach the index; everything else falls back to a full scan. This fixes three real wrong-answer bugs - a quoted-key mismatch that "searched for a quoted key and matched nothing", a Utf8 literal driving an Int64 btree into a panic, and an unsound range because "quoting is not order-preserving (ab < ab! but "ab" > "ab!")". The cost is silent: a json_extract workload that used to hit an index now scans, with no error and no plan warning. Rewrite those predicates onto the typed accessors.

The Fragment Reuse Index gained a versioned on-disk contract (#9136). InlineContent field 1 was renamed versions -> legacy_versions and a tagged transitions list added at field 2, gated on index_version >= 1; mappings are now a oneof of OrderedCompaction or StablePartition. A stable partition "assigns source rows to destination fragments while preserving their relative source order within each destination", which lets FRI reuse existing indices after reclustering - a second use case the v0 model had no concept of. Its physical form is an immutable row-map Lance file with uint16 labels and an LSPC-magic counts matrix. Two hard rules: stable row IDs and tagged FRI are mutually exclusive ("writers must not publish index_version >= 1 on them"), and cleanup "must retain intermediate transitions still needed to translate old addresses". Upstream also softened the old claim - "FRI does not remove conflicts between overlapping rewrites". Section 11.

Net-new, non-breaking: Dataset::frag_reuse_index() is public (#9112) and documented in the performance guide; FileFragment::write_overlay returns a real OverlayWriter (#8761, still env-gated); an hf:// object store with hf_enable_resolve_cache (#9236); Python lance.bitmap.Bitmap, deep_clone() (#9181), base_paths() (#9191) and update_columns(with_offsets=True) (#8891); Java DataStorageVersion, FileWriteOptions and ScanOptions.indexSegments; and namespace table listing finally bounded by read_dir_page (#9165). inline_optimization_enabled flipped true -> false (#9180), which upstream justifies with a -49% write-p50 measurement at 1M entries.

Performance questions

For anything performance-shaped - slow scans or searches, remote/object-storage cost, index maintenance cost, memory sizing, version bloat, benchmarking - load references/performance.md first. Part A routes to the official guidance plus the undocumented source-derived changes; Part B is field-verified practice against S3-compatible storage. The governing rule stays minimize remote calls - fewer commits, fewer scans, fewer round trips - because that is where the order-of-magnitude wins are. The official "Tuning remote scans" section (v11, unchanged at v12) gives a starting point for cross-region or public-internet access, where the cloud default of 64 concurrent requests is too aggressive: LANCE_IO_THREADS=8, fragment_readahead=1, batch_readahead=2, io_buffer_size=64MB. It is a legitimate second move once call volume is already minimized.

AMX-FP16 (#8540, beta.16) is the one v11 performance change that alters results, not just speed: where it engages, IVF partition assignment becomes exact instead of approximate, so recall improves and assignments differ from an older build. It is shape-gated (float16 + dot, dimension >= 32, num_centroids >= 32); everything else keeps the previous path. LANCE_DISABLE_AMX=1 disables it, but reverts assignment to the approximate path too - so an index built with it set is not equivalent to one built without it.

Two cache facts to know before tuning anything remote: Lance has no resident data cache (a Session holds only index and metadata caches, never decoded values, so repeated point reads re-pay object-store IO), and one Arc<Session> shared via DatasetBuilder::with_session lets datasets share it. Cold first search is dominated by paging indexes in - prewarm_index is the remedy. Note that #7465 changes the metadata cache key shape, so the first run after a v13 upgrade re-pays that paging even against a warm or persisted cache. Details and build-time requirements in references/performance.md.

Time travel is not an archive mechanism. Versions look like free history, but the default cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design that treats old versions as the durable record loses it on the first maintenance pass. Keep an explicit archive if you need one.

Official docs mirror

references/docs/ mirrors docs/src of lance-format/lance at the tracked tag, verbatim - 45 markdown files plus 4 diagrams, all directly readable.

Directory	Files	Covers
guide/	14	CRUD, performance, object store, distributed write + indexing, JSON, tokenizers, data types, data evolution, blob, arrays, tags/branches, migration, observability
quickstart/	4	First dataset, vector search, full-text search, versioning
format/	1	Spec-stack overview
format/file/	3	Container spec, structural encodings + compression, format versions
format/table/	9	Layout, schema, transactions (conflict-resolution matrix), versioning, row-id lineage, branch/tag, MemWAL, data overlay files
format/index/	1 + 4 svg	Index lifecycle, fragment coverage, compaction interplay
format/index/scalar/	9	fts, fmindex, ngram, btree, bitmap, bloom_filter, label_list (array_has_any/all), zonemap, rtree
format/index/vector/	1	IVF / PQ / SQ / RQ / HNSW concepts and storage layout
format/index/system/	2	Fragment reuse index, MemWAL system index
integrations/	1	DataFusion SQL over Lance, incl. JSON functions

Not mirrored: docs/src/images/ (PNG/GIF assets), so image links in the mirrored pages do not resolve - the prose is self-contained, and the four .drawio.svg diagrams are mirrored. Also out by design: community/, examples/, integrations/{index,pytorch,tensorflow}.md; and the landing stubs and contributor files (format/AGENTS.md, format/CLAUDE.md).

A whole tier of docs is not in this repo at all, so it cannot be mirrored and cannot be enumerated from a clone. docs/make-full-website.sh assembles format/catalog, format/namespace, and the integrations/{duckdb,huggingface,spark,ray,trino,context} sections at build time from six sibling repos with their own version lines - the checked-in integrations/index.md links spark/, duckdb and trino as if they were local, but those paths do not exist in the tree. Lance Context and the HuggingFace integration docs are whole nav sections that exist only on the built site. For any of those, read lance.org rather than this mirror. Protobuf message bodies are likewise expanded at build time from protos/ by mkdocs_protobuf, so the mirrored spec pages show %%% proto.message.X %%% placeholders where the site shows a rendered schema.