docs: define file format stability contract (#7779)

This codifies how Lance contributors should reason about stable and
unstable file format changes. Stable formats remain backward- and
forward-compatible, while unstable formats may evolve without
compatibility shims for unreleased intermediate states.

It also scopes protobuf compatibility accordingly, keeping decisions
anchored to the latest released stable format instead of transient
behavior on a development branch or `main`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added guidance for evaluating file-format stability and compatibility.
* Clarified backward-compatibility requirements for stable and persisted
formats.
* Documented that unstable-only formats do not require compatibility
migrations or safeguards.
* Added verification steps before making potentially breaking Protobuf
schema changes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: Xuanwo

AGENTS.md

@@ -30,6 +30,12 @@ Rust workspace with Python and Java bindings:
 
 Key technical traits: async-first (tokio), Arrow-native, versioned writes with manifest tracking, custom ML-optimized encodings, unified object store interface (local/S3/Azure/GCS).
 
+## File Format Stability and Compatibility
+
+- Treat every file format marked stable as a durable compatibility contract. All changes to a stable format must preserve both backward and forward compatibility.
+- Treat every file format marked unstable as disposable. It may change freely; do not add compatibility code, migrations, fallbacks, or tests for files written by earlier unstable revisions.
+- Evaluate compatibility against the latest released stable version while continuing to honor all stable format contracts. Changes that exist only on the current branch or `main` are not compatibility constraints; do not compromise a cleaner or more complete design to preserve those intermediate states.
+
 ## Development Commands
 
 ### Rust

protos/AGENTS.md

@@ -4,7 +4,8 @@ Also see [root AGENTS.md](../AGENTS.md) for cross-language standards.
 
 ## Compatibility
 
-- All changes must be backwards compatible. Never re-use or change field numbers of existing fields.
+- Protobuf schemas that are part of a stable file format or any other stable persisted contract must remain backwards compatible. Never reuse or change their existing field numbers.
+- Protobuf schemas used exclusively by an unstable file format follow the root file-format stability contract: do not preserve compatibility with prior unstable revisions. Before making a breaking protobuf change, verify that the schema is not shared with a stable format or another persisted contract.
 
 ## Schema Design