docs: document DataFile column_indices changes in 2.1 format (#6416)

## Summary

- Add a 5.0.0 section to the migration guide documenting how
`DataFile.column_indices` changed with data storage version 2.1:
non-leaf fields (structs, lists) now get `-1` instead of sequential
column indices
- Add an admonition to the table format spec's Data Files section noting
the version difference
- Includes a concrete before/after example and opt-out instructions

Closes #6411

## Test plan

- [x] Docs build successfully with `mkdocs build`
- [ ] Verify rendered migration guide section and table format
admonition look correct

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: westonpace

docs/src/format/table/index.md

@@ -121,6 +121,17 @@ or independently of column indices due to variable encoding widths (for Lance fi
 
 </details>
 
+!!! note "Field-to-column mapping differs between data storage versions"
+
+    In **2.0**, all fields (including non-leaf fields like struct and list containers) are assigned
+    sequential column indices in `column_indices`.
+
+    In **2.1+**, non-leaf fields (unpacked structs, list containers) are assigned `-1` in
+    `column_indices` because their validity information is folded into repetition/definition
+    levels. Only leaf fields and packed structs have column indices.
+
+    See the [5.0.0 migration guide](../../guide/migration.md#500) for a detailed example.
+
 ## Deletion Files
 
 Deletion files (a.k.a. deletion vectors) track deleted rows without rewriting data files.

docs/src/guide/migration.md

@@ -6,6 +6,41 @@ stable and breaking changes should generally be communicated (via warnings) for
 give users a chance to migrate.  This page documents the breaking changes between releases and gives advice on how to
 migrate.
 
+## 5.0.0
+
+* The default data storage version changed from 2.0 to 2.1. This affects the `column_indices`
+  field in the `DataFile` protobuf message. In 2.0, every field (including non-leaf fields like
+  struct containers and list containers) was assigned a sequential column index. In 2.1, non-leaf
+  fields (unpacked structs, list containers) are assigned `-1` instead since their validity
+  information is now folded into repetition/definition levels. Only leaf fields and packed structs
+  are assigned column indices.
+
+    For example, given the schema:
+
+    ```
+    x: i32, y: [f32], z: { a: i32 }
+    ```
+
+    The fields (in depth-first order) are:
+
+    | Field ID | Field         |
+    |----------|---------------|
+    | 0        | `x` (i32)     |
+    | 1        | `y` (list)    |
+    | 2        | `y.item` (f32)|
+    | 3        | `z` (struct)  |
+    | 4        | `z.a` (i32)   |
+
+    In **2.0**, `column_indices` = `[0, 1, 2, 3, 4]` — every field gets a column.
+
+    In **2.1**, `column_indices` = `[0, -1, 1, -1, 2]` — non-leaf fields (`y` and `z`) get `-1`.
+
+* This change only affects advanced users who construct `DataFile` messages directly, for example
+  when building operations by hand for `Dataset.commit`. Normal read and write paths are
+  unaffected.
+
+* To opt back to 2.0 format, set `data_storage_version="2.0"` when creating a dataset.
+
 ## 1.0.0
 
 * The `SearchResult` returned by scalar indices must now output information about null values.