backup: #904 v12 - remove stale .fsm on self-test mismatch

## Codex P2 v10: self-test mismatch leaves orphaned stale .fsm

When --output already exists from a prior successful run and the new
--self-test invocation detects a mismatch, writeAndPublish returned
errSelfTestMismatch without removing the stale <output>.fsm. encodeOne
then wrote a fresh <output>.encode_info.json with self_test.matched=false
and the NEW SHA pointing to the unpublished temp snapshot, leaving:

- <output>.fsm: STALE bytes from the prior successful run
- <output>.encode_info.json: NEW SHA + matched=false
- <output>.mismatch.txt: from the new run

The sidecar describes an FSM that does not exist on disk. This
violates the CLI contract that "a self-test failure leaves no
restore-visible FSM" and breaks runbooks that consume the sidecar's
SHA to verify FSM integrity (they would find the stale FSM's SHA
does not match the sidecar's NEW SHA).

Fix: writeAndPublish, on the self-test-mismatch branch, now
os.Remove(cfg.outputPath) before returning. After cleanup the
artifacts are internally consistent: the sidecar describes a FAILED
encode attempt, mismatch.txt has the diff, and the FSM path is
absent (matching the sidecar's intent).

errors.Is(rerr, os.ErrNotExist) is treated as success — a
self-test mismatch from a first-ever encode (no prior FSM) leaves
the path absent, which is the same end state.

## Surgical scope — only self-test mismatch wipes stale .fsm

Other exit-2 paths (manifest-floor regression, adapter-data error,
unsupported DDB layout) preserve any prior <output>.fsm because
those failures occur BEFORE writeAndPublish runs. This preserves
the runbook ergonomic of "operator typo'd --last-commit-ts; last
good FSM still on disk to retry against."

## Test infrastructure

To drive a deterministic self-test mismatch end-to-end through the
CLI's writeAndPublish, a minimal test seam was added to package
backup: EncodeOptions.SetSelfTestCorruptHookForTest(func(*os.File))
exposes the previously-package-private corruptBufferForTest field
to callers outside package backup. Production code MUST NOT call
this; the godoc explicitly names cmd/elastickv-snapshot-encode tests
as the sole intended caller.

Pinned by:
- TestCLIWriteAndPublishRemovesStaleFSMOnSelfTestMismatch: pre-places
  a stale .fsm, injects corruption via the new seam, asserts
  publishErr == errSelfTestMismatch, the stale .fsm is GONE, and
  mismatch.txt is present.
- TestCLINonSelfTestExitTwoPreservesPriorFSM: pre-places a stale
  .fsm, drives a manifest-floor regression (exit-2 BEFORE
  writeAndPublish), asserts the stale .fsm is byte-for-byte
  preserved. Pins the surgical scope of the cleanup.

## Caller audit per CLAUDE.md semantic-change rule

- writeAndPublish: sole production caller is encodeOne. On self-test
  mismatch, encodeOne still writes the sidecar (publishErr matches
  errSelfTestMismatch) and returns publishErr. The new os.Remove
  happens before that, so the caller sees IDENTICAL error semantics
  and the same sidecar-write path. The only difference is on-disk
  state for <output>.fsm: was stale, is now absent.
- EncodeOptions.SetSelfTestCorruptHookForTest: new exported method.
  No production callers anywhere; one test caller in
  cmd/elastickv-snapshot-encode/main_test.go. In-package backup
  tests continue to set corruptBufferForTest directly.

Tests + lint green.

Author: bootjp

cmd/elastickv-snapshot-encode/main.go

@@ -335,6 +335,16 @@ func writeAndPublish(cfg *config, encodeOpts backup.EncodeOptions, mismatchPath
 		if werr := os.WriteFile(mismatchPath, result.SelfTestMismatchTxt, mismatchTxtPerm); werr != nil {
 			logger.Warn("write mismatch.txt", "err", werr)
 		}
+		// Remove the stale <output>.fsm if one exists from a prior
+		// successful run. encodeOne is about to write a fresh
+		// <output>.encode_info.json with self_test.matched=false and
+		// a NEW SHA pointing to the unpublished temp snapshot; leaving
+		// the old bytes on disk would make the sidecar describe an
+		// FSM that does not exist and violate the "self-test failure
+		// leaves no restore-visible FSM" contract (codex P2 v10 #904).
+		if rerr := os.Remove(cfg.outputPath); rerr != nil && !errors.Is(rerr, os.ErrNotExist) {
+			logger.Warn("remove stale .fsm on self-test mismatch", "err", rerr)
+		}
 		return result, errors.Wrap(errSelfTestMismatch, "self-test diff (see "+mismatchPath+")")
 	}
 	if err := os.Rename(tempPath, cfg.outputPath); err != nil {

cmd/elastickv-snapshot-encode/main_test.go

@@ -12,6 +12,7 @@ import (
 	"time"
 
 	"github.com/bootjp/elastickv/internal/backup"
+	"github.com/cockroachdb/errors"
 )
 
 // isWindows is true on Windows builds; perm-bit tests skip on Windows
@@ -300,6 +301,130 @@ func readSidecar(t *testing.T, output string) backup.EncodeInfo {
 	return info
 }
 
+// TestCLIWriteAndPublishRemovesStaleFSMOnSelfTestMismatch pins codex
+// P2 v10 #904: when a prior successful run left an <output>.fsm on
+// disk and a new --self-test invocation produces a mismatch,
+// writeAndPublish must remove that stale .fsm. Otherwise encodeOne
+// writes a fresh sidecar (matched=false, NEW SHA) alongside the OLD
+// bytes — violating the CLI contract that a self-test failure leaves
+// no restore-visible FSM, and making the sidecar describe an FSM that
+// is not on disk.
+//
+// To drive a deterministic self-test mismatch end-to-end through the
+// CLI's writeAndPublish, the test uses the backup package's exported
+// test seam (SetSelfTestCorruptHookForTest) to flip bytes in the
+// disk-backed self-test buffer between WriteTo and the re-decode.
+func TestCLIWriteAndPublishRemovesStaleFSMOnSelfTestMismatch(t *testing.T) {
+	t.Parallel()
+	rawIn := t.TempDir()
+	writeSQSFixture(t, rawIn)
+	emitMinimalManifest(t, rawIn, 7000)
+	canonicalIn := canonicalizeInput(t, rawIn, 7000)
+
+	out := filepath.Join(t.TempDir(), "out.fsm")
+	// Pre-place a stale .fsm — what a prior successful run would have
+	// left behind. The codex P2 v10 contract is that a subsequent
+	// self-test mismatch invalidates this file.
+	stalePayload := []byte("STALE FSM FROM PRIOR SUCCESSFUL RUN")
+	if err := os.WriteFile(out, stalePayload, 0o600); err != nil {
+		t.Fatalf("WriteFile stale: %v", err)
+	}
+
+	scratchBase := t.TempDir()
+	encodeOpts := backup.EncodeOptions{
+		InputRoot:            canonicalIn,
+		Adapters:             backup.AdapterSet{SQS: true},
+		LastCommitTS:         7000,
+		ManifestLastCommitTS: 7000,
+		SelfTest:             true,
+		SelfTestDecodeOptions: backup.DecodeOptions{
+			OutRoot:  scratchBase,
+			Adapters: backup.AdapterSet{SQS: true},
+		},
+	}
+	// Flip bytes past the EKVPBBL1 header so the re-decode trips on
+	// a malformed entry length and the self-test returns matched=false.
+	// Pattern mirrors flipBytesPastHeaderHelper in the library test.
+	encodeOpts.SetSelfTestCorruptHookForTest(func(f *os.File) {
+		info, ferr := f.Stat()
+		if ferr != nil {
+			t.Fatalf("temp Stat: %v", ferr)
+		}
+		const headerSkip = 200
+		if info.Size() <= headerSkip {
+			t.Fatalf("temp file too small to corrupt past header: %d bytes", info.Size())
+		}
+		buf := make([]byte, info.Size()-headerSkip)
+		if _, rerr := f.ReadAt(buf, headerSkip); rerr != nil {
+			t.Fatalf("ReadAt: %v", rerr)
+		}
+		for i := 0; i < len(buf); i += 13 {
+			buf[i] ^= 0xFF
+		}
+		if _, werr := f.WriteAt(buf, headerSkip); werr != nil {
+			t.Fatalf("WriteAt: %v", werr)
+		}
+	})
+
+	cfg := &config{
+		inputPath:  canonicalIn,
+		outputPath: out,
+		adapters:   backup.AdapterSet{SQS: true},
+		selfTest:   true,
+	}
+	mismatchPath := out + ".mismatch.txt"
+
+	_, publishErr := writeAndPublish(cfg, encodeOpts, mismatchPath, quietLogger())
+	if !errors.Is(publishErr, errSelfTestMismatch) {
+		t.Fatalf("publishErr = %v, want errSelfTestMismatch", publishErr)
+	}
+	if _, statErr := os.Stat(out); !os.IsNotExist(statErr) {
+		t.Errorf("stale .fsm at %s not removed after self-test mismatch (codex P2 v10)", out)
+	}
+	// The mismatch.txt should be present as the operator-visible
+	// record of the failed encode attempt.
+	if _, statErr := os.Stat(mismatchPath); statErr != nil {
+		t.Errorf("mismatch.txt missing after self-test mismatch: %v", statErr)
+	}
+}
+
+// TestCLINonSelfTestExitTwoPreservesPriorFSM pins the surgical scope
+// of the codex P2 v10 fix: non-self-test exit-2 paths (e.g. the
+// manifest-floor HLC regression that fails BEFORE writeAndPublish)
+// must NOT remove a prior <output>.fsm. Only self-test mismatch
+// triggers the cleanup; a runbook recovering from a typo'd
+// --last-commit-ts still has its last good FSM on disk.
+func TestCLINonSelfTestExitTwoPreservesPriorFSM(t *testing.T) {
+	t.Parallel()
+	in := t.TempDir()
+	emitMinimalManifest(t, in, 1000)
+	out := filepath.Join(t.TempDir(), "out.fsm")
+	stalePayload := []byte("STALE FSM FROM PRIOR SUCCESSFUL RUN")
+	if err := os.WriteFile(out, stalePayload, 0o600); err != nil {
+		t.Fatalf("WriteFile stale: %v", err)
+	}
+	// Manifest-floor regression → exit-2 from resolveLastCommitTS,
+	// before writeAndPublish runs. Stale .fsm should be preserved.
+	code, err := run([]string{
+		"--input", in,
+		"--output", out,
+		"--last-commit-ts", "500", // below manifest 1000
+	}, quietLogger())
+	if err == nil {
+		t.Fatalf("run succeeded; want manifest-floor regression")
+	}
+	if code != exitDataErr {
+		t.Errorf("exit code = %d, want %d", code, exitDataErr)
+	}
+	body, rerr := os.ReadFile(out)
+	if rerr != nil {
+		t.Fatalf("read stale .fsm: %v (must be preserved on non-self-test exit-2)", rerr)
+	}
+	if !bytes.Equal(body, stalePayload) {
+		t.Errorf("stale .fsm mutated; want preserved on manifest-floor regression")
+	}
+}
+
 // TestCLIRoundTripSelfTestAllAdapters is the gold-standard CLI-level
 // end-to-end test: a real adapter fixture, encoder runs with
 // --self-test, exit 0, matched:true in the sidecar.

internal/backup/encode_snapshot.go

@@ -135,6 +135,21 @@ type EncodeOptions struct {
 	corruptBufferForTest func(*os.File)
 }
 
+// SetSelfTestCorruptHookForTest installs a same-process hook that
+// fires against the on-disk self-test buffer between WriteTo and the
+// re-decode call. The hook can WriteAt into the file to inject
+// corruption so the subsequent self-test mismatches deterministically.
+//
+// Production code MUST NOT call this; it is exclusively a test seam
+// for callers OUTSIDE package backup (specifically the
+// cmd/elastickv-snapshot-encode CLI tests, which need to drive a real
+// end-to-end self-test mismatch to verify the stale-.fsm cleanup
+// path — codex P2 v10 #904). In-package tests should set
+// EncodeOptions.corruptBufferForTest directly.
+func (o *EncodeOptions) SetSelfTestCorruptHookForTest(hook func(*os.File)) {
+	o.corruptBufferForTest = hook
+}
+
 // EncodeResult is the public return value from EncodeSnapshot. Mirrors
 // the decoder's DecodeResult shape.
 type EncodeResult struct {