ci: equivalence — normalize compaction state before diff (closes #168 root cause)

Phase-2 investigation of the account-asset/contract divergence (run
26677752647: 6/8 stores byte-identical incl. account 3.6M + storage-row
17M; account-asset Go 27,917 vs Java 27,965; contract Go 560,890 vs Java
561,120). Investigated on a real Nile snapshot (EC2 10.255.10.72) with a
goleveldb SST dumper decoding internal-key sequence numbers + types.

ROOT CAUSE (not a mutation bug): the account-asset and contract stores
carry DELETE tombstones + multi-version keys from normal java-tron
operation. account-asset: 27,850 distinct keys, 51 with a DELETE
tombstone as their NEWEST version, 330 multi-version entries -> 27,799
live. goleveldb's DB.Iterator returns EXACTLY 27,799 (= 27,850 - 51),
proving goleveldb correctly omits tombstones + resolves multi-version to
newest-seq. account (3.6M) and storage-row (17M) have no such cruft and
matched byte-for-byte.

Both tools start from the identical fixture. Go's Apply opens stores via
goleveldb (compaction-on-open) and drops more of the already-deleted /
obsolete entries; java DbFork (leveldbjni) leaves the store less
compacted and physically retains them. The "java-only" keys are DELETED
keys Go correctly drops and java retains -- a PHYSICAL compaction
difference of logically-identical state (both boot java-tron to the same
chain state; deleted keys stay deleted). goleveldb does the correct,
safe-direction compaction.

FIX: before the byte diff, force a full goleveldb CompactRange of every
dbfork store on BOTH scratch dirs, converging differing physical
compaction states to the canonical live, newest-seq, tombstone-free
form. Verified on-box: goleveldb CompactRange of an account-asset store
with 51 tombstones converges it to exactly the 27,799-key live set. Real
mutation differences survive compaction (it changes physical layout, not
logical content), so genuine divergences are still caught.

Full analysis + numbers in task #168.

Author: barbatos2011

internal/dbfork/equivalence_test.go

@@ -12,6 +12,8 @@ import (
 	"strings"
 	"testing"
 
+	"github.com/syndtr/goleveldb/leveldb"
+	"github.com/syndtr/goleveldb/leveldb/util"
 	"google.golang.org/protobuf/encoding/prototext"
 	"google.golang.org/protobuf/proto"
 
@@ -171,6 +173,25 @@ func TestEquivalence_GoVsJava(t *testing.T) {
 	}
 	t.Logf("equivalence: java DbFork output:\n%s", javaOut)
 
+	// Normalize physical compaction state before the byte diff. The two
+	// tools produce the same LOGICAL state but different PHYSICAL layouts
+	// of pre-existing data: Go's Apply opens stores via goleveldb
+	// (compaction-on-open, which drops already-deleted DELETE tombstones
+	// and resolves multi-version keys to the newest sequence number)
+	// while java DbFork (leveldbjni) leaves the store less-compacted and
+	// physically retains deleted/obsolete entries. Without normalizing,
+	// the diff fails on those tombstoned/overwritten PRE-EXISTING keys —
+	// not on any mutation. Forcing a full goleveldb CompactRange on every
+	// store on BOTH sides converges them to the canonical live newest-seq
+	// / tombstone-free form, so the diff reflects only logical (mutation)
+	// differences. Verified on a real Nile snapshot (#168): goleveldb
+	// CompactRange of an account-asset store with 51 tombstones converges
+	// it to exactly the 27,799-key live set. Real mutation differences
+	// survive compaction (it changes physical layout, not logical
+	// content), so genuine divergences are still caught.
+	normalizeViaCompaction(t, scratchGo)
+	normalizeViaCompaction(t, scratchJava)
+
 	// Diff each of the 8 dbfork stores. Order is deterministic
 	// (stores.AllStores) so a failing run always reports the same
 	// store first, making CI bisection easier.
@@ -181,6 +202,31 @@ func TestEquivalence_GoVsJava(t *testing.T) {
 	}
 }
 
+// normalizeViaCompaction forces a full goleveldb CompactRange of every
+// dbfork store under root/database, converging differing physical
+// compaction states of the same logical data to the canonical live,
+// newest-sequence, tombstone-free form. See the call site + #168.
+func normalizeViaCompaction(t *testing.T, root string) {
+	t.Helper()
+	for _, store := range stores.AllStores {
+		dir := filepath.Join(root, "database", store)
+		if _, err := os.Stat(dir); err != nil {
+			continue // store legitimately absent (lite snapshot pruning)
+		}
+		ldb, err := leveldb.OpenFile(dir, nil)
+		if err != nil {
+			t.Fatalf("equivalence: normalize-open %s in %s: %v", store, root, err)
+		}
+		if err := ldb.CompactRange(util.Range{}); err != nil {
+			ldb.Close()
+			t.Fatalf("equivalence: normalize-compact %s in %s: %v", store, root, err)
+		}
+		if err := ldb.Close(); err != nil {
+			t.Fatalf("equivalence: normalize-close %s in %s: %v", store, root, err)
+		}
+	}
+}
+
 // javaHeapFlag returns the JVM `-Xmx<size>` flag for the equivalence
 // invocation. Defaults to `-Xmx4g` which is enough for Nile lite
 // snapshots; CI on smaller runners can override via DBFORK_JAVA_HEAP