Skip to content

Commit 08af0d5

Browse files
committed
fix(golden): make zip-artifact hashing OS-deterministic for package pins
Real `sam package` zips the build artifact directory before "uploading" it. The harness's S3Uploader.upload stub was hashing the resulting zip bytes — but raw zip bytes embed OS-specific metadata (DOS timestamps, Unix mode bits, central directory ordering), so the same source content produced different digests on Linux vs Windows. The Windows runner saw the package pins regenerate with new hashes and the corpus tests failed on byte-exact diff: - CodeUri: s3://golden-bucket/dd2bf2d8... + CodeUri: s3://golden-bucket/1c1f1e20... Hash zip files via a normalized representation: sorted member names plus each member's content sha256. The envelope metadata is excluded from the digest, so identical source content produces an identical hash on every OS. Non-zip artifacts continue to be hashed by raw bytes. Detection uses zipfile.is_zipfile so .jar / .war and other zip-format artifacts are treated the same as a Lambda zip. Pins regenerated once with the new hash; from now on Linux and Windows runners agree.
1 parent 6b6d93b commit 08af0d5

4 files changed

Lines changed: 42 additions & 5 deletions

File tree

tests/golden/harness.py

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434

3535
import hashlib
3636
import os
37+
import zipfile
3738
from pathlib import Path
3839
from tempfile import TemporaryDirectory
3940
from typing import Any, Dict, Optional, Tuple
@@ -140,12 +141,48 @@ def _fake_s3_upload(self: S3Uploader, file_name: str, key: Optional[str] = None)
140141
URIs, but identical content always produces the same URI. The bucket
141142
name is fixed to :data:`GOLDEN_BUCKET` regardless of the uploader's own
142143
bucket attribute so pinned outputs do not depend on local config.
144+
145+
Zip files (the common artifact shape `sam package` uploads) are hashed
146+
over a normalized representation — sorted member names plus each
147+
member's content sha256 — instead of the raw zip bytes. Raw zip bytes
148+
embed OS-specific metadata (DOS timestamps, Unix mode bits, sometimes
149+
path separators) which would otherwise produce different digests on
150+
Windows vs. POSIX runners.
143151
"""
144-
with open(file_name, "rb") as f:
145-
digest = hashlib.sha256(f.read()).hexdigest()
152+
digest = _hash_zip_normalized(file_name) if _looks_like_zip(file_name) else _hash_raw_bytes(file_name)
146153
return f"s3://{GOLDEN_BUCKET}/{digest}"
147154

148155

156+
def _looks_like_zip(file_name: str) -> bool:
157+
"""True if ``file_name`` looks like a zip-format artifact."""
158+
try:
159+
return zipfile.is_zipfile(file_name)
160+
except OSError:
161+
return False
162+
163+
164+
def _hash_raw_bytes(file_name: str) -> str:
165+
with open(file_name, "rb") as f:
166+
return hashlib.sha256(f.read()).hexdigest()
167+
168+
169+
def _hash_zip_normalized(file_name: str) -> str:
170+
"""Hash a zip file by its sorted member names + per-member content hashes.
171+
172+
This skips the zip envelope metadata (DOS timestamps, Unix permission
173+
bits, central directory ordering) that varies by OS and makes raw
174+
byte-level hashes non-deterministic across runners.
175+
"""
176+
hasher = hashlib.sha256()
177+
with zipfile.ZipFile(file_name, "r") as zf:
178+
for name in sorted(zf.namelist()):
179+
hasher.update(name.encode("utf-8"))
180+
hasher.update(b"\x00")
181+
with zf.open(name) as member:
182+
hasher.update(hashlib.sha256(member.read()).digest())
183+
return hasher.hexdigest()
184+
185+
149186
def _fake_s3_upload_with_dedup(
150187
self: S3Uploader,
151188
file_name: str,

tests/golden/templates/language_extensions/foreach_static_zip/expected.package.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Resources:
66
- Beta
77
- ${WorkerName}Function:
88
Properties:
9-
CodeUri: s3://golden-bucket/dd2bf2d87ad97f55f64a8691a0e0c55ec010e7791d6815e0c0f0037348b8d5db
9+
CodeUri: s3://golden-bucket/b38f16354b937b1d3c3d74e0c9d24ab809eee8b0622ec5b6408fee535b822d42
1010
Handler: app.handler
1111
Runtime: python3.11
1212
Type: AWS::Serverless::Function

tests/golden/templates/packageable_resources/lambda_function_zip/expected.package.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Resources:
66
Properties:
77
Code:
88
S3Bucket: golden-bucket
9-
S3Key: 7187d20310f5514ad6d32ad02c3f6a8200a0b82949bcb4c63a751bc4ce730095
9+
S3Key: 22186578b8d9d69e7157330653df80d6801d2e1bc7af9c2a7622640de339ae63
1010
FunctionName: hello
1111
Handler: app.handler
1212
MemorySize: 128

tests/golden/templates/sam_resources/serverless_function_zip/expected.package.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Resources:
44
Metadata:
55
SamResourceId: HelloFunction
66
Properties:
7-
CodeUri: s3://golden-bucket/7187d20310f5514ad6d32ad02c3f6a8200a0b82949bcb4c63a751bc4ce730095
7+
CodeUri: s3://golden-bucket/22186578b8d9d69e7157330653df80d6801d2e1bc7af9c2a7622640de339ae63
88
Handler: app.handler
99
MemorySize: 128
1010
Runtime: python3.11

0 commit comments

Comments
 (0)