Skip to content

Commit 6a7ddab

Browse files
minitrigaclaudegmazoyer
authored
feat: add SHA-1 idempotency primitives for CoreFileObject (#963)
--------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Guillaume Mazoyer <guillaume@opsmill.com>
1 parent 191e024 commit 6a7ddab

8 files changed

Lines changed: 1115 additions & 13 deletions

File tree

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Added SHA-1 idempotency primitives for `CoreFileObject` nodes:
2+
3+
- `InfrahubNode.matches_local_checksum(source)` / sync variant — compare a local `bytes | Path | BinaryIO` source against the node's server-stored checksum without invoking a transfer.
4+
- `InfrahubNode.upload_if_changed(source, name=None)` / sync variant — stage + save only when the local source differs from the server, returning an `UploadResult(was_uploaded, checksum)` dataclass.
5+
- `download_file(..., skip_if_unchanged=True)` — short-circuit the download when `dest` already exists on disk with a matching SHA-1. Returns `0` bytes written when skipped.
6+
7+
A shared `sha1_of_source` helper (streaming, 64 KiB chunks) centralises the hashing convention in `infrahub_sdk.file_handler`.

docs/docs/python-sdk/sdk_ref/infrahub_sdk/node/node.mdx

Lines changed: 233 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,22 @@ artifact_fetch(self, name: str) -> str | dict[str, Any]
4040
#### `download_file`
4141

4242
```python
43-
download_file(self, dest: Path | None = None) -> bytes | int
43+
download_file(self, dest: None = None, skip_if_unchanged: bool = ...) -> bytes
44+
```
45+
46+
<details>
47+
<summary>Show 2 other overloads</summary>
48+
49+
#### `download_file`
50+
51+
```python
52+
download_file(self, dest: Path, skip_if_unchanged: bool = ...) -> int
53+
```
54+
55+
#### `download_file`
56+
57+
```python
58+
download_file(self, dest: Path | None = None, skip_if_unchanged: bool = False) -> bytes | int
4459
```
4560

4661
Download the file content from this FileObject node.
@@ -54,16 +69,25 @@ The node must have been saved (have an id) before calling this method.
5469
directly to this path (memory-efficient for large files) and the
5570
number of bytes written will be returned. If not provided, the
5671
file content will be returned as bytes.
72+
- `skip_if_unchanged`: When ``True``, compute the SHA-1 of the file at
73+
``dest`` (which must be provided) and compare against the
74+
node's ``checksum`` attribute. If they match, return ``0``
75+
without hitting the network. The ``checksum`` is the value
76+
loaded when this node was fetched — a later server-side
77+
change to the file will not be detected unless the caller
78+
re-fetches the node first.
5779

5880
**Returns:**
5981

6082
- If ``dest`` is None: The file content as bytes.
6183
- If ``dest`` is provided: The number of bytes written to the file.
84+
- If ``skip_if_unchanged=True`` and the local file matches the server checksum: ``0``.
6285

6386
**Raises:**
6487

6588
- `FeatureNotSupportedError`: If this node doesn't inherit from CoreFileObject.
66-
- `ValueError`: If the node hasn't been saved yet or file not found.
89+
- `ValueError`: If the node hasn't been saved yet, file not found, or
90+
``skip_if_unchanged=True`` was passed without a ``dest``.
6791
- `AuthenticationError`: If authentication fails.
6892

6993
**Examples:**
@@ -73,8 +97,88 @@ The node must have been saved (have an id) before calling this method.
7397
>>> content = await contract.download_file()
7498
>>> # Stream to file (memory-efficient for large files)
7599
>>> bytes_written = await contract.download_file(dest=Path("/tmp/contract.pdf"))
100+
>>> # Skip download if local file already matches server checksum
101+
>>> bytes_written = await contract.download_file(
102+
... dest=Path("/tmp/contract.pdf"), skip_if_unchanged=True
103+
... )
104+
```
105+
106+
</details>
107+
#### `matches_local_checksum`
108+
109+
```python
110+
matches_local_checksum(self, source: bytes | Path | BinaryIO) -> bool
111+
```
112+
113+
Return True if ``source``'s SHA-1 matches this node's server checksum.
114+
115+
Only available for nodes inheriting from ``CoreFileObject``. Callers
116+
that want to branch on the comparison without invoking a transfer
117+
should use this primitive instead of reading ``node.checksum.value``
118+
and hashing ``source`` themselves, so the hashing convention stays
119+
centralised in the SDK.
120+
121+
The comparison is against the ``checksum`` attribute as loaded
122+
when this node was retrieved from the server. If the server's
123+
file has been replaced since the node was fetched, this method
124+
will not see that change — re-fetch the node to refresh the
125+
checksum before comparing.
126+
127+
**Args:**
128+
129+
- `source`: Local content to hash and compare. Accepts the same
130+
shapes as \:func\:`infrahub_sdk.file_handler.sha1_of_source`.
131+
132+
**Returns:**
133+
134+
- True if the local digest equals the server's stored checksum.
135+
136+
**Raises:**
137+
138+
- `FeatureNotSupportedError`: Node is not a ``CoreFileObject``.
139+
- `ValueError`: Node has no server-side checksum yet (unsaved or
140+
file never attached).
141+
142+
#### `upload_if_changed`
143+
144+
```python
145+
upload_if_changed(self, source: bytes | Path | BinaryIO, name: str | None = None) -> UploadResult
76146
```
77147

148+
Upload ``source`` only if its SHA-1 differs from the server checksum.
149+
150+
Composes :meth:`matches_local_checksum` with :meth:`upload_from_path`
151+
(or :meth:`upload_from_bytes`) and :meth:`save`. For unsaved nodes or
152+
nodes that have no prior server-side file, the upload is always
153+
performed — there is nothing to compare against.
154+
155+
Idempotency is content-only: when the local SHA-1 matches the server
156+
checksum the upload is skipped even if ``name`` differs from the
157+
server-side filename. Use a regular :meth:`upload_from_path` /
158+
:meth:`save` round-trip if you need to rename without changing
159+
content.
160+
161+
**Args:**
162+
163+
- `source`: Content to upload. ``bytes`` and ``BinaryIO`` sources
164+
must supply ``name``; for a ``Path`` the filename is derived
165+
from ``source.name`` when ``name`` is omitted.
166+
- `name`: Filename to use on the server. Required for ``bytes`` /
167+
``BinaryIO`` sources.
168+
169+
**Returns:**
170+
171+
- class:`UploadResult` with ``was_uploaded=False`` (skipped) or
172+
- ``was_uploaded=True`` (transfer occurred), and the resulting server
173+
- checksum (``None`` only when no server checksum was available
174+
- after the operation).
175+
176+
**Raises:**
177+
178+
- `FeatureNotSupportedError`: Node is not a ``CoreFileObject``.
179+
- `ValueError`: ``source`` is ``bytes`` or ``BinaryIO`` and no
180+
``name`` was supplied.
181+
78182
#### `delete`
79183

80184
```python
@@ -221,7 +325,22 @@ artifact_fetch(self, name: str) -> str | dict[str, Any]
221325
#### `download_file`
222326

223327
```python
224-
download_file(self, dest: Path | None = None) -> bytes | int
328+
download_file(self, dest: None = None, skip_if_unchanged: bool = ...) -> bytes
329+
```
330+
331+
<details>
332+
<summary>Show 2 other overloads</summary>
333+
334+
#### `download_file`
335+
336+
```python
337+
download_file(self, dest: Path, skip_if_unchanged: bool = ...) -> int
338+
```
339+
340+
#### `download_file`
341+
342+
```python
343+
download_file(self, dest: Path | None = None, skip_if_unchanged: bool = False) -> bytes | int
225344
```
226345

227346
Download the file content from this FileObject node.
@@ -235,16 +354,25 @@ The node must have been saved (have an id) before calling this method.
235354
directly to this path (memory-efficient for large files) and the
236355
number of bytes written will be returned. If not provided, the
237356
file content will be returned as bytes.
357+
- `skip_if_unchanged`: When ``True``, compute the SHA-1 of the file at
358+
``dest`` (which must be provided) and compare against the
359+
node's ``checksum`` attribute. If they match, return ``0``
360+
without hitting the network. The ``checksum`` is the value
361+
loaded when this node was fetched — a later server-side
362+
change to the file will not be detected unless the caller
363+
re-fetches the node first.
238364

239365
**Returns:**
240366

241367
- If ``dest`` is None: The file content as bytes.
242368
- If ``dest`` is provided: The number of bytes written to the file.
369+
- If ``skip_if_unchanged=True`` and the local file matches the server checksum: ``0``.
243370

244371
**Raises:**
245372

246373
- `FeatureNotSupportedError`: If this node doesn't inherit from CoreFileObject.
247-
- `ValueError`: If the node hasn't been saved yet or file not found.
374+
- `ValueError`: If the node hasn't been saved yet, file not found, or
375+
``skip_if_unchanged=True`` was passed without a ``dest``.
248376
- `AuthenticationError`: If authentication fails.
249377

250378
**Examples:**
@@ -254,8 +382,88 @@ The node must have been saved (have an id) before calling this method.
254382
>>> content = contract.download_file()
255383
>>> # Stream to file (memory-efficient for large files)
256384
>>> bytes_written = contract.download_file(dest=Path("/tmp/contract.pdf"))
385+
>>> # Skip download if local file already matches server checksum
386+
>>> bytes_written = contract.download_file(
387+
... dest=Path("/tmp/contract.pdf"), skip_if_unchanged=True
388+
... )
389+
```
390+
391+
</details>
392+
#### `matches_local_checksum`
393+
394+
```python
395+
matches_local_checksum(self, source: bytes | Path | BinaryIO) -> bool
257396
```
258397

398+
Return True if ``source``'s SHA-1 matches this node's server checksum.
399+
400+
Only available for nodes inheriting from ``CoreFileObject``. Callers
401+
that want to branch on the comparison without invoking a transfer
402+
should use this primitive instead of reading ``node.checksum.value``
403+
and hashing ``source`` themselves, so the hashing convention stays
404+
centralised in the SDK.
405+
406+
The comparison is against the ``checksum`` attribute as loaded
407+
when this node was retrieved from the server. If the server's
408+
file has been replaced since the node was fetched, this method
409+
will not see that change — re-fetch the node to refresh the
410+
checksum before comparing.
411+
412+
**Args:**
413+
414+
- `source`: Local content to hash and compare. Accepts the same
415+
shapes as \:func\:`infrahub_sdk.file_handler.sha1_of_source`.
416+
417+
**Returns:**
418+
419+
- True if the local digest equals the server's stored checksum.
420+
421+
**Raises:**
422+
423+
- `FeatureNotSupportedError`: Node is not a ``CoreFileObject``.
424+
- `ValueError`: Node has no server-side checksum yet (unsaved or
425+
file never attached).
426+
427+
#### `upload_if_changed`
428+
429+
```python
430+
upload_if_changed(self, source: bytes | Path | BinaryIO, name: str | None = None) -> UploadResult
431+
```
432+
433+
Upload ``source`` only if its SHA-1 differs from the server checksum.
434+
435+
Composes :meth:`matches_local_checksum` with :meth:`upload_from_path`
436+
(or :meth:`upload_from_bytes`) and :meth:`save`. For unsaved nodes or
437+
nodes that have no prior server-side file, the upload is always
438+
performed — there is nothing to compare against.
439+
440+
Idempotency is content-only: when the local SHA-1 matches the server
441+
checksum the upload is skipped even if ``name`` differs from the
442+
server-side filename. Use a regular :meth:`upload_from_path` /
443+
:meth:`save` round-trip if you need to rename without changing
444+
content.
445+
446+
**Args:**
447+
448+
- `source`: Content to upload. ``bytes`` and ``BinaryIO`` sources
449+
must supply ``name``; for a ``Path`` the filename is derived
450+
from ``source.name`` when ``name`` is omitted.
451+
- `name`: Filename to use on the server. Required for ``bytes`` /
452+
``BinaryIO`` sources.
453+
454+
**Returns:**
455+
456+
- class:`UploadResult` with ``was_uploaded=False`` (skipped) or
457+
- ``was_uploaded=True`` (transfer occurred), and the resulting server
458+
- checksum (``None`` only when no server checksum was available
459+
- after the operation).
460+
461+
**Raises:**
462+
463+
- `FeatureNotSupportedError`: Node is not a ``CoreFileObject``.
464+
- `ValueError`: ``source`` is ``bytes`` or ``BinaryIO`` and no
465+
``name`` was supplied.
466+
259467
#### `delete`
260468

261469
```python
@@ -369,6 +577,27 @@ extract(self, params: dict[str, str]) -> dict[str, Any]
369577

370578
Extract some data points defined in a flat notation.
371579

580+
### `UploadResult`
581+
582+
Outcome of an idempotent upload attempt.
583+
584+
Returned by :meth:`InfrahubNode.upload_if_changed` and its sync twin.
585+
``was_uploaded`` tells the caller whether a network transfer actually
586+
happened; ``checksum`` carries the SHA-1 of the content held on the
587+
server after the operation — on skip paths that is the server's
588+
pre-existing value, on upload paths it is the locally-computed SHA-1
589+
used as a proxy (which matches what a standard CoreFileObject server
590+
stores, since the server computes SHA-1 of received bytes). ``None``
591+
only when no server checksum was available (either the node was
592+
unsaved and nothing was transferred, or the save returned no checksum
593+
value).
594+
595+
The comparison used by ``upload_if_changed`` reads the node's
596+
``checksum`` attribute, which was populated when the node was
597+
fetched via ``client.get(...)``. A server-side change to the file
598+
between the fetch and the call will not be detected unless the
599+
caller re-fetches the node first.
600+
372601
### `InfrahubNodeBase`
373602

374603
Base class for InfrahubNode and InfrahubNodeSync

infrahub_sdk/file_handler.py

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
from __future__ import annotations
22

3+
import hashlib
34
from dataclasses import dataclass
45
from io import BytesIO
56
from pathlib import Path
@@ -13,6 +14,53 @@
1314
if TYPE_CHECKING:
1415
from .client import InfrahubClient, InfrahubClientSync
1516

17+
_SHA1_CHUNK_BYTES = 64 * 1024
18+
19+
20+
def sha1_of_source(source: bytes | Path | BinaryIO) -> str:
21+
"""Compute the SHA-1 hex digest of an upload/download source.
22+
23+
Accepts the same shapes as :meth:`FileHandlerBase.prepare_upload` so
24+
callers can compare local content against a server-stored checksum
25+
without materialising the full file in memory.
26+
27+
Args:
28+
source: The content to hash. ``bytes`` are hashed in one shot.
29+
A ``Path`` is read in 64 KiB chunks. A ``BinaryIO`` is read
30+
from its current position, then rewound so downstream
31+
callers can re-read it.
32+
33+
Returns:
34+
Lowercase SHA-1 hex digest, matching the algorithm Infrahub
35+
stores in ``CoreFileObject.checksum``.
36+
37+
Raises:
38+
TypeError: If ``source`` is not one of the supported types.
39+
40+
"""
41+
hasher = hashlib.sha1(usedforsecurity=False)
42+
43+
if isinstance(source, bytes):
44+
hasher.update(source)
45+
return hasher.hexdigest()
46+
47+
if isinstance(source, Path):
48+
with source.open("rb") as fh:
49+
while chunk := fh.read(_SHA1_CHUNK_BYTES):
50+
hasher.update(chunk)
51+
return hasher.hexdigest()
52+
53+
if hasattr(source, "read") and hasattr(source, "seek"):
54+
start = source.tell()
55+
try:
56+
while chunk := source.read(_SHA1_CHUNK_BYTES):
57+
hasher.update(chunk)
58+
finally:
59+
source.seek(start)
60+
return hasher.hexdigest()
61+
62+
raise TypeError(f"sha1_of_source expects bytes, Path, or BinaryIO; got {type(source).__name__}")
63+
1664

1765
@dataclass
1866
class PreparedFile:

0 commit comments

Comments
 (0)