Feature: Allow using inode in ETags

[bb4242]

Issue Details

It would be nice to be able to configure caddy to include file inodes when calculating ETags, similar to Apache.

My use case involves serving static files from the nix store, (see NixOS/nixpkgs#222354), where the files' modification times are always 1, per nix's reproducibility guarantees. In this case, caddy skips ETag generation entirely (#5548), which can cause caching problems where browsers fail to detect that files have changed. The currently recommended approach in the nix world seems to be to strip the Last-Modified header also, which forces browsers to re-fetch assets every time. This works, but leads to unnecessary overhead.

I think it would be nice if it were possible to configure caddy to still calculate ETags, based on file size and inode number, rather than skipping ETag generation entirely. Using inode number has some downsides, such as in load balancing scenarios, but would be very useful in single server cases.

Assistance Disclosure

AI not used

If AI was used, describe the extent to which it was used.

No response

[steadytao]

This seems legitimate to me. The Nix-store case is a real gap, mtimes of 1 are intentionally not meaningful so the current safeguard from #5548 avoids unsafe stale caching but leaves users without a useful validator.

I would be cautious about inode-based ETags as a default though. Inodes are local filesystem identity not content identity and can be wrong or noisy across load-balanced servers, containers, bind mounts, copied stores or filesystems with different inode semantics. If we support this, I think it should be explicit opt-in and documented as suitable only when the operator knows all requests are served from the same filesystem identity domain.

A configurable ETag strategy may be cleaner than a single include inode knob, if it fits the existing ETag override mechanism. For Nix specifically, a content-hash or store-path-aware strategy might be more semantically correct but inode is probably the cheaper practical option for single-server deployments.

[bb4242]

Thanks @steadytao! I think it's a great idea to make any potential inode based ETag behavior explicitly opt-in, and to clearly document when this is a suitable choice.

I also like the idea to make ETags even more generally configurable, including the possibility of a nix-specific strategy. At one point, there was a proposal in nixpkgs for such a thing, but it was never merged. I'm not sure whether maintaining nix specific behavior in caddy is in alignment with caddy's goals, which is why I proposed the inode based approach. But as far as I'm concerned, a nix specific solution would be ideal.

The Nix-store case is a real gap, mtimes of 1 are intentionally not meaningful so the current safeguard from #5548 avoids unsafe stale caching but leaves users without a useful validator.

In my understanding, this isn't quite the full story when considering the case of using caddy, in the default configuration, to serve content from the nix store. It's true that #5548 prevents caddy from serving ETag headers in this case, which is the safe option. However, golang's http stack still sets the Last-Modified header when mtime is 1. These Last-Modified headers are harmful for caching, since they never change. Unless the nix user knows they need to configure caddy to strip these headers (which isn't the default), unsafe stale caching will occur.

[steadytao]

Thanks, I did not fully notice that part or think through the Last-Modified side.

Looking at the file-server path, Caddy skips its own ETag when mtime.Unix() is 0 or 1 but it still passes info.ModTime() to http.ServeContent so the standard library can still emit a Last-Modified header with the Nix-store timestamp. That means the default Nix-store case can still have an unsafe stale validator even though Caddy avoids generating its own ETag.

So there may be two separate pieces here:

1. Caddy probably needs a way to suppress or avoid Last-Modified when the mtime is known meaningless, matching the intent of Consider dropping ETag, Last-Modified-Time generation for files where epoch <= mtime <= epoch+1 #5548 more completely.
2. Separately, Caddy could offer an explicit opt-in ETag strategy for deployments that want a useful validator anyway with inode-based being one possible cheap local-filesystem strategy.

I still think inode should be opt-in and documented carefully but I agree the current behaviour is worse than simply having no validator, it can still produce a misleading Last-Modified validator unless the user strips it manually, and with no clear documentation on the matter that likely is not the case.

[bb4242]

Thanks @steadytao! 🙂

Looking at the file-server path, Caddy skips its own ETag when mtime.Unix() is 0 or 1 but it still passes info.ModTime() to http.ServeContent so the standard library can still emit a Last-Modified header with the Nix-store timestamp. That means the default Nix-store case can still have an unsafe stale validator even though Caddy avoids generating its own ETag.

So there may be two separate pieces here:

1. Caddy probably needs a way to suppress or avoid `Last-Modified` when the mtime is known meaningless, matching the intent of [Consider dropping ETag, Last-Modified-Time generation for files where epoch <= mtime <= epoch+1 #5548](https://github.com/caddyserver/caddy/issues/5548) more completely.

Agreed, I think that would match the intended semantics of #5548. I took a quick shot at doing that here: #7740. That should at least fix the unsafe stale caching on nix for now, by forcing clients to re-fetch resources every time.

2. Separately, Caddy could offer an explicit opt-in ETag strategy for deployments that want a useful validator anyway with inode-based being one possible cheap local-filesystem strategy.

I still think inode should be opt-in and documented carefully but I agree the current behaviour is worse than simply having no validator, it can still produce a misleading Last-Modified validator unless the user strips it manually, and with no clear documentation on the matter that likely is not the case.

Great! I agree with all of this also. I'm happy to help with this in any way that would be useful to the project - it seems like there might be a bit more design work up front to decide which validators we want (i.e. inode based, nix store path based, any others?), but if we can decide on a good path forward, I'd be glad to help out with code. I'm also happy to leave it to others if that's more appropriate. Thanks for your collaboration on this, @steadytao!

[steadytao]

Regarding #7740, it seems like the right shape, just needs some final polish before merge. And in regards to the design work, we have established a fairly solid baseline thus far, I will leave this open for anyone who wishes to comment for some time before we decide on a base structure.