Problem
On Windows workers using the native build directory, bb_worker hardlinks cached CAS blobs from the local cache into each action's input root (hardlinkingFileFetcher). NTFS caps a single file at 1023 hard links. A hot shared blob — e.g. a common toolchain input needed by many actions — gets hardlinked into more than 1023 input roots and CreateHardLink fails with ERROR_TOO_MANY_LINKS (0x476). buildbarn wraps this as codes.Internal, so the action aborts mid-build:
Failed to create hardlink to cached file "...":
An attempt was made to create more links on a file than the file system supports.
This is reproducible with a fan-out build (many actions sharing one large toolchain input) on a worker whose C:\worker cache/build directories are on NTFS.
Why native (and not the virtual filesystem)
The virtual build directory avoids hardlinks, but on Windows its backend is WinFSP, which requires installing the WinFSP kernel-mode driver on the worker host. On managed Kubernetes (e.g. AKS Windows node pools) you cannot install a kernel driver on the node, and the worker base image (nanoserver) has no driver. So native is the only viable build directory for these deployments, and the link cap is a hard wall there.
Proposal
When the hardlink fails because the file has reached the filesystem's maximum link count (ERROR_TOO_MANY_LINKS on Windows, EMLINK on POSIX), fall back to copying the cached file into the input root instead of failing. A copy is an independent inode with its own link count, so materialization succeeds; only the few over-limit hot blobs pay the copy cost, and correctness is unchanged because the input root only needs the file's contents.
Status
I have a working patch (small, single-module — no bb-storage change needed), with a unit test, that passes bazel test //pkg/cas:cas_test, the windows_amd64 cross-build, golint, and the reformat/gofmt gate locally. Happy to open a PR if you're open to this approach.
Would you accept this, or do you see a preferred way to handle the link cap on Windows native workers?
Problem
On Windows workers using the
nativebuild directory,bb_workerhardlinks cached CAS blobs from the local cache into each action's input root (hardlinkingFileFetcher). NTFS caps a single file at 1023 hard links. A hot shared blob — e.g. a common toolchain input needed by many actions — gets hardlinked into more than 1023 input roots andCreateHardLinkfails withERROR_TOO_MANY_LINKS(0x476). buildbarn wraps this ascodes.Internal, so the action aborts mid-build:This is reproducible with a fan-out build (many actions sharing one large toolchain input) on a worker whose
C:\workercache/build directories are on NTFS.Why
native(and not the virtual filesystem)The
virtualbuild directory avoids hardlinks, but on Windows its backend is WinFSP, which requires installing the WinFSP kernel-mode driver on the worker host. On managed Kubernetes (e.g. AKS Windows node pools) you cannot install a kernel driver on the node, and the worker base image (nanoserver) has no driver. Sonativeis the only viable build directory for these deployments, and the link cap is a hard wall there.Proposal
When the hardlink fails because the file has reached the filesystem's maximum link count (
ERROR_TOO_MANY_LINKSon Windows,EMLINKon POSIX), fall back to copying the cached file into the input root instead of failing. A copy is an independent inode with its own link count, so materialization succeeds; only the few over-limit hot blobs pay the copy cost, and correctness is unchanged because the input root only needs the file's contents.Status
I have a working patch (small, single-module — no
bb-storagechange needed), with a unit test, that passesbazel test //pkg/cas:cas_test, thewindows_amd64cross-build, golint, and the reformat/gofmt gate locally. Happy to open a PR if you're open to this approach.Would you accept this, or do you see a preferred way to handle the link cap on Windows
nativeworkers?