Skip to content

Commit 2cd39d4

Browse files
committed
Merge branch 'codex/fix-browser-ci-duration'
# Conflicts: # AGENTS.md # Tests/ManagedCode.Storage.Tests/Storages/Browser/BrowserServerStorageIntegrationTests.cs # Tests/ManagedCode.Storage.Tests/Storages/Browser/BrowserWasmStorageIntegrationTests.cs # docs/Development/setup.md # docs/Testing/strategy.md
2 parents d785e3d + 5b76d87 commit 2cd39d4

18 files changed

+330
-41
lines changed

.github/workflows/ci.yml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ jobs:
1313
build-and-test:
1414
name: build-and-test
1515
runs-on: ubuntu-latest
16+
timeout-minutes: 15
1617

1718
steps:
1819
- name: Checkout
@@ -47,3 +48,32 @@ jobs:
4748
token: ${{ secrets.CODECOV_TOKEN }}
4849
files: ./**/coverage.cobertura.xml
4950
fail_ci_if_error: false
51+
52+
browser-stress:
53+
name: browser-stress
54+
runs-on: ubuntu-latest
55+
timeout-minutes: 15
56+
57+
steps:
58+
- name: Checkout
59+
uses: actions/checkout@v5
60+
61+
- name: Setup .NET
62+
uses: actions/setup-dotnet@v4
63+
with:
64+
dotnet-version: ${{ env.DOTNET_VERSION }}
65+
66+
- name: Restore dependencies
67+
run: dotnet restore ManagedCode.Storage.slnx
68+
69+
- name: Restore .NET tools
70+
run: dotnet tool restore
71+
72+
- name: Build
73+
run: dotnet build ManagedCode.Storage.slnx --configuration Release --no-restore
74+
75+
- name: Install Playwright browser
76+
run: dotnet playwright -p Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj install chromium
77+
78+
- name: Browser stress test
79+
run: dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release --no-build --verbosity normal --filter "Category=BrowserStress"

.github/workflows/release.yml

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ jobs:
1212
build:
1313
name: Build and Test
1414
runs-on: ubuntu-latest
15+
timeout-minutes: 20
1516

1617
outputs:
1718
version: ${{ steps.version.outputs.version }}
@@ -57,9 +58,40 @@ jobs:
5758
path: ./artifacts/*.nupkg
5859
retention-days: 5
5960

61+
browser-stress:
62+
name: Browser Stress
63+
runs-on: ubuntu-latest
64+
timeout-minutes: 15
65+
66+
steps:
67+
- name: Checkout
68+
uses: actions/checkout@v5
69+
70+
- name: Setup .NET
71+
uses: actions/setup-dotnet@v4
72+
with:
73+
dotnet-version: ${{ env.DOTNET_VERSION }}
74+
75+
- name: Restore dependencies
76+
run: dotnet restore ManagedCode.Storage.slnx
77+
78+
- name: Restore .NET tools
79+
run: dotnet tool restore
80+
81+
- name: Build
82+
run: dotnet build ManagedCode.Storage.slnx --configuration Release --no-restore
83+
84+
- name: Install Playwright browser
85+
run: dotnet playwright -p Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj install chromium
86+
87+
- name: Browser stress test
88+
run: dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release --no-build --verbosity normal --filter "Category=BrowserStress"
89+
6090
publish-nuget:
6191
name: Publish to NuGet
62-
needs: build
92+
needs:
93+
- build
94+
- browser-stress
6395
runs-on: ubuntu-latest
6496
if: github.ref == 'refs/heads/main'
6597

AGENTS.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -136,8 +136,9 @@ If the stack is `.NET`, follow these skill-management rules explicitly:
136136

137137
- `restore`: `dotnet restore ManagedCode.Storage.slnx`
138138
- `build`: `dotnet build ManagedCode.Storage.slnx`
139-
- `test`: `dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release`
140-
- `coverage`: `dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release /p:CollectCoverage=true /p:CoverletOutput=coverage /p:CoverletOutputFormat=opencover`
139+
- `test`: `dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release --filter "Category!=BrowserStress"`
140+
- `browser-stress`: `dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release --filter "Category=BrowserStress"`
141+
- `coverage`: `dotnet test Tests/ManagedCode.Storage.Tests/ManagedCode.Storage.Tests.csproj --configuration Release --filter "Category!=BrowserStress" /p:CollectCoverage=true /p:CoverletOutput=coverage /p:CoverletOutputFormat=opencover`
141142
- `format`: `dotnet format ManagedCode.Storage.slnx`
142143

143144
Toolchain notes:
@@ -240,6 +241,7 @@ Toolchain notes:
240241
- Coverage uses the repo-defined `coverlet.msbuild` flow and must not regress without a written exception.
241242
- Place provider suites under `Tests/ManagedCode.Storage.Tests/Storages/` and reuse `Tests/ManagedCode.Storage.Tests/Common/` helpers for Testcontainers infrastructure such as Azurite, LocalStack, and FakeGcsServer.
242243
- For browser providers, put end-to-end Playwright coverage in `Tests/ManagedCode.Storage.Tests/Storages/Browser/` and keep the executable test hosts under `Tests/ManagedCode.Storage.BrowserServerHost/` and `Tests/ManagedCode.Storage.BrowserWasmHost/`.
244+
- Keep browser large-file verification tiered: the default `test` path keeps the fast browser large-file lane, while `browser-stress` runs the heavier explicit browser stress checks separately and must stay automated in CI and release.
243245

244246
### Storage Platform
245247

@@ -323,9 +325,11 @@ Ask first:
323325

324326
- Repository-facing docs, especially `README.md`, should stay in English and describe only the current supported behavior, not transitional legacy or fallback paths.
325327
- Temporary root-level `*.plan.md` files should be removed once a task is complete and their contents are no longer needed.
328+
- Browser large-file coverage should use tiered automation: a fast default CI lane plus an explicit stress lane, so merge confidence stays high without accepting 30+ minute default runs.
326329

327330
### Dislikes
328331

329332
- Template-generated scaffolding in tests; keep test hosts and verification surfaces minimal, hand-written, and purpose-built.
330333
- CI regressions that inflate `build-and-test` far beyond the historical baseline; browser large-file coverage must stay meaningful without turning the default GitHub Actions path into a 30+ minute run.
334+
- Disabling meaningful regression tests in default CI or release flows just to hide performance or flakiness problems; fix the runtime cost, or move them into a separate explicit required lane instead of silently dropping coverage.
331335
- Unnecessary product-code fallbacks; prefer one clear production path unless backward compatibility is an explicit requirement for the task.

Directory.Build.props

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@
2929
<RepositoryUrl>https://github.com/managedcode/Storage</RepositoryUrl>
3030
<PackageProjectUrl>https://github.com/managedcode/Storage</PackageProjectUrl>
3131
<Product>Managed Code - Storage</Product>
32-
<Version>10.0.4</Version>
33-
<PackageVersion>10.0.4</PackageVersion>
32+
<Version>10.0.5</Version>
33+
<PackageVersion>10.0.5</PackageVersion>
3434

3535
</PropertyGroup>
3636

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ Cloud storage vendors expose distinct SDKs, option models, and authentication pa
106106
- `ManagedCode.Storage.Client` brings streaming uploads/downloads, CRC32 helpers, and MIME discovery via `MimeHelper` to any .NET app.
107107
- Strongly typed option objects (`UploadOptions`, `DownloadOptions`, `DeleteOptions`, `MetadataOptions`, `LegalHoldOptions`, etc.) let you configure directories, metadata, and legal holds in one place.
108108
- Virtual File System package provides a file/directory API (`IVirtualFileSystem`) on top of the configured `IStorage` and can cache metadata for faster repeated operations, including browser storage verified through real Playwright flows in both Blazor WebAssembly and Interactive Server hosts.
109-
- Comprehensive automated test suite with cross-provider sync fixtures, multi-gigabyte streaming simulations (4 MB units per "GB"), ASP.NET controller harnesses, SFTP/local filesystem coverage, and Playwright browser verification for browser storage small-file overwrites, concurrent tabs, VFS flows, and `1 GiB` round-trips in both Interactive Server and Blazor WebAssembly hosts.
109+
- Comprehensive automated test suite with cross-provider sync fixtures, multi-gigabyte streaming simulations (4 MB units per "GB"), ASP.NET controller harnesses, SFTP/local filesystem coverage, and Playwright browser verification for browser storage small-file overwrites, concurrent tabs, VFS flows, a fast `128 MiB` browser large-file lane, and a separate `256 MiB` browser stress lane in both Interactive Server and Blazor WebAssembly hosts.
110110
- ManagedCode.Storage.TestFakes package plus Testcontainers-based fixtures make it easy to run offline or CI tests without touching real cloud accounts.
111111

112112
## Packages
@@ -979,7 +979,7 @@ If an MVC or Razor Pages application needs the packaged browser module path for
979979
980980
> Browser payloads use the browser Origin Private File System (OPFS). IndexedDB stays in the design only for blob metadata and list or lookup operations. If OPFS is unavailable in the current browser, uploads fail fast instead of silently falling back to a second payload backend.
981981
982-
> The real Playwright browser hosts in this repo verify small-file saves and overwrites, concurrent tabs, VFS flows, and `1 GiB` round-trips in both Blazor WebAssembly and Interactive Server. The large-file flows emit progress logs every `100 MiB`, and both the small-file and large-file paths assert that payload storage resolves to OPFS.
982+
> The real Playwright browser hosts in this repo verify small-file saves and overwrites, concurrent tabs, VFS flows, a default `128 MiB` large-file path, and a separate `256 MiB` browser stress lane in both Blazor WebAssembly and Interactive Server. The large-file flows emit progress logs every `100 MiB`, and both the small-file and large-file paths assert that payload storage resolves to OPFS.
983983
984984
</details>
985985

Storages/ManagedCode.Storage.Browser/wwwroot/browserStorage.js

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import {
33
appendPayloadChunksInternal,
44
beginPayloadWriteInternal,
55
completePayloadWriteInternal,
6+
getPayloadDigestInternal,
67
opfsFileExistsInternal,
78
readPayloadRangeInternal,
89
tryDeletePayloadFileInternal
@@ -209,6 +210,20 @@ export async function getPayloadStoreByFullName(databaseName, fullName) {
209210
return null;
210211
}
211212

213+
export async function getPayloadDigestByFullName(databaseName, fullName) {
214+
const blob = (await getAllBlobs(databaseName)).find((item) => item.fullName === fullName);
215+
if (!blob) {
216+
return null;
217+
}
218+
219+
const payloadKey = resolvePayloadKey(blob);
220+
if (!payloadKey) {
221+
return null;
222+
}
223+
224+
return await getPayloadDigestInternal(databaseName, payloadKey);
225+
}
226+
212227
const browserStorageApi = {
213228
containerExists,
214229
createContainer,
@@ -224,7 +239,8 @@ const browserStorageApi = {
224239
completePayloadWrite,
225240
abortPayloadWrite,
226241
readPayloadRange,
227-
getPayloadStoreByFullName
242+
getPayloadStoreByFullName,
243+
getPayloadDigestByFullName
228244
};
229245

230246
if (typeof window !== "undefined") {

Storages/ManagedCode.Storage.Browser/wwwroot/browserStorage.opfs.js

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,14 @@ export async function readPayloadRangeInternal(databaseName, blobKey, offset, co
100100
return result ? normalizeBytes(result) : null;
101101
}
102102

103+
export async function getPayloadDigestInternal(databaseName, blobKey) {
104+
if (!supportsOpfs()) {
105+
return null;
106+
}
107+
108+
return await postToOpfsWorker("getFileDigest", { databaseName, blobKey });
109+
}
110+
103111
export async function opfsFileExistsInternal(databaseName, blobKey) {
104112
if (!supportsOpfs()) {
105113
return false;

Storages/ManagedCode.Storage.Browser/wwwroot/browserStorage.worker.js

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
const sessions = new Map();
22
const maxWriteBlockBytes = 1024 * 1024;
3+
const maxDigestReadBlockBytes = 4 * 1024 * 1024;
4+
const crc32Table = buildCrc32Table();
35

46
self.onmessage = async (event) => {
57
const { id, command, payload } = event.data ?? {};
@@ -31,6 +33,8 @@ async function dispatchAsync(command, payload) {
3133
return true;
3234
case "readRange":
3335
return await readRangeAsync(payload.databaseName, payload.blobKey, payload.offset, payload.count);
36+
case "getFileDigest":
37+
return await getFileDigestAsync(payload.databaseName, payload.blobKey);
3438
case "deleteFile":
3539
return await deleteFileAsync(payload.databaseName, payload.blobKey);
3640
case "fileExists":
@@ -126,6 +130,41 @@ async function deleteFileAsync(databaseName, blobKey) {
126130
}
127131
}
128132

133+
async function getFileDigestAsync(databaseName, blobKey) {
134+
const fileHandle = await getBlobFileHandleAsync(databaseName, blobKey, false);
135+
const accessHandle = await fileHandle.createSyncAccessHandle();
136+
137+
try {
138+
const size = accessHandle.getSize();
139+
if (size <= 0) {
140+
return { length: 0, crc: 0 };
141+
}
142+
143+
const buffer = new Uint8Array(Math.min(maxDigestReadBlockBytes, size));
144+
let offset = 0;
145+
let crc = 0xffffffff;
146+
147+
while (offset < size) {
148+
const bytesToRead = Math.min(buffer.byteLength, size - offset);
149+
const bytesRead = normalizeBytesRead(
150+
accessHandle.read(buffer.subarray(0, bytesToRead), { at: offset }),
151+
bytesToRead,
152+
blobKey,
153+
offset);
154+
155+
crc = updateCrc32(crc, buffer, bytesRead);
156+
offset += bytesRead;
157+
}
158+
159+
return {
160+
length: Number(size),
161+
crc: completeCrc32(crc)
162+
};
163+
} finally {
164+
accessHandle.close();
165+
}
166+
}
167+
129168
async function fileExistsAsync(databaseName, blobKey) {
130169
try {
131170
await getBlobFileHandleAsync(databaseName, blobKey, false);
@@ -183,6 +222,19 @@ function normalizeBytesWritten(value, expectedBytes, blobKey, position) {
183222
return written;
184223
}
185224

225+
function normalizeBytesRead(value, expectedBytes, blobKey, position) {
226+
if (!Number.isFinite(value)) {
227+
throw new Error(`Invalid OPFS read result for ${blobKey} at ${position}: ${String(value)}.`);
228+
}
229+
230+
const bytesRead = Math.trunc(value);
231+
if (bytesRead <= 0 || bytesRead > expectedBytes) {
232+
throw new Error(`Invalid OPFS read result for ${blobKey} at ${position}: read ${bytesRead} of ${expectedBytes} bytes.`);
233+
}
234+
235+
return bytesRead;
236+
}
237+
186238
async function getDatabaseDirectoryAsync(databaseName, create) {
187239
const root = await navigator.storage.getDirectory();
188240
return await root.getDirectoryHandle(getDatabaseDirectoryName(databaseName), { create });
@@ -200,3 +252,33 @@ function getDatabaseDirectoryName(databaseName) {
200252
function getBlobFileName(blobKey) {
201253
return encodeURIComponent(blobKey);
202254
}
255+
256+
function buildCrc32Table() {
257+
const table = new Uint32Array(256);
258+
259+
for (let index = 0; index < table.length; index++) {
260+
let value = index;
261+
262+
for (let bit = 0; bit < 8; bit++) {
263+
value = (value & 1) === 0 ? value >>> 1 : (value >>> 1) ^ 0xedb88320;
264+
}
265+
266+
table[index] = value >>> 0;
267+
}
268+
269+
return table;
270+
}
271+
272+
function updateCrc32(crc, buffer, length) {
273+
let current = crc >>> 0;
274+
275+
for (let index = 0; index < length; index++) {
276+
current = (current >>> 8) ^ crc32Table[(current ^ buffer[index]) & 0xff];
277+
}
278+
279+
return current >>> 0;
280+
}
281+
282+
function completeCrc32(crc) {
283+
return (crc ^ 0xffffffff) >>> 0;
284+
}

Tests/ManagedCode.Storage.Tests/AGENTS.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ Parent: `../../AGENTS.md`
2525
## Project Commands
2626

2727
- `build`: `dotnet build ManagedCode.Storage.Tests.csproj`
28-
- `test`: `dotnet test ManagedCode.Storage.Tests.csproj --configuration Release`
29-
- `coverage`: `dotnet test ManagedCode.Storage.Tests.csproj --configuration Release /p:CollectCoverage=true /p:CoverletOutput=coverage /p:CoverletOutputFormat=opencover`
28+
- `test`: `dotnet test ManagedCode.Storage.Tests.csproj --configuration Release --filter "Category!=BrowserStress"`
29+
- `browser-stress`: `dotnet test ManagedCode.Storage.Tests.csproj --configuration Release --filter "Category=BrowserStress"`
30+
- `coverage`: `dotnet test ManagedCode.Storage.Tests.csproj --configuration Release --filter "Category!=BrowserStress" /p:CollectCoverage=true /p:CoverletOutput=coverage /p:CoverletOutputFormat=opencover`
3031
- `format`: `dotnet format ../../ManagedCode.Storage.slnx`
3132
- Active test framework: `xUnit`
3233
- Runner model: `VSTest`
@@ -54,3 +55,4 @@ Parent: `../../AGENTS.md`
5455
- Every new public behavior needs integration coverage that asserts observable outcomes, not only successful method calls.
5556
- Keep architecture dependency rules in `Architecture/` focused on durable package boundaries so they stay stable as implementation details move.
5657
- Do not weaken assertions or skip suites to get a green run; fix the real regression or document an explicit exception.
58+
- Browser stress tests are an explicit lane, not hidden debt: keep them automated via the dedicated `browser-stress` command or workflow instead of folding them into the fast default test path.
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
using Xunit;
2+
3+
namespace ManagedCode.Storage.Tests.Storages.Browser;
4+
5+
[CollectionDefinition(Name, DisableParallelization = true)]
6+
public sealed class BrowserIntegrationCollection : ICollectionFixture<BrowserServerHostFixture>, ICollectionFixture<BrowserWasmHostFixture>
7+
{
8+
public const string Name = nameof(BrowserIntegrationCollection);
9+
}

0 commit comments

Comments
 (0)