Skip to content

Commit 7021008

Browse files
NucsNucs
authored
Merge pull request #609 from SciSharp/worktree-mstests
[Tests] Migrate from TUnit to MSTest 3 for LLM Compatibility
2 parents 8da286e + 299a4b4 commit 7021008

292 files changed

Lines changed: 9212 additions & 9165 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/CLAUDE.md‎

Lines changed: 34 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -345,8 +345,7 @@ dotnet build -v q --nologo "-clp:NoSummary;ErrorsOnly" -p:WarningLevel=0
345345

346346
### Running Tests
347347

348-
Tests use **TUnit** framework with source-generated test discovery.
349-
- `dotnet_test_tunit --filter "..."`: MSTest-style filter for TUnit (Category=, Name~, ClassName~, FullyQualifiedName~)
348+
Tests use **MSTest v3** framework with source-generated test discovery.
350349

351350
```bash
352351
# Run from test directory
@@ -356,10 +355,19 @@ cd test/NumSharp.UnitTest
356355
dotnet test --no-build
357356

358357
# Exclude OpenBugs (CI-style - only real failures)
359-
dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category!=OpenBugs]"
358+
dotnet test --no-build --filter "TestCategory!=OpenBugs"
360359

361360
# Run ONLY OpenBugs tests
362-
dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
361+
dotnet test --no-build --filter "TestCategory=OpenBugs"
362+
363+
# Exclude multiple categories
364+
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"
365+
366+
# Run specific test class
367+
dotnet test --no-build --filter "ClassName~BinaryOpTests"
368+
369+
# Run specific test method
370+
dotnet test --no-build --filter "Name~Add_Int32_SameType"
363371
```
364372

365373
### Output Formatting
@@ -369,10 +377,10 @@ dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
369377
dotnet test --no-build 2>&1 | grep -E "^(failed|skipped|Test run| total:| failed:| succeeded:| skipped:| duration:)"
370378

371379
# Results with messages (no stack traces)
372-
dotnet test --no-build 2>&1 | grep -v "^ at " | grep -v "^ at " | grep -v "^ ---" | grep -v "^ from K:" | sed 's/TUnit.Engine.Exceptions.TestFailedException: //' | sed 's/AssertFailedException: //'
380+
dotnet test --no-build 2>&1 | grep -v "^ at " | grep -v "^ at " | grep -v "^ ---" | grep -v "^ from K:" | sed 's/AssertFailedException: //'
373381

374-
# Detailed output (shows passed tests too)
375-
dotnet test --no-build -- --output Detailed
382+
# Verbose output (shows passed tests too)
383+
dotnet test --no-build -v normal
376384
```
377385

378386
## Test Categories
@@ -383,36 +391,38 @@ Tests use typed category attributes defined in `TestCategory.cs`. Adding new bug
383391
|----------|-----------|---------|-------------|
384392
| `OpenBugs` | `[OpenBugs]` | Known-failing bug reproductions. Remove when fixed. | **EXCLUDED** via filter |
385393
| `Misaligned` | `[Misaligned]` | Documents NumSharp vs NumPy behavioral differences. | Runs (tests pass) |
386-
| `WindowsOnly` | `[WindowsOnly]` | Requires GDI+/System.Drawing.Common | Runtime platform check |
394+
| `WindowsOnly` | `[WindowsOnly]` | Requires GDI+/System.Drawing.Common | Excluded on non-Windows |
395+
| `HighMemory` | `[HighMemory]` | Requires 8GB+ RAM | **EXCLUDED** via filter |
387396

388-
### How CI Excludes OpenBugs
397+
### How CI Excludes Categories
389398

390-
The CI pipeline (`.github/workflows/build-and-release.yml`) uses TUnit's `--treenode-filter` to exclude `OpenBugs`:
399+
The CI pipeline (`.github/workflows/build-and-release.yml`) uses MSTest's `--filter` to exclude categories:
391400

392401
```yaml
393402
- name: Test (net10.0)
394403
run: |
395-
dotnet run --project test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
404+
dotnet test test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
396405
--configuration Release --no-build --framework net10.0 \
397-
-- --treenode-filter '/*/*/*/*[Category!=OpenBugs]'
406+
--filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"
398407
```
399408
400-
This filter excludes all tests with `[OpenBugs]` attribute from CI runs. Tests pass locally when the bug is fixed — then remove the `[OpenBugs]` attribute.
409+
This filter excludes all tests with `[OpenBugs]` or `[HighMemory]` attributes from CI runs. Tests pass locally when the bug is fixed — then remove the `[OpenBugs]` attribute.
401410

402411
### Usage
403412

404413
```csharp
405414
// Class-level (all tests in class)
415+
[TestClass]
406416
[OpenBugs]
407417
public class BroadcastBugTests { ... }
408418
409419
// Method-level
410-
[Test]
420+
[TestMethod]
411421
[OpenBugs]
412-
public async Task BroadcastWriteCorruptsData() { ... }
422+
public void BroadcastWriteCorruptsData() { ... }
413423
414424
// Documenting behavioral differences (NOT excluded from CI)
415-
[Test]
425+
[TestMethod]
416426
[Misaligned]
417427
public void BroadcastSlice_MaterializesInNumSharp() { ... }
418428
```
@@ -421,13 +431,16 @@ public void BroadcastSlice_MaterializesInNumSharp() { ... }
421431

422432
```bash
423433
# Exclude OpenBugs (same as CI)
424-
dotnet test -- --treenode-filter "/*/*/*/*[Category!=OpenBugs]"
434+
dotnet test --filter "TestCategory!=OpenBugs"
425435
426436
# Run ONLY OpenBugs tests (to verify fixes)
427-
dotnet test -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
437+
dotnet test --filter "TestCategory=OpenBugs"
428438
429439
# Run ONLY Misaligned tests
430-
dotnet test -- --treenode-filter "/*/*/*/*[Category=Misaligned]"
440+
dotnet test --filter "TestCategory=Misaligned"
441+
442+
# Combine multiple exclusions
443+
dotnet test --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory&TestCategory!=WindowsOnly"
431444
```
432445

433446
**OpenBugs files**: `OpenBugs.cs` (general), `OpenBugs.Bitmap.cs` (bitmap), `OpenBugs.ApiAudit.cs` (API audit), `OpenBugs.ILKernelBattle.cs` (IL kernel).
@@ -599,10 +612,10 @@ A: Core ops (`dot`, `matmul`) in `LinearAlgebra/`. Advanced decompositions (`inv
599612
## Q&A - Development
600613

601614
**Q: What's in the test suite?**
602-
A: TUnit framework in `test/NumSharp.UnitTest/`. Many tests adapted from NumPy's own test suite. Decent coverage but gaps in edge cases. Uses source-generated test discovery (no special flags needed).
615+
A: MSTest v3 framework in `test/NumSharp.UnitTest/`. Many tests adapted from NumPy's own test suite. Decent coverage but gaps in edge cases. Uses source-generated test discovery (no special flags needed).
603616

604617
**Q: What .NET version is targeted?**
605-
A: Library multi-targets `net8.0` and `net10.0`. Tests require .NET 9+ runtime (TUnit requirement).
618+
A: Library multi-targets `net8.0` and `net10.0`. Tests also multi-target both frameworks.
606619

607620
**Q: What are the main dependencies?**
608621
A: No external runtime dependencies. `System.Memory` and `System.Runtime.CompilerServices.Unsafe` (previously NuGet packages) are built into the .NET 8+ runtime.

‎.claude/skills/np-tests/SKILL.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ To interact/develop/create tests for np.* functions, high-level development cycl
1212
Definition of Done:
1313
- At the end of this step you understand 100% what numpy tests: inputs, outputs, edge cases, error conditions, dtype behaviors.
1414
- You have identified all test files and test methods related to your function.
15-
2. Migrate numpy's tests to C# following TUnit framework patterns in test/NumSharp.UnitTest. Match numpy's test structure and assertions exactly.
15+
2. Migrate numpy's tests to C# following MSTest v3 framework patterns in test/NumSharp.UnitTest. Match numpy's test structure and assertions exactly.
1616
Definition of Done:
1717
- Every numpy test case has a corresponding C# test.
1818
- We cover all dtypes NumSharp supports (Boolean, Byte, Int16, UInt16, Int32, UInt32, Int64, UInt64, Char, Single, Double, Decimal).
@@ -32,8 +32,8 @@ Use battletesting to validate behavior matches numpy: 'dotnet run << 'EOF'' and
3232

3333
### Test Patterns
3434
```csharp
35-
[Test]
36-
public async Task FunctionName_Scenario_Dtype()
35+
[TestMethod]
36+
public void FunctionName_Scenario_Dtype()
3737
{
3838
// Arrange
3939
var input = np.array(new[] { 3, 1, 2 });
@@ -42,8 +42,8 @@ public async Task FunctionName_Scenario_Dtype()
4242
var result = np.sort(input);
4343

4444
// Assert - values from running actual numpy
45-
Assert.That(result.IsContiguous, Is.True);
46-
Assert.That(result.GetAtIndex<int>(0), Is.EqualTo(1));
45+
Assert.IsTrue(result.IsContiguous);
46+
Assert.AreEqual(1, result.GetAtIndex<int>(0));
4747
}
4848
```
4949

‎.git-blame-ignore-revs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# TUnit to MSTest v3 attribute migration
2+
ac020336
3+
4+
# TUnit assertions to AwesomeAssertions migration
5+
b1d1d543
6+
7+
# Add [TestClass] attributes for MSTest discovery
8+
e0db3c3e
9+
10+
# Convert async Task to void for sync test methods
11+
4eea9644

‎.github/workflows/build-and-release.yml‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -43,26 +43,39 @@ jobs:
4343
- name: Build
4444
run: dotnet build test/NumSharp.UnitTest/NumSharp.UnitTest.csproj --configuration Release -p:NoWarn=${{ env.DOTNET_NOWARN }}
4545

46-
# NOTE: Test project currently only targets net10.0. See TODO in NumSharp.UnitTest.csproj
47-
# to re-enable net8.0 when TUnit compatibility is resolved.
48-
#
4946
# Test filtering:
5047
# - OpenBugs: excluded (known-failing bug reproductions)
5148
# - HighMemory: excluded (requires 8GB+ RAM, too much for CI runners)
52-
# - WindowsOnly: auto-skipped at runtime via [SkipOnNonWindows] attribute
49+
# - WindowsOnly: excluded on non-Windows runners
50+
- name: Test (net8.0)
51+
shell: bash
52+
timeout-minutes: 10
53+
run: |
54+
echo "Starting test run (net8.0)..."
55+
echo "dotnet version: $(dotnet --version)"
56+
echo "Available memory: $(free -h 2>/dev/null || echo 'N/A')"
57+
FILTER="TestCategory!=OpenBugs&TestCategory!=HighMemory"
58+
if [[ "$RUNNER_OS" != "Windows" ]]; then
59+
FILTER="$FILTER&TestCategory!=WindowsOnly"
60+
fi
61+
dotnet test test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
62+
--configuration Release --no-build --framework net8.0 \
63+
--filter "$FILTER" --logger "trx"
64+
5365
- name: Test (net10.0)
5466
shell: bash
5567
timeout-minutes: 10
5668
run: |
57-
echo "Starting test run..."
69+
echo "Starting test run (net10.0)..."
5870
echo "dotnet version: $(dotnet --version)"
5971
echo "Available memory: $(free -h 2>/dev/null || echo 'N/A')"
60-
# Note: TUnit --treenode-filter doesn't seem to exclude class-level categories.
61-
# The HighMemory filter was attempted but didn't reduce test count.
62-
# Ubuntu failures are allowed via continue-on-error while investigating.
63-
dotnet run --project test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
72+
FILTER="TestCategory!=OpenBugs&TestCategory!=HighMemory"
73+
if [[ "$RUNNER_OS" != "Windows" ]]; then
74+
FILTER="$FILTER&TestCategory!=WindowsOnly"
75+
fi
76+
dotnet test test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
6477
--configuration Release --no-build --framework net10.0 \
65-
-- --treenode-filter '/*/*/*/*[Category!=OpenBugs]' --report-trx
78+
--filter "$FILTER" --logger "trx"
6679
6780
- name: Upload Test Results
6881
uses: actions/upload-artifact@v4

‎docs/MSTEST_FILTER_GUIDE.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# MSTest `--filter` Guide
2+
3+
## Filter Syntax
4+
5+
MSTest uses a simple property-based filter syntax:
6+
7+
```
8+
--filter "Property=Value"
9+
--filter "Property!=Value"
10+
--filter "Property~Value" # Contains
11+
--filter "Property!~Value" # Does not contain
12+
```
13+
14+
Combine with `&` (AND) or `|` (OR):
15+
```
16+
--filter "Property1=A&Property2=B" # AND
17+
--filter "Property1=A|Property2=B" # OR
18+
```
19+
20+
## Available Properties
21+
22+
| Property | Description | Example |
23+
|----------|-------------|---------|
24+
| `TestCategory` | Category attribute | `TestCategory=OpenBugs` |
25+
| `ClassName` | Test class name | `ClassName~BinaryOpTests` |
26+
| `Name` | Test method name | `Name~Add_Int32` |
27+
| `FullyQualifiedName` | Full namespace.class.method | `FullyQualifiedName~Backends.Kernels` |
28+
29+
## 5 Concrete Examples
30+
31+
### 1. Exclude OpenBugs (CI-style run)
32+
33+
```bash
34+
dotnet test --no-build --filter "TestCategory!=OpenBugs"
35+
```
36+
37+
Runs all tests EXCEPT those marked `[OpenBugs]`. This is what CI uses.
38+
39+
### 2. Run ONLY OpenBugs (verify bug fixes)
40+
41+
```bash
42+
dotnet test --no-build --filter "TestCategory=OpenBugs"
43+
```
44+
45+
Runs only failing bug reproductions to check if your fix works.
46+
47+
### 3. Run single test class
48+
49+
```bash
50+
dotnet test --no-build --filter "ClassName~CountNonzeroTests"
51+
```
52+
53+
Runs all tests in classes containing `CountNonzeroTests`.
54+
55+
### 4. Run single test method
56+
57+
```bash
58+
dotnet test --no-build --filter "Name=Add_TwoNumbers_ReturnsSum"
59+
```
60+
61+
Runs only the test named exactly `Add_TwoNumbers_ReturnsSum`.
62+
63+
### 5. Run tests by namespace pattern
64+
65+
```bash
66+
dotnet test --no-build --filter "FullyQualifiedName~Backends.Kernels"
67+
```
68+
69+
Runs all tests in the `Backends.Kernels` namespace.
70+
71+
## Quick Reference
72+
73+
| Goal | Filter |
74+
|------|--------|
75+
| Exclude category | `TestCategory!=OpenBugs` |
76+
| Include category only | `TestCategory=OpenBugs` |
77+
| Single class (exact) | `ClassName=BinaryOpTests` |
78+
| Class contains | `ClassName~BinaryOp` |
79+
| Method contains | `Name~Add_` |
80+
| Namespace contains | `FullyQualifiedName~Backends.Kernels` |
81+
| Multiple categories (AND) | `TestCategory!=OpenBugs&TestCategory!=WindowsOnly` |
82+
| Multiple categories (OR) | `TestCategory=OpenBugs\|TestCategory=Misaligned` |
83+
84+
## Operators
85+
86+
| Op | Meaning | Example |
87+
|----|---------|---------|
88+
| `=` | Equals | `TestCategory=Unit` |
89+
| `!=` | Not equals | `TestCategory!=Slow` |
90+
| `~` | Contains | `Name~Integration` |
91+
| `!~` | Does not contain | `ClassName!~Legacy` |
92+
| `&` | AND | `TestCategory!=A&TestCategory!=B` |
93+
| `\|` | OR | `TestCategory=A\|TestCategory=B` |
94+
95+
**Important:**
96+
- Use `\|` (escaped pipe) for OR in bash
97+
- Parentheses are NOT needed for combining filters
98+
- Filter values are case-sensitive
99+
100+
## NumSharp Categories
101+
102+
| Category | Purpose | CI Behavior |
103+
|----------|---------|-------------|
104+
| `OpenBugs` | Known-failing bug reproductions | **Excluded** |
105+
| `HighMemory` | Requires 8GB+ RAM | **Excluded** |
106+
| `Misaligned` | NumSharp vs NumPy differences (tests pass) | Runs |
107+
| `WindowsOnly` | Requires GDI+/System.Drawing | Excluded on Linux/macOS |
108+
| `LongIndexing` | Tests > int.MaxValue elements | Runs |
109+
110+
## Useful Commands
111+
112+
```bash
113+
# Stop on first failure (MSTest v3)
114+
dotnet test --no-build -- --fail-on-failure
115+
116+
# Verbose output (see passed tests too)
117+
dotnet test --no-build -v normal
118+
119+
# List tests without running
120+
dotnet test --no-build --list-tests
121+
122+
# CI-style: exclude OpenBugs and HighMemory
123+
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"
124+
125+
# Windows CI: full exclusion list
126+
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"
127+
128+
# Linux/macOS CI: also exclude WindowsOnly
129+
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory&TestCategory!=WindowsOnly"
130+
```
131+
132+
## Advanced Filter Examples
133+
134+
### Example A: Specific Test Method
135+
136+
```bash
137+
dotnet test --no-build --filter "FullyQualifiedName=NumSharp.UnitTest.Backends.Kernels.VarStdComprehensiveTests.Var_2D_Axis0"
138+
```
139+
140+
**Result:** 1 test
141+
142+
### Example B: Pattern Matching Multiple Classes
143+
144+
```bash
145+
dotnet test --no-build --filter "ClassName~Comprehensive&Name~_2D_"
146+
```
147+
148+
**Result:** Tests in `*Comprehensive*` classes with `_2D_` in method name
149+
150+
### Example C: Namespace + Category Filter
151+
152+
```bash
153+
dotnet test --no-build --filter "FullyQualifiedName~Backends.Kernels&TestCategory!=OpenBugs"
154+
```
155+
156+
**Result:** All Kernels tests except OpenBugs
157+
158+
### Example D: Multiple Categories (OR)
159+
160+
```bash
161+
dotnet test --no-build --filter "TestCategory=OpenBugs|TestCategory=Misaligned"
162+
```
163+
164+
**Result:** Tests that have EITHER `[OpenBugs]` OR `[Misaligned]` attribute
165+
166+
## Migration from TUnit
167+
168+
| TUnit Filter | MSTest Filter |
169+
|--------------|---------------|
170+
| `--treenode-filter "/*/*/*/*[Category!=X]"` | `--filter "TestCategory!=X"` |
171+
| `--treenode-filter "/*/*/ClassName/*"` | `--filter "ClassName~ClassName"` |
172+
| `--treenode-filter "/*/*/*/MethodName"` | `--filter "Name=MethodName"` |
173+
| `--treenode-filter "/*/Namespace/*/*"` | `--filter "FullyQualifiedName~Namespace"` |

0 commit comments

Comments
 (0)