Skip to content

Commit 968b00e

Browse files
aaronziCopilot
aaronzi
and
Copilot
authored
Makes integration tests independent of each other (#534)
* Improves integration tests to be independent of each other Co-authored-by: Copilot <copilot@github.com> * Improves integration tests for remaining components Co-authored-by: Copilot <copilot@github.com> * Fixes ts errors in the sm repo tests Co-authored-by: Copilot <copilot@github.com> * adresses review remarks Co-authored-by: Copilot <copilot@github.com> * Fixes further remarks --------- Co-authored-by: Copilot <copilot@github.com>
1 parent 1245b8c commit 968b00e

9 files changed

Lines changed: 2379 additions & 916 deletions

‎.github/workflows/test-sdk.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,9 @@ jobs:
5858
- name: Install Dependencies
5959
run: pnpm install --frozen-lockfile
6060

61+
- name: Type-check Integration Tests
62+
run: pnpm exec tsc -p src/integration-tests/tsconfig.json --noEmit
63+
6164
- name: Pull Docker Images
6265
run: docker-compose -f ci/docker-compose.yml pull
6366

‎README.md‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,25 +105,25 @@ pnpm test:engine:list-components
105105
Run against one component endpoint:
106106

107107
```bash
108-
pnpm test:engine -- --component submodel-repository --url http://localhost:8082
108+
pnpm test:engine --component submodel-repository --url http://localhost:8082
109109
```
110110

111111
Select output formats (`console`, `json`, `junit`, `markdown`):
112112

113113
```bash
114-
pnpm test:engine -- --component aas-repository --url http://localhost:8081 --report console,json,junit,markdown
114+
pnpm test:engine --component aas-repository --url http://localhost:8081 --report console,json,junit,markdown
115115
```
116116

117117
Strict known-issues mode (waived known issues become failures):
118118

119119
```bash
120-
pnpm test:engine -- --component aas-discovery --url http://localhost:8086 --strict-known-issues
120+
pnpm test:engine --component aas-discovery --url http://localhost:8086 --strict-known-issues
121121
```
122122

123123
Use a custom artifact directory:
124124

125125
```bash
126-
pnpm test:engine -- --component aasx-file-server --url http://localhost:8087 --report-dir coverage/test-engine
126+
pnpm test:engine --component aasx-file-server --url http://localhost:8087 --report-dir coverage/test-engine
127127
```
128128

129129
### Report Behavior
@@ -139,6 +139,24 @@ Exit code behavior:
139139
- non-zero if any integration check fails or OpenAPI critical findings exist.
140140
- in strict mode, waived known issues also fail the run.
141141

142+
### Known Backend Deviations (Integration)
143+
144+
The integration suites include some checks that are currently waived or adapted due to backend behavior that does not
145+
fully match expected API behavior.
146+
147+
Observed and tracked:
148+
149+
- `submodel-repository` `GetAllSubmodels-Path` can return an empty `result` array even when a fresh submodel has been
150+
created in the same run. The test now validates success and response shape, but does not require `result.length > 0`.
151+
- `submodel-repository` value-only cursor not-found behavior is tracked as `known-specification-bug`.
152+
153+
Maintenance rule for new findings:
154+
155+
- If a backend deviation is confirmed, annotate the affected test with a short reason (`known-backend-bug` or
156+
`known-specification-bug`), and add a short note in this section.
157+
- Prefer narrowing assertions to contract-safe checks (status and shape) instead of asserting unstable payload details.
158+
- Keep strict mode available (`--strict-known-issues`) so waived issues are still visible in CI when needed.
159+
142160
## Import Styles
143161

144162
The SDK supports multiple import styles for flexibility:

‎src/integration-tests/aasDiscovery.integration.test.ts‎

Lines changed: 99 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -2,41 +2,89 @@ import { SpecificAssetId as CoreSpecificAssetId } from '@aas-core-works/aas-core
22
import { AasDiscoveryClient } from '../clients/AasDiscoveryClient';
33
import { Configuration } from '../generated';
44
import { base64Encode } from '../lib/base64Url';
5-
import {
6-
createTestShell,
7-
createTestSpecificAssetId1,
8-
createTestSpecificAssetId2,
9-
} from './fixtures/aasDiscoveryFixtures';
5+
import { createTestSpecificAssetId1, createTestSpecificAssetId2 } from './fixtures/aasDiscoveryFixtures';
6+
import { createPerTestCleanupRunner } from './fixtures/testCleanup';
107
import { getIntegrationBasePath } from './testEngineConfig';
118

129
describe('AAS Discovery Integration Tests', () => {
1310
const client = new AasDiscoveryClient();
14-
const testShell = createTestShell();
15-
const testSpecificAssetId1 = createTestSpecificAssetId1();
16-
const testSpecificAssetId2 = createTestSpecificAssetId2();
1711
const configuration = new Configuration({
1812
basePath: getIntegrationBasePath('aasDiscovery'),
1913
});
14+
const { track } = createPerTestCleanupRunner();
15+
2016
const uniqueSuffix = (): string => `${Date.now()}-${Math.random().toString(36).slice(2)}`;
2117
const unavailableCursor = (): string =>
2218
base64Encode(`https://example.com/ids/non-existing-cursor-${uniqueSuffix()}`);
2319

20+
const createUniqueShellId = (): string => `https://example.com/ids/aas/discovery-${uniqueSuffix()}`;
21+
const createUniqueSpecificAssetId1 = () => {
22+
const specificAssetId = createTestSpecificAssetId1();
23+
specificAssetId.value = `https://example.com/ids/asset/discovery-1-${uniqueSuffix()}`;
24+
return specificAssetId;
25+
};
26+
const createUniqueSpecificAssetId2 = () => {
27+
const specificAssetId = createTestSpecificAssetId2();
28+
specificAssetId.value = `https://example.com/ids/asset/discovery-2-${uniqueSuffix()}`;
29+
return specificAssetId;
30+
};
31+
32+
const registerAssetLinkCleanup = (aasIdentifier: string) => {
33+
track(async () => {
34+
const cleanupResponse = await client.deleteAllAssetLinksById({
35+
configuration,
36+
aasIdentifier,
37+
});
38+
39+
if (!cleanupResponse.success && cleanupResponse.statusCode !== 404) {
40+
throw new Error(
41+
`Failed to cleanup asset links for ${aasIdentifier} (status ${cleanupResponse.statusCode})`
42+
);
43+
}
44+
});
45+
};
46+
47+
const seedAssetLinks = async () => {
48+
const aasIdentifier = createUniqueShellId();
49+
const specificAssetId1 = createUniqueSpecificAssetId1();
50+
const specificAssetId2 = createUniqueSpecificAssetId2();
51+
52+
const createResponse = await client.postAllAssetLinksById({
53+
configuration,
54+
aasIdentifier,
55+
specificAssetId: [specificAssetId1, specificAssetId2],
56+
});
57+
58+
expect(createResponse.success).toBe(true);
59+
if (!createResponse.success) {
60+
throw new Error('Failed to seed AAS discovery asset links fixture');
61+
}
62+
63+
registerAssetLinkCleanup(aasIdentifier);
64+
return { aasIdentifier, specificAssetId1, specificAssetId2 };
65+
};
66+
2467
/**
2568
* @operation PostAllAssetLinksById
2669
* @status 201
2770
*/
2871
test('should create specific asset identifiers linked to an Asset Administration Shell', async () => {
72+
const aasIdentifier = createUniqueShellId();
73+
const specificAssetId1 = createUniqueSpecificAssetId1();
74+
const specificAssetId2 = createUniqueSpecificAssetId2();
75+
2976
const response = await client.postAllAssetLinksById({
3077
configuration,
31-
aasIdentifier: testShell.id,
32-
specificAssetId: [testSpecificAssetId1, testSpecificAssetId2],
78+
aasIdentifier,
79+
specificAssetId: [specificAssetId1, specificAssetId2],
3380
});
3481

3582
expect(response.success).toBe(true);
3683
if (response.success) {
84+
registerAssetLinkCleanup(aasIdentifier);
3785
expect(response.statusCode).toBe(201);
3886
expect(response.data).toBeDefined();
39-
expect(response.data).toEqual([testSpecificAssetId1, testSpecificAssetId2]);
87+
expect(response.data).toEqual([specificAssetId1, specificAssetId2]);
4088
}
4189
});
4290

@@ -45,16 +93,18 @@ describe('AAS Discovery Integration Tests', () => {
4593
* @status 200
4694
*/
4795
test('should fetch a list of specific asset identifiers based on an Asset Administration Shell ID', async () => {
96+
const { aasIdentifier, specificAssetId1, specificAssetId2 } = await seedAssetLinks();
97+
4898
const response = await client.getAllAssetLinksById({
4999
configuration,
50-
aasIdentifier: testShell.id,
100+
aasIdentifier,
51101
});
52102

53103
expect(response.success).toBe(true);
54104
if (response.success) {
55105
expect(response.statusCode).toBe(200);
56106
expect(response.data).toBeDefined();
57-
expect(response.data).toEqual([testSpecificAssetId1, testSpecificAssetId2]);
107+
expect(response.data).toEqual([specificAssetId1, specificAssetId2]);
58108
}
59109
});
60110

@@ -96,17 +146,19 @@ describe('AAS Discovery Integration Tests', () => {
96146
* @status 200
97147
*/
98148
test('should fetch a list of Asset Administration Shell IDs', async () => {
149+
const { aasIdentifier, specificAssetId1, specificAssetId2 } = await seedAssetLinks();
150+
99151
const response = await client.getAllAssetAdministrationShellIdsByAssetLink({
100152
configuration,
101-
assetIds: [testSpecificAssetId1, testSpecificAssetId2],
153+
assetIds: [specificAssetId1, specificAssetId2],
102154
});
103155

104156
expect(response.success).toBe(true);
105157
if (response.success) {
106158
expect(response.statusCode).toBe(200);
107159
expect(response.data).toBeDefined();
108160
expect(response.data.result.length).toBeGreaterThan(0);
109-
expect(response.data.result).toContainEqual(testShell.id);
161+
expect(response.data.result).toContainEqual(aasIdentifier);
110162
}
111163
});
112164

@@ -115,9 +167,12 @@ describe('AAS Discovery Integration Tests', () => {
115167
* @status 400
116168
*/
117169
test('should reject invalid AAS discovery lookup limit with bad request', async () => {
170+
const specificAssetId1 = createUniqueSpecificAssetId1();
171+
const specificAssetId2 = createUniqueSpecificAssetId2();
172+
118173
const response = await client.getAllAssetAdministrationShellIdsByAssetLink({
119174
configuration,
120-
assetIds: [testSpecificAssetId1, testSpecificAssetId2],
175+
assetIds: [specificAssetId1, specificAssetId2],
121176
limit: -1,
122177
});
123178

@@ -133,9 +188,12 @@ describe('AAS Discovery Integration Tests', () => {
133188
* @status 200
134189
*/
135190
test('should return an empty AAS discovery lookup page for an unavailable cursor', async () => {
191+
const specificAssetId1 = createUniqueSpecificAssetId1();
192+
const specificAssetId2 = createUniqueSpecificAssetId2();
193+
136194
const response = await client.getAllAssetAdministrationShellIdsByAssetLink({
137195
configuration,
138-
assetIds: [testSpecificAssetId1, testSpecificAssetId2],
196+
assetIds: [specificAssetId1, specificAssetId2],
139197
cursor: unavailableCursor(),
140198
});
141199

@@ -152,15 +210,17 @@ describe('AAS Discovery Integration Tests', () => {
152210
* @status 200
153211
*/
154212
test('should fetch a list of Asset Administration Shell IDs via search endpoint', async () => {
213+
const { aasIdentifier, specificAssetId1, specificAssetId2 } = await seedAssetLinks();
214+
155215
const response = await client.searchAllAssetAdministrationShellIdsByAssetLink({
156216
configuration,
157-
assetLink: [testSpecificAssetId1, testSpecificAssetId2],
217+
assetLink: [specificAssetId1, specificAssetId2],
158218
});
159219

160220
expect(response.success).toBe(true);
161221
if (response.success) {
162222
expect(response.statusCode).toBe(200);
163-
expect(response.data.result).toContainEqual(testShell.id);
223+
expect(response.data.result).toContainEqual(aasIdentifier);
164224
}
165225
});
166226

@@ -171,7 +231,7 @@ describe('AAS Discovery Integration Tests', () => {
171231
test('should reject malformed search payload with bad request', async () => {
172232
const response = await client.searchAllAssetAdministrationShellIdsByAssetLink({
173233
configuration,
174-
assetLink: [{ name: 'globalAssetId' } as any],
234+
assetLink: [{ name: 'globalAssetId' } as unknown as CoreSpecificAssetId],
175235
});
176236

177237
expect(response.success).toBe(false);
@@ -186,9 +246,12 @@ describe('AAS Discovery Integration Tests', () => {
186246
* @status 200
187247
*/
188248
test('should return an empty AAS discovery search page for an unavailable cursor', async () => {
249+
const specificAssetId1 = createUniqueSpecificAssetId1();
250+
const specificAssetId2 = createUniqueSpecificAssetId2();
251+
189252
const response = await client.searchAllAssetAdministrationShellIdsByAssetLink({
190253
configuration,
191-
assetLink: [testSpecificAssetId1, testSpecificAssetId2],
254+
assetLink: [specificAssetId1, specificAssetId2],
192255
cursor: unavailableCursor(),
193256
});
194257

@@ -206,10 +269,11 @@ describe('AAS Discovery Integration Tests', () => {
206269
*/
207270
test('should reject invalid specific asset IDs with bad request', async () => {
208271
const invalidSpecificAssetId = new CoreSpecificAssetId('', '');
272+
const aasIdentifier = createUniqueShellId();
209273

210274
const response = await client.postAllAssetLinksById({
211275
configuration,
212-
aasIdentifier: testShell.id,
276+
aasIdentifier,
213277
specificAssetId: [invalidSpecificAssetId],
214278
});
215279

@@ -225,10 +289,12 @@ describe('AAS Discovery Integration Tests', () => {
225289
* @status 405
226290
*/
227291
test('should reject trailing slash (empty aasIdentifier) for post by id with method not allowed', async () => {
292+
const specificAssetId1 = createUniqueSpecificAssetId1();
293+
228294
const response = await client.postAllAssetLinksById({
229295
configuration,
230296
aasIdentifier: '',
231-
specificAssetId: [testSpecificAssetId1],
297+
specificAssetId: [specificAssetId1],
232298
});
233299

234300
expect(response.success).toBe(false);
@@ -243,10 +309,12 @@ describe('AAS Discovery Integration Tests', () => {
243309
* @status 404 [known-specification-bug]
244310
*/
245311
test.skip('should return not found when creating links for a non-existing AAS (spec mismatch: endpoint behaves as create-or-update)', async () => {
312+
const specificAssetId1 = createUniqueSpecificAssetId1();
313+
246314
const response = await client.postAllAssetLinksById({
247315
configuration,
248316
aasIdentifier: 'https://example.com/ids/aas/does-not-exist',
249-
specificAssetId: [testSpecificAssetId1],
317+
specificAssetId: [specificAssetId1],
250318
});
251319

252320
expect(response.success).toBe(false);
@@ -261,10 +329,12 @@ describe('AAS Discovery Integration Tests', () => {
261329
* @status 409 [known-specification-bug]
262330
*/
263331
test.skip('should surface conflict status for duplicate link creation when backend enforces it', async () => {
332+
const { aasIdentifier, specificAssetId1 } = await seedAssetLinks();
333+
264334
const response = await client.postAllAssetLinksById({
265335
configuration,
266-
aasIdentifier: testShell.id,
267-
specificAssetId: [testSpecificAssetId1],
336+
aasIdentifier,
337+
specificAssetId: [specificAssetId1],
268338
});
269339

270340
if (response.success) {
@@ -281,9 +351,11 @@ describe('AAS Discovery Integration Tests', () => {
281351
* @status 204
282352
*/
283353
test('should delete all specific asset identifiers for an Asset Administration Shell ID', async () => {
354+
const { aasIdentifier } = await seedAssetLinks();
355+
284356
const response = await client.deleteAllAssetLinksById({
285357
configuration,
286-
aasIdentifier: testShell.id,
358+
aasIdentifier,
287359
});
288360

289361
expect(response.success).toBe(true);

‎src/integration-tests/aasRegistry.integration.test.ts‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,26 +8,44 @@ import {
88
createTestShellDescriptor,
99
createTestSubmodelDescriptor,
1010
} from './fixtures/aasregistryFixtures';
11+
import { createPerTestCleanupRunner } from './fixtures/testCleanup';
1112
import { getIntegrationBasePath } from './testEngineConfig';
1213

1314
describe('AAS Registry Integration Tests', () => {
1415
const client = new AasRegistryClient();
1516
const configuration = new Configuration({
1617
basePath: getIntegrationBasePath('aasRegistry'),
1718
});
19+
const { track } = createPerTestCleanupRunner();
1820

1921
const uniqueSuffix = (): string => `${Date.now()}-${Math.random().toString(36).slice(2)}`;
2022
const unavailableCursor = (): string =>
2123
base64Encode(`https://example.com/ids/non-existing-cursor-${uniqueSuffix()}`);
2224

23-
const createUniqueShellDescriptor = () => {
25+
const createUniqueShellDescriptor = (withCleanup = true) => {
2426
const descriptor = createTestShellDescriptor();
2527
descriptor.id = `${descriptor.id}-${uniqueSuffix()}`;
28+
29+
if (withCleanup) {
30+
track(async () => {
31+
const cleanupResponse = await client.deleteAssetAdministrationShellDescriptorById({
32+
configuration,
33+
aasIdentifier: descriptor.id,
34+
});
35+
36+
if (!cleanupResponse.success && cleanupResponse.statusCode !== 404) {
37+
throw new Error(
38+
`Failed to cleanup AAS descriptor ${descriptor.id} (status ${cleanupResponse.statusCode})`
39+
);
40+
}
41+
});
42+
}
43+
2644
return descriptor;
2745
};
2846

2947
const createInvalidShellDescriptorWithEmptyId = () => {
30-
const descriptor = createUniqueShellDescriptor();
48+
const descriptor = createUniqueShellDescriptor(false);
3149
descriptor.id = '';
3250
descriptor.assetKind = AssetKind.Type;
3351
descriptor.assetType = 'Type';

0 commit comments

Comments
 (0)