diff --git a/CODEOWNERS b/CODEOWNERS index c9eed84e1..5efa04b2b 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -89,11 +89,11 @@ /skills/uipath-tasks/ @UiPath/ActionCenterDev /tests/tasks/uipath-tasks/ @UiPath/ActionCenterDev -# Admin skill (Identity Server management) -/skills/uipath-admin/ @sriramva-uipath @bansal-anushree @jianjunwang2 @t-hsia @ZuerWang99 @litheon @IsabellaCapriottiUIPath @uipathswapnil @ashleyetheridge -/tests/tasks/uipath-admin/ @sriramva-uipath @bansal-anushree @jianjunwang2 @t-hsia @ZuerWang99 @litheon @IsabellaCapriottiUIPath @uipathswapnil @ashleyetheridge +# Admin skill (Identity Server + Audit Service management) +/skills/uipath-admin/ @sriramva-uipath @bansal-anushree @jianjunwang2 @t-hsia @ZuerWang99 @litheon @IsabellaCapriottiUIPath @uipathswapnil @ashleyetheridge @yadvender-uipath @CalinMPopa @chandhanshanth +/tests/tasks/uipath-admin/ @sriramva-uipath @bansal-anushree @jianjunwang2 @t-hsia @ZuerWang99 @litheon @IsabellaCapriottiUIPath @uipathswapnil @ashleyetheridge @yadvender-uipath @CalinMPopa @chandhanshanth # IXP skill /skills/uipath-ixp/ @cezara98t @paul-ciobanu @misupantea-uipath @andrei-uipath /skills/uipath-ixp/ @cezara98t @paul-ciobanu @misupantea-uipath @andrei-uipath -/tests/tasks/uipath-ixp/ @cezara98t @paul-ciobanu @misupantea-uipath @andrei-uipath \ No newline at end of file +/tests/tasks/uipath-ixp/ @cezara98t @paul-ciobanu @misupantea-uipath @andrei-uipath diff --git a/skills/uipath-admin/SKILL.md b/skills/uipath-admin/SKILL.md index 401a768df..8e5c20c14 100644 --- a/skills/uipath-admin/SKILL.md +++ b/skills/uipath-admin/SKILL.md @@ -1,17 +1,19 @@ --- name: uipath-admin -description: "UiPath Admin — Identity Server management via uip admin. Users, groups, robot accounts, external apps (OAuth2), credential generation (Client ID/Secret). For Orchestrator folders/jobs→uipath-platform. For RPA workflows→uipath-rpa." -allowed-tools: Bash, Read, Write, Edit, Glob, Grep +description: "Always invoke for `uip admin` commands or audit-investigation prompts (who/when/where on a resource, login history, compliance dumps). UiPath Admin via `uip admin ` — Identity Server (users, groups, robot accounts, external OAuth2 apps, credential generation) and Audit Service (event sources, paginated event queries, long-term-store ZIP exports). For Orchestrator folders/jobs→uipath-platform. For RPA workflows→uipath-rpa." +allowed-tools: Bash, Read, Write, Edit, Glob, Grep, AskUserQuestion --- # UiPath Admin > **Preview** — Under active development. Command coverage will expand. -Identity Server management via `uip admin`. Users, groups, robot accounts, external OAuth2 apps. +Administrative operations on UiPath via `uip admin` — Identity Server (users, groups, robot accounts, external OAuth2 apps) and Audit Service (event sources, event queries, long-term-store ZIP exports). ## When to Use This Skill +### Identity + - **Manage identity users** — list, create, invite, update, delete - **Manage groups** — CRUD + add/remove members - **Manage robot accounts** — create, update, delete unattended robot identities @@ -21,8 +23,17 @@ Identity Server management via `uip admin`. Users, groups, robot accounts, exter - **Identity concepts** — partitions, organizations, OAuth2 scopes - **Generate Client ID/Secret** — credentials for API or robot authentication +### Audit + +Activate on both **explicit audit requests** and **natural-language investigation intent** — users rarely say "audit events" by name. + +- **Explicit** — `uip admin audit` commands; list sources / targets / types; query, filter, paginate, or export events; CSV/ZIP dump of audit history for a window. +- **Investigation intent** — "Who deleted the X folder last Tuesday?", "Show me failed logins for user Y this month.", "What changed on tenant Z between Jan 1 and Feb 1?", "Give me the audit log for the last 30 days.", "Was the API key rotated by someone in our org?", "Export everything for compliance for Q4." + ## Critical Rules +### Identity + 1. **Verify login first.** Run `uip login status --output json`. If not logged in: `uip login`. 2. **Organization ID is resolved automatically from login.** CLI reads org ID from active session. 3. **Discover before creating.** `list` before `create` to avoid duplicates. Applies to robot accounts, groups, and external apps — not to `users invite`. @@ -33,12 +44,28 @@ Identity Server management via `uip admin`. Users, groups, robot accounts, exter 8. **Confirm before delete.** Always confirm with user before running `delete` on users, groups, robot accounts, or external apps. 9. **Stop on error (interactive use).** If any command fails, show error to user. Do not retry auth failures — ask user to run `uip login`. +### Audit + +10. **Pick scope before any other audit call — and don't silently default.** Use the [Disambiguation rule](#audit-scope-disambiguation) below: if the prompt is vague about `org` vs `tenant` AND there's no prior conversational context, **ask** which scope (and which tenant if `tenant`). `tenant` requires either a tenant in the login context or `--tenant-id `. `org` ignores `--tenant-id`. +11. **Discover source IDs with `sources` before filtering `events`.** Never invent GUIDs for `--source`/`--target`/`--type` — the SDK won't help you guess. +12. **`events` returns `{ auditEvents, next, previous }` — NOT a bare array.** Cursor naming is **chronological**: `next` = newer, `previous` = older. The default newest-backward walk follows `previous`. +13. **`events` server-clamps `maxCount` to `[10, 200]`.** When the user wants more than 200, the tool paginates internally — pass `--limit N` and the tool fetches `ceil(N/200)` pages. Do **not** re-implement pagination in the agent. +14. **`export` writes a ZIP from the long-term store.** `--from-date`, `--to-date`, and `--output-file` are required and ISO 8601 (for the dates). Never overwrite a path the user did not explicitly approve. +15. **ISO 8601 for time bounds, UTC by default.** Date-only (`2026-04-01`) or with time (`2026-04-01T14:30:00Z`). `--to-date` is inclusive of the exact instant — pass the start of the next day (or `T23:59:59.999Z`) to capture a full final day. + ## What NOT to Do 1. **Never delete built-in groups.** `type: "BuiltIn"` groups cannot be deleted. Only custom groups. 2. **Never pass IDs as flags.** Resource IDs and names are positional arguments: `groups members add --user-ids ...`, NOT `--group-id `. Same for all `get`, `update`, `delete`, `create` subcommands. +3. **Do NOT assume audit `events` returns a bare array.** It's `{auditEvents, next, previous}`. +4. **Do NOT loop on `--from-date`/`--to-date` to "paginate".** Bump `--limit` and the CLI handles cursor pagination internally. +5. **Do NOT silently default audit scope** to `tenant` or `org` when the prompt is ambiguous. Ask once, then proceed. +6. **Do NOT invent audit source/target/type GUIDs.** Always discover via `sources` first. +7. **Do NOT call audit `events` with no time bound** on a noisy tenant — default to a bounded window. +8. **Do NOT pass `--tenant-id` to `org`-scoped audit commands** — it's silently ignored. If you find yourself doing this, you probably meant `tenant` scope. +9. **Do NOT retry on 401 auth errors.** The token is missing the required scope (`Audit.Read` for audit). Tell the user to `uip logout && uip login` so the new scope is included. -## Quick Start +## Quick Start — Identity The most common identity flow is **user management** — inviting users, assigning them to groups, and managing access. @@ -76,6 +103,58 @@ uip admin groups members add \ --output json ``` +## Quick Start — Audit + +For the canonical "find events in a window then export" flow. For specific scenarios jump to the [Task Navigation](#task-navigation) table. + +### Audit scope disambiguation + +The `org` vs `tenant` choice matters — they hit different basePaths and surface different events. + +| User says... | Likely scope | Why | +|---|---|---| +| "who joined / left the organization", "who was made an admin", "license changes", "cross-tenant audit" | **org** | Org-level admin events (memberships, license, tenant lifecycle) live under `/orgaudit_`. | +| "what happened on tenant X", "logins on this tenant", "policy changes within a tenant", "asset/queue/folder edits" | **tenant** | Tenant-scoped events (Orchestrator, AOps, AI Trust, etc.) live under `/{tenantId}/tenantaudit_`. | +| "everything everywhere" | **both** — run the same flow once per scope and present combined results. | + +If the prompt is **vague about scope** AND no prior turn has established it, **stop and ask** (one yes/no question, two clarifications max). Don't assume `tenant` just because it's the more common case. + +### Step 1 — Verify scope, then discover sources + +```bash +# Tenant-scoped (most common) +uip admin audit tenant sources --output json > sources.json + +# Org-scoped (admin events: tenant lifecycle, license, memberships) +uip admin audit org sources --output json > sources-org.json +``` + +Each entry has `id` (a GUID — pass to `events --source`), `name` (human-readable), and `eventTargets[]` (each with their own GUIDs and `eventTypes[]`). + +### Step 2 — Query events with filters + +```bash +uip admin audit tenant events \ + --source \ + --from-date 2026-04-22T00:00:00Z \ + --to-date 2026-04-29T00:00:00Z \ + --limit 50 \ + --output json +``` + +The response is `{ "auditEvents": [...], "next": null, "previous": "..." }`. For more than 200 events, pass `--limit 500` (or larger) — the tool paginates internally. Do **not** write a manual loop in the agent. + +### Step 3 — Export for compliance / sharing + +```bash +uip admin audit tenant export \ + --from-date 2026-01-01 \ + --to-date 2026-02-01 \ + --output-file ./audit-jan.zip +``` + +One HTTP call per UTC day inside the window, aggregated into a single flat ZIP at `--output-file`. The result envelope reports `{Path, Bytes, Format: "zip", Days, NonEmptyDays}`. On any chunk failure (e.g. HTTP 504), no file is written and the error identifies which day failed. + ## Key Concepts ### Organization Hierarchy @@ -101,9 +180,26 @@ These are separate concepts — do not conflate them. Robot credentials are provisioned automatically by Orchestrator when connecting a robot to a machine — not by creating external apps. +### Audit scope → basePath + +- `org` → `{baseUrl}/{orgId}/orgaudit_/api/Query/...` +- `tenant` → `{baseUrl}/{orgId}/{tenantId}/tenantaudit_/api/Query/...` + +Same `QueryApi` underneath; the only difference is which segment the SDK puts in the URL. + +### Audit `Data` shape varies by verb + +| Verb | `Data` shape | +|---|---| +| `audit sources` | array of `AuditEventSourceDto` | +| `audit events` | object `{auditEvents, next, previous}` | +| `audit export` | object `{Path, Bytes, Format, Days, NonEmptyDays}` | + +`events` is the one verb that legitimately returns an object — pagination cursors live alongside the rows. + ## Completion Output -After any mutation (create, update, delete, invite, members add, members revoke, generate-secret): +### After identity mutations (create, update, delete, invite, members add/revoke, generate-secret) 1. Show the command result (success or failure) 2. For creates: display the new resource ID @@ -113,20 +209,34 @@ After any mutation (create, update, delete, invite, members add, members revoke, - After creating an external app → "Generate an additional secret?" - After inviting a user → "Check user list to see when they accept?" +### After an audit query or export + +1. **Operation & result** — e.g. `Found 47 audit events on tenant T in the last 7 days` or `Wrote 123,456 bytes to /path/to/audit.zip (3 days, 2 non-empty)`. +2. **Scope used** (`org` or `tenant`) and any `--tenant-id` override. +3. **Time window** — explicit ISO bounds, even if they came from a relative phrase ("last 7 days"). +4. **Filters applied** — sources, types, users, status. +5. **Cursor state** — for `events`, mention whether `Data.previous` is null (start of audit history) or populated (more older events available — re-run with a larger `--limit`). +6. **Next step** — "Want me to widen the window?", "Want me to export this slice?", "Want me to filter by user X?". Wait for the user's choice; do not chain mutations. + ## Task Navigation | I need to... | Read first | |---|---| -| **Full CLI command reference** | [references/identity-commands.md](references/identity-commands.md) | +| **Full CLI command reference (identity)** | [references/identity-commands.md](references/identity-commands.md) | | **Manage users** (list, create, invite, update, delete) | [references/user-management.md](references/user-management.md) | | **Manage groups** (CRUD + membership) | [references/group-management.md](references/group-management.md) | | **Manage robot accounts** | [references/robot-account-management.md](references/robot-account-management.md) | | **Manage external apps** (OAuth2 + secrets) | [references/external-app-management.md](references/external-app-management.md) | +| **Full CLI command reference (audit)** | [references/audit-commands.md](references/audit-commands.md) | +| **Investigation playbooks** (who-did-X / login-history / date-range dump / overview) | [references/audit-workflow-guide.md](references/audit-workflow-guide.md) | +| **Paginate audit events beyond 200** | [references/audit-commands.md](references/audit-commands.md) (events flag table) + Critical Rule #13 | ## References -- **[identity-commands.md](references/identity-commands.md)** — Complete CLI reference for all `uip admin` commands with flags, arguments, and output codes +- **[identity-commands.md](references/identity-commands.md)** — Complete CLI reference for all `uip admin` identity commands with flags, arguments, and output codes - **[user-management.md](references/user-management.md)** — User lifecycle workflows: discover, create, invite, update, delete, pagination, sorting - **[group-management.md](references/group-management.md)** — Group CRUD, membership management (add/remove members), built-in vs custom groups - **[robot-account-management.md](references/robot-account-management.md)** — Robot account lifecycle, relationship to external apps - **[external-app-management.md](references/external-app-management.md)** — OAuth2 client management, secret generation/rotation, scope reference +- **[audit-commands.md](references/audit-commands.md)** — Single source of truth for every `uip admin audit` subcommand: signature, every flag with required/optional, the `Code` value, and the exact `Data` shape returned +- **[audit-workflow-guide.md](references/audit-workflow-guide.md)** — Narrative playbook for the four canonical investigations: who-did-X, login-history, date-range-dump, and org-vs-tenant comparison diff --git a/skills/uipath-admin/references/audit-commands.md b/skills/uipath-admin/references/audit-commands.md new file mode 100644 index 000000000..a8481265a --- /dev/null +++ b/skills/uipath-admin/references/audit-commands.md @@ -0,0 +1,206 @@ +# `uip admin audit` — CLI Command Reference + +Single source of truth for every `uip admin audit` subcommand, its flags, and its output shape. All commands return `{ "Result": "Success"|"Failure", "Code": "...", "Data": ... }`. Use `--output json` for programmatic parsing — every command in this skill should pass it. + +> For task workflows (investigate → query → export), see [audit-workflow-guide.md](./audit-workflow-guide.md). This file only documents the command surface. + +The command tree: + +``` +uip admin audit +├── org +│ ├── sources +│ ├── events +│ └── export +└── tenant + ├── sources + ├── events + └── export +``` + +`org` and `tenant` are **subject subgroups**. Same three verbs under each. The two trees are 100% verb-symmetric — any flag valid on `tenant events` is also valid on `org events` (except `--tenant-id`, which is tenant-only). + +--- + +## uip admin audit `` sources + +List the audit event sources visible at this scope. Each source is a top-level event category (Identity, Tenant, Robot, Governance, …) with nested `eventTargets[]` and `eventTypes[]`. + +```bash +uip admin audit org sources --output json +uip admin audit tenant sources --output json +``` + +**Flags:** + +| Flag | Required | Description | +|---|---|---| +| `--login-validity ` | no | Refresh the bearer if its remaining lifetime is below this threshold. Rarely needed. | +| `--tenant-id ` | no | **Tenant scope only.** Override the tenant from login context. Silently rejected on `org`. | + +**Output `Code`:** `AuditOrgSources` / `AuditTenantSources`. + +**Output `Data`:** Array of `AuditEventSourceDto`: + +```json +[ + { + "id": "692a7634-bdfc-4c77-a7ee-a8c7eef10457", + "name": "Organization Management", + "eventTargets": [ + { + "id": "355b521a-4384-4d1b-8f39-e2769840d2d5", + "name": "Org Membership", + "eventTypes": [ + { "id": "7fb44323-cb87-40ea-bee6-8dd14f5b2c06", "name": "User Manually Joined An Org Through Invite" } + ] + } + ] + } +] +``` + +`Data` is **always** an array, even when empty. Pass the inner `id` GUIDs to `events --source/--target/--type`. + +--- + +## uip admin audit `` events + +Query audit events with filters and cursor pagination. + +```bash +uip admin audit tenant events --from-date 2026-04-22T00:00:00Z --to-date 2026-04-29T00:00:00Z --limit 50 --output json +``` + +**Flags:** + +| Flag | Required | Description | +|---|---|---| +| `--from-date ` | no | Start of time interval, ISO 8601. Inclusive. Recommended on any non-trivial query. | +| `--to-date ` | no | End of time interval, ISO 8601. Inclusive **of the exact instant** — pass the start of the next day (e.g. `2026-02-01`) or `T23:59:59.999Z` to capture a full final day. See [workflow-guide gotchas](./audit-workflow-guide.md#common-gotchas). | +| `--source ` | no | Filter by event source IDs. Repeatable. Discover with `sources`. | +| `--target ` | no | Filter by event target IDs. Repeatable. | +| `--type ` | no | Filter by event type IDs. Repeatable. | +| `--user-id ` | no | Filter by acting user IDs. Repeatable. | +| `--search ` | no | Server-side substring search across event content. | +| `--status ` | no | Case-insensitive labels or the raw `AuditEventStatus` enum values. | +| `--limit ` | no | Total events to return. Server clamps each individual API call to **200**; values >200 trigger client-side pagination automatically (cursor handled internally). Omitted = single call with the server's default page size (typically up to 200 events). | +| `--login-validity ` | no | Token-refresh hint. | +| `--tenant-id ` | no | **Tenant scope only.** Override the active tenant. | + +**Output `Code`:** `AuditOrgEvents` / `AuditTenantEvents`. + +**Output `Data`:** Object — NOT a bare array — with three fields: + +```json +{ + "auditEvents": [ + { + "id": "...", + "createdOn": "2026-04-29T17:46:07.123Z", + "organizationId": "...", + "organizationName": "Acme Corp", + "tenantId": "...", // null on org-scope events + "tenantName": "Acme-Prod", // null on org-scope events + "actorId": "...", + "actorName": "Jane Doe", + "actorEmail": "jane@example.com", + "eventType": "Edited policy", + "eventSource": "Governance", + "eventTarget": "Policy management", + "eventDetails": "{...}", // JSON-encoded string with type-specific fields + "eventSummary": "...", + "status": 0, // 0=Success, 1=Failure + "clientInfo": { // optional; absent on server-originated events + "ipAddress": "203.0.113.42", + "ipCountry": "US" + } + } + ], + "next": null, // cursor URL — events newer than this page + "previous": "/{org}/{tenant}/tenantaudit_/api/query/events?to=...&before=...&beforeId=...&maxCount=...&qw=..." +} +``` + +**Cursor naming is chronological.** `next` = newer (often null when the page is anchored at "now"); `previous` = older (typically the one you follow to scroll backwards through history). + +--- + +## uip admin audit `` export + +Stream a ZIP from the long-term audit store covering `[--from-date, --to-date]`. + +```bash +uip admin audit tenant export \ + --from-date 2026-01-01 \ + --to-date 2026-02-01 \ + --output-file ./audit-jan.zip \ + --output json +``` + +**Flags:** + +| Flag | Required | Description | +|---|---|---| +| `--output-file ` | **yes** | Where to write the ZIP. Parent dir is created if missing; existing file is overwritten. Resolved to absolute internally. | +| `--from-date ` | **yes** | Start of time interval. Both bounds are required by Commander before any HTTP call. | +| `--to-date ` | **yes** | End of time interval. | +| `--login-validity ` | no | Token-refresh hint. | +| `--tenant-id ` | no | **Tenant scope only.** Override the active tenant. | + +**Output `Code`:** `AuditOrgExport` / `AuditTenantExport`. + +**Output `Data`:** + +```json +{ + "Path": "C:\\absolute\\path\\to\\audit-jan.zip", + "Bytes": 1841, + "Format": "zip", + "Days": 31, + "NonEmptyDays": 27 +} +``` + +**Implementation notes (worth knowing for diagnostic conversations):** + +- The CLI issues **one HTTP call per UTC day** inside `[from, to]` and aggregates the per-day responses into a single output ZIP. Mirrors the `audit-dowload-from-longterm-store.sh` pattern in the AuditService repo. +- Per-day entries land at the **root of the output ZIP** named `.txt` (no folder prefix). Each `.txt` is a JSON array of events with PascalCase keys (`Id`, `CreatedOn`, `EventType`, …). +- On any single-day HTTP failure, **no file is written** and the error message identifies which day failed. Earlier successful chunks are not preserved. +- `Days` reports the total number of UTC days requested; `NonEmptyDays` reports how many actually had events. A long export with `NonEmptyDays: 0` means the window was entirely idle, not that the export failed. + +--- + +## Cross-cutting flags from the CLI host + +These appear on every command (not just audit) — set at the program level by the `uip` host: + +| Flag | Description | +|---|---| +| `--output ` | Format of the success/failure envelope on stdout. Defaults to `json`. The ZIP body of `export` is unaffected — `--output` only controls the metadata envelope. | +| `--output-filter ` | JMESPath query applied to the envelope before printing. Useful for `events`/`sources`. Less useful for `export` (envelope is small). | +| `--log-level ` | Logger threshold. Logs go to stderr. | +| `--log-file ` | Redirect logs from stderr to a file. | +| `--help, -h` | Show help. | + +--- + +## Error envelope + +```json +{ + "Result": "Failure", + "Message": "Audit export failed for 2026-04-02 (HTTP 504): Gateway Timeout", + "Instructions": "Ensure you are logged in with 'uip login' and have access to the audit service." +} +``` + +Common failure modes and what they mean: + +| `Message` snippet | Likely cause | Fix | +|---|---|---| +| `Not logged in. Run 'uip login' first.` | No cached login state | `uip login` | +| `Tenant ID required for tenant-scoped audit calls.` | `tenant` scope but no tenant in login context | Add `--tenant-id ` or re-`uip login` selecting a tenant | +| `HTTP 401 / WWW-Authenticate: Bearer` | Token's `aud` claim missing `Audit` | `uip logout && uip login` to mint a fresh token (the `Audit.Read` scope is in `DEFAULT_SCOPES` post-onboarding) | +| `HTTP 504` on a single export day | Long-term store query timed out for that day's volume | Re-run the export — the failed day will be retried; OR narrow the window | +| `Audit export failed for YYYY-MM-DD (HTTP 504)` | Single-day chunk failed during multi-day export | The whole export is rolled back. Re-run, or narrow `--from-date/--to-date` to skip the bad day. | diff --git a/skills/uipath-admin/references/audit-workflow-guide.md b/skills/uipath-admin/references/audit-workflow-guide.md new file mode 100644 index 000000000..39a673f4f --- /dev/null +++ b/skills/uipath-admin/references/audit-workflow-guide.md @@ -0,0 +1,244 @@ +# Audit Investigation Workflow Guide + +Four canonical investigations the `uipath-audit` skill should drive. Each starts from a natural-language question and produces a reproducible answer using `uip admin audit` — typically `sources` (discover) → `events` (filter) → `export` (capture). + +> Every command below assumes the user has run `uip login` and the active token includes the `Audit.Read` scope. Every command should pass `--output json` so the agent can parse the envelope. + +--- + +## Investigation 1 — "Who did X to resource Y?" + +**User asks:** "Who deleted the `Sales-Reports` folder last Tuesday?" / "Who edited the AI Trust Layer policy this week?" / "Who turned off MFA for the org admin group?" + +**Approach:** Discover the source/target IDs that match the resource, narrow `events` by source + target + time window, then format actor + timestamp. + +### Step 1 — Discover sources at the right scope + +The "what changed" determines scope. Folder/queue/asset edits are **tenant** scope. Membership/license/tenant-lifecycle is **org** scope. Most resource-touching investigations are tenant. + +```bash +uip admin audit tenant sources --output json > /tmp/sources.json +``` + +### Step 2 — Locate the matching source/target + +Find a source whose `name` matches the resource type, then drill into its `eventTargets[]` for the specific entity type, and `eventTypes[]` for the verb. + +For "deleted folder": + +- Source: `Folders` (or similar) +- Target: `Folder` +- Type: `Deleted folder` + +The agent should grep the JSON for these names to extract the GUIDs: + +```bash +jq -r '.Data[] | select(.name == "Folders") | .id' /tmp/sources.json +jq -r '.Data[] | select(.name == "Folders") | .eventTargets[] | select(.name == "Folder") | .id' /tmp/sources.json +``` + +### Step 3 — Query events with filters + +```bash +uip admin audit tenant events \ + --source \ + --target \ + --type \ + --from-date 2026-04-22T00:00:00Z \ + --to-date 2026-04-29T00:00:00Z \ + --limit 50 \ + --output json +``` + +> **Need more than 50 results?** Bump `--limit` directly — the CLI paginates internally for `--limit > 200` (server clamps each individual call to 200; the tool follows `previous` automatically). Do **not** loop on `--from-date` / `--to-date` or chase cursor flags from the agent. + +### Step 4 — Present + +For each `auditEvent` in `Data.auditEvents`, surface: + +- `createdOn` (UTC) +- `actorName` (fall back to `actorEmail` or `actorId`) +- `eventDetails` — parse the JSON-encoded string for the resource-specific fields (e.g., folder name) +- `status` — translate `0`→`Success`, `1`→`Failure` + +If `Data.previous` is non-null, mention "older results available — extend the window with `--from-date ` to see more." + +--- + +## Investigation 2 — "Show me logins for user X" + +**User asks:** "Show me failed logins for jane.doe@example.com this month." / "When did Bob last log in?" / "Was anyone logging in from outside the US last week?" + +**Approach:** Filter events by user ID + the `User Login` event type, optionally with `--status Failure`. Present chronologically with `ipAddress`/`ipCountry` from `clientInfo`. + +### Step 1 — Find the user's GUID + +The user's `actorId` GUID is required by `--user-id`. It's not visible from email alone. Two ways: + +1. If the user has at least one prior audit event you've already loaded, grab `actorId` from there. +2. Otherwise, run a search-only query (no `--user-id`) for `--search jane.doe` and take `actorId` from the first match. + +```bash +uip admin audit tenant events \ + --search "jane.doe@example.com" \ + --limit 5 --output json --output-filter "Data.auditEvents[0].actorId" +``` + +### Step 2 — Find the User Login type GUID + +```bash +jq -r '.Data[] | select(.name == "Identity") | .eventTargets[] | select(.name == "Authentication") | .eventTypes[] | select(.name == "User Login") | .id' /tmp/sources.json +``` + +Names like `Identity`, `Authentication`, `User Login` are illustrative — confirm against your actual `sources.json` output before committing the selector. They can vary across UiPath cloud versions and regions; if the `jq` returns empty, list the candidate names with `jq -r '.Data[].name'` and `.eventTargets[].name` and pick the closest match. + +### Step 3 — Query + +For "all logins this month": + +```bash +uip admin audit tenant events \ + --user-id \ + --type \ + --from-date 2026-04-01T00:00:00Z \ + --to-date 2026-04-29T23:59:59Z \ + --limit 200 \ + --output json +``` + +For "failed logins only": + +```bash +uip admin audit tenant events \ + --user-id \ + --type \ + --status Failure \ + --from-date 2026-04-01T00:00:00Z \ + --to-date 2026-04-29T23:59:59Z \ + --output json +``` + +### Step 4 — Present + +For each event: + +- `createdOn` +- `clientInfo.ipAddress` / `clientInfo.ipCountry` (parse from `clientInfo` field) +- `eventDetails` — typically contains `AuthenticationProvider` and session info + +--- + +## Investigation 3 — "Give me an audit dump for January" + +**User asks:** "Export everything for compliance for Q4." / "I need the full audit log for January for the security review." / "Pull last month's events for tenant X as a ZIP." + +**Approach:** `export` straight to a ZIP. No need to query `events` first unless the user wants a preview. + +### Step 1 — Confirm scope and window + +If the user is ambiguous about scope, ask once. For compliance reviews, **both** scopes are typically needed (separately). + +### Step 2 — Export + +```bash +# Tenant scope — most events +uip admin audit tenant export \ + --from-date 2026-01-01 \ + --to-date 2026-02-01 \ + --output-file ./audit-tenant-2026-01.zip \ + --output json + +# Org scope — admin events (memberships, license, tenant lifecycle) +uip admin audit org export \ + --from-date 2026-01-01 \ + --to-date 2026-02-01 \ + --output-file ./audit-org-2026-01.zip \ + --output json +``` + +The CLI issues one HTTP call per UTC day under the hood and aggregates daily responses into a flat ZIP. `Days` and `NonEmptyDays` in the result tell you how many calendar days had events. + +### Step 3 — Verify the ZIP + +```bash +unzip -l ./audit-tenant-2026-01.zip +``` + +Typical layout (one entry per UTC day with events): + +``` +audit-tenant-2026-01.zip +├── 2026-01-01.txt +├── 2026-01-02.txt +├── ... +└── 2026-01-31.txt +``` + +Each `.txt` is a JSON array of audit events with **PascalCase** keys (`Id`, `CreatedOn`, `OrganizationId`, `ActorId`, `ActorName`, `EventType`, …) — different from the camelCase shape returned by the live `events` endpoint. Note this in the user's hand-off if they're going to feed the dump into other tooling. + +Edge cases the CLI handles automatically — surface in your hand-off only if they appear: + +- **Nested ZIPs**: when a single day's response is itself a ZIP-of-ZIPs, inner files are renamed `_.txt` rather than collapsed. +- **Same-name collisions**: duplicate basenames across days get an `_` suffix (and `__2`, `_3`, … if needed) to keep entries unique. + +### Step 4 — Hand off + +Report: + +- The absolute path written +- Total bytes +- `Days` requested vs `NonEmptyDays` +- Whether the user wants the agent to also run an org export (if they only asked for tenant) + +If `Days > 365` or the export failed mid-stream, suggest narrowing to monthly chunks: the daily-chunked downloader is robust to multi-month windows but a single bad day in a multi-year export forces a full re-run. + +--- + +## Investigation 4 — "What's been happening at the org / tenant level?" + +**User asks:** "Show me the most recent admin activity in our org." / "What's happened on tenant T this past week?" / "I'm new — give me the audit history overview." + +**Approach:** Run both scopes, fetch a bounded recent window with no filters, summarize event-type frequencies. + +### Step 1 — Recent events at each scope + +```bash +uip admin audit org events --from-date 2026-04-22 --to-date 2026-04-29 --limit 100 --output json > /tmp/org-events.json +uip admin audit tenant events --from-date 2026-04-22 --to-date 2026-04-29 --limit 100 --output json > /tmp/tenant-events.json +``` + +### Step 2 — Group by event type/source + +```bash +jq -r '.Data.auditEvents | group_by(.eventType) | map({eventType: .[0].eventType, count: length}) | sort_by(-.count)' /tmp/org-events.json +jq -r '.Data.auditEvents | group_by(.eventType) | map({eventType: .[0].eventType, count: length}) | sort_by(-.count)' /tmp/tenant-events.json +``` + +### Step 3 — Present + +- Top 5 event types per scope +- Most active actors (group by `actorName`) +- Time distribution (events per day) + +If the user wants more depth on any one event type, drill in with Investigation 1's pattern (filter by `--source` and `--type`). + +--- + +## Picking the right investigation + +| User intent signal | Investigation | +|---|---| +| "who" / "did" + a verb on a resource | **1** — Who did X to Y | +| "logged in" / "login" / "authenticated" / a user's email | **2** — Login history | +| "export" / "dump" / "ZIP" / "CSV" / a date range | **3** — Date-range dump | +| "overview" / "what's happening" / "recent activity" / "audit summary" | **4** — Overview | + +Two or more signals? Run them in sequence and stitch the results in the final report. Don't make the user re-ask. + +## Common gotchas + +- **`tenant` events without an active tenant fail loudly.** If `uip login` has no tenant selected, every tenant-scoped command throws. Either re-`uip login` and pick a tenant, or pass `--tenant-id ` on every call. +- **`events` cursor pagination is chronologically reversed from intuition.** `next` = newer (often null), `previous` = older (the typical "load more"). The CLI tool follows `previous` automatically when you bump `--limit > 200` — don't re-implement this in the agent. +- **Date-only ISO strings are interpreted as UTC midnight.** `--from-date 2026-01-01` means `2026-01-01T00:00:00Z`. To capture the full final day in `--to-date`, use `2026-02-01` (exclusive next day) or `2026-01-31T23:59:59.999Z`. +- **The export ZIP's per-day files are JSON, not CSV, and use PascalCase keys.** Different from the camelCase live `events` endpoint. Don't paste an export directly into a parser expecting the live shape. +- **Org sources and tenant sources are different sets.** Don't reuse a GUID from `org sources` in a `tenant events` query — the filter will silently match nothing. diff --git a/tests/tasks/uipath-admin/audit_events_basic_smoke.yaml b/tests/tasks/uipath-admin/audit_events_basic_smoke.yaml new file mode 100644 index 000000000..7aa8922c8 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_events_basic_smoke.yaml @@ -0,0 +1,54 @@ +task_id: skill-admin-audit-events-basic-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to query + audit events with a small bounded time window and a small --limit + that does NOT trigger client-side pagination. Validates the simpler + events form most users will write — `--from-date`/`--to-date` ISO 8601 + bounds, `--limit <= 200`, `--output json`. + + Platform note: runs without an authenticated tenant — commands will + fail with auth errors. That is acceptable; what matters is correct + command invocation with correct flags. +tags: [uipath-admin, smoke, mode:diagnose, audit] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + A reviewer wants the most recent 25 tenant audit events from the + last 24 hours. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit tenant events` with both --from-date and --to-date ISO 8601 bounds" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent passed --limit with a value <= 200 (single-call form, no pagination needed)" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--limit\s+(?:[1-9]|[1-9]\d|1\d\d|200)\b' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent used --output json on the events call" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--output\s+json' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_events_pagination_smoke.yaml b/tests/tasks/uipath-admin/audit_events_pagination_smoke.yaml new file mode 100644 index 000000000..bf68e4ba1 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_events_pagination_smoke.yaml @@ -0,0 +1,53 @@ +task_id: skill-admin-audit-events-pagination-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to query + audit events with a bounded time window and a >200 limit, exercising + the skill's pagination rule. The CLI tool paginates internally for + --limit > 200 (server-side maxCount cap), so the agent should pass + --limit, NOT roll its own cursor loop with --before / --before-id. + + Platform note: commands fail with auth errors in this environment; + what matters is the agent's CLI invocation pattern. +tags: [uipath-admin, smoke, mode:diagnose, audit, pagination] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + A compliance reviewer wants the 500 most recent tenant audit events + from the last 30 days. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit tenant events` with both --from-date and --to-date ISO 8601 bounds" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent passed --limit 500 (or any value > 200) — single call, not a manual loop" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--limit\s+(?:[2-9]\d{2,}|[1-9]\d{3,})' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent used --output json on the events call" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--output\s+json' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_export_basic_smoke.yaml b/tests/tasks/uipath-admin/audit_export_basic_smoke.yaml new file mode 100644 index 000000000..93115f609 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_export_basic_smoke.yaml @@ -0,0 +1,47 @@ +task_id: skill-admin-audit-export-basic-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to run + a minimal `export` against a small window. Validates the three + required flags (`--from-date`, `--to-date`, `--output-file`) and + that the agent picks a reasonable output path inside the working + directory. + + Platform note: runs without an authenticated tenant — commands will + fail with auth errors. That is acceptable; what matters is correct + command invocation with correct flags. +tags: [uipath-admin, smoke, mode:operate, audit, export] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + An admin needs a ZIP of all tenant audit events from yesterday for + archival. Write the ZIP to ./audit-yesterday.zip in the current + working directory. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit tenant export` with --from-date, --to-date, AND --output-file" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+export\b.*--from-date\s+\S+.*--to-date\s+\S+.*--output-file\s+\S+' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent's export targeted the requested ./audit-yesterday.zip path" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+export\b.*--output-file\s+\.?/?audit-yesterday\.zip' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_export_e2e.yaml b/tests/tasks/uipath-admin/audit_export_e2e.yaml new file mode 100644 index 000000000..e54bef8c7 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_export_e2e.yaml @@ -0,0 +1,81 @@ +task_id: skill-admin-audit-export-e2e +description: > + End-to-end skill-guided evaluation: agent uses the uipath-admin skill + to drive a complete investigation — discover sources, query events + in a window, then export the same window to a ZIP. Validates the + full sources → events → export chain plus the export's required + --from-date / --to-date / --output-file flags. + + Platform note: commands fail with auth errors in this environment. + E2e tests still validate the CLI invocation surface — the agent must + produce the right command sequence with the right flags. +tags: [uipath-admin, e2e, mode:operate, audit, export] +max_iterations: 2 + +agent: + type: claude-code + permission_mode: acceptEdits + max_turns: 40 + turn_timeout: 600 + allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep"] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + A security reviewer needs the full audit history for the last 7 days + on the active tenant. They want to: + + 1. See what audit event sources are visible (so they know what's + being captured). + 2. Get a quick preview of the most recent events for context. + 3. Get the entire 7-day window as a ZIP for offline review. + + Output the ZIP to ./audit-last-7d.zip in the current working + directory. + + Do NOT ask for approval, confirmation, or feedback. + Do NOT pause between planning and implementation. + + Important: + - The `uip` CLI is available but is NOT connected to a live tenant. + Commands will fail with auth errors — that is expected. Run each + command exactly once and continue. Do NOT retry or attempt to login. + +success_criteria: + # 1. sources discovery happened first + - type: command_executed + description: "Agent invoked `uip admin audit tenant sources` (discovery before query)" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+sources\b.*--output\s+json' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + # 2. events query with a bounded window + - type: command_executed + description: "Agent invoked `uip admin audit tenant events` with --from-date and --to-date ISO bounds" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + # 3. export with both required bounds AND --output-file + - type: command_executed + description: "Agent invoked `uip admin audit tenant export` with --from-date, --to-date, AND --output-file" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+export\b.*--from-date\s+\S+.*--to-date\s+\S+.*--output-file\s+\S+' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + # 4. export written to the requested path + - type: command_executed + description: "Agent's export targeted ./audit-last-7d.zip" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+export\b.*--output-file\s+\.?/?audit-last-7d\.zip' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_login_history_e2e.yaml b/tests/tasks/uipath-admin/audit_login_history_e2e.yaml new file mode 100644 index 000000000..f7b0ba927 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_login_history_e2e.yaml @@ -0,0 +1,71 @@ +task_id: skill-admin-audit-login-history-e2e +description: > + End-to-end test: agent runs Investigation 2 from + audit-workflow-guide.md ("login history for user X, failed attempts + only"). Sequence is: list `sources` to find the User Login type GUID + (Identity → Authentication → User Login), then query `events` with + `--user-id`, `--type`, and `--status Failure` inside a bounded + window. Validates the agent uses server-side filters instead of + pulling everything and post-filtering. + + Command-construction only — no live tenant available for + side-effect verification. +tags: [uipath-admin, e2e, mode:diagnose, audit, investigation, login-history] +max_iterations: 2 + +agent: + type: claude-code + permission_mode: acceptEdits + max_turns: 40 + turn_timeout: 600 + allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep"] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + Show me failed login attempts for jane.doe@example.com on the + active tenant during April 2026. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit tenant sources` to discover the User Login type GUID" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+sources\b.*--output\s+json' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent invoked `uip admin audit tenant events` with --status Failure" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--status\s+(?i:Failure|1)' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent passed --user-id (or --search to resolve the user) on the events call" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*(?:--user-id\s+\S+|--search\s+\S+)' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent kept a bounded time window via --from-date/--to-date covering April 2026" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_org_events_smoke.yaml b/tests/tasks/uipath-admin/audit_org_events_smoke.yaml new file mode 100644 index 000000000..dcdf220a1 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_org_events_smoke.yaml @@ -0,0 +1,49 @@ +task_id: skill-admin-audit-org-events-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to query + ORG-scoped audit events for a bounded window. Mirrors the tenant-events + smoke but exercises the `org` code path (different basePath, different + event sources). The skill teaches that org-scope is for admin events + (memberships, license, tenant lifecycle) and silently rejects + --tenant-id — this test verifies the agent picks the right scope from + the prompt and does not pass --tenant-id with `org`. + + Platform note: runs without an authenticated tenant — commands will + fail with auth errors. That is acceptable; what matters is correct + command invocation with correct flags. +tags: [uipath-admin, smoke, mode:diagnose, audit, scope-org] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + An admin wants the 50 most recent organization-level audit events + from the last 7 days — they're checking who joined the org and any + recent admin-role changes. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit org events` (org scope, not tenant) with --from-date/--to-date" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+org\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent used --output json on the events call" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+org\s+events\b.*--output\s+json' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_org_export_smoke.yaml b/tests/tasks/uipath-admin/audit_org_export_smoke.yaml new file mode 100644 index 000000000..155c926be --- /dev/null +++ b/tests/tasks/uipath-admin/audit_org_export_smoke.yaml @@ -0,0 +1,48 @@ +task_id: skill-admin-audit-org-export-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to run + `export` at ORG scope (not tenant). Mirrors the tenant-export basic + smoke. Org-scope exports cover admin events (memberships, license, + tenant lifecycle) that don't live under any single tenant — picking + the wrong scope returns the wrong data silently. + + Platform note: runs without an authenticated tenant — commands will + fail with auth errors. That is acceptable; what matters is correct + command invocation with correct flags. +tags: [uipath-admin, smoke, mode:operate, audit, export, scope-org] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + An admin needs a ZIP of all ORGANIZATION-level audit events + (memberships, license changes, tenant lifecycle) from yesterday for + compliance archival. Write the ZIP to ./audit-org-yesterday.zip in + the current working directory. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit org export` (org scope) with --from-date, --to-date, AND --output-file" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+org\s+export\b.*--from-date\s+\S+.*--to-date\s+\S+.*--output-file\s+\S+' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent's export targeted the requested ./audit-org-yesterday.zip path" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+org\s+export\b.*--output-file\s+\.?/?audit-org-yesterday\.zip' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_sources_smoke.yaml b/tests/tasks/uipath-admin/audit_sources_smoke.yaml new file mode 100644 index 000000000..371678c54 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_sources_smoke.yaml @@ -0,0 +1,47 @@ +task_id: skill-admin-audit-sources-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to list audit + event sources at both org and tenant scopes. Tests whether the skill + teaches the correct CLI namespace (`uip admin audit sources`, + not legacy alternatives), the `--output json` convention, and the + scope-discipline rule (org vs tenant basePath). + + Platform note: runs without an authenticated tenant — commands will + fail with auth errors. That is acceptable; what matters is correct + command invocation with correct flags. +tags: [uipath-admin, smoke, mode:diagnose, audit] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + An admin wants to discover what kinds of audit events are visible at + each scope (organization-level and tenant-level) before drilling in. + Fetch the source catalog at BOTH scopes. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent invoked `uip admin audit org sources` with --output json" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+org\s+sources\b.*--output\s+json' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent invoked `uip admin audit tenant sources` with --output json" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+sources\b.*--output\s+json' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_status_filter_smoke.yaml b/tests/tasks/uipath-admin/audit_status_filter_smoke.yaml new file mode 100644 index 000000000..b72f77db2 --- /dev/null +++ b/tests/tasks/uipath-admin/audit_status_filter_smoke.yaml @@ -0,0 +1,53 @@ +task_id: skill-admin-audit-status-filter-smoke +description: > + Skill-guided evaluation: agent uses the uipath-admin skill to filter + events by status (Failure-only). Validates that the agent uses + `--status Failure` (the canonical label) rather than inventing + alternate spellings or omitting the filter and post-filtering in jq. + + Platform note: runs without an authenticated tenant — commands will + fail with auth errors. That is acceptable; what matters is correct + command invocation with correct flags. +tags: [uipath-admin, smoke, mode:diagnose, audit, status] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + Show me all FAILED audit events on the active tenant from the last + 7 days. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent passed --status Failure on the events call" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--status\s+(?i:Failure|1)' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent kept a bounded time window via --from-date/--to-date" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent used --output json" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--output\s+json' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-admin/audit_who_did_x_e2e.yaml b/tests/tasks/uipath-admin/audit_who_did_x_e2e.yaml new file mode 100644 index 000000000..32872de7d --- /dev/null +++ b/tests/tasks/uipath-admin/audit_who_did_x_e2e.yaml @@ -0,0 +1,69 @@ +task_id: skill-admin-audit-who-did-x-e2e +description: > + End-to-end test: agent runs Investigation 1 from audit-workflow-guide.md + ("who did X to resource Y"). Sequence is: discover the source/target + GUIDs via `sources`, then narrow `events` by --source AND --target + inside a bounded time window. Validates the agent uses the skill's + documented workflow rather than scanning everything client-side. + + Command-construction only — no live tenant available for + side-effect verification. +tags: [uipath-admin, e2e, mode:diagnose, audit, investigation] +max_iterations: 2 + +agent: + type: claude-code + permission_mode: acceptEdits + max_turns: 40 + turn_timeout: 600 + allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep"] + +sandbox: + driver: tempdir + python: {} + +initial_prompt: | + Find out who deleted the `Sales-Reports` folder on the active tenant + in the last 7 days. + + Important: + - The `uip` CLI is available but the `admin` subcommand may not be + installed and/or the CLI is not connected to a live tenant. + Commands WILL fail — that is expected and acceptable. + Run each command exactly once regardless of errors. + Do NOT retry, do NOT attempt to login, do NOT try to install tools, + do NOT troubleshoot errors. + - Do NOT prompt the user for confirmation — this is an automated test. + +success_criteria: + - type: command_executed + description: "Agent checked login status before any audit call" + tool_name: "Bash" + command_pattern: 'uip\s+login\s+status' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent invoked `uip admin audit tenant sources` (discovery before query)" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+sources\b.*--output\s+json' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent invoked `uip admin audit tenant events` with a bounded window AND at least one of --source/--target/--type" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+tenant\s+events\b.*--from-date\s+\S+.*--to-date\s+\S+.*--(?:source|target|type)\s+' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent consistently used --output json" + tool_name: "Bash" + command_pattern: 'uip\s+admin\s+audit\s+.*--output\s+json' + min_count: 2 + weight: 1.0 + pass_threshold: 1.0