Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ test = [
"pytest>=8.0",
"pytest-asyncio>=0.24",
"aioresponses>=0.7",
# aioresponses 0.7.8 is not compatible with aiohttp 3.14's ClientResponse constructor.
"aiohttp<3.14",
"pytest-cov>=6.0",
]

Expand All @@ -43,4 +45,4 @@ asyncio_mode = "auto"

[project.urls]
Homepage = "https://github.com/Autohive-AI/integrations-sdk"
Issues = "https://github.com/Autohive-AI/integrations-sdk/issues"
Issues = "https://github.com/Autohive-AI/integrations-sdk/issues"
13 changes: 11 additions & 2 deletions skills/building-integration/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,16 @@ Only runs tests in `test_*_unit.py` files (configured in `pyproject.toml`).

If the integration has `test_*_integration.py` files, check whether they reference environment variables via `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers. Every referenced variable must appear as a blank template entry in the repository root `.env.example`.

This is a manual review/build requirement: `.env` stays local and uncommitted, but `.env.example` is the committed contract that tells humans, reviewers, and automation what to provide.
This is a manual review/build requirement: `.env` stays local and uncommitted, but `.env.example` is the committed contract that tells humans, reviewers, and automation what to provide. Missing entries should be treated as a validation failure for PRs adding or changing integration tests.

Use a focused cross-check:

```bash
grep -RInE 'env_credentials\(|os\.environ|getenv\(' <name>/tests/test_*_integration.py
grep -nE 'INTEGRATION_PREFIX|SERVICE_PREFIX' .env.example
```

Document required credentials, optional test IDs, and destructive-test-only variables. If an env var read is unused, remove it rather than documenting unnecessary setup.

### Step 6 — Auto-fix lint and formatting issues

Expand Down Expand Up @@ -201,4 +210,4 @@ Key settings:
| Config-code sync | Ensure actions in `config.json` match handlers in source code |
| Missing test files | Create `tests/test_<name>_unit.py` (see `writing-unit-tests` skill) |
| Import errors | Check `requirements.txt` has all dependencies; check `__init__.py` exports |
| Integration tests use env vars but `.env.example` missing entries | Add blank entries for every referenced env var to root `.env.example` |
| Integration tests use env vars but `.env.example` missing entries | Add blank entries for every referenced env var to root `.env.example`; treat as blocking for PRs adding/changing integration tests |
19 changes: 15 additions & 4 deletions skills/reviewing-integration-prs/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The goal is not just a generic code review. Validate the PR against the relevant
- Review the actual diff and the surrounding integration code; do not rely only on CI status.
- Treat security and secret leakage as blockers.
- Treat failed tooling/validation as blockers unless there is a documented, intentional exception.
- Treat missing root `.env.example` entries as blockers when a PR adds or changes integration tests that read env vars.
- Use the existing skills as sub-skill checklists when their context applies.
- Prefer actionable review comments with file/line references and concrete fixes.
- Separate blockers from should-fix items and minor suggestions.
Expand Down Expand Up @@ -104,6 +105,8 @@ For migrations to public, apply the `migrating-private-integration` security/saf

### 5. Tests and `.env.example`

The root `.env.example` check is mandatory for any PR that adds or changes `test_*_integration.py`. Do not rely on CI to catch this; review it manually against the test file.

For unit tests, apply `writing-unit-tests`:

- Tests are named `test_*_unit.py`.
Expand All @@ -118,8 +121,16 @@ For integration tests, apply `writing-integration-tests`:
- `live_context` returns `FetchResponse` for SDK 2.x `context.fetch` paths.
- Tests skip cleanly when credentials or required test IDs are missing.
- Every env var read by `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers is listed as a blank template entry in root `.env.example`.
- Optional test IDs, destructive-test-only variables, and module-level env constants count; either document them in root `.env.example` or remove unused reads.

Use a focused search while reviewing:

```bash
grep -RInE 'env_credentials\(|os\.environ|getenv\(' <integration>/tests/test_*_integration.py
grep -nE 'INTEGRATION_PREFIX|SERVICE_PREFIX' .env.example
```

Missing `.env.example` entries are a should-fix at minimum, and often a blocker for PRs whose main purpose is adding/changing integration tests.
Missing `.env.example` entries are blockers when the PR adds or changes integration tests that read env vars. If the PR only exposes a pre-existing omission outside the changed tests, call it a should-fix unless it blocks the changed behavior.

### 6. SDK v2 upgrade review

Expand Down Expand Up @@ -147,8 +158,8 @@ Check:

Use these categories:

- **🚫 Blocker** — must fix before merge. Security leaks, broken validation, schema/action mismatch, new integration input drift, missing required files, broken imports/tests, unsafe public migration.
- **⚠️ Should-fix** — strongly recommended before merge. Missing `.env.example` entries, incomplete test coverage for changed behavior, unclear docs, overly broad error handling, avoidable warnings.
- **🚫 Blocker** — must fix before merge. Security leaks, broken validation, schema/action mismatch, new integration input drift, missing required files, broken imports/tests, unsafe public migration, or missing root `.env.example` entries for env vars read by newly added/changed integration tests.
- **⚠️ Should-fix** — strongly recommended before merge. Pre-existing `.env.example` omissions not introduced by the PR, incomplete test coverage for changed behavior, unclear docs, overly broad error handling, avoidable warnings.
- **💡 Suggestion** — optional improvements. Naming clarity, small refactors, additional examples, extra edge-case tests.

## Review Output Template
Expand Down Expand Up @@ -184,5 +195,5 @@ Before completing the review, verify:
- [ ] Tooling/CI status was checked or local validation was run.
- [ ] Config/code/schema sync was inspected.
- [ ] Unit/integration test changes were reviewed against the appropriate test skill.
- [ ] Root `.env.example` was checked when integration tests reference env vars.
- [ ] Root `.env.example` was checked when integration tests reference env vars; missing entries introduced by the PR were treated as blockers.
- [ ] Findings are actionable and severity-labelled.
6 changes: 3 additions & 3 deletions skills/upgrading-sdk-v2/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ async def real_fetch(url, *, method="GET", json=None, headers=None, **kwargs):
)
```

If the integration tests read credentials or test resource IDs from environment variables, also verify the repository root `.env.example` lists every variable used by those tests. Add missing entries in the same PR. This applies to variables read through `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers. The local `.env` file remains uncommitted; `.env.example` is the committed setup contract for humans and tooling.
If the integration tests read credentials or test resource IDs from environment variables, also verify the repository root `.env.example` lists every variable used by those tests. Add missing entries in the same PR. This applies to variables read through `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers. Required credentials, optional test IDs, and destructive-test-only variables all count. The local `.env` file remains uncommitted; `.env.example` is the committed setup contract for humans and tooling. Missing entries should block SDK-upgrade PRs that add or change integration tests.

### Step 7 — Local validation (required before pushing)

Expand Down Expand Up @@ -265,7 +265,7 @@ Before considering an integration upgraded, verify:
- [ ] Unit test error assertions use `result.type == ResultType.ACTION_ERROR` and `result.result.message`
- [ ] `pytest.raises(ValidationError)` replaced with `result.type == ResultType.VALIDATION_ERROR`
- [ ] Integration test `real_fetch` returns `FetchResponse(...)` (if applicable)
- [ ] Integration test env vars are documented in root `.env.example` (if integration tests read env vars)
- [ ] Integration test env vars are documented in root `.env.example` (if integration tests read env vars, including optional and destructive-test-only variables)
- [ ] `FetchResponse` and `ResultType` are imported where needed
- [ ] All unit tests pass
- [ ] `ruff check` and `ruff format --config ../autohive-integrations-tooling/ruff.toml` pass
Expand Down Expand Up @@ -303,4 +303,4 @@ Before considering an integration upgraded, verify:

10. **PyPI package name collision**: If your integration folder name matches a PyPI package the source imports (e.g. an integration in `supadata/` that does `from supadata import Supadata`), an empty `<integration>/__init__.py` will shadow the real PyPI package and every test fails with `ImportError`. Drop `<integration>/__init__.py` — the validator's "missing __init__.py" warning is correct to ignore in this case, and the Lambda runtime is unaffected (the entry point is the action source file, not the package). See the `writing-unit-tests` skill for the matching test-side guidance.

11. **Missing `.env.example` entries after touching integration tests**: SDK upgrades often touch `live_context` and credential handling. If the integration test file references `FOO_ACCESS_TOKEN`, `FOO_TEST_ID`, or similar, root `.env.example` must contain `FOO_ACCESS_TOKEN=`, `FOO_TEST_ID=`, etc. Do not rely on future contributors or automation to grep test code for setup requirements.
11. **Missing `.env.example` entries after touching integration tests**: SDK upgrades often touch `live_context` and credential handling. If the integration test file references `FOO_ACCESS_TOKEN`, `FOO_TEST_ID`, or similar, root `.env.example` must contain `FOO_ACCESS_TOKEN=`, `FOO_TEST_ID=`, etc. Optional IDs and destructive-test-only variables count too. Do not rely on future contributors or automation to grep test code for setup requirements; missing entries should block PRs that add or change those tests.
29 changes: 25 additions & 4 deletions skills/writing-integration-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,18 @@ description: "Writes pytest end-to-end integration tests for an Autohive integra

# Writing Integration Tests for an Integration

## Required: root `.env.example` update

If a PR adds or changes integration tests that read environment variables, it must update the repository root `.env.example` in the same PR. This is part of the integration-test deliverable, not optional documentation.

This applies to every variable read through `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers, including:

- Required credentials such as access tokens and API keys
- Optional test resource IDs used to avoid list→get chaining
- Variables used only by destructive tests

If a variable is read by the test file, either add a blank template entry to root `.env.example` or remove the unused env-var read. Missing entries should block integration-test PRs until fixed.

## Prerequisites

- The integration must be on **SDK 2.0.0** (`autohive-integrations-sdk~=2.0.0` in `requirements.txt`)
Expand Down Expand Up @@ -134,7 +146,16 @@ Why this matters:
- Automated and AI tooling can discover which credentials, test IDs, and optional knobs it needs to ask for.
- Reviewers have one canonical place to verify the local test contract.

Add every variable used by `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers. Include both required credentials and optional test resource IDs. Keep real secrets only in local `.env`; never commit `.env`.
Add every variable used by `env_credentials(...)`, `os.environ.get(...)`, `os.getenv(...)`, `os.environ[...]`, or equivalent helpers. Include required credentials, optional test resource IDs, and destructive-test-only variables. Keep real secrets only in local `.env`; never commit `.env`.

Before finishing, run a focused cross-check:

```bash
grep -RInE 'env_credentials\(|os\.environ|getenv\(' <integration>/tests/test_*_integration.py
grep -nE 'MYINTEGRATION|SERVICE_PREFIX' .env.example
```

Every env var found in the integration test file must be present as a blank template entry in root `.env.example`. If the variable is no longer used by tests, remove the env-var read instead of documenting unnecessary setup.

Use a clearly labeled block in root `.env.example`:

Expand All @@ -145,7 +166,7 @@ MYINTEGRATION_TEST_ITEM_ID=
MYINTEGRATION_TEST_PROJECT_ID=
```

Before finishing, cross-check the integration test file against `.env.example`: every env var referenced in tests must appear in `.env.example`, even if the test skips when it is missing.
Before finishing, cross-check the integration test file against `.env.example`: every env var referenced in tests must appear in `.env.example`, even if the test skips when it is missing or the variable is optional.

## The live_context Fixture

Expand Down Expand Up @@ -466,7 +487,7 @@ Write actions (create/update/delete) need at least:
1. **Read the integration source** — understand each action and its auth mechanism
2. **Check `config.json`** — determine auth type (none / API key / platform OAuth)
3. **Choose the right `live_context` variant** — see the three variants above
4. **Document env vars in root `.env.example` (required)** — add every env var referenced by the integration test file in the same PR
4. **Document env vars in root `.env.example` (required)** — add every env var referenced by the integration test file in the same PR; missing entries should block integration-test PRs
5. **Write read-only tests first** — these are safe to run repeatedly
6. **Add destructive tests** with `@pytest.mark.destructive` for write actions
7. **Run read-only tests**:
Expand Down Expand Up @@ -494,7 +515,7 @@ Write actions (create/update/delete) need at least:

5. **Destructive test cleanup**: Always clean up created data at the end of lifecycle tests. Use `os.getpid()` in created object names to avoid collisions when running tests in parallel.

6. **Missing `.env.example` entries**: If you add `env_credentials("FOO")`, `os.environ.get("FOO")`, or similar test env-var usage, add `FOO=` to root `.env.example` before you commit. Do not make reviewers or future contributors grep test files to discover setup requirements.
6. **Missing `.env.example` entries**: If you add `env_credentials("FOO")`, `os.environ.get("FOO")`, `os.getenv("FOO")`, `os.environ["FOO"]`, or similar test env-var usage, add `FOO=` to root `.env.example` before you commit. Do not make reviewers or future contributors grep test files to discover setup requirements. Optional IDs and destructive-test-only variables count too.

7. **OAuth token expiry**: Platform OAuth tokens expire. Document the refresh process in the integration's README or root `.env.example`.

Expand Down
Loading