Skip to content

Commit c5da48c

Browse files
achamayouachamayou
authored
Improve copilot instructions (#7762)
1 parent e1dd74b commit c5da48c

2 files changed

Lines changed: 194 additions & 26 deletions

File tree

‎.github/copilot-instructions.md‎

Lines changed: 120 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -3,44 +3,138 @@
33
- This document provides guidance for AI coding and review agents working in the CCF (Confidential Consortium Framework) repository
44
- **CCF** is an open-source framework for building secure, highly available, and performant applications focused on multi-party compute and data. It's designed for confidential, distributed systems running on secure hardware.
55

6+
## Architecture
7+
8+
CCF is a replicated state machine where application state lives in an in-memory **key-value store** (`src/kv/`). Writes are serialised to an append-only **ledger** and replicated across nodes via the **AFT consensus protocol** (a Raft variant in `src/consensus/aft/`). The node lifecycle — startup, join, recovery, reconfiguration — is managed by the **node state machine** (`src/node/node_state.h`).
9+
10+
Applications are either **C++ endpoint registries** (subclassing `ccf::UserEndpointRegistry`) or **JavaScript/TypeScript bundles** executed by an embedded QuickJS runtime (`src/js/`). Both register HTTP endpoints that read/write the KV store through transaction objects (`ccf::Tx`).
11+
12+
Governance is handled by a built-in **member-driven constitution** system — proposals are submitted as JavaScript and executed against the KV. The crypto subsystem (`src/crypto/`, `include/ccf/crypto/`) wraps OpenSSL for TLS, x.509, COSE signatures, and Merkle tree operations.
13+
614
**Key directories**:
715

8-
- `src/` - Core CCF implementation in C++ (consensus, crypto, KV store, HTTP, TLS, JavaScript runtime), including unit tests in subdirs
9-
- `include/ccf/` - Public C++ API headers
10-
- `tests/` - Python-based end-to-end test suite
11-
- `python/` - CCF Python SDK and client libraries
12-
- `doc/` - Sphinx-based RST documentation
13-
- `3rdparty/` - Third-party dependencies
16+
- `src/` — Core C++ implementation, including unit tests in subdirectories
17+
- `consensus/aft/` — AFT (Raft variant) consensus protocol
18+
- `kv/` — Replicated key-value store and transaction machinery
19+
- `node/` — Node state machine, governance, historical queries, snapshots
20+
- `crypto/` — Cryptographic primitives (OpenSSL wrappers, COSE, Merkle)
21+
- `endpoints/` — HTTP endpoint registration and dispatch
22+
- `js/` — QuickJS-based JavaScript runtime for JS applications
23+
- `http/` — HTTP/1.1 and HTTP/2 parser and session management
24+
- `tls/` — TLS session handling
25+
- `ds/` — Data structures and utilities (logging, serialisation helpers)
26+
- `service/` — Internal service tables and governance tables
27+
- `include/ccf/` — Public C++ API headers (the stable interface for app developers)
28+
- `tests/` — Python-based end-to-end test suite and infrastructure (`tests/infra/`)
29+
- `python/` — CCF Python SDK (ledger parsing, COSE signing, receipts)
30+
- `doc/` — Sphinx-based RST documentation
31+
- `samples/` — Example C++ and JS applications
32+
- `tla/` — TLA+ formal specifications for consensus and disaster recovery
33+
- `cmake/` — CMake build helpers (`common.cmake`, `ccf_app.cmake`)
34+
35+
## Build, test, and lint
36+
37+
### Building
38+
39+
```bash
40+
mkdir build && cd build
41+
cmake -GNinja .. # RelWithDebInfo by default
42+
cmake -GNinja -DCMAKE_BUILD_TYPE=Debug .. # Debug with clang-tidy: add -DCLANG_TIDY=ON
43+
ninja # Build all targets
44+
```
45+
46+
### Testing
47+
48+
See [testing skill](/.github/skills/testing.md) for how to run tests (unit, e2e, SDK), test labels, coverage, and patterns for writing new e2e tests.
49+
50+
### Linting and formatting
51+
52+
See [formatting-and-linting skill](/.github/skills/formatting-and-linting.md) for which scripts to run for each file type, and which support auto-fix.
53+
54+
### Documentation
55+
56+
```bash
57+
pip install -r doc/requirements.txt -r doc/historical_ccf_requirements.txt
58+
sphinx-build --fail-on-warning -b html doc doc/html
59+
```
1460

1561
## Code changes
1662

17-
- Coding style is enforced by the `ci-checks.sh` script, which runs clang-format for C++ and black for Python.
18-
- Linters and static analysis tools are also run as part of CI, clang-tidy for C++ and ruff for Python.
19-
- Run `ci-checks.sh -f` to automatically apply fixes (formatting and supported lint fixes).
2063
- `ci-checks.sh` must run successfully before any commit is pushed.
21-
- Match the existing coding style for naming and casing conventions. This is not automatically enforced, so pay attention to surrounding code for guidance.
2264
- All tests in `ci.yml` must pass before a PR can be merged. Consider which are likely to be affected by your changes and run those locally before pushing.
2365
- Take particular care with any changes that may affect compatibility with older releases, and ensure these are tested, via the `lts_compatibility` test with `LONG_TESTS=1` enabled.
2466
- Take particular care with changes to the consensus and crypto code, as these are critical for security and correctness. Ensure you have a thorough understanding of the existing code and the implications of your changes before proceeding.
25-
- Any changes to user-facing APIs or behaviour must be documented in `CHANGELOG.md`. When adding a new version to `CHANGELOG.md`, be sure to update `pyproject.toml` to match.
67+
- Any changes to user-facing APIs or behaviour must be documented in `CHANGELOG.md` (Keep a Changelog format, under the current `[Unreleased]` or dev version, in `Added`/`Changed`/`Fixed`/`Removed` sections). When adding a new version to `CHANGELOG.md`, be sure to update `python/pyproject.toml` to match.
2668

2769
### C++
2870

29-
- C++ changes must be built and tested locally before creating a PR. Use cmake and ninja to build, and refer to CI files for any further build configurations. For example:
30-
```bash
31-
mkdir build && cd build
32-
cmake -GNinja .. # RelWithDebInfo by default
33-
ninja # Build all targets
34-
```
35-
- Both unit tests and end-to-end tests can be run using ctest, but should be invoked via the `tests.sh` wrapper script to ensure the correct environment is set up and used. For example:
36-
```bash
37-
cd build
38-
./tests.sh # Run all tests
39-
./tests.sh -VV # Verbose output
40-
./tests.sh -R pattern # Run tests matching pattern
41-
```
71+
- C++ changes must be built and tested locally before creating a PR.
4272
- Most changes should be accompanied by new or updated tests. End-to-end tests are required for any changes that affect the user-visible behaviour.
43-
- Use modern features where appropriate, but be wary of newer features that may not be fully supported.
73+
74+
#### Naming conventions
75+
76+
- **Classes/structs**: `PascalCase` (`EndpointRegistry`, `TypedMap`, `NodeState`)
77+
- **Methods/functions**: `snake_case` (`make_endpoint`, `set_auto_schema`, `get_path_param`)
78+
- **Member variables**: `snake_case`, no prefix (`uri_path`, `forwarding_required`)
79+
- **Constants**: `UPPER_SNAKE_CASE` (`PRIVATE_RECORDS`, `JOIN_TIMEOUT`)
80+
- **Namespaces**: `snake_case` (`ccf::kv`, `ccf::endpoints`, `ccf::crypto`)
81+
- **Files**: `snake_case` (`node_state.h`, `endpoint_registry.cpp`)
82+
- **Header guards**: Always `#pragma once`, never `#ifndef`
83+
84+
#### JSON serialisation
85+
86+
Use the `DECLARE_JSON_*` macros from `ccf/ds/json.h` for struct serialisation:
87+
88+
```cpp
89+
DECLARE_JSON_TYPE(MyStruct);
90+
DECLARE_JSON_REQUIRED_FIELDS(MyStruct, field_a, field_b);
91+
92+
DECLARE_JSON_TYPE_WITH_OPTIONAL_FIELDS(MyConfig);
93+
DECLARE_JSON_REQUIRED_FIELDS(MyConfig, name);
94+
DECLARE_JSON_OPTIONAL_FIELDS(MyConfig, description, timeout);
95+
96+
DECLARE_JSON_TYPE_WITH_BASE(DerivedType, BaseType);
97+
DECLARE_JSON_REQUIRED_FIELDS(DerivedType, extra_field);
98+
```
99+
100+
#### Endpoint registration
101+
102+
Endpoints are registered in `init_handlers()` using a fluent builder pattern:
103+
104+
```cpp
105+
make_endpoint("/records/{key}", HTTP_PUT, handler, {ccf::user_cert_auth_policy})
106+
.set_auto_schema<RequestType, ResponseType>()
107+
.set_forwarding_required(ccf::endpoints::ForwardingRequired::Never)
108+
.install();
109+
110+
make_read_only_endpoint("/records/{key}", HTTP_GET, ro_handler, {ccf::user_cert_auth_policy})
111+
.install();
112+
```
113+
114+
Use `make_endpoint` for read-write, `make_read_only_endpoint` for read-only, and `make_command_endpoint` for operations that don't access the KV store.
115+
116+
#### Logging
117+
118+
Use the macro-based logging system. For application code:
119+
120+
```cpp
121+
CCF_APP_INFO("Processing request for key {}", key); // INFO level, "app" tag
122+
CCF_APP_FAIL("Failed to process: {}", error_msg); // FAIL level
123+
CCF_APP_TRACE("Debug detail: {}", detail); // TRACE level
124+
```
125+
126+
For framework-internal code, use `LOG_INFO_FMT`, `LOG_DEBUG_FMT`, `LOG_FAIL_FMT`, `LOG_FATAL_FMT` (from `src/ds/internal_logger.h`). Log levels in order of decreasing verbosity: `TRACE`, `DEBUG`, `INFO`, `FAIL`, `FATAL`.
127+
128+
#### KV store
129+
130+
Maps are typed with key/value serialisers and accessed through transaction handles:
131+
132+
```cpp
133+
using MyMap = ccf::kv::Map<std::string, std::vector<uint8_t>>;
134+
auto* handle = ctx.tx.template rw<MyMap>("my_map"); // Read-write handle
135+
handle->put(key, value);
136+
auto val = handle->get(key); // Returns std::optional
137+
```
44138

45139
### Python
46140

@@ -58,7 +152,7 @@
58152

59153
- **No secrets in code**: Avoid committing API keys, passwords, or other secrets. Some certificates and keys are included in the repository for testing purposes, but if adding more ensure these are freshly created and properly documented as test-only artifacts.
60154
- **Input validation**: Always validate and sanitize external inputs
61-
- **Cryptographic operations**: Use CCF's crypto library (`include/ccf/crypto/`) - don't roll your own
155+
- **Cryptographic operations**: Use CCF's crypto library (`include/ccf/crypto/`) — don't roll your own
62156
- **Memory safety**: Use RAII, smart pointers, and avoid manual memory management
63157

64158
## Reviews

‎.github/skills/testing.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
name: Testing
3+
description: How to run unit tests, end-to-end tests, and Python SDK tests. Covers test labels, the tests.sh wrapper, e2e test infrastructure, and patterns for writing new tests.
4+
---
5+
6+
# Testing
7+
8+
## Running tests
9+
10+
Tests must be run via the `tests.sh` wrapper (in the build directory), which sets up a Python venv, installs the SDK and test dependencies, then invokes `ctest`:
11+
12+
```bash
13+
cd build
14+
./tests.sh # Run all tests
15+
./tests.sh -VV # Verbose output
16+
./tests.sh -R <pattern> # Run tests matching a name regex
17+
./tests.sh -L unit # Run only unit tests
18+
./tests.sh -L e2e # Run only end-to-end tests
19+
./tests.sh -L partitions # Run partition tests (requires NET_ADMIN)
20+
./tests.sh --timeout 360 -R recovery_test # Single e2e test with timeout
21+
```
22+
23+
Test labels: `unit`, `e2e`, `partitions`, `perf`, `benchmark`, `raft_scenario`, `suite`, `lts_compatibility`, `snp`.
24+
25+
Python SDK tests (separate from e2e):
26+
27+
```bash
28+
cd python && pytest
29+
```
30+
31+
## Code coverage
32+
33+
Build with `-DCOVERAGE=ON`, run tests, then:
34+
35+
```bash
36+
scripts/coverage.sh # Print summary
37+
scripts/coverage.sh --html report/ # Generate HTML report
38+
```
39+
40+
## End-to-end test infrastructure
41+
42+
E2e tests use the infrastructure in `tests/infra/`. The key classes are:
43+
44+
- `infra.network.Network` — manages a multi-node CCF network (start, stop, find primary/backup, add/remove nodes)
45+
- `infra.node.Node` — represents a single CCF node process
46+
- `infra.consortium.Consortium` — member governance operations (proposals, votes)
47+
- `infra.runner.ConcurrentRunner` — runs multiple test functions against separate networks in parallel
48+
49+
## Writing e2e tests
50+
51+
Test functions take `(network, args)` parameters and are decorated with requirement annotations:
52+
53+
```python
54+
@reqs.description("Write/Read messages on primary")
55+
@reqs.supports_methods("/app/log/private")
56+
@reqs.at_least_n_nodes(2)
57+
def test_example(network, args):
58+
primary, _ = network.find_primary()
59+
with primary.client("user0") as c:
60+
r = c.post("/app/log/private", body={"id": 42, "msg": "hello"})
61+
assert r.status_code == http.HTTPStatus.OK
62+
return network
63+
```
64+
65+
Tests are assembled in `ConcurrentRunner` at the bottom of test files:
66+
67+
```python
68+
if __name__ == "__main__":
69+
cr = ConcurrentRunner()
70+
cr.add("test_name", test_function, package="samples/apps/logging/logging", nodes=...)
71+
cr.run()
72+
```
73+
74+
When a test needs its own network configuration, deep-copy `const_args`, set a distinct `args.label`, and create a standalone `Network` in a separate `run_*` function.

0 commit comments

Comments
 (0)