Skip to content

Commit 039f009

Browse files
Merge remote-tracking branch 'origin/main' into feat/tx-result-xdr
# Conflicts: # docs/architecture.md # docs/node-semantics.md # docs/notes.md # docs/server.md # src/komet_node/kdist/node.md # src/komet_node/scval.py # src/komet_node/server.py # src/tests/integration/test_server.py
2 parents 64354eb + fa5499c commit 039f009

18 files changed

Lines changed: 3064 additions & 240 deletions

README.md

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -130,16 +130,68 @@ curl -s http://localhost:8000 -H 'Content-Type: application/json' \
130130
"jsonrpc": "2.0",
131131
"id": 1,
132132
"result": [
133+
{
134+
"pos": null, "instr": ["callContract"],
135+
"from": {"type": "address", "addrType": "account", "value": "03a107bff3ce10be1d70dd18e74bc09967e4d6309ba50d5f1ddc8664125531b8"},
136+
"to": {"type": "address", "addrType": "contract", "value": "6a20fec1a9081773a5f23ce370f925f236346e510438ddd6d40f6b2711c134e0"},
137+
"function": "foo", "args":[], "depth":1, "storage":[]
138+
},
133139
{"pos": 3, "instr": ["const", "i32", 1048576], "stack": [], "locals": {}},
134140
{"pos": 11, "instr": ["const", "i32", 1048576], "stack": [], "locals": {}},
135141
{"pos": 19, "instr": ["const", "i32", 1048576], "stack": [], "locals": {}},
136142
{"pos": null, "instr": ["block"], "stack": [], "locals": {}},
137-
{"pos": 3, "instr": ["const", "i64", 2], "stack": [], "locals": {}}
143+
{"pos": 3, "instr": ["const", "i64", 2], "stack": [], "locals": {}},
144+
{"pos": null, "instr": ["endWasm"], "success":true, "depth":1, "result": {"type": "void"}}
138145
]
139146
}
140147
```
141148

142-
Each trace record captures the VM state at instruction entry: `pos` is the instruction's byte offset in the binary (`null` for synthetic instructions), `instr` is the instruction and its operands, and `stack`/`locals` are the value stack and locals as `[type, value]` pairs. See [docs/interpreter.md](docs/interpreter.md) for the full trace format.
149+
A trace can contain five kinds of records:
150+
151+
- `callContract`
152+
- Wasm instruction records
153+
- `hostCall`
154+
- `contractData`
155+
- `endWasm`
156+
157+
The example above only has three of these: `callContract`, instruction records, and `endWasm`. `foo()` doesn't touch storage or call any host functions, so no `contractData` or `hostCall` records show up.
158+
159+
Here's what each record type carries:
160+
161+
- `callContract`: logged for each contract call in the transaction, including contract-to-contract calls. Records the caller, the callee, the function name, the arguments, the call depth, and the callee's storage before the call runs.
162+
- Instruction records: logged at each WebAssembly instruction's entry. `pos` is the instruction's byte offset in the binary (`null` for synthetic instructions), `instr` is the instruction and its operands, and `stack`/`locals` are the value stack and locals as `[type, value]` pairs.
163+
- `hostCall`: logged when the contract calls a host function. `instr` gives `["hostCall", moduleId, functionId]`, identifying which host function ran. `locals` holds the function's arguments, indexed by position. Host calls don't use the stack, so `stack` is absent.
164+
Here's a `hostCall` record for a call to `put_contract_data`, module id `l`, function id `_`:
165+
```jsonc
166+
{
167+
"pos": null,
168+
"instr": ["hostCall", "l", "_"],
169+
"locals": {"2": ["i64",0], "1": ["i64",530242871224172548], "0": ["i64",45954062]}
170+
}
171+
```
172+
173+
- `contractData`: logged for storage updates. Gives the contract and the storage type (`instance`, `persistent`, or `temporary`). A `put` carries the key and value as its two args; a `del` carries only the key.
174+
Here's a `contractData` record for a `put`, followed by a `del` on the same key:
175+
```jsonc
176+
{
177+
"pos": null,
178+
"instr": ["contractData", "put", "temporary"],
179+
"contract": {"type": "address", "addrType": "contract", "value": "746573742d7363"},
180+
"args": [{"type": "symbol", "value": "foo"}, {"type": "u32", "value": 123456789}]
181+
}
182+
{
183+
"pos": null,
184+
"instr": ["contractData", "del", "temporary"],
185+
"contract": {"type": "address", "addrType": "contract", "value": "746573742d7363"},
186+
"args": [{"type": "symbol", "value": "foo"}]
187+
}
188+
```
189+
190+
- `endWasm`: logged once at the end of a call. Records whether the call succeeded, its depth, and its result.
191+
192+
193+
194+
See [docs/interpreter.md](docs/interpreter.md) for the full trace format.
143195

144196
---
145197

docs/architecture.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ flowchart TB
5151

5252
**[Detailed documentation](server.md)**
5353

54-
The server implements six RPC methods — `getHealth`, `getNetwork`, `getLatestLedger`, `sendTransaction`, `getTransaction`, and `traceTransaction` — and the K semantics answer all of them.
54+
The server implements eleven RPC methods — `getHealth`, `getNetwork`, `getLatestLedger`, `getVersionInfo`, `getFeeStats`, `sendTransaction`, `getTransaction`, `getTransactions`, `getLedgers`, `getLedgerEntries`, and the komet-specific `traceTransaction` extension — and the K semantics answer all of them. JSON-RPC framing (single calls, batch arrays, notifications) is handled in Python; the semantics see one request envelope per invocation. For `getLedgerEntries` the semantics' answer is an intermediate shape: K performs the state lookups, and `ledger_entries.py` translates between the base64 XDR wire format (`LedgerKey` in, `LedgerEntryData` out) and the JSON the semantics exchange, since K cannot parse or produce XDR.
5555

5656
`sendTransaction` always returns `PENDING` and clients poll `getTransaction` for the result — matching the Stellar RPC async pattern even though the transaction executes synchronously. See [server.md](server.md) for details.
5757

@@ -91,18 +91,20 @@ All of the server's input and output artifacts live in one directory, the *io di
9191
| `metadata.json` | persistent | the K semantics | `{"latest_ledger": N}` — the server ledger counter, bumped by 1 per committed transaction. |
9292
| `receipts/receipt_<hash>.json` | persistent | the semantics and the server (on success — the server rewrites the semantics' internal `returnValue` field into `resultXdr`/`resultMetaXdr`), or the server alone (on failure) | one stored receipt per transaction, keyed by tx hash, answering `getTransaction`. Each is `{status, ledger, createdAt, envelopeXdr, resultXdr, resultMetaXdr?}`. |
9393
| `traces/trace_<hash>.jsonl` | persistent | the semantics | one execution trace per transaction, keyed by tx hash — the instruction-level records, one JSON object per line. `traceTransaction` returns this file's contents. |
94+
| `ledgers/ledger_<seq>.json` | persistent | the server (on success) | one record per closed ledger — `{sequence, txHash, closedAt, hash, headerXdr, metadataXdr}` — the ledger→transaction index behind `getTransactions` and `getLedgers`. Written in Python because the header artifacts are XDR, which K cannot construct. |
9495
| `requests/request_<n>.json` | persistent | the server | an archive of each incoming JSON-RPC request, numbered by a monotonic counter, kept for debugging. |
96+
| `wasms/<hash>.wasm` | persistent | the server | the raw bytes of each successfully uploaded wasm module, keyed by hex sha256. The K state stores modules parsed (`ModuleDecl`), so `getLedgerEntries` CONTRACT_CODE entries read the original bytes from here. |
9597
| `request.json` | transient | the server | the request envelope for the call in flight (`method`, `id`, `now`, and method-specific fields). The semantics remove it once they respond. |
9698
| `response.json` | transient | the semantics | the JSON-RPC response (`{jsonrpc, id, result}`) for the most recent call. The server reads it back; it is absent when a transaction gets stuck. |
9799

98-
Receipts, traces, and request archives are split into one file per item — keyed by tx hash, or numbered — so that no single file grows without bound as the chain advances. The server creates the `receipts/`, `traces/`, and `requests/` directories before the semantics run, because the K file-system hooks open files with POSIX `open()`, which does not create parent directories.
100+
Receipts, traces, ledger records, and request archives are split into one file per item — keyed by tx hash, ledger sequence, or a counter — so that no single file grows without bound as the chain advances. The server creates the `receipts/`, `traces/`, `ledgers/`, and `requests/` directories before the semantics run, because the K file-system hooks open files with POSIX `open()`, which does not create parent directories.
99101

100102
The world state stays in KORE (rather than a JSON snapshot) because an uploaded wasm module is a `ModuleDecl` that the semantics cannot reconstruct from bytes — only `wasm2kast` (Python) can produce it. The receipts and the ledger counter, by contrast, are plain data and live in JSON files, which the semantics read and write directly via the file-system hooks.
101103

102104
```mermaid
103105
flowchart TB
104106
boot(["server start"]) --> exists{"state.kore exists?"}
105-
exists -->|"no"| init["empty_config() builds the idle K config in KORE<br/>write state.kore · seed metadata.json {latest_ledger: 0}<br/>create receipts/ traces/ requests/"]
107+
exists -->|"no"| init["empty_config() builds the idle K config in KORE<br/>write state.kore · seed metadata.json {latest_ledger: 0}<br/>create receipts/ traces/ requests/ wasms/"]
106108
exists -->|"yes"| reuse["use existing state.kore<br/>seed metadata.json if missing · ensure artifact dirs exist"]
107109
init --> ready(["ready for requests"])
108110
reuse --> ready
@@ -175,7 +177,7 @@ sequenceDiagram
175177

176178
## What's not yet implemented
177179

178-
- `simulateTransaction` (dry-run without state mutation)
179-
- `getEvents`, `getLedgerEntries`, `getFeeStats` and other read-only RPC methods
180+
- `simulateTransaction` of upload/deploy host functions, auth recording, resource metering, `events`/`restorePreamble`/`stateChanges` (only invoke-contract dry runs with the return value are supported)
181+
- `getEvents` and other read-only RPC methods (`getLedgerEntries` is implemented for the entry types the K state tracks: `ACCOUNT`, `CONTRACT_DATA`, `CONTRACT_CODE`)
180182
- `ExtendFootprintTTL` and `RestoreFootprint` operations
181183
- `SCVec` / `SCMap` contract-argument types in the request encoder (`scval_to_json`)

docs/node-semantics.md

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ The semantics communicate with the Python process through files in the working d
1717
| `metadata.json` | K ↔ K | `{"latest_ledger": N}` — the ledger counter |
1818
| `receipts/receipt_<hash>.json` | K → Python | one stored receipt per transaction, keyed by tx hash |
1919
| `traces/trace_<hash>.jsonl` | K → Python | one execution trace per transaction (per-instruction records), keyed by tx hash |
20+
| `ledgers/ledger_<seq>.json` | Python → K | one record per closed ledger (`sequence`, `txHash`, `closedAt`, and the ledger-header XDR artifacts), written by the server and read back to serve `getTransactions`/`getLedgers` |
2021

2122
---
2223

@@ -39,7 +40,8 @@ insert-handleRequestFile → handleRequestFile
3940
4041
#dispatchMethod(method, request) ← routes on the "method" field
4142
42-
├─ getHealth / getNetwork / getLatestLedger / getTransaction / traceTransaction → #respond(...)
43+
├─ getHealth / getNetwork / getLatestLedger / getVersionInfo / getFeeStats
44+
│ / getTransaction / getTransactions / getLedgers / getLedgerEntries / traceTransaction → #respond(...)
4345
4446
└─ sendTransaction → #runTx → run steps
4547
→ #finalizeTx → record receipt + bump ledger → #respond(...)
@@ -56,14 +58,18 @@ If `request.json` is absent, `insert-handleRequestFile` does not fire and K halt
5658

5759
## Dispatch and the read-only methods
5860

59-
`#dispatch` reads the `method` field and routes to a per-method rule. The read-only methods answer directly from constants and the bookkeeping files:
61+
`#dispatch` reads the `method` field and routes to a per-method rule. The read-only methods answer directly from constants and the bookkeeping files. Response shapes follow real stellar-rpc's serialization: ledger sequences and `protocolVersion` are JSON numbers; close-time fields are int64-as-string (JSON strings holding a decimal integer).
6062

61-
- `getHealth``{ "status": "healthy" }`
62-
- `getNetwork``{ "friendbotUrl": null, "passphrase": ..., "protocolVersion": ... }` (passphrase/version come from the request, keeping the semantics network-agnostic)
63-
- `getLatestLedger` → reads `metadata.json` and returns `{ "id": <64 zeros>, "protocolVersion": ..., "sequence": <latest_ledger> }`
64-
- `getTransaction` → reads the hash's `receipts/receipt_<hash>.json` file; returns the stored receipt merged with the current `latestLedger`/`latestLedgerCloseTime`, or `{ "status": "NOT_FOUND", ... }` when the file is absent
63+
- `getHealth``{ "status": "healthy", "latestLedger": N, "latestLedgerCloseTime": <now>, "oldestLedger": 0, "oldestLedgerCloseTime": "0", "ledgerRetentionWindow": N+1 }` (everything since genesis is retained)
64+
- `getNetwork``{ "passphrase": ..., "protocolVersion": ... }` (passphrase/version come from the request, keeping the semantics network-agnostic; `friendbotUrl` is omitted — no friendbot)
65+
- `getLatestLedger` → reads `metadata.json` and returns `{ "id": #ledgerId(seq), "protocolVersion": ..., "sequence": <latest_ledger> }`; `#ledgerId` derives a deterministic per-ledger 64-hex id from the sequence (the semantics have no hash hooks, so it is a fixed odd-constant multiply mod 2^256, which is injective)
66+
- `getVersionInfo` → echoes the version fields the server put into the envelope (`version`, `commitHash`, `buildTimestamp`, `captiveCoreVersion` — package metadata lives on the Python side); `protocolVersion` is a JSON number
67+
- `getFeeStats` → constant `#feeDistribution` objects (komet-node has no fee market) for `sorobanInclusionFee`/`inclusionFee`, plus a live `latestLedger` number from `metadata.json`; all distribution fields except `ledgerCount` are decimal strings, matching real stellar-rpc's Go `,string` encoding
68+
- `getTransaction` → reads the hash's `receipts/receipt_<hash>.json` file; returns the stored receipt merged with the ledger-range fields (`latestLedger`/`latestLedgerCloseTime`/`oldestLedger`/`oldestLedgerCloseTime`), or `{ "status": "NOT_FOUND", ... }` (with the same ledger-range fields) when the file is absent
69+
- `getTransactions` / `getLedgers` → walk the per-ledger index files `ledgers/ledger_<seq>.json` from the envelope's `startSeq` up to the latest ledger, taking at most `limit` records (`#txInfos` / `#ledgerInfos`), and format the history page (`#txHistoryPage` / `#ledgerHistoryPage`). Parameter validation and cursor resolution happen in the server; the response `cursor` (`#pageCursor`) names the page's last record when the page is full — a TOID for transactions, the plain sequence for ledgers — and is empty otherwise. Serialization matches real stellar-rpc, including its quirks: per-transaction `createdAt` is a JSON number (unlike singular `getTransaction`), per-ledger `ledgerCloseTime` is a decimal string, and empty `resultXdr`/`resultMetaXdr` stubs are omitted (`#optXdrEntry`)
70+
- `getLedgerEntries` → looks each *key descriptor* of the request up in the world-state cells (`<accounts>`, `<contracts>`, `<contractData>`, `<contractCodes>`) and responds with `{ "entries": [...], "latestLedger": <int> }`. The server decodes the base64 `LedgerKey` XDR into the descriptors beforehand and re-encodes the found entries as `LedgerEntryData` XDR afterwards (`ledger_entries.py`) — the entries in K's response are an intermediate JSON shape (per-kind payloads such as `balance`, `wasmHash`, or an ScVal value serialised by `#scVal2JSON`, the inverse of `#decodeArg`). Keys that match nothing are skipped via an `[owise]` rule — per the spec they are not an error
6571

66-
`#respond(ID, RESULT)` is the shared terminal: it writes the JSON-RPC envelope to `response.json`, removes `request.json`, and sets the exit code to 0.
72+
`#respond(ID, RESULT)` is the shared terminal: it writes the JSON-RPC envelope to `response.json`, removes `request.json`, and sets the exit code to 0. `#respondError(ID, CODE, MSG)` is its error counterpart, writing `{jsonrpc, id, error: {code, message}}` instead; the unknown-method fallback uses it to answer `-32601 Method not found` (a safety net — the Python server filters unknown methods before they reach K).
6773

6874
---
6975

@@ -83,18 +89,20 @@ If `request.json` is absent, `insert-handleRequestFile` does not fire and K halt
8389

8490
1. writes `metadata.json` with `latest_ledger + 1`,
8591
2. writes the receipt to `receipts/receipt_<hash>.json`:
86-
`{ status: "SUCCESS", ledger, createdAt, envelopeXdr, returnValue }``returnValue` is
92+
`{ status: "SUCCESS", applicationOrder: 1, feeBump: false, envelopeXdr, ledger, createdAt, returnValue }``returnValue` is
8793
the contract call's return `ScVal`, JSON-encoded by `#scValToJSON` (or `null` when the
8894
transaction made no contract call), read off the `<hostStack>` before the host is reset,
8995
3. responds with `{hash, status: "PENDING", latestLedger, latestLedgerCloseTime}`.
9096

9197
The internal `returnValue` field never reaches an RPC client: the Python server immediately rewrites it into the spec-mandated `resultXdr`/`resultMetaXdr` base64 XDR fields (`_attach_result_xdr` in `server.py`), because K cannot construct XDR. To keep the return value observable here, `uncheckedCallTx` (unlike komet's `callTx`) does not `#resetHost` after the call; `#recordAndRespond` serialises the stack top and resets the host instead.
9298

99+
A `sendTransaction` whose hash already has a receipt never reaches `#runTx`: dispatch answers `{hash, status: "DUPLICATE", latestLedger, latestLedgerCloseTime}` directly, without executing anything or advancing the ledger. (For wasm uploads the Python server also suppresses the `<program>` injection on duplicates, since those steps would run before dispatch.) The Python `_attach_result_xdr` rewrite is skipped for duplicates, so a resubmission never disturbs the stored receipt.
100+
93101
The trace is not part of the receipt — the executing steps already appended it to `traces/trace_<hash>.jsonl`. Reaching `#finalizeTx` means the steps completed without getting stuck, so the status is `SUCCESS`. A failed transaction gets stuck before this point, `response.json` is never written, and the Python server records the `FAILED` receipt instead.
94102

95103
### traceTransaction
96104

97-
`traceTransaction` is a read-only lookup. It takes a `hash` (the same parameter `getTransaction` takes) and responds with the contents of `traces/trace_<hash>.jsonl`, or `null` when no trace file exists for that hash. Because tracing is always on, every `sendTransaction` writes this file.
105+
`traceTransaction` is a komet-specific extension, not part of the Stellar RPC specification (see [server.md](server.md#tracetransaction-komet-specific-extension)). It is a read-only lookup: it takes a `hash` (the same parameter `getTransaction` takes) and responds with the contents of `traces/trace_<hash>.jsonl`, or `null` when no trace file exists for that hash. Because tracing is always on, every `sendTransaction` writes this file.
98106

99107
### Two ways steps are delivered
100108

@@ -109,7 +117,8 @@ The trace is not part of the receipt — the executing steps already appended it
109117

110118
- `#getJSON(key, obj[, default])`, `#getString(key, obj)`, `#getInt(key, obj)` — read a field
111119
- `#concatJSONs(a, b)` — append object entries (used to merge `latestLedger` fields into a stored receipt)
112-
- `#receiptFile(hash)`, `#traceFile(hash)` — build the per-transaction file paths (`receipts/receipt_<hash>.json`, `traces/trace_<hash>.jsonl`)
120+
- `#receiptFile(hash)`, `#traceFile(hash)`, `#ledgerFile(seq)` — build the per-item file paths (`receipts/receipt_<hash>.json`, `traces/trace_<hash>.jsonl`, `ledgers/ledger_<seq>.json`)
121+
- `#readJSONFile(path)`, `#asInt(json)`, `#lengthJSONs(list)`, `#lastIntIn(key, list)` — small conveniences for the history methods
113122

114123
These complement the **order-sensitive** step decoders below.
115124

@@ -145,6 +154,10 @@ rule HexBytes(S) => Int2Bytes(lengthString(S) /Int 2, String2Base(S, 16), BE)
145154
requires lengthString(S) >Int 0
146155
```
147156

157+
### `Bytes2Hex(Bytes) → String`
158+
159+
The inverse direction is K's built-in `Bytes2Hex` (hook `BYTES.bytes2hex`): it encodes `Bytes` as a lowercase, zero-padded hex string. The `getLedgerEntries` rules use it to report hashes, addresses, and `ScBytes` values to the server.
160+
148161
### `string2WasmToken(String) → WasmStringToken`
149162

150163
`string2WasmToken` wraps a K `String` into a `WasmStringToken` (`hook(STRING.string2token)`). It is required because `callTx` expects a `WasmString` for the function name.

0 commit comments

Comments
 (0)