Skip to content

Commit 0e2a429

Browse files
achamayoumaxtropetsCopiloteddyashton
authored
Ledger Chunk download API (#7550)
Co-authored-by: Max <maxtropets@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Eddy Ashton <edashton@microsoft.com>
1 parent cfe79fe commit 0e2a429

21 files changed

Lines changed: 1138 additions & 224 deletions

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
66
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
77

8+
## [7.0.0-dev10]
9+
10+
[7.0.0-dev10]: https://github.com/microsoft/CCF/releases/tag/ccf-7.0.0-dev10
11+
12+
### Added
13+
14+
- `GET` and `HEAD` `/node/ledger-chunk?since={seqno}` and `/node/ledger-chunk/{chunk_name}` endpoints, gated by the `LedgerChunkDownload` RPC interface operator feature. See [documentation](https://microsoft.github.io/CCF/main/operations/ledger_snapshot.html#download-endpoints) for more detail.
15+
816
## [7.0.0-dev9]
917

1018
[7.0.0-dev9]: https://github.com/microsoft/CCF/releases/tag/ccf-7.0.0-dev9

‎doc/operations/ledger_snapshot.rst‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,74 @@ The listing below is an example of what a ledger directory may look like:
4444
- While the :doc:`/operations/recovery` procedure is in progress, new ledger files are suffixed with ``.recovery``. These files are automatically renamed (i.e. recovery suffix removed) once the recovery procedure is complete. ``.recovery`` files are automatically discarded on node startup so that a failed recovery attempt does not prevent further recoveries.
4545
- A new ledger chunk can also be created by the ``trigger_ledger_chunk`` governance action, which will automatically produce a new chunk at the following signature transaction.
4646

47+
Download Endpoints
48+
~~~~~~~~~~~~~~~~~~
49+
50+
In order to faciliate long term backup of the ledger files (also called chunks), nodes can enable HTTP endpoints that allow a client to download committed ledger files.
51+
The `LedgerChunkDownload` feature must be added to `enabled_operator_features` on the relevant `rpc_interfaces` entries in the node configuration.
52+
53+
1. :http:GET:`/node/ledger-chunk/{chunk_name}` and :http:HEAD:`/node/ledger-chunk/{chunk_name}`
54+
55+
These endpoints allow downloading a specific ledger chunk by name, where `<chunk-name>` is of the form `ledger_<start_seqno>-<end_seqno>.committed`.
56+
They support the HTTP `Range` header for partial downloads, and the `HEAD` method for clients to query metadata such as the total size without downloading the full chunk.
57+
They also populate the `x-ms-ccf-ledger-chunk-name` response header with the name of the chunk being served.
58+
59+
2. :http:GET:`/node/ledger-chunk` and :http:HEAD:`/node/ledger-chunk`, both taking a `seqno` query parameter.
60+
61+
These endpoints can be used by a client to download the next ledger chunk including a given sequence number `<seqno>`.
62+
The redirects to the appropriate chunk if it exists, using the previous set of endpoints, or returns a `404 Not Found` response if no such chunk is available.
63+
64+
In the usual case, a downloading client will first hit a Backup, and will eventually want to download files recent enough that only the primary can provide them:
65+
66+
.. mermaid::
67+
68+
sequenceDiagram
69+
Note over Client: Client asks for chunk starting at index
70+
Client->>+Backup: GET /node/ledger-chunk?since=index
71+
Backup->>-Client: 308 Location: /node/ledger-chunk/ledger_startIndex_endIndex.committed
72+
Note over Backup: Backup node has that chunk
73+
Client->>+Backup: GET /node/ledger-chunk/ledger_startIndex_endIndex.committed
74+
Backup->>-Client: 200 <Chunk Contents>
75+
Client->>+Backup: GET /node/ledger-chunk?since=endIndex+1
76+
Note over Backup: Backup node does not yet have a committed chunk starting at endIndex+1
77+
Backup->>-Client: 308 Location: https://primary/node/ledger-chunk?since=endIndex+1
78+
Client->>+Primary: GET /node/ledger-chunk?since=endIndex+1
79+
Primary->>-Client: 308 Location: /node/ledger-chunk/ledger_endIndex+1_nextEndIndex.committed
80+
Client->>+Primary: GET /node/ledger-chunk/ledger_startIndex_endIndex.committed
81+
Note over Primary: But the Primary node has the most recent chunk already
82+
Primary->>-Client: 200 <Chunk Contents>
83+
84+
But it is also possible for a client to first hit a node that has recently started from a snapshot, and does not have some past chunks as a result.
85+
If the Primary started from `snapshot_100.committed` and locally has:
86+
87+
.. code-block:: bash
88+
89+
ledger_1-50.committed
90+
ledger_101-150.committed
91+
92+
and Backup has:
93+
94+
.. code-block:: bash
95+
96+
ledger_1-50.committed
97+
ledger_51-100.committed
98+
99+
then the following sequence can occur:
100+
101+
.. mermaid::
102+
103+
sequenceDiagram
104+
Client->>+Primary: GET /node/ledger-chunk?since=51
105+
Primary->>-Client: 308 Location: https://backup/node/ledger-chunk?since=51
106+
Client->>+Backup: GET /node/ledger-chunk?since=51
107+
Backup->>-Client: 308 Location: /node/ledger-chunk/ledger_51-100.committed
108+
Client->>+Backup: GET /node/ledger-chunk/ledger_51-100.committed
109+
Backup->>-Client: 200 <Chunk Contents>
110+
Client->>+Backup: GET /node/ledger-chunk?since=101
111+
Note over Backup: Backup node does not have 101-150
112+
Backup->>-Client: 308 Location: https://primary/node/ledger-chunk?since=51
113+
Client->>+Primary: GET /node/ledger-chunk?since=101
114+
47115
Snapshots
48116
---------
49117

‎doc/schemas/node_openapi.json‎

Lines changed: 99 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -918,7 +918,7 @@
918918
"info": {
919919
"description": "This API provides public, uncredentialed access to service and node state.",
920920
"title": "CCF Public Node API",
921-
"version": "4.16.0"
921+
"version": "5.0.3"
922922
},
923923
"openapi": "3.0.0",
924924
"paths": {
@@ -1166,6 +1166,104 @@
11661166
}
11671167
}
11681168
},
1169+
"/node/ledger-chunk": {
1170+
"get": {
1171+
"description": "Redirect to the corresponding /node/ledger-chunk/{chunk_name} endpoint for the ledger chunk including the sequence number specified in the 'since' query parameter.",
1172+
"operationId": "GetNodeLedgerChunk",
1173+
"parameters": [
1174+
{
1175+
"in": "query",
1176+
"name": "since",
1177+
"required": true,
1178+
"schema": {
1179+
"$ref": "#/components/schemas/uint64"
1180+
}
1181+
}
1182+
],
1183+
"responses": {
1184+
"200": {
1185+
"description": "Default response description"
1186+
},
1187+
"default": {
1188+
"$ref": "#/components/responses/default"
1189+
}
1190+
},
1191+
"summary": "Download ledger chunk",
1192+
"x-ccf-forwarding": {
1193+
"$ref": "#/components/x-ccf-forwarding/never"
1194+
}
1195+
},
1196+
"head": {
1197+
"description": "Redirect to the corresponding /node/ledger-chunk/{chunk_name} endpoint for the ledger chunk including the sequence number specified in the 'since' query parameter.",
1198+
"operationId": "HeadNodeLedgerChunk",
1199+
"parameters": [
1200+
{
1201+
"in": "query",
1202+
"name": "since",
1203+
"required": true,
1204+
"schema": {
1205+
"$ref": "#/components/schemas/uint64"
1206+
}
1207+
}
1208+
],
1209+
"responses": {
1210+
"200": {
1211+
"description": "Default response description"
1212+
},
1213+
"default": {
1214+
"$ref": "#/components/responses/default"
1215+
}
1216+
},
1217+
"summary": "Ledger chunk metadata",
1218+
"x-ccf-forwarding": {
1219+
"$ref": "#/components/x-ccf-forwarding/never"
1220+
}
1221+
}
1222+
},
1223+
"/node/ledger-chunk/{chunk_name}": {
1224+
"get": {
1225+
"description": "Download a specific ledger chunk by name. Supports HTTP Range header for partial downloads.",
1226+
"operationId": "GetNodeLedgerChunkChunkName",
1227+
"responses": {
1228+
"200": {
1229+
"description": "Default response description"
1230+
},
1231+
"default": {
1232+
"$ref": "#/components/responses/default"
1233+
}
1234+
},
1235+
"summary": "Download ledger chunk",
1236+
"x-ccf-forwarding": {
1237+
"$ref": "#/components/x-ccf-forwarding/never"
1238+
}
1239+
},
1240+
"head": {
1241+
"description": "Metadata about a specific ledger chunk (Content-Length and x-ms-ccf-ledger-chunk-name)",
1242+
"operationId": "HeadNodeLedgerChunkChunkName",
1243+
"responses": {
1244+
"200": {
1245+
"description": "Default response description"
1246+
},
1247+
"default": {
1248+
"$ref": "#/components/responses/default"
1249+
}
1250+
},
1251+
"summary": "Ledger chunk metadata",
1252+
"x-ccf-forwarding": {
1253+
"$ref": "#/components/x-ccf-forwarding/never"
1254+
}
1255+
},
1256+
"parameters": [
1257+
{
1258+
"in": "path",
1259+
"name": "chunk_name",
1260+
"required": true,
1261+
"schema": {
1262+
"type": "string"
1263+
}
1264+
}
1265+
]
1266+
},
11691267
"/node/memory": {
11701268
"get": {
11711269
"operationId": "GetNodeMemory",

‎include/ccf/http_consts.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ namespace ccf
2626

2727
static constexpr auto CCF_TX_ID = "x-ms-ccf-transaction-id";
2828
static constexpr auto CCF_SNAPSHOT_NAME = "x-ms-ccf-snapshot-name";
29+
static constexpr auto CCF_LEDGER_CHUNK_NAME =
30+
"x-ms-ccf-ledger-chunk-name";
2931
}
3032

3133
namespace headervalues::contenttype

‎js/ccf-app/package.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,5 +37,8 @@
3737
},
3838
"bin": {
3939
"ccf-build-bundle": "scripts/build_bundle.js"
40+
},
41+
"dependencies": {
42+
"colors": "1.4.0"
4043
}
4144
}

‎python/pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "ccf"
7-
version = "7.0.0-dev9"
7+
version = "7.0.0-dev10"
88
authors = [
99
{ name="CCF Team", email="CCF-Sec@microsoft.com" },
1010
]

‎src/enclave/enclave.h‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
#include "ds/internal_logger.h"
1111
#include "ds/oversized.h"
1212
#include "ds/work_beacon.h"
13+
#include "host/ledger.h"
1314
#include "indexing/enclave_lfs_access.h"
1415
#include "indexing/historical_transaction_fetcher.h"
1516
#include "interface.h"
@@ -23,6 +24,7 @@
2324
#include "node/rpc/custom_protocol_subsystem.h"
2425
#include "node/rpc/forwarder.h"
2526
#include "node/rpc/gov_effects.h"
27+
#include "node/rpc/ledger_subsystem.h"
2628
#include "node/rpc/member_frontend.h"
2729
#include "node/rpc/network_identity_subsystem.h"
2830
#include "node/rpc/node_frontend.h"
@@ -81,7 +83,8 @@ namespace ccf
8183
size_t chunk_threshold,
8284
const ccf::consensus::Configuration& consensus_config,
8385
const ccf::crypto::CurveID& curve_id,
84-
ccf::ds::WorkBeaconPtr work_beacon_) :
86+
ccf::ds::WorkBeaconPtr work_beacon_,
87+
asynchost::Ledger& ledger_) :
8588
circuit(std::move(circuit_)),
8689
basic_writer_factory(std::move(basic_writer_factory_)),
8790
writer_factory(std::move(writer_factory_)),
@@ -135,6 +138,10 @@ namespace ccf
135138
context->install_subsystem(cpss);
136139
rpcsessions->set_custom_protocol_subsystem(cpss);
137140

141+
auto ledger_subsystem =
142+
std::make_shared<ccf::ReadLedgerSubsystem>(ledger_);
143+
context->install_subsystem(ledger_subsystem);
144+
138145
static constexpr size_t max_interpreter_cache_size = 10;
139146
auto interpreter_cache =
140147
std::make_shared<ccf::js::InterpreterCache>(max_interpreter_cache_size);

‎src/enclave/entry_points.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
#include "common/enclave_interface_types.h"
66
#include "ds/work_beacon.h"
7+
#include "host/ledger.h"
78

89
#include <cstdint>
910

@@ -18,7 +19,8 @@ namespace ccf
1819
StartType start_type,
1920
ccf::LoggerLevel log_level,
2021
size_t num_worker_thread,
21-
const ccf::ds::WorkBeaconPtr& work_beacon);
22+
const ccf::ds::WorkBeaconPtr& work_beacon,
23+
asynchost::Ledger& ledger);
2224

2325
bool enclave_run();
2426
}

‎src/enclave/main.cpp‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
#include "common/enclave_interface_types.h"
88
#include "ds/internal_logger.h"
99
#include "enclave.h"
10+
#include "host/ledger.h"
1011

1112
#include <chrono>
1213
#include <cstdint>
@@ -33,7 +34,8 @@ namespace ccf
3334
StartType start_type,
3435
ccf::LoggerLevel log_level,
3536
size_t num_worker_threads,
36-
const ccf::ds::WorkBeaconPtr& work_beacon)
37+
const ccf::ds::WorkBeaconPtr& work_beacon,
38+
asynchost::Ledger& ledger)
3739
{
3840
std::lock_guard<ccf::pal::Mutex> guard(create_lock);
3941

@@ -107,7 +109,8 @@ namespace ccf
107109
ccf_config.ledger.chunk_size,
108110
ccf_config.consensus,
109111
ccf_config.node_certificate.curve_id,
110-
work_beacon);
112+
work_beacon,
113+
ledger);
111114
// NOLINTEND(cppcoreguidelines-owning-memory)
112115
}
113116
catch (const std::exception& exc)

0 commit comments

Comments
 (0)