Skip to content

Commit e874678

Browse files
mhutchinsonmhutchinson
authored
Use the rsc implementation of the patricia trie (#126)
This required changing the API structs. We have tagged a v0.1.0 that operators of the old code should pin to. This implementation has comparable disk and memory implementations: ``` go test -bench=BenchmarkBuild -benchmem ./vindex goos: linux goarch: amd64 pkg: github.com/transparency-dev/incubator/vindex cpu: AMD EPYC 7B12 BenchmarkBuild_InMemory-24 1 1006305591 ns/op 13464448 B/op 244751 allocs/op BenchmarkBuild_OnDisk-24 1 1001417280 ns/op 14245768 B/op 261960 allocs/op PASS ok github.com/transparency-dev/incubator/vindex 22.217s ``` Compared with old implementation: ``` New: BenchmarkBuild_InMemory-24 1 1006305591 ns/op 13464448 B/op 244751 allocs/op Old: BenchmarkBuild_InMemory-24 1 1004559670 ns/op 38192304 B/op 588597 allocs/op New: BenchmarkBuild_OnDisk-24 1 1001417280 ns/op 14245768 B/op 261960 allocs/op Old: BenchmarkBuild_OnDisk-24 1 4810209571 ns/op 2268211632 B/op 1541229 allocs/op ``` Given that the disk performance is now so good, and superior for restarts, the vindex implementations have been changed to use persistence by default (though support a flag override to use the memory version). Comparing real world performance of this vs the previous version is most useful. For a full read of SumDB: ``` OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir 280.92s user 131.52s system 176% cpu 3:54.03 total OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir 164.35s user 105.55s system 147% cpu 3:03.23 total OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir 320.98s user 134.93s system 180% cpu 4:13.08 total ```
1 parent 0a38ce8 commit e874678

11 files changed

Lines changed: 163 additions & 132 deletions

File tree

‎go.mod‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ module github.com/transparency-dev/incubator
33
go 1.25.0
44

55
require (
6-
filippo.io/torchwood v0.9.0
76
github.com/cockroachdb/pebble v1.1.5
87
github.com/go-git/go-git/v5 v5.19.1
98
github.com/google/go-cmp v0.7.0
@@ -14,6 +13,7 @@ require (
1413
golang.org/x/mod v0.36.0
1514
golang.org/x/sync v0.20.0
1615
k8s.io/klog/v2 v2.140.0
16+
rsc.io/tmp/mpt v0.2.0
1717
)
1818

1919
require (
@@ -31,7 +31,6 @@ require (
3131
github.com/cockroachdb/redact v1.1.5 // indirect
3232
github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 // indirect
3333
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
34-
github.com/dustin/go-humanize v1.0.1 // indirect
3534
github.com/emirpasic/gods v1.18.1 // indirect
3635
github.com/getsentry/sentry-go v0.27.0 // indirect
3736
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
@@ -42,24 +41,20 @@ require (
4241
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
4342
github.com/golang/protobuf v1.5.4 // indirect
4443
github.com/golang/snappy v0.0.4 // indirect
45-
github.com/google/uuid v1.6.0 // indirect
4644
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
4745
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
4846
github.com/kevinburke/ssh_config v1.2.0 // indirect
4947
github.com/klauspost/compress v1.18.0 // indirect
5048
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
5149
github.com/kr/pretty v0.3.1 // indirect
5250
github.com/kr/text v0.2.0 // indirect
53-
github.com/mattn/go-isatty v0.0.20 // indirect
5451
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
55-
github.com/ncruces/go-strftime v0.1.9 // indirect
5652
github.com/pjbgf/sha1cd v0.6.0 // indirect
5753
github.com/pkg/errors v0.9.1 // indirect
5854
github.com/prometheus/client_golang v1.15.0 // indirect
5955
github.com/prometheus/client_model v0.3.0 // indirect
6056
github.com/prometheus/common v0.42.0 // indirect
6157
github.com/prometheus/procfs v0.9.0 // indirect
62-
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
6358
github.com/rogpeppe/go-internal v1.14.1 // indirect
6459
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
6560
github.com/skeema/knownhosts v1.3.1 // indirect
@@ -75,9 +70,4 @@ require (
7570
golang.org/x/text v0.36.0 // indirect
7671
google.golang.org/protobuf v1.36.11 // indirect
7772
gopkg.in/warnings.v0 v0.1.2 // indirect
78-
modernc.org/libc v1.65.7 // indirect
79-
modernc.org/mathutil v1.7.1 // indirect
80-
modernc.org/memory v1.11.0 // indirect
81-
modernc.org/sqlite v1.37.1 // indirect
82-
zombiezen.com/go/sqlite v1.4.2 // indirect
8373
)

‎go.sum‎

Lines changed: 2 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
dario.cat/mergo v1.0.0 h1:AGCNq9Evsj31mOgNPcLyXc+4PNABt905YmuqPYYpBWk=
22
dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
3-
filippo.io/torchwood v0.9.0 h1:2W156cI7K3MyxEyNTuS1C9lYEW7y1u7PoHLmvgNsiZc=
4-
filippo.io/torchwood v0.9.0/go.mod h1:zOJguxdmaODUQScAvC80bV6N0SOA9U+bFZG1DwJU6N8=
53
github.com/DataDog/zstd v1.4.5 h1:EndNeuB0l9syBZhut0wns3gV1hL8zX8LIu6ZiVHWLIQ=
64
github.com/DataDog/zstd v1.4.5/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo=
75
github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
@@ -41,8 +39,6 @@ github.com/cyphar/filepath-securejoin v0.6.1/go.mod h1:A8hd4EnAeyujCJRrICiOWqjS1
4139
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
4240
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
4341
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
44-
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
45-
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
4642
github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o=
4743
github.com/elazarl/goproxy v1.7.2/go.mod h1:82vkLNir0ALaW14Rc399OTTjyNREgmdL2cVoIbS6XaE=
4844
github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
@@ -78,8 +74,6 @@ github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
7874
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
7975
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
8076
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
81-
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
82-
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
8377
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
8478
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
8579
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
@@ -101,12 +95,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
10195
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
10296
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
10397
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
104-
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
105-
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
10698
github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
10799
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
108-
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
109-
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
110100
github.com/onsi/gomega v1.34.1 h1:EUMJIKUjM8sKjYbtxQI9A4z2o+rruxnzNvpknOXie6k=
111101
github.com/onsi/gomega v1.34.1/go.mod h1:kU1QgUvBDLXBJq618Xvm2LUX6rSAfRaFRTcdOeDLwwY=
112102
github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
@@ -126,8 +116,6 @@ github.com/prometheus/common v0.42.0 h1:EKsfXEYo4JpWMHH5cg+KOUWeuJSov1Id8zGR8eeI
126116
github.com/prometheus/common v0.42.0/go.mod h1:xBwqVerjNdUDjgODMpudtOMwlOwf2SaTr1yjz4b7Zbc=
127117
github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJfhI=
128118
github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY=
129-
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
130-
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
131119
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
132120
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
133121
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
@@ -193,7 +181,6 @@ golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7w
193181
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
194182
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
195183
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
196-
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
197184
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
198185
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
199186
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -228,31 +215,5 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
228215
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
229216
k8s.io/klog/v2 v2.140.0 h1:Tf+J3AH7xnUzZyVVXhTgGhEKnFqye14aadWv7bzXdzc=
230217
k8s.io/klog/v2 v2.140.0/go.mod h1:o+/RWfJ6PwpnFn7OyAG3QnO47BFsymfEfrz6XyYSSp0=
231-
lukechampine.com/blake3 v1.4.1 h1:I3Smz7gso8w4/TunLKec6K2fn+kyKtDxr/xcQEN84Wg=
232-
lukechampine.com/blake3 v1.4.1/go.mod h1:QFosUxmjB8mnrWFSNwKmvxHpfY72bmD2tQ0kBMM3kwo=
233-
modernc.org/cc/v4 v4.26.1 h1:+X5NtzVBn0KgsBCBe+xkDC7twLb/jNVj9FPgiwSQO3s=
234-
modernc.org/cc/v4 v4.26.1/go.mod h1:uVtb5OGqUKpoLWhqwNQo/8LwvoiEBLvZXIQ/SmO6mL0=
235-
modernc.org/ccgo/v4 v4.28.0 h1:rjznn6WWehKq7dG4JtLRKxb52Ecv8OUGah8+Z/SfpNU=
236-
modernc.org/ccgo/v4 v4.28.0/go.mod h1:JygV3+9AV6SmPhDasu4JgquwU81XAKLd3OKTUDNOiKE=
237-
modernc.org/fileutil v1.3.1 h1:8vq5fe7jdtEvoCf3Zf9Nm0Q05sH6kGx0Op2CPx1wTC8=
238-
modernc.org/fileutil v1.3.1/go.mod h1:HxmghZSZVAz/LXcMNwZPA/DRrQZEVP9VX0V4LQGQFOc=
239-
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
240-
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
241-
modernc.org/libc v1.65.7 h1:Ia9Z4yzZtWNtUIuiPuQ7Qf7kxYrxP1/jeHZzG8bFu00=
242-
modernc.org/libc v1.65.7/go.mod h1:011EQibzzio/VX3ygj1qGFt5kMjP0lHb0qCW5/D/pQU=
243-
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
244-
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
245-
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
246-
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
247-
modernc.org/opt v0.1.4 h1:2kNGMRiUjrp4LcaPuLY2PzUfqM/w9N23quVwhKt5Qm8=
248-
modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
249-
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
250-
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
251-
modernc.org/sqlite v1.37.1 h1:EgHJK/FPoqC+q2YBXg7fUmES37pCHFc97sI7zSayBEs=
252-
modernc.org/sqlite v1.37.1/go.mod h1:XwdRtsE1MpiBcL54+MbKcaDvcuej+IYSMfLN6gSKV8g=
253-
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
254-
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
255-
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
256-
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
257-
zombiezen.com/go/sqlite v1.4.2 h1:KZXLrBuJ7tKNEm+VJcApLMeQbhmAUOKA5VWS93DfFRo=
258-
zombiezen.com/go/sqlite v1.4.2/go.mod h1:5Kd4taTAD4MkBzT25mQ9uaAlLjyR0rFhsR6iINO70jc=
218+
rsc.io/tmp/mpt v0.2.0 h1:EORjerXrh5u8VU0INROMFgclr2HVO2XzSx9j4wmsTak=
219+
rsc.io/tmp/mpt v0.2.0/go.mod h1:aP1ILCaUW9I5rFbt2z0/8KYz/0FDaYfPW2zEArs+OIQ=

‎vindex/api/api.go‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,6 @@ type LookupResponse struct {
4444
// These values represent the lookup operation in the index at the root hash
4545
// committed to by OutputLogLeaf. The values contain all indices for the given
4646
// key, and the proof binds these values at this key at the index root hash.
47-
IndexValue []uint64 `json:"index_value"`
48-
IndexProof []IndexNode `json:"index_proof"`
49-
}
50-
51-
// IndexNode is a node in the Verifiable Index.
52-
type IndexNode struct {
53-
LabelBitLen uint32 `json:"label_bit_len"`
54-
LabelPath []byte `json:"label_path"`
55-
Hash [sha256.Size]byte `json:"hash"`
47+
IndexValue []uint64 `json:"index_value"`
48+
IndexProof []byte `json:"index_proof"`
5649
}

‎vindex/client/client.go‎

Lines changed: 30 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
package client
1717

1818
import (
19+
"bytes"
1920
"context"
2021
"crypto/sha256"
2122
"encoding/binary"
@@ -27,7 +28,6 @@ import (
2728
"net/http"
2829
"net/url"
2930

30-
"filippo.io/torchwood/prefix"
3131
"github.com/transparency-dev/formats/log"
3232
"github.com/transparency-dev/incubator/vindex"
3333
"github.com/transparency-dev/incubator/vindex/api"
@@ -37,15 +37,16 @@ import (
3737
"github.com/transparency-dev/tessera/client"
3838
"golang.org/x/mod/sumdb/note"
3939
"k8s.io/klog/v2"
40+
"rsc.io/tmp/mpt"
4041
)
4142

42-
func VerifyLookupResponse(keyHash [sha256.Size]byte, resp api.LookupResponse, outV note.Verifier) ([]uint64, []byte, error) {
43+
func VerifyLookupResponse(keyHash [sha256.Size]byte, resp api.LookupResponse, inV, outV note.Verifier) ([]uint64, []byte, error) {
4344
// Currently the response contains the RFC6962 style response type; leaf, proof, etc.
4445
// What if we flip this all around, and the OutputLog part of the response
4546
// only returns an index into the output log, and the client has to look up
4647
// that leaf, checkpoint, and generate inclusion proof?
4748

48-
cp, _, _, err := log.ParseCheckpoint(resp.OutputLogCP, outV.Name(), outV)
49+
olcp, _, _, err := log.ParseCheckpoint(resp.OutputLogCP, outV.Name(), outV)
4950
if err != nil {
5051
return nil, nil, fmt.Errorf("failed to parse output log checkpoint: %v", err)
5152
}
@@ -54,8 +55,8 @@ func VerifyLookupResponse(keyHash [sha256.Size]byte, resp api.LookupResponse, ou
5455
for i := range olp {
5556
olp[i] = resp.OutputLogProof[i][:]
5657
}
57-
oli := cp.Size - 1 // TODO(mhutchinson): include this in the response?
58-
if err := proof.VerifyInclusion(rfc6962.DefaultHasher, oli, cp.Size, outLeafHash[:], olp, cp.Hash); err != nil {
58+
oli := olcp.Size - 1 // TODO(mhutchinson): include this in the response?
59+
if err := proof.VerifyInclusion(rfc6962.DefaultHasher, oli, olcp.Size, outLeafHash[:], olp, olcp.Hash); err != nil {
5960
return nil, nil, fmt.Errorf("failed to verify inclusion in output log: %v", err)
6061
}
6162

@@ -72,26 +73,28 @@ func VerifyLookupResponse(keyHash [sha256.Size]byte, resp api.LookupResponse, ou
7273
}
7374
vindexLeafHash := idxLeafHash.Sum(nil)
7475

75-
pns := make([]prefix.ProofNode, len(resp.IndexProof))
76-
for i, p := range resp.IndexProof {
77-
label, err := prefix.NewLabel(p.LabelBitLen, p.LabelPath)
78-
if err != nil {
79-
return nil, nil, fmt.Errorf("failed to create label: %v", err)
80-
}
81-
pns[i] = prefix.ProofNode{
82-
Label: label,
83-
Hash: p.Hash,
84-
}
76+
origin := inV.Name()
77+
// Hack for sumDB - it's the only major log with a different origin than verifier string.
78+
// It's easier to have a special case here, than to route through options to allow overriding.
79+
if origin == "sum.golang.org" {
80+
origin = "go.sum database tree"
81+
}
82+
ilcp, _, _, err := log.ParseCheckpoint(inCp, origin, inV)
83+
if err != nil {
84+
return nil, nil, fmt.Errorf("failed to parse input log checkpoint: %v", err)
8585
}
8686

87-
if len(resp.IndexValue) > 0 {
88-
if err := prefix.VerifyMembershipProof(sha256.Sum256, keyHash, [32]byte(vindexLeafHash), pns, mapRoot); err != nil {
89-
return nil, nil, fmt.Errorf("failed to verify membership: %v", err)
90-
}
91-
} else {
92-
if err := prefix.VerifyNonMembershipProof(sha256.Sum256, keyHash, pns, mapRoot); err != nil {
93-
return nil, nil, fmt.Errorf("failed to verify non-membership: %v", err)
94-
}
87+
snap := mpt.Snapshot{
88+
Version: int64(ilcp.Size),
89+
Hash: mapRoot,
90+
}
91+
kvHash, ok, err := mpt.Verify(snap, keyHash, resp.IndexProof)
92+
if err != nil {
93+
return nil, nil, fmt.Errorf("mpt.Verify(): %v", err)
94+
}
95+
96+
if ok && !bytes.Equal(kvHash[:], vindexLeafHash) {
97+
return nil, nil, fmt.Errorf("failed to verify membership: %x != %x", kvHash, vindexLeafHash)
9598
}
9699

97100
return resp.IndexValue, inCp, nil
@@ -100,7 +103,7 @@ func VerifyLookupResponse(keyHash [sha256.Size]byte, resp api.LookupResponse, ou
100103
// NewVIndexClient returns a client that can perform verified lookups into the index at the
101104
// given base URL, using the supplied verifier to check checkpoint signatures on the output
102105
// log.
103-
func NewVIndexClient(vindexUrl string, outV note.Verifier) (*VIndexClient, error) {
106+
func NewVIndexClient(vindexUrl string, inV, outV note.Verifier) (*VIndexClient, error) {
104107
viu, err := url.Parse(vindexUrl)
105108
if err != nil {
106109
return nil, fmt.Errorf("failed to parse URL: %v", err)
@@ -109,14 +112,15 @@ func NewVIndexClient(vindexUrl string, outV note.Verifier) (*VIndexClient, error
109112

110113
return &VIndexClient{
111114
lookupURL: lookupURL,
115+
inV: inV,
112116
outV: outV,
113117
}, nil
114118
}
115119

116120
// VIndexClient allows verified lookups into a verifiable index.
117121
type VIndexClient struct {
118122
lookupURL *url.URL
119-
outV note.Verifier
123+
inV, outV note.Verifier
120124
}
121125

122126
// Lookup returns all indices, in ascending order, where the given key appears in the Input Log.
@@ -137,7 +141,7 @@ func (c VIndexClient) Lookup(ctx context.Context, key string) ([]uint64, []byte,
137141
return nil, nil, fmt.Errorf("lookup failed: %v", err)
138142
}
139143

140-
return VerifyLookupResponse(kh, resp, c.outV)
144+
return VerifyLookupResponse(kh, resp, c.inV, c.outV)
141145
}
142146

143147
func (c VIndexClient) lookupUnverified(ctx context.Context, kh [sha256.Size]byte) (api.LookupResponse, error) {

‎vindex/cmd/client/client.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,11 +104,15 @@ func newVIndexClientFromFlags() *client.VIndexClient {
104104
if *outLogPubKey == "" {
105105
klog.Exitf("out_log_pub_key must be provided")
106106
}
107+
inV, err := note.NewVerifier(*inLogPubKey)
108+
if err != nil {
109+
klog.Exitf("failed to construct input log verifier: %v", err)
110+
}
107111
outV, err := note.NewVerifier(*outLogPubKey)
108112
if err != nil {
109113
klog.Exitf("failed to construct output log verifier: %v", err)
110114
}
111-
c, err := client.NewVIndexClient(*vindexBaseURL, outV)
115+
c, err := client.NewVIndexClient(*vindexBaseURL, inV, outV)
112116
if err != nil {
113117
klog.Exitf("failed to construct VIndex Client: %v", err)
114118
}

‎vindex/cmd/logandmap/main.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ var (
5353
inputLogPrivKeyFile = flag.String("input_log_private_key_path", "", "Location of private key file. If unset, uses the contents of the INPUT_LOG_PRIVATE_KEY environment variable.")
5454
outputLogPrivKeyFile = flag.String("output_log_private_key_path", "", "Location of private key file. If unset, uses the contents of the OUTPUT_LOG_PRIVATE_KEY environment variable.")
5555
storageDir = flag.String("storage_dir", "", "Root directory in which to store the data for the demo. This will create subdirectories for the Input Log, Output Log, and allocate space to store the verifiable map persistence.")
56+
persistIndex = flag.Bool("persist_index", true, "Set to false to use a memory-based implementation of the verifiable index.")
5657
listen = flag.String("listen", ":8088", "Address to set up HTTP server listening on")
5758
)
5859

@@ -101,7 +102,9 @@ func run(ctx context.Context) error {
101102
outputLog, outputCloser := outputLogOrDie(ctx, outputLogDir)
102103
defer outputCloser()
103104

104-
vi, err := vindex.NewVerifiableIndex(ctx, inputLog, mapFnFromFlags(), outputLog, mapRoot, vindex.Options{})
105+
vi, err := vindex.NewVerifiableIndex(ctx, inputLog, mapFnFromFlags(), outputLog, mapRoot, vindex.Options{
106+
PersistIndex: *persistIndex,
107+
})
105108
if err != nil {
106109
return fmt.Errorf("failed to create vindex: %v", err)
107110
}

‎vindex/cmd/sumdbindex/README.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,3 +181,56 @@ Use the left and right arrow keys to browse, `g` to jump to a specific index, an
181181

182182
[tlog-tiles]: https://c2sp.org/tlog-tiles
183183

184+
## Performance
185+
186+
### Setup
187+
188+
This creates a local clone of SumDB in a tlog-tiles format. Having a local copy removes a lot of variance when running performance checks.
189+
190+
```
191+
# Run a tlog-proxy for sumdb
192+
go run github.com/transparency-dev/incubator/sumdb/cmd@main --listen=":8089" &
193+
194+
# Clone it
195+
go run github.com/transparency-dev/tessera/cmd/experimental/migrate/posix@main \
196+
--storage_dir ~/log-clones/sumdb \
197+
--source_url http://localhost:8089/ \
198+
--save_checkpoint
199+
200+
# Check it
201+
echo sum.golang.org+033de0ae+Ac4zctda0e5eza+HJyk9SxEdh+s3Ux18htTTAD8OuAn8 > ~/.go.sum.vkey
202+
go run github.com/transparency-dev/tessera/cmd/fsck@main \
203+
--storage_url file:///$HOME/log-clones/sumdb/ \
204+
--public_key ~/.go.sum.vkey \
205+
--origin "go.sum database tree"
206+
```
207+
208+
### Building the VIndex
209+
210+
For performance testing, we use:
211+
- The local copy of the log for performance reasons
212+
- `--oneshot` and `time` to get clear timing figures for initialization
213+
214+
Building the VIndex for the first time needs to consume everything in the log:
215+
```shell
216+
time OUTPUT_LOG_PRIVATE_KEY=PRIVATE+KEY+SumDBIndex+a5ed0e81+AYT6tfHpqGaSoH0gYpM7fhj1tEkM3wwYR/IhtiYh1pnj \
217+
go run ./vindex/cmd/sumdbindex \
218+
--storage_dir ~/vindex-sumdb/ \
219+
--oneshot \
220+
--input_override_url=file:///$HOME/log-clones/sumdb/ \
221+
--persist_index=true
222+
...
223+
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir 164.35s user 105.55s system 147% cpu 3:03.23 total
224+
```
225+
226+
Building the VIndex for the second time reloads state from file and is much faster:
227+
```shell
228+
time OUTPUT_LOG_PRIVATE_KEY=PRIVATE+KEY+SumDBIndex+a5ed0e81+AYT6tfHpqGaSoH0gYpM7fhj1tEkM3wwYR/IhtiYh1pnj \
229+
go run ./vindex/cmd/sumdbindex \
230+
--storage_dir ~/vindex-sumdb/ \
231+
--oneshot \
232+
--input_override_url=file:///$HOME/log-clones/sumdb/ \
233+
--persist_index=true
234+
...
235+
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir 19.17s user 5.37s system 122% cpu 20.058 total
236+
```

0 commit comments

Comments
 (0)