Skip to content

Commit 4e56a0f

Browse files
committed
feat(storage): add native GCS driver
For deployments where org policy disables service-account key creation (the default on most managed GCP orgs), HMAC keys for the S3-compat shim aren't an option. The native GCS driver auths via Application Default Credentials — on Cloud Run that's the runtime SA via the metadata server, no key files anywhere. The driver implements the full Storage interface (Put, Get, Delete, Exists, SignedURL). SignedURL falls back to returning '' rather than erroring when the runtime SA can't self-sign — callers can then proxy reads through the engine instead.
1 parent bcb2635 commit 4e56a0f

4 files changed

Lines changed: 304 additions & 74 deletions

File tree

go.mod

Lines changed: 37 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ module github.com/hallelx2/vectorless-engine
33
go 1.25.0
44

55
require (
6+
cloud.google.com/go/storage v1.62.1
67
github.com/aws/aws-sdk-go-v2 v1.41.6
78
github.com/aws/aws-sdk-go-v2/config v1.32.16
89
github.com/aws/aws-sdk-go-v2/credentials v1.19.15
@@ -24,15 +25,20 @@ require (
2425
)
2526

2627
require (
27-
cloud.google.com/go v0.116.0 // indirect
28+
cel.dev/expr v0.25.1 // indirect
29+
cloud.google.com/go v0.123.0 // indirect
2830
cloud.google.com/go/ai v0.7.0 // indirect
29-
cloud.google.com/go/aiplatform v1.69.0 // indirect
30-
cloud.google.com/go/auth v0.14.0 // indirect
31-
cloud.google.com/go/auth/oauth2adapt v0.2.7 // indirect
32-
cloud.google.com/go/compute/metadata v0.6.0 // indirect
33-
cloud.google.com/go/iam v1.2.2 // indirect
34-
cloud.google.com/go/longrunning v0.6.2 // indirect
31+
cloud.google.com/go/aiplatform v1.120.0 // indirect
32+
cloud.google.com/go/auth v0.19.0 // indirect
33+
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
34+
cloud.google.com/go/compute/metadata v0.9.0 // indirect
35+
cloud.google.com/go/iam v1.7.0 // indirect
36+
cloud.google.com/go/longrunning v0.9.0 // indirect
37+
cloud.google.com/go/monitoring v1.24.3 // indirect
3538
cloud.google.com/go/vertexai v0.12.0 // indirect
39+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 // indirect
40+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.55.0 // indirect
41+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.55.0 // indirect
3642
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.9 // indirect
3743
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.22 // indirect
3844
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.22 // indirect
@@ -48,16 +54,20 @@ require (
4854
github.com/aws/aws-sdk-go-v2/service/sts v1.42.0 // indirect
4955
github.com/cespare/xxhash/v2 v2.3.0 // indirect
5056
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
57+
github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 // indirect
5158
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
5259
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
5360
github.com/dlclark/regexp2 v1.10.0 // indirect
61+
github.com/envoyproxy/go-control-plane/envoy v1.36.0 // indirect
62+
github.com/envoyproxy/protoc-gen-validate v1.3.0 // indirect
5463
github.com/felixge/httpsnoop v1.0.4 // indirect
64+
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
5565
github.com/go-logr/logr v1.4.3 // indirect
5666
github.com/go-logr/stdr v1.2.2 // indirect
5767
github.com/google/generative-ai-go v0.15.1 // indirect
5868
github.com/google/s2a-go v0.1.9 // indirect
59-
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
60-
github.com/googleapis/gax-go/v2 v2.14.1 // indirect
69+
github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
70+
github.com/googleapis/gax-go/v2 v2.21.0 // indirect
6171
github.com/hhrutter/lzw v1.0.0 // indirect
6272
github.com/hhrutter/pkcs7 v0.2.2 // indirect
6373
github.com/hhrutter/tiff v1.0.3 // indirect
@@ -67,38 +77,42 @@ require (
6777
github.com/mattn/go-runewidth v0.0.23 // indirect
6878
github.com/pkg/errors v0.9.1 // indirect
6979
github.com/pkoukk/tiktoken-go v0.1.6 // indirect
80+
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
7081
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
7182
github.com/redis/go-redis/v9 v9.14.1 // indirect
7283
github.com/riverqueue/river/riverdriver v0.35.0 // indirect
7384
github.com/riverqueue/river/rivershared v0.35.0 // indirect
7485
github.com/riverqueue/river/rivertype v0.35.0 // indirect
7586
github.com/robfig/cron/v3 v3.0.1 // indirect
76-
github.com/rogpeppe/go-internal v1.14.1 // indirect
7787
github.com/spf13/cast v1.10.0 // indirect
88+
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
7889
github.com/stretchr/testify v1.11.1 // indirect
7990
github.com/tidwall/gjson v1.18.0 // indirect
8091
github.com/tidwall/match v1.2.0 // indirect
8192
github.com/tidwall/pretty v1.2.1 // indirect
8293
github.com/tidwall/sjson v1.2.5 // indirect
8394
github.com/tmc/langchaingo v0.1.14 // indirect
84-
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
85-
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
95+
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
96+
go.opentelemetry.io/contrib/detectors/gcp v1.39.0 // indirect
97+
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0 // indirect
8698
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
87-
go.opentelemetry.io/otel v1.36.0 // indirect
88-
go.opentelemetry.io/otel/metric v1.36.0 // indirect
89-
go.opentelemetry.io/otel/trace v1.36.0 // indirect
99+
go.opentelemetry.io/otel v1.43.0 // indirect
100+
go.opentelemetry.io/otel/metric v1.43.0 // indirect
101+
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
102+
go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
103+
go.opentelemetry.io/otel/trace v1.43.0 // indirect
90104
go.uber.org/goleak v1.3.0 // indirect
91105
golang.org/x/crypto v0.50.0 // indirect
92106
golang.org/x/image v0.39.0 // indirect
93-
golang.org/x/oauth2 v0.30.0 // indirect
107+
golang.org/x/oauth2 v0.36.0 // indirect
94108
golang.org/x/sys v0.43.0 // indirect
95109
golang.org/x/text v0.36.0 // indirect
96-
golang.org/x/time v0.14.0 // indirect
97-
google.golang.org/api v0.218.0 // indirect
98-
google.golang.org/genproto v0.0.0-20241118233622-e639e219e697 // indirect
99-
google.golang.org/genproto/googleapis/api v0.0.0-20241209162323-e6fa225c2576 // indirect
100-
google.golang.org/genproto/googleapis/rpc v0.0.0-20250122153221-138b5a5a4fd4 // indirect
101-
google.golang.org/grpc v1.70.0 // indirect
102-
google.golang.org/protobuf v1.36.10 // indirect
110+
golang.org/x/time v0.15.0 // indirect
111+
google.golang.org/api v0.274.0 // indirect
112+
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
113+
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
114+
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
115+
google.golang.org/grpc v1.80.0 // indirect
116+
google.golang.org/protobuf v1.36.11 // indirect
103117
gopkg.in/yaml.v2 v2.4.0 // indirect
104118
)

0 commit comments

Comments
 (0)