Skip to content

Commit 2bdedad

Browse files
committed
Phase 4 (partial): Cloud storage clients — Azure Blob, AWS S3, GitHub-owned multipart
Port three storage backends used for migration archive upload: ## Azure Blob Storage (pkg/storage/azure/) - BlobService interface for testability over Azure SDK - Upload: creates container migration-archives-<uuid>, uploads blob with 4MB block size, generates SAS URL (48h read-only expiry) - Download: HTTP GET returning bytes - Progress logging throttled to every 10 seconds - 15 tests ## AWS S3 (pkg/storage/aws/) - S3Uploader/S3Presigner interfaces for testability - Upload from stream or file path, returns 48h pre-signed URL - Session token support (SessionCredentials vs BasicCredentials) - Timeout detection wrapped as user-friendly error - Progress logging with 10-second throttle - 10 tests ## GitHub-owned storage multipart (pkg/storage/ghowned/) - Small archives (<= 100 MiB): single POST, returns URI from JSON - Large archives: 3-phase multipart (Start POST → Part PATCHes with Location header following → Complete PUT) - Configurable part size via env var (min 5 MiB) - UserError wrapping for all failure modes - 15 tests Dependencies added: azure-sdk-for-go/sdk/storage/azblob, aws-sdk-go-v2, google/uuid. All 40 tests passing, 0 lint issues.
1 parent 470dc87 commit 2bdedad

9 files changed

Lines changed: 1892 additions & 1 deletion

File tree

go.mod

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,35 @@ module github.com/github/gh-gei
33
go 1.25.4
44

55
require (
6+
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4
67
github.com/avast/retry-go/v4 v4.7.0
8+
github.com/aws/aws-sdk-go-v2 v1.41.5
9+
github.com/aws/aws-sdk-go-v2/credentials v1.19.13
10+
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3
711
github.com/google/go-github/v68 v68.0.0
12+
github.com/google/uuid v1.6.0
813
github.com/spf13/cobra v1.10.2
914
github.com/stretchr/testify v1.11.1
1015
)
1116

1217
require (
18+
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0 // indirect
19+
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
20+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
21+
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
22+
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
23+
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 // indirect
24+
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect
25+
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 // indirect
26+
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 // indirect
27+
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 // indirect
28+
github.com/aws/smithy-go v1.24.2 // indirect
1329
github.com/davecgh/go-spew v1.1.1 // indirect
1430
github.com/google/go-querystring v1.1.0 // indirect
1531
github.com/inconshreveable/mousetrap v1.1.0 // indirect
1632
github.com/pmezard/go-difflib v1.0.0 // indirect
1733
github.com/spf13/pflag v1.0.9 // indirect
34+
golang.org/x/net v0.47.0 // indirect
35+
golang.org/x/text v0.31.0 // indirect
1836
gopkg.in/yaml.v3 v3.0.1 // indirect
1937
)

go.sum

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,69 @@
1+
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0 h1:JXg2dwJUmPB9JmtVmdEB16APJ7jurfbY5jnfXpJoRMc=
2+
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0/go.mod h1:YD5h/ldMsG0XiIw7PdyNhLxaM317eFh5yNLccNfGdyw=
3+
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 h1:Hk5QBxZQC1jb2Fwj6mpzme37xbCDdNTxU7O9eb5+LB4=
4+
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1/go.mod h1:IYus9qsFobWIc2YVwe/WPjcnyCkPKtnHAqUYeebc8z0=
5+
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 h1:9iefClla7iYpfYWdzPCRDozdmndjTm8DXdpCzPajMgA=
6+
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2/go.mod h1:XtLgD3ZD34DAaVIIAyG3objl5DynM3CQ/vMcbBNJZGI=
7+
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig=
8+
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA=
9+
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 h1:jWQK1GI+LeGGUKBADtcH2rRqPxYB1Ljwms5gFA2LqrM=
10+
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4/go.mod h1:8mwH4klAm9DUgR2EEHyEEAQlRDvLPyg5fQry3y+cDew=
11+
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs=
12+
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk=
113
github.com/avast/retry-go/v4 v4.7.0 h1:yjDs35SlGvKwRNSykujfjdMxMhMQQM0TnIjJaHB+Zio=
214
github.com/avast/retry-go/v4 v4.7.0/go.mod h1:ZMPDa3sY2bKgpLtap9JRUgk2yTAba7cgiFhqxY2Sg6Q=
15+
github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY=
16+
github.com/aws/aws-sdk-go-v2 v1.41.5/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
17+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 h1:eBMB84YGghSocM7PsjmmPffTa+1FBUeNvGvFou6V/4o=
18+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
19+
github.com/aws/aws-sdk-go-v2/credentials v1.19.13 h1:mA59E3fokBvyEGHKFdnpNNrvaR351cqiHgRg+JzOSRI=
20+
github.com/aws/aws-sdk-go-v2/credentials v1.19.13/go.mod h1:yoTXOQKea18nrM69wGF9jBdG4WocSZA1h38A+t/MAsk=
21+
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 h1:Rgg6wvjjtX8bNHcvi9OnXWwcE0a2vGpbwmtICOsvcf4=
22+
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21/go.mod h1:A/kJFst/nm//cyqonihbdpQZwiUhhzpqTsdbhDdRF9c=
23+
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 h1:PEgGVtPoB6NTpPrBgqSE5hE/o47Ij9qk/SEZFbUOe9A=
24+
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21/go.mod h1:p+hz+PRAYlY3zcpJhPwXlLC4C+kqn70WIHwnzAfs6ps=
25+
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 h1:rWyie/PxDRIdhNf4DzRk0lvjVOqFJuNnO8WwaIRVxzQ=
26+
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22/go.mod h1:zd/JsJ4P7oGfUhXn1VyLqaRZwPmZwg44Jf2dS84Dm3Y=
27+
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 h1:5EniKhLZe4xzL7a+fU3C2tfUN4nWIqlLesfrjkuPFTY=
28+
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
29+
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 h1:JRaIgADQS/U6uXDqlPiefP32yXTda7Kqfx+LgspooZM=
30+
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13/go.mod h1:CEuVn5WqOMilYl+tbccq8+N2ieCy0gVn3OtRb0vBNNM=
31+
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 h1:c31//R3xgIJMSC8S6hEVq+38DcvUlgFY0FM6mSI5oto=
32+
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21/go.mod h1:r6+pf23ouCB718FUxaqzZdbpYFyDtehyZcmP5KL9FkA=
33+
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 h1:ZlvrNcHSFFWURB8avufQq9gFsheUgjVD9536obIknfM=
34+
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21/go.mod h1:cv3TNhVrssKR0O/xxLJVRfd2oazSnZnkUeTf6ctUwfQ=
35+
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3 h1:HwxWTbTrIHm5qY+CAEur0s/figc3qwvLWsNkF4RPToo=
36+
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3/go.mod h1:uoA43SdFwacedBfSgfFSjjCvYe8aYBS7EnU5GZ/YKMM=
37+
github.com/aws/smithy-go v1.24.2 h1:FzA3bu/nt/vDvmnkg+R8Xl46gmzEDam6mZ1hzmwXFng=
38+
github.com/aws/smithy-go v1.24.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
339
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
440
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
541
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
42+
github.com/golang-jwt/jwt/v5 v5.3.0 h1:pv4AsKCKKZuqlgs5sUmn4x8UlGa0kEVt/puTpKx9vvo=
43+
github.com/golang-jwt/jwt/v5 v5.3.0/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
644
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
745
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
846
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
947
github.com/google/go-github/v68 v68.0.0 h1:ZW57zeNZiXTdQ16qrDiZ0k6XucrxZ2CGmoTvcCyQG6s=
1048
github.com/google/go-github/v68 v68.0.0/go.mod h1:K9HAUBovM2sLwM408A18h+wd9vqdLOEqTUCbnRIcx68=
1149
github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=
1250
github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=
51+
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
52+
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
1353
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
1454
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
55+
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
56+
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
57+
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
58+
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
59+
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
60+
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
61+
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
62+
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
1563
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
1664
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
65+
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
66+
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
1767
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
1868
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
1969
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
@@ -22,8 +72,17 @@ github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An
2272
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
2373
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
2474
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
75+
golang.org/x/crypto v0.45.0 h1:jMBrvKuj23MTlT0bQEOBcAE0mjg8mK9RXFhRH6nyF3Q=
76+
golang.org/x/crypto v0.45.0/go.mod h1:XTGrrkGJve7CYK7J8PEww4aY7gM3qMCElcJQ8n8JdX4=
77+
golang.org/x/net v0.47.0 h1:Mx+4dIFzqraBXUugkia1OOvlD6LemFo1ALMHjrXDOhY=
78+
golang.org/x/net v0.47.0/go.mod h1:/jNxtkgq5yWUGYkaZGqo27cfGZ1c5Nen03aYrrKpVRU=
79+
golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
80+
golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
81+
golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
82+
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
2583
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
26-
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
2784
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
85+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
86+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
2887
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
2988
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

pkg/storage/aws/client.go

Lines changed: 274 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
// Package aws provides an S3 client for uploading migration archives to AWS.
2+
package aws
3+
4+
import (
5+
"context"
6+
"errors"
7+
"fmt"
8+
"io"
9+
"os"
10+
"sync"
11+
"time"
12+
13+
"github.com/aws/aws-sdk-go-v2/aws"
14+
"github.com/aws/aws-sdk-go-v2/credentials"
15+
"github.com/aws/aws-sdk-go-v2/service/s3"
16+
)
17+
18+
const (
19+
// presignExpiry is the duration for pre-signed URLs (48 hours, matching C# AUTHORIZATION_TIMEOUT_IN_HOURS).
20+
presignExpiry = 48 * time.Hour
21+
22+
// progressReportInterval is the minimum time between progress log messages.
23+
progressReportInterval = 10 * time.Second
24+
)
25+
26+
// S3Uploader abstracts S3 PutObject for testability.
27+
type S3Uploader interface {
28+
PutObject(ctx context.Context, bucket, key string, body io.Reader) error
29+
}
30+
31+
// S3Presigner abstracts S3 presigned URL generation for testability.
32+
type S3Presigner interface {
33+
PresignGetObject(ctx context.Context, bucket, key string, expires time.Duration) (string, error)
34+
}
35+
36+
// ProgressLogger is the subset of logger used for upload progress reporting.
37+
type ProgressLogger interface {
38+
LogInfo(format string, args ...interface{})
39+
}
40+
41+
// Client provides S3 upload operations with pre-signed URL generation.
42+
type Client struct {
43+
uploader S3Uploader
44+
presigner S3Presigner
45+
logger ProgressLogger
46+
47+
mu sync.Mutex
48+
nextProgressTime time.Time
49+
}
50+
51+
// Option configures Client construction.
52+
type Option func(*clientConfig)
53+
54+
type clientConfig struct {
55+
region string
56+
sessionToken string
57+
logger ProgressLogger
58+
}
59+
60+
// WithRegion sets the AWS region.
61+
func WithRegion(region string) Option {
62+
return func(c *clientConfig) {
63+
c.region = region
64+
}
65+
}
66+
67+
// WithSessionToken sets an AWS session token for temporary credentials.
68+
func WithSessionToken(token string) Option {
69+
return func(c *clientConfig) {
70+
c.sessionToken = token
71+
}
72+
}
73+
74+
// WithLogger sets the progress logger.
75+
func WithLogger(l ProgressLogger) Option {
76+
return func(c *clientConfig) {
77+
c.logger = l
78+
}
79+
}
80+
81+
// NewClient creates a new AWS S3 Client using the provided credentials.
82+
func NewClient(accessKey, secretKey string, opts ...Option) (*Client, error) {
83+
cfg := &clientConfig{}
84+
for _, o := range opts {
85+
o(cfg)
86+
}
87+
88+
var creds aws.CredentialsProvider
89+
if cfg.sessionToken != "" {
90+
creds = credentials.NewStaticCredentialsProvider(accessKey, secretKey, cfg.sessionToken)
91+
} else {
92+
creds = credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")
93+
}
94+
95+
s3Client := s3.New(s3.Options{
96+
Credentials: creds,
97+
Region: cfg.region,
98+
})
99+
presignClient := s3.NewPresignClient(s3Client)
100+
101+
return &Client{
102+
uploader: &sdkUploader{client: s3Client},
103+
presigner: &sdkPresigner{client: presignClient},
104+
logger: cfg.logger,
105+
}, nil
106+
}
107+
108+
// NewClientFromInterfaces creates a Client from pre-built interfaces (for testing).
109+
func NewClientFromInterfaces(uploader S3Uploader, presigner S3Presigner, logger ProgressLogger) *Client {
110+
return &Client{
111+
uploader: uploader,
112+
presigner: presigner,
113+
logger: logger,
114+
}
115+
}
116+
117+
// Upload uploads data from an io.Reader to S3 and returns a pre-signed URL.
118+
func (c *Client) Upload(ctx context.Context, bucket, key string, data io.Reader) (string, error) {
119+
if err := c.uploader.PutObject(ctx, bucket, key, data); err != nil {
120+
if ctx.Err() != nil || isTimeout(err) {
121+
return "", fmt.Errorf("upload to AWS timed out: %w", err)
122+
}
123+
return "", fmt.Errorf("failed to upload to S3: %w", err)
124+
}
125+
126+
url, err := c.presigner.PresignGetObject(ctx, bucket, key, presignExpiry)
127+
if err != nil {
128+
return "", fmt.Errorf("failed to generate pre-signed URL: %w", err)
129+
}
130+
131+
return url, nil
132+
}
133+
134+
// UploadFile opens a file and uploads it to S3, returning a pre-signed URL.
135+
// Catches timeout/context errors and wraps them with a user-friendly message
136+
// (matching C# behavior for TaskCanceledException/TimeoutException).
137+
func (c *Client) UploadFile(ctx context.Context, bucket, key, filePath string) (string, error) {
138+
f, err := os.Open(filePath)
139+
if err != nil {
140+
return "", fmt.Errorf("failed to open file %q: %w", filePath, err)
141+
}
142+
defer f.Close()
143+
144+
// Wrap with progress reporting if logger is available
145+
var reader io.Reader = f
146+
if c.logger != nil {
147+
info, statErr := f.Stat()
148+
if statErr == nil && info.Size() > 0 {
149+
reader = c.newProgressReader(f, info.Size())
150+
}
151+
}
152+
153+
url, err := c.Upload(ctx, bucket, key, reader)
154+
if err != nil {
155+
if isTimeout(err) {
156+
return "", fmt.Errorf("upload of archive %q to AWS timed out: %w", filePath, err)
157+
}
158+
return "", err
159+
}
160+
return url, nil
161+
}
162+
163+
// isTimeout checks whether an error represents a timeout.
164+
func isTimeout(err error) bool {
165+
if errors.Is(err, context.DeadlineExceeded) {
166+
return true
167+
}
168+
type timeouter interface {
169+
Timeout() bool
170+
}
171+
var te timeouter
172+
if errors.As(err, &te) {
173+
return te.Timeout()
174+
}
175+
return false
176+
}
177+
178+
// --- Progress reader ---
179+
180+
type progressReader struct {
181+
reader io.Reader
182+
total int64
183+
transferred int64
184+
client *Client
185+
}
186+
187+
func (c *Client) newProgressReader(r io.Reader, total int64) *progressReader {
188+
return &progressReader{
189+
reader: r,
190+
total: total,
191+
client: c,
192+
}
193+
}
194+
195+
func (pr *progressReader) Read(p []byte) (int, error) {
196+
n, err := pr.reader.Read(p)
197+
pr.transferred += int64(n)
198+
pr.client.logProgress(pr.transferred, pr.total)
199+
return n, err
200+
}
201+
202+
func (c *Client) logProgress(transferred, total int64) {
203+
if c.logger == nil {
204+
return
205+
}
206+
207+
c.mu.Lock()
208+
now := time.Now()
209+
if now.Before(c.nextProgressTime) {
210+
c.mu.Unlock()
211+
return
212+
}
213+
c.nextProgressTime = now.Add(progressReportInterval)
214+
c.mu.Unlock()
215+
216+
if total > 0 {
217+
percent := int(float64(transferred) / float64(total) * 100)
218+
c.logger.LogInfo(
219+
"Archive upload in progress, %s out of %s (%d%%) completed...",
220+
formatBytes(transferred), formatBytes(total), percent,
221+
)
222+
} else {
223+
c.logger.LogInfo("Archive upload in progress...")
224+
}
225+
}
226+
227+
// formatBytes returns a human-friendly size string.
228+
func formatBytes(b int64) string {
229+
const (
230+
kb = 1024
231+
mb = kb * 1024
232+
gb = mb * 1024
233+
)
234+
switch {
235+
case b >= gb:
236+
return fmt.Sprintf("%.2f GB", float64(b)/float64(gb))
237+
case b >= mb:
238+
return fmt.Sprintf("%.2f MB", float64(b)/float64(mb))
239+
case b >= kb:
240+
return fmt.Sprintf("%.2f KB", float64(b)/float64(kb))
241+
default:
242+
return fmt.Sprintf("%d bytes", b)
243+
}
244+
}
245+
246+
// --- SDK adapter implementations ---
247+
248+
type sdkUploader struct {
249+
client *s3.Client
250+
}
251+
252+
func (u *sdkUploader) PutObject(ctx context.Context, bucket, key string, body io.Reader) error {
253+
_, err := u.client.PutObject(ctx, &s3.PutObjectInput{
254+
Bucket: &bucket,
255+
Key: &key,
256+
Body: body,
257+
})
258+
return err
259+
}
260+
261+
type sdkPresigner struct {
262+
client *s3.PresignClient
263+
}
264+
265+
func (p *sdkPresigner) PresignGetObject(ctx context.Context, bucket, key string, expires time.Duration) (string, error) {
266+
req, err := p.client.PresignGetObject(ctx, &s3.GetObjectInput{
267+
Bucket: &bucket,
268+
Key: &key,
269+
}, s3.WithPresignExpires(expires))
270+
if err != nil {
271+
return "", err
272+
}
273+
return req.URL, nil
274+
}

0 commit comments

Comments
 (0)