Skip to content

Commit d858aeb

Browse files
Implement fetch subcommand and internal/fetch package (closes #4)
- Add internal/fetch/fetch.go: - HTTP GET with configurable timeout - Auth header via env var (--auth-env, never on CLI) - Path traversal prevention via internal/safety - Redirect control: disabled by default, same-site only when enabled, cross-site opt-in via --allow-cross-site-redirects - Insecure TLS opt-in via --insecure-tls - Add internal/fetch/fetch_test.go with 12 tests: - success, auth header, empty auth env, missing URL, missing output, path traversal, HTTP error, insecure TLS, no-follow redirects, same-site redirects, nested output dir, context cancellation - Add internal/cmd/fetch.go: fetch subcommand with retry integration, --url, --output, --workdir, --auth-env, --insecure-tls, --follow-redirects, --allow-cross-site-redirects, retry flags - Add internal/cmd/fetch_test.go with 9 command-level tests - Register fetch command in cmd/initium/main.go - Update docs/usage.md with full fetch section - Update CHANGELOG.md
1 parent 02cd94b commit d858aeb

7 files changed

Lines changed: 892 additions & 3 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `fetch` subcommand and `internal/fetch` package: fetch secrets/config from HTTP(S) endpoints with auth header via env var, retry with backoff, TLS options, redirect control (same-site by default), and path traversal prevention
1112
- `render` subcommand and `internal/render` package: render templates into config files with `envsubst` (default) and Go `text/template` modes, path traversal prevention, and automatic intermediate directory creation
1213
- `seed` subcommand: run database seed commands with structured logging and exit code forwarding (no idempotency — distinct from `migrate`)
1314
- `migrate` subcommand: run database migration commands with structured logging, exit code forwarding, and optional idempotency via `--lock-file`

cmd/initium/main.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ and security guardrails.`,
3636
root.AddCommand(cmd.NewMigrateCmd(log))
3737
root.AddCommand(cmd.NewSeedCmd(log))
3838
root.AddCommand(cmd.NewRenderCmd(log))
39+
root.AddCommand(cmd.NewFetchCmd(log))
3940
if err := root.Execute(); err != nil {
4041
log.Error(err.Error())
4142
os.Exit(1)

docs/usage.md

Lines changed: 55 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -196,14 +196,66 @@ initium render --template /tpl/db.conf.tmpl --output config/db.conf --workdir /w
196196
| `0` | Render succeeded |
197197
| `1` | Invalid arguments, missing template, template syntax error, or path traversal |
198198

199-
### fetch _(coming soon)_
199+
### fetch
200200

201-
Fetch secrets or config from HTTP endpoints.
201+
Fetch a resource from an HTTP(S) endpoint and write the response body to a file.
202+
203+
Supports optional authentication via an environment variable (to avoid leaking
204+
credentials in process argument lists), TLS verification skipping, redirect
205+
control, and retries with exponential backoff.
202206

203207
```bash
204-
initium fetch --url https://vault:8200/v1/secret/data/app --output secrets.json --workdir /work
208+
# Fetch a config file
209+
initium fetch --url http://config-service:8080/app.json --output app.json
210+
211+
# Fetch from Vault with auth token
212+
initium fetch --url https://vault:8200/v1/secret/data/app --output secrets.json \
213+
--auth-env VAULT_TOKEN --insecure-tls
214+
215+
# Fetch with retries
216+
initium fetch --url http://api:8080/config --output config.json \
217+
--max-attempts 10 --initial-delay 2s
218+
219+
# Follow redirects (same-site only by default)
220+
initium fetch --url http://cdn/config --output config.json --follow-redirects
221+
222+
# Allow cross-site redirects
223+
initium fetch --url http://cdn/config --output config.json \
224+
--follow-redirects --allow-cross-site-redirects
205225
```
206226

227+
**Flags:**
228+
229+
| Flag | Default | Description |
230+
|------|---------|-------------|
231+
| `--url` | _(required)_ | Target URL to fetch |
232+
| `--output` | _(required)_ | Output file path relative to workdir |
233+
| `--workdir` | `/work` | Working directory for output files |
234+
| `--auth-env` | _(none)_ | Name of env var containing the Authorization header value |
235+
| `--insecure-tls` | `false` | Skip TLS certificate verification |
236+
| `--follow-redirects` | `false` | Follow HTTP redirects |
237+
| `--allow-cross-site-redirects` | `false` | Allow cross-site redirects (requires `--follow-redirects`) |
238+
| `--timeout` | `5m` | Overall timeout |
239+
| `--max-attempts` | `3` | Maximum retry attempts |
240+
| `--initial-delay` | `1s` | Initial delay between retries |
241+
| `--max-delay` | `30s` | Maximum delay between retries |
242+
| `--backoff-factor` | `2.0` | Backoff multiplier |
243+
| `--jitter` | `0.1` | Jitter fraction (0.0–1.0) |
244+
| `--json` | `false` | Enable JSON log output |
245+
246+
**Security notes:**
247+
248+
- The `--auth-env` flag takes the **name** of an environment variable, not the token itself, to avoid leaking credentials in process argument lists or shell history.
249+
- Redirects are disabled by default. When enabled with `--follow-redirects`, cross-site redirects are blocked unless `--allow-cross-site-redirects` is also set.
250+
- TLS verification is enabled by default; `--insecure-tls` must be explicitly set.
251+
252+
**Exit codes:**
253+
254+
| Code | Meaning |
255+
|------|---------|
256+
| `0` | Fetch succeeded |
257+
| `1` | Invalid arguments, HTTP error, timeout, or path traversal |
258+
207259
### exec _(coming soon)_
208260

209261
Run arbitrary commands with structured logging.

internal/cmd/fetch.go

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
package cmd
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"time"
7+
8+
"github.com/kitstream/initium/internal/fetch"
9+
"github.com/kitstream/initium/internal/logging"
10+
"github.com/kitstream/initium/internal/retry"
11+
"github.com/spf13/cobra"
12+
)
13+
14+
func NewFetchCmd(log *logging.Logger) *cobra.Command {
15+
var (
16+
urlFlag string
17+
output string
18+
workdir string
19+
authEnv string
20+
insecureTLS bool
21+
followRedirects bool
22+
allowCrossSiteRedirect bool
23+
timeout time.Duration
24+
maxAttempts int
25+
initialDelay time.Duration
26+
maxDelay time.Duration
27+
backoffFactor float64
28+
jitterFraction float64
29+
jsonLogs bool
30+
)
31+
32+
cmd := &cobra.Command{
33+
Use: "fetch",
34+
Short: "Fetch secrets or config from HTTP(S) endpoints",
35+
Long: `Fetch a resource from an HTTP(S) endpoint and write the response body to a
36+
file within the working directory.
37+
38+
Supports optional authentication via an environment variable (to avoid leaking
39+
credentials in process argument lists), TLS verification skipping, redirect
40+
control, and retries with exponential backoff.`,
41+
Example: ` # Fetch a config file
42+
initium fetch --url http://config-service:8080/app.json --output app.json
43+
44+
# Fetch from Vault with auth token
45+
initium fetch --url https://vault:8200/v1/secret/data/app --output secrets.json \
46+
--auth-env VAULT_TOKEN --insecure-tls
47+
48+
# Fetch with retries
49+
initium fetch --url http://api:8080/config --output config.json \
50+
--max-attempts 10 --initial-delay 2s
51+
52+
# Follow redirects (same-site only by default)
53+
initium fetch --url http://cdn/config --output config.json --follow-redirects`,
54+
SilenceUsage: true,
55+
SilenceErrors: true,
56+
RunE: func(cmd *cobra.Command, args []string) error {
57+
if jsonLogs {
58+
log.SetJSON(true)
59+
}
60+
61+
if urlFlag == "" {
62+
return fmt.Errorf("--url is required")
63+
}
64+
if output == "" {
65+
return fmt.Errorf("--output is required")
66+
}
67+
68+
retryCfg := retry.Config{
69+
MaxAttempts: maxAttempts,
70+
InitialDelay: initialDelay,
71+
MaxDelay: maxDelay,
72+
BackoffFactor: backoffFactor,
73+
JitterFraction: jitterFraction,
74+
}
75+
if err := retryCfg.Validate(); err != nil {
76+
return fmt.Errorf("invalid retry config: %w", err)
77+
}
78+
79+
fetchCfg := fetch.Config{
80+
URL: urlFlag,
81+
OutputPath: output,
82+
Workdir: workdir,
83+
AuthEnv: authEnv,
84+
InsecureTLS: insecureTLS,
85+
FollowRedirects: followRedirects,
86+
AllowCrossSiteRedirect: allowCrossSiteRedirect,
87+
Timeout: timeout,
88+
}
89+
90+
if err := fetchCfg.Validate(); err != nil {
91+
return err
92+
}
93+
94+
ctx, cancel := context.WithTimeout(cmd.Context(), timeout)
95+
defer cancel()
96+
97+
log.Info("fetching", "url", urlFlag, "output", output)
98+
99+
result := retry.Do(ctx, retryCfg, func(ctx context.Context, attempt int) error {
100+
log.Debug("fetch attempt", "attempt", fmt.Sprintf("%d", attempt+1))
101+
return fetch.Do(ctx, fetchCfg)
102+
})
103+
104+
if result.Err != nil {
105+
log.Error("fetch failed", "url", urlFlag, "error", result.Err.Error())
106+
return fmt.Errorf("fetch %s failed: %w", urlFlag, result.Err)
107+
}
108+
109+
log.Info("fetch completed", "url", urlFlag, "output", output, "attempts", fmt.Sprintf("%d", result.Attempt+1))
110+
return nil
111+
},
112+
}
113+
114+
cmd.Flags().StringVar(&urlFlag, "url", "", "Target URL to fetch (required)")
115+
cmd.Flags().StringVar(&output, "output", "", "Output file path relative to workdir (required)")
116+
cmd.Flags().StringVar(&workdir, "workdir", "/work", "Working directory for output files")
117+
cmd.Flags().StringVar(&authEnv, "auth-env", "", "Name of env var containing the Authorization header value")
118+
cmd.Flags().BoolVar(&insecureTLS, "insecure-tls", false, "Skip TLS certificate verification")
119+
cmd.Flags().BoolVar(&followRedirects, "follow-redirects", false, "Follow HTTP redirects")
120+
cmd.Flags().BoolVar(&allowCrossSiteRedirect, "allow-cross-site-redirects", false, "Allow cross-site redirects (requires --follow-redirects)")
121+
cmd.Flags().DurationVar(&timeout, "timeout", 5*time.Minute, "Overall timeout")
122+
cmd.Flags().IntVar(&maxAttempts, "max-attempts", 3, "Maximum retry attempts")
123+
cmd.Flags().DurationVar(&initialDelay, "initial-delay", time.Second, "Initial delay between retries")
124+
cmd.Flags().DurationVar(&maxDelay, "max-delay", 30*time.Second, "Maximum delay between retries")
125+
cmd.Flags().Float64Var(&backoffFactor, "backoff-factor", 2.0, "Backoff multiplier")
126+
cmd.Flags().Float64Var(&jitterFraction, "jitter", 0.1, "Jitter fraction (0.0-1.0)")
127+
cmd.Flags().BoolVar(&jsonLogs, "json", false, "Enable JSON log output")
128+
129+
return cmd
130+
}

0 commit comments

Comments
 (0)