fix(timeout): 120s provisioning deadline so slow provisions don't 409 (#21)

Provisioning is synchronous server-side: POST /db/new (and the other
/*/new endpoints plus /deploy/new) blocks while the real Postgres /
Redis / Mongo / NATS / bucket / pod is created. Under prod hot-pool
contention a fresh Postgres provision can exceed the old client-wide
30s timeout; when the client gave up, the server kept working and held
a 60s in-flight idempotency marker, so the caller's retry hit
409 idempotency_key_in_progress instead of succeeding.

Give provisioning + deploy a dedicated HTTP client with no client-wide
Timeout cap and a 120s per-request context deadline (provisionContext),
comfortably outliving both the slow provision and the server in-flight
window. Reads (list/get/claim/delete/rotate) stay capped at 30s via the
existing httpClient.Timeout, so the guarded read-path timeout tests are
unchanged. WithTimeout now governs both budgets; a tighter caller
context deadline is always honoured (provisionContext only lengthens an
open-ended context, never overrides a shorter caller deadline).

Tests (package instant, run under -race in CI):
- TestProvisioningTimeoutDefaults: pins read=30s / provision=120s split
  + provisionClient.Timeout==0.
- TestProvisioningOutlivesReadCap: behavioral proof — a provision
  succeeds against a server that stalls past the read-path cap while a
  read against the same server times out (scaled to ms).
- TestProvisioningHonorsTighterCallerDeadline / TestWithTimeoutGoverns
  Provisioning / TestWithHTTPClientTimeoutGovernsProvisioning /
  TestProvisionContext{OpenEndedGetsBudget,ZeroTimeoutFallsBack} /
  TestProvisionJSONMarshalError.

README (Provisioning + Client configuration) and CHANGELOG document the
120s provisioning default and the WithTimeout override.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: mastermanas805

CHANGELOG.md

@@ -9,6 +9,21 @@ existing callers.
 
 ### Fixed
 
+- **Provisioning + deploy calls now default to a 120 s per-request timeout
+  (reads stay at 30 s).** Provisioning is synchronous server-side: `POST /db/new`
+  (and the other `/*/new` endpoints plus `/deploy/new`) blocks while the real
+  Postgres / Redis / Mongo / NATS / bucket / pod is created. Under prod hot-pool
+  contention a *fresh* Postgres provision can exceed the old 30 s client timeout;
+  when the client gave up the server kept working and held a 60 s in-flight
+  idempotency marker, so the caller's retry hit `409 idempotency_key_in_progress`
+  instead of succeeding. The provisioning + deploy paths now run on a dedicated
+  HTTP client with no client-wide cap and a 120 s per-request context deadline,
+  comfortably outliving both the slow provision and the server's in-flight
+  window. Read calls (list/get/claim/delete/rotate) are unchanged at 30 s.
+  `WithTimeout` still overrides both (it now governs the provisioning deadline
+  too), and a tighter deadline on the caller's `context.Context` is always
+  honoured — the SDK only lengthens an open-ended context, never overrides a
+  shorter caller deadline.
 - **`Client.Claim` now sends the canonical `token` wire field instead of the
   deprecated `jwt` alias.** The api ClaimRequest doc names the Go SDK as one
   of three drift sources for the legacy `jwt` name (alongside the dashboard

README.md

@@ -112,6 +112,32 @@ q, err := client.ProvisionQueue(ctx, &instant.ProvisionOpts{Name: "app-queue"})
 Anonymous resources expire after **24 hours**. Claim them permanently with a free account
 (see [Claim](#claim) below or visit the URL in `result.Note`).
 
+### Timeouts: provisioning runs longer than reads
+
+Provisioning is **synchronous** — `ProvisionDatabase` / `Cache` / `MongoDB` /
+`Queue` / `Storage` / `Webhook` and `Deploy` block while the API creates the
+real backend. Under production hot-pool contention a *fresh* Postgres provision
+can take **more than 30 seconds**. If the client gave up at 30 s, the server
+kept working and held a 60 s in-flight idempotency marker, so the next retry hit
+`409 idempotency_key_in_progress` instead of succeeding.
+
+To avoid that, the SDK gives provisioning + deploy calls a **120 s** per-request
+deadline by default, while read calls (list, get, claim, delete, rotate) keep
+the shorter **30 s** default. You do not need to do anything — the split is
+automatic.
+
+Override either with `WithTimeout`, which sets a single budget governing **both**
+read and provisioning calls:
+
+```go
+// One 90 s budget for every call — set high enough to outlive a slow provision.
+client := instant.New(instant.WithTimeout(90 * time.Second))
+```
+
+A deadline you set on the `context.Context` you pass in is always honoured if it
+is *tighter* than the SDK's provisioning budget — the SDK only lengthens an
+open-ended context, it never overrides a shorter caller deadline.
+
 ---
 
 ## Deploy
@@ -217,7 +243,7 @@ fmt.Println("team_id:", result.TeamID)
 client := instant.New(
     instant.WithAPIKey("inst_live_..."),           // default: INSTANT_API_KEY env var
     instant.WithBaseURL("http://localhost:8080"),   // default: INSTANT_API_URL or https://api.instanode.dev (port-forward svc/instant-api for local k8s)
-    instant.WithTimeout(15 * time.Second),         // default: 30s
+    instant.WithTimeout(15 * time.Second),         // governs reads AND provisioning; defaults: reads 30s, provisioning 120s
     instant.WithHTTPClient(myClient),              // custom transport (tracing, TLS, etc.)
     instant.WithLogger(slog.Default()),            // advisory notices and upgrade prompts
 )

instant/cache.go

@@ -38,7 +38,7 @@ func (c *Client) ProvisionCache(ctx context.Context, opts *ProvisionOpts) (*Prov
 	}
 
 	var result ProvisionResult
-	if err := c.postJSONWithHeaders(ctx, "/cache/new", body, provisionHeaders(opts), &result); err != nil {
+	if err := c.provisionJSONWithHeaders(ctx, "/cache/new", body, provisionHeaders(opts), &result); err != nil {
 		return nil, fmt.Errorf("ProvisionCache: %w", err)
 	}
 	if result.Token == "" {

instant/client.go

@@ -47,18 +47,54 @@ const (
 	// TLS, full OpenAPI surface.
 	DefaultBaseURL = "https://api.instanode.dev"
 
-	// defaultTimeout is applied to every HTTP request.
+	// defaultTimeout is the per-request deadline applied to read-style calls
+	// (list, get, claim, delete, rotate). 30 s is plenty for a JSON round trip.
 	defaultTimeout = 30 * time.Second
+
+	// defaultProvisioningTimeout is the per-request deadline applied to the
+	// synchronous provisioning + deploy endpoints.
+	//
+	// Provisioning is synchronous on the server: POST /db/new (and the other
+	// /*/new endpoints, plus /deploy/new) blocks the handler while the real
+	// Postgres / Redis / Mongo / NATS / bucket / pod is created. Under prod
+	// hot-pool contention a *fresh* Postgres provision can exceed 30 s. When
+	// the client gave up at the read-path 30 s the server kept working and
+	// held a 60 s in-flight idempotency marker, so the caller's retry hit
+	// `409 idempotency_key_in_progress` instead of succeeding. A 120 s
+	// provisioning deadline comfortably outlives both the slow provision and
+	// the server's in-flight window, so the first call returns the resource
+	// rather than orphaning it behind a conflicting retry.
+	//
+	// Overridable: WithTimeout sets an explicit per-request deadline that
+	// governs BOTH read and provisioning calls (see [WithTimeout]).
+	defaultProvisioningTimeout = 120 * time.Second
 )
 
 // Client is the instant.dev API client.
 // Construct one with [New]; all methods are safe for concurrent use.
 type Client struct {
-	baseURL    string
-	apiKey     string
+	baseURL string
+	apiKey  string
+
+	// httpClient governs read-style calls. Its Timeout is the read-path
+	// per-request deadline (defaultTimeout unless WithTimeout overrides it).
 	httpClient *http.Client
-	userAgent  string
-	logger     *slog.Logger
+
+	// provisionClient governs the synchronous provisioning + deploy calls. It
+	// shares httpClient's transport (so WithHTTPClient's transport chaining,
+	// auth header, and User-Agent all apply identically) but carries no
+	// client-wide Timeout cap — the per-request provisioning deadline is
+	// enforced via the request context instead. This lets provisioning run for
+	// up to provisionTimeout even though reads stay capped at the shorter
+	// read-path Timeout.
+	provisionClient *http.Client
+
+	// provisionTimeout is the per-request deadline applied to provisioning +
+	// deploy calls (defaultProvisioningTimeout unless WithTimeout overrides it).
+	provisionTimeout time.Duration
+
+	userAgent string
+	logger    *slog.Logger
 }
 
 // Option configures a [Client]. Pass options to [New].
@@ -101,9 +137,22 @@ func WithHTTPClient(hc *http.Client) Option {
 	}
 }
 
-// WithTimeout sets the per-request HTTP timeout. Default is 30 s.
+// WithTimeout sets the per-request HTTP timeout.
+//
+// It governs BOTH read-style calls and the synchronous provisioning / deploy
+// calls. Without this option reads default to 30 s and provisioning defaults to
+// 120 s (synchronous provisioning can exceed 30 s under prod hot-pool
+// contention — see [defaultProvisioningTimeout]). Passing WithTimeout collapses
+// both onto the single value you supply, so set it high enough to outlive a
+// slow provision if you provision through this client:
+//
+//	// give every call — reads and provisioning alike — a 90 s budget
+//	client := instant.New(instant.WithTimeout(90 * time.Second))
 func WithTimeout(d time.Duration) Option {
-	return func(c *Client) { c.httpClient.Timeout = d }
+	return func(c *Client) {
+		c.httpClient.Timeout = d
+		c.provisionTimeout = d
+	}
 }
 
 // WithLogger sets the structured logger used for advisory notices and upgrade prompts.
@@ -127,6 +176,10 @@ func New(opts ...Option) *Client {
 		httpClient: &http.Client{
 			Timeout: defaultTimeout,
 		},
+		// 0 is the "not explicitly set" sentinel. WithTimeout overwrites it;
+		// otherwise it is resolved below to either the default provisioning
+		// timeout or a caller-supplied WithHTTPClient Timeout.
+		provisionTimeout: 0,
 	}
 
 	// Environment defaults (explicit options below override these)
@@ -141,6 +194,21 @@ func New(opts ...Option) *Client {
 		opt(c)
 	}
 
+	// Resolve the provisioning deadline. Precedence:
+	//  1. WithTimeout — already stamped both httpClient.Timeout and
+	//     provisionTimeout to the caller's explicit value (non-zero here).
+	//  2. WithHTTPClient with a non-zero Timeout — the caller set a deliberate
+	//     per-request budget on their own client; honour it for provisioning
+	//     too rather than silently widening it to 120 s.
+	//  3. Neither — provisioning gets the 120 s default while reads keep 30 s.
+	if c.provisionTimeout == 0 {
+		if c.httpClient.Timeout != defaultTimeout && c.httpClient.Timeout > 0 {
+			c.provisionTimeout = c.httpClient.Timeout
+		} else {
+			c.provisionTimeout = defaultProvisioningTimeout
+		}
+	}
+
 	// Wire the auth transport on top of whatever the caller supplied. We
 	// preserve every field on the caller's *http.Client (Timeout,
 	// CheckRedirect, Jar, Transport) and chain the caller's Transport as the
@@ -154,21 +222,55 @@ func New(opts ...Option) *Client {
 	if base == nil {
 		base = http.DefaultTransport
 	}
-	wrapped := &http.Client{
+	sharedTransport := &authTransport{
+		base:      base,
+		apiKey:    c.apiKey,
+		userAgent: c.userAgent,
+	}
+	c.httpClient = &http.Client{
 		Timeout:       c.httpClient.Timeout,
 		CheckRedirect: c.httpClient.CheckRedirect,
 		Jar:           c.httpClient.Jar,
-		Transport: &authTransport{
-			base:      base,
-			apiKey:    c.apiKey,
-			userAgent: c.userAgent,
-		},
+		Transport:     sharedTransport,
+	}
+
+	// provisionClient shares the same (auth-wrapped) transport but carries NO
+	// client-wide Timeout. Provisioning deadlines are enforced per-request via
+	// the request context (see provisionContext) so a slow synchronous
+	// provision can run for the full provisionTimeout instead of being killed
+	// at the shorter read-path Timeout. CheckRedirect / Jar are preserved so
+	// the two clients behave identically apart from the timeout cap.
+	c.provisionClient = &http.Client{
+		Timeout:       0,
+		CheckRedirect: c.httpClient.CheckRedirect,
+		Jar:           c.httpClient.Jar,
+		Transport:     sharedTransport,
 	}
-	c.httpClient = wrapped
 
 	return c
 }
 
+// provisionContext derives a child context carrying the provisioning deadline.
+//
+// It never *extends* a deadline the caller already set: if ctx already has an
+// earlier deadline (the caller passed context.WithTimeout themselves) that
+// tighter deadline wins. It only adds the provisioning budget when the caller
+// left the context open-ended. The returned cancel func must always be called.
+func (c *Client) provisionContext(ctx context.Context) (context.Context, context.CancelFunc) {
+	d := c.provisionTimeout
+	if d <= 0 {
+		d = defaultProvisioningTimeout
+	}
+	if existing, ok := ctx.Deadline(); ok {
+		// Caller set their own deadline; only shorten ours to honour it, never
+		// override a tighter caller budget.
+		if remaining := time.Until(existing); remaining <= d {
+			return context.WithCancel(ctx)
+		}
+	}
+	return context.WithTimeout(ctx, d)
+}
+
 // ─── internal HTTP helpers ────────────────────────────────────────────────────
 
 // get executes a GET request and decodes the JSON response into out.
@@ -187,19 +289,45 @@ func (c *Client) postJSON(ctx context.Context, path string, body any, out any) e
 }
 
 // postJSONWithHeaders is like postJSON but also sets extra request headers.
-// Used by the provisioning helpers to forward the optional Idempotency-Key.
+// It runs on the read-path client (defaultTimeout). Provisioning helpers must
+// use provisionJSONWithHeaders instead so the synchronous provision gets the
+// longer provisioning deadline.
 func (c *Client) postJSONWithHeaders(ctx context.Context, path string, body any, headers map[string]string, out any) error {
-	var r io.Reader
-	if body != nil {
-		b, err := json.Marshal(body)
-		if err != nil {
-			return fmt.Errorf("marshalling request body: %w", err)
-		}
-		r = bytes.NewReader(b)
+	r, err := jsonBodyReader(body)
+	if err != nil {
+		return err
 	}
 	return c.doWithHeaders(ctx, http.MethodPost, path, r, headers, out)
 }
 
+// provisionJSONWithHeaders POSTs a JSON body for a synchronous provisioning
+// call. It routes through provisionClient (no client-wide Timeout cap) and
+// applies the longer provisioning deadline via the request context, so a slow
+// hot-pool provision can complete instead of timing out at the read-path 30 s
+// and orphaning the resource behind a 409 idempotency_key_in_progress retry.
+func (c *Client) provisionJSONWithHeaders(ctx context.Context, path string, body any, headers map[string]string, out any) error {
+	r, err := jsonBodyReader(body)
+	if err != nil {
+		return err
+	}
+	pctx, cancel := c.provisionContext(ctx)
+	defer cancel()
+	return c.doWithClient(pctx, c.provisionClient, http.MethodPost, path, r, headers, out)
+}
+
+// jsonBodyReader marshals body to a re-readable JSON reader. A nil body yields
+// a nil reader (no Content-Type is set downstream in that case).
+func jsonBodyReader(body any) (io.Reader, error) {
+	if body == nil {
+		return nil, nil
+	}
+	b, err := json.Marshal(body)
+	if err != nil {
+		return nil, fmt.Errorf("marshalling request body: %w", err)
+	}
+	return bytes.NewReader(b), nil
+}
+
 // delete executes a DELETE request and decodes the JSON response into out.
 func (c *Client) delete(ctx context.Context, path string, out any) error {
 	return c.do(ctx, http.MethodDelete, path, nil, out)
@@ -211,8 +339,18 @@ func (c *Client) do(ctx context.Context, method, path string, body io.Reader, ou
 }
 
 // doWithHeaders is like do but allows the caller to attach extra request
-// headers (e.g. Idempotency-Key). headers may be nil.
+// headers (e.g. Idempotency-Key). headers may be nil. It runs on the read-path
+// httpClient (read-path Timeout).
 func (c *Client) doWithHeaders(ctx context.Context, method, path string, body io.Reader, headers map[string]string, out any) error {
+	return c.doWithClient(ctx, c.httpClient, method, path, body, headers, out)
+}
+
+// doWithClient executes an HTTP request on the supplied client, retrying once
+// on a transport error or 5xx. It is the shared core behind doWithHeaders
+// (read path) and provisionJSONWithHeaders (provisioning path); the only
+// difference between the two is the *http.Client (hence the timeout regime)
+// and the request context.
+func (c *Client) doWithClient(ctx context.Context, hc *http.Client, method, path string, body io.Reader, headers map[string]string, out any) error {
 	rawURL := c.baseURL + path
 
 	// If we have a body reader we need to be able to re-read it on retry.
@@ -246,7 +384,7 @@ func (c *Client) doWithHeaders(ctx context.Context, method, path string, body io
 			}
 		}
 
-		resp, err := c.httpClient.Do(req)
+		resp, err := hc.Do(req)
 		if err != nil {
 			lastErr = fmt.Errorf("request failed: %w", err)
 			if attempt == 0 {

instant/database.go

@@ -34,7 +34,7 @@ func (c *Client) ProvisionDatabase(ctx context.Context, opts *ProvisionOpts) (*P
 	}
 
 	var result ProvisionResult
-	if err := c.postJSONWithHeaders(ctx, "/db/new", body, provisionHeaders(opts), &result); err != nil {
+	if err := c.provisionJSONWithHeaders(ctx, "/db/new", body, provisionHeaders(opts), &result); err != nil {
 		return nil, fmt.Errorf("ProvisionDatabase: %w", err)
 	}
 	if result.Token == "" {

instant/deploy.go

@@ -171,8 +171,16 @@ func (c *Client) Deploy(ctx context.Context, opts DeployOpts) (*Deployment, erro
 		return nil, fmt.Errorf("instant: closing multipart writer: %w", err)
 	}
 
+	// Deploy is a synchronous create like the /*/new provisioning endpoints:
+	// the API blocks while the Kaniko build is kicked off and the row is
+	// written. Use the provisioning client (no 30 s read-path cap) plus the
+	// longer provisioning deadline so a slow accept under load doesn't time out
+	// client-side and strand the build behind a 409 idempotency conflict.
+	pctx, cancel := c.provisionContext(ctx)
+	defer cancel()
+
 	url := c.baseURL + "/deploy/new"
-	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, &buf)
+	req, err := http.NewRequestWithContext(pctx, http.MethodPost, url, &buf)
 	if err != nil {
 		return nil, fmt.Errorf("instant: building deploy request: %w", err)
 	}
@@ -181,7 +189,7 @@ func (c *Client) Deploy(ctx context.Context, opts DeployOpts) (*Deployment, erro
 		req.Header.Set("Idempotency-Key", opts.IdempotencyKey)
 	}
 
-	resp, err := c.httpClient.Do(req)
+	resp, err := c.provisionClient.Do(req)
 	if err != nil {
 		return nil, fmt.Errorf("instant: deploy request failed: %w", err)
 	}

instant/mongodb.go

@@ -34,7 +34,7 @@ func (c *Client) ProvisionMongoDB(ctx context.Context, opts *ProvisionOpts) (*Pr
 	}
 
 	var result ProvisionResult
-	if err := c.postJSONWithHeaders(ctx, "/nosql/new", body, provisionHeaders(opts), &result); err != nil {
+	if err := c.provisionJSONWithHeaders(ctx, "/nosql/new", body, provisionHeaders(opts), &result); err != nil {
 		return nil, fmt.Errorf("ProvisionMongoDB: %w", err)
 	}
 	if result.Token == "" {