Add http_headers and http_path_prefix provider configuration options for HTTP proxy support

NEXT_CHANGELOG.md

@@ -8,6 +8,9 @@
 
 * Add `databricks_users` data source ([#4028](https://github.com/databricks/terraform-provider-databricks/pull/4028))
 * Improve `databricks_service_principals` data source ([#5164](https://github.com/databricks/terraform-provider-databricks/pull/5164))
+* Add `http_headers` and `http_path_prefix` provider configuration options for HTTP proxy support ([#XXXX](https://github.com/databricks/terraform-provider-databricks/pull/XXXX)).
+
+  These options enable the provider to work with HTTP proxies that require custom authentication headers or URL path rewriting.
 
 ### Bug Fixes
 

docs/index.md

@@ -142,6 +142,28 @@ The provider supports additional configuration parameters not related to authent
 * `debug_headers` - (optional, environment variable `DATABRICKS_DEBUG_HEADERS`) Applicable only when `TF_LOG=DEBUG` is set. Debug HTTP headers of requests made by the provider. Default is *false*. We recommend turning this flag on only under exceptional circumstances, when troubleshooting authentication issues. Turning this flag on will log first `debug_truncate_bytes` of any HTTP header value in cleartext.
 * `skip_verify` - skips SSL certificate verification for HTTP calls. *Use at your own risk.* Default is *false* (don't skip verification).
 
+The following parameters can be used for HTTP proxy scenarios:
+
+* `http_headers` - (optional) A map of custom HTTP headers to add to all API requests. This is useful when working with HTTP proxies that require custom authentication headers. This field is marked as sensitive.
+* `http_path_prefix` - (optional) A path prefix to prepend to all API request URLs. This is useful when working with HTTP proxies that use path-based routing.
+
+### HTTP Proxy Configuration Example
+
+When using an HTTP proxy that requires custom headers or path rewriting:
+
+```hcl
+provider "databricks" {
+  host  = "https://proxy.example.com"
+  token = "your-token"
+
+  http_headers = {
+    "X-Custom-Auth" = "auth-value"
+  }
+
+  http_path_prefix = "/proxy"
+}
+```
+
 !> **Warning** Sensitive credentials are printed to the log when `debug_headers` is `true`. Use it for troubleshooting purposes only.
 
 ### `host` argument
@@ -173,7 +195,7 @@ There are currently a number of supported methods to [authenticate](https://docs
 !> **Warning** Please be aware that hard coding any credentials in plain text is not something that is recommended. We strongly recommend using a Terraform backend that supports encryption. Please use [environment variables](#environment-variables), `~/.databrickscfg` file, encrypted `.tfvars` files or secret store of your choice (Hashicorp [Vault](https://www.vaultproject.io/), AWS [Secrets Manager](https://aws.amazon.com/secrets-manager/), AWS [Param Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html), Azure [Key Vault](https://azure.microsoft.com/en-us/services/key-vault/))
 
 ### Authenticating with GitHub OpenID Connect (OIDC)
-The arguments `host` and `client_id` are used for the authentication which maps to the `github-oidc` authentication type. 
+The arguments `host` and `client_id` are used for the authentication which maps to the `github-oidc` authentication type.
 
 These can be declared in the provider block or set in the environment variables `DATABRICKS_HOST` and `DATABRICKS_CLIENT_ID` respectively. Example:
 
@@ -200,7 +222,7 @@ provider "databricks" {
 
 ### Authenticating with Databricks CLI
 
-The provider can authenticate using the Databricks CLI. After logging in with the `databricks auth login` command to your account or workspace, you only need to specify the name of the profile in your provider configuration. Terraform will automatically read and reuse the cached OAuth token to interact with the Databricks REST API. See [the user-to-machine authentication guide](https://docs.databricks.com/aws/en/dev-tools/cli/authentication#oauth-user-to-machine-u2m-authentication) for more details. 
+The provider can authenticate using the Databricks CLI. After logging in with the `databricks auth login` command to your account or workspace, you only need to specify the name of the profile in your provider configuration. Terraform will automatically read and reuse the cached OAuth token to interact with the Databricks REST API. See [the user-to-machine authentication guide](https://docs.databricks.com/aws/en/dev-tools/cli/authentication#oauth-user-to-machine-u2m-authentication) for more details.
 
 You can specify a [CLI connection profile](https://docs.databricks.com/aws/en/dev-tools/cli/profiles) through `profile` parameter or `DATABRICKS_CONFIG_PROFILE` environment variable:
 
@@ -275,7 +297,7 @@ provider "databricks" {
 
 ### Authenticating with Workload Identity Federation (WIF)
 
-Workload Identity Federation can be used to authenticate Databricks from automated workflows. This is done through the tokens issued by the automation environment. For more details on environment variables regarding the specific environments, please see: https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-provider. 
+Workload Identity Federation can be used to authenticate Databricks from automated workflows. This is done through the tokens issued by the automation environment. For more details on environment variables regarding the specific environments, please see: https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-provider.
 
 To create resources at both the account and workspace levels, you can create two providers as shown below:
 
@@ -300,7 +322,7 @@ provider "databricks" {
 }
 ```
 
-Note: `auth_type` for Github Actions would be "github-oidc". For more details, please see the document linked above. 
+Note: `auth_type` for Github Actions would be "github-oidc". For more details, please see the document linked above.
 
 
 ## Special configurations for Azure
@@ -351,7 +373,7 @@ resource "databricks_user" "my-user" {
 }
 ```
 
-Follow the [Configuring OpenID Connect in Azure](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-azure). You can then use the Azure service principal to authenticate in databricks. 
+Follow the [Configuring OpenID Connect in Azure](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-azure). You can then use the Azure service principal to authenticate in databricks.
 
 
 There are `ARM_*` environment variables provide a way to share authentication configuration using the `databricks` provider alongside the [`azurerm` provider](https://registry.terraform.io/providers/hashicorp/azurerm/latest).
@@ -462,7 +484,7 @@ To make Databricks Terraform Provider generally available, we've moved it from [
 You should have [`.terraform.lock.hcl`](https://github.com/databrickslabs/terraform-provider-databricks/blob/v0.6.2/scripts/versions-lock.hcl) file in your state directory that is checked into source control. terraform init will give you the following warning.
 
 ```text
-Warning: Additional provider information from registry 
+Warning: Additional provider information from registry
 
 The remote registry returned warnings for registry.terraform.io/databrickslabs/databricks:
 - For users on Terraform 0.13 or greater, this provider has moved to databricks/databricks. Please update your source in required_providers.

internal/providers/client/databricks_client.go

@@ -3,6 +3,8 @@ package client
 import (
 	"context"
 	"fmt"
+	"net/http"
+	"strings"
 
 	"github.com/databricks/databricks-sdk-go/client"
 	"github.com/databricks/databricks-sdk-go/config"
@@ -11,17 +13,47 @@ import (
 	"github.com/hashicorp/terraform-plugin-log/tflog"
 )
 
+// HTTPConfig holds configuration for custom HTTP transport behavior.
+type HTTPConfig struct {
+	// Headers are custom HTTP headers added to all API requests.
+	Headers map[string]string
+	// PathPrefix is prepended to all API request URL paths.
+	PathPrefix string
+}
+
+// httpTransportWrapper wraps an http.RoundTripper to add custom headers
+// and path prefix to all requests.
+type httpTransportWrapper struct {
+	base       http.RoundTripper
+	headers    map[string]string
+	pathPrefix string
+}
+
+// RoundTrip implements http.RoundTripper interface.
+func (t *httpTransportWrapper) RoundTrip(req *http.Request) (*http.Response, error) {
+	// Add custom headers
+	for key, val := range t.headers {
+		req.Header.Set(key, val)
+	}
+	// Add path prefix
+	if t.pathPrefix != "" {
+		req.URL.Path = strings.TrimSuffix(t.pathPrefix, "/") + req.URL.Path
+	}
+	return t.base.RoundTrip(req)
+}
+
 // PrepareDatabricksClient makes some common adjustments to the config that apply in all cases
 // and returns a ready-to-use Databricks client. This includes:
 // - mapping deprecated auth types to their newer counterparts
 // - ensuring the config is resolved
 // - setting a default retry timeout if not set
 // - setting a default HTTP timeout if not set
+// - configuring custom HTTP headers and path prefix if provided
 //
 // TODO: this should be colocated with the definition of DatabricksClient in common/client.go, but
 // this isn't possible without introducing a circular dependency. Fixing this will require refactoring
 // DatabricksClient out of the common package.
-func PrepareDatabricksClient(ctx context.Context, cfg *config.Config, configCustomizer func(*config.Config) error) (*common.DatabricksClient, error) {
+func PrepareDatabricksClient(ctx context.Context, cfg *config.Config, configCustomizer func(*config.Config) error, httpConfig *HTTPConfig) (*common.DatabricksClient, error) {
 	if cfg.AuthType != "" {
 		// mapping from previous Google authentication types
 		// and current authentication types from Databricks Go SDK
@@ -53,6 +85,24 @@ func PrepareDatabricksClient(ctx context.Context, cfg *config.Config, configCust
 			return nil, err
 		}
 	}
+	// Set up custom HTTP transport if headers or path prefix are specified
+	if httpConfig != nil && (len(httpConfig.Headers) > 0 || httpConfig.PathPrefix != "") {
+		baseTransport := cfg.HTTPTransport
+		if baseTransport == nil {
+			baseTransport = http.DefaultTransport
+		}
+		cfg.HTTPTransport = &httpTransportWrapper{
+			base:       baseTransport,
+			headers:    httpConfig.Headers,
+			pathPrefix: httpConfig.PathPrefix,
+		}
+		if len(httpConfig.Headers) > 0 {
+			tflog.Debug(ctx, fmt.Sprintf("Custom HTTP headers configured: %d header(s)", len(httpConfig.Headers)))
+		}
+		if httpConfig.PathPrefix != "" {
+			tflog.Debug(ctx, fmt.Sprintf("Custom HTTP path prefix configured: %s", httpConfig.PathPrefix))
+		}
+	}
 	client, err := client.New(cfg)
 	if err != nil {
 		return nil, err

internal/providers/client/databricks_client_test.go

@@ -0,0 +1,163 @@
+package client
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestHttpTransportWrapper_RoundTrip_Headers(t *testing.T) {
+	// Create a test server to capture the request
+	var capturedHeaders http.Header
+	var capturedPath string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		capturedHeaders = r.Header.Clone()
+		capturedPath = r.URL.Path
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	// Create the transport wrapper with custom headers
+	wrapper := &httpTransportWrapper{
+		base: http.DefaultTransport,
+		headers: map[string]string{
+			"X-Custom-Header": "custom-value",
+			"X-Actor-Id":      "actor-123",
+		},
+	}
+
+	// Create a request and send it
+	req, err := http.NewRequest("GET", server.URL+"/api/test", nil)
+	require.NoError(t, err)
+
+	client := &http.Client{Transport: wrapper}
+	resp, err := client.Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+
+	// Verify headers were added
+	assert.Equal(t, "custom-value", capturedHeaders.Get("X-Custom-Header"))
+	assert.Equal(t, "actor-123", capturedHeaders.Get("X-Actor-Id"))
+	assert.Equal(t, "/api/test", capturedPath)
+}
+
+func TestHttpTransportWrapper_RoundTrip_PathPrefix(t *testing.T) {
+	// Create a test server to capture the request
+	var capturedPath string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		capturedPath = r.URL.Path
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	// Create the transport wrapper with path prefix
+	wrapper := &httpTransportWrapper{
+		base:       http.DefaultTransport,
+		pathPrefix: "/proxy/env-01",
+	}
+
+	// Create a request and send it
+	req, err := http.NewRequest("GET", server.URL+"/api/test", nil)
+	require.NoError(t, err)
+
+	client := &http.Client{Transport: wrapper}
+	resp, err := client.Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+
+	// Verify path prefix was added
+	assert.Equal(t, "/proxy/env-01/api/test", capturedPath)
+}
+
+func TestHttpTransportWrapper_RoundTrip_PathPrefixWithTrailingSlash(t *testing.T) {
+	// Create a test server to capture the request
+	var capturedPath string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		capturedPath = r.URL.Path
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	// Create the transport wrapper with path prefix that has trailing slash
+	wrapper := &httpTransportWrapper{
+		base:       http.DefaultTransport,
+		pathPrefix: "/proxy/env-01/",
+	}
+
+	// Create a request and send it
+	req, err := http.NewRequest("GET", server.URL+"/api/test", nil)
+	require.NoError(t, err)
+
+	client := &http.Client{Transport: wrapper}
+	resp, err := client.Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+
+	// Verify trailing slash is trimmed from prefix
+	assert.Equal(t, "/proxy/env-01/api/test", capturedPath)
+}
+
+func TestHttpTransportWrapper_RoundTrip_HeadersAndPathPrefix(t *testing.T) {
+	// Create a test server to capture the request
+	var capturedHeaders http.Header
+	var capturedPath string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		capturedHeaders = r.Header.Clone()
+		capturedPath = r.URL.Path
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	// Create the transport wrapper with both headers and path prefix
+	wrapper := &httpTransportWrapper{
+		base: http.DefaultTransport,
+		headers: map[string]string{
+			"Authorization": "Bearer token123",
+		},
+		pathPrefix: "/environments/staging",
+	}
+
+	// Create a request and send it
+	req, err := http.NewRequest("GET", server.URL+"/api/2.0/clusters/list", nil)
+	require.NoError(t, err)
+
+	client := &http.Client{Transport: wrapper}
+	resp, err := client.Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+
+	// Verify both headers and path prefix were applied
+	assert.Equal(t, "Bearer token123", capturedHeaders.Get("Authorization"))
+	assert.Equal(t, "/environments/staging/api/2.0/clusters/list", capturedPath)
+}
+
+func TestHttpTransportWrapper_RoundTrip_EmptyConfig(t *testing.T) {
+	// Create a test server to capture the request
+	var capturedPath string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		capturedPath = r.URL.Path
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	// Create the transport wrapper with no customizations
+	wrapper := &httpTransportWrapper{
+		base:    http.DefaultTransport,
+		headers: nil,
+	}
+
+	// Create a request and send it
+	req, err := http.NewRequest("GET", server.URL+"/api/test", nil)
+	require.NoError(t, err)
+
+	client := &http.Client{Transport: wrapper}
+	resp, err := client.Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+
+	// Verify no modifications
+	assert.Equal(t, "/api/test", capturedPath)
+}

internal/providers/pluginfw/pluginfw.go

@@ -92,6 +92,17 @@ func providerSchemaPluginFramework() schema.Schema {
 			}
 		}
 	}
+	// Custom HTTP proxy configuration options
+	ps["http_headers"] = schema.MapAttribute{
+		Optional:    true,
+		Sensitive:   true,
+		ElementType: types.StringType,
+		Description: "Custom HTTP headers to add to all API requests. Useful for HTTP proxies that require custom authentication headers.",
+	}
+	ps["http_path_prefix"] = schema.StringAttribute{
+		Optional:    true,
+		Description: "Path prefix to prepend to all API request URLs. Useful for HTTP proxies that use path-based routing.",
+	}
 	return schema.Schema{
 		Attributes: ps,
 	}
@@ -169,7 +180,36 @@ func (p *DatabricksProviderPluginFramework) configureDatabricksClient(ctx contex
 	} else {
 		tflog.Info(ctx, "(plugin framework) No attributes specified in provider configuration")
 	}
-	databricksClient, err := client.PrepareDatabricksClient(ctx, cfg, p.configCustomizer)
+	// Read custom HTTP configuration
+	var httpConfig *client.HTTPConfig
+	var httpHeaders types.Map
+	resp.Diagnostics.Append(req.Config.GetAttribute(ctx, path.Root("http_headers"), &httpHeaders)...)
+	var httpPathPrefix types.String
+	resp.Diagnostics.Append(req.Config.GetAttribute(ctx, path.Root("http_path_prefix"), &httpPathPrefix)...)
+	if resp.Diagnostics.HasError() {
+		return nil
+	}
+	if !httpHeaders.IsNull() && !httpHeaders.IsUnknown() || !httpPathPrefix.IsNull() && !httpPathPrefix.IsUnknown() {
+		headers := make(map[string]string)
+		if !httpHeaders.IsNull() && !httpHeaders.IsUnknown() {
+			for k, v := range httpHeaders.Elements() {
+				if strVal, ok := v.(types.String); ok && !strVal.IsNull() {
+					headers[k] = strVal.ValueString()
+				}
+			}
+		}
+		pathPrefix := ""
+		if !httpPathPrefix.IsNull() && !httpPathPrefix.IsUnknown() {
+			pathPrefix = httpPathPrefix.ValueString()
+		}
+		if len(headers) > 0 || pathPrefix != "" {
+			httpConfig = &client.HTTPConfig{
+				Headers:    headers,
+				PathPrefix: pathPrefix,
+			}
+		}
+	}
+	databricksClient, err := client.PrepareDatabricksClient(ctx, cfg, p.configCustomizer, httpConfig)
 	if err != nil {
 		resp.Diagnostics.AddError("Failed to configure Databricks client", err.Error())
 		return nil