Use X-Databricks-Workspace-Id for workspace routing (#1688)

## Summary

Workspace-scoped API calls now identify the target workspace via the
`X-Databricks-Workspace-Id` request header instead of
`X-Databricks-Org-Id`. The value source is unchanged: it still comes
from `Config.WorkspaceID` (or the `DATABRICKS_WORKSPACE_ID` environment
variable). The response-side header read in `CurrentWorkspaceID` is left
on the legacy name for now. A new `CurrentWorkspaceIdentifier` helper is
added to support workspace identifier formats other than classic numeric
IDs.

## Why

On unified Databricks hosts that serve multiple workspaces, the SDK
sends a routing header on every workspace-scoped HTTP request so the
gateway can dispatch it to the correct workspace. Until now that header
was `X-Databricks-Org-Id`.

The Databricks platform is consolidating workspace addressing onto a
single header, `X-Databricks-Workspace-Id`, which accepts a broader
range of workspace identifier formats and replaces the older
single-purpose channels going forward. The previous header continues to
be accepted by the platform, so older SDK versions keep working, but new
SDK versions should use the new name.

This PR is the Go SDK's part of that migration on the **request** side.
The platform's response-side echo header has not yet been renamed, so
the SDK continues to read the workspace ID back from the legacy
`X-Databricks-Org-Id` response header.

## What changed

### Interface changes

- **New**: `WorkspaceClient.CurrentWorkspaceIdentifier(ctx
context.Context) (string, error)` — returns the workspace identifier
exactly as the server echoes it back, without attempting to parse it as
a number. Use this in preference to `CurrentWorkspaceID` if your code
may run against workspaces whose identifier is not a classic numeric ID.
- `Config.WorkspaceID` keeps the same field name, type (`string`), and
environment variable (`DATABRICKS_WORKSPACE_ID`). The doc comment is
widened to note that the value may now be either a classic numeric
workspace ID or another workspace identifier format that the server
understands.
- `CurrentWorkspaceID(ctx) (int64, error)` is unchanged signature-wise
but is now a thin `ParseInt` adapter over `CurrentWorkspaceIdentifier`.
Existing callers keep working byte-for-byte against classic numeric
workspaces; for non-numeric identifiers it returns the parse error from
`strconv.ParseInt`.

### Behavioral changes

- Workspace-scoped requests no longer send `X-Databricks-Org-Id`. They
now send `X-Databricks-Workspace-Id` (with the same value, gated on
`Config.WorkspaceID` being non-empty). Account-scoped requests are
unaffected.
- `WorkspaceClient.CurrentWorkspaceID` and the new
`CurrentWorkspaceIdentifier` send `X-Databricks-Workspace-Id` as the
routing header and read the workspace ID back from the legacy
`X-Databricks-Org-Id` response header on `/api/2.0/preview/scim/v2/Me` —
the server still emits the legacy name on responses, and the
response-side migration is on a separate, later schedule.

### Internal changes

- Regenerated `service/*/impl.go` and
`internal/testspecs/service/*/impl.go` from the corresponding generator
template change: 34 files, 944 line swaps, no logic change.
- Updated the hand-written sites that emit the same header on
workspace-scoped calls but are not covered by the generator:
`service/iam/ext_impl.go`, `service/sharing/ext_api.go`,
`service/workspace/ext_utilities.go`, and `workspace_functions.go`.
- Manually patched 16 occurrences in newly-merged generated code
(`service/bundle/impl.go`, `service/dashboards/impl.go:733`,
`service/postgres/impl.go:625`) that came in via an auto-codegen update
made before the generator template change was available. This will be a
no-op against the next regen after the generator change lands upstream.
- Renamed the two `TestCurrentWorkspaceID*` test functions so the names
match the new header on the request side.

## How is this tested?

- `make fmt`, `make lint`, and `make test` all pass locally. Full unit
suite is green.
- The four `workspace_functions_test.go` tests cover both helpers: they
assert the new `X-Databricks-Workspace-Id` request header is sent, that
the SDK reads the workspace ID back from the legacy
`X-Databricks-Org-Id` response header, and that
`CurrentWorkspaceIdentifier` returns non-numeric identifiers verbatim
while `CurrentWorkspaceID` returns a parse error for those.
- Spot-checked the regenerated diff: every changed line in
`service/*/impl.go` is exclusively the request-header literal swap; no
collateral changes.
- End-to-end verified against a staging workspace: the SDK sends
`X-Databricks-Workspace-Id` correctly and the gateway routes the request
to the expected workspace. The response on `/api/2.0/preview/scim/v2/Me`
echoes the workspace ID on `x-databricks-org-id`, which is why the
response read site stays on the legacy name.
- Added a new acceptance test `TestAccUnifiedHostSendsWorkspaceIdHeader`
in `internal/unified_host_test.go`. It hooks a capturing
`http.RoundTripper` into a workspace client built against a unified host
and verifies on the wire that (1) the SDK sends
`X-Databricks-Workspace-Id` and not `X-Databricks-Org-Id` on the
request, (2) the server still echoes `X-Databricks-Org-Id` on the
response, and (3) the SCIM Me call succeeds end-to-end. Skips when the
workspace-test environment variables (`UNIFIED_HOST`,
`THIS_WORKSPACE_ID`, `TEST_ACCOUNT_ID`, `DATABRICKS_CLIENT_ID`,
`DATABRICKS_CLIENT_SECRET`) are missing.

---------

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>

Author: Divyansh-db

NEXT_CHANGELOG.md

@@ -12,4 +12,6 @@
 
 ### Internal Changes
 
+* Switch workspace addressing header on workspace-scoped API calls from `X-Databricks-Org-Id` to `X-Databricks-Workspace-Id`. The value continues to come from `Config.WorkspaceID` (`DATABRICKS_WORKSPACE_ID`), and now accepts either a classic numeric workspace ID or a CPDR connection ID (server disambiguates). Part of the DECO + Unified Workspace Addressing initiative.
+
 ### API Changes

config/config.go

@@ -123,7 +123,9 @@ type Config struct {
 	// Databricks Account ID for Accounts API. This field is used in dependencies.
 	AccountID string `name:"account_id" env:"DATABRICKS_ACCOUNT_ID"`
 
-	// Databricks Workspace ID for Workspace clients when working with unified hosts
+	// Databricks Workspace ID for Workspace clients when working with unified hosts.
+	// Accepts either a classic numeric workspace ID or a CPDR connection ID; the
+	// server disambiguates.
 	WorkspaceID string `name:"workspace_id" env:"DATABRICKS_WORKSPACE_ID"`
 
 	Token    string `name:"token" env:"DATABRICKS_TOKEN" auth:"pat,sensitive"`

internal/testspecs/service/httpcallv2/impl.go

@@ -0,0 +1,100 @@
+package internal
+
+import (
+	"context"
+	"net/http"
+	"testing"
+
+	"github.com/databricks/databricks-sdk-go"
+	"github.com/databricks/databricks-sdk-go/config"
+	"github.com/databricks/databricks-sdk-go/service/iam"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// captureHeadersTransport snapshots the request and response headers for a
+// single matching URL path and lets every other request through unchanged.
+type captureHeadersTransport struct {
+	inner http.RoundTripper
+	path  string
+
+	reqHdr  http.Header
+	respHdr http.Header
+}
+
+func (t *captureHeadersTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+	matched := req.URL.Path == t.path
+	if matched {
+		t.reqHdr = req.Header.Clone()
+	}
+	resp, err := t.inner.RoundTrip(req)
+	if matched && resp != nil {
+		t.respHdr = resp.Header.Clone()
+	}
+	return resp, err
+}
+
+// TestAccUnifiedHostSendsWorkspaceIdHeader verifies, against a real unified
+// (SPOG) host, that:
+//  1. the SDK sends the new X-Databricks-Workspace-Id routing header on
+//     workspace-scoped requests, and no longer sends the legacy
+//     X-Databricks-Org-Id, and
+//  2. the server still echoes the workspace ID on the legacy
+//     X-Databricks-Org-Id response header — the response-side migration has
+//     not yet happened, which is why CurrentWorkspaceID still reads the
+//     legacy name in workspace_functions.go.
+//
+// Uses OAuth M2M auth (ClientID/ClientSecret) and probes the SCIM Me endpoint
+// as the workspace-scoped call. A transport-level header capture is hooked in
+// to verify which routing header actually crosses the wire.
+//
+// Runs against the workspace test environment (TEST_ENVIRONMENT_TYPE in
+// {WORKSPACE, UC_WORKSPACE}). Requires UNIFIED_HOST, THIS_WORKSPACE_ID,
+// TEST_ACCOUNT_ID, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET in the
+// environment.
+func TestAccUnifiedHostSendsWorkspaceIdHeader(t *testing.T) {
+	loadDebugEnvIfRunsFromIDE(t, "workspace")
+	if envType := testEnvironmentType(); envType != "" && envType != "WORKSPACE" && envType != "UC_WORKSPACE" {
+		skipf(t)("Skipping workspace test: TEST_ENVIRONMENT_TYPE=%s", envType)
+	}
+
+	const apiPath = "/api/2.0/preview/scim/v2/Me"
+	transport := &captureHeadersTransport{
+		inner: http.DefaultTransport,
+		path:  apiPath,
+	}
+	workspaceID := GetEnvOrSkipTest(t, "THIS_WORKSPACE_ID")
+	cfg := &config.Config{
+		Host:               GetEnvOrSkipTest(t, "UNIFIED_HOST"),
+		AccountID:          GetEnvOrSkipTest(t, "TEST_ACCOUNT_ID"),
+		ClientID:           GetEnvOrSkipTest(t, "DATABRICKS_CLIENT_ID"),
+		ClientSecret:       GetEnvOrSkipTest(t, "DATABRICKS_CLIENT_SECRET"),
+		WorkspaceID:        workspaceID,
+		HTTPTransport:      transport,
+		HTTPTimeoutSeconds: 60,
+	}
+	if err := cfg.EnsureResolved(); err != nil {
+		skipf(t)("config resolve: %s", err)
+	}
+	t.Parallel()
+
+	ctx := context.Background()
+	w := databricks.Must(databricks.NewWorkspaceClient((*databricks.Config)(cfg)))
+
+	// (1) workspace-scoped probe against the unified host: must succeed.
+	me, err := w.CurrentUser.Me(ctx, iam.MeRequest{})
+	require.NoError(t, err, "workspace-scoped SCIM Me should pass through the unified host gateway with the new routing header")
+	assert.NotEmpty(t, me.UserName, "Me should return a non-empty UserName")
+
+	// (2) On the request side, the SDK must send the new header and not the legacy one.
+	require.NotNil(t, transport.reqHdr, "transport did not observe a request to %s", apiPath)
+	assert.Equal(t, workspaceID, transport.reqHdr.Get("X-Databricks-Workspace-Id"),
+		"SDK must send X-Databricks-Workspace-Id with Config.WorkspaceID")
+	assert.Empty(t, transport.reqHdr.Get("X-Databricks-Org-Id"),
+		"SDK must no longer send the legacy X-Databricks-Org-Id request header")
+
+	// (3) On the response side, the server still echoes the legacy header name during the migration.
+	require.NotNil(t, transport.respHdr, "transport did not observe a response from %s", apiPath)
+	assert.NotEmpty(t, transport.respHdr.Get("X-Databricks-Org-Id"),
+		"server is expected to still echo the legacy X-Databricks-Org-Id response header during the migration")
+}