SPOG: short-circuit CurrentWorkspaceID and add org-id header to SharesAPI (#1635)

## Changes

Two SPOG fixes for hand-written SDK extension methods that bypass the
generated code's per-call `X-Databricks-Org-Id` header logic.

### 1. Short-circuit `WorkspaceClient.CurrentWorkspaceID()`

Return `Config.WorkspaceID` directly when it is non-empty, instead of
making a `GET` to `/api/2.0/preview/scim/v2/Me` purely to read the
workspace id from the `X-Databricks-Org-Id` response header.

In the CLI bundle flow, `Config.WorkspaceID` is already populated before
this call runs, from any of:

- the matched profile in `.databrickscfg` (`workspace_id = ...`)
- the `?o=<id>` query parameter extracted from the bundle host
- the `DATABRICKS_WORKSPACE_ID` env var
- host metadata at `/.well-known/databricks-config`

Calling `Me()` to re-learn that value is a wasted round-trip, and on
SPOG
hosts it also breaks `databricks bundle open`:

```
Error: Unable to load OAuth Config (400 UNKNOWN)
Endpoint: GET https://<spog-host>/api/2.0/preview/scim/v2/Me
HTTP Status: 400 Bad Request
```

The previous implementation used `httpclient.WithResponseHeader` to read
`X-Databricks-Org-Id` from the response, but did not set it on the
request. SPOG's proxy rejects un-routed requests with the exact 400
above.

#### Why the short-circuit is sufficient

On SPOG, `WorkspaceID` is the routing key. To call `Me()` the request
must carry `X-Databricks-Org-Id`.

- If `Config.WorkspaceID` is set, return it without any API call.
- If `Config.WorkspaceID` is empty, we have no value to put in the
  header, so `Me()` on SPOG would fail regardless. A header-setting
  patch would add no reachable value.

On non-SPOG hosts the fallback API call still works without the header,
preserving pre-existing behavior.

### 2. Add `X-Databricks-Org-Id` to `SharesAPI.internalList()`

`service/sharing/ext_api.go` `SharesAPI.internalList()` is a
hand-written
extension that calls `a.client.Do()` directly and was missing the
`X-Databricks-Org-Id` request header. On SPOG hosts this causes
`shares.List(Paginated|All)` to fail with the same `Unable to load OAuth
Config` error. Matches the fix in #1634 for
`service/workspace/ext_utilities.go` `Upload()` / `Download()`.

Pointed out by Hector Castejon Diaz.

## Tests

### Unit tests

Added `workspace_functions_test.go`:

- `TestCurrentWorkspaceIDShortCircuitsWhenConfigHasWorkspaceID`: fails
  the test if `Me()` is called when `Config.WorkspaceID` is set, and
  asserts the returned value matches the configured id.
- `TestCurrentWorkspaceIDFallsBackToAPIWhenConfigMissingWorkspaceID`:
  asserts the pre-existing API fallback still works and parses the
  `X-Databricks-Org-Id` response header correctly.

No dedicated unit test added for the sharing header change; matches
#1634 which applied the same pattern without one. `make fmt test lint`
clean on the full SDK (1344 tests pass, 103 skipped).

### Empirical SPOG verification

Same bearer token, same endpoint, toggling only the request header:

| Request | Response |
| ------------------------------------------ |
-------------------------------------- |
| `GET /Me` (no `X-Databricks-Org-Id`) | HTTP 400 `Unable to load OAuth
Config` |
| `GET /Me` with `X-Databricks-Org-Id: <id>` | HTTP 200 with valid user
JSON |

Confirms the missing request header is the cause.

### End-to-end with the CLI

Built the CLI against this branch via `go mod edit -replace` and ran on
a
real SPOG workspace (`db-deco-test-chrisst`, host
`https://db-deco-test.databricks.com`):

- `databricks bundle validate -t dev`: OK
- `databricks bundle deploy -t dev`: OK
- `databricks bundle run hello_world -t dev`: OK, job TERMINATED SUCCESS
- `databricks bundle open hello_world -t dev`: OK, opens the job URL.
  Previously failed with `Unable to load OAuth Config (400 UNKNOWN)`.

Signed-off-by: simon <simon.faltum@databricks.com>

---------

Signed-off-by: simon <simon.faltum@databricks.com>

Author: simonfaltum

NEXT_CHANGELOG.md

@@ -9,6 +9,8 @@
 ### Bug Fixes
 
  * Add `X-Databricks-Org-Id` header to `Workspace.Download()` and `Workspace.Upload()` for SPOG host compatibility.
+ * `WorkspaceClient.CurrentWorkspaceID()` now returns `Config.WorkspaceID` directly when set, instead of calling `/api/2.0/preview/scim/v2/Me`. This removes an API round-trip on every call where the workspace ID is already known (profile, `?o=` query param, env var, or host metadata) and fixes a failure on SPOG hosts where the unauthenticated probe request was rejected with `Unable to load OAuth Config`.
+ * Add `X-Databricks-Org-Id` header to `SharesAPI.internalList()` (`service/sharing/ext_api.go`) for SPOG host compatibility. Same class of bug as #1634 - a hand-written extension method was calling `a.client.Do()` without the header that the generated code paths set per-call.
 
 ### Documentation
 

service/sharing/ext_api.go

@@ -71,6 +71,10 @@ func (a *SharesAPI) internalList(ctx context.Context, request ListSharesRequest)
 	queryParams := make(map[string]any)
 	headers := make(map[string]string)
 	headers["Accept"] = "application/json"
+	cfg := a.client.Config
+	if cfg.WorkspaceID != "" {
+		headers["X-Databricks-Org-Id"] = cfg.WorkspaceID
+	}
 	err := a.client.Do(ctx, http.MethodGet, path, headers, queryParams, request, &listSharesResponse)
 	return &listSharesResponse, err
 }

workspace_functions.go

@@ -9,7 +9,15 @@ import (
 
 // CurrentWorkspaceID returns the workspace ID of the workspace that this client is
 // connected to.
+//
+// If Config.WorkspaceID is already set (from the databrickscfg profile, the
+// DATABRICKS_WORKSPACE_ID env var, host metadata, or a ?o= query param extracted
+// by the caller), it is returned without an API round-trip. Otherwise the ID is
+// fetched from the X-Databricks-Org-Id response header on /api/2.0/preview/scim/v2/Me.
 func (w *WorkspaceClient) CurrentWorkspaceID(ctx context.Context) (int64, error) {
+	if w.Config != nil && w.Config.WorkspaceID != "" {
+		return strconv.ParseInt(w.Config.WorkspaceID, 10, 64)
+	}
 	var workspaceIdStr string
 	err := w.apiClient.Do(ctx, "GET", "/api/2.0/preview/scim/v2/Me", httpclient.WithResponseHeader("X-Databricks-Org-Id", &workspaceIdStr))
 	if err != nil {

workspace_functions_test.go

@@ -0,0 +1,63 @@
+package databricks
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestCurrentWorkspaceIDShortCircuitsWhenConfigHasWorkspaceID(t *testing.T) {
+	// When Config.WorkspaceID is already populated (from profile, ?o= query
+	// param, env var, or host metadata), CurrentWorkspaceID returns it without
+	// hitting the API. This avoids a round-trip and sidesteps SPOG's routing
+	// requirement that requests to /api/2.0/preview/scim/v2/Me carry an
+	// X-Databricks-Org-Id header.
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/api/2.0/preview/scim/v2/Me" {
+			t.Errorf("Me() should not be called when Config.WorkspaceID is set")
+		}
+		http.NotFound(w, r)
+	}))
+	defer server.Close()
+
+	w, err := NewWorkspaceClient(&Config{
+		Host:        server.URL,
+		Token:       "token",
+		WorkspaceID: "7474644166319138",
+	})
+	require.NoError(t, err)
+
+	got, err := w.CurrentWorkspaceID(t.Context())
+	require.NoError(t, err)
+	assert.Equal(t, int64(7474644166319138), got)
+}
+
+func TestCurrentWorkspaceIDFallsBackToAPIWhenConfigMissingWorkspaceID(t *testing.T) {
+	var meCalls int
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/api/2.0/preview/scim/v2/Me" {
+			meCalls++
+			w.Header().Set("X-Databricks-Org-Id", "7474644166319138")
+			w.Write([]byte(`{}`))
+			return
+		}
+		// Other bootstrap calls (e.g. /.well-known/databricks-config) are not
+		// relevant to this test; respond with a benign 404.
+		http.NotFound(w, r)
+	}))
+	defer server.Close()
+
+	w, err := NewWorkspaceClient(&Config{
+		Host:  server.URL,
+		Token: "token",
+	})
+	require.NoError(t, err)
+
+	got, err := w.CurrentWorkspaceID(t.Context())
+	require.NoError(t, err)
+	assert.Equal(t, int64(7474644166319138), got)
+	assert.Equal(t, 1, meCalls)
+}