feat(csharp/src/Drivers/Databricks): Implement ClientCredentialsProvider

[toddmeng-db]

Implements OAuth 2.0 Client Credentials flow for machine-to-machine (M2M) authentication in the Databricks ADBC driver.

This is the first PR in a series to add complete OAuth client credentials support. This PR focuses on the core token
provider infrastructure.

Changes

• New OAuthClientCredentialsProvider class: Handles OAuth token lifecycle
  • Fetches access tokens from Databricks OAuth service using client credentials
  • Implements automatic token refresh based on expiration
  • Caches tokens to minimize authentication requests
  • Thread-safe token management
• Unit tests in OAuthClientCredentialsProviderTests:
  • Token retrieval and caching
  • Automatic refresh when token expires
  • Error handling for invalid credentials
• Test configuration support for integration testing

Testing

Unit Tests

Run the OAuth client credentials tests:

cd csharp
dotnet test --filter "FullyQualifiedName~OAuthClientCredentialsServiceTests"

Integration Testing

To test with real credentials:

1. Create a databricks_test_config.json:
{
  "oauth_client_id": "your-client-id",
  "oauth_client_secret": "your-client-secret",
  "host": "your-workspace.databricks.com"
}
2. Set the environment variable:
  - macOS/Linux:
  export DATABRICKS_TEST_CONFIG_FILE=/path/to/databricks_test_config.json
  - Windows PowerShell:
  $env:DATABRICKS_TEST_CONFIG_FILE = "C:\path\to\databricks_test_config.json"
3. Run the tests (see command above)

Follow-up Work

- Integrate OAuth client credentials provider with the Databricks driver connection flow
- Add support for OAuth scopes configuration
- Integrate with existing authentication parameter handling

Authentication Flow

The provider implements the standard OAuth 2.0 Client Credentials grant:
1. Client sends credentials to OAuth token endpoint
2. Server validates and returns access token with expiration
3. Provider caches token and reuses until near expiration
4. Provider automatically refreshes token when needed

[eric-wang-1990]

Can we rename this, it is not a service maybe call it a provider

[eric-wang-1990]

Why do we need it to be async? This is get credentials, which should be blocking?

[toddmeng-db]

Seems like it's a bit awkward to implement a non-async method when there's an http call involved in c#?

public Credentials GetCredentials()
{
    var response = Task.Run(() => _httpClient.GetAsync(...))
                       .Unwrap()
                       .GetAwaiter()
                       .GetResult();
    // process response...
    return credentials;
}

[eric-wang-1990]

Yeah, I guess we need to take a look at the callsite, maybe check SparkHttpConnection to see how we pass those token as the header, I believe those functions are all sync functions, which means it should call sync functions as well.
Maybe we should just expose a sync version of GetAccessToken() which return GetAccessTokenAsync().GetAwaiter().GetResult();

[toddmeng-db]

Just curious, what's the concern with using async here?

[eric-wang-1990]

There is no concern of using async, but c# appears need to follow a pattern than only async function can call async function. If you integrate with the rest of the driver you can see all the callsite is not async functions. So you must somehow return some results immediately, or else you will only get the Task.

[toddmeng-db]

I see, I noticed that too...

[CurtHagenlocher]

Thanks! I take it there will be a followup where this is hooked up to the driver?