Add Salesforce Data 360 (Data Cloud) API support

[Copilot]

Adds typed client support for Salesforce Data Cloud (Data 360) APIs, covering SQL query, async query management, vector search (RAG), unified profile lookup, and metadata discovery. Also adds the Data Cloud token exchange flow required to obtain a TSE-scoped access token. Includes real-org integration tests and CI/scratch org fixes.

sf-auth: Data Cloud token exchange

• DataCloudTokenResponse — holds the TSE URL + Data Cloud access token; access_token and signature are redacted in Debug
• OAuthClient::exchange_for_data_cloud(sf_access_token, login_url) — performs the RFC 8693 token exchange (grant_type=urn:salesforce:grant-type:external:cdp). The client_id is intentionally omitted from the request body — the subject token alone identifies the caller for this grant type.

sf-rest: DataCloudClient + types

New DataCloudClient struct targeting the TSE URL (separate infrastructure from standard SF REST). Default API version 64.0.

Method	Endpoint
query_sql()	POST /services/data/v{ver}/ssot/query-sql
query_status()	GET …/query-sql/{queryId}
query_rows()	GET …/query-sql/{queryId}/rows
vector_search()	POST …/ssot/search-vector
profile(dmo, filters)	GET /api/v1/profile/{dmo}?filters=…
metadata(entity_type)	GET /api/v1/metadata?entityType=…

New types: DataCloudQueryRequest/Response, QueryMetadata, ColumnInfo, AsyncQueryStatus, VectorSearchRequest/Response/Result, DataCloudMetadataResponse/Object. ColumnInfo preserves original API casing (relevant since the 2026 casing change from lowercase to e.g. ssot__Id__c).

Integration tests (tests/integration/data_cloud.rs)

Seven real-org integration tests covering the full Data Cloud surface. All tests fail loudly if Data Cloud is not provisioned (no silent skipping):

Test	What it verifies
test_data_cloud_token_exchange_returns_tse_url_and_token	RFC 8693 exchange returns a ****** + TSE URL
test_data_cloud_metadata_discovery_returns_entities	/api/v1/metadata?entityType=DataModelObject returns ≥1 DMO
test_data_cloud_metadata_discovery_without_filter	Unfiltered metadata endpoint is reachable
test_data_cloud_sql_query_returns_column_metadata	SELECT * FROM <dmo> LIMIT 0 returns column schema (no data dependency)
test_data_cloud_async_sql_query_returns_query_id	Async query returns queryId; status poll + row fetch succeed
test_data_cloud_vector_search_endpoint_is_reachable	Vector search endpoint responds; index-not-found is accepted for fresh orgs
test_data_cloud_profile_endpoint_is_reachable	Profile endpoint responds; DMO-not-found is accepted for fresh orgs

DMO names from the metadata API are validated as safe SQL identifiers before interpolation.

Scratch org definition (config/project-scratch-def.json)

Added "DataCloud" to the scratch org features list so orgs created via sf org create scratch -f config/project-scratch-def.json have Salesforce Data Cloud provisioned.

CI fix (.github/workflows/ci.yml)

Added environment: GitHub composable-delivery and the git-credentials configuration step to the MSRV job, matching what Clippy/Test/Docs already have. The job was always failing because it had no access to GH_TOKEN_BUILDS, which is required to resolve the busbar-capability optional dependency in sf-bridge.

Note: The Test/Clippy/Docs/Integration/WASM Bridge failures are caused by the GH_TOKEN_BUILDS PAT having expired. Rotating that secret in the GitHub composable-delivery environment will resolve those jobs.

Usage

// 1. Exchange SF access token → Data Cloud token
//    OAuthConfig consumer key is not required for this grant type.
let oauth = OAuthClient::new(OAuthConfig::new(""));
let dc = oauth.exchange_for_data_cloud(&sf_token, &login_url).await?;

// 2. Build client with TSE URL + Data Cloud token
let client = DataCloudClient::new(&dc.instance_url, &dc.access_token)?;

// 3. Query
let resp = client.query_sql(&DataCloudQueryRequest {
    sql: "SELECT ssot__Id__c FROM Individual__dlm LIMIT 10".into(),
    page_size: None,
    r#async: None,
}).await?;

// 4. Vector search for RAG
let hits = client.vector_search(&VectorSearchRequest {
    index_name: "KnowledgeIndex".into(),
    query_text: "How do I reset my API key?".into(),
    top_k: Some(5),
}).await?;

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add Data 360 API endpoints</issue_title>
<issue_description>### Salesforce API

REST API (sf-rest)

Endpoint(s)

1. Data Cloud Query API (SQL/SOQL)

• Base URL: https://{TSE_Instance_Url}/services/data/v64.0/ssot/
• Synchronous SQL Query: POST /query-sql
  • Usage: Executing standard ANSI SQL against Data Model Objects (DMOs).
• Asynchronous Query Management:
  • POST /query-sql (with async: true) returns a queryId.
  • GET /query-sql/{queryId}: To check status.
  • GET /query-sql/{queryId}/rows: To fetch the paginated result set.

2. Profile & Calculated Insights API

• Base URL: https://{TSE_Instance_Url}/api/v1/
• Profile Lookup: GET /profile/{dataModelName}
  • Note: Use this for OData-style filtering (e.g., ?filters=[EmailAddress__c='user@example.com']).
• Calculated Insights: GET /calculated-insights/{insightName}
  • Usage: Retrieving specific KPIs calculated by the Data Cloud engine.

3. Vector Search API (For RAG)

• Vector Search Index: POST /services/data/v64.0/ssot/search-vector
  • Payload: Includes queryText and indexName. The engine returns chunks of text and similarity scores which your Rust host will then inject into the LLM prompt.

---

Implementation Mapping for busbar-sf-api

Handle the Token Exchange flow. Data Cloud often requires a "secondary" token (the Offcore Token) for the TSE endpoints.

Functionality	Endpoint Path	Method	Purpose in AgentScript
Identity Resolution	/api/v1/profile/Individual__dlm	GET	Resolve a user to a Unified ID.
Ad-hoc Query	/services/data/v64.0/ssot/query-sql	POST	Agent asks: "How many cases does this user have?"
Semantic Search	/services/data/v64.0/ssot/search-vector	POST	Agent asks: "How do I reset my password?" (RAG)
Token Exchange	/services/oauth2/token	POST	Exchange a Salesforce session for a Data Cloud TSE token.

Response Schemas

Note that as of late 2025, Salesforce rebranded these as Data 360 APIs, but the internal URI paths usually retain ssot or c360a.

1. The Core Query Endpoint (SQL)

This is the most critical endpoint for your petgraph engine to fetch structured context.

• Endpoint: POST /services/data/v64.0/ssot/query-sql
• Purpose: Executes ANSI SQL against your DMOs (Data Model Objects).
• JSON Schema Requirement:
  • Request: Contains the sql string and optional pageSize.
  • Response: Since 2025/2026, Salesforce has standardized on a "Metadata + Rows" structure.

Recommended Rust Structs (serde):

Rust

#[derive(Deserialize, Debug)]
pub struct QueryResponse {
    pub data: Vec<serde_json::Value>, // Array of objects (rows)
    pub metadata: QueryMetadata,
    pub done: bool,
    pub queryId: Option<String>,
    pub nextBatchId: Option<String>,
}

#[derive(Deserialize, Debug)]
pub struct QueryMetadata {
    pub columns: Vec<ColumnInfo>,
}

#[derive(Deserialize, Debug)]
pub struct ColumnInfo {
    pub name: String,
    pub type: String, // e.g., "numeric", "varchar", "timestamp_with_timezone"
}

Warning for 2026: Be sure your parser handles case-insensitive column names. Recent updates changed the default output from lowercase to the original API name casing (e.g., ssot__Id__c instead of ssot__id__c).

---

2. The Vector Search Endpoint (RAG)

• Endpoint: POST /services/data/v64.0/ssot/search-vector
• Request Payload:

JSON

{
  "indexName": "Knowledge_Articles_Index",
  "queryText": "How do I reset my API key?",
  "topK": 5
}

• Response: Returns a list of chunks, scores, and the original record IDs. Your runtime will use these IDs to then fetch the full record via the Query API if needed.

---

3. The Unified Profile API (v1)

• Endpoint: GET /api/v1/profile/{dataModelName}
• Key Filters: You need to support the ?filters=[Field='Value'] syntax.

4. Metadata Discovery API

If you want your Rust engine to "know" what fields are available (for auto-complete or validation), you need the metadata endpoint.

• Endpoint: GET /api/v1/metadata
• Filter: ?entityType=DataModelObject
• Response: Provides the primary keys, field types, and relationships (essential for building your petgraph representation of the data schema).

Summary Table for busbar-sf-api

Feature	Path	Method	Essential for...
SQL Query	/services/data/v64.0/ssot/query-sql	POST	General data retrieval and logic branching.
Vector Search	/services/data/v64.0/ssot/search-vector	POST	Semantic search and RAG grounding.
Profile Access	/api/v1/profile/{DMO}	GET	Personalization and Identity Resolution.
Discovery	/api/v1/metadata	GET	Val...

• Fixes Add Data 360 API endpoints #93

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[jlantz]

@copilot please resolve the build failures for clippy, docs, integration tests, MSRV, Test, and WASM Bridge Integration Tests. And we need to have integration tests that test against a real scratch org. If we need changes to the scratchdef, that should be available here in the repo

[Copilot]

@copilot please resolve the build failures for clippy, docs, integration tests, MSRV, Test, and WASM Bridge Integration Tests. And we need to have integration tests that test agains...

Addressed in commit 361cec8. Here's what was done for each failure:

MSRV (pre-existing): Added environment: GitHub composable-delivery and the git-credentials configuration step to the MSRV job in ci.yml. It was always failing because it had no access to GH_TOKEN_BUILDS (needed to resolve the busbar-capability optional dependency in sf-bridge).

Test / Clippy / Docs: These use --all-features which triggers busbar-capability (private repo in sf-bridge). The GH_TOKEN_BUILDS PAT in the GitHub composable-delivery environment appears to have expired between the last passing CI run (Feb 4) and now (Mar 2). The credential steps are already configured in these jobs — rotating the GH_TOKEN_BUILDS secret in the environment settings will resolve them.

Integration Tests / WASM Bridge Tests: The cargo run --bin setup-scratch-org step fails for the same expired PAT reason (Cargo resolves the whole workspace dependency graph, including busbar-capability). Same fix: rotate GH_TOKEN_BUILDS.

Data Cloud integration tests (tests/integration/data_cloud.rs): Added 7 real-org integration tests covering token exchange, metadata discovery, SQL query (sync + async), vector search reachability, and profile lookup. Tests fail loudly if Data Cloud is not provisioned (following the no-silent-skip policy).

Scratch org definition (config/project-scratch-def.json): Added DataCloud to the features list so new scratch orgs created for integration testing have Salesforce Data Cloud provisioned.