databases tables load: 401 Invalid api key on large file uploads

[eddietejeda]

Summary

Users are seeing a 401 "Invalid api key" error when running hotdata databases tables load with large files (e.g. 10 GB). Two distinct failure paths were identified and reproduced via unit tests.

Reproduction

hotdata databases tables load test_10gb --database <db_id> --file /path/to/test_10gb.parquet

Error output:

Invalid api key
current database: <db_id>  use 'hotdata databases set' to change

Root Cause — Two Paths

Path 1: Token expiry during upload

ApiClient::new mints a JWT access token once at startup and stores it as a plain String. It is never refreshed mid-command. For large files, the upload to POST /files can exceed the JWT TTL (~5 min). When the streaming upload finally completes, the same now-expired bearer is reused for the subsequent POST /connections/{conn}/schemas/{schema}/tables/{table}/loads — which the server rejects with Invalid api key.

Relevant code in src/databases.rs:

let api = ApiClient::new(Some(workspace_id));      // token minted here
let db = resolve_database(&api, &database);
let upload_id = upload_parquet_file(&api, path);   // can take > 5 min for large files
// same api, same expired token:
let (status, resp_body) = api.post_raw(&path, &body);

Path 2: X-Database-Id sent on file upload

When the user has a current database set via hotdata databases set, ApiClient::new includes X-Database-Id on every request, including POST /files. If the server requires the bearer to be a database-scoped JWT when this header is present, the upload is rejected immediately — regardless of token age.

Relevant code in src/api.rs:

if let Some(ref db_id) = self.database_id {
    req = req.header("X-Database-Id", db_id);
}

/files is a general upload endpoint that does not belong to a specific database, so sending X-Database-Id there appears unintentional.

Reproduction Tests Added

Three new unit tests in src/databases.rs exercise both failure paths using mockito:

• upload_401_surfaces_server_message — server rejects the JWT on POST /files
• load_endpoint_401_after_successful_upload_same_token — upload succeeds but the subsequent load POST returns 401 with the same stale token (no refresh in between)
• database_id_header_sent_on_upload_when_db_set_in_client — verifies X-Database-Id is sent to /files when a current database is configured

Suggested Fixes

For Path 1: Re-create the ApiClient after the upload completes so a fresh JWT is used for the load request, or add token-refresh logic inside ApiClient itself.

For Path 2: Strip X-Database-Id from requests to POST /files. A dedicated upload method on ApiClient that omits that header would be the cleanest fix.