Skip to content

Latest commit

 

History

History
1103 lines (931 loc) · 23.7 KB

File metadata and controls

1103 lines (931 loc) · 23.7 KB

Pangolin API - Curl Examples

This document provides practical curl examples for all Pangolin API endpoints.

Note: For Iceberg REST API endpoints (/v1/*), refer to the Apache Iceberg REST Catalog specification.

Table of Contents


Authentication

No Auth Mode (Development)

# Set environment variable
export PANGOLIN_NO_AUTH=true

# No authentication headers needed
curl http://localhost:8080/api/v1/tenants

JWT Authentication

# Login to get JWT token
curl -X POST http://localhost:8080/api/v1/users/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "admin",
    "password": "password"
  }'

# Use token in subsequent requests
export TOKEN="eyJhbGciOiJIUzI1NiIs..."
curl http://localhost:8080/api/v1/tenants \
  -H "Authorization: Bearer $TOKEN"

Generate Long-Lived Token

# Generate token for automation/scripts
curl -X POST http://localhost:8080/api/v1/tokens \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "550e8400-e29b-41d4-a716-446655440000",
    "username": "automation-user",
    "expires_in_hours": 720
  }'

Response:

{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "expires_at": "2026-01-17T10:00:00Z"
}

Revoke Token

# Revoke your current token
curl -X POST http://localhost:8080/api/v1/auth/revoke \
  -H "Authorization: Bearer $TOKEN"

API Key Authentication (Service Users)

# Use API key header
export API_KEY="pgl_key_abc123..."
curl http://localhost:8080/api/v1/catalogs \
  -H "X-API-Key: $API_KEY"

Tenant Management

List All Tenants

curl http://localhost:8080/api/v1/tenants \
  -H "Authorization: Bearer $TOKEN"

Response:

[
  {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "name": "acme-corp",
    "properties": {},
    "created_at": "2025-12-14T10:00:00Z"
  }
]

Create Tenant

curl -X POST http://localhost:8080/api/v1/tenants \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "new-tenant",
    "properties": {
      "department": "analytics",
      "owner": "data-team"
    }
  }'

Get Tenant

curl http://localhost:8080/api/v1/tenants/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer $TOKEN"

Update Tenant

curl -X PUT http://localhost:8080/api/v1/tenants/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "updated-tenant-name",
    "properties": {
      "department": "data-engineering"
    }
  }'

Delete Tenant

curl -X DELETE http://localhost:8080/api/v1/tenants/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer $TOKEN"

Warehouse Management

List Warehouses

curl http://localhost:8080/api/v1/warehouses \
  -H "Authorization: Bearer $TOKEN"

Create S3 Warehouse (Static Credentials)

curl -X POST http://localhost:8080/api/v1/warehouses \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "s3-warehouse",
    "use_sts": false,
    "storage_config": {
      "type": "s3",
      "bucket": "my-data-bucket",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
      "endpoint": "http://localhost:9000"
    }
  }'

Create S3 Warehouse (AWS STS)

curl -X POST http://localhost:8080/api/v1/warehouses \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "s3-sts-warehouse",
    "use_sts": true,
    "storage_config": {
      "type": "s3",
      "bucket": "my-data-bucket",
      "region": "us-east-1",
      "role_arn": "arn:aws:iam::123456789:role/DataAccess",
      "external_id": "optional-external-id"
    }
  }'

Create Azure Warehouse (OAuth2)

curl -X POST http://localhost:8080/api/v1/warehouses \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "azure-warehouse",
    "use_sts": true,
    "storage_config": {
      "type": "azure",
      "tenant_id": "your-azure-tenant-id",
      "client_id": "your-azure-client-id",
      "client_secret": "your-azure-client-secret",
      "account_name": "mystorageaccount",
      "container": "data"
    }
  }'

Create GCP Warehouse (Service Account)

curl -X POST http://localhost:8080/api/v1/warehouses \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gcp-warehouse",
    "use_sts": true,
    "storage_config": {
      "type": "gcs",
      "service_account_key": "{\"type\":\"service_account\",\"project_id\":\"my-project\",...}",
      "project_id": "my-project",
      "bucket": "my-bucket"
    }
  }'

Get Warehouse

curl http://localhost:8080/api/v1/warehouses/s3-warehouse \
  -H "Authorization: Bearer $TOKEN"

Update Warehouse

curl -X PUT http://localhost:8080/api/v1/warehouses/s3-warehouse \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "use_sts": true,
    "storage_config": {
      "role_arn": "arn:aws:iam::123456789:role/NewRole"
    }
  }'

Delete Warehouse

curl -X DELETE http://localhost:8080/api/v1/warehouses/s3-warehouse \
  -H "Authorization: Bearer $TOKEN"

Catalog Management

List Catalogs

curl http://localhost:8080/api/v1/catalogs \
  -H "Authorization: Bearer $TOKEN"

Create Catalog

curl -X POST http://localhost:8080/api/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production",
    "warehouse_name": "s3-warehouse",
    "storage_location": "s3://my-bucket/warehouse",
    "properties": {
      "owner": "data-team"
    }
  }'

Get Catalog

curl http://localhost:8080/api/v1/catalogs/production \
  -H "Authorization: Bearer $TOKEN"

Update Catalog

curl -X PUT http://localhost:8080/api/v1/catalogs/production \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "warehouse_name": "new-warehouse",
    "properties": {
      "environment": "production"
    }
  }'

Delete Catalog

curl -X DELETE http://localhost:8080/api/v1/catalogs/production \
  -H "Authorization: Bearer $TOKEN"

User Management

List Users

curl http://localhost:8080/api/v1/users \
  -H "Authorization: Bearer $TOKEN"

Create User

curl -X POST http://localhost:8080/api/v1/users \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "john.doe",
    "email": "john@example.com",
    "password": "secure-password",
    "tenant_id": "550e8400-e29b-41d4-a716-446655440000"
  }'

Get User

curl http://localhost:8080/api/v1/users/USER_ID \
  -H "Authorization: Bearer $TOKEN"

Update User

curl -X PUT http://localhost:8080/api/v1/users/USER_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "newemail@example.com"
  }'

Delete User

curl -X DELETE http://localhost:8080/api/v1/users/USER_ID \
  -H "Authorization: Bearer $TOKEN"

Credential Vending

Get Table Credentials (AWS STS)

curl http://localhost:8080/v1/production/namespaces/db/tables/users/credentials \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "storage-credentials": [
    {
      "prefix": "s3://my-bucket/warehouse/",
      "config": {
        "access-key": "ASIA...",
        "secret-key": "...",
        "session-token": "...",
        "expiration": "2025-12-14T17:00:00Z",
        "s3.region": "us-east-1"
      }
    }
  ]
}

Get Table Credentials (Azure OAuth2)

curl http://localhost:8080/v1/production/namespaces/db/tables/users/credentials \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "storage-credentials": [
    {
      "prefix": "abfss://data@mystorageaccount.dfs.core.windows.net/",
      "config": {
        "adls.auth.type": "OAuth2",
        "adls.oauth2.token": "eyJ0eXAiOiJKV1QiLCJhbGci..."
      }
    }
  ]
}

Federated Catalogs

List Federated Catalogs

curl http://localhost:8080/api/v1/federated-catalogs \
  -H "Authorization: Bearer $TOKEN"

Create Federated Catalog

curl -X POST http://localhost:8080/api/v1/federated-catalogs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "partner-catalog",
    "config": {
      "properties": {
        "uri": "https://partner.example.com",
        "token": "partner-api-token"
      }
    }
  }'

Get Federated Catalog

curl http://localhost:8080/api/v1/federated-catalogs/partner-catalog \
  -H "Authorization: Bearer $TOKEN"

Test Federated Catalog Connection

curl -X POST http://localhost:8080/api/v1/federated-catalogs/partner-catalog/test \
  -H "Authorization: Bearer $TOKEN"

Delete Federated Catalog

curl -X DELETE http://localhost:8080/api/v1/federated-catalogs/partner-catalog \
  -H "Authorization: Bearer $TOKEN"

Service Users (API Keys)

Create Service User

curl -X POST http://localhost:8080/api/v1/service-users \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ci-cd-pipeline",
    "description": "Service user for CI/CD automation",
    "role": "TenantUser",
    "expires_in_days": 30
  }'

Response:

{
  "id": "...",
  "name": "ci-cd-pipeline",
  "api_key": "pgl_key_abc123...",
  "created_at": "2025-12-14T10:00:00Z",
  "expires_at": "2026-12-31T23:59:59Z"
}

Important: Save the api_key - it's only shown once!

List Service Users

curl http://localhost:8080/api/v1/service-users \
  -H "Authorization: Bearer $TOKEN"

Delete Service User

curl -X DELETE http://localhost:8080/api/v1/service-users/SERVICE_USER_ID \
  -H "Authorization: Bearer $TOKEN"

Roles & Permissions

List Roles

curl http://localhost:8080/api/v1/roles \
  -H "Authorization: Bearer $TOKEN"

Create Role

curl -X POST http://localhost:8080/api/v1/roles \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "DataEngineer",
    "description": "Can manage tables and schemas",
    "permissions": ["read", "write", "create"]
  }'

List Permissions

curl http://localhost:8080/api/v1/permissions \
  -H "Authorization: Bearer $TOKEN"

Grant Permission

curl -X POST http://localhost:8080/api/v1/permissions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "user-id": "USER_ID",
    "scope": {
      "type": "catalog",
      "catalog-id": "550e8400-e29b-41d4-a716-446655440000"
    },
    "actions": ["Read", "Write"]
  }'

Revoke Permission

curl -X DELETE http://localhost:8080/api/v1/permissions/PERMISSION_ID \
  -H "Authorization: Bearer $TOKEN"

OAuth

Initiate OAuth Flow (Google)

# Open in browser or redirect
curl http://localhost:8080/oauth/authorize/google

Initiate OAuth Flow (GitHub)

# Open in browser or redirect
curl http://localhost:8080/oauth/authorize/github

Note: OAuth flows require browser interaction. The callback endpoint /oauth/callback/{provider} is handled automatically.



Complete Workflow Example

1. Create Complete Environment

# Set base URL and token
export BASE_URL="http://localhost:8080"
export TOKEN="your-jwt-token"

# Create tenant
TENANT_RESPONSE=$(curl -s -X POST $BASE_URL/api/v1/tenants \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"acme-corp"}')

TENANT_ID=$(echo $TENANT_RESPONSE | jq -r '.id')

# Create warehouse
curl -X POST $BASE_URL/api/v1/warehouses \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name":"s3-warehouse",
    "use_sts":true,
    "storage_config":{
      "type":"s3",
      "bucket":"my-bucket",
      "region":"us-east-1",
      "role_arn":"arn:aws:iam::123:role/DataAccess"
    }
  }'

# Create catalog
curl -X POST $BASE_URL/api/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name":"production",
    "warehouse_name":"s3-warehouse"
  }'

# Create namespace (Iceberg REST API)
curl -X POST $BASE_URL/v1/production/namespaces \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace":["analytics"]
  }'

echo "✅ Environment created successfully!"

Error Handling

Common Error Responses

401 Unauthorized:

{
  "error": "Unauthorized",
  "message": "Invalid or missing authentication token"
}

404 Not Found:

{
  "error": "Not Found",
  "message": "Warehouse 'nonexistent' not found"
}

500 Internal Server Error:

{
  "error": "Internal Server Error",
  "message": "Failed to connect to database"
}

Tips & Best Practices

  1. Use jq for JSON parsing:

    curl http://localhost:8080/api/v1/catalogs | jq '.[] | .name'
  2. Save tokens in environment variables:

    export TOKEN=$(curl -s -X POST http://localhost:8080/api/v1/auth/login \
      -H "Content-Type: application/json" \
      -d '{"username":"admin","password":"password"}' | jq -r '.token')
  3. Use -v for debugging:

    curl -v http://localhost:8080/api/v1/tenants
  4. Pretty print JSON responses:

    curl http://localhost:8080/api/v1/catalogs | jq '.'

Related Documentation


Branch Operations

List Branches

curl http://localhost:8080/api/v1/branches \
  -H "Authorization: Bearer $TOKEN"

Create Branch

curl -X POST http://localhost:8080/api/v1/branches \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "dev",
    "catalog_name": "production",
    "source_branch": "main"
  }'

Get Branch

curl http://localhost:8080/api/v1/branches/dev \
  -H "Authorization: Bearer $TOKEN"

List Branch Commits

curl http://localhost:8080/api/v1/branches/dev/commits \
  -H "Authorization: Bearer $TOKEN"

Merge Branch

curl -X POST http://localhost:8080/api/v1/branches/merge \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source_branch": "dev",
    "target_branch": "main",
    "catalog_name": "production"
  }'

Tag Operations

List Tags

curl http://localhost:8080/api/v1/tags \
  -H "Authorization: Bearer $TOKEN"

Create Tag

curl -X POST http://localhost:8080/api/v1/tags \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "v1.0.0",
    "catalog_name": "production",
    "commit_id": "abc123..."
  }'

Delete Tag

curl -X DELETE http://localhost:8080/api/v1/tags/v1.0.0 \
  -H "Authorization: Bearer $TOKEN"

Merge Operations & Conflict Resolution

List Merge Operations

curl http://localhost:8080/api/v1/catalogs/production/merge-operations \
  -H "Authorization: Bearer $TOKEN"

Get Merge Operation

curl http://localhost:8080/api/v1/merge-operations/OPERATION_ID \
  -H "Authorization: Bearer $TOKEN"

List Merge Conflicts

curl http://localhost:8080/api/v1/merge-operations/OPERATION_ID/conflicts \
  -H "Authorization: Bearer $TOKEN"

Resolve Conflict

curl -X POST http://localhost:8080/api/v1/conflicts/CONFLICT_ID/resolve \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "strategy": "TakeSource"
  }'

Complete Merge

curl -X POST http://localhost:8080/api/v1/merge-operations/OPERATION_ID/complete \
  -H "Authorization: Bearer $TOKEN"

Abort Merge

curl -X POST http://localhost:8080/api/v1/merge-operations/OPERATION_ID/abort \
  -H "Authorization: Bearer $TOKEN"

Business Metadata & Access Requests

Add Business Metadata

curl -X POST http://localhost:8080/api/v1/assets/ASSET_ID/metadata \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "owner": "data-team",
    "classification": "PII",
    "tags": ["customer", "sensitive"]
  }'

Get Business Metadata

curl http://localhost:8080/api/v1/assets/ASSET_ID/metadata \
  -H "Authorization: Bearer $TOKEN"

Search Assets

curl http://localhost:8080/api/v1/assets/search?q=customer \
  -H "Authorization: Bearer $TOKEN"

Request Access to Asset

curl -X POST http://localhost:8080/api/v1/assets/ASSET_ID/request-access \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Need access for analytics project"
  }'

List Access Requests

curl http://localhost:8080/api/v1/access-requests \
  -H "Authorization: Bearer $TOKEN"

Update Access Request

curl -X PUT http://localhost:8080/api/v1/access-requests/REQUEST_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "approved"
  }'

Audit Logs

List Audit Events

curl http://localhost:8080/api/v1/audit \
  -H "Authorization: Bearer $TOKEN"

Response:

[
  {
    "id": "...",
    "timestamp": "2025-12-18T10:00:00Z",
    "user_id": "...",
    "action": "table.create",
    "resource": "production.analytics.users",
    "details": {}
  }
]

Token Management (Admin)

List User Tokens

# List all tokens for a specific user (admin only)
curl http://localhost:8080/api/v1/users/USER_ID/tokens \
  -H "Authorization: Bearer $TOKEN"

Response:

[
  {
    "id": "token-uuid-123",
    "user_id": "user-uuid-456",
    "created_at": "2025-12-19T10:00:00Z",
    "expires_at": "2026-01-19T10:00:00Z",
    "is_valid": true
  }
]

Delete Token

# Delete a specific token by ID (admin only)
curl -X DELETE http://localhost:8080/api/v1/tokens/TOKEN_ID \
  -H "Authorization: Bearer $TOKEN"

System Configuration (Admin)

Get System Settings

# Get all system configuration settings (admin only)
curl http://localhost:8080/api/v1/config/settings \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "allow_public_signup": false,
  "default_retention_days": 30,
  "default_warehouse_bucket": "my-default-bucket",
  "smtp_host": "smtp.example.com",
  "smtp_port": 587,
  "smtp_user": "noreply@example.com",
  "smtp_password": "***"
}

Update System Settings

# Update system configuration settings (admin only)
curl -X PUT http://localhost:8080/api/v1/config/settings \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "allow_public_signup": true,
    "default_retention_days": 90,
    "smtp_host": "smtp.newprovider.com"
  }'

Federated Catalog Operations

Sync Federated Catalog

# Trigger immediate metadata sync for federated catalog
curl -X POST http://localhost:8080/api/v1/federated-catalogs/partner-catalog/sync \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "status": "Sync triggered",
  "catalog_name": "partner-catalog",
  "triggered_at": "2025-12-19T10:00:00Z"
}

Get Federated Catalog Stats

# Get sync statistics and status for federated catalog
curl http://localhost:8080/api/v1/federated-catalogs/partner-catalog/stats \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "catalog_name": "partner-catalog",
  "last_synced_at": "2025-12-19T09:00:00Z",
  "sync_status": "success",
  "tables_synced": 42,
  "namespaces_synced": 5,
  "last_error": null,
  "next_scheduled_sync": "2025-12-19T12:00:00Z"
}

Data Explorer

Get Namespace Tree

# Get hierarchical namespace tree structure for a catalog
curl http://localhost:8080/api/v1/catalogs/production/namespaces/tree \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "root": [
    {
      "name": ["analytics"],
      "children": [
        {
          "name": ["analytics", "sales"],
          "children": []
        },
        {
          "name": ["analytics", "marketing"],
          "children": []
        }
      ]
    },
    {
      "name": ["staging"],
      "children": []
    }
  ]
}

Complete Examples

Token Management Workflow

# 1. List all tokens for a user (admin)
curl http://localhost:8080/api/v1/users/$USER_ID/tokens \
  -H "Authorization: Bearer $TOKEN"

# 2. Delete a specific token
curl -X DELETE http://localhost:8080/api/v1/tokens/$TOKEN_ID \
  -H "Authorization: Bearer $TOKEN"

System Configuration Workflow

# 1. Get current settings
SETTINGS=$(curl -s http://localhost:8080/api/v1/config/settings \
  -H "Authorization: Bearer $TOKEN")

echo $SETTINGS | jq '.'

# 2. Update specific settings
curl -X PUT http://localhost:8080/api/v1/config/settings \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "allow_public_signup": true,
    "default_retention_days": 60
  }'

Federated Catalog Monitoring

# 1. Check sync stats
curl http://localhost:8080/api/v1/federated-catalogs/partner-catalog/stats \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# 2. Trigger manual sync if needed
curl -X POST http://localhost:8080/api/v1/federated-catalogs/partner-catalog/sync \
  -H "Authorization: Bearer $TOKEN"

# 3. Wait and check stats again
sleep 30
curl http://localhost:8080/api/v1/federated-catalogs/partner-catalog/stats \
  -H "Authorization: Bearer $TOKEN" | jq '.last_synced_at'

Search & Optimization

Unified Search

Search across catalogs, namespaces, and tables.

curl "http://localhost:8080/api/v1/search?q=sales&tenant_id=550e8400-e29b-41d4-a716-446655440000" \
  -H "Authorization: Bearer $TOKEN"

Validate Table Name

Check if a name is valid and available.

curl -X POST http://localhost:8080/api/v1/validate/names \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resource_type": "catalog",
    "names": ["new_catalog_v2"]
  }'

### Bulk Asset Deletion
```bash
curl -X POST http://localhost:8080/api/v1/bulk/assets/delete \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "asset_ids": ["uuid-1", "uuid-2"]
  }'