Skip to content

Port correct streaming SQL TVF patterns into restructured SDP skill #434

@calreynolds

Description

@calreynolds

Context

PR #320 by @sienkiewiczPat fixed several real issues in the SDP streaming SQL documentation, but the skill was restructured in #368 and the target files no longer exist. Closing #320 but porting the fixes here.

Supersedes #304 (which reported the same broken TVFs).

What needs to be added

The following corrections from #320 should be applied to the current databricks-skills/databricks-spark-declarative-pipelines/references/sql/ directory:

  1. Kafka ingestion: Use read_kafka() TVF (not read_stream()), with correct parameter names (bootstrapServers, backtick-quoted dot-options like `kafka.security.protocol`), and serviceCredential auth option
  2. Kinesis ingestion: Use read_kinesis() TVF with correct parameter names (streamName, initialPosition) and all three auth patterns (explicit credentials, IAM role, environment variables)
  3. Event Hub ingestion: Use read_kafka() with SASL/SSL JAAS config (no read_eventhub TVF exists; Event Hub uses Kafka-compatible endpoint)
  4. Secret syntax: Use secret('scope', 'key') function — the {{secrets/scope/key}} template syntax is not valid in SDP SQL
  5. Advanced config: Add pipelines.reset.allowed config key

Reference

See PR #320 diff for the exact corrections: https://github.com/databricks-solutions/ai-dev-kit/pull/320/files

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions