build(kernel): add optional [kernel] extra for use_kernel=True

[vikrantpuppala]

What

databricks-sql-kernel is now published to PyPI (and resolvable via the db-pypi JFrog proxy CI uses), so the Rust kernel backend (use_kernel=True) can ship as an optional dependency instead of a local-dev-only build. This adds the [kernel] extra:

pip install "databricks-sql-connector[kernel]"

sql.connect(server_hostname=host, http_path=http_path, access_token=token, use_kernel=True)

Changes

• pyproject.toml
  • Declare databricks-sql-kernel = {version = "^0.1.0", optional = true, python = ">=3.10"}. Gated to Python ≥ 3.10 because the wheel is cp310-abi3 (Requires-Python: >=3.10). The connector's own floor (3.8) is unchanged; on 3.8/3.9 the extra resolves to nothing.
  • Add kernel = ["databricks-sql-kernel", "pyarrow"]. pyarrow is listed explicitly: the kernel result path (backend/kernel/result_set.py) imports it unconditionally to wrap the Arrow batches the kernel returns. It's already pulled transitively via the kernel wheel's pyarrow>=23.0.1,<24, but naming it documents the connector-side requirement and lets pip co-resolve both constraints at install time.
• backend/kernel/_errors.py — the use_kernel=True ImportError now points at pip install "databricks-sql-connector[kernel]" and notes Python ≥ 3.10 (was the obsolete "not yet published, build locally" hint).
• README.md — document the extra, usage, and the Python ≥ 3.10 / pyarrow notes.

How it works

use_kernel=True imports databricks_sql_kernel at backend load. With the extra → import succeeds, kernel path is live. Without it → friendly ImportError telling the user to install [kernel]. The kernel stays a soft dependency: default installs don't pull the Rust wheel.

Verification (all green, against the published wheel)

Installed databricks-sql-connector[kernel] from this source with the kernel resolved purely from the proxy (no local wheel):

• Installs databricks-sql-kernel 0.1.2 + pyarrow 23.0.1; use_kernel=True runs a live query end-to-end (backend KernelDatabricksClient, results 42 and 99 in two test venvs).
• Plain pip install databricks-sql-connector → use_kernel=True raises the friendly ImportError (mentions [kernel] + Python ≥ 3.10).
• scripts/dependency_manager.py (CI's unit-test dep generator) correctly excludes the optional kernel dep from the generated requirements — verified default and --include-optional runs don't emit it, so no install attempt on the 3.8/3.9 test matrix.

poetry.lock

Not regenerated in this commit: my environment's poetry can only reach public PyPI (DNS-blocked here), so I can't produce a correct lock locally. CI's setup-poetry action runs poetry lock itself against the db-pypi JFrog source on every run (and poetry install --all-extras), so the lockfile is regenerated in CI with the kernel entry. If a committed-and-consistent poetry.lock is required by repo policy, please run poetry lock on a network-capable machine and add it to this branch.

Follow-up (optional, not blocking)

Consider a CI job that installs the [kernel] extra and smoke-tests import databricks_sql_kernel (or extend kernel-e2e.yml, which currently source-builds at KERNEL_REV) so the published-wheel path is exercised in CI. Note: if such a job uses dependency_manager.py --include-optional, the script's dict-constraint branch ignores the python marker — handle the ≥3.10 gate there to avoid a 3.8/3.9 failure.