Skip to content

Commit 438da3d

Browse files
yokoflyyokofly
committed
docs: refine Python External Stream version gates (#627)
1 parent 5b1485d commit 438da3d

2 files changed

Lines changed: 95 additions & 0 deletions

File tree

‎docs/sql-create-external-stream.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,96 @@ Please check the [Pulsar External Stream](/pulsar-source) for more details.
5555

5656
Please check the [NATS JetStream External Stream](/nats-jetstream-source) for more details.
5757

58+
## Python External Stream
59+
60+
Python External Stream lets you read from and write to arbitrary sources by embedding a Python body directly in the DDL. It is available in **Timeplus Enterprise 3.1.1+**.
61+
62+
The `init_function_name`, `deinit_function_name`, and `init_function_parameters` lifecycle hooks require **Timeplus Enterprise 3.2.1+**.
63+
64+
The `__timeplus_local_api_user` / `__timeplus_local_api_password` injected module globals require **Timeplus Enterprise 3.2.2+**.
65+
66+
```sql
67+
CREATE EXTERNAL STREAM [IF NOT EXISTS] stream_name (<col_name1> <col_type>)
68+
AS $$
69+
def read_fn():
70+
...
71+
72+
def write_fn(col1, ...):
73+
...
74+
75+
def init_fn(config): # optional, 3.2.1+
76+
...
77+
78+
def deinit_fn(): # optional, 3.2.1+
79+
...
80+
$$
81+
SETTINGS
82+
type = 'python', -- required
83+
read_function_name = '..', -- defaults to the stream name
84+
write_function_name = '..', -- defaults to read_function_name
85+
init_function_name = '..', -- 3.2.1+
86+
init_function_parameters = '..', -- 3.2.1+ (requires init_function_name)
87+
deinit_function_name = '..', -- 3.2.1+
88+
mode = 'auto' -- 'auto' (default), 'streaming', or 'batch'
89+
```
90+
91+
Settings:
92+
* **type**: must be `'python'`. Required.
93+
* **read_function_name**: name of the Python function used when the stream is read from. Defaults to the stream name.
94+
* **write_function_name**: name of the Python function used when the stream is written to (sink). Defaults to `read_function_name`.
95+
* **init_function_name** *(Timeplus Enterprise 3.2.1+)*: name of a Python function called once before read/write processing begins. Use it to open connections, warm caches, or stash state on `builtins` for the entry function to consume.
96+
* **init_function_parameters** *(Timeplus Enterprise 3.2.1+)*: a single string passed as the only argument to the init function. Any format works (JSON, `key=value`, or a plain string) — parsing is up to your Python code. Requires `init_function_name`; otherwise the stream fails to create with `Setting 'init_function_parameters' requires 'init_function_name' to be configured`.
97+
* **deinit_function_name** *(Timeplus Enterprise 3.2.1+)*: name of a Python function called once after read/write processing completes, for cleanup.
98+
* **mode**: Python table execution mode — `'auto'` (default), `'streaming'`, or `'batch'`.
99+
100+
:::info
101+
Attempting to use `init_function_name`, `deinit_function_name`, or `init_function_parameters` on versions earlier than 3.2.1 fails with:
102+
```
103+
Code: 115. DB::Exception: Unknown setting init_function_name: for storage ExternalStream.
104+
```
105+
Upgrade to 3.2.1 or later to use these hooks.
106+
:::
107+
108+
### Local API credentials *(Timeplus Enterprise 3.2.2+)*
109+
110+
When the local API user is enabled on the server, Timeplus injects two module-level globals into every Python External Stream module so your code can authenticate back to the same timeplusd over the native TCP protocol or the REST HTTP interface without hard-coding credentials:
111+
112+
* `__timeplus_local_api_user` — the ephemeral local API username.
113+
* `__timeplus_local_api_password` — the matching token. Treat this as a secret; do not log it.
114+
115+
Both globals are available as bare names inside the Python body — no `os.environ` lookup needed. They are regenerated on every server restart and never written to disk.
116+
117+
### Example: init / deinit hooks
118+
119+
```sql
120+
CREATE EXTERNAL STREAM py_cookie_counter
121+
(
122+
previous_cleanup_count int32,
123+
secret_flavor string
124+
)
125+
AS $$
126+
import builtins, json
127+
128+
def open_bakery(config):
129+
builtins._tp_cookie_secret_flavor = json.loads(config)["flavor"]
130+
131+
def close_bakery():
132+
if hasattr(builtins, "_tp_cookie_secret_flavor"):
133+
del builtins._tp_cookie_secret_flavor
134+
135+
def serve_cookie_report():
136+
return [(0, getattr(builtins, "_tp_cookie_secret_flavor", ""))]
137+
$$
138+
SETTINGS
139+
type = 'python',
140+
read_function_name = 'serve_cookie_report',
141+
init_function_name = 'open_bakery',
142+
init_function_parameters = '{"flavor":"double-chocolate"}',
143+
deinit_function_name = 'close_bakery';
144+
```
145+
146+
See [Python UDF](/py-udf) for more about the embedded Python runtime and library management.
147+
58148
## Timeplus External Stream
59149
```sql
60150
CREATE EXTERNAL STREAM [IF NOT EXISTS] stream_name (<col_name1> <col_type>)

‎tools/spellchecker/dic.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -886,6 +886,8 @@ DefaultTableEngine
886886
defaultValue
887887
defaultValueOfArgumentType
888888
defaultValueOfTypeName
889+
deinit
890+
deinit_function_name
889891
DelayedInserts
890892
DELETEs
891893
delim
@@ -1316,6 +1318,7 @@ globalNotIn
13161318
GlobalThread
13171319
GlobalThreadActive
13181320
globalVariable
1321+
globals
13191322
globbing
13201323
glushkovds
13211324
gofakeit
@@ -1526,6 +1529,8 @@ infty
15261529
Ingestions
15271530
ingressClassName
15281531
init
1532+
init_function_name
1533+
init_function_parameters
15291534
init_sql
15301535
initcap
15311536
initcapUTF

0 commit comments

Comments
 (0)