Skip to content

Commit f55ee13

Browse files
committed
integrations part cleared from rst syntax
1 parent 71b2c10 commit f55ee13

6 files changed

Lines changed: 67 additions & 120 deletions

File tree

mddocs/docs/en/integrations/airflow/index.md

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
(overview-setup-airflow)=
2-
3-
# Apache Airflow integration
1+
# Apache Airflow integration { #overview-setup-airflow }
42

53
Using [OpenLineage integration with Apache Airflow](https://openlineage.io/docs/integrations/airflow/).
64

@@ -23,12 +21,14 @@ Using [OpenLineage integration with Apache Airflow](https://openlineage.io/docs/
2321

2422
```console
2523
$ pip install "apache-airflow-providers-openlineage>=2.3.0" "openlineage-python[kafka]>=1.34.0" zstd
24+
...
2625
```
2726

2827
- For Airflow 2.1.x-2.6.x, use [OpenLineage integration for Airflow](https://openlineage.io/docs/integrations/airflow/) 1.19.0 or higher
2928

3029
```console
3130
$ pip install "openlineage-airflow>=1.34.0" "openlineage-python[kafka]>=1.34.0" zstd
31+
...
3232
```
3333

3434
## Setup
@@ -105,15 +105,12 @@ Browse frontend page [Jobs](http://localhost:3000/jobs) to see what information
105105

106106
### Job list page
107107

108-
```{image} ./job_list.png
109-
```
108+
![job list](job_list.png)
110109

111110
### DAG run details page
112111

113-
```{image} ./dag_run_details.png
114-
```
112+
![dag run details](dag_run_details.png)
115113

116114
### Task run details page
117115

118-
```{image} ./task_run_details.png
119-
```
116+
![task run details](task_run_details.png)

mddocs/docs/en/integrations/dbt/index.md

Lines changed: 13 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
(overview-setup-dbt)=
2-
3-
# dbt integration
1+
# dbt integration { #overview-setup-dbt }
42

53
Using [OpenLineage integration with dbt](https://openlineage.io/docs/integrations/dbt).
64

@@ -19,6 +17,7 @@ Using [OpenLineage integration with dbt](https://openlineage.io/docs/integration
1917

2018
```console
2119
$ pip install "openlineage-dbt>=1.34.0" "openlineage-python[kafka]>=1.34.0" zstd
20+
...
2221
```
2322

2423
## Setup
@@ -52,14 +51,18 @@ Replace `dbt` CLI commands:
5251

5352
```shell
5453
$ dbt run myproject
54+
...
5555
$ dbt test myproject
56+
...
5657
```
5758

5859
with `dbt-ol` CLI:
5960

6061
```shell
6162
$ dbt-ol run myproject
63+
...
6264
$ dbt-ol test myproject
65+
...
6366
```
6467

6568
Lineage will be send to Data.Rentgen automatically by OpenLineage integration.
@@ -70,35 +73,28 @@ Browse frontend page [Jobs](http://localhost:3000/jobs) to see what information
7073

7174
### Job list page
7275

73-
```{image} ./job_list.png
74-
```
76+
![job list](job_list.png)
7577

7678
### Job details page
7779

78-
```{image} ./job_details.png
79-
```
80+
![job details](job_details.png)
8081

8182
### Job-level lineage
8283

83-
```{image} ./job_lineage.png
84-
```
84+
![job lineage](job_lineage.png)
8585

8686
### Run details
8787

88-
```{image} ./run_details.png
89-
```
88+
![run details](run_details.png)
9089

9190
### Run lineage
9291

93-
```{image} ./run_lineage.png
94-
```
92+
![run lineage](run_lineage.png)
9593

9694
### Operation details
9795

98-
```{image} ./operation_details.png
99-
```
96+
![operation details](operation_details.png)
10097

10198
### Operation lineage
10299

103-
```{image} ./operation_lineage.png
104-
```
100+
![operation lineage](operation_lineage.png)

mddocs/docs/en/integrations/flink1/index.md

Lines changed: 12 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
(overview-setup-flink1)=
2-
3-
# Apache Flink 1.x integration
1+
# Apache Flink 1.x integration { #overview-setup-flink1 }
42

53
Using [OpenLineage integration with Apache Flink 1.x](https://openlineage.io/docs/integrations/flink/flink1).
64

@@ -22,17 +20,15 @@ Using [OpenLineage integration with Apache Flink 1.x](https://openlineage.io/doc
2220

2321
- Add dependencies [openlineage-flink](https://mvnrepository.com/artifact/io.openlineage/openlineage-flink) and [kafka-clients](https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients) to your Flink job:
2422

25-
```{code-block} groovy
26-
:caption: build.gradle
23+
```groovy title="build.gradle"
2724
2825
implementation "io.openlineage:openlineage-flink:1.34.0"
2926
implementation "org.apache.kafka:kafka-clients:3.9.0"
3027
```
3128

3229
- Register `OpenLineageFlinkJobListener` in the code of your Flink job:
3330

34-
```{code-block} java
35-
:caption: MyFlinkJob.java
31+
```java title="MyFlinkJob.java"
3632

3733
import io.openlineage.flink.OpenLineageFlinkJobListener;
3834

@@ -48,16 +44,14 @@ Using [OpenLineage integration with Apache Flink 1.x](https://openlineage.io/doc
4844

4945
- Modify Flink `config.yaml` to include:
5046

51-
```{code-block} yaml
52-
:caption: config.yaml
47+
```yaml title="config.yaml"
5348

5449
execution.attached: true # capture job stop events
5550
```
5651
5752
- Create `openlineage.yml` file with content like:
5853

59-
```{code-block} yaml
60-
:caption: openlineage.yml
54+
```yaml title="openlineage.yml"
6155
6256
job:
6357
namespace: http://some.host.name:18081 # set namespace to match Flink address
@@ -92,8 +86,7 @@ Using [OpenLineage integration with Apache Flink 1.x](https://openlineage.io/doc
9286

9387
At the end, this should look like this (see [Official documentation](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/resource-providers/standalone/docker/)):
9488

95-
```{code-block} yaml
96-
:caption: docker-compose.yml
89+
```yaml title="docker-compose.yml"
9790
9891
services:
9992
jobmanager:
@@ -129,30 +122,24 @@ Browse frontend pages [Jobs](http://localhost:3000/jobs) to see what information
129122

130123
### Job list page
131124

132-
```{image} ./job_list.png
133-
```
125+
![job list](job_list.png)
134126

135127
### Job details page
136128

137-
```{image} ./job_details.png
138-
```
129+
![job details](job_details.png)
139130

140131
### Run details page
141132

142-
```{image} ./run_details.png
143-
```
133+
![run details](run_details.png)
144134

145135
### Dataset level lineage
146136

147-
```{image} ./dataset_lineage.png
148-
```
137+
![dataset lineage](dataset_lineage.png)
149138

150139
### Job level lineage
151140

152-
```{image} ./job_lineage.png
153-
```
141+
![job lineage](job_lineage.png)
154142

155143
### Run level lineage
156144

157-
```{image} ./run_lineage.png
158-
```
145+
![run lineage](run_lineage.png)

mddocs/docs/en/integrations/flink2/index.md

Lines changed: 11 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
(overview-setup-flink2)=
2-
3-
# Apache Flink 2.x integration
1+
# Apache Flink 2.x integration { #overview-setup-flink2 }
42

53
Using [OpenLineage integration with Apache Flink 2.x](https://openlineage.io/docs/integrations/flink/flink2).
64

@@ -31,8 +29,7 @@ Using [OpenLineage integration with Apache Flink 2.x](https://openlineage.io/doc
3129

3230
- Configure Flink `JobManager` to load these dependencies using its own ClassLoader:
3331

34-
```{code-block} yaml
35-
:caption: config.yaml
32+
```yaml title="config.yaml"
3633

3734
classloader.parent-first-patterns.additional: ["io.openlineage.", "org.apache.kafka.","com.github.luben."]
3835
```
@@ -50,8 +47,7 @@ Using [OpenLineage integration with Apache Flink 2.x](https://openlineage.io/doc
5047
5148
- Add `OpenLineageJobStatusChangedListenerFactory` to Flink `config.yaml`:
5249

53-
```{code-block} yaml
54-
:caption: config.yaml
50+
```yaml title="config.yaml"
5551
5652
classloader.parent-first-patterns.additional: ["io.openlineage.", "org.apache.kafka.","com.github.luben."]
5753
execution.job-status-changed-listeners: io.openlineage.flink.listener.OpenLineageJobStatusChangedListenerFactory # capture job event
@@ -62,8 +58,7 @@ Using [OpenLineage integration with Apache Flink 2.x](https://openlineage.io/doc
6258

6359
- Create `openlineage.yml` file with content like:
6460

65-
```{code-block} yaml
66-
:caption: openlineage.yml
61+
```yaml title="openlineage.yml"
6762
6863
# Send RUNNING event every 1 hour.
6964
# Using default interval (1 minute) just floods Kafka with useless RUNNING events.
@@ -94,8 +89,7 @@ Using [OpenLineage integration with Apache Flink 2.x](https://openlineage.io/doc
9489

9590
At the end, this should look like this (see [Official documentation](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/resource-providers/standalone/docker/)):
9691

97-
```{code-block} yaml
98-
:caption: docker-compose.yml
92+
```yaml title="docker-compose.yml"
9993
10094
services:
10195
jobmanager:
@@ -132,30 +126,24 @@ Browse frontend pages [Jobs](http://localhost:3000/jobs) to see what information
132126

133127
### Job list page
134128

135-
```{image} ../flink1/job_list.png
136-
```
129+
![job list](../flink1/job_list.png)
137130

138131
### Job details page
139132

140-
```{image} ../flink1/job_details.png
141-
```
133+
![job details](../flink1/job_details.png)
142134

143135
### Run details page
144136

145-
```{image} ../flink1/run_details.png
146-
```
137+
![run details](../flink1/run_details.png)
147138

148139
### Dataset level lineage
149140

150-
```{image} ../flink1/dataset_lineage.png
151-
```
141+
![dataset lineage](../flink1/dataset_lineage.png)
152142

153143
### Job level lineage
154144

155-
```{image} ../flink1/job_lineage.png
156-
```
145+
![job lineage](../flink1/job_lineage.png)
157146

158147
### Run level lineage
159148

160-
```{image} ../flink1/run_lineage.png
161-
```
149+
![run lineage](../flink1/run_lineage.png)

mddocs/docs/en/integrations/hive/index.md

Lines changed: 15 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
(overview-setup-hive)=
2-
3-
# Apache Hive integration
1+
# Apache Hive integration { #overview-setup-hive }
42

53
Using [OpenLineage integration with Apache Hive](https://openlineage.io/docs/integrations/hive/).
64

@@ -131,54 +129,46 @@ Change `hive-site.xml` configuration file:
131129
Connect to you HiveServer2 instance JDBC interface, e.g. using `beeline` or DBeaver.
132130
After query was executed, integration will send lineage events to DataRentgen.
133131

134-
:::{note}
135-
By default, Job is created with name `{username}@{clientIp}`. You can override this name by executing this statement:
132+
!!! note
136133

137-
```sql
138-
SET hive.openlineage.job.name=my_session_name;
139-
```
140-
:::
134+
By default, Job is created with name `{username}@{clientIp}`. You can override this name by executing this statement:
135+
136+
```sql
137+
SET hive.openlineage.job.name=my_session_name;
138+
```
141139

142140
## See results
143141

144142
Browse frontend pages [Jobs](http://localhost:3000/jobs) to see what information was extracted by OpenLineage & DataRentgen.
145143

146144
### Job list page
147145

148-
```{image} ./job_list.png
149-
```
146+
![job list](job_list.png)
150147

151148
### Job details page
152149

153-
```{image} ./job_details.png
154-
```
150+
![job details](job_details.png)
155151

156152
### Run details page
157153

158-
```{image} ./run_details.png
159-
```
154+
![run details](run_details.png)
160155

161156
### Operation details page
162157

163-
```{image} ./operation_details.png
164-
```
158+
![operation details](operation_details.png)
165159

166160
### Dataset level lineage
167161

168-
```{image} ./dataset_lineage.png
169-
```
162+
![dataset lineage](dataset_lineage.png)
170163

171164
### Job level lineage
172165

173-
```{image} ./job_lineage.png
174-
```
166+
![job lineage](job_lineage.png)
175167

176168
### Run level lineage
177169

178-
```{image} ./run_lineage.png
179-
```
170+
![run lineage](run_lineage.png)
180171

181172
### Operation level lineage
182173

183-
```{image} ./operation_lineage.png
184-
```
174+
![operation lineage](operation_lineage.png)

0 commit comments

Comments
 (0)