-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathindex.html
More file actions
406 lines (332 loc) · 23.1 KB
/
Copy pathindex.html
File metadata and controls
406 lines (332 loc) · 23.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
<!DOCTYPE html>
<html lang="en">
<head>
<meta name="generator" content="Hugo 0.82.0" />
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="alternate" href="/index.xml" type="application/rss+xml" title="Flokkr.">
<link rel="icon" href="https://flokkr.github.io/favicon.ico">
<title>Flokkr.</title>
<link rel="stylesheet" href="https://flokkr.github.io/css/highlight/default.css">
<link rel="stylesheet" href="https://flokkr.github.io/css/bootstrap.min.css">
<link rel="stylesheet" href="https://flokkr.github.io/css/bootstrap-theme.min.css">
<link rel="stylesheet" href="https://flokkr.github.io/css/theme.css">
<link rel="stylesheet" href="https://flokkr.github.io/css/bootie-docs.css">
<link rel="stylesheet" href="https://flokkr.github.io/css/site.css">
</head>
<body role="document">
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="https://flokkr.github.io/">Flokkr.</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav">
<li class="active"><a href="https://flokkr.github.io/">Home</a></li>
</ul>
</div>
</div>
</nav>
<div class="container">
<main role="main">
<div class="row">
<div class="jumbotron">
<h1><span style="color:rgb(22, 21, 47)">(Apache) Big data <span style="color:red">❤</span> Kubernetes</span></h1>
<p></p>
<p>Flokkr is a containerization project for<b>Apache Flink, Kafka, Ozone, Spark</b>
and other big data project to run them in <b>Kubernetes</b> with a GitOps based approach.</p>
<p>It provides:</br></br>
1. ready to use containers for various Bigdata project</br>
2. framework to generate Kubernetes resources with any type of customization</br>
3. various configuration set for different use cases (Kerberos, HA, TDE, etc.)<br/>
4. helper tools to use the projects in cloud-native environments
</p>
<p>Please note that this project is <b>NOT</b> an official <a href="https://apache.org">Apache</a> project</p>
</div>
</div>
<h2>Containers</h2>
<table class="table">
<tr>
<td>name</td>
<td>version (latest)</td>
<td>Example K8s deployments</td>
<td>Kubernetes test tatus</td>
</tr>
<tr>
<td>
flink
</td>
<td>
[1.12.20 1.11.3]
</td>
<td>
<a href="https://github.com/flokkr/docker-flink/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-flink/workflows/test/badge.svg"/>
</td>
</tr>
<tr>
<td>
hadoop
</td>
<td>
[3.3.0 3.2.2 3.1.4 2.7.3 2.7.0]
</td>
<td>
<a href="https://github.com/flokkr/docker-hadoop/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-hadoop/workflows/test/badge.svg"/>
</td>
</tr>
<tr>
<td>
hbase
</td>
<td>
[2.4.2 2.3.5]
</td>
<td>
<a href="https://github.com/flokkr/docker-hbase/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-hbase/workflows/test/badge.svg"/>
</td>
</tr>
<tr>
<td>
kafka
</td>
<td>
[2.8.0 2.7.0]
</td>
<td>
<a href="https://github.com/flokkr/docker-kafka/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-kafka/workflows/test/badge.svg"/>
</td>
</tr>
<tr>
<td>
ozone
</td>
<td>
[1.1.0]
</td>
<td>
<a href="https://github.com/flokkr/docker-ozone/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-ozone/workflows/test/badge.svg"/>
</td>
</tr>
<tr>
<td>
spark
</td>
<td>
[3.0.0 2.4.5 2.4.6]
</td>
<td>
<a href="https://github.com/flokkr/docker-spark/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-spark/workflows/test/badge.svg"/>
</td>
</tr>
<tr>
<td>
zookeeper
</td>
<td>
[3.6.2]
</td>
<td>
<a href="https://github.com/flokkr/docker-zookeeper/tree/master/examples">Kubernetes example</a>
</td>
<td>
<img src="https://github.com/flokkr/docker-zookeeper/workflows/test/badge.svg"/>
</td>
</tr>
</table>
<p>Note: there are some other containers on dockerhub, but they are either experimental or retired. The latest version from this table are tested with CI.x
<h2>Getting started</h2>
<div>
<p>Flokkr provides tools and building elements to create your own cluster. It’s based on <a href="https://github.com/elek/flekszible">Flekszible</a> which is a highly flexible Kubernetes resource generator.</p>
<p>To start, install Flekszible and register Flokkr sources:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">>flekszible source search</span>
<span style="color:#f92672">Available flekszible repositories</span>:
<span style="color:#ae81ff">+------------------------------------+-----------------------------------------------------------------------------+</span>
<span style="color:#ae81ff">| name | description |</span>
<span style="color:#ae81ff">+------------------------------------+-----------------------------------------------------------------------------+</span>
<span style="color:#ae81ff">| github.com/flokkr/k8s | Flekszible based kubernetes manfiest templates for Apache bigdata projects. |</span>
<span style="color:#ae81ff">| github.com/flokkr/infra-flekszible | Flekszible based Kubernetes recipes for logging/monitoring/ci |</span>
<span style="color:#ae81ff">| github.com/elek/ozone-flekszible | Apache Hadoop Ozone deployment definitions with flekszible |</span>
<span style="color:#ae81ff">+------------------------------------+-----------------------------------------------------------------------------+</span>
<span style="color:#ae81ff">Add flekszible topic to your repository to show your repository here.</span>
</code></pre></div><p>Register main Flokkr repositories. The repositories contain the Kubernetes resource definitions (together with optional transformations) for the specific projects.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">> flekszible source add github.com/flokkr/k8s</span>
<span style="color:#ae81ff">> flekszible source add github.com/elek/> ozone-flekszible</span>
<span style="color:#ae81ff">> flekszible source add github.com/flokkr/infra-flekszible</span>
</code></pre></div><p>Now you can list the available components:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">> flekszible app search</span>
<span style="color:#f92672">INFO[0000] Input dir: /tmp, output dir</span>: <span style="color:#ae81ff">/tmp</span>
<span style="color:#ae81ff">INFO[0000] Reading resources from /tmp/resources</span>
<span style="color:#ae81ff">+--------------------------+------------------------------------------------------------------+</span>
<span style="color:#ae81ff">| path | description |</span>
<span style="color:#ae81ff">+--------------------------+------------------------------------------------------------------+</span>
<span style="color:#ae81ff">| flink | Apache Flink |</span>
<span style="color:#ae81ff">| grafana | Grafana dashboard server |</span>
<span style="color:#ae81ff">| hdfs | Apache Hadoop HDFS base setup |</span>
<span style="color:#ae81ff">| hdfs-ha | Apache Hadoop HDFS, HA setup |</span>
<span style="color:#ae81ff">| jaeger | Jaeger tracing server |</span>
<span style="color:#ae81ff">| kafka | Apache Kafka |</span>
<span style="color:#ae81ff">| kafka-demo | Simple console producer / consumer for Kafka |</span>
<span style="color:#ae81ff">| krb5-dev | Unsecure MIT kerberos server for DEVELOPMENT only |</span>
<span style="color:#ae81ff">| krb5-dev/getkeystore | Sidecar definition to import java trust/keystore from vault |</span>
<span style="color:#ae81ff">| monitor | K8s level monitoring |</span>
<span style="color:#ae81ff">| pv-test | Nginx example deployment with persistent volume claim. |</span>
<span style="color:#ae81ff">| zookeeper | Scalable Apache Zookeeper setup |</span>
<span style="color:#ae81ff">| ozone | Apache Hadoop Ozone |</span>
<span style="color:#ae81ff">| ozone/freon | Load test tool for Apache Hadoop Ozone |</span>
<span style="color:#ae81ff">| anonymous-proxy | permission to access proxy url by anonymous users |</span>
<span style="color:#ae81ff">| cadvisor | CAdvisor node level container metrics |</span>
<span style="color:#ae81ff">| grafana | Grafana dashboard server |</span>
<span style="color:#ae81ff">| jaeger | Jaeger tracing server |</span>
<span style="color:#ae81ff">| kube-dashboard/fulladmin | Full admin privilege for kube-dashboard |</span>
<span style="color:#ae81ff">| kube-state-metrics | Kubernetes metrics exporter |</span>
<span style="color:#ae81ff">| kubernetes-monitoring | prometheus instance to be configured for k8s cluster monitoring. |</span>
<span style="color:#ae81ff">| loki | loki based log collector |</span>
<span style="color:#ae81ff">| minio | Simple MINIO S3 server |</span>
<span style="color:#ae81ff">| node-exporter | Prometheus Node Exporter |</span>
<span style="color:#ae81ff">| prometheus | Prometheus monitoring |</span>
<span style="color:#ae81ff">| sleep | Forever sleeping test containers |</span>
<span style="color:#ae81ff">+--------------------------+------------------------------------------------------------------+</span>
</code></pre></div><p>And add everything what you need:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">> flekszible app add zookeeper</span>
<span style="color:#ae81ff">> flekszible app add flink</span>
<span style="color:#ae81ff">> flekszible app add kafka</span>
</code></pre></div><p>Finally you can generate the Kubernetes resources files:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">> flekszible generate</span>
</code></pre></div><p>It generates all the required yaml files.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">> ls -lah</span>
<span style="color:#ae81ff">.rwxr-xr-x 194 elek 15 Dec 9:00 Flekszible</span>
<span style="color:#ae81ff">.rw-r-xr-x 177 elek 15 Dec 9:02 flink-config-configmap.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 230 elek 15 Dec 9:02 flink-jobmanager-service.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 627 elek 15 Dec 9:02 flink-jobmanager-statefulset.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 233 elek 15 Dec 9:02 flink-taskmanager-service.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 634 elek 15 Dec 9:02 flink-taskmanager-statefulset.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 174 elek 15 Dec 9:02 kafka-broker-service.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 723 elek 15 Dec 9:02 kafka-broker-statefulset.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 331 elek 15 Dec 9:02 kafka-config-configmap.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 506 elek 15 Dec 9:02 zookeeper-config-configmap.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 177 elek 15 Dec 9:02 zookeeper-service.yaml</span>
<span style="color:#ae81ff">.rw-r-xr-x 766 elek 15 Dec 9:02 zookeeper-statefulset.yaml</span>
</code></pre></div><p>Finally you can install it:</p>
<pre><code>kubectl apply -f .
</code></pre><p>For more customization check the <a href="https://flekszible.netlify.com/">documentation of Flekszible</a></p>
</div>
<h2>Next steps</h2>
<div>
<p>The next steps after the first cluster is the customization. It can be done by adding any kind of custom transformation or reuse ready-to-use transformations.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#ae81ff">> flekszible transformation search</span>
<span style="color:#ae81ff">+---------------------+--------------------------------------------------------------------------------------------+</span>
<span style="color:#ae81ff">| name | description |</span>
<span style="color:#ae81ff">+---------------------+--------------------------------------------------------------------------------------------+</span>
<span style="color:#ae81ff">| Namespace | Use explicit namespace |</span>
<span style="color:#ae81ff">| Pipe | Transform content with external shell command. |</span>
<span style="color:#ae81ff">| Remove | Remove yaml fragment from an existing k8s resources |</span>
<span style="color:#ae81ff">| ozone/emptydir | Add empty dir based ephemeral persistence |</span>
<span style="color:#ae81ff">| ozone/onenode | remove scheduling rules to make it possible to run multiple datanode on the same k8s node. |</span>
<span style="color:#ae81ff">| ozone/persistence | Add real PVC based persistence |</span>
<span style="color:#ae81ff">| ozone/profiler | Enable profiler endpoint. |</span>
<span style="color:#ae81ff">| Add | Extends yaml fragment to an existing k8s resources |</span>
<span style="color:#ae81ff">| Image | Replaces the docker image definition |</span>
<span style="color:#ae81ff">| Prefix | Add same prefix to all the k8s names |</span>
<span style="color:#ae81ff">| ozone/devtracing | Enable jaeger tracing for ALL the requests (100% sampling) |</span>
<span style="color:#ae81ff">| ozone/grafana | Enable grafana for ozone dashboards |</span>
<span style="color:#ae81ff">| ozone/memdisk | Use memdisks for empty dirs |</span>
<span style="color:#ae81ff">| ozone/ozonefs | copy ozonefs jar file to a temporary emptydir volume |</span>
<span style="color:#ae81ff">| Change | Replace existing value literal in the yaml struct |</span>
<span style="color:#ae81ff">| ConfigHash | Add labels to the k8s resources with the hash of the used configmaps |</span>
<span style="color:#ae81ff">| DaemonToStatefulSet | Converts daemonset to statefulset |</span>
<span style="color:#ae81ff">| K8sWriter | Internal transformation to print out k8s resources as yaml |</span>
<span style="color:#ae81ff">| PublishService | Creates additional service for internal services |</span>
<span style="color:#ae81ff">| Replace | Replace a yaml subtree with an other one. |</span>
<span style="color:#ae81ff">| ozone/tracing | Enable jaeger tracing |</span>
<span style="color:#ae81ff">| PublishStatefulSet | Creates additional NodeType service for StatefulSet internal services |</span>
<span style="color:#ae81ff">| zookeeper/scale | Set the number of the zookeeper replicas. |</span>
<span style="color:#ae81ff">| ozone/prometheus | Enable prometheus monitoring in Ozone |</span>
<span style="color:#ae81ff">+---------------------+--------------------------------------------------------------------------------------------+</span>
</code></pre></div><p>The prefixed transformations (like <code>zookeeper/scale</code>) are usually combined, pre-defined transformation, you can apply it with command line (<code>flekszible transformation add ozone/emptydir</code>) or with editing the <code>Flekszible</code> descriptor.</p>
<p>(Note: all the previous command line just modified this descriptor)</p>
<p>In this example we imported Ozone app, with a transformation which adds emptyDir based persistence:</p>
<p>Content of <code>Flekszible</code>:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#f92672">import</span>:
- <span style="color:#f92672">path</span>: <span style="color:#ae81ff">ozone</span>
<span style="color:#f92672">transformations</span>:
- <span style="color:#f92672">type</span>: <span style="color:#ae81ff">ozone/emptydir</span>
</code></pre></div><p>In the next example we imported kafka, kafka-demo, flink, but the flink resources are transformed to add a custom <code>imagePullPolicy</code>:</p>
<p>Content of <code>Flekszible</code>:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#f92672">import</span>:
- <span style="color:#f92672">path</span>: <span style="color:#ae81ff">kafka</span>
- <span style="color:#f92672">path</span>: <span style="color:#ae81ff">kafka-demo</span>
- <span style="color:#f92672">path</span>: <span style="color:#ae81ff">flink</span>
<span style="color:#f92672">transformations</span>:
- <span style="color:#f92672">type</span>: <span style="color:#ae81ff">add</span>
<span style="color:#f92672">path</span>:
- <span style="color:#ae81ff">spec</span>
- <span style="color:#ae81ff">template</span>
- <span style="color:#ae81ff">spec</span>
- <span style="color:#ae81ff">containers</span>
- <span style="color:#e6db74">".*"</span>
<span style="color:#f92672">value</span>:
<span style="color:#f92672">imagePullPolicy</span>: <span style="color:#ae81ff">IfNotPresent</span>
</code></pre></div>
</div>
<h2>News</h2>
<div>
<ul class="doc-list-category">
<li><a href="/post/spring2020/">Project update</a></li>
<li><a href="/post/cleanup/">The big cleanup</a></li>
<li><a href="/post/byteman/">Byteman: X-ray for Hadoop</a></li>
<li><a href="/post/katacoda/">Katacoda</a></li>
<li><a href="/post/mrrobot/">Integration tests with robot framework</a></li>
<li><a href="/post/hadoop-hdfs-ha-docker-compose/">Start Hadoop Hdfs HA cluster with docker-compose</a></li>
</ul>
</div>
<h2>Presentations</h2>
<div>
<ul>
<li><a href="https://flokkr.github.io/presentations/201711-docker-meetup.html">4 ways to Dockerize Apache bigdata project</a> (Docker meetup, Budapest)</li>
<li><a href="https://flokkr/github.io/presentations/bb/index.html">From docker to Kubernetes: Running Hadoop in a cloud-natie way</a> (Berlin Buzzwords 2018)</li>
<li>Apache Hadoop Ozone in the cloud-native word: Use Hadoop as a Kubernetes persistent storage provider (Apache Roadshow EU, 2018)</li>
</ul>
</div>
</main>
<hr />
<div class="row">
<div class="col-sm-8">
<p class="doc-footer-em"><a href="#" onclick="resetSidebarPos()">Back to TOP</a></p>
</div>
</div>
</div>
<footer class="doc-footer">
<p>Powered by <strong><a href="https://github.com/progrhyme/hugo-theme-bootie-docs">Bootie Docs</a></strong> - theme for <a href="http://gohugo.io/">Hugo</a> by <a href="https://github.com/progrhyme/">progrhyme</a>.</p>
</footer>
<script src="https://flokkr.github.io/js/jquery.min.js"></script>
<script src="https://flokkr.github.io/js/bootstrap.min.js"></script>
<script src="https://flokkr.github.io/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<script src="https://flokkr.github.io/js/ie10-viewport-bug-workaround.js"></script>
<script src="https://flokkr.github.io/js/bootie-docs.js"></script>
</body>
</html>