You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Logging is important for observability of the platform. Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform. Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well.
6
+
Logging is important for observability of the platform.
7
+
Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform.
8
+
Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well.
5
9
6
10
image::logging_overview.drawio.svg[]
7
11
8
12
== Motivation
9
13
10
-
**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts. For this, logs should share the same structure, and should be viewable in a central location. Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason.
14
+
**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts.
15
+
For this, logs should share the same structure, and should be viewable in a central location.
16
+
Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason.
11
17
12
-
**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform. The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs. This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.
18
+
**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform.
19
+
The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs.
20
+
This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.
13
21
14
-
**Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs. Logging for each product is configured in the ProductCluster resource. It is still supported to supply custom logging configuration files, these are then product specific.
22
+
**Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs.
23
+
Logging for each product is configured in the ProductCluster resource.
24
+
It is still supported to supply custom logging configuration files, these are then product specific.
15
25
16
26
== Architecture
17
27
18
-
Below you can see the overall architecture using ZooKeeper as an example. Stackable uses https://vector.dev/[Vector] for log aggregation and https://opensearch.org/[OpenSearch] as the sink to persist logs in.
28
+
Below you can see the overall architecture using ZooKeeper as an example.
29
+
Stackable uses https://vector.dev/[Vector] for log aggregation and any of the supported https://vector.dev/docs/reference/configuration/sinks/[sinks] can be used to persist the logs.
19
30
20
31
image::logging_architecture.drawio.svg[]
21
32
22
-
You configure your logging in the resource describing the cluster, seen in the top left in the diagram (see the <<configuration, configuration>> section below). The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration. The ConfigMap is then mounted into the containers. The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container]. All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator].
33
+
=== Log source
23
34
24
-
The aggregator is configured to use the OpenSearch sink. It sends all logs to the sink. If the sink is unavailable, it will buffer the log messages. It is also the single point where the sink is configured, so the sink is decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform.
35
+
You configure your logging settings in the resource describing the cluster (ZookeeperCluster in this example), seen in the top left in the diagram (see the <<configuration, configuration>> section below).
36
+
The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration.
37
+
The ConfigMap is then mounted into the containers.
38
+
The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container].
39
+
All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator].
40
+
41
+
=== Aggregator and sinks
42
+
43
+
The aggregator is configured to use one or multiple https://vector.dev/docs/reference/configuration/sinks/[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks.
44
+
If a sink is unavailable, it will buffer the log messages.
45
+
It is also the single point where the sinks are configured, so the sinks are decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform.
25
46
26
47
[#configuration]
27
-
== Identical configuration in each product
48
+
== Configuration
49
+
50
+
=== Identical configuration structure in each product
28
51
29
-
Logging configuration usually requires configuration what to log, where to log it and in which format. Usually you want to persist logs but also view them on the fly in a console, and possibly both of these in different formats. Stackable provides configuration for this out of the box. You only need to enable log aggregation and configure _what_ you want to log. As part of the logging configuration, individual modules can be configured with different log levels:
52
+
Logging configuration usually requires configuration what to log, where to log it and in which format.
53
+
Usually you want to persist logs but also view them on the fly in a console, and possibly both of these in different formats.
54
+
Stackable provides configuration for this out of the box.
55
+
You only need to enable log aggregation and configure _what_ you want to log.
56
+
As part of the logging configuration, individual modules can be configured with different log levels:
<5> Console and file appenders can have different log level thresholds.
58
85
<6> Setting log levels for individual modules is also possible.
59
86
60
-
**Log levels per module** - In the `loggers` section the log levels for each module are configured. Note the `ROOT` module. This is a special module and refers to the default log level for the container. The log levels are the commonly used ones (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) as well as the NONE level to turn of logging. The default log level is `INFO`.
87
+
**Log levels per module** - In the `loggers` section the log levels for each module are configured.
88
+
Note the `ROOT` module.
89
+
This is a special module and refers to the default log level for the container.
90
+
The log levels are the commonly used ones (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) as well as the NONE level to turn of logging.
91
+
The default log level is `INFO`.
61
92
62
-
**Log levels for console and file** - The configuration supports setting another level threshold for the console and the file output from which the aggregation is done. A typical use case is that the log level for the console is set to a more detailed level to allow for detailed inspection of a container, but less information in the aggregated log store.
93
+
**Log levels for console and file** - The configuration supports setting another level threshold for the console and the file output from which the aggregation is done.
94
+
A typical use case is that the log level for the console is set to a more detailed level to allow for detailed inspection of a container, but less information in the aggregated log store.
63
95
64
-
**Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them. Also the log levels per module are container specific.
96
+
**Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them.
97
+
Also the log levels per module are container specific.
65
98
66
99
Following the Stackable xref::roles-and-role-groups.adoc[roles and role groups] concept, this configuration fragment can be set either at role or role group level.
67
100
101
+
=== Configuring the Aggregator
102
+
103
+
Follow the https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator[installation instructions] for the aggregator.
104
+
Configure a https://vector.dev/docs/reference/configuration/sources/vector/[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs.
105
+
68
106
=== Configuring the aggregator location
69
107
70
-
Every product also has a `vectorAggregatorConfigMapName` property in its spec that needs to be set to the name of the ConfigMap that contains the address of the aggregator. The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if you're running the aggregator behind a service named `vector-aggregator`.
108
+
Every product also has a `vectorAggregatorConfigMapName` property in its spec that needs to be set to the name of the ConfigMap that contains the address of the aggregator.
109
+
The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if you're running the aggregator behind a service named `vector-aggregator`.
71
110
72
111
== Custom overrides
73
112
@@ -85,6 +124,7 @@ logging:
85
124
<1> The vector logging agent is not deployed
86
125
<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`
87
126
88
-
== Further Reading
127
+
== Further reading
89
128
90
-
To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo].
129
+
To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo] or follow the xref:tutorials:logging-vector-aggregator.adoc[logging tutorial].
130
+
The Vector documentation contains more information about the https://vector.dev/docs/setup/deployment/topologies/#centralized[deployment topology] and https://vector.dev/docs/reference/configuration/sinks/[sinks].
This tutorial teaches you how to deploy a Vector aggregator together with a product - in this case ZooKeeper - and how to configure both of them so the logs are sent from the product to the aggregator.
6
+
Logging on the Stackable Data Platform is always configured in the same way, so you can use this knowledge to configure logging in any product that you want to deploy.
7
+
8
+
Prerequisites:
9
+
10
+
* a k8s cluster available, or https://kind.sigs.k8s.io/[kind] installed
This is a minimal working configuration. The source should be defined in this way, but you can configure different sinks, depending on your needs.
40
+
You can find an https://vector.dev/docs/reference/configuration/sinks/[overview] of all sinks in the Vector documentation, specifically the https://vector.dev/docs/reference/configuration/sinks/elasticsearch/[Elasticsearch] sink might be useful, it also works when configured with OpenSearch.
41
+
42
+
To make the Vector aggregator discoverable to ZooKeeper, deploy a xref:concepts:service_discovery.adoc[discovery ConfigMap] called `vector-aggregator-discovery`.
43
+
Create a file called `vector-aggregator-discovery.yaml`:
2023-11-06 10:30:40,223 [myid:1] - INFO [main:o.a.z.Environment@98] - Server environment:zookeeper.version=3.8.0-5a02a05eddb59aee6ac762f7ea82e92a68eb9c0f, built on 2022-02-25 08:49 UTC
110
+
111
+
Congratulations, this concludes the tutorial!
112
+
113
+
== What's next?
114
+
115
+
Look into different sink configurations which are more suited to production use in the https://vector.dev/docs/reference/configuration/sinks/[sinks overview documetation] or learn more about how logging works on the platform in the xref:concepts:logging.adoc[concepts documentation].
0 commit comments