Skip to content

Commit 277ebc8

Browse files
mjfrycmjfryc
committed
Issue #26: Structured metadata
* Structured metadata support. * v1.1.0
1 parent 3aa1cb5 commit 277ebc8

17 files changed

Lines changed: 462 additions & 59 deletions

‎README.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
![Branches](.github/badges/branches.svg)
1616

1717
Tiny [Grafana Loki](https://grafana.com/oss/loki/) client (log sender) written in pure Java 1.8 without any external
18-
dependencies.
18+
dependencies. One of Grafana Loki third-party clients mentioned in [documentation](https://grafana.com/docs/loki/v3.4.x/send-data/).
1919

2020
* Implements JSON variant of [Loki API](https://grafana.com/docs/loki/latest/api/#post-lokiapiv1push)
2121
* Works with **Android** and **Java SE**
@@ -122,6 +122,14 @@ Click the [Packages section](https://github.com/mjfryc?tab=packages&repo_name=mj
122122
implementation files(project.rootDir.absolutePath + '/libs/mjaron-tinyloki-java-1.0.0.jar')
123123
```
124124

125+
## Features description
126+
127+
### Structured metadata
128+
129+
The [structured metadata](https://grafana.com/docs/loki/v3.4.x/get-started/labels/structured-metadata/) has been enabled
130+
by default in [Grafana Loki 3.0.0](https://grafana.com/docs/loki/v3.4.x/setup/upgrade/#loki-300).
131+
To put structured metadata to the log line, add it as the last argument of logging method.
132+
125133
## API design (outdated)
126134

127135
```mermaid

‎build.gradle‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ plugins {
66
}
77

88
group 'io.github.mjfryc'
9-
version '1.0.1'
9+
version '1.1.0'
1010

1111
repositories {
1212
mavenCentral()

tinyloki-server/README.md renamed to integration-test-server/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,15 @@ Following services are available:
1414

1515
* Check if Loki server is ready: <http://localhost:3100/ready>
1616
* Open Grafana dashboard: <http://localhost:3000>
17+
18+
## This and that
19+
20+
### Running shell of Loki container
21+
22+
```bash
23+
docker run --rm --entrypoint /bin/sh -it grafana/loki
24+
```
25+
26+
```bash
27+
/usr/bin/loki --help
28+
```

tinyloki-server/docker-compose.yaml renamed to integration-test-server/docker-compose.yaml

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,22 @@
1-
version: "3"
1+
version: "3.3"
2+
3+
# https://grafana.com/docs/loki/latest/setup/install/docker/
4+
# https://raw.githubusercontent.com/grafana/loki/v3.4.1/production/docker-compose.yaml
25

36
networks:
47
loki:
58

69
services:
710
loki:
8-
image: grafana/loki:2.9.2
11+
image: grafana/loki:3.4.2
912
ports:
1013
- "3100:3100"
1114
command: -config.file=/etc/loki/local-config.yaml
1215
networks:
1316
- loki
1417

1518
promtail:
16-
image: grafana/promtail:2.9.2
19+
image: grafana/promtail:latest
1720
volumes:
1821
- /var/log:/var/log
1922
command: -config.file=/etc/promtail/config.yml
@@ -25,6 +28,7 @@ services:
2528
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
2629
- GF_AUTH_ANONYMOUS_ENABLED=true
2730
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
31+
- GF_FEATURE_TOGGLES_ENABLE=alertingSimplifiedRouting,alertingQueryAndExpressionsStepMode
2832
entrypoint:
2933
- sh
3034
- -euc

‎src/main/java/pl/mjaron/tinyloki/ILogCollector.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,14 @@ public interface ILogCollector {
2121
*/
2222
void configureBufferingManager(IBuffering bufferingManager);
2323

24+
/**
25+
* If label settings are not <c>null</c>, the structured metadata is supported, else structured metadata should not be collected.
26+
*
27+
* @param structuredMetadataLabelSettings Settings for structured metadata validation.
28+
* @since 1.1.0
29+
*/
30+
void configureStructuredMetadata(LabelSettings structuredMetadataLabelSettings);
31+
2432
/**
2533
* Configures all values in single command.
2634
*

‎src/main/java/pl/mjaron/tinyloki/ILogStream.java‎

Lines changed: 34 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,53 @@
11
package pl.mjaron.tinyloki;
22

33
/**
4+
* Represents a Grafana Loki <a href="https://grafana.com/docs/loki/v3.4.x/get-started/overview/#loki-overview">stream</a>.
45
* Top-level log interface.
5-
* Single stream has fixed label values.
6+
* <p>
7+
* Single stream has fixed {@link Labels}.
8+
* <p>
9+
* Optionally, custom <a href="https://grafana.com/docs/loki/v3.4.x/get-started/labels/structured-metadata/">structured metadata</a> may be added to the particular log entry.
610
*/
711
public interface ILogStream {
812

913
/**
1014
* Thread-safe method used to write log messages to a stream.
1115
*
12-
* @param timestampMs Usually System.currentTimeMillis().
13-
* @param line Log content.
16+
* @param timestampMs Usually System.currentTimeMillis().
17+
* @param line Log content.
18+
* @param structuredMetadata The optional (nullable) {@link Labels} containing <a href="https://grafana.com/docs/loki/v3.4.x/get-started/labels/structured-metadata/">structured metadata</a> values.
19+
* @since 1.1.0
1420
*/
15-
void log(final long timestampMs, final String line);
21+
void log(final long timestampMs, final String line, final Labels structuredMetadata);
22+
23+
/**
24+
* Thread-safe log line with custom time.
25+
*
26+
* @param line Log content.
27+
* @since 1.1.0
28+
*/
29+
default void log(final long timestampMs, final String line) {
30+
log(timestampMs, line, null);
31+
}
1632

1733
/**
1834
* Thread-safe log line with current time.
1935
*
2036
* @param line Log content.
2137
*/
2238
default void log(final String line) {
23-
log(System.currentTimeMillis(), line);
39+
log(System.currentTimeMillis(), line, null);
40+
}
41+
42+
/**
43+
* Thread-safe log line with current time and structured metadata.
44+
*
45+
* @param line Log content.
46+
* @param structuredMetadata The optional (nullable) {@link Labels} containing <a href="https://grafana.com/docs/loki/v3.4.x/get-started/labels/structured-metadata/">structured metadata</a> values.
47+
* @since 1.1.0
48+
*/
49+
default void log(final String line, final Labels structuredMetadata) {
50+
log(System.currentTimeMillis(), line, structuredMetadata);
2451
}
2552

2653
/**
@@ -29,6 +56,8 @@ default void log(final String line) {
2956
* <p>
3057
* Note that if stream is released too fast after logging,
3158
* the logs may be not consumed by {@link IExecutor} and not sent to the server.
59+
* <p>
60+
* If needed, call {@link TinyLoki#sync()} to synchronize all logs occurring up to this point.
3261
*/
3362
void release();
3463
}

‎src/main/java/pl/mjaron/tinyloki/JsonLogCollector.java‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ public class JsonLogCollector implements ILogCollector {
1616
private int logEntriesCount = 0;
1717
private ILogListener logListener = null;
1818
private IBuffering bufferingManager = null;
19+
private LabelSettings structuredMetadataLabelSettings = null;
1920

2021
@Override
2122
public void configureLogListener(final ILogListener logListener) {
@@ -27,6 +28,21 @@ public void configureBufferingManager(final IBuffering bufferingManager) {
2728
this.bufferingManager = bufferingManager;
2829
}
2930

31+
@Override
32+
public void configureStructuredMetadata(LabelSettings structuredMetadataLabelSettings) {
33+
this.structuredMetadataLabelSettings = structuredMetadataLabelSettings;
34+
}
35+
36+
/**
37+
* Provides label settings for structured metadata.
38+
*
39+
* @return Label settings for structured metadata. It may be <c>null</c>.
40+
* @since 1.1.0
41+
*/
42+
public LabelSettings getStructuredMetadataLabelSettings() {
43+
return structuredMetadataLabelSettings;
44+
}
45+
3046
/**
3147
* Creates new instance of stream which will notify this collector about new logs.
3248
* This collector will syncAnd logs from the stream.
@@ -110,13 +126,13 @@ public String contentType() {
110126
return CONTENT_TYPE;
111127
}
112128

113-
synchronized void logImplementation(JsonLogStream stream, final long timestampMs, final String line) {
129+
synchronized void logImplementation(JsonLogStream stream, final long timestampMs, final String line, final Labels structuredMetadata) {
114130
final int logCandidateSize = 25 + line.length();
115131
if (!bufferingManager.beforeLog(logCandidateSize)) {
116132
return;
117133
}
118134

119-
final int acceptedLogSize = stream.logUnsafe(timestampMs, line);
135+
final int acceptedLogSize = stream.logUnsafe(timestampMs, line, structuredMetadata);
120136

121137
bufferingManager.logAccepted(acceptedLogSize);
122138
++logEntriesCount;

‎src/main/java/pl/mjaron/tinyloki/JsonLogStream.java‎

Lines changed: 33 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,14 @@
66
* Writes logs to JSON-formatted string.
77
*/
88
public class JsonLogStream implements ILogStream {
9+
10+
private final static String SEQUENCE_OPENING_STREAM = "{\"stream\":{";
11+
private final static String SEQUENCE_CLOSING_STREAM = "]}";
12+
913
private final JsonLogCollector collector;
10-
private final StringBuilder b = new StringBuilder("{\"stream\":{");
14+
private final StringBuilder b = new StringBuilder(SEQUENCE_OPENING_STREAM);
1115
private final String initialSequenceWithHeaders;
12-
private final String finalSequenceClosingStreams = "]}";
16+
1317
private int cachedLogsCount = 0; // Must be used in synchronized methods only.
1418

1519
/**
@@ -30,17 +34,15 @@ public JsonLogStream(final JsonLogCollector collector, final Labels labels) {
3034
}
3135
b.append('"');
3236
b.append(entry.getKey());
33-
b.append('"');
34-
b.append(':');
35-
b.append('"');
37+
b.append("\":\"");
3638
Utils.escapeJsonString(b, entry.getValue());
3739
b.append('"');
3840
}
3941
b.append("},\"values\":[");
4042
initialSequenceWithHeaders = b.toString();
4143
}
4244

43-
int logUnsafe(final long timestampMs, final String line) {
45+
int logUnsafe(final long timestampMs, final String line, final Labels structuredMetadata) {
4446
final int beforeSize = b.length();
4547
if (cachedLogsCount != 0) {
4648
b.append(',');
@@ -50,13 +52,34 @@ int logUnsafe(final long timestampMs, final String line) {
5052
b.append(timestampMs);
5153
b.append("000000\",\"");
5254
Utils.escapeJsonString(b, line);
53-
b.append("\"]");
55+
b.append('"');
56+
57+
if (structuredMetadata != null && collector.getStructuredMetadataLabelSettings() != null) {
58+
final Labels prettified = Labels.prettify(structuredMetadata, collector.getStructuredMetadataLabelSettings());
59+
b.append(",{");
60+
boolean isFirst = true;
61+
for (final Map.Entry<String, String> entry : prettified.getMap().entrySet()) {
62+
if (isFirst) {
63+
isFirst = false;
64+
} else {
65+
b.append(',');
66+
}
67+
b.append('"');
68+
b.append(entry.getKey());
69+
b.append("\":\"");
70+
Utils.escapeJsonString(b, entry.getValue());
71+
b.append('"');
72+
}
73+
b.append("}");
74+
}
75+
76+
b.append("]");
5477
return b.length() - beforeSize;
5578
}
5679

5780
@Override
58-
public void log(final long timestampMs, final String line) {
59-
collector.logImplementation(this, timestampMs, line);
81+
public void log(final long timestampMs, final String line, final Labels structuredMetadata) {
82+
collector.logImplementation(this, timestampMs, line, structuredMetadata);
6083
}
6184

6285
@Override
@@ -68,7 +91,7 @@ public void release() {
6891
* Appends JSON tags which closes streams array and JSON root object.
6992
*/
7093
public void closeStreamsEntryTag() {
71-
b.append(finalSequenceClosingStreams);
94+
b.append(SEQUENCE_CLOSING_STREAM);
7295
}
7396

7497
/**

‎src/main/java/pl/mjaron/tinyloki/LabelSettings.java‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,39 +3,51 @@
33
/**
44
* Defines parameters of label name and value.
55
* These parameters allow preparing values accepted by Grafana Loki server.
6+
* <p>
7+
* Based on: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
68
*
9+
* @see #DEFAULT_MAX_LABEL_NAME_LENGTH
10+
* @see #DEFAULT_MAX_LABEL_VALUE_LENGTH
711
* @since 0.3.0
812
*/
913
public class LabelSettings {
1014

1115
/**
12-
* Determines safe length of label name, based on default configuration of loki server.
13-
* Based on: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
16+
* Determines safe length of label name, based on default configuration of loki server at <c>limits_config</c>:
17+
* <pre>{@code
18+
* [max_label_name_length: <int> | default = 1024]
19+
* }</pre>
20+
* See: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
1421
*
1522
* @since 0.3.0
1623
*/
1724
public static final int DEFAULT_MAX_LABEL_NAME_LENGTH = 1024;
1825

1926
/**
20-
* Determines safe length of label value, based on default configuration of loki server.
21-
* Based on: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
27+
* Determines safe length of label value, based on default configuration of loki server at <c>limits_config</c>:
28+
* <pre>{@code
29+
* [max_label_value_length: <int> | default = 2048]
30+
* }</pre>
31+
* See: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
2232
*
2333
* @since 0.3.0
2434
*/
2535
public static final int DEFAULT_MAX_LABEL_VALUE_LENGTH = 2048;
2636

2737
/**
2838
* Determines accepted length of added label name, based on default configuration of loki server.
29-
* Based on: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
39+
* See: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
3040
*
41+
* @see #DEFAULT_MAX_LABEL_NAME_LENGTH
3142
* @since 0.3.0
3243
*/
3344
private int maxLabelNameLength = DEFAULT_MAX_LABEL_NAME_LENGTH;
3445

3546
/**
3647
* Determines accepted length of added label value, based on default configuration of loki server.
37-
* Based on: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
48+
* See: <a href="https://grafana.com/docs/loki/latest/configuration/">Grafana Loki limits_config</a>
3849
*
50+
* @see #DEFAULT_MAX_LABEL_VALUE_LENGTH
3951
* @since 0.3.0
4052
*/
4153
private int maxLabelValueLength = DEFAULT_MAX_LABEL_VALUE_LENGTH;

0 commit comments

Comments
 (0)